Title: Factorized Tensor Networks for Multi-Task and Multi-Domain Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: multi-task learning

multi-domain learning

low-rank tensors

multipath networks
Abstract: Multi-task and multi-domain learning methods seek to learn multiple tasks/domains, jointly or one after another, using a single unified network. The key challenge and opportunity is to exploit shared information across tasks and domains to improve the efficiency of the unified network. The efficiency can be in terms of accuracy, storage cost, computation, or sample complexity. In this paper, we propose a factorized tensor network (FTN) that can achieve accuracy comparable to independent single-task/domain networks with a small number of additional parameters. FTN uses a frozen backbone network from a source model and incrementally adds task/domain-specific low-rank tensor factors to the shared frozen network. This approach can adapt to a large number of target domains and tasks without catastrophic forgetting. Furthermore, FTN requires a significantly smaller number of task-specific parameters compared to existing methods. We performed experiments on widely used multi-domain and multi-task datasets. We observed that FTN achieves similar accuracy as single-task/domain methods while using 2--6\% additional parameters per task. We also demonstrate the effectiveness of FTN with domain adaptation for image generation.

Abstract

Multi-task and multi-domain learning methods seek to learn multiple tasks/domains, jointly or one after another, using a single unified network. The key challenge and opportunity is to exploit shared information across tasks and domains to improve the efficiency of the unified network. The efficiency can be in terms of accuracy, storage cost, computation, or sample complexity. In this paper, we propose a factorized tensor network (FTN) that can achieve accuracy comparable to independent single-task/domain networks with a small number of additional parameters. FTN uses a frozen backbone network from a source model and incrementally adds task/domain-specific low-rank tensor factors to the shared frozen network. This approach can adapt to a large number of target domains and tasks without catastrophic forgetting. Furthermore, FTN requires a significantly smaller number of task-specific parameters compared to existing methods. We performed experiments on widely used multi-domain and multi-task datasets. We observed that FTN achieves similar accuracy as single-task/domain methods while using 2-6% additional parameters per task. We also demonstrate the effectiveness of FTN with domain adaptation for image generation.

1 Introduction

The primary objective in multi-task learning (MTL) is to train a single model to learn multiple related tasks, either jointly or sequentially. Multi-domain learning (MDL) aims to achieve the same learning objective across multiple domains. MTL and MDL techniques seek to improve overall performance by leveraging shared information across multiple tasks and domains. On the other hand, single-task or single-domain learning do not have that opportunity. Furthermore, the storage and computational cost associated with single-task/domain models quickly grows as the number of tasks/domains increases. In contrast, MTL and MDL methods can use the same network resources for multiple tasks/domains, which keeps the overall computational and storage cost small [1-10].

In general, MTL and MDL can have different input/output configurations, but we model them as task/domain-specific network representation problems. Let us represent a network for MTL or MDL as the following general function:

where F t represents a function for task/domain t that maps input x to output y t . We further assume 29 that F represents a network with a fixed architecture and W t and h t represent the parameters for 30 task/domain-specific feature extraction and classification/inference heads, respectively. The function 31 in (1) can represent the network for specific task/domain t using the respective W t , h t . In the case 32 of MTL, with T tasks, we can have T outputs y 1 , . . . , y T for a given input x . In the case of MDL, 33 we usually have a single output for a given input, conditioned on the domain t . Our main goal is to 34

35

Layer 1

Layer 2

Head 1

Input 1

Output 1

Layer 1

Layer 2

Head 2

Input 2

Output 2

(b) Feature-Extractor

Head 1

Output 1

Head 2

Input 1

Output 2

Shared

shared

(c) Factorized Tensor Network (FTN)

Head 1

Head 2

Input 1

shared

shared

1

1

2

2

Frozen

backbone

weight

Low-rank tensor

BN

W l

Relu

C out

C in

k 2

k 2

C in

C out

rank,1

rank,r

C out

C in

k 2

k 2

C in

C out

rank,1

rank,r

C out

C in

k 2

k 2

C in

C out

rank,1

rank,r

Frozen

backbone

weight

Low-rank tensor

W l+1

C out

C in

k 2

k 2

C in

C out

rank,1

rank,r

C out

C in

k 2

k 2

C in

C out

rank,1

rank,r

C out

C in

k 2

k 2

C in

C out

rank,1

rank,r

Head

Output 1

Output 2

Low-rank tensors

for task 1

Low-rank tensors

for task 2

(a) Fine-Tuning

(d) Detailed overview of FTN architecture.

learn the

W

t

, h

t

for all

t

that maximize the performance of MTL/MDL with minimal computation

and memory overhead compared to single task/domain learning. 36

Figure 1(a),(b),(c) illustrate three typical approaches for MTL/MDL. First, we can start with a pre37

trained network and fine-tune all the parameters ( W t ) to learn a target task/domain, as shown in 38

Figure 1(a). Fine-Tuning approaches can transfer some knowledge from the pretrained network to the 39

target task/domain, but they effectively use an independent network for every task/domain [1, 5, 1140

14]. Second, we can reduce the parameter and computation complexity by using a completely 41

shared Feature-Extractor (i.e., W t = W shared for all t ) and learning task/domain-specific heads as 42

last layers, as shown in Figure 1(b). While such approaches reduce the number of parameters, they 43

often result in poor overall performance because of limited network capacity and interference among 44

features for different tasks/domains [1, 4, 5, 15]. Third, we can divide the network into shared and 45

task/domain-specific parameters or pathways, as shown in Figure 1(c). Such an approach can increase 46

the network capacity, provide interference-free paths for task/domain-specific feature extraction, and 47

enable knowledge sharing across the tasks/domains. In recent years, a number of such methods 48

have been proposed for MTL/MDL [1, 4, 5, 9, 16-22]. While existing methods provide performance 49

comparable to single-task/domain learning, they require a significantly large number of parameters. 50

In this paper, we propose a new method to divide network into shared and task/domain-specific 51

parameters using a factorized tensor network (FTN). In particular, our method learns task/domain52

specific low-rank tensor factors and normalization layers. An illustration of our proposed method is 53

shown in Figure 1(d), where we represent network parameters as W t = W shared +∆ W t , where ∆ W t 54

is a low-rank tensor. Furthermore, we also learn task/domain-specific normalization parameters. We 55

demonstrate the effectiveness of our method using different MTL and MDL datasets. Our method 56

can achieve accuracy comparable to a single-task/domain network with a small number of additional 57

parameters. Existing parameter-efficient MTL/MDL methods [1, 2, 23] introduce small task/domain58

specific parameters while others [15, 24] add many parameters to boost the performance irrespective 59

of the task complexity. In our work, we demonstrate the flexibility of FTNs by selecting the rank 60

according to the complexity of the task. Finally, we also present an experiment for multi-domain 61

image generation using FTNs. 62

Contributions. The main contributions of this paper can be summarized as follows. 63

64

65

We propose a new method for MTL and MDL, called factorized tensor networks (FTN), that adds task/domain-specific low-rank tensors to shared weights.

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

We demonstrate that by using as little as 2-6% additional parameters per task/domain, FTNs can achieve similar performance as the single-task/domain methods.

Our proposed FTNs can be viewed as a plug-in module that can be added to any pretrained network and layer.

We performed empirical analysis to show that the FTNs enable flexibility by allowing us to vary the rank of the task-specific tensors based on the complexity of the problem.

Limitations. Our proposed method requires a small memory overhead to represent the MTL/MDL networks compared to the single task/domain networks. The proposed method does not affect the computational cost because we need to compute features for each task/domain using separate functional pathways. In our experiments, we used a fixed rank for each layer. In principle, we can adaptively select the rank for different layers to further reduce the parameters. MTL/MDL models often suffer from task interference or negative transfer learning when multiple conflicting tasks are trained jointly. Our method can have similar drawbacks as we did not investigate which tasks/domains should be learned jointly.

2 Related Work

Multi-task learning (MTL) methods commonly leverage shared and task-specific layers in a unified network to solve related tasks [16-18, 25-32]. These methods learn shared and task-specific representation through their respective modules. Optimization based methods [33-37] devise a principled way to evaluate gradients and losses in multi-task settings. MTL networks that incrementally learn new taks were proposed in [9, 10]. ASTMT [10] proposed a network that emphasizes or suppresses features depending on the task at hand. RCM [9] reparameterizes the convolutional layer into non-trainable and task-specific trainable modules. We compare our proposed method with these incrementally learned networks. Adashare [8] is another related work in MTL that jointly learns multiple tasks. It learns task-specific polices and network pathways [38].

Multi-domain learning (MDL) focuses on adapting one network to multiple unseen domains or tasks. MDL setup trains models on task-specific modules built upon the frozen backbone network. This setup helps MDL networks to avoid negative transfer learning or catastrophic forgetting, which is common among multi-task learning methods. The work by [3, 6] introduces the task-specific parameters called residual adapters. The architecture introduces these adapters as a series or parallel connection on the backbone for a downstream task. Inspired by pruning techniques, Packnet [2] learns on multiple domains sequentially on a single task to decrease the overhead storage, which comes at the cost of performance. Similarly, the Piggyback [1] method uses binary masks as the module for task-specific parameters. These masks are applied to the weights of the backbone to adapt them to new domains. To extend this work, WTPB [7] uses the affine transformations of the binary mask on their backbone to extend the flexibility for better learning. BA 2 [4] proposed a budget-constrained MDL network that selects the feature channels in the convolutional layer. It gives parameter efficient network by dropping the feature channels based on budget but at the cost of performance. Spot-Tune [24] learns a policy network, which decides whether to pass each image through Fine-Tuning or pre-trained networks. It neglects the parameter efficiency factor and emphasises more on performance. TAPS [5] adaptively learns to change a small number of layers in pre-trained backbone network for the downstream task.

Our proposed method, FTN, achieves performance comparable to or better than other methods by utilizing a fraction of the parameters. We demonstrated that, unlike other methods, easy domains do not require the transformation of the backbone by the same complex module, and we can choose the flexibility of the task-specific parameter for a given domain.

Domain adaptation and transfer learning. The work in this field usually focuses on learning a network from a given source domain to a closely related target domain. The target domains under this kind of learning typically have the same category of classes as source domains [11]. Due to this, it benefits from exploiting the labels of source domains to learn about multiple related target domains[12, 39]. Some work has a slight domain shift between source and target data like different

camera views [40]. At the same time, recent papers have worked on large domain shifts like converting 116 targets into sketch or art domains [12, 41]. Transfer learning is related to MDL or domain adaptation, 117 but focuses on how to generalize better on target tasks [13, 14, 42]. Most of the work in this field 118 uses the popular Imagenet as a source dataset to learn feature representation and learn to transfer to 119 target datasets. 120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

3 Technical Details

In our proposed method, we use task/domain-specific low-rank tensors to adapt every convolutional layer of a pretrained backbone network to new tasks and domains. Let us assume the backbone network has L convolution layers that are shared across all task/domains. We represent the shared network weights as W shared = { W 1 , . . . , W L } and the low-rank network updates for task/domain t as ∆ W t = { ∆ W 1 ,t , . . . , ∆ W L,t } . To compute features for task/domain t , we update weights at every layer as W shared +∆ W t = { W 1 +∆ W 1 ,t , . . . , W L +∆ W L,t } .

To keep our notations simple, let us only consider l th convolution layer that has k × k filters, C in channels for input feature tensor, and C out channels for output feature tensor. We represent the corresponding W l as a tensor of size k 2 × C in × C out . We represent the low-rank tensor update as a summation of R rank-1 tensors as

where w r 1 ,t , w r 2 ,t , w r 3 ,t represent vectors of length k 2 , C in , C out , respectively, and ⊗ represents the Kronecker product.

Apart from low-rank tensor update, we also optimize over batchnorm layers (BN) for each task/domain [43, 44]. The BN layer learns two vectors Γ and β , each of length C out . The BN operation along C out dimension can be defined as element-wise multiplication and addition:

We represent the output of l th convolution layer for task/domain t as

where Y l -1 ,t represents the input tensor and Z l,t represents the output tensor for l th layer. In our proposed FTN, we learn the task/domain-specific factors { w r 1 ,t , w r 2 ,t , w r 3 ,t } R r =1 , and Γ t , and β t for every layer in the backbone network.

In the FTN method, since we are learning over only ∆ W and BN parameters, the rank, R , plays an important role in defining the expressivity of our network. We can define a complex ∆ W by increasing the rank R of the low-rank tensor and taking their linear combination. Our experiments showed that this has resulted in a significant performance gain.

Initialization. In our approach, the initialization of the low-rank parameter layers and the pre-trained weights of the backbone network plays a crucial role due to their sensitivity towards performance. To establish a favorable starting point, we adopt a strategy that minimizes substantial modifications to the frozen backbone network weights during the initialization of the task-specific parameter layers. To achieve this, we initialize each parameter layer from the xavier uniform distribution [45], thereby generating ∆ W values close to 0 before their addition to the frozen weights. This approach ensures the maintenance of a similar initialization state to the frozen weights at iteration 0.

To acquire an effective initialization for our backbone network, we leverage the pre-trained weights obtained from ImageNet. We aim to establish a robust and capable feature extractor for our specific task by incorporating these pre-trained weights into our backbone network.

Number of parameters. In a Fine-Tuning setup with T tasks/domains, the total number of required parameters at convolutional layer l can be calculated as T · ( k 2 × C in × C out ) . Whereas using our proposed FTNs, the total number of frozen backbone ( W l ) and low-rank R tensor (∆ W l,t ) parameters are given by ( C out × C in × k 2 ) + T · R · ( C out + C in + k 2 ) . In our results section, we

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

have shown that the absolute number of parameters required by our method is a fraction of what the Fine-Tuning counterpart needs.

Effect of batch normalization. In our experiment section, under the 'FC and BN only' setup, we have shown that having task-specific batchnorm layers in the backbone network significantly affects the performance of a downstream task/domain. For all the experiments with our proposed approach, we include batch normalization layers as task-specific along with low-rank tensors and classification/decoder layer.

4 Experiments and Results

We evaluated the performance of our proposed FTN on several MTL/MDL datasets for three different experiments: 1. Multi-domain classification , 2. Multi-task dense prediction , and 3. Multi-domain image generation . For each set of benchmarks we report the performance of FTN with different rank increments and compare with results from existing methods. All experiments are run on a single NVIDIA GeForce RTX 2080 Ti GPU with 12GB memory.

4.1 Multi-domain classification

Datasets. We use two MTL/MDL classification-based benchmark datasets. First, ImageNet-toSketch [1, 2, 5, 7], which contains five different domains: Flowers [46], Cars [47], Sketch [48], Caltech-UCSD Birds (CUBs) [49], and WikiArt [50], with different classes. Second, DomainNet [51], which contains six domains: Clipart, Sketch, Painting (Paint), Quickdraw (Quick), Inforgraph (Info), and Real, with each domain containing an equal 345 classes. The datasets are prepared using augmentation techniques as adopted by [5].

Training details. For each benchmark, we report the performance of FTN for various choices for ranks, along with several benchmark-specific comparative and baseline methods. The backbone weights are pretrained from ImageNet, using ResNet-50 [52] for the ImageNet-to-Sketch benchmarks, and ResNet-34 on the DomainNet benchmarks to keep the same setting as [5]. On ImageNet-toSketch we run FTNs for ranks, R ∈ { 1 , 5 , 10 , 15 , 20 , 25 , 50 } and on DomainNet dataset for ranks, R ∈ { 1 , 5 , 10 , 20 , 30 , 40 } . In the supplementary material, we provide the hyperparameter details to train our network.

Results. We report the top-1% accuracy for each domain and the mean accuracy across all domains for each collection of benchmark experiments. We also report the number of frozen and learnable parameters in the backbone network. Table 1 compares the FTN method with other methods in terms of accuracy and number of parameters (also see Figure 2). FTN outperforms every other method while using 36.02 million parameters in the backbone with rank-50 updates for all domains. The

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

mean accuracy performance is better than other methods and is close to Spot-Tune [24], which requires nearly 165M parameters. On the Wikiart dataset, we outperform the top-1 accuracy with other baseline methods. The performance of baseline methods is taken from TAPS [5] since we are running the experiments under the same settings.

Table 2 shows the results on the DomainNet dataset, which we compare with TAPS [5] and Adashare [8]. Again, using FTN, we significantly outperform comparison methods along the required parameters (rank-40 needs 25.22 million parameters only). Also, FTN rank-40 attains better top-1% accuracy on the Infograph and Real domain, while it attains similar performance on all other domains. On DomainNet with resnet-34 and Imagenet-to-Sketch with resnet-50 backbone, the rank-1 low-rank tensors require only 16,291 and 49,204 parameters per task, respectively. We have shown additional experiments on this dataset under a joint optimization setup in the supplementary material.

1

2

3

4

5

6

7

Parameter multiplier with baseline

75

80

85

90

95

Top-1 accuracy [%]

mean

Flowers

Wikiart

Sketch

Cars

CUBS

Spot-Tune

WTPB

BA2

TAPS

FTN, R=1

FTN, R=25

FTN, R=50

(a)

Imagenet-to-sketch dataset

1

2

3

4

5

Parameter multiplier with baseline

40

50

60

70

80

Top-1 accuracy [%]

mean

Sketch

Painting

Quickdraw

Infograph

Real

Adashare

TAPS

FTN, R=1

FTN, R=40

(b)

DomainNet dataset

Analysis on rank. We showed the effect of rank on FTNs by performing experiments with multiple ranks on both datasets. Figure 3 shows the accuracy vs. ranks plot, where we observe a trend of performance improvement as we increase the rank from 1 to 50 on the ImageNet-to-Sketch and from 1 to 40 on the DomainNet dataset. Also, not all domains need a high rank, as Figure 3 shows that the Flowers and Cars domain attains good accuracy at rank 20 and 15, respectively. We can argue that, unlike prior works [24, 23], which consume the same task-specific module for easy and complex tasks, we can provide different flexibility to each task. Also, in supplementary material, we have a heatmap plot showing the adaption of low-rank tensor at every layer upon increasing the rank.

4.2 Multi-task dense prediction

Dataset. The widely-used NYUD dataset [53] with 795 training and 654 testing images of indoor scenes is used for dense prediction experiments in multi-task learning. The dataset contains four tasks: edge detection (Edge), semantic segmentation (SemSeg), surface normals estimation (Normals), and depth estimation (Depth). We follow the same data-augmentation technique as used by [9].

Metrics. On the tasks of the NYUD dataset, we report mean intersection over union for semantic segmentation, mean error for surface normal estimation, optimal dataset F-measure [54] for edge

217

218

219

220

221

222

223

224

225

226

227

228

229

R=1 R=5

R=50

Differnet ranks

60

65

70

75

80

85

90

95

Top-1 accuracy [%]

Flower

Wikiart

Sketch

Cars

CUBS

Fine-Tuning

(a) Imagenet-to-sketch dataset

R=1

R=5

R=30

R=40

Differnet ranks

10

20

30

40

50

60

70

80

Top-1 accuracy [%]

Sketch

Painting

Quickdraw

Infograph

Real

Fine-Tuning

(b) DomainNet dataset

100

detection, and root mean squared error for depth estimation. We also report the number of parameters used in the backbone for each method.

Training details. ResNet-18 is used as the backbone network, and DeepLabv3+ [55] as the decoder architecture. The Fine-Tuning and Feature-Extractor experiments are implemented in the same way as in the classification-based experiments above. We showed experiments for FTNs with R ∈ { 1 , 10 , 20 , 30 } . Further details are in the supplementary material.

Results. Table 3 shows the performance of FTN with various ranks and of other baseline comparison methods for dense prediction tasks on the NYUD dataset. We observe performance improvement by increasing flexibility through higher rank. FTN with rank-30 performs better than all comparison methods and utilizes the least number of parameters. Also, on the Depth and Edge task we can attain good performance by using only rank-20. We take the performance of baseline comparison methods from the RCM paper [9] as we run our experiments under the same setting.

4.3 Multi-domain image generation

A deep generative network G , parameterized by W , can learn to map a low-dimensional latent code 230 z to a high-dimensional natural image x [56-58]. To find a latent representation for a set of images 231 given a pre-trained generative network, we can solve the following optimization problem: 232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

Summer

(original)

Summer

(generated)

Winter

(generated)

Spring

(generated)

Autumn

(generated)

The work in [58] as well as our experimental results show that this approach is very limited in handling complex and diverse images. If x is an image that belongs to a domain that is different from the source domain used to train the generator, we are not guaranteed to find a latent vector z ∗ such that x ≈ G ( z ∗ ) . We showed FTNs can be used to expand the range of G by reparametrizing it as G ( z i ; W , ∆ W t ) , where ∆ W t are the domain-specific low-rank factors and Batch Normalization parameters. By optimizing over the latent vectors z i and domain specific parameters, we learned to generate images from new domains.

Dataset. We used the multi-domain Transient Attributes [59] dataset that contains outdoor scenes under different weather, lighting, and seasons. We extracted season information, "summer", "winter", "spring", and "autumn", for each image and categorized them accordingly. Our goal is to learn a single FTN network that can generate images from all seasons.

Training details. Our base network follows the BigGAN architecture [60]. We compared our proposed FTN network with models trained under two different setups. In the first setup, which serves as our baseline, we fine-tuned a pre-trained generator on images from all seasons. For the second setup, we fine-tuned the same network separately for each season. Additional training details can be found in the supplementary material.

Results. Table 4 shows the performance, in terms of Peak Signal-to-Noise Ratio (PSNR), for three methods. The baseline model is a single network trained on all images and shows overall poor performance. Our proposed FTN network achieved a comparable performance to the single-domain networks. Each single domain network has 71 . 4 Mtrainable parameters, while the FTN network adds an additional 3 . 9 M

parameters per domain over the base network. Figure 4 shows examples of images generated by our proposed FTN network.

5 Conclusion

We have proposed a simple, architecture-agnostic, and easy-to-implement FTN method that adapts 262 to new unseen domains/tasks by using low-rank task-specific tensors. In our work, we have shown 263 FTN requires the least number of parameters than other baseline methods in MDL/MTL experiments 264 and attains better or comparable performance. We can adapt the backbone network with different 265

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

References

A. Mallya, D. Davis, and S. Lazebnik, 'Piggyback: Adapting a single network to multiple tasks by learning to mask weights,' in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 67-82.

A. Mallya and S. Lazebnik, 'Packnet: Adding multiple tasks to a single network by iterative pruning,' in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , 2018, pp. 7765-7773.

S.-A. Rebuffi, H. Bilen, and A. Vedaldi, 'Learning multiple visual domains with residual adapters,' Advances in neural information processing systems , vol. 30, 2017.

R. Berriel, S. Lathuillere, M. Nabi, T. Klein, T. Oliveira-Santos, N. Sebe, and E. Ricci, 'Budgetaware adapters for multi-domain learning,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 382-391.

M. Wallingford, H. Li, A. Achille, A. Ravichandran, C. Fowlkes, R. Bhotika, and S. Soatto, 'Task adaptive parameter sharing for multi-task learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 7561-7570.

S.-A. Rebuffi, H. Bilen, and A. Vedaldi, 'Efficient parametrization of multi-domain deep neural networks,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 8119-8127.

M. Mancini, E. Ricci, B. Caputo, and S. Rota Bulo, 'Adding new tasks to a single network with weight transformations using binary masks,' in Proceedings of the European Conference on Computer Vision (ECCV) Workshops , 2018, pp. 0-0.

X. Sun, R. Panda, R. Feris, and K. Saenko, 'Adashare: Learning what to share for efficient deep multi-task learning,' Advances in Neural Information Processing Systems , vol. 33, pp. 8728-8740, 2020.

M. Kanakis, D. Bruggemann, S. Saha, S. Georgoulis, A. Obukhov, and L. V. Gool, 'Reparameterizing convolutions for incremental multi-task learning without task interference,' in European Conference on Computer Vision . Springer, 2020, pp. 689-707.

K.-K. Maninis, I. Radosavovic, and I. Kokkinos, 'Attentive single-tasking of multiple tasks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 1851-1860.

E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, 'Adversarial discriminative domain adaptation,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 7167-7176.

H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, 'Deep hashing network for unsupervised domain adaptation,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 5018-5027.

B. Mustafa, A. Loh, J. Freyberg, P. MacWilliams, M. Wilson, S. M. McKinney, M. Sieniek, J. Winkens, Y. Liu, P. Bui et al. , 'Supervised transfer learning at scale for medical imaging,' arXiv preprint arXiv:2101.05913 , 2021.

A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, 'Big transfer (bit): General visual representation learning,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16 . Springer, 2020, pp. 491-507.

J. O. Zhang, A. Sax, A. Zamir, L. Guibas, and J. Malik, 'Side-tuning: a baseline for network adaptation via additive side networks,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 . Springer, 2020, pp. 698-714.

I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, 'Cross-stitch networks for multi-task 316 learning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 317 2016, pp. 3994-4003. 318

319

320

321

S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard, 'Latent multi-task architecture learning,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, no. 01, 2019, pp. 4822-4829.

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

Y. Gao, J. Ma, M. Zhao, W. Liu, and A. L. Yuille, 'Nddr-cnn: Layerwise feature fusing in multitask cnns by neural discriminative dimensionality reduction,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 3205-3214.

G. Strezoski, N. v. Noord, and M. Worring, 'Many task learning with task routing,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 1375-1384.

J. Liang, E. Meyerson, and R. Miikkulainen, 'Evolutionary architecture search for deep multitask networks,' in Proceedings of the genetic and evolutionary computation conference , 2018, pp. 466-473.

Y. Gao, H. Bai, Z. Jie, J. Ma, K. Jia, and W. Liu, 'Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning,' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, 'Gradient surgery for multitask learning,' Advances in Neural Information Processing Systems , vol. 33, pp. 5824-5836, 2020.

Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou, 'Revisiting batch normalization for practical domain adaptation,' arXiv preprint arXiv:1603.04779 , 2016.

Y. Guo, H. Shi, A. Kumar, K. Grauman, T. Rosing, and R. Feris, 'Spottune: transfer learning through adaptive fine-tuning,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 4805-4814.

S. Liu, E. Johns, and A. J. Davison, 'End-to-end multi-task learning with attention,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 1871-1880.

S. Vandenhende, S. Georgoulis, B. De Brabandere, and L. Van Gool, 'Branched multi-task networks: Deciding what layers to share,' Proceedings BMVC , 20202.

D. Bruggemann, M. Kanakis, S. Georgoulis, and L. Van Gool, 'Automated search for resourceefficient branched multi-task networks,' Proceedings BMVC , 2020.

P. Guo, C.-Y. Lee, and D. Ulbricht, 'Learning to branch for multi-task learning,' in International Conference on Machine Learning . PMLR, 2020, pp. 3854-3863.

D. Xu, W. Ouyang, X. Wang, and N. Sebe, 'Pad-net: Multi-tasks guided prediction-anddistillation network for simultaneous depth estimation and scene parsing,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 675-684.

Z. Zhang, Z. Cui, C. Xu, Y. Yan, N. Sebe, and J. Yang, 'Pattern-affinitive propagation across depth, surface normal and semantic segmentation,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 4106-4115.

Z. Zhang, Z. Cui, C. Xu, Z. Jie, X. Li, and J. Yang, 'Joint task-recursive learning for semantic segmentation and depth estimation,' in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 235-251.

S. Vandenhende, S. Georgoulis, and L. V. Gool, 'Mti-net: Multi-scale task interaction networks 359 for multi-task learning,' in European Conference on Computer Vision . Springer, 2020, pp. 360 527-543. 361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, 'Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,' in International conference on machine learning . PMLR, 2018, pp. 794-803.

A. Kendall, Y. Gal, and R. Cipolla, 'Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 7482-7491.

M. Guo, A. Haque, D.-A. Huang, S. Yeung, and L. Fei-Fei, 'Dynamic task prioritization for multitask learning,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 270-287.

X. Lin, H.-L. Zhen, Z. Li, Q.-F. Zhang, and S. Kwong, 'Pareto multi-task learning,' Advances in neural information processing systems , vol. 32, 2019.

Z. Chen, J. Ngiam, Y. Huang, T. Luong, H. Kretzschmar, Y. Chai, and D. Anguelov, 'Just pick a sign: Optimizing deep multitask models with gradient sign dropout,' Advances in Neural Information Processing Systems , vol. 33, pp. 2039-2050, 2020.

E. Jang, S. Gu, and B. Poole, 'Categorical reparameterization with gumbel-softmax,' in International Conference on Learning Representations , 2017.

K. Li, C. Liu, H. Zhao, Y. Zhang, and Y. Fu, 'Ecacl: A holistic framework for semi-supervised domain adaptation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 8578-8587.

K. Saenko, B. Kulis, M. Fritz, and T. Darrell, 'Adapting visual category models to new domains,' in European Conference on Computer Vision (ECCV) . Springer, 2010, pp. 213-226.

Y. Zhao, H. Ali, and R. Vidal, 'Stretching domain adaptation: How far is too far?' arXiv preprint arXiv:1712.02286 , 2017.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, 'An image is worth 16x16 words: Transformers for image recognition at scale,' International Conference on Learning Representations , 2021.

S. Ioffe and C. Szegedy, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift,' in International conference on machine learning . pmlr, 2015, pp. 448-456.

Q. Pham, C. Liu, and H. Steven, 'Continual normalization: Rethinking batch normalization for online continual learning,' in International Conference on Learning Representations , 2022.

X. Glorot and Y. Bengio, 'Understanding the difficulty of training deep feedforward neural networks,' in Proceedings of the thirteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2010, pp. 249-256.

M.-E. Nilsback and A. Zisserman, 'Automated flower classification over a large number of classes,' in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing . IEEE, 2008, pp. 722-729.

J. Krause, M. Stark, J. Deng, and L. Fei-Fei, '3d object representations for fine-grained categorization,' in Proceedings of the IEEE international conference on computer vision workshops , 2013, pp. 554-561.

M. Eitz, J. Hays, and M. Alexa, 'How do humans sketch objects?' ACM Transactions on graphics (TOG) , vol. 31, no. 4, pp. 1-10, 2012.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, 'The caltech-ucsd birds-200-2011 dataset,' 2011.

B. Saleh and A. Elgammal, 'Large-scale classification of fine-art paintings: Learning the right 407 metric on the right feature,' International Journal for Digital Art History , no. 2, 2016. 408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, 'Moment matching for multi-source domain adaptation,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 1406-1415.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.

N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, 'Indoor segmentation and support inference from rgbd images.' ECCV (5) , vol. 7576, pp. 746-760, 2012.

D. R. Martin, C. C. Fowlkes, and J. Malik, 'Learning to detect natural image boundaries using local brightness, color, and texture cues,' IEEE transactions on pattern analysis and machine intelligence , vol. 26, no. 5, pp. 530-549, 2004.

L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, 'Encoder-decoder with atrous separable convolution for semantic image segmentation,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 801-818.

D. Ulyanov, A. Vedaldi, and V. Lempitsky, 'Deep image prior,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 9446-9454.

J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros, 'Generative visual manipulation on the natural image manifold,' in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 2016, pp. 597-613.

X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, and P. Luo, 'Exploiting deep generative prior for versatile image restoration and manipulation,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 11, pp. 7474-7489, 2021.

P.-Y. Laffont, Z. Ren, X. Tao, C. Qian, and J. Hays, 'Transient attributes for high-level understanding and editing of outdoor scenes,' ACM Transactions on Graphics (proceedings of SIGGRAPH) , vol. 33, no. 4, 2014.

A. Brock, J. Donahue, and K. Simonyan, 'Large scale gan training for high fidelity natural image synthesis,' in International Conference on Learning Representations , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a factorized tensor network (FTN), that adds task/domain-specific low-rank tensors to shared weights, to perform lightweight MTL/MDL. Specifically, the low-rank tensors can form a weight additive to the shared backbone weight so that each task can have its own parameters to achieve good accuracy.\n\nSTRENGTHS:\n- The paper is easy to follow.\n- The basic idea of adding additional task-specific low-rank vectors is clear to understand.\n\nWEAKNESSES:\n- About the method:\n    - My high-level concern about this idea is not truly an MTL/MDL work in my understanding.   \n    If I understand the proposed method correctly, it is equivalent to fine-tuning a completely shared feature extractor on each task. After fine-tuning, we can get the weights offset $\\Delta W = W_t - W$ between the fine-tuned weights $W_t$ for task $t$ and the original backbone weights $W$. The low-rank tensors are just a parameter-efficient way to store $\\Delta W$ by three separate vectors.\n    What I believe is that the proposed method is still a traditional transfer learning method with some low-rank storage design for the weight difference.\n\n    - If it is an MTL work, it fails to obtain the two basic advantages of using MTL.\n        - First, in MTL, we believe a good parameters-sharing may improve the generalization ability, i.e., the task accuracy, when training multiple tasks jointly. However, in this paper, the shared backbone weights will be frozen (as shown in Figure 1d) when training the additional low-rank tensors for each task, which means there is no truly joint training/parameter sharing across the multiple tasks. As I described in the first bullet point, each task is just fine-tuned from the shared backbone and has no communication with other tasks during training, and the final model weights for each task will be independent to each other at the end.\n        - Second, the wide adaptation of MTL relies on its advantage of lower computation cost and latency. However, the proposed method in this paper has no latency saving since all the tasks still have their own parameters and have to execute sequentially even if they share the same input images.\n\n    - About the method itself, I feel it lacks good motivation. There is no reason why we can have three low-rank tensors to get $\\Delta W$ for each task in each layer. The authors may consider adding some literature that can support this choice, or have some ablation studies. For example, what will happen if we use the full-rank $\\Delta W$ directly? How about using PCA or SVD on $\\Delta W$ to get low-rank matrics?\n    \n    - More importantly, this paper has a very similar idea as [1]. The only difference is the choice of low-rank representation.    \n    [1] Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[J]. arXiv preprint arXiv:2106.09685, 2021.\n\n - About the literature:\n    - Missing papers and comparisons with adaptor-based MTL methods. Adaptor-based MTL is very close to this paper, in which task-specific adaptors are parameter-efficient and plug-in modules on top of an all-shared feature extractor. For examples,    \n    [2] Rosenfeld, Amir, and John K. Tsotsos. \"Incremental learning through deep adaptation.\" IEEE transactions on pattern analysis and machine intelligence 42.3 (2018): 651-663.     \n    [3] Zhao, Hanbin, et al. \"What and where: Learn to plug adapters via nas for multidomain learning.\" IEEE Transactions on Neural Networks and Learning Systems 33.11 (2021): 6532-6544.\n\n    - Missing recent papers in MTL/MDL.    \n    [4] Zhang, Lijun, et al. \"Rethinking hard-parameter sharing in multi-domain learning.\" 2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2022.     \n    [5] Zhang, Lijun, Xiao Liu, and Hui Guan. \"Automtl: A programming framework for automating efficient multi-task learning.\" Advances in Neural Information Processing Systems 35 (2022): 34216-34228.\n    \n - About the experiments:\n    - The authors may want to compare with adaptor-based MTL methods, which are the most related works to this paper.\n    - It's necessary to have more experiments for MTL, including more baselines, more backbone models, and more datasets. More baselines could be the representative and new methods like MTAN and AutoMTL, rather than just RA and ASTMT. Backbone models could be some efficient backbone like MobileNet or EfficientNet, rather than just ResNet variants. Datasets could be CityScapes and Taskonomy, rather than just NYUD.\n    - AdaShare is a typical MTL model. It's weird to compare it in MDL setting only.\n\nQUESTIONS:\nMy main concerns are in the weakness part. Thanks.\n\nLIMITATIONS:\nThe social impact is not mentioned either in the main paper or the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the Factorized Tensor Network (FTN) as a solution to the challenge of learning multiple tasks/domains using a single unified network. The authors claim that FTN achieves comparable accuracy to independent single-task/domain networks with a smaller number of additional parameters. The method incorporates task/domain-specific low-rank tensor factors into a shared frozen backbone network. Experimental results on multi-domain and multi-task datasets demonstrate the effectiveness of FTN.\n\nSTRENGTHS:\n1. `Efficient Approach for Knowledge Transfer`: The Factorized Tensor Network (FTN) proposed in the paper offers an effective mechanism for transferring knowledge across multiple tasks and domains. By incorporating task/domain-specific low-rank tensor factors into a shared frozen backbone network, FTN allows for efficient utilization of existing knowledge, reducing the need for extensive retraining or independent models.\n\n2. `Minimal Parameter Requirement`: FTN demonstrates the ability to achieve comparable accuracy to independent single-task/domain networks while requiring a smaller number of additional parameters. This parameter efficiency is advantageous when dealing with a large number of tasks or domains, as it helps reduce storage costs and computational complexity, making the approach more scalable and practical.\n\n3. `Strong Validation Results`: The paper presents experimental results on widely used multi-domain and multi-task datasets. These results showcase the effectiveness of FTN, demonstrating similar accuracy compared to single-task/domain methods while utilizing only a small percentage (2-6%) of additional parameters per task. The validation experiments provide empirical evidence supporting the proposed approach and its potential for practical application.\n\nWEAKNESSES:\n1. `Limited Novelty`: The paper falls short in highlighting the novelty and distinctiveness of FTN compared to other parameter-efficient tuning methods, especially KAdaptation [A] and SSF [B] . This missed opportunity leaves readers wondering about the specific advantages and unique contributions of FTN. A more comprehensive analysis of its novelty would have evoked excitement and clarified its standout features.\n\n2. `Limited Comparison`: The paper fails to sufficiently contrast FTN with other parameter-efficient fine-tuning (PEFT) methods, such as adaptors and low-rank approximation (LORA). This omission leaves unanswered questions about FTN's comparative strengths and weaknesses. A thorough evaluation against these methods would have added depth and fostered confidence in the proposed approach. Additionally, the paper's baseline for multi-domain image translation appears relatively weak, limiting the scope of the evaluation. \n\n3. `Overlooking Recent Trends in Modular Learning`: The paper overlooks recent advancements in modular learning highlighted in references [C] and learning factorized knowledge with different modules [D]. \n\n[A] He X, Li C, Zhang P, et al. Parameter-Efficient Model Adaptation for Vision Transformers[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(1): 817-825. \n\n[B] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123. \n\n[C] Pfeiffer J, Ruder S, Vulić I, et al. Modular deep learning[J]. arXiv preprint arXiv:2302.11529, 2023. \n\n[D] Yang X, Ye J, Wang X. Factorizing knowledge in neural networks[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 73-91.\n\nQUESTIONS:\nThe paper mentions that FTN achieves accuracy comparable to independent single-task/domain networks with a small number of additional parameters. Could you provide more insights into how FTN manages to strike this balance between accuracy and parameter efficiency? What are the underlying mechanisms or strategies employed by FTN that enable it to achieve such promising results?\n\nLIMITATIONS:\nThe author should discuss some of the limitation of FTN in the paper", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
The paper tackles multi-task learning (MTL) and multi-domain learning (MDL). The author proposes to a general method to add task/domain-specific weights to the common shared backbone. Specifically, they add the factorized task/domain specific tensors which introduce very low storage cost for each task and domain while maintaining the original performance. They experiment in three different settings: multi-task learning with dense-prediction tasks, multi-domain learning for image classification and multi-domain learning for image generalization. They also ablate on the relationship between the rank of factorization and the accuracy.

STRENGTHS:
(1) The paper is well written and easy to understand.

(2) The experiment section is well designed and they show the effectiveness of their proposed methods in three different scenarios including multi-task learning / multi-domain learning, deterministic and generative tasks. 

(3) The design of the proposed method is reasonable, easy to implement but effective to preserve the performance of each task/domain and save the storage cost. It scales well with more tasks and domains

WEAKNESSES:
The paper is based on the convolutional network and the author uses big gan in the generation tasks. These designs are somehow in the old fashion. Even though it is mentioned in the conclusion that the method can be extended to transformer based architecture, it is lack of experimental proof.

QUESTIONS:
How does the model applied to the transformer based architecture? 

Does the method work for diffusion models when tackling generative tasks?

LIMITATIONS:
They have included limitation in the main paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of adapting a single pretrained network to multiple tasks or domains while introducing as little parameters as possible. More specifically, first a backbone model is trained which contains a shared backbone. Then,  the proposed method (Factorized Tensor Networks, **FTN**) is applied to further adapt the shared backbone to each task with a few added parameters. All task specific parameters are trained alongside each other (i.e. the BatchNorm parameter, the low-rank FTN parameters as well as the task heads)\n\nGiven a weight $W$ from the shared backbone, FTN adds low-rank parameters $\\Delta W_t$ for each task/domain $t$ such that the forward pass for task $t$ uses the shifted weights $W + \\Delta W_t$. Setting the rank $r$ of the low-rank parameters allow to control the trade-off between parameter efficiency and model capacity. In addition, FTN also tune batch norm layers parameters (scale and bias) for each task independently.\n\nThe proposed FTN is compared to previous methods on adapting weights during finetuning (e.g. residual adapters) on mainly three benchmarks:\n- Multidomain classification from a pretrained ImageNet model to the 5 domains in ImageNet-to-Sketch\n- Multidomain classification from a pretrained ImageNet model to the 6 domains in DomainNet\n- Multitask from a pretrained ImageNet model to the 3 domains in NYU\n\nSTRENGTHS:\n* The proposed method is simple to implement and compared to other adapter methods it is extremely parameter efficient\n\n* The method is experimented on both multi-domain (different inputs) and multi-task (different outputs) settings\n\n* While the method is only evaluated on convolutional networks, there is no strong assumptions on the backbone architectures, hence it could also be used on transformers for instances\n\nWEAKNESSES:\n- **Missing important related work**: As far as I can tell, the proposed FTN is quite similar to the low-rank finetuning method from \"LoRA: Low-Rank Adaptation of Large Language Models\". While LoRA is only evaluated on NLP/transformers in the original paper, it should be mentioned and discussed in related work\n\n- **Unclear how the number of parameters is computed** The original $BA^2$ paper reports a parameter cost of \"1.03x\" in their Table 3. While this paper reports a cost of \"3.8x [1.71x]\". It seems like the baselines results reported in this paper are directly taken from the Table 1 in the  \"Task Adaptive Parameter Sharing for Multi-Task Learning\" paer:   It would be nice to clarify how baselines evaluation was conducted and where the discrepancy may come from\n\n- **Missing results from Visual Decathlon**: Manyworks on adapters report results on Visual Decathlon introduced in the original residual adapters work where an ImageNet pretrained model is adapted to a wide variety of very different downstream tasks\n\nQUESTIONS:\n* In **Table 3**: Why is the cost of \"single task\" equal to 4x the one of \"decoder only\" ? From Figure 1, it seems that \"decoder only\" would be an architecture contains one R18 backbone + 4 task heads. While \"single task\" would contain 4 R18 backbones, but still 4 decoders / task heads: So when comparing the absolute number of parameters (backbone + task head), shouldn't the factor be smaller ?\n\nLIMITATIONS:\nLimitations are explicitly mentioned in an independent paragraph and fairl ydescribe some drawbacks of the methods.\nAlthough I partially disagree with the statement that *(line 73) \"The proposed method does not affect the computational cost because we need to compute features for each task/domain using separate functional pathways\"*: This is fine for the MDL applications, however most MTL works operate under the \"feature -extractor\" paradigm instead, where the multiple tasks can be solved with a single forward pass on the fully shared backbones, followed by lightweight task heads: In comparison, FTN is much less efficient as it requires individual forward passes of the encoder for each task.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}