Joint Data-Task Generation for Auxiliary Learning
Authors: Hong Chen
Xin Wang
Yuwei Zhou
Yijian Qin
Chaoyu Guan
Wenwu Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: auxiliary learning
data-task joint generation

Abstract: Current auxiliary learning methods mainly adopt the methodology of reweighing losses for the manually collected auxiliary data and tasks. However, these methods heavily rely on domain knowledge during data collection, which may be hardly available in reality. Therefore, current methods will become less effective and even do harm to the primary task when unhelpful auxiliary data and tasks are employed. To tackle the problem, we propose a joint data-task generation framework for auxiliary learning (DTG-AuxL), which can bring benefits to the primary task by generating the new auxiliary data and task in a joint manner. The proposed DTG-AuxL framework contains a joint generator and a bi-level optimization strategy. Specifically, the joint generator contains a feature generator and a label generator, which are designed to be applicable  and expressive for various auxiliary learning scenarios. The bi-level optimization strategy optimizes the joint generator and the task learning model, where the joint generator is effectively optimized in the upper level via the implicit gradient from the primary loss and the explicit gradient of our proposed instance regularization, while the task learning model is optimized in the lower level by the generated data and task. Extensive experiments show that our proposed DTG-AuxL framework consistently outperforms existing methods in various auxiliary learning scenarios, particularly when the manually collected auxiliary data and tasks are unhelpful.

Joint Data-Task Generation for Auxiliary Learning
Hong Chen', Xin : Yuwei Zhou', Yijian Qin', Chaoyu Guan', Wenwu Zhu 1.2+
'Department of Computer Science and Technology, Tsinghua University Beijing National Research Center for Information Science and Technology, Tsinghua (h- cb-chen20, zhou-yu21, gung...ggyn111)mmmaaalll tsinghua. edu. cn {xin_wand wizhhu edu. cn
edw..
Abstract
Current auxiliary learning methods mainly adopt the methodology of reweighing losses for the manually collected auxiliary data and tasks. However, these methods heavily rely on domain knowledge during data collection, which may be hardly available in reality. Therefore, current methods will become less effective and even do harm the primary task when unhelpful auxiliary data and tasks are employed. To tackle the problem, we propose joint data-task generation framework for auxiliary learning (DTG-AuxL), which can bring benefits the primary task by generating the new auxiliary data and task in joint manner. The proposed DTG- AuxL framework contains a joint generator and bi-level optimization strategy. Specifically, the joint generator contains feature generator and a label generator, which are designed to be applicable and expressive for various auxiliary learning scenarios. The bi- optimization strategy optimizes the joint generator and the task learning model, where the joint generator is effectively optimized in the upper level via the implicit gradient from the primary loss and the explicit gradient of our proposed instance regularization, while the task learning model is optimized the lower level by the generated data and task. Extensive experiments show that our proposed DTG-AuxL. framework consistently outperforms existing methods in various auxiliary learning scenarios, particularly when the manually collected auxiliary data and tasks are unhelpful.
Introduction
Auxiliary learning aims to improve the model generalization ability on the primary task with the help of related auxiliary tasks 2 This learning paradigm has been widely adopted and has shown its effectiveness in various areas, like image classification (3). recommendation [4] and reinforcement
learning [5] 6). Different auxiliary tasks are often chosen manually according the primary task, e.g., [7] utilize the task of visual attribute classification to help the fine-grained bird classification and improve the click conversion rate prediction with the help of click-through rate prediction task.
Most existing works utilize the auxiliary information by first reweighing the losses of the auxiliary data and tasks, and then the of the weighted losses together with the primary loss optimize the task learning model. The weights are employed to balance the primary loss and the auxiliary losses avoid negative auxiliary transfer, which tuned with HPO tools 44] 3 g) More recent works 9
propose to dynamically weigh different auxiliary losses during training.
However, the existing methods require that there exists beneficial information in the auxiliary data and tasks, so that the beneficial loss terms can be selected through the loss reweighing process. This condition cannot always be satisfied because whether the auxiliary data task is beneficial depends on many factors including the chosen auxiliary task, the scale of primary task dataset and the selected
"Coresponding Authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

learning model for the tasks [ making it difficult to manually collect the beneficial auxiliary data and task via prior knowledge. Therefore, existing approaches may adopt useless auxiliary information and finally do harm to the primary task when the involved auxiliary data and tasks are improperly collected, as observed [ 12). Although 2 propose to generate fine-grained auxiliary classification tasks, this method can be only applied to the classification primary task. Additionally, they only generate the new task without considering new data, while the data-level information is pointed out to be important factor in auxiliary learning (Л
To address the problem, in this paper, we propose to simultaneously generate the auxiliary data and task for auxiliary learning in a joint manner. However, there are three challenges for the joint generation. First, it is challenging to design a generic framework that can accommodate various tasks with different inputs. This is because the types of data and tasks are quite diversified in different auxiliary learning scenarios, e.g., the primary task of image classification takes visual images as input and outputs categorical labels for classification 77], while the primary task of rating prediction for recommendation takes tabular data as input and outputs numeric labels for regression 4 Second, is challenging to develop generation framework that is expressive enough to produce beneficial data and tasks for the primary task. Finally, to guarantee the jointly generated auxiliary data and task are beneficial to primary task, how to effectively optimize the parameters in the generation framework is a challenging problem as well.
To tackle these challenges. we propose joint Data-Task Generation framework for Auxiliary Learning (DTG-AuxL), which involves a joint generator and bi-level optimization strategy. Specif- ically, the joint generator consists of feature generator and label generator, which generates the new auxiliary data and new task based on the existing manually collected data and tasks. The data generation process is conducted in the feature space so that it can accommodate various input types, while the label generator architecture is inspired by the model in recommendation which can accommodate both categorical labels for classification and numeric labels for regression. Moreover, we introduce nonlinear interaction terms in the joint generator, making it more expressive to produce beneficial auxiliary data and tasks. To effectively optimize the joint generator and the task learning model, we propose the bi-level optimization strategy with instance regularization. In the lower optimization, the task learning model is optimized by the generated auxiliary data and task. In the upper optimization for the joint generator, we not only utilize the implicit gradient from the primary loss but also the explicit gradient from the proposed instance regularization, avoid label generation mode collapse. Extensive experiments show that DTGG-AxL. outperforms existing methods in various auxiliary learning scenarios, especially when the manually collected auxiliary data and task are unhelpful to the primary task. We summarize our contributions as follows,
We propose to simultaneously generate auxiliary data and tasks in a joint manner in auxiliary learning for the first time, to the best of our knowledge.
We propose DTG-AuXL, a joint data-task generation framework applicable in various auxiliary learning scenarios, containing a joint generator and a bi-level optimization strategy.
Extensive experiments in various auxiliary learning scenarios demonstrate the superiority of our proposed DTG-AuxL framework. We further analyze when and how DTG--uxL works to bring performance boost.
2 Related Work
Auxiliary Learning The most widely adopted way in auxiliary learning is combine the loss of the primary task and the losses of the auxiliary tasks in a linear way 3
7
4
5) where the
linear weights are tuned manually or with HPO methods. More recent works 9 6 10 propose to automatically assign dynamic weights to the auxiliary losses. [9] propose to assign weights to each loss based on the cosine gradient similarity between the primary loss and each auxiliary loss. Later work [00 aims to make the weighted auxiliary gradient sum closest to the gradient of the primary loss. 1 utilize the li-le optimization strategy to learn the auxiliary weights, where I even propose to combine the losses not only limited to a linear form, but also a nonlinear form. point out that only considering the task- information is not sufficient, so they jointly consider the weights of different tasks and different data samples within the same task through joint selector. As aforementioned, these reweighing methods will easily fail to bring improvement when the chosen auxiliary tasks and data are unhelpful. There are also works [2] [ that generate
2

---

Task Learning (LKM)
Ingut dada
Fentures
lower ustimization For
taok leaming modwi
Prarary Head
TLM(A)
K3(4)
Backbonn
Aux Head
update
Waha Mo
For
Aux Head
TLM
feature
far
Aux Head
Vall
Va 2dd update
Genwented label
Vglly + Tybrep
TLM
Feature Encar torn
Liner Embadiiing
Yatau
Label Incer torn
(E2))
Yalan
Fsd
TLM
Kex,
Ygu
Pgle N" update
Feature norl tar
Yxu
(EN,)
MLP:
TLM(A)
IG($1
Fiberdiing
uppir opilinccation gar
Joint Generator(]G] $
Model Design
Optimization
Figure I: The overall DTG-AuxL framework. In the model design part, the joint generator contains a feature generator and label generator. The feature generator utilizes the primary and auxiliary features to generate the new auxiliary feature fg.j. whose label D9.5 generated by the label generator by combining the information of all the auxiliary and primary labels. The optimization part shows that we optimize the task learning model and the joint generator in an alternating bi-level manner.
fine-grained classification auxiliary tasks for the primary classification task. However, they can only be applied to a classification problem. Additionally, they only generate auxiliary tasks without new data, limiting their performance especially when the data of the primary task is inadequate, which is an often encountered scenario in auxiliary learning
Multi-task learning Multi-task learning aims to share information among tasks to improve model performance. However, the goal of multi-task learning is to obtain good performance for all the tasks, while auxiliary learning only focuses on the primary task. Multi-task learning methods can be categorized into three parts [15]: multi-task architecture design [16] [17, multi-task optimization [18]
19] and multi-task relationship learning (20), where the multi-task optimization methods involve techniques for optimizing several losses, like loss reweighing [8] and gradient modulation 2
3
The Proposed Method
The overall DTG-AuxL framework is shown in Figure Next, we give the problem formulation, describe the designs of the joint generator, and present the proposed bi-level optimization strategy.
3.1 Preliminaries and Problem Formulation
In auxiliary learning, we have one primary task Tp, and totally K auxiliary tasks {Tas) Each of these tasks has its corresponding training dataset, including the primary dataset Dp ==1 ((Fp.: Mp.s)) ) and the dataset for each auxiliary task D.: = ((Fai )at.)) where denotes the data sample number of the dataset. the auxiliary tasks share the same input with the primary task, which is a widely encountered and studied scenario in previous works [ 9 [0]. we have Toi.j Eps and [aas parameterized by 8 which is used to learn all the tasks together. There is also a validation dataset D. which is used to evaluate the model performance on the primary task. With these notations, the widely adopted training objective of auxiliary learning is:
1K
Dpll- Besides the datasets, we have a task learning model
Lo
Lai(Der;
(I)
where Cp and Las indicate the loss functions of the primary and each auxiliary task. Current reweighing methods focus on how to decide w, that the task learning model can achieve the best

---

performance on D.. However, as aforementioned, the terms (Di 0) are defined by the manually collected auxiliary data and tasks, with no guarantee bring benefits for the primary task. Therefore, we propose to simultaneously generate the new beneficial auxiliary data and task in a joint manner, with the following training objective:
6) = (o - Σ (Dais + (P
(2)
where the last term is the loss on the generated auxiliary data and task. (Ø( {T Naaii:aaaal ) / is generated dataset which contains the new data together with the new label defining the new task, and & denotes the parameters of the joint generator that are used to generate Dg- Since we only care about the performance of the primary task in auxiliary learning, we keep Lg the same loss function as Cp. which can be cross entropy MSE loss, etc. Note that still keep the original auxiliary loss terms our training objective, so that it can accommodate the scenario where the cxisting manually collected auxiliary data and tasks beneficial. The task weights w; and Wy are also learnable, which will be optimized together with the generator parameters, and we uniformly denote them all as & for convenience. Next, we discuss the details of the generator and how optimize & and 0.
3.2 Joint Generator Design
The joint generator involves a feature generator to generate new features and label generator to generate new task for the new feature, as shown in the Model Design of Figure
3.2.1 Feature Generator
Since we expect that our data generator can tackle different types data, a neat and elegant solution is to conduct the generation process in the feature space. In the feature space, different types of input data are transformed to vectors, so we can tackle these vectors in a unified way. Specifically, for the input from different tasks [Tp. Fal.j. ".. Fak.yl the task learning model will first map them to the feature space with an encoder which is generally noted as "backbone", and then use different task-specific heads to tackle each of the tasks. We generate the features based on the features extracted via the backbone. In another word, the input of the feature generator is feature list [pp.j. fal.j: fuk,j), where fp.j Sbacktome (5 0) is a d-dimension feature. We next describe how the new features are generated.
Feature linear term. A natural way to generate new features is to combine different features with linear masks. We assign an individual feature mask to each the K + features, i.c., we have mask list pm Mal," "-. mak]. where each mask is d-dimension vector. Then, we denote the subscript set for the task IDs {p, al,"............... ... aak} } as S. and the linear combination is conducted as follows:
(
for. = Σ m, figs m. K
1 1.....d.
SE
where gl indicates "generated lincarly", means the element-wise multiplication, and my is the normalized mask. m,[k] is the 4thh element of the mask m,, and is normalized by softmax with the elements in the same dimension in other masks. This linear combination with normalized mask works as feature selection from all the input features, and then combines them into one.
Feature nonlinear term. To make the generated features more expressive, we introduce an MLP Muult--Layer-Peeceptroo to capture the nonlinear feature interaction:
(4) where gn indicates "generated nonlinearly". All the features are concatenated together and the nonlinear feature interactions are modeled by an MLP, whose output dimension is d. Finally, the generated feature the sum of the linear and nonlinear term, i.e., So S9 + s
fgn.j = ML.P ([fp.s fat.i. fak,));
3.2.2 Label Generator
The label generator aims to generate proper label for the feature produced by the feature generator. We use the labels of the input data as the input of the label generator to make the generated label expressive enough, i.c., the input of the label generator is list of labels [3p.. Bal.j, yak.].

---

However, this label list may contain both numeric labels (if the corresponding task is regression) and categorical labels (if the corresponding task classification). How to utilize different types of label information to generate a new label is the key problem. Inspired by the CTR (click-through rate) model recommendation [22 23 which predicts the user's preference towards an item also based on both their numeric and categorical features, we design the label generator with both linear terms and deep nonlinear terms similar to those in the CTR model.
Label linear term. The linear term models the direct and independent relationship between each input label and the generated label. We simply keep the dimension of the generated label the same as the primary label Vp.j and more adaptive ways to choose the dimension can be an interesting future work. Here, we assume that the dimension of V/p.j is m, where m = if the primary task T, is regression, or m equals to the number of the total categories if Tい is classification. If the primary task regression, the generated label is also scalar value for regression, while if the primary task is classification, the generated label is an m-dimension probability distribution vector. Specifically, for each task ID i E S. if T; is a classification task and it has totally d, categories, there is an embedding table E, with dimension d: x m for this task. We directly map the label Vi.i in task T; to its m-dimension embedding space through the embedding table:
(5) If T; is a regression task, we follow 23 to map y to its m-dimension embedding. Specifically, we also have an embedding table E, for this task T., whose dimension is d: m, where d: H is hyper-paranen parameter which is fixed to 10 in our experiments. Then Vi., is mapped into the m-dimension embedding space as follows:
Vgl.i.j = Ec(ys,))
H
Cij = Softtma
E R" Vol..
Σ a [NJE)
(6)
where the numeric label Vi.j is linearly transformed to H- H-dimension vector Ctjj. which is used to weigh all the embeddings in the embedding table to obtain the final emhedding. Then the final label linear term is obtained by the sum of all the input label embeddings Vgl.j Eies Vgl.i.j
Label nonlinear term. In addition to the label linear term capturing how each input label indepen- dently influences the final generated label, we also propose nonlinear term to model the influence of more complex label interactions on the generated label. Specifically, we introduce another group of embedding tables for all the input tasks {EN,) Jigs where the ways to obtain the embeddings of the categorical and numeric labels are the same as eq. (5) and eq. (6). respectively. The dimension of EN, is d2 X mn, where d, equals to the number of total categories if Ti is classification. d, equals to H (0) for regression task, and m, is a apper-paramttr.. With these embedding tables, we can map all the input label list [pp.j: Ja "* -aa j] an embedding list (ep.j, Cal.j." " [eak]]. Then the nonlinear label interactions are captured through an MLP:
Vgn.. = M LMP eai.ji ;eax,s)).
(7)
The final generated label for the generated feature ʃg.j is the sum of the linear and nonlinear terms,
Vg.j
ygt.J + ygn..-
(8)
Label hias term. Besides the linear and nonlinear terms, we also introduce label bias term that guides the generated label to possess similar semantic meaning to the label from the target primary task, yp.. As such, we add you as the label bias term as follows,
Úg. = 0yp.j + (1-a) norm(ys.))
where a E (0, 1) is a learnable parameter initialized as 0.5 and norm(-) is the normalization function. If the primary task is classification, norm(-) will be sof (tmaa(-) and yoj will be converted to the one-hot form. If the primary task is regression with range (a, b), then norm(() is set to be (b sigmoid((() + a. This bias term makes the generated label lie in the same space as the original label, enabling us to more conveniently explore its semantic meaning, which is also verified to improve model performance in experiments.
3.2.3 Discussion about the Share Input Scenario
The proposed generator can generate the new feature by combining the features from the primary and auxiliary tasks. However, in the most widely studied auxiliary learning scenario 19 M. all the
5

---

tasks share the same imput data. We only have the training dataset (Zp.). My.i Vol..'.. , BoK j=1 Therefore, cannot obtain the auxiliary data [ _aak.j] from the auxiliary tasks to conduct the feature generation. To tackle this problem, we obtain the new auxiliary data by randomly sampling another data sample in the dataset, which is easy implement with randomly shuffling the training batch to match another sample for the original sample. Specifically, for a sample (Tp..1 Mp.j Val.ji".. ".. :j) we consider another sample (Tp.52, 3r. Val _j2. -ak,jja) in the dataset to provide new auxiliary data information. The imput of the feature generator is [fp.j. f2,12], which are the features extracted by the backbone with [xp.j. 22.52] as input. When generating new labels, we need to combine all the existing labels of the two samples, i.c., the input of the label generator is [Vp.j. yal.j.... YaK.j j. 32 Val.j2: ... |ya,,y2], and then the label generation process is the same as before. Note that there are two small details: i) During the label embedding process, yas, and yoi.z2 share the same embedding table, because they both belong to the same task. ii) For the bias term in eq. (9), we now have two labels from the primary task, Vp.j and Up.j2, and then eq. (9) can be adjusted to:
Pa.s = a(3es + (1 B) + (1 -a norm((s..)
(10)
where 3 E (0, 1) is a learnable parameter of the generator. Till now, we have described the designs of the joint generator, which is applicable in various auxiliary learning scenarios. However, the generator has several parameters to be optimized, like the masks and MLPs. We denote all the learnable parameters in the generator together with the task weights wy in eq. 2 é. Next, we will elaborate how we optimize the task learning model parameters and the generator parameters o.
3.3 Optimization Strategy
Bi-level optimization. In our whole framework, the task learning model parameters O are expected to minimize the loss of all the selected and generated tasks, while the generator parameters aim to make 0 achieve the best performance on the primary task. These two different goals give rise to the following li-le optimization problem:
argmin (p(00 Do):
6
(11)
s.t. r (0) = arg min L, 6).
where Le(0, is the objective in eq. and (0) D.) is the primary task loss of the task learning model on the validation dataset. The lower optimization is easy, we can directly obtain the gradient of 0 as (o o). However, in the upper optimization, (6) Do) directly relies A
(0);
instead of d. Assuming that the Hessian V. (Ø), Φ) is positive-definite we can use the implicit theorem to obtain its implicit gradient V (0" D.).
Ve (0); D.) VoL, (V.C))-.
(12)
Detailed derivation can be found in Appendix 1. Since the inverse of the Hessian is often intractable, we follow (24] to use truncated Neumann series to approximate it. (V) ~ E/ - C, and n is fixed to 3 in all our experiments. Thus, the complete implicit gradient is approximated as:
oo (0): (a): VoLp E ([--vah VoCa)' VoLoL.
(13)
Instance regularization. In the upper optimization, we additionally introduce an instance regulariza- tion for the parameterized label Uي.و in eq.
(8) as follows, to prevent generation mode collapse.
Lreg (D) Σ 1Dy Σ ve cos( calegorical
1
(14)
Pal j= (o.glo((o..)
reumerical
where if the generated label is categorical, we expect that the cosine similarity of different generated labels is small. This regularization means expect that the generated label can keep its instance- level uniqueness, preventing the generated labels of all the generated features from being the same. Similarly, if the generated label is numerical, since is I-dimension, we the entropy regularization to achieve this goal, where Vg.j = cVa.. / E)pe c²a..'_ Then in the upper optimization, the gradient of o is the sum of the implicit gradient and explicit gradient from the regularization:
& (o), + Vo
(15)

---

Now the gradients of f and é are obtained, and we follow previous works [1 to alternatingly update and Φ. The update loop will continue until convergence. where in each loop we first update @ for N times, and then update e using the upper gradient eq. [15) as shown in Figure [ N is the interval between two upper updates. We summarize the complete algorithm in Appendix 2, where we do not require an additional validation dataset Da to calculate the upper objective but reuse the training primary set Dp as done in [1 14.
4 Experiments
4.1
Experimental Setup
Task and Dataset. We conduct our experiments on two scenarios to validate the generalization ability of our method. One is the most widely studied scenario in previous auxiliary learning works, where the auxiliary tasks share the same input as the primary task (share input). The other one is that the inputs of the primary and auxiliary tasks are different (different input). In the Share Input setting, (i) CUB 25:: we follow previous works [ to use the bird visual attribute classification (e.g., whether the bird has white belly) to help the bird species classification primary task, where both the auxiliary and primary labels are categorical. (ii) CIFAR100 (26]: it is a widely adopted image classification dataset, where there are totally 100 categories. Additionally, each image has a coarse class, e.g., a car belongs to the "vehicles 1" coarse class. We use the coarse classification as the auxiliary task to help the 100-clasiffication primary task. (iii) Besides the classification problem, we also focus on the regression problem, where we follow previous works [2] to regard the rating prediction in recommender system as the primary task, and the CTR prediction as the auxiliary task. The primary task is regression and the auxiliary task is binary classification. We choose the widely used Amazon Toys and Movies (27) datasets, where we use the user ID, item ID and item category as the input data. In the Different Input setting, (i) CIFAR10-100 is a setting where our primary task is the CIFAR1C 10 classification problem, and the auxiliary task is the CIFAR 100 classification problem. (ii) Pet-CUB is similar setting where our primary task is fine fineggriined pet classification on the Pet (28) dataset, while our auxiliary task is the bird species classification in the previously mentioned CUB dataset. More detailed data information is presented in Appendix 4.
Baselines. We compare with SOTA auxiliary learning methods, including the reweighing methods and the auxiliary task generation method. Single task learnin(STLL) is natural baseline where only train on the primary task. Equal is a baseline where we assign equal weights 1.0 to all the tasks. Uncert (18) is dynamic weighting method for multi-task learning based on task uncertainty. GCS [9] and AuxL. [1] dynamically reweight the auxiliary losses, and JTDS [[]] not only reweighs the tasks but also each data sample within each task. MAXI. D is a method that automatically generates a fine-grained auxiliary task for the primary classification task. We provide detailed differences between our work and the baselines in Appendix 3.1.
Implementation details. In CUB dataset, we respectively adopt ResNet18 22] and ResNet50 as our backbone. In CIFAR 100, we respectively adopt R and 4-yye ConyNet composed of Convolution, Batch Normaliztion and Relu layers as our backbone. In Amazon Toys and Movies, adopt AutoINT [00 as the backbone. In 1000 the backbone is the 4-layer ConvNet and in Pet-CUB the backbone is ResNet18. For the head of each task, we adopt Multi-Layer- Perceptron(MI.P whose layer is searched from (1, 2). In the generator, the embedding dimension m,, is searched from (32, 64), ), and the layer number of the MI.P is searched from (2, 3, 4). More details are presented in Appendix 4.
4.2 Experimental Results
4.2.1 Method Performance
Main results. Table | presents the overall experimental results, showing that our proposed method consistently outperforms existing methods on the diversified auxiliary learning scenarios. During training in the CUB dataset and Pet-CUB dataset, different from [II] that does not use the learning rate scheduler, we found that applying a learning rate scheduler to all the baselines can obtain better or at least on-par performance. Therefore, we report the results with the scheduler. From the results, we have the following observations. (i) As expected, our proposed method brings benefits for the primary task when the original manually collected auxiliary data and task are unhelpful. For example,
7

---

Table 1: Overall performance. We report the results over 5 random seeds. Note that JTDS cannot handle the different input scenarios and MAXL cannot generate auxiliary tasks for the primary regression task. The metric for classification is accuracy(Acc) and the metric for the rating regression in recommendation is RMSE. Higher accuracy and lower RMSE indicate better results.
Dataset
CUB
CIFAR100
Toys
Movies
CIFAR1O-100
Pet-CUB
Metrie
Acc( %T
Acc( STT
(cc(%)
RMSEL
RMSET
Backhone
ResNet18
ResNe150
ConvNet
ResNer18
AutolNT
AutoINT
ConvNet
ResNet18
STL
76.16u.70
80.46..42
50.35u.30
55.79
0,9188p.000s
1,0466 o00s
79.35p.41
69,476.3
Equal
74.25..17
28.28..21
49. 49.56u.31
56.42..u5
0.9213
1.0459..um
78.99u.42
67.15
Uncert
73.94a.00
77.030.24
48. 48.90a.10
56.950.91
0.9171000000
1.04950.0018
80.13p.17
63,58..13
GCS
74.1
78.54u.5
49. 49.26
56.570.21
0.9224p.000s
1.045900018
79.59p.ou
67.3
AuxL
75.39p.59
78.000.55
49.39p.82
57.140.25
0,9186p.0005
1.04830.0019
79.69p.41
66.177.37
JTDS
76.50g.47
79.34..18
49.00p.33
57.280.20
0.9187p.0004
1.0504..0aa
MAXL
75.79..4
78.48.
49.71
56.32.
80.02
68 18.4%.uv so
ours
77.75u27
81.73..00
50.94..00
67.84..20
0.91530.0004
1.0426 0000
80.64
70,48,
Table 2: Acc(%) on CUB Fewshot Experiments with ResNet50.
Method
STL
Equal
Uncert
GCS
AuxL
JTDS
MAXL
ours
5 shot
44.9 44,99
45.37
45.63%.08
45.68 6z
45.07
46. 46.13p
45.47, 08
52.33
10 shot
63.25u.6s
60.58p.a1
59.30n.ss
60.81..74
63.79b.11
63.39 03.39o.18
61.70p.11
67.68p.33
in the CUB dataset with ResNet50 backbone, Movies, and Pet-CUB, the three scenarios, all the methods that utilize auxiliary tasks perform worse than the STL baseline, indicating that the manually chosen auxiliary data and tasks are not beneficial to the primary task. This phenomenon is also observed in previous works 2 However, our proposed method can utilize the information in the originally harmful auxiliary task to generate new and beneficial auxiliary data and task, so that the model performance can be significantly improved. The ability of our method to convert originally harmful auxiliary labels into beneficial forms is further validated in Appendix 5.2. Although MAXL generates new fine-graines classification task for the primary task, it brings no improvement in these settings. This is possibly because it requires a hierarchical dataset structure in the primary task. However, in the CUB and Pet-CUB setting, the primary task itself is fine-grained image classification problem and a more fine-grained meanwhile heneficial task is hard to be automatically generated. Additionally, it does not consider generating new data during generation. (ii) When the original auxiliary task is beneficial to the primary task, our proposed method inherits these benefits and can bring further improvement. For example, in CIFARIO0 with ResNet18 as backbone and CIFAR10-100 settings, almost all the methods that utilize the auxiliary information outperform the STL baseline, and our proposed method outperforms all the existing auxiliary methods. (iii) Whether the auxiliary data and task are beneficial to the primary task depends on the backbone we choose. In the CIFAR100 dataset, when we the use ConvNet backbone, the original auxiliary data and tasks are harmful because all the existing auxiliary methods perform worse than STL, but when we use ResNet18 as the backbone, the auxiliary data and tasks are beneficial.
Fewshot results. Since previous works а indicate auxiliary learning is more effective when the data of the primary task is inadequate, we conduct experiments on CUB where we only use 5/10 shots of each category for the primary task in Table [ We can see that when the data of the primary task is 5 shot, all methods that utilize the auxiliary task perform better than STL, while in T when the primary task has full data, the auxiliary data and task are harmful, indicating whether the auxiliary information beneficial or not is related to the primary dataset scale. Among all the auxiliary learning methods, our proposed method brings the most significant improvement. Compared to the full dataset setting, the improvement of our method becomes more significant the fewshot setting. The improvement largely results from that we not only generate new tasks but also new data. The effectiveness of generating new data is further validated Appendix 5.3.
4.2.2 Ablation Studies
To further understand how our proposed method works, we provide the following ablation studies.
8

---

Table Generation visualization. The first two rows from Pet-CUB and the last two rows are from CUB. The first and second columns are the images from the primary and auxiliary tasks used to generate new feature and label. The third column is the generated label, where the x-axis represents the category and the y-axis represents the probability of the generated feature belonging to this category. In the fourth column, we visualize the images with maximum probabilities (largest peaks) in the generated label, where besides the image from the peak guided by the bias term, we visualize images from the largest 2 peaks among the small peaks.
4mmage-P P
AEmage-Aux
4G-Label
#Max Prub Imagees
Generation visualization. We visualize the generation process in Table 3 The first two rows from the Pet-CUB dataset in the Different Input setting, and the last two rows are from the CUB see the generated label has one obvious
dataset in the Share Input setting. In Pet-CUB, peak from the label bias term, which represents the original category of the primary image. There are also some other small peaks. Specifically, from the first row, we can see that when our feature generator combines a white cat and a yellow-and-black bird, the label generator thinks that the label of the generated feature has the maximum probability of being the original white cat category, the second largest probability of being the yellow-an and-black cat category and then a similar color dog. The pattern in the second row similar to the first row, the texture and color of the bird are combined with the black cat, so that the label generator thinks the generated feature should first belong to the original black cat, then the brown cat with stripes from the bird, and then similar color dog. In CUB, the generated label has two obvious peaks, which represent the two categories of the primary and auxiliary images. Also, there are some other small peaks, we can see that the images from the small peaks are similar to the primary and auxiliary images, which have reasonable semantic meanings.
Effectiveness of the framework design. Our whole framework contains the generator and the optimization strategy. We validate their effectiveness in Table To make the generator expressive, we incorporate nonlinear terms in the generator, and we also incorporate the label bias term in the label generator. To optimize the generator, we propose the bi-level optimization strategy with instance regularization in the upper update. We respectively remove each of these components and observe performance degradation, where w/o F-nonlinear, w/o L-nonlinear, w/o L-bias, w/o bi-level and w/o instance reg respectively represents the variant where we remove the feature nonlinear term, label nonlinear term, label bias term, the overall upper update, and the instance regularization. We find that the non-linear terms indeed help to generate more beneficial data and tasks by making the generator more expressive. Also, the instance regularization is very important in the bi-level optimization. Appendix 5.4.
Furthermore, we observe label generation mode collapse without the regularization
Comparison with MixUp During the generation visualization, we find that in the CUB setting, the generation process behaves like MixUp where the inputs are combined in linear way and the labels are also combined in a linear way. Therefore, we compare our proposed method with different MixUp methods in Table 5 Mix Up 31 means we directly utilize the data and labels generated by Mix Up as auxiliary tasks. F-MixUp 32 means the linear combination does not happen in the input, but WiMM/Miitto---MMMA means the linear weight of MixUp/F-MixUp is learned automatically with the bi-level optimization, and the task weights are also learnable, i.e, utilizing learnable MixUp//-MixUp to replace our joint generator. We can see that our proposed method outperforms these MixUp methods. The reasons are quite obvious, as shown in Table 3, our proposed method not only has the two peaks in Mix but also other semantically reasonable small peaks. Compared to MixUp, the ad advantages of our proposed method are three-fold. (i) Our method
in the manifold feature space.

---

Table Effectiveness of framework designs. The first three rows explore the effectiveness of the generator components, while the rest show the effectiveness of the training strategies.
Dataset
CUB
Pet-CUB
Toys
Metric
Acc( % yf
Acc( % I
RMSE,
Backbone
ResNet50
ResNet18
AutoINT
w/o F-nonlincar
81.22o.27
67.760.17
0.91950.cm 0021
w/o L-nonlinear
80,960.37
66,581.12
0.92130.002
w/o L-bias
79.280.34
69.366.37
0.9204o.oom
w/ bi-level
80.32o.4z
68.53u.45
0,9163d.000r
w/o instance reg
80, 80.440.54
67.53o.c 95
0.9174o.000
complete
81.73p.20
70.480.2 28
0.91530.co
can not only combine information from the label the same task but also label from other tasks. (ii) Our method can not only model linear relations in generation but also non-lincar relations. (iii) Our method can handle the Different Input scenario, while MixUp cannot.
To further show the advantage of our proposed generation compared to MixUp, we apply Auto-F-u --MMix-ixx on top of auxiliary learning methods (AuxL. JTDS, MAXL) and the experimental results are presented in Table 6 The results show that both directly combining auxiliary weighting methods(AuxL, JTDS) with Auto-F-Mix and combining the current auxiliary generation method (MAXL) with Auto-F-mix are not effective as our method, indicating the advantage of our proposed joint generation.
Table Comparison with MixUp Methods. Note that MixUp cannot be applied to the recommenda- tion scenario where the input are categorical features.
Dataset
CUB
Movies RMSE
Metric
Acc(%))
Backbone
ResNet18
AutoINT
MixUp
75.880.50
Auto-Mix
76.53o.38
F-MixUp
75.330.67
1.0460p.com
Auto-Fi Mixx-F-Mix
76.88n.52
1.0464..o00
ours
77.750.27
1.0426..
Table 6: Comparison to the combination of Auto-F-Mix and current auxiliary learning methods. We denote Auto-F-Mix AFM.
Dataset
CUB
CUB-5shot
Toys
Metric
Acc( % %f
Acc(%)
RMSEJ
Backbone
ResNet50
ResNet50
AutoINT
AFM
80.30p.s7
46.97o.95
0.91870.000s
AUXL+AFM
79.701.ur .u7
47.841.25
0.9195p.uu11
MAXL+AFM
80.76p.as
47.96u.89
JTDS+AFM
80.52a.90
14..44
0.9189 6.089.
ours
81.730.20
52.331.36
0.9153p.00m.
Conclusion
In this paper, we propose to jointly generate beneficial auxiliary data and tasks for auxiliary learning. so that the primary task can still obtain benefits when the manually collected auxiliary data and tasks are unhelpful. We propose the DTGL framework with joint generator and a bi-level optimization strategy, which can be applied in various auxiliary learning scenarios. Future works like designing more adaptive generators and more efficient bi-level optimization algorithms can further improve the generation.
10

---

Acknowledgement
This work was supported by the National Key Research and Development Program of China No. 2020AAA01003000, National Natural Science Foundation of China (No. 62222209, 62250008, 62102222), Beijing National Research Center for Information Science and Technology under Grant No. BNR2023RC01003, BNR2023TD03006, and Beijing Key Lab of Networked Multimedia.
References
[1] Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Felaya. Auxiliary learning by implicit differentiation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[2] Shikun Liu, Andrew J. Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Prucessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1677-1687, 2019.
[3] Lucas Beyer, Xiaohua Zhai, Avital Oliver, and Alexander Kolesnikov. S4L: self-supervised semi-supervised learning. In ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pages 1476-1485 2019.
[4] Hong Wen, Jing Zhang, Yuan Wang. Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. Entire space multi-task modeling via post-click behavior decomposition for conversion rate prediction. In SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 2377-2386, 2020.
[5] Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss is its own reward: Self- supervision for reinforcement learning. In ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017.
[6] Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2019.
[7] Yabin Zhang, Hui Tang, and Kui Jia. Fine-grained visual categorization using meta-learning optimization with sample selection of auxiliary data. In Computer Vision ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII, pages 241-256, 2018.
[8] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference computer vision, pages 94-108, 2014.
Yunshu Du Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, and Balaji Lakshmi- narayanan. Adapting auxiliary losses using gradient similarity. CoRR, 2018.
[10] Baifeng Shi, Judy Hoffman, Kate Saeıko, Trevor Darrell, and Huijuan Xu. Auxiliary task reweighting for minimum-data learning. In NeurlPS 2020. December 6- 2020, virtual, 2020.
[11] Hong Chen, Xin Wang, Chaoyu Guan, Yue Liu, and Wenwu Zhu. Auxiliary learning with joint task and data scheduling. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pages 3634-3666 2022.
[12] Hong Chen. Xin Wang, Yue Liu, Yuwei Zhou, Chaoyu Guan, and Wenwu Zhu. Module-aware optimization for auxiliary learning. In Advances in Neural Information Processing Systems, 2022.
[13] Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.0309) 2022.
[14] Hong Chen, Xin Wang, Ruobing Xie, Yuwei 7hou, and Wenwu 7hu. Cross-domain recommendation with behavioral importance perception. In Proceedings the ACM Web Conference 2023, pages 1294-1304, 2023.
[15] Michael Crawshaw. Multi-task learning with deep neural networks: A survey, arXiv preprint
arXiv:2009009996 2020.
[16] Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Lue Van Gool. Branched multi-task networks: deciding what layers to share. arXiv preprint 1rX.v21904.02900, 2019.
11

---

[17] Pengsheng Guo, Chen-Yu Yu Lee. and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning, pages 3854-3863. PMLR, 2020.
[18] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7482-7491, 2018.
[19] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich, Gradnorm: Gradient normal- ization for adaptive loss balancing deep multitask networks. ICML 2018, pages 793 802. PMLR, 2018.
[20] Song. Yixin Chen, Xinchao Wang, Chengchao Shen, and Mingli Song. Deep model transferability from attribution maps. Advances in Neural Information Processing Systems, 32, 2019.
[21] Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[22] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7 10, 2016.
[23] Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, and Xiuqiang He. An embedding learning framework for numerical features in prediction. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2910-2918 18, 2021.
[24] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Ouline [Palermo, Sicily, Italy], Proceedings of Machine Learning Research, pages 15502 PMLR, 2020.
[25] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-201 1-011, California Institute of Technology, 2011.
[26] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4), 2009.
[27] Ruining He and Julian J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Weh, WWW 2016, Montreal, Canada, April 11 15, 2016, pages 507-517. ACM, 2016.
[28] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505, 2012.
[29] Kaiming He,, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, IEEE Computer Society, 2016.
[30] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019, pages 1161-1170, ACM, 2019.
[31] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171.0997 2017.
[32] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Toannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
12


Reviews:
[{"review_id": "unV8sl1VF1", "reviewer": "Reviewer_SrwV", "summary": "This paper proposes an auxiliary learning framework that jointly generates new auxiliary data and new labels, to tackle the negative transfer problem in auxiliary learning. The framework is designed to be generic to various auxiliary scenarios, both multiple input and single input, regression and classification. The bi-level optimization strategy is designed to optimize the task learning model and the joint generation model. Experiments on several benchmarks show that the proposed method outperforms existing methods.", "questions": "Please refer to the weakness part. Although I think this paper is overall of good quality, I want to know how it can adapt to the current pretrained models.", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper is very clear and well-written.\n2.\tThe idea to generate new data and task to avoid the potential negative transfer in manually designed auxiliary tasks is novel and reasonable.\n3.\tThe relations and differences compared to previous works are clearly discussed.\n4.\tThe proposed method is generic and makes sense.\n5.\tExperiments are well conducted, and the effectiveness of each component is well validated. The visualizations are reasonable and interesting.\n", "weaknesses": "1.\tIn the single input scenario, the proposed method behaves like MixUp, and I think the comparison with MixUp is important and should be put to the main manuscript instead of the appendix, although I understand the page is limited.\n2.\tConsidering that current large pretrained models usually have a lot of parameters, if we want to finetune these models, as the limitation part discussed, we will need more computational cost. Whether this method can serve the current pretrain-tuning paradigm will largely influences its significance.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the constructive comments and suggestions. We address your concerns point by point as follows\n\n1. **Rely to Weakness 1**:  \n    **Comparison with MixUp**: \n    We agree with the reviewer that comparison with MixUp is important in the shared-input scenario, so in Appendix 5.4, we conduct extensive experiments and list the advantages of our method as the reviewer pointed out. We are glad to include it into the main manuscript in our final version.\n\n2. **Reply to Weakness 2**: \n\n    **Applying our method to large pretrained models**: Applying our method to directly finetune all the parameters in large pretrained models will indeed result in high computational cost, which is the limitation of bi-level optimization. However, luckily, there are many parameter-efficient finetuning methods, like LoRA[1], Adapter[2], or even we can choose partial parameters to finetune while fixing most of the pretrained parameters, then the computational cost will be largely reduced, and our method can be easily applied to large pretrained models. It is a very interesting future work and we will try to extend this work to large pretrained models. Thanks for your suggestions.\n\nWe are glad to answer further questions if necessary.\n\n[1] Houlsby, Neil, et al. \"Parameter-efficient transfer learning for NLP.\" International Conference on Machine Learning. PMLR, 2019.\n\n[2] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021)."}, {"review_id": "uYD68C8QdZ", "reviewer": "Reviewer_HDHq", "summary": "Motivated by the manually collected auxiliary data and tasks in auxiliary learning may have negative impacts on the primary task, this paper proposes an auxiliary learning framework that jointly generates new auxiliary data and task. To generate new data for various auxiliary scenarios, the authors design an adaptive feature generator and a label generator, which considers both linear and nonlinear relations. Extensive experiments show that the proposed method can bring benefits to the primary task, even when the manually collected auxiliary data and task are potentially harmful.", "questions": "See above.", "limitations": "I don't see any concerns regarding the societal impact in this work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "\n1.\tThe paper is well-motivated and tackles an important problem in auxiliary learning. To collect the beneficial auxiliary data and task really needs human trials and domain knowledge.\n2.\tThe paper is well organized with clear discussion about related works, reasonable method formulation, and experiments to support the claims. \n3.\tThe experiments are conducted across different domains and datasets to show the effectiveness and generality of the proposed method. \n4.\tSome ablations bring interesting insights.", "weaknesses": "\n1.\tSince this work focuses on using generation to tackle the auxiliary learning problem, why not use some pretrained generative models like Stable Diffusion? The Stable Diffusion can also use conditional generation which gives similar data as provided by Table 3, and the paper can be strengthen with some discussion about this.\n2.\tThe most advanced models are pretrained, whether the proposed method can be applied to pretrained models is not clear.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful comments and constructive suggestions. We address your concerns point by point as follows.\n\n1. **Reply to Weakness1**:\n\n    **Discussion about generation with stable diffusion**: Stable Diffusion is a pretrained text-to-image generation model, and it relies on large datasets to train. It can be used to generate image data and with the text as the labels. Our method is superior to Stable Diffusion in auxiliary learning in the following two ways:\n    + Stable diffusion can be only used to generate auxiliary data for image classification task, but our proposed method can generate auxiliary data and tasks for different types of data, such as image, and tabular data.\n\n    + Even in the image generation scenario, the generated data by Stable diffusion is not necessarily helpful to the primary task, but the bi-level optimization in our work guarantees that the generated data and task are beneficial to the primary task.\n\n    Therefore, our proposed method is quite meaningful in auxiliary learning despite the existence of Stable Diffusion. We will include the discussion in our final version. Thanks for your suggestions.\n\n2. **Reply to Weakness2**:\n\n    **Whether the proposed method works when the model is pretrained**: In our CUB and CUB-fewshot experiments, the used backbones are pretrained ResNet18 and ResNet50, and we finetune them on CUB dataset, and this detailed experimental setting is given in Appendix 4.1. Additionally, we follow the setting of [1] to use the pretrained EfficientNet to finetune on the NYUv2 dataset to help the semantic segmentation primary task. The results are shown in the following table, which show that our proposed method can work well when conduct auxiliary finetuning. Therefore, our method can work well when applied to pretrained models.\n    |  NYUv2   | SLL  | Equal | HPO-tune | Uncert | GCS| AuxL  |MAOAL | ours|\n    |  ----  | ----  | ----|----  | ----  | ----|----  | ----  | ----|\n    | m-IOU(%) $\\uparrow$  |$32.61_{0.21}$  | $33.15_{0.15}$| $33.09_{0.25}$| $32.93_{0.32}$| $32.18_{0.34}$| $33.06_{0.14}$| $33.88_{0.23}$|  $\\textbf{34.46}_{0.22}$|\n    | pixel-acc(%) $\\uparrow$ | $68.42_{0.62}$ |$68.11_{0.24}$|$68.52_{0.52}$| $68.74_{0.62}$| $ 67.58_{0.80}$|$68.36_{0.53}$| $69.17_{0.44}$| $\\textbf{70.12}_{0.53}$|\n\nWe are glad to answer further questions if necessary.\n\n[1] Chen, Hong, et al. \"Module-Aware Optimization for Auxiliary Learning.\" Advances in Neural Information Processing Systems 35 (2022): 31827-31840.\n"}, {"review_id": "5tWVBSX0hC", "reviewer": "Reviewer_iWiK", "summary": "This paper studies the problem of learning with auxiliary tasks. Motivated by the observation that in most scenario the auxiliary data is hard to collect, the authors propose to train a model that can automatically generate auxiliary data (and labels) for any task at hand. Specifically, the authors propose a data generator network that generates new data based on the current data, and the network is learned by a bi-level optimization where the final objective is to maximize the performance on the validation set of the primary task. The authors show the advantage of the proposed method on several datasets and settings.", "questions": "See Weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear and sensitive and the proposed method is sound. \n\n- The idea of auxiliary data generation might not be completely new but the authors propose a unified way that can generate data for different tasks and different input/output format, which is a good contribution to me. \n\n- The paper is overall well written and easy to follow.", "weaknesses": "- To tackle with tasks with different input format, the authors propose to generate the features instead of the raw data. This may be a problem because in this way only the task-specific heads are trained on the generated data, but the backbone is still only learned on the primary task (as well as the already existed auxiliary tasks). This is suboptimal and might limit the impact of the generated data. Maybe the authors could comment on this?\n\n- In the feature linear interpolation, are the masks (L143) input-independent? \n\n- The instance regularization loss for numerical labels (Equation 14), if the generated labels have one label that is extremely large, and other labels that are the same and extremely small, then the $\\bar{y}_{g,j}$ will be 1 for one label and 0 for others, which will minimize the regularization loss. This seems not encouraging the diversity of the labels.\n\n- One question I have is about the visualization of the generated data (Table 3). I wonder where do the Max-Prob Images come from? Are they the images that have the maximal probability (predicted by the trained model) on the label? Have the authors tried to visualize the images in the training set which have the feature that is closest to the generated feature, i.e., using the generated feature to do a knn search in the training set?\n\n- Have the authors tried the setting where there is no ready-to-use auxiliary tasks and only the primary task is available?\n\n\n===========\n\nPost rebuttal: The rebuttal partially address my concerns and I will keep my score of weak accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the detailed comments and constructive suggestions. We address your concerns point by point as follows.\n1. **Reply to Weakness 1**:\n\n    **The impact of generated data and task on the backbone**: The generated data not only influences the task-specific head but also the backbone. Note that in Figure 1, the loss of the generated auxiliary task relies on the generated feature and label, and the generated feature is obtained from the existing features through the feature generator, and the existing features are extracted using the backbone. Therefore, the loss of the generated auxiliary task will also have its gradient on the backbone, and improves the performance of the backbone on the primary task. Thanks for your careful reading and thinking, and we will add one paragraph to explain it in our final version. \n\n2. **Reply to Weakness 2**:\n\n    **Whether the mask(L143) is input-independent**: Yes, in our implementation, we use a global mask for all the data and find it works well for various scenarios, but it will be an interesting problem whether designing some sample-specific masks can further improve the performance in our future work. \n\n3. **Reply to Weakness 3**:\n    \n    **Instance regularization for numerical labels**: If only using the regularization, the generated label will be not diverse as the reviewer pointed out. However, there is also the implicit gradient to make the generated label helpful to the primary task. Both the implicit gradient and the explicit gradient working together can cause a beneficial joint generation as shown in Table 4, without using the instance regularization will do harm to the recommendation performance. Additionally, in some other particular scenarios, users may add a hyperparameter to control the impact of the regularization.    \n\n4. **Reply to Weakness 4**:\n    \n    **About the visualization**: We find the Max-Prob images with the following process: we first find the largest peak of the generated label, e.g., 7 in the 2nd row in Table 3, and then we will see in the training set what images have label 7, and then put the image of category 7 in Table 3. Then, for the second largest peak which is label 4, we also find its corresponding image in the training set with category 4. \n    \n    The generation visualization aims to find whether the generated task (label) has some semantic meanings, and the answer is yes as explained in L330-356. Considering that the generated task and the primary task do not share the same task head (indicating that their classification boundaries for the features are different, and the features of the auxiliary task and the primary task are not in the same space), visualizing the images searched by features may not be that meaningful.\n\n5. **Reply to Weakness 5**:\n\n    **Whether we try there are no ready-to-use auxiliary tasks**: In Appendix 5.2, we try to remove the auxiliary labels, and for reading convenience, we report the results in the following table (best performance is bolded and the second is underlined). We find that:\n    + Our method without auxiliary labels, ours (w/o aux), outperforms STL and all the baselines, indicating our method can be applied when there are no auxiliary labels.\n    + Our complete method with the auxiliary labels, further improves the performance by converting the information in the auxiliary labels into a beneficial form to the primary task. \n    \n    |                      | STL           |    Equal      | Uncert        | GCS           | AuxL          | JTDS          | MAXL       | ours (w/o aux) | ours          |\n    |----------------------|---------------|---------------|---------------|---------------|---------------|---------------|------------|----------------|---------------|\n    | CUB(ResNet50)/Acc(%) $\\uparrow$ | $80.46_{0.42}$    | $78.28_{0.23}$    | $77.03_{0.24}$    | $78.54_{0.54}$    | $78.00_{0.55}$    | $79.34_{0.19}$    | $78.48_{0.88}$ | $\\underline{80.84}_{0.69}$     | $\\textbf{81.73}_{0.20}$    |\n    | Movies/RMSE    $\\downarrow$      | $1.0456_{0.0008}$ | $1.0459_{0.0009}$ | $1.0495_{0.0018}$ | $1.0459_{0.0018}$ | $1.0483_{0.0019}$ | $1.0504_{0.0023}$ | -          | $\\underline{1.0442}_{0.0003}$  | $\\textbf{1.0426}_{0.0009}$ |\n    We are glad to answer further questions if necessary.\n\n"}, {"review_id": "dJ1KgCmKY7", "reviewer": "Reviewer_Gctb", "summary": "This paper focuses on Auxiliary Learning, a technique aiming to improve the model generalization ability on the primary task with the help of related auxiliary tasks, and proposes a novel joint data-task generation framework for auxiliary learning (DTG-AuxL). DTG-AuxL is designed to mitigate certain issues present in existing methods. Specifically, existing methods necessitate the presence of beneficial information in the auxiliary data and tasks; otherwise, the useless auxiliary information will do harm to the primary task when the involved auxiliary data and tasks are improperly collected. DTG-AuxL features a joint generator and a bi-level optimization strategy and can simultaneously generate auxiliary data and tasks in a joint manner. Extensive experiments show that DTG-AuxL outperforms existing methods in various auxiliary learning scenarios.", "questions": "See my questions in Weaknesses.", "limitations": "No comments.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper proposes DTG-AuxL, a joint data-task generation framework applicable in various auxiliary learning scenarios. DTG-AuxL contains a joint generator and a bi-level optimization strategy and can simultaneously generate auxiliary data and tasks in a joint manner, which is the first attempt to do so.\n\nDTG-AuxL can adapt to different kinds of input data, such as tabular data and visual images. Because the data generation process is conducted in the feature space and the label generator can accommodate both categorical labels for classification and numeric labels for regression. Consequently, DTG-AuxL has wide applicability.\n\nHow to effectively optimize the parameters in the generation framework is a challenging problem to guarantee the generated data and task are beneficial to the primary task. This paper proposes a bi-level optimization strategy as well as instance regularization to tackle this challenge.\n\nExtensive experiments on two scenarios validate the generalization ability of DTG-AuxL. The first scenario is that the auxiliary tasks share the same input as the primary task, and the second scenario is that the inputs of the primary and auxiliary tasks are different. Experimental results show that DTG-AuxL outperforms existing methods on diversified scenarios, including few-shot learning scenarios. Specifically, DTG-AuxL is able to convert originally harmful auxiliary labels into beneficial forms.\n", "weaknesses": "Table 3 and Table 4 is not appropriate.\n\nThis paper conducts experiments on the following scenarios:\n\n1. Share Input: visual inputs & classification task (primary) + visual inputs & classification task (auxiliary)\n2. Share Input: tabular inputs & regression task (primary) + tabular inputs & classification task (auxiliary)\n3. Different Input: visual inputs & classification task (primary) + visual inputs & classification task (auxiliary)\n\nBut in order to verify the broad applicability of DTG-AuxL in the paper, more different types of experiments need to be done, such as Different Input: tabular inputs & classification task (primary) + tabular inputs & classification task (auxiliary).\n\nFinally, I am a little confused, does the so-called task generation refer to label generation? Why can't label be regarded as part of data? Shouldn't task generation literally include loss functions or other ingredients?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the detailed comments and constructive suggestions. We address your concerns point by point as follows.\n\n1.  **About Table 3 and Table 4**: We will make them better formatted in our final version. Thanks for your suggestion.\n\n2. **More experiments**: We follow the reviewer's suggestions to conduct experiments on the setting Different Input: Tabular inputs, primary classification + auxiliary classification. \n\n    Specifically, we consider the cross-domain CTR prediction in recommendation, i.e., using the user behaviors in the Movies domain to help the target Toys domain CTR prediction, and evaluate the results with AUC metric(higher indicates better). The results are shown as follows,\n\n    |   Method  | STL  | Equal | Uncert | GCS | AuxL | JTDS | MAXL| ours |\n    |  ----  | ----  | ----|----  | ----|----| ----| ----| ----|\n    |  AUC $\\uparrow$ | $0.6508_{0.0053}$| $0.6486_{0.0071}$| $0.6685_{0.0050}$ | $0.6661_{0.6664}$| $0.6513_{0.0058}$|__|$0.6628_{0.0049}$|$\\textbf{0.6787}_{0.0068}$|\n   \n   Additionally, we follow [1] to conduct experiments on the NYUv2 dataset to use the depth prediction and normal prediction to help the image segmentation, which is share input, image data, primary classification, and auxiliary regression, and the results are shown as follows (Note that JTDS and MAXL cannot be applied to this scenario),\n   \n    |  NYUv2   | SLL  | Equal | HPO-tune | Uncert | GCS| AuxL  |MAOAL | ours|\n    |  ----  | ----  | ----|----  | ----  | ----|----  | ----  | ----|\n    | m-IOU(%) $\\uparrow$  |$32.61_{0.21}$  | $33.15_{0.15}$| $33.09_{0.25}$| $32.93_{0.32}$| $32.18_{0.34}$| $33.06_{0.14}$| $33.88_{0.23}$|  $\\textbf{34.46}_{0.22}$|\n    | pixel-acc(%) $\\uparrow$ | $68.42_{0.62}$ |$68.11_{0.24}$|$68.52_{0.52}$| $68.74_{0.62}$| $ 67.58_{0.80}$|$68.36_{0.53}$| $69.17_{0.44}$| $\\textbf{70.12}_{0.53}$|\n\n    The results show that our proposed method outperforms all the baselines, which further shows the effectiveness of our method. We believe these results can better support our paper. Thanks for your suggestions.\n\n3. **About task generation:** Yes, we follow previous works [2,3] to call the label generation as task generation, and they generate a new label for the input data and state that they generate an auxiliary task for the input data. As a matter of fact, in many shared-input multi-task learning scenarios, and we have several labels for the input (e.g., in NYUv2, one input image, and the segmentation map, the normal vector and the depth map as the labels). Predicting each of the labels defines a task (e.g., semantic segmentation, depth prediction and normal prediction), and predicting all of them together is called multi-task learning. Considering these literature, we call generating a new label as task generation.\n\n[1] Chen, Hong, et al. \"Module-Aware Optimization for Auxiliary Learning.\" Advances in Neural Information Processing Systems 35 (2022): 31827-31840.\n\n[2] Liu, Shikun, Andrew Davison, and Edward Johns. \"Self-supervised generalisation with meta auxiliary learning.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[3] Navon, Aviv, et al. \"Auxiliary Learning by Implicit Differentiation.\" International Conference on Learning Representations. 2020."}, {"review_id": "9ZZj5VHkbn", "reviewer": "Reviewer_81dw", "summary": "This paper tackles the problem of auxiliary learning, specifically for cases where the provided auxiliary data actually harms the primary task.\nA joint data-task generation framework is proposed, which generates new auxiliary tasks during the optimization process.\nThe label generator generates data in the feature space and includes both linear and nonlinear interaction terms to produce more expressive data.\nBi-level optimization is used to optimize the entire structure, with instance regularization preventing mode collapse of the generated data.\nExperimental results show that the proposed structure can outperform existing methods in a variety of auxiliary learning setups and even prevent performance degradation in certain cases.", "questions": "See weaknesses.\n\nWriting:\n- The proposed method is akin to MixUp in the feature space, thus I believe the experimental results in section 5.4 in the supplementary material is important and can perhaps be moved to the main paper.\n- Experiments:\n  - Please add citations for the methods in the experiment tables so it is easier to refer to the corresponding method.\n  - Add performance of [3] to Table 1/2.\n  - Please fix the formatting of Table 3 and 4, as their current positioning makes this page difficult to read. Table 3 should be a figure and can be column width. The space between Table 4 and the text is so small.\n\n[3] Chen, Hong, et al. \"Module-Aware Optimization for Auxiliary Learning.\" Advances in Neural Information Processing Systems 35 (2022): 31827-31840.", "limitations": "The limitations of the work is provided in the supplementary material. I suggest the authors to include at least a short summary in the main paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method has strong results in auxiliary learning benchmarks and even prevents performance drop with harmful auxiliary tasks, as shown in the Movies and Pet-CUB experiments. Table 2 also shows its effectiveness in low-data regimes.\n- Evaluation and training settings are fully explained in the paper and supplementary material, which helps with understanding and reproducing the results.", "weaknesses": "- The idea of bi-level optimization strategy has been proposed and used by prior work [1,2,3]. Thus, this contribution cannot really be claimed by this work. Also, instance regularization is simply an application of the regularization loss introduced in [1].\n- Experiments are conducted in limited scale:\n  - Most tasks only have a single auxiliary task and the tasks are simple classification or regression problems. Can more auxiliary tasks be used and more complex tasks also be studied as done in previous work, such as on NYUv2 [2]?\n  - The extension to dense prediction (e.g., semantic segmentation in NYUv2) seems not so trivial with the proposed framework. It will be beneficial if the authors can demonstrate how this can work with the proposed framework.\n- While the improvement even in the presence of harmful auxiliary tasks is a good result, the explanation for why this can be achieved and why previous methods fail is lacking.\n  - In section 5.2 of the supplementary material, it seems like even removing auxiliary label information the performance is still superior to STL. Why is this?\n\n\n[1] Liu, Shikun, Andrew Davison, and Edward Johns. \"Self-supervised generalisation with meta auxiliary learning.\" Advances in Neural Information Processing Systems 32 (2019).\n[2] Navon, Aviv, et al. \"Auxiliary learning by implicit differentiation.\" arXiv preprint arXiv:2007.02693 (2020).\n[3] Chen, Hong, et al. \"Module-Aware Optimization for Auxiliary Learning.\" Advances in Neural Information Processing Systems 35 (2022): 31827-31840.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the detailed comments and constructive suggestions. We address your concerns point by point as follows.\n\n1. **About the optimization strategy**: The bi-level optimization has been adopted by previous works[1,2,3], but the instance regularization we adopt is quite different from the regularization in [1] in the following two ways:\n    + [1] calculate the regularization in the class level, which aims to prevent the labels from being a uniform distribution on different classes, but our regularization focuses on the sample level, we want to prevent the labels of all samples from being the same to a fixed point as shown in Table A7 in Appendix 5.5.\n    + The regularization in [1] can be only applied to categorical labels, but our method can be applied to both categorical and numeric labels.\n\n2. **Extension to NYUv2**: It is not hard to extend our method to NYUv2. First, NYUv2 belongs to the shared input scenarios. Our framework can handle this scenario. Specifically, for each input image, we obtain its feature, and use the input feature with another image's feature to generate a new feature with the feature generator, which is not hard. Additionally, for each image and another image, we have 3 labels for each of them, segmentation map with size (height, width, category_num), depth map with size (height, width, 1) and normal label with size (height, width, 3), when generating new labels, only the final dimension contains label information, therefore, the label generator can generate a (height, width, category_num)-size new label which is the same size as that of the primary segmentation task. That is to say, the label generator uses different labels in a pixel and uses their linear and nonlinear relations to generate a new label for the same pixel. With this principle, we follow [3] to use the EfficientNet as the backbone and the results are shown in the following table,\n\n    |  NYUv2   | SLL  | Equal | HPO-tune | Uncert | GCS| AuxL  |MAOAL | ours|\n    |  ----  | ----  | ----|----  | ----  | ----|----  | ----  | ----|\n    | m-IOU(%) $\\uparrow$  |$32.61_{0.21}$  | $33.15_{0.15}$| $33.09_{0.25}$| $32.93_{0.32}$| $32.18_{0.34}$| $33.06_{0.14}$| $33.88_{0.23}$|  $\\textbf{34.46}_{0.22}$|\n    | pixel-acc(%) $\\uparrow$ | $68.42_{0.62}$ |$68.11_{0.24}$|$68.52_{0.52}$| $68.74_{0.62}$| $ 67.58_{0.80}$|$68.36_{0.53}$| $69.17_{0.44}$| $\\textbf{70.12}_{0.53}$|\n\n    The results show that our proposed method outperforms all the baselines, and our method can be applied to this scenario.\n\n3. **Why our method achieves improvement**: Compared to the existing methods, our advantage lies in the information brought by the generated new data and task. Specifically, current weighting methods limit their performance because the existing auxiliary data and tasks are unhelpful. Current auxiliary generation methods have limited performance because they ignore the data generation and limit their label to fine-grained labels, which may be not proper for all the scenarios. \n    + In Appendix 5.2, we remove the auxiliary labels from the input of the label generator, which means we only use the primary labels to generate the new label (Note that in Figure 1 in the main manuscript, both existing the primary labels and auxiliary labels will be used as the input of the label generator.). The performance of w/o aux is better than STL means that only using the primary labels to generate new auxiliary label, our method can still bring performance improvement. This phenomenon also means our method can be used to generate auxiliary data and tasks even when there are only primary labels but no auxiliary labels. \n\n4. **Writing**: \n    + We are glad to include section 5.4 in our final version.\n    + We will add citations to the table.\n    + Part of the performance of [3] is listed in the following table and we will include the results on other settings and datasets in our final version.\n    \n    | ResNet50    |MAOAL[3]  | ours | \n    |  ----  | ----  | ----|\n    | CUB  |$79.68_{0.45}$  | $81.73_{0.20}$| \n    | CUB-5shot | $45.53_{0.67}$ |$52.33_{1.36}$|\n    + We will make the tables well formatted in our final version.\n    + We will include a summary of the limitations in the main paper in our final version.\n\nThanks for your suggestions. We believe the experiments in the rebuttal will further strengthen our work and the writing suggestions will improve the readability. We are glad to answer further questions if necessary. \n\n\n[1] Liu, Shikun, Andrew Davison, and Edward Johns. \"Self-supervised generalisation with meta auxiliary learning.\" Advances in Neural Information Processing Systems 32 (2019). \n\n[2] Navon, Aviv, et al. \"Auxiliary learning by implicit differentiation.\" arXiv preprint arXiv:2007.02693 (2020). \n\n[3] Chen, Hong, et al. \"Module-Aware Optimization for Auxiliary Learning.\" Advances in Neural Information Processing Systems 35 (2022): 31827-31840."}]

Author rebuttal:

Meta-review: This paper studies the problem of learning with auxiliary tasks, specifically where the provided auxiliary data harms the primary task. Based on the observation that auxiliary data is hard to collect in most cases, a joint data-task generation framework (DTG-AuxL) is proposed and the authors design a feature generator and a label generator for various auxiliary scenarios. Extensive experiments successfully validate the effectiveness of the proposed method and some ablations bring interesting insights. Overall, all reviewers agreed with the contribution of this paper and the significance to the community can be further improved if DTG-AuxL is still effective on large pre-trained models.
Decision: Accept (poster)