Title: Joint Data-Task Generation for Auxiliary Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: auxiliary learning

data-task joint generation
Abstract: Current auxiliary learning methods mainly adopt the methodology of reweighing losses for the manually collected auxiliary data and tasks. However, these methods heavily rely on domain knowledge during data collection, which may be hardly available in reality. Therefore, current methods will become less effective and even do harm to the primary task when unhelpful auxiliary data and tasks are employed. To tackle the problem, we propose a joint data-task generation framework for auxiliary learning (DTG-AuxL), which can bring benefits to the primary task by generating the new auxiliary data and task in a joint manner. The proposed DTG-AuxL framework contains a joint generator and a bi-level optimization strategy. Specifically, the joint generator contains a feature generator and a label generator, which are designed to be applicable  and expressive for various auxiliary learning scenarios. The bi-level optimization strategy optimizes the joint generator and the task learning model, where the joint generator is effectively optimized in the upper level via the implicit gradient from the primary loss and the explicit gradient of our proposed instance regularization, while the task learning model is optimized in the lower level by the generated data and task. Extensive experiments show that our proposed DTG-AuxL framework consistently outperforms existing methods in various auxiliary learning scenarios, particularly when the manually collected auxiliary data and tasks are unhelpful.

Joint Data-Task Generation for Auxiliary Learning

Abstract

Current auxiliary learning methods mainly adopt the methodology of reweighing losses for the manually collected auxiliary data and tasks. However, these methods heavily rely on domain knowledge during data collection, which may be hardly available in reality. Therefore, current methods will become less effective and even do harm to the primary task when unhelpful auxiliary data and tasks are employed. To tackle the problem, we propose a joint data-task generation framework for auxiliary learning (DTG-AuxL), which can bring benefits to the primary task by generating the new auxiliary data and task in a joint manner. The proposed DTGAuxL framework contains a joint generator and a bi-level optimization strategy. Specifically, the joint generator contains a feature generator and a label generator, which are designed to be applicable and expressive for various auxiliary learning scenarios. The bi-level optimization strategy optimizes the joint generator and the task learning model, where the joint generator is effectively optimized in the upper level via the implicit gradient from the primary loss and the explicit gradient of our proposed instance regularization, while the task learning model is optimized in the lower level by the generated data and task. Extensive experiments show that our proposed DTG-AuxL framework consistently outperforms existing methods in various auxiliary learning scenarios, particularly when the manually collected auxiliary data and tasks are unhelpful.

1 Introduction

Auxiliary learning aims to improve the model generalization ability on the primary task with the help of related auxiliary tasks [1, 2]. This learning paradigm has been widely adopted and has shown its effectiveness in various areas, like image classification [3], recommendation [4] and reinforcement learning [5, 6]. Different auxiliary tasks are often chosen manually according to the primary task, e.g., [7] utilize the task of visual attribute classification to help the fine-grained bird classification and [4] improve the click conversion rate prediction with the help of click-through rate prediction task.

Most existing works utilize the auxiliary information by first reweighing the losses of the auxiliary data and tasks, and then use the sum of the weighted losses together with the primary loss to optimize the task learning model. The weights are employed to balance the primary loss and the auxiliary losses to avoid negative auxiliary transfer, which are tuned with HPO tools [4, 3, 8]. More recent works [9, 6, 10, 1, 11] propose to dynamically weigh different auxiliary losses during training.

However, the existing methods require that there exists beneficial information in the auxiliary data and tasks, so that the beneficial loss terms can be selected through the loss reweighing process. This condition cannot always be satisfied because whether the auxiliary data or task is beneficial depends on many factors including the chosen auxiliary task , the scale of primary task dataset and the selected

learning model for the tasks [1, 11], making it difficult to manually collect the beneficial auxiliary data and task via prior knowledge. Therefore, existing approaches may adopt useless auxiliary information and finally do harm to the primary task when the involved auxiliary data and tasks are improperly collected, as observed in [1, 12]. Although [2] propose to generate fine-grained auxiliary classification tasks, this method can be only applied to the classification primary task. Additionally, they only generate the new task without considering new data, while the data-level information is pointed out to be an important factor in auxiliary learning [11].

To address the problem, in this paper, we propose to simultaneously generate the auxiliary data and task for auxiliary learning in a joint manner. However, there are three challenges for the joint generation. First, it is challenging to design a generic framework that can accommodate various tasks with different inputs. This is because the types of data and tasks are quite diversified in different auxiliary learning scenarios, e.g., the primary task of image classification takes visual images as input and outputs categorical labels for classification [7], while the primary task of rating prediction for recommendation takes tabular data as input and outputs numeric labels for regression [4]. Second, it is challenging to develop a generation framework that is expressive enough to produce beneficial data and tasks for the primary task. Finally, to guarantee the jointly generated auxiliary data and task are beneficial to primary task, how to effectively optimize the parameters in the generation framework is a challenging problem as well.

To tackle these challenges, we propose a joint Data-Task Generation framework for Auxiliary Learning (DTG-AuxL), which involves a joint generator and a bi-level optimization strategy. Specifically, the joint generator consists of a feature generator and a label generator, which generates the new auxiliary data and new task based on the existing manually collected data and tasks. The data generation process is conducted in the feature space so that it can accommodate various input types, while the label generator architecture is inspired by the model in recommendation which can accommodate both categorical labels for classification and numeric labels for regression. Moreover, we introduce nonlinear interaction terms in the joint generator, making it more expressive to produce beneficial auxiliary data and tasks. To effectively optimize the joint generator and the task learning model, we propose the bi-level optimization strategy with instance regularization. In the lower optimization, the task learning model is optimized by the generated auxiliary data and task. In the upper optimization for the joint generator, we not only utilize the implicit gradient from the primary loss but also the explicit gradient from the proposed instance regularization, to avoid label generation mode collapse. Extensive experiments show that DTG-AuxL outperforms existing methods in various auxiliary learning scenarios, especially when the manually collected auxiliary data and task are unhelpful to the primary task. We summarize our contributions as follows,

We propose to simultaneously generate auxiliary data and tasks in a joint manner in auxiliary learning for the first time, to the best of our knowledge.

We propose DTG-AuxL, a joint data-task generation framework applicable in various auxiliary learning scenarios, containing a joint generator and a bi-level optimization strategy.

Extensive experiments in various auxiliary learning scenarios demonstrate the superiority of our proposed DTG-AuxL framework. We further analyze when and how DTG-AuxL works to bring performance boost.

2 Related Work

Auxiliary Learning The most widely adopted way in auxiliary learning is to combine the loss of the primary task and the losses of the auxiliary tasks in a linear way [3, 7, 4, 5], where the linear weights are tuned manually or with HPO methods. More recent works [9, 6, 10] propose to automatically assign dynamic weights to the auxiliary losses. [9] propose to assign weights to each loss based on the cosine gradient similarity between the primary loss and each auxiliary loss. Later work [10] aims to make the weighted auxiliary gradient sum closest to the gradient of the primary loss. [13, 1] utilize the bi-level optimization strategy to learn the auxiliary weights, where [1] even propose to combine the losses not only limited to a linear form, but also a nonlinear form. [11, 14] point out that only considering the task-level information is not sufficient, so they jointly consider the weights of different tasks and different data samples within the same task through a joint selector. As aforementioned, these reweighing methods will easily fail to bring improvement when the chosen auxiliary tasks and data are unhelpful. There are also works [2, 1] that generate

𝑥

!,#

𝑥

$%,#

𝑥

$&,#

Backbone

…

𝑓

!,#

…

𝑓

$%,#

𝑓

$&,#

Primary Head

Aux Head 1

Aux Head K

…

𝑟

!,#

𝑟

$%,#

𝑟

$&,#

…

𝑦

!,#

𝑦

$%,#

𝑦

$&,#

…

Prediction

Label

𝐿

!

𝐿

$%

𝐿

$&

𝑓

!,#

…

𝑓

$%,#

𝑓

$&,#

𝑚

!

𝑚

"#

𝑚

"$

*

*

*

MLP

F

+ +

𝑓

",$

Aux Head G

𝑟

",$

𝑦

!,#

𝑦

$%,#

𝑦

$&,#

…

{

𝐸

%

}

$ 𝑦

&,$

𝐿

'

…

{

𝐸𝑁

%

}

…

𝑦

'(,!,#

𝑦

'(,$%,#

𝑦

'(,$&,#

…

𝑦

'(,#

𝑒

$&,#

𝑒

!,#

+

…

MLP

L

𝑦

'),#

+

𝑦

!,#

+

1 - α

𝐿

*

= 𝐿

!

+,

+,%

&

𝑤

+

𝐿

$+

+𝑤

'

𝐿

'

α

𝑦

',#

Task Learning Model(TLM)

𝜃

Joint Generator(JG)

𝜙

Feature Generator

TLM(

𝜃

)

∇

'

𝐿

(

TLM(

𝜃

)

∇

'

𝐿

(

TLM(

𝜃

)

TLM(

𝜃

)

∇

'

𝐿

(

…

TLM(

𝜃

)

1

st

update

2

nd

update

N

th

update

JG(

𝜙

)

JG(

𝜙

)

∇

)

𝐿

*

+∇

)

𝐿

+,&

Linear Embedding

Nonlinear Embedding

Input data

Features

bias

Label linear term

Label nonlinear term

Feature linear term

Feature nonlinear term

Generated feature

Generated label

upper optimization

for joint generator

lower optimization for

task learning model

Model Design

Optimization

fine-grained classification auxiliary tasks for the primary classification task. However, they can only be applied to a classification problem. Additionally, they only generate auxiliary tasks without new data, limiting their performance especially when the data of the primary task is inadequate, which is an often encountered scenario in auxiliary learning [9, 1].

Multi-task learning Multi-task learning aims to share information among tasks to improve model performance. However, the goal of multi-task learning is to obtain good performance for all the tasks, while auxiliary learning only focuses on the primary task. Multi-task learning methods can be categorized into three parts [15]: multi-task architecture design [16, 17], multi-task optimization [18, 19] and multi-task relationship learning [20], where the multi-task optimization methods involve techniques for optimizing several losses, like loss reweighing [18] and gradient modulation [21] .

3 The Proposed Method

The overall DTG-AuxL framework is shown in Figure 1. Next, we give the problem formulation, describe the designs of the joint generator, and present the proposed bi-level optimization strategy.

3.1 Preliminaries and Problem Formulation

In auxiliary learning, we have one primary task T p , and totally K auxiliary tasks { T ai } K i =1 . Each of these tasks has its corresponding training dataset, including the primary dataset D p = { ( x p,j , y p,j ) } | D p | j =1 , and the dataset for each auxiliary task D ai = { ( x ai,j , y ai,j ) } | D ai | j =1 , where | · | denotes the data sample number of the dataset. If the auxiliary tasks share the same input with the primary task, which is a widely encountered and studied scenario in previous works [11, 1, 9, 10], we have x ai,j = x p,j and | D ai | = | D p | . Besides the datasets, we have a task learning model parameterized by θ which is used to learn all the tasks together. There is also a validation dataset D v which is used to evaluate the model performance on the primary task. With these notations, the widely adopted training objective of auxiliary learning is:

where L p and L ai indicate the loss functions of the primary and each auxiliary task. Current reweighing methods focus on how to decide w i so that the task learning model θ can achieve the best

performance on D v . However, as aforementioned, the terms L ai ( D ai , θ ) are defined by the manually collected auxiliary data and tasks, with no guarantee to bring benefits for the primary task. Therefore, we propose to simultaneously generate the new beneficial auxiliary data and task in a joint manner, with the following training objective:

where the last term is the loss on the generated auxiliary data and task. D g ( ϕ ) = { x g,j , ˆ y g,j } | D g | j =1 is the generated dataset which contains the new data together with the new label defining the new task, and ϕ denotes the parameters of the joint generator that are used to generate D g . Since we only care about the performance of the primary task in auxiliary learning, we keep L g the same loss function as L p , which can be cross entropy or MSE loss, etc. Note that we still keep the original auxiliary loss terms in our training objective, so that it can accommodate the scenario where the existing manually collected auxiliary data and tasks are beneficial. The task weights w i and w g are also learnable, which will be optimized together with the generator parameters, and we uniformly denote them all as ϕ for convenience. Next, we discuss the details of the generator and how we optimize ϕ and θ .

3.2 Joint Generator Design

The joint generator involves a feature generator to generate new features and a label generator to generate a new task for the new feature, as shown in the Model Design of Figure 1.

3.2.1 Feature Generator

Since we expect that our data generator can tackle different types of data, a neat and elegant solution is to conduct the generation process in the feature space. In the feature space, different types of input data are transformed to vectors, so we can tackle these vectors in a unified way. Specifically, for the input from different tasks [ x p,j , x a 1 ,j , · · · , x aK,j ] , the task learning model will first map them to the feature space with an encoder which is generally noted as 'backbone', and then use different task-specific heads to tackle each of the tasks. We generate the new features based on the features extracted via the backbone. In another word, the input of the feature generator is a feature list [ f p,j , f a 1 ,j , · · · , f aK,j ] , where f p,j = f backbone ( x p,j ; θ ) is a d -dimension feature. We next describe how the new features are generated.

Feature linear term. A natural way to generate new features is to combine different features with linear masks. We assign an individual feature mask to each of the K +1 features, i.e., we have a mask list [ m p , m a 1 , · · · , m aK ] , where each mask is a d -dimension vector. Then, we denote the subscript set for the task IDs { p, a 1 , · · · , aK } as S , and the linear combination is conducted as follows:

where gl indicates 'generated linearly', ∗ means the element-wise multiplication, and ˆ m i is the normalized mask. m i [ k ] is the k th element of the mask m i , and is normalized by softmax with the elements in the same dimension in other masks. This linear combination with normalized mask works as feature selection from all the input features, and then combines them into a new one.

Feature nonlinear term. To make the generated features more expressive, we introduce an MLP (Multi-Layer-Perceptron) to capture the nonlinear feature interaction:

where gn indicates 'generated nonlinearly'. All the features are concatenated together and the nonlinear feature interactions are modeled by an MLP, whose output dimension is d . Finally, the generated feature is the sum of the linear and nonlinear term, i.e., f g,j = f gl,j + f gn,j .

3.2.2 Label Generator

The label generator aims to generate a proper label for the feature produced by the feature generator. We use the labels of all the input data as the input of the label generator to make the generated label expressive enough, i.e., the input of the label generator is a list of labels [ y p,j , y a 1 ,j , · · · , y aK,j ] .

However, this label list may contain both numeric labels (if the corresponding task is regression) and categorical labels (if the corresponding task is classification). How to utilize different types of label information to generate a new label is the key problem. Inspired by the CTR (click-through rate) model in recommendation [22, 23], which predicts the user's preference towards an item also based on both their numeric and categorical features, we design the label generator with both linear terms and deep nonlinear terms similar to those in the CTR model.

Label linear term. The linear term models the direct and independent relationship between each input label and the generated label. We simply keep the dimension of the generated label the same as the primary label y p,j and more adaptive ways to choose the dimension can be an interesting future work. Here, we assume that the dimension of y p,j is m , where m = 1 if the primary task T p is regression, or m equals to the number of the total categories if T p is classification. If the primary task is regression, the generated label is also a scalar value for regression, while if the primary task is classification, the generated label is an m -dimension probability distribution vector. Specifically, for each task ID i ∈ S , if T i is a classification task and it has totally d i categories, there is an embedding table E i with dimension d i × m for this task. We directly map the label y i,j in task T i to its m -dimension embedding space through the embedding table:

If T i is a regression task, we follow [23] to map y i,j to its m -dimension embedding. Specifically, we also have an embedding table E i for this task T i , whose dimension is d i × m , where d i = H is a hyper-parameter which is fixed to 10 in our experiments. Then y i,j is mapped into the m -dimension embedding space as follows:

where the numeric label y i,j is linearly transformed to a H -dimension vector c i,j , which is used to weigh all the embeddings in the embedding table to obtain the final embedding. Then the final label linear term is obtained by the sum of all the input label embeddings y gl,j = ∑ i ∈ S y gl,i,j .

Label nonlinear term. In addition to the label linear term capturing how each input label independently influences the final generated label, we also propose a nonlinear term to model the influence of more complex label interactions on the generated label. Specifically, we introduce another group of embedding tables for all the input tasks { EN i } i ∈ S , where the ways to obtain the embeddings of the categorical and numeric labels are the same as eq. (5) and eq. (6), respectively. The dimension of EN i is d i × m n , where d i equals to the number of total categories if T i is classification. d i equals to H (i.e.,10) for regression task, and m n is a hyper-parameter. With these embedding tables, we can map all the input label list [ y p,j , y a 1 ,j , · · · , y aK,j ] to an embedding list [ e p,j , e a 1 ,j , · · · , e aK,j ] . Then the nonlinear label interactions are captured through an MLP:

The final generated label for the generated feature f g,j is the sum of the linear and nonlinear terms,

Label bias term. Besides the linear and nonlinear terms, we also introduce a label bias term that guides the generated label to possess similar semantic meaning to the label from the target primary task, y p,j . As such, we add y p,j as the label bias term as follows,

where α ∈ (0 , 1) is a learnable parameter initialized as 0.5 and norm ( · ) is the normalization function. If the primary task is classification, norm ( · ) will be softmax ( · ) and y p,j will be converted to the one-hot form. If the primary task is regression with range ( a, b ) , then norm ( · ) is set to be ( b -a ) sigmoid ( · ) + a . This bias term makes the generated label lie in the same space as the original label, enabling us to more conveniently explore its semantic meaning, which is also verified to improve model performance in our experiments.

3.2.3 Discussion about the Share Input Scenario

The proposed generator can generate the new feature by combining the features from the primary and auxiliary tasks. However, in the most widely studied auxiliary learning scenario [9, 1, 11], all the

tasks share the same input data. We only have the training dataset { ( x p,j , y p,j , y a 1 ,j , · · · , y aK,j ) } | D p | j =1 . Therefore, we cannot obtain the auxiliary data [ x a 1 ,j , · · · , x aK,j ] from the auxiliary tasks to conduct the feature generation. To tackle this problem, we obtain the new auxiliary data by randomly sampling another data sample in the dataset, which is easy to implement with randomly shuffling the training batch to match another sample for the original sample. Specifically, for a sample ( x p,j , y p,j , y a 1 ,j , · · · , y aK,j ) , we consider another sample ( x p,j 2 , y p,j 2 , y a 1 ,j 2 , · · · , y aK,j 2 ) in the dataset to provide new auxiliary data information. The input of the feature generator is [ f p,j , f p,j 2 ] , which are the features extracted by the backbone with [ x p,j , x p,j 2 ] as input. When generating new labels, we need to combine all the existing labels of the two samples, i.e., the input of the label generator is [ y p,j , y a 1 ,j , · · · , y aK,j , y p,j 2 , y a 1 ,j 2 , · · · , y aK,j 2 ] , and then the label generation process is the same as before. Note that there are two small details: i) During the label embedding process, y ai,j and y ai,j 2 share the same embedding table, because they both belong to the same task. ii) For the bias term in eq. (9), we now have two labels from the primary task, y p,j and y p,j 2 , and then eq. (9) can be adjusted to:

where β ∈ (0 , 1) is a learnable parameter of the generator. Till now, we have described the designs of the joint generator, which is applicable in various auxiliary learning scenarios. However, the generator has several parameters to be optimized, like the masks and MLPs. We denote all the learnable parameters in the generator together with the task weights w i in eq. (2) as ϕ . Next, we will elaborate how we optimize the task learning model parameters θ and the generator parameters ϕ .

3.3 Optimization Strategy

Bi-level optimization. In our whole framework, the task learning model parameters θ are expected to minimize the loss of all the selected and generated tasks, while the generator parameters ϕ aim to make θ achieve the best performance on the primary task. These two different goals give rise to the following bi-level optimization problem:

where L t ( θ, ϕ ) is the objective in eq. (2), and L p ( θ ∗ ( ϕ ); D v ) is the primary task loss of the task learning model on the validation dataset. The lower optimization is easy, we can directly obtain the gradient of θ as ∇ θ L t ( θ, ϕ ) . However, in the upper optimization, L p ( θ ∗ ( ϕ ); D v ) directly relies on θ instead of ϕ . Assuming that the Hessian ∇ 2 θ L t ( θ ∗ ( ϕ ) , ϕ ) is positive-definite, we can use the implicit theorem to obtain its implicit gradient ∇ ϕ L p ( θ ∗ ( ϕ ); D v ) ,

Detailed derivation can be found in Appendix 1. Since the inverse of the Hessian is often intractable, we follow [24] to use truncated Neumann series to approximate it. ( ∇ 2 θ L t ) -1 ≈ ∑ n i =0 ( I -∇ 2 θ L t ) i , and n is fixed to 3 in all our experiments. Thus, the complete implicit gradient is approximated as:

Instance regularization. In the upper optimization, we additionally introduce an instance regularization for the parameterized label y g,j in eq. (8) as follows, to prevent generation mode collapse.

̸

where if the generated label is categorical, we expect that the cosine similarity of different generated labels is small. This regularization means we expect that the generated label can keep its instancelevel uniqueness, preventing the generated labels of all the generated features from being the same. Similarly, if the generated label is numerical, since it is 1-dimension, we use the entropy regularization to achieve this goal, where ¯ y g,j = e y g,j / ∑ | D g | j ′ =1 e y g,j ′ . Then in the upper optimization, the gradient of ϕ is the sum of the implicit gradient and explicit gradient from the regularization:

Now the gradients of θ and ϕ are obtained, and we follow previous works [1, 11] to alternatingly update θ and ϕ . The update loop will continue until convergence, where in each loop we first update θ for N times, and then update ϕ using the upper gradient eq. (15) as shown in Figure 1. N is the interval between two upper updates. We summarize the complete algorithm in Appendix 2, where we do not require an additional validation dataset D v to calculate the upper objective but reuse the training primary set D p as done in [13, 14].

4 Experiments

4.1 Experimental Setup

Task and Dataset. We conduct our experiments on two scenarios to validate the generalization ability of our method. One is the most widely studied scenario in previous auxiliary learning works, where the auxiliary tasks share the same input as the primary task (share input). The other one is that the inputs of the primary and auxiliary tasks are different (different input). In the Share Input setting, (i) CUB [25]: we follow previous works [1, 11] to use the bird visual attribute classification (e.g., whether the bird has white belly) to help the bird species classification primary task, where both the auxiliary and primary labels are categorical. (ii) CIFAR100 [26]: it is a widely adopted image classification dataset, where there are totally 100 categories. Additionally, each image has a coarse class, e.g., a car belongs to the 'vehicles 1' coarse class. We use the coarse classification as the auxiliary task to help the 100-classification primary task. (iii) Besides the classification problem, we also focus on the regression problem, where we follow previous works [11, 12] to regard the rating prediction in recommender system as the primary task, and the CTR prediction as the auxiliary task. The primary task is regression and the auxiliary task is binary classification. We choose the widely used Amazon Toys and Movies [27] datasets, where we use the user ID, item ID and item category as the input data. In the Different Input setting, (i) CIFAR10-100 is a setting where our primary task is the CIFAR10 classification problem, and the auxiliary task is the CIFAR100 classification problem. (ii) Pet-CUB is a similar setting where our primary task is fine-grained pet classification on the Pet [28] dataset, while our auxiliary task is the bird species classification in the previously mentioned CUB dataset. More detailed data information is presented in Appendix 4.

Baselines. We compare with SOTA auxiliary learning methods, including the reweighing methods and the auxiliary task generation method. Single task learning(STL) is a natural baseline where we only train on the primary task. Equal is a baseline where we assign equal weights 1.0 to all the tasks. Uncert [18] is a dynamic weighting method for multi-task learning based on task uncertainty. GCS [9] and AuxL [1] dynamically reweight the auxiliary losses, and JTDS [11] not only reweighs the tasks but also each data sample within each task. MAXL [1] is a method that automatically generates a fine-grained auxiliary task for the primary classification task. We provide detailed differences between our work and the baselines in Appendix 3.1.

Implementation details. In CUB dataset, we respectively adopt ResNet18 [29] and ResNet50 as our backbone. In CIFAR100, we respectively adopt ResNet18 and a 4-layer ConvNet composed of Convolution, Batch Normaliztion and Relu layers as our backbone. In Amazon Toys and Movies, we adopt AutoINT [30] as the backbone. In CIFAR10-100, the backbone is the 4-layer ConvNet and in Pet-CUB the backbone is ResNet18. For the head of each task, we adopt Multi-LayerPerceptron(MLP) whose layer is searched from {1, 2}. In the generator, the embedding dimension m n is searched from {32, 64}, and the layer number of the MLP is searched from {2, 3, 4}. More details are presented in Appendix 4.

4.2 Experimental Results

4.2.1 Method Performance

Main results. Table 1 presents the overall experimental results, showing that our proposed method consistently outperforms existing methods on the diversified auxiliary learning scenarios. During training in the CUB dataset and Pet-CUB dataset, different from [11] that does not use the learning rate scheduler, we found that applying a learning rate scheduler to all the baselines can obtain better or at least on-par performance. Therefore, we report the results with the scheduler. From the results, we have the following observations. (i) As expected, our proposed method brings benefits for the primary task when the original manually collected auxiliary data and task are unhelpful. For example,

in the CUB dataset with ResNet50 backbone, Movies, and Pet-CUB, the three scenarios, all the methods that utilize auxiliary tasks perform worse than the STL baseline, indicating that the manually chosen auxiliary data and tasks are not beneficial to the primary task. This phenomenon is also observed in previous works [1, 12]. However, our proposed method can utilize the information in the originally harmful auxiliary task to generate new and beneficial auxiliary data and task, so that the model performance can be significantly improved. The ability of our method to convert originally harmful auxiliary labels into beneficial forms is further validated in Appendix 5.2. Although MAXL generates a new fine-grained classification task for the primary task, it brings no improvement in these settings. This is possibly because it requires a hierarchical dataset structure in the primary task. However, in the CUB and Pet-CUB setting, the primary task itself is a fine-grained image classification problem and a more fine-grained meanwhile beneficial task is hard to be automatically generated. Additionally, it does not consider generating new data during generation. (ii) When the original auxiliary task is beneficial to the primary task, our proposed method inherits these benefits and can bring further improvement. For example, in CIFAR100 with ResNet18 as backbone and CIFAR10-100 settings, almost all the methods that utilize the auxiliary information outperform the STL baseline, and our proposed method outperforms all the existing auxiliary methods. (iii) Whether the auxiliary data and task are beneficial to the primary task depends on the backbone we choose. In the CIFAR100 dataset, when we the use ConvNet backbone, the original auxiliary data and tasks are harmful because all the existing auxiliary methods perform worse than STL, but when we use ResNet18 as the backbone, the auxiliary data and tasks are beneficial.

Fewshot results. Since previous works [1, 11] indicate auxiliary learning is more effective when the data of the primary task is inadequate, we conduct experiments on CUB where we only use 5/10 shots of each category for the primary task in Table 2. We can see that when the data of the primary task is 5 shot, all methods that utilize the auxiliary task perform better than STL, while in Table 1, when the primary task has full data, the auxiliary data and task are harmful, indicating whether the auxiliary information is beneficial or not is related to the primary dataset scale. Among all the auxiliary learning methods, our proposed method brings the most significant improvement. Compared to the full dataset setting, the improvement of our method becomes more significant in the fewshot setting. The improvement largely results from that we not only generate new tasks but also new data. The effectiveness of generating new data is further validated in Appendix 5.3.

4.2.2 Ablation Studies

To further understand how our proposed method works, we provide the following ablation studies.

#Image-P

#Image-Aux

#G-Label

9'0

0.5

0.4

0.3

0.2

0.1

0.0

0

OT

20

0.5

0.4

0.3

0.0

0

510 15 20 25 30 35

0.30

0.25

0.35

0.20

0.15

0.10

0.05

0.00

025 50 75 100125150175200

0.30

0.25

0.35

0.20

0.15

0.10

0.05

0.00

0 25 50 75 100125150175200

Generation visualization. We visualize the generation process in Table 3. The first two rows are from the Pet-CUB dataset in the Different Input setting, and the last two rows are from the CUB dataset in the Share Input setting. In Pet-CUB, we can see the generated label has one obvious peak from the label bias term, which represents the original category of the primary image. There are also some other small peaks. Specifically, from the first row, we can see that when our feature generator combines a white cat and a yellow-and-black bird, the label generator thinks that the label of the generated feature has the maximum probability of being the original white cat category, the second largest probability of being the yellow-and-black cat category and then a similar color dog. The pattern in the second row is similar to the first row, the texture and color of the bird are combined with the black cat, so that the label generator thinks the generated feature should first belong to the original black cat, then the brown cat with stripes from the bird, and then a similar color dog. In CUB, the generated label has two obvious peaks, which represent the two categories of the primary and auxiliary images. Also, there are some other small peaks, we can see that the images from the small peaks are similar to the primary and auxiliary images, which have reasonable semantic meanings.

Effectiveness of the framework design. Our whole framework contains the generator and the optimization strategy. We validate their effectiveness in Table 4. To make the generator expressive, we incorporate nonlinear terms in the generator, and we also incorporate the label bias term in the label generator. To optimize the generator, we propose the bi-level optimization strategy with instance regularization in the upper update. We respectively remove each of these components and observe performance degradation, where w/o F-nonlinear, w/o L-nonlinear, w/o L-bias, w/o bi-level and w/o instance reg respectively represents the variant where we remove the feature nonlinear term, label nonlinear term, label bias term, the overall upper update, and the instance regularization. We find that the non-linear terms indeed help to generate more beneficial data and tasks by making the generator more expressive. Also, the instance regularization is very important in the bi-level optimization. Furthermore, we observe label generation mode collapse without the regularization in Appendix 5.4.

Comparison with MixUp During the generation visualization, we find that in the CUB setting, the generation process behaves like MixUp where the inputs are combined in a linear way and the labels are also combined in a linear way. Therefore, we compare our proposed method with different MixUp methods in Table 5. MixUp [31] means we directly utilize the data and labels generated by MixUp as auxiliary tasks. F-MixUp [32] means the linear combination does not happen in the input, but in the manifold feature space. Auto-Mix/Auto-F-Mix means the linear weight of MixUp/F-MixUp is learned automatically with the bi-level optimization, and the task weights are also learnable, i.e, utilizing learnable MixUp/F-MixUp to replace our joint generator. We can see that our proposed method outperforms these MixUp methods. The reasons are quite obvious, as shown in Table 3, our proposed method not only has the two peaks in MixUp, but also other semantically reasonable small peaks. Compared to MixUp, the advantages of our proposed method are three-fold. (i) Our method

can not only combine information from the label of the same task but also label from other tasks. (ii) Our method can not only model linear relations in generation but also non-linear relations. (iii) Our method can handle the Different Input scenario, while MixUp cannot.

To further show the advantage of our proposed generation compared to MixUp, we apply Auto-F-Mix on top of auxiliary learning methods (AuxL, JTDS, MAXL) and the experimental results are presented in Table 6. The results show that both directly combining auxiliary weighting methods(AuxL, JTDS) with Auto-F-Mix and combining the current auxiliary generation method (MAXL) with Auto-F-mix are not as effective as our method, indicating the advantage of our proposed joint generation.

5 Conclusion

In this paper, we propose to jointly generate beneficial auxiliary data and tasks for auxiliary learning, so that the primary task can still obtain benefits when the manually collected auxiliary data and tasks are unhelpful. We propose the DTG-AuxL framework with a joint generator and a bi-level optimization strategy, which can be applied in various auxiliary learning scenarios. Future works like designing more adaptive generators and more efficient bi-level optimization algorithms can further improve the generation.

References

Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

Shikun Liu, Andrew J. Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 1677-1687, 2019.

Lucas Beyer, Xiaohua Zhai, Avital Oliver, and Alexander Kolesnikov. S4L: self-supervised semi-supervised learning. In ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pages 1476-1485, 2019.

Hong Wen, Jing Zhang, Yuan Wang, Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. Entire space multi-task modeling via post-click behavior decomposition for conversion rate prediction. In SIGIR 2020, Virtual Event, China, July 25-30, 2020 , pages 2377-2386, 2020.

Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Selfsupervision for reinforcement learning. In ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings , 2017.

Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 4773-4784, 2019.

Yabin Zhang, Hui Tang, and Kui Jia. Fine-grained visual categorization using meta-learning optimization with sample selection of auxiliary data. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII , pages 241-256, 2018.

Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference on computer vision , pages 94-108, 2014.

Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. CoRR , 2018.

Baifeng Shi, Judy Hoffman, Kate Saenko, Trevor Darrell, and Huijuan Xu. Auxiliary task reweighting for minimum-data learning. In NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Hong Chen, Xin Wang, Chaoyu Guan, Yue Liu, and Wenwu Zhu. Auxiliary learning with joint task and data scheduling. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , pages 3634-3647, 2022.

Hong Chen, Xin Wang, Yue Liu, Yuwei Zhou, Chaoyu Guan, and Wenwu Zhu. Module-aware optimization for auxiliary learning. In Advances in Neural Information Processing Systems , 2022.

Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.03091 , 2022.

Hong Chen, Xin Wang, Ruobing Xie, Yuwei Zhou, and Wenwu Zhu. Cross-domain recommendation with behavioral importance perception. In Proceedings of the ACM Web Conference 2023 , pages 1294-1304, 2023.

Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796 , 2020.

Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: deciding what layers to share. arXiv preprint arXiv:1904.02920 , 2019.

Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning , pages 3854-3863. PMLR, 2020.

Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 7482-7491, 2018.

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML 2018 , pages 793-802. PMLR, 2018.

Jie Song, Yixin Chen, Xinchao Wang, Chengchao Shen, and Mingli Song. Deep model transferability from attribution maps. Advances in Neural Information Processing Systems , 32, 2019.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems , pages 7-10, 2016.

Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, and Xiuqiang He. An embedding learning framework for numerical features in ctr prediction. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2910-2918, 2021.

Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy] , Proceedings of Machine Learning Research, pages 1540-1552. PMLR, 2020.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases , 1(4), 2009.

Ruining He and Julian J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, WWW2016, Montreal, Canada, April 11 - 15, 2016 , pages 507-517. ACM, 2016.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 3498-3505, 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016.

Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019 , pages 1161-1170. ACM, 2019.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , pages 6438-6447. PMLR, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of auxiliary learning, specifically for cases where the provided auxiliary data actually harms the primary task.\nA joint data-task generation framework is proposed, which generates new auxiliary tasks during the optimization process.\nThe label generator generates data in the feature space and includes both linear and nonlinear interaction terms to produce more expressive data.\nBi-level optimization is used to optimize the entire structure, with instance regularization preventing mode collapse of the generated data.\nExperimental results show that the proposed structure can outperform existing methods in a variety of auxiliary learning setups and even prevent performance degradation in certain cases.\n\nSTRENGTHS:\n- The proposed method has strong results in auxiliary learning benchmarks and even prevents performance drop with harmful auxiliary tasks, as shown in the Movies and Pet-CUB experiments. Table 2 also shows its effectiveness in low-data regimes.\n- Evaluation and training settings are fully explained in the paper and supplementary material, which helps with understanding and reproducing the results.\n\nWEAKNESSES:\n- The idea of bi-level optimization strategy has been proposed and used by prior work [1,2,3]. Thus, this contribution cannot really be claimed by this work. Also, instance regularization is simply an application of the regularization loss introduced in [1].\n- Experiments are conducted in limited scale:\n  - Most tasks only have a single auxiliary task and the tasks are simple classification or regression problems. Can more auxiliary tasks be used and more complex tasks also be studied as done in previous work, such as on NYUv2 [2]?\n  - The extension to dense prediction (e.g., semantic segmentation in NYUv2) seems not so trivial with the proposed framework. It will be beneficial if the authors can demonstrate how this can work with the proposed framework.\n- While the improvement even in the presence of harmful auxiliary tasks is a good result, the explanation for why this can be achieved and why previous methods fail is lacking.\n  - In section 5.2 of the supplementary material, it seems like even removing auxiliary label information the performance is still superior to STL. Why is this?\n\n\n[1] Liu, Shikun, Andrew Davison, and Edward Johns. \"Self-supervised generalisation with meta auxiliary learning.\" Advances in Neural Information Processing Systems 32 (2019).\n[2] Navon, Aviv, et al. \"Auxiliary learning by implicit differentiation.\" arXiv preprint arXiv:2007.02693 (2020).\n[3] Chen, Hong, et al. \"Module-Aware Optimization for Auxiliary Learning.\" Advances in Neural Information Processing Systems 35 (2022): 31827-31840.\n\nQUESTIONS:\nSee weaknesses.\n\nWriting:\n- The proposed method is akin to MixUp in the feature space, thus I believe the experimental results in section 5.4 in the supplementary material is important and can perhaps be moved to the main paper.\n- Experiments:\n  - Please add citations for the methods in the experiment tables so it is easier to refer to the corresponding method.\n  - Add performance of [3] to Table 1/2.\n  - Please fix the formatting of Table 3 and 4, as their current positioning makes this page difficult to read. Table 3 should be a figure and can be column width. The space between Table 4 and the text is so small.\n\n[3] Chen, Hong, et al. \"Module-Aware Optimization for Auxiliary Learning.\" Advances in Neural Information Processing Systems 35 (2022): 31827-31840.\n\nLIMITATIONS:\nThe limitations of the work is provided in the supplementary material. I suggest the authors to include at least a short summary in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on Auxiliary Learning, a technique aiming to improve the model generalization ability on the primary task with the help of related auxiliary tasks, and proposes a novel joint data-task generation framework for auxiliary learning (DTG-AuxL). DTG-AuxL is designed to mitigate certain issues present in existing methods. Specifically, existing methods necessitate the presence of beneficial information in the auxiliary data and tasks; otherwise, the useless auxiliary information will do harm to the primary task when the involved auxiliary data and tasks are improperly collected. DTG-AuxL features a joint generator and a bi-level optimization strategy and can simultaneously generate auxiliary data and tasks in a joint manner. Extensive experiments show that DTG-AuxL outperforms existing methods in various auxiliary learning scenarios.\n\nSTRENGTHS:\nThis paper proposes DTG-AuxL, a joint data-task generation framework applicable in various auxiliary learning scenarios. DTG-AuxL contains a joint generator and a bi-level optimization strategy and can simultaneously generate auxiliary data and tasks in a joint manner, which is the first attempt to do so.\n\nDTG-AuxL can adapt to different kinds of input data, such as tabular data and visual images. Because the data generation process is conducted in the feature space and the label generator can accommodate both categorical labels for classification and numeric labels for regression. Consequently, DTG-AuxL has wide applicability.\n\nHow to effectively optimize the parameters in the generation framework is a challenging problem to guarantee the generated data and task are beneficial to the primary task. This paper proposes a bi-level optimization strategy as well as instance regularization to tackle this challenge.\n\nExtensive experiments on two scenarios validate the generalization ability of DTG-AuxL. The first scenario is that the auxiliary tasks share the same input as the primary task, and the second scenario is that the inputs of the primary and auxiliary tasks are different. Experimental results show that DTG-AuxL outperforms existing methods on diversified scenarios, including few-shot learning scenarios. Specifically, DTG-AuxL is able to convert originally harmful auxiliary labels into beneficial forms.\n\nWEAKNESSES:\nTable 3 and Table 4 is not appropriate.\n\nThis paper conducts experiments on the following scenarios:\n\n1. Share Input: visual inputs & classification task (primary) + visual inputs & classification task (auxiliary)\n2. Share Input: tabular inputs & regression task (primary) + tabular inputs & classification task (auxiliary)\n3. Different Input: visual inputs & classification task (primary) + visual inputs & classification task (auxiliary)\n\nBut in order to verify the broad applicability of DTG-AuxL in the paper, more different types of experiments need to be done, such as Different Input: tabular inputs & classification task (primary) + tabular inputs & classification task (auxiliary).\n\nFinally, I am a little confused, does the so-called task generation refer to label generation? Why can't label be regarded as part of data? Shouldn't task generation literally include loss functions or other ingredients?\n\nQUESTIONS:\nSee my questions in Weaknesses.\n\nLIMITATIONS:\nNo comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning with auxiliary tasks. Motivated by the observation that in most scenario the auxiliary data is hard to collect, the authors propose to train a model that can automatically generate auxiliary data (and labels) for any task at hand. Specifically, the authors propose a data generator network that generates new data based on the current data, and the network is learned by a bi-level optimization where the final objective is to maximize the performance on the validation set of the primary task. The authors show the advantage of the proposed method on several datasets and settings.\n\nSTRENGTHS:\n- The motivation is clear and sensitive and the proposed method is sound. \n\n- The idea of auxiliary data generation might not be completely new but the authors propose a unified way that can generate data for different tasks and different input/output format, which is a good contribution to me. \n\n- The paper is overall well written and easy to follow.\n\nWEAKNESSES:\n- To tackle with tasks with different input format, the authors propose to generate the features instead of the raw data. This may be a problem because in this way only the task-specific heads are trained on the generated data, but the backbone is still only learned on the primary task (as well as the already existed auxiliary tasks). This is suboptimal and might limit the impact of the generated data. Maybe the authors could comment on this?\n\n- In the feature linear interpolation, are the masks (L143) input-independent? \n\n- The instance regularization loss for numerical labels (Equation 14), if the generated labels have one label that is extremely large, and other labels that are the same and extremely small, then the $\\bar{y}_{g,j}$ will be 1 for one label and 0 for others, which will minimize the regularization loss. This seems not encouraging the diversity of the labels.\n\n- One question I have is about the visualization of the generated data (Table 3). I wonder where do the Max-Prob Images come from? Are they the images that have the maximal probability (predicted by the trained model) on the label? Have the authors tried to visualize the images in the training set which have the feature that is closest to the generated feature, i.e., using the generated feature to do a knn search in the training set?\n\n- Have the authors tried the setting where there is no ready-to-use auxiliary tasks and only the primary task is available?\n\n\n===========\n\nPost rebuttal: The rebuttal partially address my concerns and I will keep my score of weak accept.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
Motivated by the manually collected auxiliary data and tasks in auxiliary learning may have negative impacts on the primary task, this paper proposes an auxiliary learning framework that jointly generates new auxiliary data and task. To generate new data for various auxiliary scenarios, the authors design an adaptive feature generator and a label generator, which considers both linear and nonlinear relations. Extensive experiments show that the proposed method can bring benefits to the primary task, even when the manually collected auxiliary data and task are potentially harmful.

STRENGTHS:
1.	The paper is well-motivated and tackles an important problem in auxiliary learning. To collect the beneficial auxiliary data and task really needs human trials and domain knowledge.
2.	The paper is well organized with clear discussion about related works, reasonable method formulation, and experiments to support the claims. 
3.	The experiments are conducted across different domains and datasets to show the effectiveness and generality of the proposed method. 
4.	Some ablations bring interesting insights.

WEAKNESSES:
1.	Since this work focuses on using generation to tackle the auxiliary learning problem, why not use some pretrained generative models like Stable Diffusion? The Stable Diffusion can also use conditional generation which gives similar data as provided by Table 3, and the paper can be strengthen with some discussion about this.
2.	The most advanced models are pretrained, whether the proposed method can be applied to pretrained models is not clear.

QUESTIONS:
See above.

LIMITATIONS:
I don't see any concerns regarding the societal impact in this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes an auxiliary learning framework that jointly generates new auxiliary data and new labels, to tackle the negative transfer problem in auxiliary learning. The framework is designed to be generic to various auxiliary scenarios, both multiple input and single input, regression and classification. The bi-level optimization strategy is designed to optimize the task learning model and the joint generation model. Experiments on several benchmarks show that the proposed method outperforms existing methods.

STRENGTHS:
1.	The paper is very clear and well-written.
2.	The idea to generate new data and task to avoid the potential negative transfer in manually designed auxiliary tasks is novel and reasonable.
3.	The relations and differences compared to previous works are clearly discussed.
4.	The proposed method is generic and makes sense.
5.	Experiments are well conducted, and the effectiveness of each component is well validated. The visualizations are reasonable and interesting.

WEAKNESSES:
1.	In the single input scenario, the proposed method behaves like MixUp, and I think the comparison with MixUp is important and should be put to the main manuscript instead of the appendix, although I understand the page is limited.
2.	Considering that current large pretrained models usually have a lot of parameters, if we want to finetune these models, as the limitation part discussed, we will need more computational cost. Whether this method can serve the current pretrain-tuning paradigm will largely influences its significance.

QUESTIONS:
Please refer to the weakness part. Although I think this paper is overall of good quality, I want to know how it can adapt to the current pretrained models.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07