SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{L}_t(\theta, \phi)$  is the objective in eq. (2), and  $\mathcal{L}_p(\theta^*(\phi); D_v)$  is the primary task loss of the task learning model on the validation dataset. The lower optimization is easy, we can directly obtain the gradient of  $\theta$  as  $\nabla_\theta \mathcal{L}_t(\theta, \phi)$ . However, in the upper optimization,  $\mathcal{L}_p(\theta^*(\phi); D_v)$  directly relies on  $\theta$  instead of  $\phi$ . Assuming that the Hessian  $\nabla_\theta^2 \mathcal{L}_t(\theta^*(\phi), \phi)$  is positive-definite, we can use the implicit theorem to obtain its implicit gradient  $\nabla_\phi \mathcal{L}_p(\theta^*(\phi); D_v)$ ,
$$
\nabla_ {\phi} \mathcal {L} _ {p} \left(\theta^ {*} (\phi); D _ {v}\right) = - \nabla_ {\theta} \mathcal {L} _ {p} \cdot \left(\nabla_ {\theta} ^ {2} \mathcal {L} _ {t}\right) ^ {- 1} \cdot \nabla_ {\phi} \nabla_ {\theta} \mathcal {L} _ {t} | _ {(\phi , \theta^ {*} (\phi))}. \tag {12}
$$
Detailed derivation can be found in Appendix 1. Since the inverse of the Hessian is often intractable, we follow [24] to use truncated Neumann series to approximate it.  $(\nabla_{\theta}^{2}\mathcal{L}_{t})^{-1}\approx \sum_{i = 0}^{n}(I - \nabla_{\theta}^{2}\mathcal{L}_{t})^{i}$  and  $n$  is fixed to 3 in all our experiments. Thus, the complete implicit gradient is approximated as:
$$
\nabla_ {\phi} \mathcal {L} _ {p} \left(\theta^ {*} (\phi); D _ {v}\right) \approx - \nabla_ {\theta} \mathcal {L} _ {p} \cdot \sum_ {i = 0} ^ {n} \left(I - \nabla_ {\theta} ^ {2} \mathcal {L} _ {t}\right) ^ {i} \cdot \nabla_ {\phi} \nabla_ {\theta} \mathcal {L} _ {t}. \tag {13}
$$
Instance regularization. In the upper optimization, we additionally introduce an instance regularization for the parameterized label  $y_{g,j}$  in eq. (8) as follows, to prevent generation mode collapse.
$$
\mathcal {L} _ {r e g} (\phi ; D _ {g}) = \left\{ \begin{array}{l l} \sum_ {j = 1} ^ {| D _ {g} |} \sum_ {j ^ {\prime} \neq j} \cos \left(y _ {g, j}, y _ {g, j ^ {\prime}}\right) & \text {c a t e g o r i c a l} \\ - \sum_ {j = 1} ^ {| D _ {g} |} \bar {y} _ {g, j} \log \left(\bar {y} _ {g, j}\right) & \text {n u m e r i c a l} \end{array} \right. \tag {14}
$$
where if the generated label is categorical, we expect that the cosine similarity of different generated labels is small. This regularization means we expect that the generated label can keep its instance-level uniqueness, preventing the generated labels of all the generated features from being the same. Similarly, if the generated label is numerical, since it is 1-dimension, we use the entropy regularization to achieve this goal, where  $\bar{y}_{g,j} = e^{y_{g,j}} / \sum_{j' = 1}^{|D_g|}e^{y_{g,j'}}$ . Then in the upper optimization, the gradient of  $\phi$  is the sum of the implicit gradient and explicit gradient from the regularization:
$$
\nabla_ {\phi} = \nabla_ {\phi} \mathcal {L} _ {p} \left(\theta^ {*} (\phi); D _ {v}\right) + \nabla_ {\phi} \mathcal {L} _ {r e g} (\phi ; D _ {g}). \tag {15}
$$
Now the gradients of  $\theta$  and  $\phi$  are obtained, and we follow previous works [1, 11] to alternatingly update  $\theta$  and  $\phi$ . The update loop will continue until convergence, where in each loop we first update  $\theta$  for  $N$  times, and then update  $\phi$  using the upper gradient eq. (15) as shown in Figure 1.  $N$  is the interval between two upper updates. We summarize the complete algorithm in Appendix 2, where we do not require an additional validation dataset  $D_v$  to calculate the upper objective but reuse the training primary set  $D_p$  as done in [13, 14].
Task and Dataset. We conduct our experiments on two scenarios to validate the generalization ability of our method. One is the most widely studied scenario in previous auxiliary learning works, where the auxiliary tasks share the same input as the primary task (share input). The other one is that the inputs of the primary and auxiliary tasks are different (different input). In the Share Input setting, (i) CUB [25]: we follow previous works [1, 11] to use the bird visual attribute classification (e.g., whether the bird has white belly) to help the bird species classification primary task, where both the auxiliary and primary labels are categorical. (ii) CIFAR100 [26]: it is a widely adopted image classification dataset, where there are totally 100 categories. Additionally, each image has a coarse class, e.g., a car belongs to the "vehicles 1" coarse class. We use the coarse classification as the auxiliary task to help the 100-classification primary task. (iii) Besides the classification problem, we also focus on the regression problem, where we follow previous works [11, 12] to regard the rating prediction in recommender system as the primary task, and the CTR prediction as the auxiliary task. The primary task is regression and the auxiliary task is binary classification. We choose the widely used Amazon Toys and Movies [27] datasets, where we use the user ID, item ID and item category as the input data. In the Different Input setting, (i) CIFAR10-100 is a setting where our primary task is the CIFAR10 classification problem, and the auxiliary task is the CIFAR100 classification problem. (ii) Pet-CUB is a similar setting where our primary task is fine-grained pet classification on the Pet [28] dataset, while our auxiliary task is the bird species classification in the previously mentioned CUB dataset. More detailed data information is presented in Appendix 4.
Baselines. We compare with SOTA auxiliary learning methods, including the reweighing methods and the auxiliary task generation method. Single task learning(STL) is a natural baseline where we only train on the primary task. Equal is a baseline where we assign equal weights 1.0 to all the tasks. Uncert [18] is a dynamic weighting method for multi-task learning based on task uncertainty. GCS [9] and AuxL [1] dynamically reweight the auxiliary losses, and JTDS [11] not only reweighs the tasks but also each data sample within each task. MAXL [1] is a method that automatically generates a fine-grained auxiliary task for the primary classification task. We provide detailed differences between our work and the baselines in Appendix 3.1.
Implementation details. In CUB dataset, we respectively adopt ResNet18 [29] and ResNet50 as our backbone. In CIFAR100, we respectively adopt ResNet18 and a 4-layer ConvNet composed of Convolution, Batch Normalization and Relu layers as our backbone. In Amazon Toys and Movies, we adopt AutoINT [30] as the backbone. In CIFAR10-100, the backbone is the 4-layer ConvNet and in Pet-CUB the backbone is ResNet18. For the head of each task, we adopt Multi-Layer-Perceptron(MLP) whose layer is searched from  $\{1,2\}$ . In the generator, the embedding dimension  $m_{n}$  is searched from  $\{32,64\}$ , and the layer number of the MLP is searched from  $\{2,3,4\}$ . More details are presented in Appendix 4.
4.2.1 Method Performance
Main results. Table 1 presents the overall experimental results, showing that our proposed method consistently outperforms existing methods on the diversified auxiliary learning scenarios. During training in the CUB dataset and Pet-CUB dataset, different from [11] that does not use the learning rate scheduler, we found that applying a learning rate scheduler to all the baselines can obtain better or at least on-par performance. Therefore, we report the results with the scheduler. From the results, we have the following observations. (i) As expected, our proposed method brings benefits for the primary task when the original manually collected auxiliary data and task are unhelpful. For example,
Table 1: Overall performance. We report the results over 5 random seeds. Note that JTDS cannot handle the different input scenarios and MAXL cannot generate auxiliary tasks for the primary regression task. The metric for classification is accuracy(Acc) and the metric for the rating regression in recommendation is RMSE. Higher accuracy and lower RMSE indicate better results.
Dataset	CUB	CIFAR100	Toys	Movies	CIFAR10-100	Pet-CUB
Metric	Acc(%)↑	Acc(%)↑	RMSE↓	RMSE↓	Acc(%)↑	Acc(%)↑
Backbone	ResNet18	ResNet50	ConvNet	ResNet18	AutoINT	AutoINT	ConvNet	ResNet18
STL	76.16	80.46	50.35	55.79	0.9188	1.0456	79.35	69.47
Equal	74.25	78.28	49.56	56.42	0.9213	1.0459	78.99	67.15
Uncert	73.94	77.03	48.90	56.95	0.9171	1.0495	80.13	63.58
GCS	74.11	78.54	49.26	56.57	0.9224	1.0459	79.59	67.31
AuxL	75.39	78.00	49.39	57.14	0.9186	1.0483	79.69	66.17
JTDS	76.50	79.34	49.00	57.28	0.9187	1.0504	-	-
MAXL	75.79	78.48	49.71	56.32	-	-	80.02	68.48
ours	77.75	81.73	50.94	57.84	0.9153	1.0426	80.64	70.48
Table 2: Acc(%) on CUB Fewshot Experiments with ResNet50.
Method	STL	Equal	Uncert	GCS	AuxL	JTDS	MAXL	ours
5 shot	44.991.15	45.370.61	45.630.08	45.680.67	45.071.31	46.130.89	45.471.06	52.331.36
10 shot	63.250.68	60.580.81	59.300.55	60.810.74	63.790.11	63.390.18	61.700.10	67.680.33
in the CUB dataset with ResNet50 backbone, Movies, and Pet-CUB, the three scenarios, all the methods that utilize auxiliary tasks perform worse than the STL baseline, indicating that the manually chosen auxiliary data and tasks are not beneficial to the primary task. This phenomenon is also observed in previous works [1, 12]. However, our proposed method can utilize the information in the originally harmful auxiliary task to generate new and beneficial auxiliary data and task, so that the model performance can be significantly improved. The ability of our method to convert originally harmful auxiliary labels into beneficial forms is further validated in Appendix 5.2. Although MAXL generates a new fine-grained classification task for the primary task, it brings no improvement in these settings. This is possibly because it requires a hierarchical dataset structure in the primary task. However, in the CUB and Pet-CUB setting, the primary task itself is a fine-grained image classification problem and a more fine-grained meanwhile beneficial task is hard to be automatically generated. Additionally, it does not consider generating new data during generation. (ii) When the original auxiliary task is beneficial to the primary task, our proposed method inherits these benefits and can bring further improvement. For example, in CIFAR100 with ResNet18 as backbone and CIFAR10-100 settings, almost all the methods that utilize the auxiliary information outperform the STL baseline, and our proposed method outperforms all the existing auxiliary methods. (iii) Whether the auxiliary data and task are beneficial to the primary task depends on the backbone we choose. In the CIFAR100 dataset, when we use ConvNet backbone, the original auxiliary data and tasks are harmful because all the existing auxiliary methods perform worse than STL, but when we use ResNet18 as the backbone, the auxiliary data and tasks are beneficial.
Fewshot results. Since previous works [1, 11] indicate auxiliary learning is more effective when the data of the primary task is inadequate, we conduct experiments on CUB where we only use 5/10 shots of each category for the primary task in Table 2. We can see that when the data of the primary task is 5 shot, all methods that utilize the auxiliary task perform better than STL, while in Table 1, when the primary task has full data, the auxiliary data and task are harmful, indicating whether the auxiliary information is beneficial or not is related to the primary dataset scale. Among all the auxiliary learning methods, our proposed method brings the most significant improvement. Compared to the full dataset setting, the improvement of our method becomes more significant in the fewshot setting. The improvement largely results from that we not only generate new tasks but also new data. The effectiveness of generating new data is further validated in Appendix 5.3.
4.2.2 Ablation Studies
To further understand how our proposed method works, we provide the following ablation studies.
Table 3: Generation visualization. The first two rows from Pet-CUB and the last two rows are from CUB. The first and second columns are the images from the primary and auxiliary tasks used to generate new feature and label. The third column is the generated label, where the x-axis represents the category and the y-axis represents the probability of the generated feature belonging to this category. In the fourth column, we visualize the images with maximum probabilities (largest peaks) in the generated label, where besides the image from the peak guided by the bias term, we visualize images from the largest 2 peaks among the small peaks.
Generation visualization. We visualize the generation process in Table 3. The first two rows are from the Pet-CUB dataset in the Different Input setting, and the last two rows are from the CUB dataset in the Share Input setting. In Pet-CUB, we can see the generated label has one obvious peak from the label bias term, which represents the original category of the primary image. There are also some other small peaks. Specifically, from the first row, we can see that when our feature generator combines a white cat and a yellow-and-black bird, the label generator thinks that the label of the generated feature has the maximum probability of being the original white cat category, the second largest probability of being the yellow-and-black cat category and then a similar color dog. The pattern in the second row is similar to the first row, the texture and color of the bird are combined with the black cat, so that the label generator thinks the generated feature should first belong to the original black cat, then the brown cat with stripes from the bird, and then a similar color dog. In CUB, the generated label has two obvious peaks, which represent the two categories of the primary and auxiliary images. Also, there are some other small peaks, we can see that the images from the small peaks are similar to the primary and auxiliary images, which have reasonable semantic meanings.
Effectiveness of the framework design. Our whole framework contains the generator and the optimization strategy. We validate their effectiveness in Table 4. To make the generator expressive, we incorporate nonlinear terms in the generator, and we also incorporate the label bias term in the label generator. To optimize the generator, we propose the bi-level optimization strategy with instance regularization in the upper update. We respectively remove each of these components and observe performance degradation, where w/o F-nonlinear, w/o L-nonlinear, w/o L-bias, w/o bi-level and w/o instance reg respectively represents the variant where we remove the feature nonlinear term, label nonlinear term, label bias term, the overall upper update, and the instance regularization. We find that the non-linear terms indeed help to generate more beneficial data and tasks by making the generator more expressive. Also, the instance regularization is very important in the bi-level optimization. Furthermore, we observe label generation mode collapse without the regularization in Appendix 5.4.
Comparison with MixUp During the generation visualization, we find that in the CUB setting, the generation process behaves like MixUp where the inputs are combined in a linear way and the labels are also combined in a linear way. Therefore, we compare our proposed method with different MixUp methods in Table 5. MixUp [31] means we directly utilize the data and labels generated by MixUp as auxiliary tasks. F-MixUp [32] means the linear combination does not happen in the input, but in the manifold feature space. Auto-Mix/Auto-F-Mix means the linear weight of MixUp/F-MixUp is learned automatically with the bi-level optimization, and the task weights are also learnable, i.e., utilizing learnable MixUp/F-MixUp to replace our joint generator. We can see that our proposed method outperforms these MixUp methods. The reasons are quite obvious, as shown in Table 3, our proposed method not only has the two peaks in MixUp, but also other semantically reasonable small peaks. Compared to MixUp, the advantages of our proposed method are three-fold. (i) Our method
Table 4: Effectiveness of framework designs. The first three rows explore the effectiveness of the generator components, while the rest show the effectiveness of the training strategies.
Dataset	CUB	Pet-CUB	Toys
Metric	Acc(%)↑	Acc(%)↑	RMSE↓
Backbone	ResNet50	ResNet18	AutoINT
w/o F-nonlinear	81.220.27	67.760.17	0.91950.0021
w/o L-nonlinear	80.960.37	66.581.12	0.92130.0026
w/o L-bias	79.280.34	69.360.37	0.92040.0011
w/o bi-level	80.320.42	68.530.49	0.91630.0008
w/o instance reg	80.440.34	67.530.95	0.91740.0012
complete	81.730.20	70.480.28	0.91530.0004
can not only combine information from the label of the same task but also label from other tasks. (ii) Our method can not only model linear relations in generation but also non-linear relations. (iii) Our method can handle the Different Input scenario, while MixUp cannot.
To further show the advantage of our proposed generation compared to MixUp, we apply Auto-F-Mix on top of auxiliary learning methods (AuxL, JTDS, MAXL) and the experimental results are presented in Table 6. The results show that both directly combining auxiliary weighting methods(AuxL, JTDS) with Auto-F-Mix and combining the current auxiliary generation method (MAXL) with Auto-F-mix are not as effective as our method, indicating the advantage of our proposed joint generation.
Table 5: Comparison with MixUp Methods. Note that MixUp cannot be applied to the recommendation scenario where the input are categorical features.
Dataset	CUB	Movies
Metric	Acc(%)↑	RMSE↓
Backbone	ResNet18	AutoINT
MixUp	75.880.50	-
Auto-Mix	76.530.36	-
F-MixUp	75.330.67	1.04600.0023
Auto-F-Mix	76.880.52	1.04640.0003
ours	77.750.27	1.04260.0009
Table 6: Comparison to the combination of Auto-F-Mix and current auxiliary learning methods. We denote Auto-F-Mix as AFM.
Dataset	CUB	CUB-5shot	Toys
Metric	Acc(%)↑	Acc(%)↑	RMSE↓
Backbone	ResNet50	ResNet50	AutoINT
AFM	80.300.57	46.970.95	0.91870.0003
AuxL+AFM	79.701.07	47.841.25	0.91950.0011
MAXL+AFM	80.760.38	47.960.89	-
JTDS+AFM	80.520.90	48.140.87	0.91890.0013
ours	81.730.20	52.331.36	0.91530.0004
In this paper, we propose to jointly generate beneficial auxiliary data and tasks for auxiliary learning, so that the primary task can still obtain benefits when the manually collected auxiliary data and tasks are unhelpful. We propose the DTG-AuxL framework with a joint generator and a bi-level optimization strategy, which can be applied in various auxiliary learning scenarios. Future works like designing more adaptive generators and more efficient bi-level optimization algorithms can further improve the generation.
This work was supported by the National Key Research and Development Program of China No. 2020AAA0106300, National Natural Science Foundation of China (No. 62222209, 62250008, 62102222), Beijing National Research Center for Information Science and Technology under Grant No. BNR2023RC01003, BNR2023TD03006, and Beijing Key Lab of Networked Multimedia.