Multi Task Learning of Different Class Label Representations for Stronger Models
Authors: Judah A Goldfeder
Boyuan Chen
Hod Lipson
Venue: Submitted to ICLR 2023 (2023)
Keywords: Label Representation
Image Classification
Representation Learning
Multi-Task Learning

Abstract: We find that the way in which class labels are represented can have a powerful effect on how well models trained on them learn. In classification, the standard way of representing class labels is as one-hot vectors. We present a new way of representing class labels called Binary Labels, where each class label is a large binary vector. We further introduce a new paradigm, multi task learning on different label representations. We train a network on two tasks. The main task is to classify images based on their one-hot label, and the auxiliary task is to classify images based on their Binary Label. We show that networks trained on both tasks have many advantages, including higher accuracy across a wide variety of datasets and architectures, both when trained from scratch and when using transfer learning. Networks trained on both tasks are also much more effective when training data is limited, and seem to do especially well on more challenging problems. 

Under review as a conference paper at ICLR 2023
MULTI TASK LEARNING OF DIFFERENT CLASS LABEL REPRESENTATIONS FOR STRONGER MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
We find that the way in which class labels represented have powerful effect on how well models trained on them learn. In classification, the standard way of representing class labels as one-hot vectors. We present a new way of representing class labels called Binary Labels, where each class label is a large bi- nary vector. We further introduce a new paradigm, multi task learning on different label representations. We train network on two tasks. The main task is to classify images based on their one-hot label, and the auxiliary task is classify images based on their Binary Label. We show that networks trained on both tasks have many advantages, including higher accuracy across wide variety of datasets and architectures, both when trained from scratch and when using transfer learning. Networks trained on both tasks are also much more effective when training data is limited, and seem to do especially well more challenging problems.
INTRODUCTION
All supervised learning problems involve three components, data, a model, and labels. tremendous amount of work has been done involving the first two parts, models, and data, but labels have been mostly ignored. Deep learning architectures get ever more complicated (Villalobos et al.) (022). and many excellent techniques exist learn strong representations of the data that is fed into the model (He et al. [022a] Chen et al., 2020), but the labels themselves remain as simple as they were 20 years ago. Let's illustrate this point with common deep learning task, image classification on ImageNet (Deng et al. 2009). The leaderboard is full of algorithms that employ powerful unsupervised pre- training methods to learn representations of the data, and make use of massive model architectures with hundreds of layers and millions of weights. But if one surveys the top state of the art models, the target labels are all simple one-hot encoded vectors (Yu et al. 2022 Wortsman et al. 2022 Chen
al. 2022
Recently, various alternative label representations have been proposed, where labels are represented dense vectors (Chen et al. (002I). While on some metries, such as robustness and data efficiency, these labels were able to outperform standard label representations, on other key metrics, such as accuracy, they performed worse than one-hot labels. Additionally, these dense labels were slower inference time, for two reasons. First, because they added more weights to the networks that used them, the forward pass takes slightly longer. Second, and more significant, is that classification on dense labels is slower by definition since it involves comparing the network output with each label to find the class with the nearest label, as opposed to one-hot labels with a softmax output where the classification is done directly. Furthermore, these dense labels were much slower to converge, often taking four times as many epochs. As such, choosing between these label representations is essentially trade off between opposing factors.
In this paper we explore the key idea of: given various ways of representing labels, why only choose one? Every way of representing labels can become different task for the network to learn. By choosing to represent the same class labels in different ways, we are reusing the supervision to create multiple learning tasks. We thus propose using recent work in the field of multi task learning to train networks to recognize multiple representations of the same labels. We find that doing so allows us mitigate the tradeoff above.
From an intuitive perspective, makes sense that learning multiple representations of the same concept can be useful. To give an analogy from data structures, while linked lists and arrays represent

---

Under review as a conference paper at ICLR 2023
the same underlying data, each data structure has inherent advantages and disadvantages, and some algorithms will rely on redundant data storage of both types to get the advantages of both. While it may be novel to study explicit redundant label representations, they commonly exist implicitly. Research on the human brain indicates we make heavy use of redundant representations (Pieszek et al. 2013), and artificial neural networks are already doing this implicitly (Doimo et al. (021).
We present a new label representation type, Binary labels, where each class label is represented as binary vector. This representation has key properties that led to think it would create a useful auxiliary task to improve accuracy, and indeed our experiments verify this,
We make the following main contributions:
1. We present the novel paradigm of using several label representations of a single label to augment our network supervision and create auxiliary tasks for the network to learn
2. We present a new label representation type, Binary Labels
3. We present results that demonstrate the strength of our approach
We hope this will inspire further research on this topic.
2 RELATED WORK
We unite work from two areas, label representation, and multi task learning.
2.1 LABEL REPRESENTATION
There are many scenarios where it makes sense for a network performing image classification to output dense representation as opposed softmax vector. For example, few-shot learning, common approach, often termed embedding learning, is to learn a lower dimensional representa- tion of the input, such that similar images are near each other in the embedding space (Wang al.] 2020a). It is also common to use autoencoders to try and learn image representations, which usu- ally involves some sort of dense intermediate representation that is used to reconstruct the original (Pinaya al. 2020). While both of these methods involve networks that output representation, they do not make any attempt to generate or make use of alternative label representations. Indeed, relatively little research has been conducted on the labels themselves or more specifically, their rep- resentation, aside from (Chen et al. 2021), where several alternative methods were proposed. They reported gains on robustness and data efficiency. However convergence was much slower, and accu- racy was slightly lower than when using standard softmax labels. Aside form that, the closes work label smoothing (Szegedy al. 2016 Sun al. 2017). although (Chen al. 202 note, is quite different.
2.2 MULTI-TASK LEARNING
Multi task learning, first proposed in 1997 (Crruaaa 1997). has gained traction as a powerful way of training single network to learn multiple tasks at the same time (Crawshaw 2020 Ruder 2017] Zhang& & Yang 202T Vandenhende et al. 202T). )). Often times, doing so leads to gains on both tasks, and recent work has explored what sorts of tasks are best learned together (Standley et al.) 2020). Auxiliary learning is a subtopic in multi task learning that deals specifically with learning a main task, which we about, and auxiliary tasks, whose only purpose is to increase performance for the main task /Vafaeikia et al. 2020 Liebel & Körner 2018).
We make use of several standard tools from multitask learning. We shared trunk approach (Crwssaaw 2020), such that all tasks share a backbone network, but have different output heads. When combining the losses of different tasks, we explored several popular methods including PC- Grad (Yu et al. 2020). GradNorm (Chen et al. 2018), Grad (Wang et al.) 2020b), and Mtadam (Malieel & Wolf 2020), but ultimately settled on Metabalance (He et al. (002b)) which is specif- We also assign weights to each task, such that the sum of all
ically aimed at auxiliary learning
weights on all tasks sum to
We believe we caught small mistake in the Metahalance paper, and notified the author. The version we is slightly di different in that it requires two forward passes correctly compute the gradicnts
2

---

Under review as a conference paper at ICLR 2023
While we make use of and build off of work in the field of Multi Task Learning, any work that tries to learn different representations of the same task simultaneously. The closest work, supervised autoencoders (Le et al. 2018), attempts to learn classification and image recon- struction at the same time. While this is an example of automatically generating an auxiliary task, and applying the tools of multi task learning. the auxiliary task is simply different representa- tion of the main classification task, but something entirely different (input reconstruction).
unaware of
Another related method is Semantic Softmax (Ridnik al. (021), which uses a hierarchical dataset to generate one-hot softmax labels for each level of the hierarchy, and trains on them together. However, this method requires additional supervision: the class hierarchy needs to be given, and in most datasets, we are given no such hierarchy. Furthermore, the various labels are learning to classify at different levels of the hierarchy, and thus are distinct classification tasks all represented as one-hot softmax vectors, opposed to our work where each classification task is identical, just using different label representations.
MULTI TASK LEARNING OF MULTIPLE REDUNDANT LABEL REPRESENTATIONS
It is well known that learning multiple tasks, when the two tasks are deeply related, often leads to improved performance on both tasks (Standley et al., 2020). It thus makes sense that training network to learn two different label representations of the same class would be useful. In this case the two tasks are not just related, they are identical, with the only difference being the chosen label representation method.
Lets define our approach more formally. First, we will start with the standard way of doing things. Lets say
given labeled dataset where there classes, where each class label is provided as a number in the range of 0 to n Say we have a dataset of I samples, where X,, y; denotes
the ith input and label respectively.
We define some function f that maps these labels to the representations we would like the network to learn. In the standard approach, this simply is one-hot encoding, but in principle this function can be anything. We further define loss function L, and network N.
The goal is to minimize the total loss, given
LIN(X.)
This is how networks are normally trained. The only thing noteworthy is that we explicitly defined f. the function that converts the class label the one-hot representation that the network learns.
In the redundant representation multi task learning setting that we propose, we extend the above. We may have several such functions fo - fr-1. where the number tasks we are trying to learn. Each function takes in number, in range 0 to 1 representing the class, and produces label.
Correspondingly, our network has one backbone N, with T' output heads ho hr-1- We will use Nr, (X;) to denote the output of the tth head input X,.
We also define T loss functions, Lo LT-1, with each one corresponding task.
The goal is to minimize the loss on all of these tasks, given as:
Σ
La NN, (x.),)(())
Thus, instead of training the network to recognize just representations, we may utilize several representations together, learning them all at the same time by treating them as different tasks. The tasks all share the main network parameters, but each task has different head that learns the
3

---

Under review as a conference paper at ICLR 2023
representation specific to it. In this way reusing single supervision source to generate many related tasks.
We further can define T task weights, wo WT-1, with the constraints that all weights positive, minimize then becomes:
and that Σ w 1. The loss
We L, (N (X.),,
should be noted, that although the high level goal is to minimize the above, when calculating the gradients for backpropogation, often times better results are achieved by using more clever combi- nation techniques than addition (Crawshaw 2020
Standard Learning Procedure
One-hol
vector encoding
Labels
1xN
Loss
Input Data
Backbone Network
Vector
Softmax
1xN
Dense Labels Learning Procedure
One-hot vector encoding
Labels
1xN
Loss
Input Data
Backbone Network
Decoder
Vector
1xE
Multi Task Learning Procedure on label representations, the first which is one-hot encoded, second dense representation
One-hot
vector
encoding
1xN
Labels
Dense Repres- entation
Loss
Task
1xE
Input
Vector
Softmax
Data
1xN
Backbone Network
Vector
Task
Decoder
1xE
N is number of classes, E is embedding dimension
Figure I: Multi task learning on two label representations
4

---

Under review as a conference paper at ICLR 2023
To make this auxiliary learning problem, define task, to the main task, and the rest as auxiliary. After training finished, we can discard all the heads of the auxiliary tasks, and are left with network with output head trained on to-
Figure/ compares a softmax classification architecture, a dense label classification architecture, and multi task learning architecture training on 2 representations, softmax, dense.
4 BINARY LABELS
Binary labels, as the name implies, are vectors of bits, with each value being either a or a 1. Thus, each class is represented as a binary vector of length E, where E' is the embedding dimension.
While Binary Labels might seem like a strange choice to represent class labels, there are actually intuitive factors that motivate this representation as useful auxiliary task improve accuracy.
To illustrate why this representation is useful, consider single dimension of the embedding space. For this dimension, each class label will either be or 1, by definition. This splits the classes into two groups, the group of classes that were labeled 1 in this dimension, and the group labeled What this means that the neuron that learns to output this dimension of the label space is being trained on binary classification task, where the classes in the dataset have been split into two groups. Now lets consider the binary labels as a whole. For a set of labels of dimension E, the network is being trained E binary classification tasks, each one potentially of different split of the classes. Another way of thinking about this is that each bit learning the commonalities between different set of classes that are grouped together
Intuitively, having each dimension solve different binary classification problem makes sense as auxiliary task, since subgroups of classes that may not have been learned otherwise.
expect the network to learn useful features that distinguish between
The number of ways can divide all classes into 2 groups increases exponentially with the number of classes. For example, if we only have four classes, A D, the following binary labels would cover all 7 possible divisions (see figure 2).
Class A
Label
0
0001000 1
B
C
0
1000100
0
D
1 1000111
Binary Classifier on A,B,C vs. D
Binary Classifier on A,D vs. B,C
Figure 2: Binary Labels
Note, each column represents the same division if all the bits inverted.

---

Under review as a conference paper at ICLR 2023
In general, for N classes, we can divide them into unevenly split groups, this becomes Ex=1 K groups.
evenly split groups, and if we also consider
If N is small, it is feasible to use all of these divisions. As seen in figure 2. when N is 4, there only 7 such groups, 3 of which are evenly split. When is 10, there are 252 evenly split groups, and 637 total groups.
In these cases, when N is small, the binary labels could be carefully constructed to utilize every division. However, as N gets large, this is infeasible. For example, with 20 classes, already have 184756 evenly split divisions, far too many for us to attempt to learn each one.
Thus, for large number of classes, wish to only sample from the number of possible divisions. One very easy way of doing this is by simply randomly generating the binary labels, where each bit in each label has equal probability of being or 1.
In summary, the motivation behind using Binary labels as an auxiliary task is that it teaches the network distinguish between the classes many different ways, and this may provide useful features that can increase accuracy on the main task.
5 EXPERIMENTAL SETUP
We ran three sets of experiments. In the first, we demonstrate the usefulness of this method when training from scratch. We then demonstrate that our method leads accuracy gains when perform- ing transfer learning as well. Finally, we analyze data efficiency, and compare Binary Labels with other auxiliary label representations.
5.1 DATASETS
For the training from scratch experiments, we evaluate model on five datasets, Cifar-100 (Krizhevsky et al. 2009). Caltech101 (Fei-Fei et al.] 2004), Sun397 (Xiao et al. 2010), FGVC- Aireraft (Maji et al. 2013), and Stanford Cars (Krause al. 2013). Stanford Cars is particularly interesting since Resnets do surprisingly poorly on this dataset when trained from scratch 2
5.2 MODELS
On Cifar-100 100, we evaluated three models, VGG19 (Simonyan & Zisserman 2014), ResNet32, and ResNetl 10 (He et al., 2016). This choice was mostly made because we wanted to reproduce and improve upon the experiments of (Chen et al. 202T). where exactly these networks were used on C00 For the other datasets, we used Resnext-50 (Xie et al., 2017), Resnet-50 and Resnet-101
He et al. 2016).
For the transfer learning experiments, we used Caltech101 and FGVC-Aireraft, and evaluated on Resnet-50, Resnet- 101, VIT-B16 (Dosovitskiy et al. 2020), Beit-B16 (Bao et al. 202T), and MLP- Mixer-B16 (Tolstikhin al. 2021).
5.3 DATA EFFICIENCY
For the final experiments, where we analyzed how Binary Labels compare with other label represen- tations in the realm of data efficiency, we used VGG19 and Resnet-110, trained on Cifar-100, once again to be consistent with (Chen et al. 2021).
5.4 MULTI TASK ARCHITECTURE AND SETUP
In all of our multi-task experiments, we trained the network on two tasks. The main task was to predict the standard softmax one-hot labels, and the auxiliary task was predict Binary Labels. While the exact value E, the embedding dimension for the Binary Labels, is somewhat arbitrary,
While we did not find anything in the literature pointing this out, several open source implementations of Resnets Stanford Cars confirm this. See for example https///hhhb..cc///yyyyyyyhhahaa Baselines

---

Under review as a conference paper at ICLR 2023
in all of our experiments in this paper, E was equal to 4096, that we would be directly comparable to (Chen et al. 2021), where labels were given as 64x64 matrices.
Each network that trained had to be modified for the multi-task setting. We kept the standard softmax output as the head that predicts one-hot labels, which we defined as the main task. For predicting Binary Labels, the auxiliary task, appended each network dense label head. After training was done, this head was discarded, and the final networks that were produced are architec- turally equivalent
the standard architectures, and all comparisons are completely fair. (This also means that at inference time, our networks are just as fast as the baseline). Accuracy is reported on the main task. The dense label head similar to the decoders used in (Chen et al., 202I). except that we added a single fully connected layer to the end. For loss, we used cross entropy for the softmax heads, and binary cross entropy for the binary label heads.
There is also an important hyperparameter at play here, the weight we assign each task. For the experiments below, we tried two weighting schemes, and report the higher value. The two task weights used were [0.5,0.5], representing equal weighing on both tasks, and [0.9, 0.1], where we give clear priority to the main task and less priority to the auxiliary task.
RESULTS
6.1 TRAINING FROM SCRATCH
We present the results of the training from scratch experiments below.
The goal of these experiments was to demonstrate the value of multi task learning on redundant label representations, and specifically to demonstrate that Binary Labels generate useful auxiliary task. For all experiments, runs were averaged. We report an increase in accuracy for all architectures and datasets tested.
On Cifar-100, to be consistent with (Chen et al. 2021). we used the same models with the same hyper parameters such learning rate, weight decay, batch size and number of epochs.
The results on Cifar- 100 are below:
Table 1: Results on Cifar- 100 (aceuracy %)
Network
Standard
MTL. Binary Labels
error reduction
VGG19
70.765
71.57
2.7%
Resnet32
69.375
69.7975
1.3%
Resnet110
71.72
72.36
2.2%
While the accuracy gains on Cifar- were modest, we also ran experiments on Stanford Cars, Caltech-101, Sun-397, and FGVC-Aircraft. with much larger gains. For these experiments, we used Resnet50, Resnext50, and R01 We trained for 90 epochs, with an initial learning rate of 0. that relaxed by 0.1 every 30 epochs. We present the results below:
Table 2: Results on four other datasets (accuracy %)
Dataset
Cars
Caltech
Sun
Aireraft
Network
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Resnet50
13.27
27.11
63.95
72.63
27.74
39.26
56.38
63.52
Resnex150
6.38
41.15
67.37
71.03
30.5
39.86
44.4
54.55
Resnet101
17.29
31.9
68,55
72.98
30.58
37.42
52.66
68.26
The accuracy gains here quite strong across wide range of datasets and models, demonstrating that our method can be used as a general accuracy enhancement. Of particular note are the results on Stanford Cars, where the baseline networks struggled to learn anything.
7

---

As these results demonstrate, our method is useful accuracy enhancer when doing transfer learning as well. We also note that the lower the baseline accuracy is (i.e. the more challenging a task is for a given network), the more our method seems worthwhile, with the exception of MLP-Mixer on the Under review as a conference paper at ICLR 2023
We also want to note, Binary Labels were the only auxiliary label representation type tested on that gave accuracy improvements. The other label representations from (Chen et al. 2021) lead to decreases in accuracy. We further note that Binary labels only led to accuracy improvements when trained as an auxiliary task. As a main task, or as the only task, the accuracy when using Binary Labels decreased compared with the baseline. Thus, the benefit of Binary Labels is only realized when used in the context of our approach.
6.2 TRANSFER LEARNING
We also demonstrate the usefulness of our technique when using transfer learning.
All networks were pretrained on ImageNet. We used Resnet50, Resnet101, VIT-B16, Beit-B16, and MLP-Mixer representing a large range of SOTA network architectures. We trained on Caltech-101 and FGVC-Airerat. Training was done for 60 epochs, using the same hyperparameters as (Ridnik al. 2021 Again, we present the average of three runs. The results given below:
Table 3: Transfer learning results (accuracy %)
Dataset
Caltech
Aircraft
Network
Baseline
Ours
Baseline
Ours
Resnet50
95.6
96.17
71.99
75.14
Resnet101
96.5
96.9
71.52
76.45
VIT-BI6
96.62
96.5
82.97
83.86
Beit-B16
96.94
97.07
84.79
85.9
MLP-Mixer
82.39
83.91
27.67
24.82
Aircraft dataset, where both models did very poorly, but ours did worse. Once again, want to point out that Binary Labels were the only label representation that led to accuracy improvement, and even then only when used an auxiliary task.
6.3 DATA EFFICIENCY
While Binary Labels were superior to the other label representations given in (Chen al.) 2021
when it came to accuracy, we now explore another key metric: data efficiency. Data efficiency has been widely researched in the context of meta learning and few shot learning (Thrun & Pratt 2012
Vilalta & Drissi] 2002, Vanschoren 2018: Wang & Yao, 2019), and indeed, (Chen et al. 2021 noted that they did not see accuracy improvements when training on their label representations, but reported strong gains in data efficiency. Thus, it is only natural to see how Binary Labels fair when it comes to data efficiency. We compare them with random labels generated from a uniform distribution, since this representation achieved strong efficiency gains in (Chen et al.) (021). Results are on Cifar- 100, using the same hyper parameters as above, with one difference: taks weights are [0.1,0.9], where give when it comes to data efficiency, these parameters lead to stronger results.
weight to the auxiliary task of learning the dense labels. We found
We present results from training 1%, 2%, 4%, and 8% of the dataset:
8

---

Under review as a conference paper at ICLR 2023
Table 4: Data efficiency: training small percentages of the dataset (aceuracy %
Percentage
1%
2%
Network
Baseline
Binary
Random
Baseline
Binary
Random
VGGT9
1.86
9.24
7.93
4.4
13.03
12.39
Resnet 10
4.3
6.32
6.11
8.5
9.25
8.83
Percentage
4%
8%
Network
Baseline
Binary 18.36
Random 18.03
Baseline 17.4
Binary 30
Random
VGG19
9.4
22.23
Resnet 10
15.31
16.11
13.05
25.1
25.61
23.66
The results are quite interesting. While both multi task methods generally did better than the base- line, Binary Labels outperformed random labels, and the difference was far more pronounced on VGG-19 than Resnet-110, 10. Given that state of the art models rely heavily large amounts of train- ing data (Krizhevsky et al., [2099] Russakovsky et al.) 2013; Kuznetsova et al. 2020), something generally not available for real world problems, this is very useful property.
CONCLUSION AND FUTURE DIRECTIONS
We present a completely new paradigm: Using alternative label representations generate auxil- iary tasks that then learned in parallel, using techniques from multi task learning, We propose Binary Labels as a simple method of obtaining an auxiliary task, and achieve strong results. Our experiments demonstrate that this is an exciting new direction.
However, this is an entirely new learning paradigm, and there remains a lot to explored. We propose the following questions, and think exploring them can lead stronger results than the ones presented in this paper:
1. What types of label representations make for good auxiliary tasks? Can we improve upon the Binary Labels presented here?
2. It is possible different representations are useful depending on the goal. Can we find repre- sentations that help with accuracy, robustness, efficiency, convergence, etc. ?
3. In this paper, only explored using two different label representations and training two tasks. However, it is possible a larger number would lead to better results. We would like to explore this direction well, especially considering that different representations may help us different ways (i.e. accuracy vs robustness).
4. We found that the benefits of the auxiliary task was very sensitive to how we combined the loss functions. What the ideal way do this?
5. For the dense label heads, we used the decoder from (Chen al. 2021 1), with an additional fully connected layer. We tried other schemes that were not as useful. Even though these output heads are discarded after training, the question remains: what output head for dense labels produces the strongest results?
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint WXKv:22006002254. 2021.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Boyuan Chen, Li, Sunand Raghupathi, and Hod Lipson. Beyond categorical label representations for image classification. arXiv preprint arXiv:2104.02226. 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1697-1607. PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener. Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint rrX1:2209.06694 2022.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pp. 794-803. PMLR, 2018.
Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 245-255. leee, 2009.
Diego Doimo, Aldo Glielmo, Sehastian Goldt, and Alessandro Laio. Representation mitosis in wide neural networks. arXiv preprint (tt::/006.0348. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint wwXiv:2010.11929, 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022a. 2022a.
Yun He, Xue Feng, Cheng Cheng, Geng Ji, Yunsong Guo, and James Caverlee. Metabalance: Improving multi-task recommendations via adapting gradient magnitudes of auxiliary tasks. In Proceedings of the ACM Web Conference 2022, pp. 2205-2215 2022b.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):196-119811 2020.
Lei Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. Advances in neural information processing systems, 31, 2018.
Lukas Liebel and Marco Körner. Auxiliary tasks in multi-task learning. 10Xiv:1005.000 06334, 2018.
arXiv preprint
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint arXiv: arXiv:1306.5111 1306.5/51, 2013.
Itzik Malkiel and Lior Wolf. Mtadam: Automatic balancing of multiple training loss terms. arXiv preprint ar arXiv:2006. 14683, 2020.
10

---

Under review as a conference paper at ICLR 2023
Marika Pieszek, Andreas Widmann, Thomas Gruber, and Erich Schröger. The human brain main- tains contradictory and redundant auditory sensory predictions. PLoS One, 8()):433334, 2013.
Walter Hugo Lopez Pinaya, Sandra Vieira, Rafael Garcia-Dias, and Andrea Mechelli. Autoencoders. In Machine learning, pp. 193-208. Elsevier, 2020.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses. arXiv preprint vrX1v:2104.10972, 2021.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 1rXiv:1706.05098, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2211-22,, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pp. 9120-911 9132. PMLR, 2020.
Xu Sun, Bingzhen Wei, Xuancheng Ren, and Shuming Ma. Label embedding network: Learning label representation for soft training of deep networks. arXiv preprint arXiv:/710. 10393, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34:24261- 24272, 2021.
Partoo Vafaeikia, Khashayar Namdar, and Farzad Khalvati. A brief review of deep multi-task learn- ing and auxiliary task learning. arXiv preprint arXiv:2007.01 126, 2020.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey, IEEE transactions on pattern analysis and machine intelligence, 2021.
Joaquin Vanschoren. Meta-learning: survey. arXiv preprint arXiv: 2018.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-leaming. Artificial intelligence review, 18(2):77-95, 2002.
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho, and Marius Hobbhahn. Machine learning model sizes and the parameter gap. arXiv preprint arXiv:2207.02885. 2022.
Yaqing Wang and Quanming Yao. Few-shot learning: A survey. 2019.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020a.
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.05874, 2020b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23998. PMLR, 2022.
11

---

Under review as a conference paper at ICLR 2023
Jianxiong Xiao, James Hays. Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485- 3492. IEEE, 2010.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500. 2017.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 2rXiv:2205.01917, 2022.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
12


Reviews:
[{"review_id": "iVzOz6hbd0k", "reviewer": "Reviewer_pCpT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose improving the classification performance of neural networks by adopting a multi-task approach to label prediction. \nIn addition to the usual one-hot encodings, they consider random binary partitions of the classes and use them as classification tasks in a multi-task setting. Experiments show small improvements over baseline methods using standard one-hot encoding, especially for classification in special domains such as distinguishing different types of cars and airplanes.  \n", "summary_of_the_review": "The authors produce a piece of interesting work on using random binary partition of class labels as auxiliary tasks to improve the prediction accuracy of multi-class classification problems. While the result alone might not be significant enough for ICLR, the authors can consider expanding the results by exploring some of the questions they listed in the conclusions section. \n", "strength_and_weaknesses": "Strengths: \n- The method is simple and it works. The use of random binary partition of classes create useful related tasks for the neural network to train on. This might be important for more difficult classification tasks (e.g. cars and airplanes in the paper), where the sampled binary classification tasks can be easier than the original multi-class problem.  \n\nWeaknesses: \n- The use of random binary partitions can be a strength but can also be a weakness, since unrelated classes can be grouped together. If class hierarchies are available (such as the ImageNet class hierarchy), they might improve upon the random binary partitions used in this work. \n\n- While the main idea of the paper is simple and effective, the contribution might not be significant enough due to the small empirical improvements, and the lack of further analysis and insights into the problem of label encoding.  \n\n- Some classic works on multi-class label coding are missing, e.g.: \nDietterich, Thomas G., and Ghulum Bakiri. \"Solving multiclass learning problems via error-correcting output codes.\" Journal of artificial intelligence research 2 (1994): 263-286.\nCrammer, Koby, and Yoram Singer. \"On the learnability and design of output codes for multiclass problems.\" Machine learning 47.2 (2002): 201-233.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "anXzPrvHK7", "reviewer": "Reviewer_gbMZ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to improve image classification performance by modifying labels into various shapes and combining them. In each variant, the classes are separated 2-fold and labeled 0 and 1, respectively. Randomly selected combinations are used for classification training in combination with the existing softmax-based loss. In some limited experiments, this algorithm shows improved performance.", "summary_of_the_review": "I think that the weaknesses must be sufficiently improved for this paper to be accepted.", "strength_and_weaknesses": "The approach of improving performance by transforming labels is novel. This approach may be a direction that can improve performance in a way that is cheaper than data and network architecture improvements.\n\nIt is not easy to agree with the authors' claim that research on labels has been ignored. For example, knowledge distillation is a representative research direction that attempts to learn the more complex distribution of labels. Also, ordinal regression is labeled in a differentiated way to a one-hot vector. The authors should have discussed many studies related to labels.\n\nThe notations are generally poor, with unnecessary content and many calculation errors. The proposed algorithm is a trial without insight or analysis. For example, on page 3, it is difficult to understand what f_0 - f_{T-1}, h_0 - h_{T-1} mean. The three equations on pages 3 and 4 have little relation with the algorithm's core, and one of these equations is sufficient.\nIn the case of Chapter 4 on page 6, according to the authors' formula, when N=4, the number of groups becomes 10 instead of 7. This does not match the example in Figure 2.\n\nExperimental results are insufficient to show the effectiveness of the algorithm. For most datasets, the baseline algorithm was set to those with too low performance. It is necessary to compare the SOTA method and the proposed algorithm.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ag3YJlpGsvE", "reviewer": "Reviewer_tGTN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an approach to train a neural network for a multiclass image classification task. The main contribution is to add an auxiliary task, where Binary Labels are used as supervision. The Binary Labels can be carefully constructed or generated randomly based on the number of classes. The proposed approach is evaluated on five datasets: Cifar-100, Caltech101, Sun397, FGVC- Aircraft, and Stanford Cars. The proposed approach also seems to improve accuracy when the training is done with a subset of the training examples. \n", "summary_of_the_review": "The technical contributions of the paper seems to be too limited. The main contribution is to train a neural network architecture with an auxiliary task that uses Binary Labels as supervision but it is difficult to understand why it improves performances. It seems there is no technical contribution on the multi-task learning part. ", "strength_and_weaknesses": "**Strengths**\n- The proposed auxiliary task seems to be quite easy to add to existing neural network architectures.\n- The proposed approach increases the accuracy on five datasets: Cifar-100, Caltech101, Sun397, FGVC- Aircraft, and Stanford Cars. The proposed approach also seems to improve accuracy when the training is done with a subset of the training examples. \n\n**Weaknesses**\n- The technical contribution of the paper seems limited. The main idea is to use Binary Labels as an auxiliary task during training. The multi-task learning architecture is basic: one independent branch for each task. \n- The paper should clearly explain how the Binary Labels are generated in the experiment section. There are multiple approaches that are presented in Section 4, but it is not clear which one is used in the experiment section.\n- I am not sure to understand what is the motivation/intuition behind the Binary Labels. The Binary Labels do not bring semantic/meaningful information about the categories so why do they improve performances. I think the quality of the paper will improve if there was a paragraph to discuss about the properties that are modelled by Binary Labels but that are not modelled by one-hot labels. In other words, if Binary Labels is the solution, what is the problem? \n- In the experiment section, it is important to report the standard deviation when the runs are averaged. It is important to understand the sensibility of the model to the random seed used to generate the Binary Labels.  \n- I wonder if data efficiency is a good metric for this paper because it is about label representation. I think that label efficiency may be a better metric. For a given label budget (i.e X labeled images and Y unlabelled images), what is the best performance? Data efficiency is maybe not a good metric because it is quite easy to collect a lot of images without labels, and then use a self-supervised learning algorithm to train a model. \n- The proposed approach seems to be limited to multiclass image classification and does not seem to generalize to multilabel image classification.\n- The proposed approach is not evaluated on large benchmarks like ImageNet. It will increase the quality of the paper to show the proposed approach generalizes to a large benchmark. \n- There is no analysis of the weighting between the two tasks. It can be useful to add an analysis to show the robustness of the model to the weighting strategy. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UADagLRpC3-", "reviewer": "Reviewer_2fNv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes ‘binary labels’, where each dimension of the embedding space is assigned a 0 or 1 for each class, which is decided randomly at label construction time, before the start of training). Intuitively, at each embedding dimension, this leads to a grouping of classes (those that have a 0 in that dimension, and those that have a 1 there). So training to predict these labels can be thought of incentivizing the model to learn commonalities between different groups of classes. They propose to predict these binary labels as an auxiliary task, jointly with the main task of predicting the usual one-hot labels. They show that in some cases, both for in-distribution generalization and transfer learning, using this auxiliary task improves performance compared to the baseline of using only the main task. They also show that it can improve data efficiency (achieving higher accuracy with fewer updates) compared to that same baseline.", "summary_of_the_review": "Overall, the authors propose a really interesting auxiliary task to co-train with the main task of predicting one-hot labels. The results are encouraging in some of the scenarios explored. However, I felt that this exploration is still too premature to meet the bar for publication. Additional baselines, analyses and experimental scenarios should be considered to gain more insights into the performance of the proposed approach. Clarity issues should also be addressed, and related work should be discussed more thoroughly.", "strength_and_weaknesses": "Strengths\n========\n[+] The paper proses an interesting idea and I agree that label representations are an understudied aspect which is orthogonal to where large gains usually come from (improving models and data)\n\n[+] In some cases, the proposed method improves results significantly over vanilla training.\n\nWeaknesses\n===========\n[-] In some cases, the performance improvement is very small compared to only training with softmax labels, and no intuition or analysis is provided to shed some light on when we expect this approach to help. For instance, is it for datasets with more finegrained labels? What are other factors that affect this?\n\n[-] The experimental section is weak. It would have been useful to compare against additional baselines, perform more analyses, explore different datasets and scenarios (concrete details in the below section)\n\n[-] The paper has some clarity issues.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject