Beyond Categorical Label Representations for Image Classification
Authors: Boyuan Chen
Yu Li
Sunand Raghupathi
Hod Lipson
Venue: ICLR 2021 Poster (2021)
Keywords: Label Representation
Image Classification
Representation Learning

Abstract: We find that the way we choose to represent data labels can have a profound effect on the quality of trained models. For example, training an image classifier to regress audio labels rather than traditional categorical probabilities produces a more reliable classification. This result is surprising, considering that audio labels are more complex than simpler numerical probabilities or text. We hypothesize that high dimensional, high entropy label representations are generally more useful because they provide a stronger error signal. We support this hypothesis with evidence from various label representations including constant matrices, spectrograms, shuffled spectrograms, Gaussian mixtures, and uniform random matrices of various dimensionalities. Our experiments reveal that high dimensional, high entropy labels achieve comparable accuracy to text (categorical) labels on standard image classification tasks, but features learned through our label representations exhibit more robustness under various adversarial attacks and better effectiveness with a limited amount of training data. These results suggest that label representation may play a more important role than previously thought.

Published as a conference paper at ICLR 2021
BEYOND CATEGORICAL LABEL REPRESENTATIONS FOR IMAGE CLASSIFICATION
Boyuan Chen, Yu Li, Sunand Raghupathi, Hod Lipson
Columbia University
nes http://eeeeeeeceeeeeeeeeeeeeeeee htw.:////aabblrrppppeeeeeettatoa http:///we.aaaaaeeeeeee///eeeeea https: /tpp:://www...
1 html
ABSTRACT
We find that the way we choose to represent data labels can have a profound ef- fect on the quality of trained models. For example, training an image classifier regress audio labels rather than traditional categorical probabilities produces a more reliable classification. This result is surprising, considering that audio labels are more complex than simpler numerical probabilities text. We hypothesize that high dimensional, high entropy label representations are generally more use- ful because they provide stronger error signal. We support this hypothesis with evidence from various label representations including constant matrices, spectro- grams, shuffled spectrograms, Gaussian mixtures, and uniform random matrices of various dimensionalities. Our experiments reveal that high dimensional, high entropy labels achieve comparable accuracy to text (categorical) labels on the stan- dard image classification task, but features learned through our label representa- tions exhibit more robustness under various adversarial attacks and better effec- tiveness with limited amount of training data. These results suggest that label representation may play more important role than previously thought.
INTRODUCTION
Image classification is a well-established task in machine learning. The standard approach takes an input image and predicts categorical distribution over the given classes. The most popular method train these neural network through cross-entropy loss with backpropagation. Deep convolu- tional neural networks (Lecun et al. 1998 Krizhevsky et al., 2012, [Simonyan & Zisserman, [014]] He et al., 2015 Huang et al. 2016) have achieved extraordinary performance on this task, while some even surpass human level performance. However, is this solved problem? The state-of-the- art performance commonly relies on large amounts of training data (Krizhevsky) [2099] [Russakovsky let al. 2015 Kuznetsova al.) (0188). and there exist many examples of networks with good perfor- mance that fail on images with imperceptible adversarial perturbations (Biggio et al. 2013 Szegedy
al. 2013 Nguyen al. 2014)
Much progress has been made in domains such as few-shot learning and meta-learning to improve the data efficiency of neural networks. There also large body of research addressing the chal- lenge of adversarial defense. Most efforts have focused on improving optimization methods, weight initialization, architecture design, and data preprocessing. In this work, find that simply replacing standard categorical labels with high dimensional, high entropy variants (e.g. an audio spectrogram pronouncing the name of the class) can lead interesting properties such as improved robustness and efficiency, without a loss of accuracy.
Our research is inspired by key observations from human learning. Humans appear to learn to recognize new objects from few examples, and are not easily fooled by the types of adversarial perturbations applied to current neural networks. There could be many reasons for the discrepancy between how humans and machines learn. One significant aspect is that humans do not output categorical probabilities on all known categories. A child shown a picture of a dog and asked "what is this a picture of?" will directly speak out the answer - "dog." Similarly, child being trained by a parent is shown a picture and then provided the associated label in the form of speech. These observations raise the question: Are supervising neural networks the best modality?

---

Published as a conference paper at ICLR 2021
22222
2222
22222 2222
222 2222
le
sd
"bird"
222 2222
222
2222
Speech
Shuffled Speech
Composite Gaussion
Constant
Figure I: Label Representations beyond Categorical Probabilities: We study the role of label repre- sentation in training neural networks for image classification. We find that high-dimensional labels with high entropy lead more robust and data-efficient feature learning.
In this paper, we take one step closer to understanding the role of label representations inside the training pipeline of deep neural networks. However, while useful properties emerge by utilizing various label representations, we do not attempt to achieve state-of-the-art performance over these metrics. Rather, we hope to provide novel research perspective the standard setup. Therefore, our study is not mutually exclusive with previous research on improving adversarial robustness and data efficiency.
An overview of our approach is shown Figure We first follow the above natural observation and modify the existing image classifiers to "speak out" the predictions instead of outputting categorical distribution. Our initial experiments show surprising results: that neural networks trained with speech labels learn more robust features against adversarial attacks, and more data-efficient when only less than 20% of training data is available.
Furthermore, we hypothesize that the improvements from the speech label representation come from its property as specific type of high-dimensiona object. To test our hypothesis, we performed large-scale systematic study with various other high-dimensional label representations including constant matrices, speech spectrograms, shuffled speech spectrograms, composition of Gaussians, and high dimensional and low dimensional uniform random vectors. Our experimental results show that nigh-dimensional label representations with high entropy generally lead to robust and data effi- cient network training. believe that our findings suggest significant role of label representations which has been largely unexplored when considering the training of deep neural networks.
Our contributions are three fold. First, we introduce new paradigm for the image classification task by using speech the supervised signal. We demonstrate that speech models achieve comparable performance to traditional models that rely on categorical outputs. Second, we quanti- tatively show that tigh-dimensional label representations with high entropy (e.g. audio spectrograms and composition of Gaussians) produce more robust and data-efficient neural networks, while high- dimensional labels with low entropy (e.g. constant matrices) and -ow-dimeniional labels with high entropy do not have these benefits and may even lead to worse performance. Finally, we present set of quantitative and qualitative analyses systematically study and understand the learned feature representations of our networks. Our visualizations suggest that speech labels encourage learning more discriminative features.
2 RELATED WORKS
Data Efficiency and Robustness Data efficiency has been widely studied problem within the context of few-shot learning and meta-learning (Thrun Pratt 2012] Vilalta Drissil schoren 2018 Wang et al.) 2019). Researchers have made exciting progress on improving methods of optimization (Ravi Larochelle 2016 Li al. (017), weight initialization (Finn et al. 2017
2002 Van-
Ravi & Larochelle 2017). and architecture design (Santoro et al. 2016a b).
There is also a large body of research addressing the challenge of adversarial defense. Adversarial training is perhaps the most common measure against adversarial attacks (Goodfellow et al. 2014
Kurakin et al. 2016; Szegedy et al. 2013] [haham et al. [2018] Madry et al. Recent works try
to tackle the problem by leveraging GANs (Samangouel et al. 2018), detecting adversarial examples
(Meng & Chen 2017 Lu et al. 2017 Metzen et al. 2017). ) and denoising or reconstruction (Song
al. 2017
Liao al. 2017)
2

---

Published as a conference paper at ICLR 2021
Most of these techniques for improving data efficiency and adversarial robustness study the problem from the perspective of model, optimizer and data. Relatively little research has been conducted on the labels themselves more specifically, their representation. Hosseini et al. (2017) augmented categorical labels with a new NULL class to allow the model to classify and reject perturbed exam- ples. Papernot et al. (2013) utilizes model distillation (Hinton et al.] 2015) for adversarial defense. Papernot & McDaniel (2017) further augment the label used to train the distilled model with the predictive uncertainty from the original model. Nevertheless, the method Hosseiii al. (2017) requires adversarial examples to train on while that of defensive distillation has been shown to be vulnerable to substitute model black-box attacks (Papernot et al.) 2017).
Label Smoothing The closest approach to our work is Label Smoothing (LS) (Szegedy et al. 2016), Here we highlight key differences between our approach and I.S. First, LS applies to discriminative outputs where both correct and incorrect class information are presented during training, while our output is generative and only correct class information is presented. That is, our outputs are not a distribution over classes. Although LS has been shown to improve adversarial robustness (Goibert) & Dohmmtob, 2020), it has not been shown to be effective for low-data learning. As we will show in our experiments, LS does not help when the amount of training data is limited, while our label representations lead to significant improvements. Therefore, our high-dimensional, high-entropy labels provide benefits beyond those provided by label smoothing.
3 BEYOND ACCURACY: EMERGENCE OF ROBUSTNESS AND EFFICIENCY
It is well-known that deep neural networks with similar accuracy on the same task may perform very differently under different evaluation scenarios. Additionally, real-world applications rely on more considerations than just accuracy. Robustness and data efficiency are two practical challenges for deep neural networks. We test the emergence of those properties under various label representations.
3. ROBUSTNESS UNDER ADVERSARIAL ATTACKS
We evaluate the robustness of all the trained models using the fast gradient sign method (FGSM) Goodfellow et al. 2014) and the iterative method (Kurakin et al., (016) across multiple widely used convolutional networks. We choose these attacks because their adversarial images Inly usually indistinguishable from the original images or do not significantly affect human evaluation, but they can be very challenging for neural networks to correctly classify. When loss function J is involved in generating the adversarial images, we use the cross-entropy loss for the text model and the smooth LI loss for the speech model.
FGSM is fast one-step attack that generates an adversarial image by adding small adversarial perturbation to the original image. The perturbation is based on the gradient of the loss with respect to the original image. The maximum magnitude of the perturbation is maintained by e:
|| - -aavllle S.
(1)
We test both untargeted and targeted versions of FGSM. The untargeted attacks increase the loss between the predicted class and the true class Yiue:
Jul- = I e [ Yiue)),
whereas the targeted attacks decrease the loss between the predicted class and a target class Yiager
Ladv = I - e Sign (1, Ymugs)).
(3)
We choose a random incorrect class as the target class for each input image, and the same target classes are used to test different models. All lavv are normalized after the perturbation.
Iterative Method As an extension to FGSM, the iterative method applies multiple steps of gradient- based updates. In experiments, initialize the adversarial image Iav to be the original image so that adv = I. Then we apply FGSM for 5 times with a small step size a = e/5. The untargeted update for each iteration becomes
N+ Clip,,, ( IN +a -iin You)))
and the targeted update becomes
N+1 Clip,, (1) a- Sign((Vz (1 Yunge)))
(5)
3

---

Published as a conference paper at ICLR 2021
where C denotes clipping the total perturbation ENN in the range of [-c,c]. We use the same targeted classes from FGSM for the evaluation on the iterative method.
3.2 LEARNING EFFICIENCY WITH LIMITED DATA
We take the most straightforward approach to evaluate data efficiency. We start with only 1% of the original training data. We always use the full amount of testing data. We then gradually increase the amount of training data to 2%, 4%. 8%, 10%, and 20% of the original perform extensive multi-scale evaluation.
4 EXPERIMENTAL SETUP
Dataset We evaluate our models on the CIFAR- 10 and CIFAR-100 datasets (Krizhevsky) 2009). We use the same training, validation and testing data split (45, 000/5, 000 10, 000) for all of exper- iments. We also keep the same random seed for data preprocessing and augmentation. Therefore, present apple to apple comparisons for all label representations.
Speech Label Generation We generate the speech labels shown in Figure by following the stan- dard practice in recent works (Naranjo-Alcazar al., 2019 Zhang al. 20199:
We first generate the English speech audio automatically with a text-to-speech (TTS) sys- tem from the text labels in the corresponding dataset. Therefore, all the speech labels pronounced consistently by the same API with the same parameters for controlled experi- ments. We leave the exploration of different languages and intonations as future work.
We save each audio file in the WAVE format with the 16-bit pulse-code modulation encod- ing, and trim the silent edges from all audio files.
Since different speech signals may last for various lengths, we preproces="each speech la- bel to generate Log Mel spectrogram to maintain the same dimension. We use a sampling rate of 22, 050 Hz, 64 Mel frequency bands, and a hop length of 256. Another advantage of this preprocessing into spectrograms is that we can then utilize convolutional neural net- works as the speech decoder to reconstruct our speech labels. Mcanwhile, we convert the amplitudes to the decibel scale.
Finally, the spectrograms are shaped into N x N matrix with values ranging from -80 to 0, where is double the dimension of the image input. Our resulting speech spectrogram can be viewed 2D image.
Given the first step in this procedure, note that for a given class (e.g. "bird") there is only one corresponding spectrogram. Therefore, the improved robustness observe is not result of any tabel-augmentation.
Other Labels For a deeper understanding of which properties of speech labels introduce dif- ferent feature learning characteristics, we replicate all the experiments using the following high- dimensional label variants. We also show visualizations for all the label variants in Figure
cruffled-speech We cut the speech spectrogram image into 64 slices along the time dimen- sion, shuffle the order of them, and then combine them back together as an image. Although the new image cannot be converted back to meaningful audio, it preserves the frequency information from the original audio. More importantly, this variant does not change the entropy or dimensionality of the original speech label.
constant-matrix contrast, the constant matrix represents another extreme of high- dimensional label representation, with all elements having the same value and zero entropy. The constant- matrix label has the same dimension as the speech label, and values are evenly spaced with range of 80 (which is also the range of the speech labels).
Gaussian- comiiaancoossiiionn We obtain this representation by plotting a composition of 2D Gaus- sians directly images. Each composition is obtained by adding 10 Gaussians with uni- formly sampled positions and orientations.
random/uniform-noa We also adopt matrix with same dimensions as the above high- dimensional labels by randomly sampling from uniform distribution. We additionally
hhtts://cloud wtwp://wwwwooooloooooooooo/ooe/e http:////odd...gog...o////ttoooe htt..cco/m//etttt--pcchhh
hhtps://ibrmsa.ghhuuii
4

---

Published as a conference paper at ICLR 2021
construct uniform random vector with low dimensionality inspect whether dimen- sionality matters. Throughout the paper, we will use random matrix and uniform matrix interchangeably to refer to the same representation.
BERT embedding We obtain the BERT embedding from the last hidden state of a pre- trained BERT model (Devlin al. 2018] |Wolf al.) 2020). We remove outlines larger than twice the standard deviation and normalize the matrix to the same range of our other high-dimensional labels. The BERT embedding results in 64 64 matrix.
CloVe embedding Similarly, we directly use the pretrained Glo Ve (Pennington et al. 2014) This This results in 50-dimensional label vector.
word vectors
Models We take three widely used convolutional networks as our base model: VGG19 (Simonyan & Zisserman 2014), ResNet-32 and ResNet- 110 (He et al., 2015). Here we define that a categorical classification model has two parts: an image encoder Te and a category (text) decoder Tza- Tradi- tionally, Le represents fully connected layers following various forms of convolutional backbones Le. Finally, softmax function is applied to the output from the last layer of Led to convert the pre- dictions to a categorical probability distribution. The prediction is retrieved from the class with the highest probability.
Similarly, we define that the models for our encoder and label decoder Iis- Throughout the paper, the same image encoder but replace the category decoder lt with one dense layer and several transpose convolutional layers as Tta. All the layers inside Jia are equipped with batch normalization (loffe & Szegedy 20155) and leaky ReLU with a 0.01 negative slope.
labels consists of two parts: an image
Overall, the majority of parameters of the network comes from the image encoder which is shared across both categorical labels and other label representations. The decoder for high-dimensional labels increases the number of parameters by a very limited amount (see Appendix for all the num- bers). Thus, our experiments are well-controlled with respect to the number of parameters.
Learning We train the categorical model to minimize the traditional cross-entropy objective func- tion. We minimize Equation 6 for other models with high-dimensional labels. We use the smooth L1 loss (Huber loss) L, shown in Equation 7 Here, yi is the predicted label matrix while s, stands for the ground-trute matrix.
min Σ C.., si)
(6)
0,
(1: si)?, y 1
Co 8:)
(7)
18 - s.l. otherwise
We optimize all the networks using stochastic gradient descent (SGD) with back-propagations and select the best model based on the accuracy on the validation set.
Evaluation For categorical models, a prediction considered correct the class with the highest probability indicates the same category as the target class. For high-dimensional labels, we provide two types of measurements. Given the model output, we select the ground-trutt label that minimizes the distance to the output. We refer to this the "nearest neighbor" (NN) choice. The other criteria is to check whether the smooth L1 loss is below a certain threshold. We use Amazon Mechanical validate that generated speech below our threshold is correctly
Turk (Sorokin & Forsyth| 2008) identihable by humans. In our experiments, we find 3.5 is reasonable threshold. The human evaluations on the speech label demonstrate that our metric captures both the numerical performance and the level of interpretability of the generated speech output. Note that we mainly rely on the NN method for evaluation and only refer to the threshold method to demonstrate the qualitative results from the speech label.
5 RESULTS AND DISCUSSION
5.1 Do ALI. THE MODELS LEARN TO CLASSIFY IMAGES?
We report the classification accuracy for all the labels in Table Speech labels, shuffled-speech and Gaussian-compositton labels achieve comparable accuracy with the traditional categorical labels,
5

---

Published as a conference paper at ICLR 2021
while the constant- matrix performs slightly worse than these representations. This suggests that the constant-matris label is harder to train with. We verify this observation by visualizing the training curves for all label representations The training curves show that the constant-matrix model takes longer to converge than the others, and converges to higher loss.
CIFAR-100 with the ResNet- 10 image encoder in Figure 2
CIFAR-10 10 Accuracy (%)
CIFAR- 100 Accuracy (%)
Labels
VGG19
ResNer32
ResNer110
VGG19
RexNet32
R1
Cutegory
92.82 + 0.08
92.34 0.25
93.23 = 0.29
70.98 0.10
68.00 08.05 + 0.72
70.03 = 0.49
Speech Threshold
91.97 + 0.17
91.90
0.04
92.44 0.11
69.13
0.75
61.08 + 0.27
67.88 + 0.16
Speech NN
92.12 +: 0.18
92.34 +: 0.01
92.73 = 0.08
70.27
0.61
64.74 + 0.36
69.51 # 0.25
Shuflle Threshold
92.27 + 0.16
91.49 + 0.22
92.44 + 0.05
67.04
0.41
51.95 + 0.85
63.00 + 1.45
Shuffle NN
92.61 + 0.19
92.72 + 0.20
92.92 = 0.24
70.88 = 0.31
64.23 + 0.80
69.41 +: 0.66
Composition Threshold
91.53 +: 0.24
91.44
0.22
92.36 + 0.02
68.06
0.28
60.54 + 0.54
67.17 0.40
Composition NN
99.91 0.22
92.39
0.17
93.07 0.03
70:20
0.23
72 + 0.44
70.62 0.20
Constant Threshold
88.27 0.63
88.50
0.25
89.27 +: 0.15
62.99 0.11
55.70 4: 0.36
58.78 0.18
Constant NN
88 $8.33-2 + 0.65
88.61
+: 0.27
89.37 # 0.13
34..99 + 2.40
19.00 1.19
24.46 +: 3.70
Table 1: Classification accuracy on CIFAR- 10 and CIFAR-100 100 for all label representations. Speech labels, shuffled speech labels, and composition of Gaussian labels all achieve comparable accuracies with categorical labels. Constant matrix labels perform slightly worse than the others.
Figure 2: Training and validation losses on CIFAR-10 dataset with ResNet-110 10 image encoder for mod- els with the speech / shuffled speech / composition of Gaussians / con- stant matrix labels (left) and categor- ical labels (right). All of the mod- els are trained to converge. The model trained with constant matrix labels converges slower than models trained with other high dimensional labels.
Highh-iimensional
Category
Constant
Smooth L1 Loss
20
Cross-entoppy Less
2
Speech
1.
1.5
Shuffle
Composite
10
Category
0.5
Ayy
0
150
300
450
600
50
100
150
200
Epoch
Epoch
5.2 FEATURE ROBUSTNESS
In order evaluate the robustness of the models, we take all the trained models (Table [ as the starting points for adversarial attacks with the FGSM and the iterative method. We apply an e from 0 0.3 with a 0.05 increment to the normalized images by following the original FGSM setup Goodfellow et al., 2014) and test the model accuracy for each value. We only run attacks on the original correctly classified images and mark the original misclassified images as incorrect when we compute the accuracy for all e values. We provide the accuracy computed on the subset of test images that are initially correctly classified in the Appendix, though the ranking among different models remains the same.
Figure 3 shows the test accuracy under various attacks. Although the accuracy of all models de- creases as the attack becomes stronger (larger c), the models with speech labels, shuffled speech labels, and composition of Gaussian labels perform consistently much better than the models with traditional categorical labels across all three image encoders, for all types of attacks, and on both CI- FAR datasets. Uniform random matrix labels perform similarly well in this setting (see the Appendix for details). Interestingly, models with constant matrix labels perform worse than all other models with high- dish-dimnssooal labels, suggesting that there are some inherent properties that enhance model robustness beyond simply high dimensionality.

---

Published as a conference paper at ICLR 2021
(lntarrated)) ("acgeted//brattio (Urrargeeedjerattee (Targeted)
(lntargated) (Twrgebed/bbrattoo (Urtargeiesjeeette (Targeled)
ResNe33
ResNer32
Reshet11D
Reshet110
VGG19
VGG19
Epsion
Epstos
Figure 3: Test accuracy under adversarial attacks on CIFAR-I0 (left four columns) and CIFAR- (right four columns). The accuracy evaluated by the threshold and the nearest neighbor is plotted in solid and dotted lines respectively. We show the results of targeted and untargeted FGSM and iterative method on three image encoders with three random seeds. The horizontal axis indicates the strength of different attacks.
5.3 FEATURE EFFECTIVENESS
With the CIFAR-10 10 dataset, we train models with various label representations using 1%, 2%, 4%, 8%, 10%, and 20% of the training data. For each amount of data, we train with the VGG19, ResNet- 32, and ResNet-1 1 10 image encoders with five different random seeds. To conduct controlled experi- ments, we use the exact same training procedures and hyperparameters
the full-data experiments, so that the only difference is the amount of training data. All models are evaluated on the same val- idation set and test set. Figure 4 reports the test accuracy. Similar to the results from the robustness evaluation, speech labels, shuffled speech labels, composition of Gaussian labels, and uniform ran- dom labels achieve higher accuracies than the models with categorical labels for both VGG19 and ResNet- 1 100 and comparable results for ResNet-32. The results demonstrate that the speech models are able to learn more generalizable and effective features with less data. This property is extremely valuable when the amount of training data is limited.
Additionally, our results suggest that label- smoothing does not provide further benefits when the amount of training data is limited, discussed above. Lastly, the performance of the models trained on constant matrix labels is consistent with that in the robustness experiment: it performs worse than all other high- -igheiinnniooall labels. We provide further analysis in the next section.
ResNet32
ResNet1 10
VGG19
Test Accuracy (96)
Amount of Data (%)
Figure 4: Test accuracy when limited training data is available. Accuracy computed using the neighst-neighhor method
5.4 WHAT IS SPECIAL ABOUT AUDIO LABELS?
Our experiments for robustness and data efficiency suggest that nigh-dimensional labels hold some interesting inherent property beyond just high-dimensionality that encourage learning of more robust
7

---

Published as a conference paper at ICLR 2021
and effective features. We hypothesize that high dith-dimensionall label representations with high entropy provide stronger learning signals which give rise to better feature representations.
To verify hypothesis, measure several standard statistics over various label representations, shown in Table 2 Specifically, we measure the normalized LI and L2 distance between pairs of labels for each representation. We further measure the entropy for each individual label.
CIFAR-
CIFAR- 100
Label Types
Entropy
1.1 Distance
1.2 Distance
Entropy
Distance
1.2 Distance
Category
0.47 + 0.00
2.00 + 0.00
1.41 + 0.00
0.08 + 0.00
2.00 = 0.00
1.41 + 0.00
Constant
0.00 + 0.00
26.07 15.72
26.07 + 15.72
0.00 + 0.00
21.76 + 15.16
21.76 + 15.16
Speech
11.35 + 0.35
23.80 + 5.16
12.95 + 2.70
11.37 + 0.32
21.45
5.53
13.15 + 2.19
Shuffle
11.35
0.35
35.29 + 2.18
18.87 + 1.53
11.37 0.32
34.40
2.59
17.81 1.24
Composite
12,00 + 0.00
24.13 + 3.36
19.41 + 1.47
12.00 + 0.00
25.75
4.72
20.60 = 2.96
BERT
11.17 0.00
5.71 + 0.94
2.06 + 0.24
11.17 0.00
7.89 + 2.84
2.63 +: 0.70
GloVe
5.64 + 0.00
7.35 + 1.69
1.30 + 0.31
5.64 + 0.00
5.62 + 0.90
0.99 + 0.16
Table 2: Different basic statistics of all types of label representations. Labels that encourage more robust and effective feature learning also have higher entropy than other label forms.
Interestingly, although the Manhattan and Euclidean distance between pairs of labels do not show any particularly useful patterns, the average entropy of the speech labels, the shuffled speech la- bels, and composition of Gaussian labels are all higher than that of the constant matrix and original categorical labels. The ranking of the entropy between these two groups exactly matches the per- formance in our robustness and data efficiency experiments shown in Figure 3 and Figure 4 This correlation suggests that high dimensional labels with high entropy may have positive impacts on robustness and data-efficient training.
We further validate that the benefits come from both high dimensionality and high entropy by train- ing model with cow-dimensional and high-entropy labels. We generated these labels by sam- pling from a uniform distribution, following the same procedure as the uniform-random matrix label described previously. We found that while models trained with this label perform comparably to ones trained with high-dimensional high-entropy labels in terms of adversarial robustness (see Appendix), high-dimensional and high-entropy label models outperform the low-dimensional high- entropy model in terms of data efficiency, as shown by the curve "Low-dim" in Figure + We find similar result for categorical models trained with label smoothing, which has been previously shown to improve adversarial robustness (Goibert & Dohmatob) 2020). In fact, high dimensionality is prerequisite for high entropy because the maximum entropy is limited by the dimensionality of the label.
Note that the model trained with label smoothing uses the standard cross-entropy loss, meanwhile the low-dimensional high-entropy model is trained with the Huber loss. As result, argue that the loss is not responsible for the improved performance of models trained with high-dimensional, high-entropy labels.
5.5 VISUALIZATIONS
Throughout the training process, we visualized the learned features immediately after the image encoder layer of ResNet-110 with t-SNE (van der Maaten & Hinton| 2008), both for audio and categorical models embedding of the learned features evolve as the training progresses. Compared with the feature embedding of the categorical model. the embedding of the speech models shows the formation of clusters at earlier stages of training. More separated clusters are also obtained towards convergence. We provide further visualizations and Grad-CAM interpretations in the Appendix.
CIFAR-10 test set. The results are shown in Figure 5 We observe that the
8

---

Published as a conference paper at ICLR 2021
Figure 5: T-SNE progression for speech (top row) and categorical (bottom row) models with ResNet- image encoder. From left to right, the plot shows 10%, 30%, 50%, 70%, and 100% progress training. The speech model develops distinctive clusters at an ear- lier stage and has better separated clusters overall.
Training Progress
CONCLUSION
We introduce a novel paradigm for the traditional image classification task by replacing categorical labels with high-dimensional, high-entropy matrices such speech spectrograms, We demonstrate comparable accuracy on the original task with our speech labels, however, models trained with our speech labels achieve superior performance under various adversarial attacks and able to leam in more data efficient manner with only small percentage of training data. We further study the inherent properties of high dimensional label representations that potentially introduce the advantages. Through a large scale systematic study on various label representations, we suggest that high-entropy, high-dimensional labels generally lead to more robust and data efficient training. Our work provides novel insights for the role of label representation in training deep neural networks.
ACKNOWLEDGMENTS
This research is supported by NSF NRI 1925157 and DARPA MTO grant L2M Program HR0011- 18-2-0020.
REFERENCES
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. Lecture Notes in Computer Science, 402, 2013. ISSN 661 -33493 doi: 101.107/977-3--62-40000 URL htt.httt:///et doi org 1000000100000000 978 622--042---9--- 25.
25.
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017.
Morgane Goibert and Elvis Dohmatob. Adversarial robustness via label-smoothing. 2020.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. arXiv e-prints, art. urXiv:1412.655 Dec 2014.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Hossein Hosseini, Yize Chen, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Blocking transferability of adversarial examples in black-box learning systems, 2017.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: (iXi::/552.03166 2015.

---

Published as a conference paper at ICLR 2021
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q- Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates, Inc.. 2012. URL http:::/papprs..iipsss.. miw.. www.:niisssccccppoom
4824- -magenet-clasiiicat http:://e.....aaa.o.......oooorr pdf
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world, 2016.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Sha- hab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv:1811.00 0e1218111.098,, 2018.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: /000:000799.3 2017.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser, 2017.
Jiajun Lu, Theerasit Issaranon, and David Forsyth. Safetynet: Detecting and rejecting adversarial examples robustly, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2017.
Dongyu Meng and Hao Chen. Magnet: two-pronged defense against adversarial examples, 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations, 2017.
Javier Naranjo-Alcazar, Sergi Perez-Castanos, Pedro Zuccarello, and Maximo Cobos. Dease 2019: Con depth analysis with different channel inputs for acoustic scene classification, 2019.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. arXiv e-prints, art. arXiv:1412.1897, Dec 2014.
Nicolas Papernot and Patrick McDaniel. Extending defensive distillation, 2017.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense adversarial perturbations against deep neural networks, 2015.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, 7. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. Proceedings of the 2017 ACM on Asía Conference on Computer and Communications Security, Apr 2017. doi: 10.1145/3052973. 11.....111.1111. 300 3053009
3053009. /tt.utttp://ax....orrr..
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, http: /ttt::/www..r www.w.wwwwwb..o.. http::/wwwwwwwwwwo////////tt/eee wwt.://wwwww.....tootee.ooeooeee
2014. URL WRRLhttp:
http://..r..g.aaooooooo//--e 1162
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. 2016.
10

---

Published as a conference paper at ICLR 2021
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In ICLR, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC),, 115()):11-2555) 252, 2015. doi: 10.1007/s1 1263-015-08166-
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models, 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentte neural networks. In Proceedings of the 33rd International Con- ference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1842-1850. JMLR.org, 2016a.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One- shot learning with meemory-augmente neural networks, 2016b.
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing
local stability of supervised models through robust optimization. Neurocomputing, 307:195 204. 2018. ISSN 0925-2312. doi: http:////............./////////.
2018.04.. URL http:
7/www.co www.sscieneediree... wwt.co/occccc.oo
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples, 2017.
Alexander Sorokin and David Forsyth. Utility data annotation with amazon mechanical turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1-8. IEEE, 2008.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv e-prints, art. arXiv:1312.6199, Dec 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of Ma- chine Learning Research, 9:2579 2008. TRLhhttp://wee jm wrg./pppers////
wandermaaten html
Joaquin Vanschoren. Meta-leamning survey, arXiv preprint arXiv:1810.03544 2018.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-leaming. Artificial intelligence review, 18(2):77-95, 2002.
Yaqing Wang, Quanming Yao, J Kwok, and LM Ni. Few-shot learning: A survey. arXiv preprint 1rX1v:1904.050466, 2019.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https: /www. a aclweb. org. httgg//thhlooy///
2020 emn 1 1p -e-deemo.

---

Published as a conference paper at ICLR 2021
Jing- Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, and Li-Rong Dai. Sequence-to- sequence acoustic modeling for voice conversion. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(3):631-644, Mar 2019. ISSN 2329-9304. doi: 10. 10.1109tuslp.2099 2892235. TRI.http::// http//xwddooo...oooooo//o///oood
http::///...oo.o//////.0. wrg. 1109/111119/TAAAA 2019 9.2892235
A APPENDIX
A.1 THRESHOLD VALIDATION
We deployed a large-scale study on Amazon Mechanical Turk to validate our choice of 3.5 as classification threshold for the speech model.
In particular, we asked workers to listen to the outputs of the speech model and choose from the set of classes (with a "None" class for unintelligible output) the one which best fits the output. We assigned 3 workers to evaluate each output from the VGG19 speech model on CIFAR-10 test set. We chose a restrictive selection criteria to ensure maximum quality of responses. Only workers with 99% approval rating and at least 10, 000 approvals were selected.
To measure agreement between humans and our speech model, for each sample in the test set we determine the decision made by the model using our pre-selected threshold (loss < 3.5 is correct, while loss 2 3.5 is incorrect). Then we compare these decisions to those of the human workers. When we count each of the three workers independently, 99.4% of the time. When we take a majority vote (2/3 humans agreeing) we find that humans agree with the model 99.8% of the time. We conclude that 3.5 is a reasonable threshold for evaluating the model.
find that humans agree with the model
A.2 SUBSET ROBUSTNESS
Additional robustness evaluation is computed using the subset of the test images that are initially correctly classified by the models without any adversarial attacks (Figure (). All test accuracy starts at 100% and decreases with stronger attacks. The strength of the attacks is described by the value of epsilon.
(Untargeted)
(Targetod)
(Uinargetad)
(Tangetd)
(Untargeted) (Thrgeted)
(Unnargeid)) (Targrio)
61D0X
Tot (esmacyy))))
Aucannc (N))
ZEPNNR
100
100
100
Epiiloe
Figure 6: Test accuracy under adversarial attacks on CIFAR- 10 (left four columns) and CIF (right four columns) for the initially correct subset of the test images. The accuracy evaluated by the threshold and the nearest neighbor is plotted in solid and dotted lines respectively. The general trend from the subset is similar to that from the full test set.
A.3 ADDITIONAL RESULTS ON ROBUSTNESS EVALUATION
We here include the full results on robustness evaluation on CIFAR-10 dataset in Figure 7
hhtps//wwww.murk.ccmm
12

---

Published as a conference paper at ICLR 2021
FGSM (Untargeted)
FGSM (Targeted)
Iterative (Untargeted)
Iterative (Targeted)
100
80
60
20
01
Test Acquaccy (%)
100
80
ReaNee32
60
20
100
80
RaNa110
20
01
0.2
0.3
0.1
0.2
0.3
0
0.1
0.2
0.3
0
0.1
0.2
0.3
Epsilon
Figure 7: Full results of the robustness evaluation on CIFAR-10
A.4 HYPERPARAMETERS
A.4.1 IMPLEMENTATION DETAILS
We train the categorical models for 200 epochs with starting learning rate of 0.1, and decay the learning rate by 0.1 at epoch 100 and 150. The high-dimensional models are trained for 600 epochs with the same initial learning rate, and drop the learning rate 0.1 epoch 300 and 450. All models are trained with a batch size of 128 using the SGD optimizer with 0.9 momentum and 0.0001 weight decay. One exception is when train categorical models with the VGG19 image encoder, use a larger weight decay, 0.0005. We implement our models using Py PyTorch,aszkk Paszke et al. (2019). All experiments are performed on single GeForce RTX 2080 GPU. The limited-date experiments are conducted using the same settings as the full data experiments.
A.4.2 ARCHITECTURE PARAMETERS
We provide the parameter counts for the all models in Table and Table 4 parameters come from the image encoders. High-dimensional models have slightly more parameters than categorical models due the high-dimensional label decoder (Table
The majority of the
CIFAR-10
Model
VGG19
ResNer32
ResNetl 10
Category
2 107
4.67 x 105
1.73 10
High-dimensione
2.01 x 107
5.80 x 10
1, 84 x 10
Table 3: Total number of parameters of the category and high-dimensional models for CIFAR-10
dataset
A.5 DATASET
To demonstrate the effectiveness of our proposed method, we evaluate our models on the CIFAR- 10 and CIFAR-100 datasets Krizhevsky (2009). For each dataset, we train different models on the same training set and evaluate the models on the same validation set using the same random seeds for fair comparisons. To preprocess the training images, randomly crop them with padding size and perform random horizontal flips. All CIFAR images are normalized with mean (0.4914, 4, 0.4822, 0.4465) and standard deviation (0.2023, 0. 1994, 0.2010) of the training
13

---

Published as a conference paper at ICLR 2021
CIFAR-100
Model
VGG19
ResNet32
ResNetl
Category
2.01 x 10?
4.73 x 10
1.74 x 10
High-dimensional
2.02 x 10?
5.80 105
1.84 x 10
Table 4: Total number of parameters of the category and high-dimensional models for CIFAR-100
dataset
Layer
Input
Output
Kernel
Stride
Padding
Dense
Le out
64
Toansrosee 2D
64
1
4
x 4
1x1
0
Conv Transpose
2D
64 x 4
32 x 8 x 8
4x4
2x2
1 X
1
ConvTranspose 2D 32 x 8 x
16 x 16 x 16
4 4
2x2
1
ConyTranspose 2D 16 x 16 16
8 x 32 x 32
4 4
2 x
1
ConvTranspose 2D
32 32
1 x x
x
1x1
Table 5: The architecture of the high-dimensional label decoder Iia- The input dimension of the first dense layer is the dimension of the output of the image encoder Ig. The output of the last ConyTranspose2d layer is the target label.
CIFAR-10 consists of 60, 000 images of size 32 x 32 uniformly distributed across 10 classes. The dataset comes with 50, 000 training images and 10, 000 test images. We use 45, 000/5, 000 train- ing/validation split.
CIFAR- also comprises 60, 000 images of size 32 32, has 100 classes each containing 600 images. The dataset is split into 50, 000 training images and 10, 000 test images. We randomly select 5, 000 images from the training images to form the validation set.
A.6 VISUALIZATIONS
In addition the progressive t-SNE plots we presented in the main context, we plot the embedding of all three types of image encoders of models trained on speech labels, categorical labels, and constant matrix labels Figure 8 We only show the results from the models with highest accuracy. Similarly, we observe that speech models have better separation of clusters. The feature embedding of the constant model worse than that of the speech model, further confirming that the speech representation is unique in addition to its dimensionality.
Grad-CAM We visualize activations from beginning, intermediate, and final layers in the image encoders for both speech and categorical models in Figure 9 We see that the input activations for speech model conform more tightly than those of the categorical model to central object of each image at all three stages. This may suggest that the speech model learns more discriminative features than the categorical model. These features are also visually more easily understandable to humans.
Uniform
Speech
Category
Uniform
Speech
Category
Uniform
Speech
Category
VGG19
(b) ResNet32
ResNetl
Figure 8: T-SNE the best uniform (left), speech (middle), and categorical (right) models trained with the same random seed. Speech models show best separated clusters across all three models.
14

---

Published as a conference paper at ICLR 2021
(a) VGG19
(b) ResNet32
(c) ResNet110
Figure 9: Grad-CAM visualization of learned features. Each triplet contains (from left to right) an unaltered image, categorical model visualization, and a speech model visualization. From top to bottom, the activations are taken from some beginning. intermediate, and final layer in the respective image encoders. Speech models learn more discriminative features than categorical models.
15


Reviews:
[{"review_id": "rgk3S5kvDmw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "########################################################################## \nSummary:\n\nThis paper presents a novel approach/perspective for improving data efficiency and robustness, other than existing research progress achieved by scientists, from model, optimizer and data perspective. \n\nIn particular, it proposes to introduce high dimensional and high entropy label representations for group truth, to improve image classification performance from two practical matters --- Robustness and data efficiency, while achieving comparable accuracy to text labels as the standard representation.  To valid its findings, the authors develop designed a set of comprehensive experiments for evaluation and comparison purposes, while making the best effort to not introducing variations from other angles, such as keeping the same data for training and testing and introducing adversary information consistently among all labeling representations. \n\n##########################################################################\nReasons for score: \n \nOverall, I vote for accepting. \nI like the idea of approaching image classification problem from a new angle that are not well explored yet. \nMy major concerns are: \n1. The clarity of the study on the underline true set of characteristics that contribute to the improvement, from speech label, shuffled-speech label, Gaussian-composition label, besides high dimension and high entropy. \n2. The logic behind the adversary image generation, target vs non-target. \n3. And the evidence/thinking process behind the pre-selected threshold of 3.5\n\nHopefully the authors can address my concern in the rebuttal period. \n \n##########################################################################\nPros: \n \n1. The paper provides a novel perspective for classification performance improvement, rather than from data, algorithms, or optimizers perspective. It shows the importance of label for supervised learning problems, can also come from the ways that we represent them, just only label quality.  For me, this approach is new and potentially expend to other applications, besides image classification \n \n2. The experiment design is also quite comprehensive as it covers all potential perspectives and variations. \n \n3. This inspiration of the idea to me is also quite natural and understandable, as for most of us, when recognizing an image, we express it not just in writing and can also in speech format. \n \n\n##########################################################################\nCons: \n \nAlthough the proposed representations have shown better performance in image classification problem, with evidence to support its out-performed robustness to adversarial attack and data efficiency -- achieving comparable accuracy with less data in training, I would still suggest the authors to conduct the following studies to enhance the quality of the paper: \n\n1. It could be valuable to future investigate the inherent property that contributes to the improvement, besides high dimensionality and high entropy. \n \n2. What’s the performance with high dimensional and high entropy label representations, comparing to text label, for other kind of the classification problems, such as NLP problems. \n\n3. For speech label, in model evaluation, what’s the performance for the model, if we choose a speech to text process to obtain its ground truth, beside the two approaches mentioned in the paper -- “nearest neighbor” and a validated loss threshold. \n \n \n\n##########################################################################\nQuestions during rebuttal period: \n \nPlease address and clarify the cons above \n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gk2nFZW30Jj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work suggests an alternative training method where the ground truth label is given as speech signal. The authors hypothesize the labels with high-dimensionality & high entropy will help the network learn better feature representation. The hypothesis is empirically examined by the authors using commonly used CIFAR-10 and CIFAR-100 datasets. The experiment results show that the proposed approach is effective in terms of adversarial robustness and data efficiency.\n\nStrength: \nThe idea of giving speech signal as a proxy for categorical label is interesting.\n\nWeakness:  Unclear source of performance for data efficiency and robustness on adversarial attack\n\nThe proposed claim in this work seems quite bold and is mostly built on empirical observation.  Also, I think the proposed approach is somewhat similar to the popular multi-modal representation learning schemes, which is quite common these days. The main difference is that the authors train the network by targeting speech signals itself. I recommend the authors to compare the proposed method with previous multi-modal learning training schemes such as [1, 2] and many more.\n\nRatings:\n\nAlthough the paper shows some interesting research direction, the way the authors show the effectiveness of the proposed method is mostly built on empirical results, which is not enough to claim such a bold argument. For this reason, I recommend rejection. \n\nReferences\n\n[1] Look, Listen and Learn (https://arxiv.org/abs/1705.08168)\n\n[2] Objects that Sound (https://arxiv.org/abs/1712.06651)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S5u8YdCk1Hd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the effect of data labels on the quality of trained models. More specifically, the authors use audio labels rather than traditional categorical probabilities for model training and get surprising and interesting results.  The results show that high dimensional and high entropy label representations are more useful, which is observed in the experiments related to robustness and a limited amount of training data. Such a result is very interesting and suggests that the label representation can be further explored and potentially plan an important role.\n\nThis paper starts an interesting direction and conducts nice experiments. The paper is easy to follow and well written. The hypothesis about the relationship between high entropy and training effectiveness is also a good observation.\n\nI am also interested in that if the audio signal is replaced by pre-trained embeddings, like glove or BERT, as label representation, how the effectiveness of labels is compared with the audio signals?", "extra_scores": {}, "extra_text": {}}, {"review_id": "7bh7nxvdEm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed to use other high-dimensional representation instead of direct one-hot class labels in image classification problem. They used spectrogram of the pronunciations (TTS-based generated speech) of class labels as a high-dimensional representation. Then, they performed regression using the spectrogram as a label. The evaluation is conducted in nearest-neighbor way, measuring the distance between the image embedding feature and groundtruth high-dimensional representation (spectrogram) and decide the predicted class label. Then, they compared it with traditional classification model with cross-entropy loss. The results showed that the proposed approach are more robust in adversarial attack and feature effectiveness. \n\nHowever, overall, I think one important related direction is somewhat missing which is zero-shot learning. In zero-shot learning, the primary goal is to make a prediction on unseen class labels. On the other hand, zero-shot learning has characteristics in that it uses an additional information to learn the embedding space. I think the proposed approach in this paper is somewhat related with it.\nSince TTS system is utilized to generate pronunciation of label, such other information is naturally used in high-dimensional labels (the information used in the TTS system). Also, the author argues that audio labels is special, but in the paper, other type of high-dimensional representation of labels that uses external information are not explored, such as word2vec. \n\nI think two factors are somewhat mixed. One is about the usefulness of high-dimensional representation (without external information, constant comparison is working at here) and the other is use of the external information. So, to verify the former factor, I think high-dimensional version of class label that does not use the external information should be added, such as after running topic modeling algorithm within the image classification data, and use them as a high-dimensional representation of a class label. Second, to verify the specialness of the audio label, other external data can also be compared, such as general word2vec, ...\n\nIn page 4, how the various length of audio produce the spectrogram of the same size? (the length difference is really small?)\n\nThe traditional classification is conducted in classification, while the high-dimensional label experiment is conducted in regression. I think the author can explore classification type loss for the latter experiments also (measuring the distance between the two matrices and put softmax over these similarity scores).\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use high dimensional representation for labels to strengthen the adversarial robustness of deep neural networks. Experimental results demonstrate that the proposed method improve adversarial robustness. All reviewer agree that the authors propose an interesting idea and this direction deserves further exploration. On the other hand, the reviewers also raise a serious question: There is a lack of explanation of why high dimensional representation of labels improve adversarial robustness. Therefore, it is not clear if the proposed method can defend refined attacks tailored to such dimensional label representation. The authors are highly encouraged to conduct deeper analysis, especially on the robustness against finer attacks.
Decision: Accept (Poster)