Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposed to use other high-dimensional representation instead of direct one-hot class labels in image classification problem. They used spectrogram of the pronunciations (TTS-based generated speech) of class labels as a high-dimensional representation. Then, they performed regression using the spectrogram as a label. The evaluation is conducted in nearest-neighbor way, measuring the distance between the image embedding feature and groundtruth high-dimensional representation (spectrogram) and decide the predicted class label. Then, they compared it with traditional classification model with cross-entropy loss. The results showed that the proposed approach are more robust in adversarial attack and feature effectiveness. 

However, overall, I think one important related direction is somewhat missing which is zero-shot learning. In zero-shot learning, the primary goal is to make a prediction on unseen class labels. On the other hand, zero-shot learning has characteristics in that it uses an additional information to learn the embedding space. I think the proposed approach in this paper is somewhat related with it.
Since TTS system is utilized to generate pronunciation of label, such other information is naturally used in high-dimensional labels (the information used in the TTS system). Also, the author argues that audio labels is special, but in the paper, other type of high-dimensional representation of labels that uses external information are not explored, such as word2vec. 

I think two factors are somewhat mixed. One is about the usefulness of high-dimensional representation (without external information, constant comparison is working at here) and the other is use of the external information. So, to verify the former factor, I think high-dimensional version of class label that does not use the external information should be added, such as after running topic modeling algorithm within the image classification data, and use them as a high-dimensional representation of a class label. Second, to verify the specialness of the audio label, other external data can also be compared, such as general word2vec, ...

In page 4, how the various length of audio produce the spectrogram of the same size? (the length difference is really small?)

The traditional classification is conducted in classification, while the high-dimensional label experiment is conducted in regression. I think the author can explore classification type loss for the latter experiments also (measuring the distance between the two matrices and put softmax over these similarity scores).


review_title: TTS-generated audio can be used to train classification model instead of one-hot class label.

final_score: rating: 7: Good paper, accept
review_content: ########################################################################## 
Summary:

This paper presents a novel approach/perspective for improving data efficiency and robustness, other than existing research progress achieved by scientists, from model, optimizer and data perspective. 

In particular, it proposes to introduce high dimensional and high entropy label representations for group truth, to improve image classification performance from two practical matters --- Robustness and data efficiency, while achieving comparable accuracy to text labels as the standard representation.  To valid its findings, the authors develop designed a set of comprehensive experiments for evaluation and comparison purposes, while making the best effort to not introducing variations from other angles, such as keeping the same data for training and testing and introducing adversary information consistently among all labeling representations. 

##########################################################################
Reasons for score: 
 
Overall, I vote for accepting. 
I like the idea of approaching image classification problem from a new angle that are not well explored yet. 
My major concerns are: 
1. The clarity of the study on the underline true set of characteristics that contribute to the improvement, from speech label, shuffled-speech label, Gaussian-composition label, besides high dimension and high entropy. 
2. The logic behind the adversary image generation, target vs non-target. 
3. And the evidence/thinking process behind the pre-selected threshold of 3.5

Hopefully the authors can address my concern in the rebuttal period. 
 
##########################################################################
Pros: 
 
1. The paper provides a novel perspective for classification performance improvement, rather than from data, algorithms, or optimizers perspective. It shows the importance of label for supervised learning problems, can also come from the ways that we represent them, just only label quality.  For me, this approach is new and potentially expend to other applications, besides image classification 
 
2. The experiment design is also quite comprehensive as it covers all potential perspectives and variations. 
 
3. This inspiration of the idea to me is also quite natural and understandable, as for most of us, when recognizing an image, we express it not just in writing and can also in speech format. 
 

##########################################################################
Cons: 
 
Although the proposed representations have shown better performance in image classification problem, with evidence to support its out-performed robustness to adversarial attack and data efficiency -- achieving comparable accuracy with less data in training, I would still suggest the authors to conduct the following studies to enhance the quality of the paper: 

1. It could be valuable to future investigate the inherent property that contributes to the improvement, besides high dimensionality and high entropy. 
 
2. What’s the performance with high dimensional and high entropy label representations, comparing to text label, for other kind of the classification problems, such as NLP problems. 

3. For speech label, in model evaluation, what’s the performance for the model, if we choose a speech to text process to obtain its ground truth, beside the two approaches mentioned in the paper -- “nearest neighbor” and a validated loss threshold. 
 
 

##########################################################################
Questions during rebuttal period: 
 
Please address and clarify the cons above 
 

review_title: Recommendation to Accept

final_score: rating: 7: Good paper, accept
review_content: The authors study the effect of data labels on the quality of trained models. More specifically, the authors use audio labels rather than traditional categorical probabilities for model training and get surprising and interesting results.  The results show that high dimensional and high entropy label representations are more useful, which is observed in the experiments related to robustness and a limited amount of training data. Such a result is very interesting and suggests that the label representation can be further explored and potentially plan an important role.

This paper starts an interesting direction and conducts nice experiments. The paper is easy to follow and well written. The hypothesis about the relationship between high entropy and training effectiveness is also a good observation.

I am also interested in that if the audio signal is replaced by pre-trained embeddings, like glove or BERT, as label representation, how the effectiveness of labels is compared with the audio signals?
review_title: Interesting observation, nice experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work suggests an alternative training method where the ground truth label is given as speech signal. The authors hypothesize the labels with high-dimensionality & high entropy will help the network learn better feature representation. The hypothesis is empirically examined by the authors using commonly used CIFAR-10 and CIFAR-100 datasets. The experiment results show that the proposed approach is effective in terms of adversarial robustness and data efficiency.

Strength: 
The idea of giving speech signal as a proxy for categorical label is interesting.

Weakness:  Unclear source of performance for data efficiency and robustness on adversarial attack

The proposed claim in this work seems quite bold and is mostly built on empirical observation.  Also, I think the proposed approach is somewhat similar to the popular multi-modal representation learning schemes, which is quite common these days. The main difference is that the authors train the network by targeting speech signals itself. I recommend the authors to compare the proposed method with previous multi-modal learning training schemes such as [1, 2] and many more.

Ratings:

Although the paper shows some interesting research direction, the way the authors show the effectiveness of the proposed method is mostly built on empirical results, which is not enough to claim such a bold argument. For this reason, I recommend rejection. 

References

[1] Look, Listen and Learn (https://arxiv.org/abs/1705.08168)

[2] Objects that Sound (https://arxiv.org/abs/1712.06651)

review_title: Interesting proposed method but not enough theoretical support.

Meta-review: This paper proposes to use high dimensional representation for labels to strengthen the adversarial robustness of deep neural networks. Experimental results demonstrate that the proposed method improve adversarial robustness. All reviewer agree that the authors propose an interesting idea and this direction deserves further exploration. On the other hand, the reviewers also raise a serious question: There is a lack of explanation of why high dimensional representation of labels improve adversarial robustness. Therefore, it is not clear if the proposed method can defend refined attacks tailored to such dimensional label representation. The authors are highly encouraged to conduct deeper analysis, especially on the robustness against finer attacks.
Decision: Accept (Poster)