Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Quality


I like the intended focus of this paper which is to perform self-supervised training of small data for downstream tasks with applications for zero and few-shot learning. However, there are many limitations.


(1) The only task considered is multi-class classification. The technique proposed in this paper revolves around learning the label embeddings that would match the input embeddings, which is quite limited to multi-classification and might not carry much value to other tasks. For example, for tasks such as entailment prediction where the labels are `true' or `false', for the model to perform well, most of it comes from the ability to process the input and does not have much to do with the model's understanding of these two labels, which is unlike in multi-class classification where label understanding is crucial. Therefore, the proposed approach seems to be quite limited to the areas of multi-class classification. 
(2) I would love to see more experiments on other tasks besides multi-class classification but wonders how applicable the framework will be. Explanations would be appreciated here.
(3) Learning label embeddings seems like a good idea and this seems like an ok way to do it. However, label embeddings are somewhat learned already in some pre-trained models such as generative sequence-to-sequence models (T5 or related models). The paper at least need to have these models as baselines for comparison. 


Clarity

The writing is at an acceptable level. However I find it a bit dense in terms of explanations and might need more illustrative figures. 

- I don't really get see the CNN architecture is used here. I have some concerns that the method might not be applicable for other architectures and would like to get some more clarity on this. 
- It is not entirely clear why the prevalent evaluation favors large self-supervised pre-training. *** 
- Too many notations (s, -, +s , SL) that makes it hard to follow in the results section. 




Originality

- The paper seems heavily inspired by SIMCLR so in my opinion the originality is low to moderate. 


Significance


The premise of this work is to avoid having to use external datasets for pre-training. However, the prevalent self-supervised training in many work (BERT, RoBerTa, to name a few) do not require labels, so I don't see why we really need to avoid that. The paper can be positioned a bit better if the paper uses these models, and performs additional learning on top with the 'dataset-internal' contrastive learning approach in order to improve performance on long-tail zero and few-shot learning.

However the paper does show that it is possible to do well on long-tail zero- and few-shot learning with only small data pretraining, so that is a nice conclusion derived from the experiments. However, I am not entirely sure about the real-world applicability. 


High-level pros and cons


Pros
- Long-tail zero and few-shot learning is an interesting direction which the paper explores

Cons
- This paper seems like a probing paper that looks into how well we can do without external pre-training, but might not be so applicable for real-world applications due to reasons above (see discussion in Quality and Significance)

review_title: Good motivation but experiments are limited

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a contrastive autoencoder approach that only requires small data to perform a multi-label classification on the long-tail problem. They introduce a matching network to compare text and label embeddings and calculate the probabilities of the label given the input. The proposed idea is very straightforward by combining a matching network with contrastive learning to give broader signals. The goal of this work is to enable zero-shot and few-shot learning with very few resources as a more sustainable approach to machine learning applications. 

In general, the idea is interesting, since it leverages the in-task data for distinguishing positive and negative samples. However, the work itself is not entirely new, there is already similar work on this [1]. I could not find enough related work on matching networks [1] and it seems the authors need to give more credit to other seminal work that has been used for many applications for few-shot learning [2]. I would ask the authors to add more relevant work to the paper

The structure of the paper can be further improved. The naming is not intuitive, such as $(SLpf, S(+S)Lpf)$. The results in Table 1 isn't easy to read (especially the learning setup). 

Strengths:
- A straightforward method for self-supervised contrastive label-embedding prediction using in-task data.
- This paper focuses on a challenging, noisy long-tail, low-resource multi-label text prediction task

Weaknesses:
- The key concern about the paper is the lack of comparison to the baselines. Since the proposed method is not the first on this topic, the authors should compare their method with other well known baselines for few-shot learning, such as matching networks and Siamese Networks
- The paper does not analyze the effectiveness of the approach on the long-tail scenario.
- No ablation study on the contrastive learning and label embeddings

***Post-rebuttal***

> I want to thank the author for addressing my concerns. However, I still have issues with the evaluation and the clarity of the paper. I think the paper requires another round of revision before it is ready for publication.

References
1. Transferable Contrastive Network for Generalized Zero-Shot Learning 2019 ICCV https://openaccess.thecvf.com/content_ICCV_2019/papers/Jiang_Transferable_Contrastive_Network_for_Generalized_Zero-Shot_Learning_ICCV_2019_paper.pdf
2. Siamese Neural Networks for One-shot Image Recognition ICML Deep Learning Workshop 2015 https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf
3. Matching Networks for One Shot Learning NeurIPS 2016 https://arxiv.org/pdf/1606.04080.pdf
review_title: Please compare your method with existing baselines

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
Summary:
This paper studied the effect of pretraining from a miniaturization perspective. The proposed, dataset-internal contrastive self-supervised learning benefits in zero- or/and few-shot learning settings. 

##########################################################################
Reasons for score: 
 
Overall, my score is marginally below the acceptance threshold. 

1. The paper provides a very in-depth survey of recent work on pretraining research and a clear scoping of the problem to be addressed in this work against the other work. I also like the framing of dataset-internal and dataset-external to distinguish between self-supervision and task-supervision, respectively. I also like the list of limitations of the prior work in the Related Work section. 
2. The description of the proposed method in Section 3 was very detailed and easy to follow. 
3. Overall messages and findings from the experiment were clear and convincing to me. 
 
Cons: 
 
1. Although the clarity of methodology description and clear motivation compared to prior work, my main concern of this work is the lack of generalizability of the proposed method to other low-resource, long-tail problems. The predicting pseudo, noisy labels itself is not a novel idea. The main novelty of the method comes from predicting new labels from word token space directly, which seems to be very limited to the task itself; free-formed tag labels for text. As described in the conclusion section, the method seems to be only applicable when input text and label tags are in the same place. 
 
2. Another concern is the unclear description of how two binary labels are optimized in a contrastive way. Usually, there should be an additional loss to maximize the difference between two similar-looking examples of different labels in contrastive learning like SIMCLR. I only find that (5) in Figure 1 is a simple aggregation of binary classifiers, rather than contrastive optimization. When the batch set of positive and negative (pseudo) labels are prepared in L_i, does the ordering of the labels matter? Also, again do you pair positive and negative labels for pair-wise optimization? 
 
3. I like to suggest changing the framing of this work from the general findings of pretraining schemes to task-specific findings. That is, Section 4 should be introduced earlier than Section 3, and describe how difficult the task is and how existing pretraining methods fail to achieve good performance. Until section 3, I had a hard time understanding whether the proposed method in Figure 1 is generalizable to any downstream tasks or specific to certain tasks. The partial examples in Figure 1, like “measuring an interaction”, “interaction”, “p-value” are mentioned without even describing what they are. Otherwise, it would be really nice to add more applications in a long-tail, low-data regime, proving that the proposed framework is general enough. 
 
 
##########################################################################
Questions during the rebuttal period: 
 
Please address and clarify the cons above.
Here are some questions:
1. In section 3, what do you mean by “Sampling pseudo-labels provides a straight-forward contrastive, partial autoencoding mechanism…”?
2. In Section 6.2, authors controlled the hyper-parameter for a better-controlled experiment. However, I wonder this is a fair game. Each training technique may have different levels of optimality. So, I guess given a fixed set of parameter ranges, providing the optimal setting might be a more fair setting. Or, at least providing them in the Appendix might be more convincing. 
 
#########################################################################
Some typos and suggestions for presentation: 
(1) Too many styles in the text (e.g., bold, italic, the numbers) make it difficult to read sometimes. I know they are used to help follow the main points but they are quite distracting.  
(2) Table Tab. 2 -> Tab. 2

review_title: Focused, well-written work but limited to specific types of tasks

Meta-review: The paper presents a self-supervised model based on a contrastive autoencoder that can make use of a small training set for upstream multi-label/class tasks.
Reviewers have several concerns, including the lack of comparisons and justification for the setting, as well as the potentially narrow setting. Overall, I found the paper to be borderline, the cons slightly greater than the pros, so I recommend to reject it.
Decision: Reject