Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: UPDATE: Based on the extensive improvements by the authors, I have updated my rating. However, I still have doubts about the potential of this approach to reach practically useful levels of accuracy.

This paper introduces a simple method to weight pretrained lexical features for use in meta learning of few-shot text classification. The method boils down to weighting word inut features, in the form of pretrained word-embeddings, by attention computed from inverse document frequency and class local mutual information. The idea is that this measure of feature informativeness transfers between tasks, whereas lexical features themselves are highly task-specific. The approach is well motivated and is empirically shown to outperform existing approaches to few-shot text classification with a significant margin.

While the improvement over existing approaches is quite substantial, I believe the paper should not be accepted to ICLR for the following reasons. First, the contribution is quite limited and not particularly novel. While two weight functions are proposed, the majority of improvement comes simply from normalizing IDF with attention. Based on existing work on delexicalized features for NLP tasks such as parsing, this is quite a straightforward extension. Given the limited contribution, a short paper seems a better fit. Second, the approach simply trades variance for bias. This brings us to the question of how likely the approach is to be a building block in bringing us towards a pratically useful few-shot classification method. Given the weak representational power of the model, I believe this is unlikely. I see a situation similar to syntactic parsing for low-resource languages, where a collection of simple techniques similar in spirit to the current approach, like delexicalization, brought results far above the naive baselines, but never approached practically useful results. I think this is a crucial point to address in meta-learning research in general to make sure we’re not just solving a toy problem with tailored heuristics.

Additional notes:

What is the motivation for using a BiLSTM to combine inverse document frequency and inverse class entropy? Is the sequence information at all useful, or would a simple projection and nonlinearity give the same result?

The theoretical analysis is completely self-evident from the definition of the feature space. Replaing a feature with an equivalent feature of course gives the same result and I don’t see the need to “mathematize” this.

The effect of approximating logistic regression with linear regression + calibration is not analyzed and it is not clear what the effect of this approximation is in the text classification scenario. I would suggest to compare to differentiating through a direct optimization of the logistic formulation, for example with Newton’s method, or plain SGD, as in Bertinetto et al. (2019).

Table 1. Why not run the attention-based feature aggregator together with all algorithms. The main contribution is at the input representation level, and this should be applicable across algorithms. In fact, if we remove the BiLSTM which seems to have a very small effect the representation function does not contain learnable parameters in itself.

Please provide the average across datasets in Table 1.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper focuses on applying meta learning approaches to text classification.

The primary contribution is an attention mechanism based on word statistics --- most importantly word frequency. This novel attention mechanism is motivated by the observation that the base units in text (lexemes) are more likely to have task specific interpretations than lower level patterns in vision. And, while a lexeme based attention mechanism trained on one task may not transfer well to other tasks, a mechanism based on coarser word statistics is less likely to focus in on task specific patterns.

A secondary contribution is the use of ridge regression [1] to perform meta-learning for text classification.

The paper presents experiments on a number of text classification tasks from the NLP literature. Aside from the new attention mechanism and the use of ridge regression, the proposed approach makes use of FastText word embeddings or BERT sentence representations, depending on the task. The paper demonstrates significant improvements over baselines that use other methods of aggregating word representations. All of baselines were implemented for this paper.

The idea of using coarse statistical signatures to calculate attention is an interesting one. However, I have concerns about both the clarity of this paper and the lack of clear comparison to previous work.

== Clarity ==

Many of the details of the model and learning approach are vaguely discussed, or relegated to Figures 4 & 5. I think the paper would benefit from a more formal definition of the entire learning procedure.

== Comparison to previous work ==

This paper seems to be following the standard FewRel experimental setup. Also, the RCV1 experiments seem to follow the [2] which was cited by the in the paper under review. However, it is not clear if the setups are the same or if the numbers are comparable.

I am not sure about the existence of comparable results for the other tasks, but for FewRel at least the baselines presented here significantly underperform other papers' reports of equivalent models.

  - The paper from [3] that introduced FewRel reported 69.2 / 84.8 for CNN based prototypical networks --- far above the 49.8 / 65.2 reported here.

  - [4] found that a BERT model with no FewRel specific training at all achieves 72.9% on the 5way/1shot task. Which is above all of the BERT based models reported in Table 2.

I may be missing something, but if these numbers are actually not comparable then this paper should contain an explanation of how the experimental setup differs. And if the setup is actually the same as previous work, I expect to see a comparison of results.

[1] https://openreview.net/pdf?id=HyxnZh0ct7
[2] https://openreview.net/forum?id=SyxMWh09KX
[3] https://www.aclweb.org/anthology/D18-1514/
[4] https://arxiv.org/abs/1906.03158
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper studies the effects of using function of ngram statistics as feature to generate attention score per word. The attention score is then used as weights to aggregate document embedding by doing a weighted average on word embedding. The output is finally fed into a ridge regressor to do the final predictions on target labels. 

Main comments:
This paper has a clear motivation and decent experimental results (though some concern on baseline models, see below). The introduction of using distributional signature to derive attention scores seems interesting and a novel contribution. However I was not able to fully understand the intuition behind the benefit of doing attention mechanism on top of ngram statistics (see my question below as well). 
Also the reference/baseline models used in the experiment might not be strong enough. If you could compare your model with some latest algorithms proposed in the few-shot-learning communities, that would be more convincing as well. 
To list a few:
* P-MAML: [Zhang et al., 2019]
* Induction-Network-Routing: [Geng et al., 2019]
* ROBUSTTC-FSL [Yu et al., 2018]

I am leaning to give a "weak reject" based on my current knowledge and understanding of the paper. But I will be willing to revisit the decision after we get feedback from the author(s). 

In particular, I would be glad if the author could clarify the questions below.

* From table 1, it seems Method IDF+RR is a competitive model. IIUC, the statistics of s(.) is highly correlated with IDF which also indicates general word importance in corpus. My questions are that, 
1) regarding ablation test "OUR w/o biLSTM", how is $h$ calculated in this case (without biLSTM)?
2) since each word is represented based on two statistical number (map function by t(.) and s(.)), can you give any intuitive explanation that why getting attention score from that makes sense?
3) do you have any experiments using the distributional signature as a common feature in standard text classification problems? In other words, is this method only (significantly) beneficial to few-short-learning? If it is also useful in general text classification task, it would be a good "plus" here.

* From table 2, can you explain why CNN+RR benefits a lot from the BERT embedding? Actually it gets more percentage of improvement than the model "OUR".

* For all the usages of pre-trained embedding (fasttext or BERT), are you further finetuning the embedding parameters during your training? Or you freeze the embedding parameters?

[Zhang et al., 2019] Ningyu Zhang et al., Improving Few-shot Text Classification via Pretrained Language Representations. arXiv preprint arXiv: 1908.08788
[Geng et al., 2019] Ruiying Geng, Binhua Li, Yongbin Li, Yuxiao Ye, Ping Jian, and Jian Sun. 2019. Few-shot text classification with induction network. arXiv preprint arXiv:1902.10482.
 [Yu et al., 2018] Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang,
and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. arXiv preprint arXiv:1805.07513
review_title: Official Blind Review #1

Meta-review: This paper proposes a meta-learning approach for few-shot text classification. The main idea is to use an attention mechanism over the distributional signatures of the inputs to weight word importance. Experiments on text classification datasets show that the proposed method improves over baselines in 1-shot and 5-shot settings.

The paper addresses an important problem of learning from a few labeled examples. The proposed approach makes sense and the results clearly show the strength of the proposed approach.

R1 had some questions regarding the proposed method and experimental details. I believe this have been addressed by the authors in their rebuttal.

R2 suggested that the authors clarified their experimental setup with respect to prior work and improved the clarity of their paper. The authors have made some adjustments based on this feedback, including adding new sections in the appendix.

R3 had concerns regarding the contribution of the approach and whether it trades variance for bias. The authors have addressed most of these concerns and R3 has updated their review accordingly.

I think all the reviewers gave valuable feedbacks that have been incorporated by the authors to improve their paper. While the overall scores remain low, I believe that they would have been increased had R1 and R2 reassessed the revised submission. I recommend to accept this paper.

Decision: Accept (Poster)