Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a novel method for few-shot text classification tasks. The method is built on top of the CNAP  The key contribution is to effectively encode the gradients of networks as the task representation so that the meta-learning stage can use such task representations to improve the few-shot generalization ability. The two key technical contributions are a task embedding network (an RNN-based layer-by-layer gradient encoder via Fisher Information Matrix) and an adaptation network for auto-regressively adapting parameters from the bottom to the top layer.  The implementation is based on BERT-like LMs with Adapter bottleneck modules, which are parameter efficient to compute. 

The authors compared the proposed method with a few meta-learning methods on a diverse set of tasks, and show that their method outperforms other methods for most of the time. The empirical analysis and ablations also provide insights for understanding the proposed method. 



main_review: 
Overall, the work is a nice contribution towards using the gradient information to encode a task instead of only using the input/output text. The gradient information implies more about the network behavior which is closer to the features of a task. The paper is mostly well written with nice figures and tables for delivering important information. I do not have major concerns about the paper.  There are a few minor notes and suggestions below:

- I think in the problem definition, Line 98, it would be better to explicitly define the learning stages and a shared text classification model. The meta-learning stage has the access to the source tasks, and the few-shot generalization stage does not. And there is a shared model after the meta-learning stage. 

- In the preamble of Section 4, it would be better to illustrate more the motivation of the design choice and define the "adaptation parameters" clearly. It is vague in the current presentation and makes people wonder about the underlying motivations of the design. 

- It would be better to visualize the task representation in some way and show the task-to-task distances so that we can inspect if the closer tasks can locate near each other. 

- Line 202, BERT-base only allows 128 tokens? I don't think so. It should be 512 if I understand correctly. Please double-check it and correct me if I was wrong about this. 

- A few typos: Line 19: "a intermediate"; Line 201 BERT-BERT. 
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed Grad2Task, a model for few-shot text classification tasks. Grad2Task first computes gradients when fine-tuning the base model on a support set; then the task embedding network decodes task representations using these gradients; finally the task representations go through MLP layers and output scaling and shifting parameters, which are plugged into the base model for task-specific adaptation. 

The paper provides comprehensive evaluation on 10 tasks. Grad2Task outperforms strong baselines (Leopard, ProtoNet-BERT) in most settings. Analysis with a toy dataset shows the task representations are informative. Ablation study justified several design choices in Grad2Task.
main_review: Strength:
- Novelty. The work is the first to explore gradient-based task representations for meta-learning. In addition to combining recent efforts (e.g., Task2Vec, meta-learning), the authors also reduces parameter size using lightweight design (e.g., adapters, shifting and scaling parameters).
- Detailed ablation study that explains the design choices in detail. The failure cases also brings interesting observations. Future work may stem from these observations.
- Strong performance when compared to strong baselines.

Weakness:
- Notation are sometimes confusing. (1) In Eq.3 the function d is overloaded. The function outputs the task embedding in the first part. It outputs RNN hidden representation in the second part. (2) In line 7-8 in Algorithm 1, e_j is a new variable that seems to be newly-introduced. Please also provide hints for the reader to link e_j with p(y|x_j), e.g., refer the reader to Eq. 1.

Questions:
- Is there any specific reason that RNN is selected as the task embedding network? 
- How long is the training process? How large is the final model? Some comparison in model size and training efficiency may be included.

Minor Suggestions:
- Eq. 7. Is the first stage done on the support set or the query set or both?
- Line 274: “w/ XY”, Table 2 “w/ X&Y”. Please keep consistent.
- I believe the right side of Fig 1 is actually zooming in the part of [e_l^{task}] —> [Bottleneck Adapter]. Please consider highlighting this information to make the figure more accessible. 

limitations_and_societal_impact: The authors provide detailed discussion on Grad2Task's limitations in Sec 7. I listed some of my concerns and suggestions in the main review.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper considers the problem of few-shot text classification. It builds on a bunch of prior works on meta-learning [1,2,3], Film adapters [4], task embeddings [5] and tries to combine them for the task of few-shot text classification. The key idea seems to be to use adapters in between transformer layers, compute task embeddings (using fisher Information matrix) for a task and use it to update Film parameters that are used to update the model. The approach is compared with a prior work [1] and prototypical networks on a subset of few-shot classification tasks selected from [1].

main_review: I like the paper’s proposal to use task-embeddings to further modulate the encoder for few-shot learning. This is not a new concept and has been widely explored in meta-learning literature. The main contribution of the paper seems to be in its specific application to text classification models and the use of the task embeddings to generate the Film parameters. 

Overall, I found the paper very hard to follow, there are numerous typos and the exposition made it really difficult to extract the key contribution, the motivations for the specific choices, and the key take-aways. The paper seems a lot like an amalgamation of many related concepts, which is not a bad thing but in the current version of the paper there is no clear motivation for the various choices. To provide just one instance, if the main motivation is to use task embeddings then there are many straight-forward ways to incorporate them in existing models, such as in the input to the encoder or the classifier, and carefully considering/evaluating such choices would have been useful in motivating the approach. Another choice made by the authors is to use only the values in FIM corresponding to the adapters. Why is it sufficient to only use these, is there any analysis on the quality of the task embedding when restricting to only the adapters?
Similarly, the motivation for using task embeddings to generate Film adapters is not clear. For instance, one can use Film adapters with regular fine-tuning on the few-shot set to learn “task-specific” Film adapters. Note that this baseline is not present in the paper. So how is including the task embedding (which also uses gradient information) helpful over fine-tuning? Motivation as well as empirical results in this direction are lacking. 

Moreover, there are several concerns regarding experiments. 
- First, as mentioned above, there are relevant baselines that need to be provided to understand the value of the proposed approach. These also include: ProtoNet with fine-tuning, ProtoNet with adapters fine-tuning, ProtoNet-BN with only Film fine-tuning. Currently, the main baselines for the paper are ProtoNet and ProtoNet-BN, none of which use gradient information like the proposed method. These baselines will help demonstrate the utility of task embeddings over fine-tuning.
- The authors use datasets from [1]. However, I see there are 17 datasets used in that paper and the authors picked 7 of them. I am not convinced that seq-length is an appropriate reason to drop 10 datasets, for instance, the authors can run the model with a larger sequence length during evaluation. Moreover, there are decisions like only using a subset of classes in one of the datasets (Huffpost), and why this is done is also not clear. As such, due to the use of non-standard datasets, I am not sure how to interpret the results.
- Ablations are carried on all the downstream tasks used for final evaluation, which makes me think the results are overfit to these tasks. I didn’t find these particularly insightful either. For example, ablations on these choices would have been more helpful: using/not-using adapters or Film adapters, hidden-size of adapters, fine-tuning adapters vs using task embeddings, alternatives to incorporate task embeddings.
- Some hyper-parameter choices are missing. For example in section 4.2, what are the hyper-parameters of the RNN d()? Similarly, what is the architecture and hyper-parameters of the MLPs a(.) in adaptation network?
- There is no analysis on what the task embeddings are capturing and why they are useful here. This would have helped motivate the paper quite a bit. 

These concerns together make me hesitant about the value of the task-embedding approach presented here. The authors should consider a thorough re-writing of the paper to provide motivation and to explain their approach clearly, and consider the suggested baselines and improvements to experiments explained above.

[1] Learning to few-shot learn across diverse natural language classification tasks.  
[2] Meta-dataset: A dataset of datasets for learning to learn from few examples.  
[3] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes.  
[4] Film: Visual reasoning with a 382 general conditioning layer.  
[5] Task2vec: Task embedding for meta-learning.


Update post rebuttal

The authors provided a lengthy response with additional experiments. These partly address some concerns but raise a few more. 
When using more tasks and relevant baselines as reported here, I can see that the margins of improvements are significantly less than what is reported in the paper. For ex, FT-PN-BN is competitive with the proposed method. Some of the new results are also very puzzling. Like why does prototypical networks after fine-tuning becomes much worse than without fine-tuning. Another related problem I hadn't noticed in the first review was the difference between prototypical and other fine-tuning baselines, this difference seems too large, contrary to previous literature, and warrants more exploration. Comparing some results in the rebuttal and in Table 1, it seems that most of the improvements for the proposed approach are coming from the 3 new introduced datasets. These results are also very surprising. For instance, what is the property of SNIPS dataset on which Leopard and PN suddenly perform at half the accuracy of Grad2Task (21, 28 vs 52)!

I have still raised my score in favor weak accept since the rebuttal did address some concerns. Authors should incorporate these results and the presentation improvements in the paper. 
limitations_and_societal_impact: There is an appropriate discussion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary:   A new method for few-shot text classification
main_review: Th paper has a good analytical presentation and a set of thorough experiments that demonstrate it is better on the selected datasets. It is not clear how many labeled instances are required, and at which threshold the method degrades. 
limitations_and_societal_impact: Not convincing; issues raised have to do with text classification n in general, b not so much with the paper
review_title: null

Meta-review: The reviewers all agreed that that the method of encoding gradients to obtain a task embedding and using FILM to adapt the text-classification network to that task. Many of the initial concerns raised primarily by reviewer WPsh were addressed in the rebuttal through additional experiments, leading the reviewer to raise their score by 1. In general, the design choices and claims are well supported by ablation studies and comparisons against various baselines on various datasets. The writing is also structured and of decent quality. That said, the authors need to address the new "puzzling" issues raised by reviewer WPsh's updated review on the fine-tuning and comparison to other methods, especially as they cast doubt on the validity of the reported numbers.
Decision: Accept (Poster)