Grad2Task: Improved Few-shot Text Classification Using Gradients for Task Representation
Year: 2022

Abstract:
Large pretrained language models (LMs) like BERT have improved performance in many disparate natural language processing (NLP) tasks. However, fine tuning such models requires a large number of training examples for each target task. Simultaneously, many realistic NLP problems are "few shot", without a sufficiently large training set. In this work, we propose a novel conditional neural process-based approach for few-shot text classification that learns to transfer from other diverse tasks with rich annotation. Our key idea is to represent each task using gradient information from a base model and to train an adaptation network that modulates a text classifier conditioned on the task representation. While previous task-aware few-shot learners represent tasks by input encoding, our novel task representation is more powerful, as the gradient captures input-output relationships of a task. Experimental results show that our approach outperforms traditional fine-tuning, sequential transfer learning, and state-of-the-art meta learning approaches on a collection of diverse few-shot tasks. We further conducted analysis and ablations to justify our design choices.

heading: 1 Introduction
text: Transformer-based pretrained large-scale language models (LMs) have achieved tremendous success on many NLP tasks [15, 34], but require a large number of in-domain labeled examples for finetuning [54]. One approach to alleviate that issue is to fine-tune the model on an intermediate (source) task that is related to the target task. While previous work has focused on this setting for a single source task [33, 50], the problem of transferring from a set of source tasks has not been thoroughly considered. Wang et al. [51] showed that a naïve combination of multiple source tasks may negatively impact target task performance. Additionally, transfer learning methods typically consider the setting where more than a medium (N>100) number of training examples are available for both the source and target tasks.
In this work we develop a method to improve a large pre-trained LM for few-shot text classification problems by transferring from multiple source tasks. While it is possible to directly adapt recent advances in meta-learning developed for few-shot image classification, a unique challenge in the NLP setting is that source tasks do no share the same task structure. Namely, on standard few-shot image classification benchmarks, the training tasks are sampled from a “single” larger dataset, and the label space contains the same task structure for all tasks. In contrast, in text classification tasks, the set of source tasks available during training can range from sentiment analysis to grammatical acceptability judgment. Different source tasks could not only be different in terms of input domain, but also their task structure (i.e. label semantics, and number of output labels).
This challenging problem requires resistance to overfitting due to its few-shot nature and more task-specific adaptation due to the distinct nature among tasks. Fine-tuning based approaches are known to suffer from the overfitting issue and approaches like MAML [17] and prototypical networks
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
(ProtoNet) [41] were not designed to meta-learn from such a diverse collection of tasks. CNAP [37], a conditional neural process [18], explicitly designed for learning from heterogeneous task distributions by adding a task conditional adaptation mechanism is better suited for solving tasks with a shifted task distribution. We design our approach based on the CNAP framework. Compared with CNAP, we use pretrained transformers for text sequence classification instead of convolutional networks for image classification. We insert adapter modules [21] into a pretrained LM to ensure efficiency and maximal parameter-sharing, and learn to modulate those adapter parameters conditioned on tasks, while CNAP learns to adapt the whole feature extractor. Additionally, we use gradient information for task representation as opposed to average input encoding in CNAP, since gradients can capture information from both input and ouput space.
In summary, our main contributions and findings are:
• We propose the use of gradients as features to represent tasks under a model-based metalearning framework. The gradient features are used to modulate a base learner for rapid generalization on diverse few-shot text classification tasks.
• We use pre-trained BERT with adapter modules [21] as the feature extractor and show that it works better than using only the BERT model on few-shot tasks, while also being more efficient to train as it has much fewer parameters to learn.
• We compare our approach with traditional fine-tuning, sequential transfer learning, and stateof-the-art meta-learning approaches on a collection of diverse few-shot text classification tasks used in [5] and three newly added tasks, and show that our approach achieves the best performance. Our codes are publicly available1.

heading: 2 Related Work

heading: 2.1 Few-shot text classification
text: There is a wide range of text classification tasks, such as user intent classification, sentiment analysis, among others. Few-shot text classification tasks refer to those that contain novel classes unseen in training tasks and where only a few labeled examples are given for each class [55]. Meta-learning approaches have been proposed for this problem [7, 19, 5]. To increase the amount and diversity of training episodes, previous work trained few-shot learners by semi-supervised learning [40] and selfsupervised learning [6]. We compare different meta-learning approaches following the experiment procedures of [5].

heading: 2.2 Meta-learning
text: Metric-based meta-learning approaches [49, 41, 45] try to learn embedding models on which simple classifiers can achieve good performance. Optimization-based approaches [17, 38, 30] are designed to learn good parameter initialization that can quickly adapt to new tasks within a few gradient descent steps. “Black box” or model-based meta-learning approaches use neural networks to embed task information and predict test examples conditioned on the task information [39, 36, 28, 31, 18, 37]. Our approach falls into this third category and is mostly related to CNAP [37]. Compared with CNAP we use a different feature extractor and different mechanisms for modulation, as well as focusing on different problems. An important difference with CNAP is that we use gradient information as features to represent tasks, instead of using average input encoding. Gradient information has also been used to generate task-dependent attenuation of parameter initialization under the MAML framework [4].

heading: 2.3 Transfer learning
text: Our work is also related to transfer learning. Instead of fine-tuning a pretrained language model directly on a target task, Phang et al. [33] and Talmor and Berant [46] showed that intermediate training on data-rich supervised tasks is beneficial for downstream performance. Wang et al. [51] found that there is no guarantee that multi-task learning or intermediate training yields better performance than direct fine-tuning on target tasks. Transferability estimation is the task of predicting whether it is
1https://github.com/jixuan-wang/Grad2Task
beneficial to transfer from some task to another one. There have been works estimating transferability of NLP tasks based on handcrafted features [8, 23], while data-driven approaches have been studied mainly for computer vision tasks [56, 47, 43, 29].
TASK2VEC is a task embedding approach based on the Fisher Information Matrix (FIM) [2]. Transferability between tasks can be easily evaluated by the distance between their task embeddings according to some simple metric. In the NLP domain, given a target task and a predefined set of source tasks, Vu et al. [50] proposed to use task embedding based on FIM to measure task similarity and predict transferability according to the distance between task embeddings. We draw inspiration from this work to use gradients as features for task representation. Instead of one-to-one transfer, our work can be seen as transferring from a set of source tasks to new tasks through meta-learning.

heading: 3 Problem Definition
text: Following the terminologies in meta learning, each task (or episode), t = (S,Q), is specified by a support set (training set) S and a query set (test set) Q, where S = {(xi, yi)}|S|i=1, Q = {(xi, yi)} |Q| i=1, xi is a text sequence and yi is a discrete value corresponding to a class. Different tasks could have different input domains, different numbers of classes, different label space, and so on.
Our goal is to train a text classification model on a set of labeled datasets {Dsi }Ni=1 of N source tasks. The model is expected to achieve good performance on new target tasks after training. For each target task, a small labeled dataset, e.g., five shots for each class, is given for adaptation or fine-tuning and the full test dataset is used for evaluation. Note that when learning on target tasks, we assume no access to the source tasks nor other target tasks. This is different with multi-task learning where target tasks are learned together with source tasks.

heading: 4 Model Design
text: To handle diverse tasks with different structures a single model may not be flexible enough. Thus, for every task, we utilize a task embedding network to capture the task nature and adapt a base model conditioned on the current task. Task-specific adaptation is done by generating shifting and scaling parameters, named adaptation parameters, that are applied on the hidden representations inside the base model. Figure 1 shows an overview of our model architecture, mainly consisting of three parts: a base model denoted by f , a task embedding network denoted by d, and an adaptation network denoted by a.
To make a prediction, our model takes the following steps: 1) given a support set, it uses the base model (Section 4.1) to compute gradients w.r.t. a subset of its parameters to use as input to the task embedding network (Section 4.2), 2) the task embedding network maps these gradients to task representations, 3) layer-wise adaptation networks (Section 4.3) take as input the task representations and output adaptation parameters, and lastly 4) the adaptation parameters are applied to the base model before it predicts on the query set.
The training of our model consists of two stages: 1) we first train the base model episodically as a prototypical network; 2) we then freeze the base model and train the task embedding network and the adaptation network using the same loss function of the first stage (Section 4.4).
Next we introduce our model in details. The list of notations used throughout the paper can be found in Appendix A.

heading: 4.1 Base Model: BERT & Bottleneck Adapters
text: Our base model is built upon transformer-based pretrained LMs. We use the pretrained BERTBASE model throughout this paper, although other transformer-based pretrained LMs are also applicable [35, 25]. Following Houlsby et al. [21], we insert two adapter modules containing bottleneck layers into each transformer layer of BERT. Each bottleneck adapter is denoted as ↵l, l = 1...2L where L is the total number of transformer layers in BERT. We use the bottleneck adapters because, first, BERT with adapters are more efficient to train and less vulnerable to overfitting – a desirable property in the few-shot setting. Additionally, this simplifies our goal of using gradients for task representation. It is infeasible to use the gradients of the whole BERT model because the dimensionality is too large.
Instead, we compute gradients w.r.t. parameters in these bottleneck layers (< 0.5% of the number of parameters in BERT).
Since text classification is a sequence classification task, we need a pooling method to represent each sequence by a single embedding before the embedding is fed into a classifier. Following Devlin et al. [15], this single embedding is obtained by applying a linear transformation on top of the contextual embedding of the special ‘[CLS]’ token inserted at the beginning of every sequence. We further apply a fully connected layer on top of the BERT output. Together, we name the pretrained BERT with adapters and the last linear layer as the ‘base model’, denoted by f . We denote the parameters of the pretrained BERT as ✓, bottleneck adapters as {↵l}2Ll=1, and linear layer as !. For the classifier, we use the ProtoNet classifier [41]. Given a support set S , a query example x and our base model f , a nearest cluster classifier can be described as follows:
pbase(y = c |x) = softmax(euc(f(x; ✓,↵,!), µc)), (1)
µc = 1 |Sc| X
xi2Sc f(xi; ✓,↵,!)
where Sc = {(xi, yi) | (xi, yi) 2 S, yi = c}, µc is the cluster center in the embedding space for support examples with class c, and euc refers to the Euclidean distance.

heading: 4.2 Task Embedding Network for Per-Layer Task Encoding
text: Our task embedding network, denoted by d, is parametrized as a recurrent neural network (RNN) over the layers. We use an RNN to summarize the gradient information from the lower layers so that the higher layers can better adapt to the current tasks. Also, using RNNs can enable parameter sharing across different layers. Concatenating gradients of all layers as input will result in an extremely large task embedding model. The task embedding network takes as input at each layer the gradient information of the adapter parameters at the given layer. Following Achille et al. [2], we use the
gradient information defined by the FIM of the base model’s parameters, denoted by ⇥ = {✓,↵,!}. The FIM is computed as:
F⇥ = Ex,y⇠p̂(x)pbase(y | x)[r⇥ log pbase(y |x)r⇥ log pbase(y |x)T ] (2) where p̂(x) is the empirical distribution defined by the dataset. Following Achille et al. [2], we only use the diagonal values of F⇥. We only use those values corresponding to the adapter parameters, denoted by g↵l for adapter layer l, l = 1 . . . 2L.
Then, the RNN-based task embedding network d, parameterized by , maps each g↵l into a task embedding:
etaskl = d o(hl; ), (3)
hl = d h(g↵l , hl 1; )
where hl is the hidden representation at layer l, etaskl is the task embedding at layer l, h0 is the learnable initial RNN hidden state, and do and dh are the functions of d that produce output and hidden states, respectively.

heading: 4.3 Adaptation Network with Auto-Regressive Adaptation
text: As shown on the right side of Figure 1, at each layer l, the adaptation network output four adaptation parameters: midl and mid l are scaling and shifting adaptation parameters applied on the hidden representation after the middle layer, and outl and out l are scaling and shifting adaptation parameters applied on the output hidden representation. The input to the adaptation network at each layer consists of both the task representation and the intermediate activation output by previous adapted layers. Due to the conditioning on previous modulated layers, we refer this adaptation method as auto-regressive adaptation, following Requeima et al. [37].
Starting from the input layer, the adaptation network a produces adaptation parameters for the first adapter layer as follows:
mid1 , mid 1 , out 1 , out 1 = a(e task 1 , x; ) (4)
where x is input to the base model. We denote the bottleneck adapter network after modulation as ↵0.
For the subsequent layers, the adaptation network takes as input the combination of the task embedding and intermediate activation of the adapted base model, and outputs adaptation parameters as follows:
midl , mid l , out l , out l = a(e task l , fl 1(x; ✓, {↵0j}l 1j=1); ), l = 2 . . . 2L (5)
where fl 1(x; ✓, {↵0j} l 1 j=1) denotes the intermediate activation before layer l from the base model with the first l 1 adapters that have been modulated. Finally, the prediction from our modulated base model is written as:
pfinal(y = c |x) = softmax(euc(f(x; ✓,↵0,!), µc)), (6)
µc = 1 |Sc| X
xi2Sc f(xi; ✓,↵
0,!)
Notice that the only difference between pfinal(y = c |x) and pbase(y = c |x) is using ↵0 or ↵, i.e. our task-specific modulation.
Our adaptation mechanism is similar with the FiLM layer [32] but different in terms of that here we only apply adaptation on the hidden representation of the “[CLS]”, since only its embedding is used as the sequence embedding. This is analogous to only adapting the the first channel of each hidden representation to a Film layer. We also tried adapting embeddings of all tokens but performance is worse, as shown in Section 6.2.

heading: 4.4 Model Training
text: Similar with CNAP [37], we train our model in two stages. In the first stage, we train the base model using episodic training [41]. The training algorithm is showed in Appendix B. Given an episode t = (S,Q), the loss is computed as:
`pn(t) = 1 |Q| X
xi2Q log pbase(yi |xi) (7)
Algorithm 1: Training the task embedding network and adaptation network for quick adaptation to new tasks. Input : T : A set of text classification tasks for training. Di: The labeled dataset of each task i. The base model trained in the first stage. Output : : Parameters of the task embedding model. : Parameters of the adaptation network.
1 while not converge do 2 Sample M episodes {ti}Mi=1, each episode ti = (Si,Qi),Si ✓ Di,Qi ✓ Di 3 for i 1 to M do 4 for s 1 to S do 5 Sample a few examples for each class from Si and build prototypes {µc} 6 Sample another subset of examples Ŝi ✓ Si 7 For each (xj , yj) 2 Ŝi, sample y0j ⇠ pbase(y |xj) // pbase is given by Equation (1) 8 g↵s = P|Ŝi| j=1 r↵ log pbase(y = y0j |xj) 2 , (xj , yj) 2 Ŝi // calculating gradient // information as in Equation (2) 9 end
10 g↵ = 1S PS s=1 g ↵ s 11 etask = d(g↵; ) // generating task representation 12 in, in, out, out = a(etask; ) // generating adaptation parameters 13 `i = 1|Qi| P|Qi| j=1 log p final(y = yj |xj) // calculating the loss on query set // pfinal is given by Equation (6) 14 end 15 { , } { , } lr ·r{ , } PM i=1 `i // gradient descent on and 16 end 17 return ,
During this stage, only the parameters of the bottleneck adapters, layer normalization parameters and the top linear layer are updated while other parameters of the base model are frozen. The parameters are learned by:
↵⇤,!⇤ = argmin ↵,! Et2T [`pn(t)] (8)
In the second stage, the task embedding network and the adaptation network are trained episodically to generate good quality task embeddings and adaptation parameters for better performance on the query set of each episode. Specifically, as shown in Algorithm 1, we freeze the encoding network and only train the task embedding network d and the adaptation network a. In this stage, the frozen base model is used to generate task-specific gradient information and also be adapted to predict the query labels. Same with the first training stage, we also use the ProtoNet loss to train d and a but use the adapted based model. The parameters of the task embedding network and the adaptation network is learned by:
⇤, ⇤ = argmin , Et2T [`0pn(t)], (9)
`0pn(t) = 1 |Q| X
xi2Q log pfinal(yi |xi)
where the loss is computed based on the modulated model.

heading: 5 Experiments and Results

heading: 5.1 Experiment setup
text: We use datasets for training and testing that have different input domains and different numbers of labels. Each dataset appears during either training or testing – not both. Following [5], we use tasks from the GLUE benchmark [52] for training. Specifically, we use WNLI (m/mm), SST-2, QQP, RTE, MRPC, QNLI, and the SNLI dataset [10], to which we refer as our ‘meta-training datasets’. The validation set of each dataset is used for hyperparameter searching and model selection. We train
our model and other meta-learning models by sampling episodes from the meta-training tasks. The sampling process first selects a dataset and then randomly selects k-shot examples for each class as the support set and another k-shot as the query set. As with Bansal et al. [5], the probability of a task being selected is proportional to the square root of its dataset size.
We use the same test datasets as Bansal et al. [5], but also add several new datasets to increase task diversity, namely the Yelp [1] dataset to predict review stars, the SNIPS dataset [13] for intent detection, and the HuffPost [26] dataset for news headline classification. We refer to this set of tasks as our ‘meta-testing datasets’. For each test task and a specific number of shot k, ten k-shot datasets are randomly sampled. Each time one of the ten datasets is given for model training or fine-tuning, the model is then tested on the full test dataset of the corresponding task. This is in line with real world applications where models built on a small training set are tested on the full test set.
Several test datasets used by Bansal et al. [5] contain very long sentences, but they limited the maximal sequence length to 128 tokens in their experiments. Truncating long sequences beyond the maximal length might lose important information for text classification, leading to noise in the test results. In order to ensure the results can truly reflect text classification, we discard a few datasets used by Bansal et al. [5] that contain many very long sentences.
See Appendix C for more details about the datasets we use. See Appendix G.1 for the results on all datasets used by Bansal et al. [5].

heading: 5.2 Few shot text classification results
text: We use the cased version of the BERTBASE model for all experiments. We compare our proposed approach with the following approaches:
BERT. The pretrained BERT is simply fine-tuned on the labeled training data of the target task and then evaluated on the corresponding test data. No transfer learning happens with this model.
MT-BERT. The pretrained BERT is first trained on the meta-training tasks via multi-task learning and then fine-tuned and evaluated on each test task.
ProtoNet-BERT. ProtoNet using pretrained BERT plus a linear layer as the feature extractor. The model is trained episodically on the meta-training datasets.
ProtoNet-BN. Similar with ProtoNet-BERT but with adapter modules inserted in the transformer layers. Only the parameters of the adapters, linear layer, and layer normalization layers inside BERT are updated during training. The model is trained episodically on the meta-training datasets.
MAML based approach. We compare with the MAML-based approach, named Leopard, proposed by Bansal et al. [5] with first-order approximation and meta-learned per-layer learning rates. We re-implemented this model, and include both their reported results and results of our implementation for fair comparison. See Appendix D.1 for more details.
Grad2Task. This is our proposed model built upon pretrained ProtoNet-BN and trained episodically to learn to adapt.
Results are shown in Table 1. Overall, our proposed method achieves the best performance. Our model is built upon ProtoNet-BN but keeps its parameters untouched and only learns to adapt the bottleneck adapter modules for different tasks. On average, our approach improves over ProtoNet-BN by 1.02%, indicating task conditioning can further improve a strong baseline.
Surprisingly, we find that ProtoNet-BERT achieves very good performance and outperforms the optimization-based method, Leopard, while Bansal et al. [5] reported the opposite results. We thus re-implemented both Leopard and ProtoNet-BERT and confirm that our MAML implementation has similar performance with Leopard but our ProtoNet-BERT results are much better than theirs (see Appendix D.1 for a more detailed comparison).
Although we observe that ProtoNet-BERT has better performance and faster convergence rates during training and validation, it is outperformed by ProtoNet-BN which has orders of magnitude fewer parameters to learn (fewer than 0.5% of the number of BERT’s parameters). We hypothesize this is because ProtoNet-BERT is more vulnerable to overfitting on the meta-training tasks.
See Appendix D.2 for the details of implementation, model size and training efficiency. We also compare with other fine-tuning approaches, like further fine-tuning a trained ProtoNet. See Appendix G.2 for the results.

heading: 6 Analysis and Ablations

heading: 6.1 Same/different task classification and task embedding visualization
text: Capturing task nature is the prerequisite for task-specific adaptation conditioned on task embeddings. We evaluate the ability of gradients as task representations through a toy experiment and visualization. The toy experiment is a binary classification task about predicting whether two few-shot datasets are sampled from the same task or not. We name this task as “same/different task classification.” For each pair of few-shot datasets, we calculate the ProtoNet loss and its gradients using the base model on the two datasets, respectively. We then feed the gradients on the two datasets into a single linear layer, respectively. Prediction is given by the cosine similarity between the two representations after linear transformation and the binary cross entropy loss is used for training.
We train the linear model on few-shot dataset pairs sampled from our meta-training datasets, and test it on dataset pairs sampled from our meta-testing datasets. Figure 2 shows the AUC curves on the testing set. Although the model is simple, it can predict same/different labels reasonably well on unseen tasks during training. With more shots, the gradient becomes more reliable, thus the model achieves better performance, resulting in an AUC score of 0.84 with 16 shots.
The results of the same/different task classification experiment confirm that gradients can be used as features to capture task nature and distinguish between tasks. To further evaluate the quality of the task embeddings learned by our model, we visualize the learned task embeddings in 2D space. We find the task embeddings form good quality clusters according to task nature at higher layers. More details can be found in Appendix E.
6.2 Ablation Study Table 2: Ablation results. Model Mean Acc.
Grad2Task w/ Gradients 45.99 ProtoNet Longer Training 45.10 Grad2Task w/ X 45.66 Grad2Task w/ X&Y 45.16 Grad2Task Adapt All 44.57 Grad2Task w/ Pretrained TaskEmb 45.68 Hypernetwork 44.79 We conduct ablation studies to justify our decision choices. We report the average accuracy on metatesting datasets of different model variants in Table 2. Results show that our approach achieves the best average accuracy among all model variants, resulting in average accuracy of 45.99%. The full results are included in Appendix F.
Does the adaptation really help? Starting from the base model trained after the first stage, we compare our approach with just training the base model using the ProtoNet loss for the same number of steps. Shown as “ProtoNet Longer Training” in Table 2, the performance of this approach is worse than our proposed approach. This justifies that we can do better by adapting the base model than training it for more steps.
How are gradients compared with other task representations? We compare our approach with other CNP-based approaches with different methods of task embedding. Similar to Requeima et al. [37], we use the average embeddings of the training sequences as representations for a task, and keep the other model architecture unchanged. We also consider using the embeddings of both input sequences and labels for task representation. We treat labels as normal text instead of discrete values and encode them using the BERT model. The label embedding and average input embeddings are concatenated as the task representation. Shown as “Grad2Task w/ X” and “Grad2Task w/ X&Y”, respectively, in Table 2, both of these two approach underperform our gradient-based method while “Grad2Task w/ X” performs closely with our method. Note that we can also combine those methods for task representation, which we leave for future work.
Adapt all or just “[CLS]"? The only difference between this model and ours is whether we apply the generated adaptation parameters only on the hidden representation of the “[CLS]" token or on all tokens in each sequence. Shown as “Grad2Task Adapt All” in Table 2, this model performs worse, resulting in 44.57% on average.
Generating model parameters or adaptation parameters? We also experimented with a hypernetwork [20]-based approach. Conditioned on the task representation, we use a neural network (hypernetwork) to output the parameters of bottleneck adapters directly, instead of freezing the bottleneck adapters and only generating scaling/shifting parameters, as in our proposed model. The model with a hypernetwork has higher flexibility for task adaptation, but the size of the hypernework is large, as it maps high-dimensional gradient information to high-dimensional model parameters, which brings challenges for optimization and suceptibility for overfitting. As shown in Table 2, it achieves accuracy of 44.79% on average and is worse than our proposed approach.
Pretraining task embedding? For this model, we first pretrain the task embedding model for the same/different tasks, then freeze and combine it with other components following our proposed approach. We observe that it has close but slightly worse performance (45.68%) than Grad2Task without task embedding pretraining. We hypothesize the task embedding trained with the same/different tasks is not good enough. We expect this model will benefit from pretraining the task embedding
model on a set of datasets with higher diversity and training the task embedding model using more advanced metric-learning approaches, which we leave for future work.

heading: 7 Discussion and Conclusion
text: In this paper, we propose a novel model-based meta-learning approach for few-shot text classification and show the feasibility of using gradient information for task conditioning. Our approach is explicitly designed for learning from diverse text classification tasks how to adapt a base model conditioned on different tasks. We use pretrained BERT with bottleneck adapters as the base model and modulate the adapters conditioned on task representations based on gradient information.
Our work is an inaugural exploration of using gradient-based task representations for meta-learning. It has several limitations. First, the way we use neural networks to encode high-dimensional data significantly increases the number of parameters to train. For future work, we will explore more efficient ways for FIM calculation and better ways to handle high-dimensional gradient information. Second, we only focus on text sequence classification tasks in this paper. There are various types of NLP tasks to explore, such as question answering and sequence labeling tasks, which present many distinct challenges. Also, we only consider transferring between text classification tasks. As shown in Vu et al. [50], positive transfer can happen among different types of NLP tasks. Thus, we intend to explore meta-learning or transfer learning approaches to learning from different types of tasks.
Ethical Concerns. Our work advances the field of few-shot text classification. While not having any direct negative societal impact, one can imagine that a few-shot text classifier is abused by a malicious user. For example, classifying trendy keywords on the internet naturally falls into the application area of our method as example phrases containing the new keyword can be limited in number. Toxic language is known to be a pervasive problem in the Internet [3], and a malicious user could use a good few-shot text classifier to further perpetuate these problems (e.g., by classifying toxic text as benign and fooling the end user). On the other hand, the same system can be used to promote fairness.
Acknowledgments and Disclosure of Funding
We would like to thank reviewers and ACs for constructive feedback and discussion. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute (www.vectorinstitute. ai/#partners). JW is supported by the RBC Graduate Fellowship. FR and MB are CIFAR AI Chairs.

Review summaries:
The paper considers the problem of few-shot text classification. It builds on a bunch of prior works on meta-learning [1,2,3], Film adapters [4], task embeddings [5] and tries to combine them for the task of few-shot text classification. The key idea seems to be to use adapters in between transformer layers, compute task embeddings (using fisher Information matrix) for a task and use it to update Film parameters that are used to update the model. The approach is compared with a prior work [1] and prototypical networks on a subset of few-shot classification tasks selected from [1]. 

The paper presents a novel method for few-shot text classification tasks. The method is built on top of the CNAP  The key contribution is to effectively encode the gradients of networks as the task representation so that the meta-learning stage can use such task representations to improve the few-shot generalization ability. The two key technical contributions are a task embedding network (an RNN-based layer-by-layer gradient encoder via Fisher Information Matrix) and an adaptation network for auto-regressively adapting parameters from the bottom to the top layer.  The implementation is based on BERT-like LMs with Adapter bottleneck modules, which are parameter efficient to compute.   The authors compared the proposed method with a few meta-learning methods on a diverse set of tasks, and show that their method outperforms other methods for most of the time. The empirical analysis and ablations also provide insights for understanding the proposed method.    

This paper proposed Grad2Task, a model for few-shot text classification tasks. Grad2Task first computes gradients when fine-tuning the base model on a support set; then the task embedding network decodes task representations using these gradients; finally the task representations go through MLP layers and output scaling and shifting parameters, which are plugged into the base model for task-specific adaptation.   The paper provides comprehensive evaluation on 10 tasks. Grad2Task outperforms strong baselines (Leopard, ProtoNet-BERT) in most settings. Analysis with a toy dataset shows the task representations are informative. Ablation study justified several design choices in Grad2Task.