Grad2Task: Improved Few-shot Text Classification Using Gradients for Task Representation
Authors: Jixuan Wang
Kuan-Chieh Wang
Frank Rudzicz
Michael Brudno
Venue: NeurIPS 2021 Poster (2021)
Keywords: few-shot text classification
meta-learning
transfer learning
conditional neural process
transformer
pretrained language models

Abstract: Large pretrained language models (LMs) like BERT have improved performance in many disparate natural language processing (NLP) tasks. However, fine tuning such models requires a large number of training examples for each target task. Simultaneously, many realistic NLP problems are "few shot", without a sufficiently large training set. In this work, we propose a novel conditional neural process-based approach for few-shot text classification that learns to transfer from other diverse tasks with rich annotation. Our key idea is to represent each task using gradient information from a base model and to train an adaptation network that modulates a text classifier conditioned on the task representation. While previous task-aware few-shot learners represent tasks by input encoding, our novel task representation is more powerful, as the gradient captures input-output relationships of a task. Experimental results show that our approach outperforms traditional fine-tuning, sequential transfer learning, and state-of-the-art meta learning approaches on a collection of diverse few-shot tasks. We further conducted analysis and ablations to justify our design choices.

Grad2Task: Improved Few-shot Text Classification Using Gradients for Task Representation
Jixuan Wang 4.233
Kuan-Chieh Wang Frank Rudzicz! 1.2.
Michael Brudno' 1.2 1,2,3
University of Toronto, Vector Institute, University Health Network, Uniiy Health Toronto {jixuan, wangkual, frank, brudno]@cs. toronto... edu
Abstract
Large pretrained language models (LMs) like BERT have improved performance in many disparate natural language processing (NLP) tasks. However, fine tuning such models requires a large number of training examples for each target task. Simultaneously. many realistic NLP problems are "few shot", without sufficiently large training set. In this work, we propose a novel conditional neural process- based approach for few-shot text classification that learns to transfer from other diverse tasks with rich annotation. Our key idea is to represent each task using gradient information from base model and to train an adaptation network that modulates a text classifier conditioned on the task representation. While previous task aware few-shot learners represent tasks by input encoding, novel task representation is more powerful, as the gradient captures input-output relationships of task. Experimental results show that our approach outperforms traditional fine-tuning, sequential transfer learning, and state-of-thh-arr meta learning approaches on a collection of diverse few-shot tasks. We further conducted analysis and ablations to justify our design choices.
Introduction
Transformer-based pretrained large-scale language models (LMs) have achieved tremendous success many NLP tasks [15, 34], but require a large number of in-domain labeled examples for fine- tuning [54]. One approach to alleviate that issue is to fine-tune the model on an intermediate (source) task that is related the target task. While previous work has focused on this setting for single source task [33, 50], the problem of transferring from a set of source tasks has not been thoroughly considered. Wang et al. [51] showed that na've combination of multiple source tasks may negatively impact target task performance. Additionally, transfer learning methods typically consider the setting where more than medium (N>100) number of training examples are available for both the source and target tasks.
In this work develop method to improve large pre-trained LM for few-shot text classification problems by transferring from multiple source tasks. While it is possible to directly adapt recent advances in meta-learning developed for few-shot image classification, unique challenge in the NLP setting is that source tasks do no share the same task structure. Namely, on standard few-shot image classification benchmarks, the training tasks sampled from "single" larger dataset, and the label space contains the same task structure for all tasks. In contrast, in text classification tasks, the set of source tasks available during training can range from sentiment analysis to grammatical acceptability judgment. Different source tasks could not only be different in terms of input domain, but also their task structure (i.e. label semantics, and number of output labels).
This challenging problem requires resistance to overfitting due to its few-shot nature and more task-specific adaptation due to the distinct nature among tasks. Fine-tuning based approaches known to suffer from the overfitting issue and approaches like MAML [17] and prototypical networks
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(ProtoNet) [41] were not designed to meta-learn from such diverse collection of tasks. CNAP [37], conditional neural process [18]. explicitly designed for learning from heterogeneous task distributions by adding a task conditional adaptation mechanism is better suited for solving tasks with a shifted task distribution. We design our approach based on the CNAP framework. Compared with CNAP, we use pretrained transformers for text sequence classification instead of convolutional networks for image classification. We insert adapter modules [21] into a pretrained LM to ensure efficiency and maximal parameter-sharing, and learn to modulate those adapter parameters conditioned on tasks, while CNAP learns to adapt the whole feature extractor. Additionally, we use gradient information for task representation as opposed to average input encoding in CNAP, since gradients capture information from both input and ouput space.
In summary, our main contributions and findings are:
We propose the use of gradients as features to represent tasks under a model-based meta- learning framework. The gradient features are used to modulate base learner for rapid generalization on diverse few-shot text classification tasks.
We use pre-trained BERT with adapter modules [21] as the feature extractor and show that it works better than using only the BERT model on few-shot tasks, while also being more efficient train it has much fewer parameters learn.
We compare approach with traditional fine-tumine tinettunigg, sequential transfer learning, and state- of-the-art meta-learning approaches on a collection of diverse few-shot text classification tasks used in [5] and three newly added tasks, and show that approach achieves the best performance. Our codes are publicly available'.
2 Related Work
2.1 Few-shot text classification
There a wide range of text classification tasks, such as user intent classification, sentiment analysis, among others. Few-shot text classification tasks refer to those that contain novel classes unseen in training tasks and where only few labeled examples are given for each class [55]. Meta-learning approaches have been proposed for this problem [7, 19, 5J. To increase the amount and diversity of training episodes, previous work trained few-shot learners by semi-supervised learning [40] and self- supervised learning [6]. We compare different meta meta-learning approaches following the experiment procedures of [5].
2.2 Meta-learning
Metric-based meta-leaming approaches [49, 41, 45] try to learn embedding models on which simple classifiers can achieve good performance. Optimization-ban learn good parameter initialization that can quickly adapt new tasks within few gradient descent steps. "Black box" or model-based meta-learning approaches use neural networks to embed task information and predict test examples conditioned on the task information [39, 36, 28, 31, 18, 37]. Our approach falls into this third category and is mostly related to CNAP [37]. Compared with CNAP we use a different feature extractor and different mechanisms for modulation, as well as focusing on different problems. An important difference with CNAP is that we use gradient information as features to represent tasks, instead of using average input encoding. Gradient information has also been used to generate task-dependent attenuation of parameter initialization under the MAML framework [4].
approaches [17, 30] are designed to
2.3 Transfer learning
Our work is also related to transfer learning. Instead of fine-tuning pretrained language model di- rectly on a target task, Phang et al. [33] and Talmor and Berant [46] showed that intermediate training on data-rich supervised tasks is beneficial for downstream performance. Wang et al. [51] found that there is no guarantee that multi-task learning or intermediate training yields better performance than direct fine-tuning on target tasks. Transferability estimation is the task of predicting whether it is
'nttps://githu/ com/ jiljj-iunn-s wannn-wwn///a/a Gaaee/GradTaaak

---

beneficial to transfer from some task to another one. There have been works estimating transferability of NLP tasks based on handcrafted features [8, 23], while data-driven upproaches have been studied mainly for computer vision tasks [56, 47, 43, 29].
TACK2VEC is task embedding approach based on the Fisher Information Matrix (FIM) [2]. Transfer- ability between tasks can be easily evaluated by the distance between their task embeddings according to some simple metrie. In the NLP domain, given target task and predefined set of source tasks, Vu et al. [50] proposed to use task embedding based on FIM to measure task similarity and predict transferability according to the distance between task embeddings. We draw inspiration from this work to use gradients as features for task representation. Instead of one-to-one transfer, our work can be seen as transferring from a set of source tasks to new tasks through meta-learning
Problem Definition
Following the terminologies in meta learning, each task (or episode). / - (S, 2), is specified by support set (training set) S and a query set (test set) 2, where S = {(T,, V1 } Q = {(ri, })) 1=11 H; is a text sequence and yı is discrete value corresponding different input domains, different numbers of classes, different label space, and so on.
class. Different tasks could have
Our goal is train text classification model set of labeled datasets {D:) of N source tasks. The model is expected to achieve good performance on new target tasks after training. For each target task, a small labeled dataset, e.g., five shots for each class, is given for adaptation or fine-tuning and the full test dataset is used for evaluation. Note that when learning on target tasks, we assume no access to the source tasks other target tasks. This is different with multi-task learning where target tasks are learned together with source tasks.
4 Model Design
To handle diverse tasks with different structures a single model may not be flexible enough. Thus, for every task, we utilize task embedding network capture the task nature and adapt a base model conditioned on the current task. specific adaptation is done by generating shifting and scaling parameters, named adaptation parameters, that are applied the hidden representations inside the base model. Figure shows an overview of our model architecture, mainly consisting of three parts: base model denoted by f, a task embedding network denoted by d, and adaptation network denoted by a.
To make prediction, our model takes the following steps: 1) given a support set, it uses the base model (Section 4.1) to compute gradients w.r.t. a subset of its parameters to use as input to the task embedding network (Section 4.2), 2) the task embedding network maps these gradients to task representations, 3) layer-wise adaptation networks (Section 4.3) take as input the task representations and output adaptation parameters, and lastly 4) the adaptation parameters are applied to the base model before it predicts on the query set.
The training of our model consists of two stages: 1) we first train the base model episodically as prototypical network; 2) we then freeze the base model and train the task embedding network and the adaptation network using the same loss function of the first stage (Section 4.4).
Next we introduce our model in details. The list of notations used throughout the paper can be found in Appendix A.
4.1 Base Model: BERT & Bottleneck Adapters
Our base model is built upon transformer-basee pretrained LMs. We use the pretrained BERT model throughout this paper, although other transformer- pretrained LMs are also applicable [35, 25]. Following Houlsby et al. [21], we insert two adapter modules containing bottleneck layers into each transformer layer of BERT. Each bottleneck adapter is denoted as a,, l 1...2L where L is the total number of transformer layers in BERT. We use the bottleneck adapters because, first, BERT with adapters are more efficient to train and less vulnerable to overfitting a desirable property in the few-shot setting. Additionally, this simplifies our goal of using gradients for task representation. is infeasible to use the gradients of the whole BERT model because the dimensionality is too large.
3

---

Colculate grodients
Adopt and predict on set
Adaptation
set
Network
PL
MLP
Unear Layer
Uneer Layer
yl
Bottleneck Adapter
Bottlerscck Adapter
Fully Connected Layer
Task Embodiding Network
Fully Connected Layer
Bottleneck Adapter
Bottleneck Adapter
Nonlinearity
Self-Attention layer
Self-Atention Liyer
MLP
O
MLP
Yi
O
Bottienack Adapter
Bottleneck Adapter
Fully Connected Layer
Fully Connected Layor
O
Bottleneck Adapter
Bottleneck Adapter
Self-ietention Layer
Self-Attetionn Layer
(0000,
O
Base Model: Pretrained BERT Bottleneck Adaptars
Bottleneck Adapters with Adaptation
Support
Support
Querr
Figure I: Model architecture overview. Left: The base model contains a pretrained BERT model with bottleneck adapters inserted in each transformer layer and a linear layer stacked the top. The task embedding network is an RNN-based model that maps gradients from the hase model into task representations in layer-wise fashion, which are used for base model adaptation. Right: The adaptation network contains MLPs that take as input the task representations and intermediate activate inside the base model, and outputs shifting and scaling parameters that are used adapt the base model. Specifically, given the task embedding for a certain layer, we concatenate it with the input hidden representation to the current layer and map them into four shifting/scaling parameters by four MLP networks, which are applied on the hidden and output representations of the adapter, referred as the rugr-segressive adaptation.
Instead, we compute gradients w.r.t. parameters in these bottleneck layers (< 0.5% of the number of parameters in BERT).
Since text classification is a sequence classification task, we need a pooling method to represent each sequence by single embedding before the embedding is fed into a classifier. Following Devlin et al. [15], this single embedding is obtained by applying linear transformation on top of the contextual embedding of the special 'CLLS' token inserted at the beginning of every sequence. We further apply a fully connected layer on top of the BERT output. Together, we name the pretrained BERT with adapters and the last linear layer as the 'base model', denoted by ʃ. We denote the parameters the pretrained BERT 0, bottleneck adapters as {ar (or)22,, and linear layer as w.
For the classifier, we use the ProtoNet classifier [41]. Given support set S, a query example æ and our base model f. a nearest cluster classifier can be described as follows:
p huse (y = c x) = softmax(euc( 0 (,,w),
(1)
1
Σ Z.ES:
((2:0,a,u
Sc1
where S [z; y:) (zx, v:) S, v4 c), μº is the cluster center in the embedding space for support examples with class c, and euc refers to the Euclidean distance.
4.2 Task Embedding Network for Per-Layer Task Encoding
Our task embedding network, denoted by d, is parametrized as recurrent neural network (RNN) over the layers. We use an RNN to summarize the gradient information from the lower layers so that the higher layers can better adapt to the current tasks. Also, using RNNs can enable parameter sharing across different layers. Concatenating gradients of all layers as input will result in an extremely large task embedding model. The task embedding network takes as input at each layer the gradient information of the adapter parameters at the given layer. Following Achille al. [2], we use the

---

gradient information defined by the FIM of the base model's ' parameters, denoted by θ = {0, w). The FIM is computed as:
Fe E, py 2) Ve log puse (y) x) Ve log p x)=
where p(z) is the empirical distribution defined by the dataset. Following Achille et al. [2], we only use the diagonal values of Fe. We only use those values corresponding to the adapter parameters, denoted by of for adapter layer 1, 1... 2T.
Then, the RNN-based task embedding network d, parameterized by o, maps each of into task embedding:
cask d' (h;; Φ),
h1 = dd (of h(-1i)
where hy is the hidden representation at layer L, ef is the task embedding at layer 1, hg is the learnable initial RNN hidden state, and dº and dh are the functions of d that produce output and hidden states, respectively.
4.3 Adaptation Network with Auto- neto--Rereesivv Adaptation
As shown on the right side of Figure 1, at each layer the adaptation network output four adaptation parameters: 7/"ind and Irrdid are scaling and shifting adaptation parameters applied on the hidden representation after the middle layer, and 7/ and Bree are scaling and shifting adaptation parameters applied on the output hidden representation. The input to the adaptation network at each layer consists of both the task representation and the intermediate activation output by previous adapted layers. Due to the conditioning on previous modulated layers, we refer this adaptation method as auto-regressive adaptation, following Requeima et al. [37].
Starting from the input layer, the adaptation network a produces adaptation parameters for the first adapter layer as follows:
71 mid ,1 71 out S.qut a(e xx; v/)
grmid
where x is input to the base model. We denote the bottleneck adapter network after modulation as a'.
For the subsequent layers, the adaptation network takes as input the combination of the task embedding and intermediate activation of the adapted base model, and outputs adaptation parameters as follows:
yimid gymid vtt sppet = alctask fi-1 (z;; 0, {a))/=)):0)) I 2.....2 2L
(5)
where fi= 0, {a (=1) denotes the intermediate activation before layer / from the base model with the first adapters that have been modulated. Finally, the prediction from our modulated base model is written as:
Ginal (y- cl cl2) softmax(euc( (f; (w,
(6)
Σ
[( 0, a','
1
2.ES.
Notice that the only difference between final (y = c x) and p huse (y = x)) is using a' or a, i.c. our task-specific modulation.
Our adaptation mechanism is similar with the FiLM layer [32] but different in terms of that here we only apply adaptation on the hidden representation of the "[CLS]". since only its embedding is used as the sequence embedding. This is analogous only adapting the the first channel of each hidden representation to Film layer. We also tried adapting embeddings of all tokens but performance is worse, as shown Section 6.2.
4.4 Model Training
Similar with CNAP [37], we train our model in two stages. In the first stage, we train the base model using episodic training [41]. The training algorithm is showed in Appendix B. Given episode
t = (S, Q), the loss is computed as:
1
9pm (t)
log p base (y: (x;)
[
5

---

Algorithm 1: Training the task embedding network and adaptation network for quick adaptation
new tasks.
Input ::: A set of text classification tasks for training. D;: The labeled dataset of each task i. The base
model trained in the first stage.
Output:o: Parameters the task embedding model. di Parameters of the adaptation network.
while not converge do
Sample episodes {44) each episode (St, Q.). C. D.
for - to .M do
for 8 to S do
Sample few examples for each class from S; and build prototypes (1e)
Sample another subset of examples S, S S.
For each (2,, {/) St, sample v/ ~ (y // paace is given by Equation (1)
D/S 16, 5= (Va log p (y J/s x,)) (2,, 81) E // calculating gradient // information as Equation (2)
g:e
end
10
g" 3 Σ 9
11
e task
d(gg; Φ)
// generating task representation // generating adaptation parameters
7 1th 7 grntt = a(clmak;:))
12
4 10.1 E loog (imal (y 12²)
// calculating the loss query set
11 p timal given by Equation (6)
14
end
15
{ } + fo, 20 - br (ooo)) En li
// gradient descent on é and eb
16
end
17
return ó, き
During this stage, only the parameters of the bottleneck adapters, layer normalization parameters and the top linear layer are updated while other parameters of the base model are frozen. The parameters learned by:
a' w = arg minE,er|(pmm
0,,0
In the second stage, the task embedding network and the adaptation network are trained episodically generate good quality task embeddings and adaptation parameters for better performance on the query set of each episode. Specifically, as shown in Algorithm 1, we freeze the encoding network and only train the task embedding network d and the adaptation network a. this stage, the frozen base model is used to generate task-specifi gradient information and also be adapted to predict the query labels. Same with the first training stage, we also use the ProtoNet loss train and but use the adapted based model. The parameters of the task embedding network ø and the adaptation network & is learned by: 6¹, 6- arg minEEETT ( (1))
d.vi
pm (t)
log log final (y: xx))
2
where the loss is computed based on the modulated model.
Experiments and Results
5.1 Experiment setup
We use datasets for training and testing that have different input domains and different numbers of labels. Each dataset appears during either training or testing - not both. Following [5], we use tasks from the GLUE benchmark [52] for training. Specifically, we use WNLI (n/mm), SST-2, QQP, RTE, MRPC, QNLI, and the SNLI dataset [10], to which we refer as our 'meta-training datasets'. The validation set of each dataset is used for hyperparameter searching and model selection. We train

---

our model and other meta-learnn meea-learning models by sampling episodes from the meta-training tasks. The sampling process first selects dataset and then randomly selects k-shot examples for each class as the support set and another k-shot as the query set. As with Bansal et al. [5], the probability of a task being selected proportional the square root of its dataset size.
We use the same test datasets as Bansal et al. [5], but also add several new datasets to increase task diversity, namely the Yelp [1] dataset to predict review stars, the SNIPS dataset [13] for intent detection, and the HuftPost [26] dataset for news headline classification. We refer to this set of tasks as our meta-testing datasets'. For each test task and a specific number of shot ten k-shot datasets are randomly sampled. Each time one of the ten datasets is given for model training or fine-tuning, the model then tested on the full test dataset of the corresponding task. This in line with real world applications where models built on a small training set are tested on the full test set.
Several test datasets used by Bansal al. [5] contain very long sentences, but they limited the maximal sequence length to 128 tokens in their experiments. Truncating long sequences beyond the maximal length might lose important information for text classification, leading to noise in the test results. In order to ensure the results can truly reflect text classification, we discard a few datasets used by Bansal et al. [5] that contain many very long sentences.
See Appendix for more details about the datasets we use. See Appendix G.I for the results on all datasets used by Bansal et al. [5].
5.2 Few shot text classification results
We use the cased version of the BERT BASE model for all experiments. We compare our proposed approach with the following approaches:
BERT. The pretrained BERT is simply fine-tuned on the labeled training data of the target task and then evaluated on the corresponding test data. No transfer learning happens with this model.
MT-BERT. The pretrained BERT first trained on the meta-training tasks via multi-task learning and then fine-tuned and evaluated on each test task.
PrOToNet-BERT. ProtoNet using pretrained BERT plus linear layer as the feature extractor. The model is trained episodically on the meta-training datasets.
ProtoNet-BN. Similar with ProtoNet-BEie BERT but with adapter modules inserted in the transformer layers. Only the parameters of the adapters, linear layer, and layer normalization layers inside BERT updated during training. The model is trained episodically the meta-training datasets.
MAML based approach. We compare with the MAML-based approach, named Leopard, proposed by Bansal et al. [5] with first-order approximation and meta-learned per-layer learning rates. We re- re-implemented this model, and include both their reported results and results of our implementation for fair comparison. See Appendix D.I for more details.
Grad2Task. This is our proposed model built upon pretrained ProtoNet-BN and trained episodically to learn to adapt.
Results are shown in Table 1. Overall, proposed method achieves the best performance. Our model is built upon ProtoNet-BN but keeps its parameters untouched and only learns to adapt the bottleneck adapter modules for different tasks. On average, our approach improves over ProtoNet-BN by 1.02%, indicating task conditioning can further improve a strong baseline.
Surprisingly, find that ProtoNet-BERT achieves very good performance and outperforms the optimization-baser method, Leopard, while Bansal et al. [5] reported the opposite results. We thus re-implemented both Leopard and ProtoNet-BERT and confirm that our MAML implementation has similar performance with Leopard but our ProtoNet-BERG results are much better than theirs (see Appendix D.1 for more detailed comparison).
Although we observe that ProtoNet-BERT has better performance and faster convergence rates during training and validation, it is outperformed by ProtoNet-BN which has orders of magnitude fewer parameters to learn (fewer than 0.5% of the number of BERT's parameters). We hypothesize this because ProtoNet-BERT is more vulnerable to overfitting on the meta-training tasks.
7

---

See Appendix D.2 for the details of implementation, model size and training efficiency. We also com- pare with other fine-tuning approaches, like further fine-tuning a trained ProtoNet. See Appendix G.2 for the results.
Table I: Results diverse few-shot text classification tasks. Results marked with * are from [5]. For Leopard, we reuse the their results on the first seven tasks and report the results of our implementation on the last three newly added tasks.
Model
BERT
MT-BERTS
Leopard
PN-BERT
PN-BN
GradZTask
8
Mean
Std
Mean
Std
Mean
Std
Mean
Std
Mean
Std
Mean
Sid
sirline
42.76
13.50
12.26
54.95
11.81
12.73
65.33
7.95
disaster
55.73
10.29
50.61
8.33
51.45
4.25
54.01
2.50
53.48
4.76
55.43
5.89
emotion
9.20
3.22
9.84
2.14
11.71
2.16
11.69
1.37
12.52
1.32
12.76
1.35
political Lundianee
$1.89
51.53
52,60
52.77
51.88
51.28
political his
54.57
5.02
54.66
3.74
60.49
6,66
58.26
10:42
61.72
5.65
58,74
9.43
political, message
15.64
15.69
20.98
21.13
rating kitchen
34.76
11.30
36.77
10.62
50.21
9.63
58.47
11:12
55.99
9.85
57.09
9.74
hutfpust 10
11.8
1.41
14.97
1.69
16.81
2.52
21.36
2.7
28.99
3.93
46,29
52,51
2.68
yelp
36.95
2.98
42.84
2.06
42.64
2.93
3.55
36.72
40.52
5.45
42.76
44.11
airline
38.00
17.06
49.81
10.86
61.44
3.90
69. 14
4.84
69.37
2.46
72.04
2.58
56.31
55.96
53.85
57.49
emotion
8.21
2.12
11.21
2.11
12,90
1.63
13.10
2.64
13.87
1.82
13.99
1.90
political_audienee Lundience
$2.80
2.72
54.34
2.88
54.31
3.95
55.17
4.28
53.08
6.08
52.60
5.55
political_aa
56.15
54.79
4.19
61.74
63.22
65.36
64.06
8
political, message
13.38
1.74
15.24
2,81
18.02
2.32
20.40
1.12
21.64
1.72
21.31
1.16
rating kitchen
34.49
47.98
53.72
$7.08
56.27
58.35
hulfpust_I 10
12.73
2.23
16.52
1.48
19.03
2.18
21.12
1.69
saigs
52.74
selp
38.31
3.52
44.7
1.68
43,83
2,45
43.66
1.65
Average
38.96
4.11
42.60
3.46
44.90
3.52
46.18
3.36
airline
58.01
57.25
9,90
62.15
71.06
69.83
72.30
diseuttrr
64.52
8:93
60.70
6.05
61.32
2.89
55.30 30
2.68
57.38
5.25
59.63
3.11
13.43
12.75
13.38
12.81
14.11
13.72
political_ aundiece
$8.45
4.98
55.14
4.57
57.71
3.52
56.16
2.81
57.23
2.73
55.46
3.34
political_
60.96
3.26
16
political_no message
20.67
3.89
19.20
2.20
18.07
21.36
0.56
24.00
1.39
22.22
1.20
kitchen kitchen
47.94
8.28
53.79
9.47
$7.00
3.69
61.00
9.17
59.45
8.33
61.72
6.38
hurfpostt_1
13.78
17.74
21.43
23.57
snips
24.49
1.62
31.84
3.08
54.64
2.06
59.47
1.91
yelp
45.5
44.78
44,87
Average
41.27
3.23
43.67
3.14
46.82
2.78
47.68
2.35
Analysis and Ablations
6.1 Same/different task classification and task embedding visualization
Capturing task nature the prerequisite for task-specific adaptation conditioned on task embeddings. We evaluate the ability of gradients task representations through toy experiment and visualization. The toy experiment is a binary classification task about predicting whether two few-shot datasets are sampled from the same task not. We name this task as "same/different task classification." For each pair of few-shot datasets, we calculate the ProtoNet loss and its gradients using the base model on the two datasets, respectively. We then feed the gradients the two datasets into a single linear layer, respectively. Prediction is given by the cosine similarity between the two representations after linear transformation and the binary cross entropy loss used for training.
We train the linear model on few-shot dataset pairs sampled from our meta-training datasets, and test it dataset pairs sampled from meta-testing datasets. Figure 2 shows the AUC curves the testing set. Although the model is simple, it can predict same/different labels reasonably well on unseen tasks during training. With more shots, the gradient becomes more reliable, thus the model achieves better performance, resulting an AUC score of 0.84 with 16 shots.
The results of the same/different task classification experiment confirm that gradients can be used as features to capture task nature and distinguish between tasks. further evaluate the quality of task embeddings learned by our model, we visualize the learned task embeddings in 2D space. We find the task embeddings form good quality clusters according to task nature at higher layers. More details can be found in Appendix E.

---

shots
shots
16 shots
1.0
1.0
1.0
Thue Nostiv
0.6
Yrue Nosiive
0.0
wesiive
0.6
2.4
2.4
9.4
AJC 0.70
AUC 0.76
AUC
0.84
2.0
0.2
0.4
U.
0.8
3.0
0.2
0.4
U.
0.8
0.2
0.4
U.0.
C.
Palse Posttive Rote
Palse Posttive Rate
1aise Posttive Rate
Figure 2: Results the same/different experiments. Each column shows the results of classifying pairs of dataset with a certain number of shots.
6.2 Ablation Study
Table 2: Ablation results.
Model
Mcan Acc.
We conduct ablation studies to justify our decision choices. We report the average accuracy on meta- testing datasets of different model variants in Ta- ble 2. Results show that our approach achieves the best average accuracy among all model variants, resulting in average accuracy of 45.99%. The full results are included in Appendix F.
Grad2Task w/ Gradients
45.99
ProtoNet Longer Training Grad2Task w/ X Grad2Task w/ X&Y
45.10
45.66
45.16
Grad2Task Adapt All
44.57
Grad2Task w/ Pretrained TaskEmb
45.68
Hypernetwork
44.79
Does the adaptation really help? Starting from the base model trained after the first stage, we com- pare our approach with just training the base model using the ProtoNet loss for the same number of steps. Shown as "ProtoNet Longer Training" in Table 2, the performance of this approach is worse than our proposed approach. This justifies that we can do better by adapting the base model than training it for more steps.
How are gradients compared with other task representations? We compare approach with other CNP-based approaches with different methods of task embedding. Similar to Requeima et al. the average embeddings of the training sequences representations for task, and keep the other model architecture unchanged. We also consider using the embeddings of both input sequences and labels for task representation. We treat labels as normal text instead of discrete values and encode them using the BERT model. The label embedding and average input embeddings are concatenated the task representation. Shown as "Grad2Task w/ X" and "Grad2Task w/ X& Y", respectively, in Table 2, both of these two approach underperform our gradient-based method while "Grad2Task w/ X" performs closely with method. Note that can also combine those methods for task representation, which we leave for future work.
[37],
Adapt all just "CCLS]"? The only difference between this model and ours is whether apply the generated adaptation parameters only on the hidden representation of the "[CLS]" token or on all tokens in each sequence. Shown as "Grad2Task Adapt All" in Table 2, this model performs worse, resulting in 44.57% on average.
Generating model parameters adaptation parameters? We also experimented with hyper- network [20]-based approach. Conditioned on the task representation, we use a neural network (hypernetwork) to output the parameters of bottleneck adapters directly, instead of freezing the bottleneck adapters and only generating scaling/shifting parameters, as in our proposed model. The model with hypernetwork has higher flexibility for task adaptation, but the size of the hypernework large, as it maps high-dimensional gradient information to high-dimensional model parameters, which brings challenges for optimization and suceptibility for overfitting. As shown in Table 2, it achieves accuracy of 44.79% on average and is worse than our proposed approach.
Pretraining task embedding? For this model, we first pretrain the task embedding model for the same/different tasks, then freeze and combine with other components following our proposed approach. We observe that it has close but slightly worse performance (45.68%) than Grad2Task with- out task embedding pretraining. We hypothesize the task embedding trained with the same/different tasks is not good enough. We expect this model will benefit from pretraining the task embedding

---

model on a set of datasets with higher diversity and training the task embedding model using more advanced metric-learning approaches, which we leave for future work.
Discussion and Conclusion
In this paper, we propose novel model-based meta-learning approach for few-shot text classification and show the feasibility of using gradient information for task conditioning, Our approach is explicitly designed for learning from diverse text classification tasks how adapt base model conditioned on different tasks. We use pretrained BERT with bottleneck adapters as the base model and modulate the adapters conditioned on task representations based on gradient information.
Our work is an inaugural exploration of using gradient- based task representations for meta-learning. It has several limitations. First, the way significantly increases the number of parameters to train. For future work, we will explore more efficient ways for FIM calculation and better ways to handle high-dimensional gradient information. Second, we only focus on text sequence classification tasks in this paper. There are various types of NLP tasks to explore, such as question answering and sequence labeling tasks, which present many distinct challenges. Also, we only consider transferring between text classification tasks. As shown in Vu al. [50], positive transfer can happen among different types of NLP tasks. Thus, intend to explore meta-learning or transfer learning approaches to learning from different types of tasks.
neural networks to encode nigh-dimensional data
Ethical Concerns. Our work advances the field of few-shot text classification. While not having any direct negative societal impact, one can imagine that few-shot text classifier is abused by malicious user. For example, classifying trendy keywords on the internet naturally falls into the application area of our method as example phrases containing the new keyword can be limited in number. Toxic language is known to be a pervasive problem in the Internet [3], and a malicious user could use good few-shot text classifier to further perpetuate these problems (e.g., by classifying toxic text as benign and fooling the end user). On the other hand, the same system can be used to promote fairness.
Acknowledgments and Disclosure of Funding
We would like to thank reviewers and ACs for constructive feedback and discussion. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute (www. vectorinstitute aa/#partners). JW supported by the RBC Graduate Fellowship. FR and MB are CIFAR AI Chairs.
References
[1] Yelp open dataset. https::/www.yeet wttppppa..mmaaaateeeeeeeeee (Accessed on 05/20/2021).
[2] A. Achille, M. Lam, R. Tewari, A. Ravichandran, Maji, C. Fowlkes, S. Soatto, and
P. Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6430-6439, 2019.
[3] R. Adragna, E. Creager, D. Madras, and R. Zemel. Fairness and robustness in invariant learning: case study in toxicity classification. arXiv preprint arXiv:20/ / I.06485 2020.
[4] S. Baik, S. Hong, and K. Lee. Learning to forget for meta-learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2379-2388 2020.
[5] T. Bansal, R. Jha, and A. McCallum. Learning to few-shot learn across diverse natural language classification tasks. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5108-5123, 2020.
[6] T. Bansal, R. Jha, T. Munkhdalai, and A. McCallum. Self-supervised meta-learning for few-shot natural language classification tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 522-534, 2020.
10

---

[7] Y. Bao, M. Wu, S. Chang, and R. Barzilay. Few-shot text classification with distributional signatures. In International Conference on Learning Representations, 2019.
[8] J. Bingel and A. Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume 2, Short Pupers, pages 164-169, 2017.
[9] J. Blitzer, M. Dredze, and F. Pereira. Biographies, Bollywood, boom-boxed and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: : anthology/P07-110
/nte://wwwnw /wwp://wwaaa..c aclweb org/
[10] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 662--62,, 2015.
[11] R. Bowman, G. Angeli, C. Potts, and C. D. Manning. large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pages 632 642, Lisbon, Portugal, Sept. 2015, Association for Computational Linguisties. doi: 10. 1/D15- URL https acwweb. org/ anthology/D16-100 1075
[12] A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, T. Gisselbrecht, Caltagirone, T. Lavril, M. Primet, and Dureau. Snips voice platform: an embedded spoken language understanding system for privateby-desipe voice interfaces. CoRR, abs//1005.1a 10190, 2018. URL http: //wpvv.rxvvvv//////////e//.aaa orgooogg/aa.. 1805. 10190.
[13] A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, T. Gissel- brecht, F. Caltagirone, T. Lavril, et al. Snips voice platform: embedded spoken language understanding system for private-by-dett voice interfaces. arXiv preprint arXiv: 1805. 10190, 2018.
[14] I. Dagan, O. Glickman, and Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer, 2005.
[15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[16] W. B. Dolan and C. Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Puraphrasing (/WP2005), 2005.
[17] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1 1126-1135 1135. PMLR, 2017.
[18] M. Garnelo, D. Rosenbaum, C. Maddison, Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional neural processes. In International Conference Machine Learning, pages 1704-1713. PMLR, 2018.
[19] R. Geng, B. Li, Y. Li, X. Zhu, P. Jian, and J. Sun. Induction networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 3895-3904, 2019.
[20] D. Ha, A. Dai, and Q Le. Hypernetorks.. arXiv preprint 1609:v1609.09100 2016.
[21] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efffcient transfer learning for NLP. International Conference on Machine Learning, pages 2790-2799, PMLR, 2019.
[22] S. Iyer, N. Dandekar, and K. Csernai. First quora dataset release: Question pairs, 2017.

---

[23] E. Kerineč, A. Søgaard, and C. Braud. When does deep multi-task learning work for loosely related document classification tasks? In Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 1-8. Association for Computational Linguistics, 2018.
[24] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[25] Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint
arXiv:1907.11692, 2019.
[26] R.
Misra. News category dataset.
news -at--aattrorraaeeeeeeeeeeee--- (Accessed on 05/20/2021).
[27] R. Misra. News category dataset. 2018.
[28] T. Munkhdalai and H. Yu. Meta networks. In International Conference on Machine Learning. pages 2554-2563. PMLR, 2017.
[29] C. Nguyen, T. Hassner, M. Seeger, and C. Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference Machine Learning, pages 7294-7305. PMLR, 2020.
[30] A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018.
[31] B. N. Oreshkin, P. Rodriguez, and A. Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 719-729, 2018.
[32] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings the AAAI Conference Artificial Intelligence, volume 32, 2018.
[33] J. Phang. T. Févry, and S. R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: I. 1.01088, 2018.
[34] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre-training, 2018.
[35] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
[36] Ravi and Larochelle. Optimization as a model for few-shot learning. 2016.
[37] Requeima, J. Gordon, J. Bronskill, S. Nowozin, and R. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems 32 (NIPS 2019).
[38] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. International Conference on Learning Representations, 2018.
[39] A. Santoro. S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory- augmene- neural networks. In International conference on machine learning, pages 1842- 1850. PMLR, 2016.
[40] Schick and H. Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. arXiv preprint arXiv:2001.07677 2020.
[41] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 4080- 4090, 2017.
12

---

[42] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631- 1642, Seattle, Washington, USA, Oct. 2013. Association for Computational Linguuttioo URL. attps://ver aclweb.......................... org/ httpgggnnttllgyo/eeeaeeoo DD1//111311111111
[43] J. Song, Y. Chen, J. Ye, X. Wang, C. Shen, F. Mao, and M. Song. Depara: Deep attribution graph for deep knowledge transferability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3922-39922 -930, 2020.
[44] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 190011999-998 2014. URL BtLhttt::/ jmwr http://mmmmrromgpeeeeeeeeeeeeeee otg/papers/ wtteerss/vv///// rr15/sriaasttva44aa html.
[45] Sung, Y. Yang, L. Zhang, T. Xiang. P. H. Torr, and T. M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018.
[46] A. Talmor and Berant. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. arXiv preprint arXiv: 1345, 2019.
[47] A. T. Tran, C. V. Nguyen, and T. Hassner. Transferability and hardness of supervised classifi- cation tasks. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 1395-1405, 2019.
[48] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[49] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3637-3645, 2016.
[50] T. Vu, T. Wang. T. Munkhdalai, A. Sordoni, A. Trischler, A. Mattarella-Mick, S. Maji, and M. lyyer. Exploring and predicting transferability across nlp tasks. arXiv preprint arXiv:2005.000 00770, 2020.
[51] A. Wang, J. Hula, P. Xia, R. Pappagari, R. T. McCoy, R. Patel, N. Kim, I. Tenney, Y. Huang, K. Yu, al. Can you tell me how to get past Sesame Street? Sentence-level pretraining beyond language modeling. arXiv preprint 18.2. 2018.
[52] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Booman. GLUE: A multi- task benchmark and analysis platform for natural language understanding. arXiv preprint 1804 07461, 2018.
[53] A. Williams, N. Nangia, and S. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18--1 110111 httpst:s: / //ww//wwwooooo/ aclweb. org/ anthology /Naeey///11-
[54] D. Yogatama, C. d. M. d' A'ttumee J. Connor, T. Kocisky, M. Chrzanowski, L. Kong, A. Lazari- dou, W. Ling, L. Yu, C. Dyer, al. Learning and evaluating general linguistic intelligence. 2019.
arXiv preprint 1901.
[55] M. Yu, X. Guo, J. Yi, S. Chang, S. Potdar, Y. Cheng, G. Tesauro, H. Wang, and B. Zhou. Diverse few-shot text classification with multiple metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1206- 1215, 2018
[56] A. R. Zamir, A. Sax, W. Shen, J. Guibas, Malik, and Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722, 2018
13


Reviews:
[{"review_id": "q2vBh4o2Hj", "reviewer": "Reviewer_WPsh", "summary": "The paper considers the problem of few-shot text classification. It builds on a bunch of prior works on meta-learning [1,2,3], Film adapters [4], task embeddings [5] and tries to combine them for the task of few-shot text classification. The key idea seems to be to use adapters in between transformer layers, compute task embeddings (using fisher Information matrix) for a task and use it to update Film parameters that are used to update the model. The approach is compared with a prior work [1] and prototypical networks on a subset of few-shot classification tasks selected from [1].\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the paper’s proposal to use task-embeddings to further modulate the encoder for few-shot learning. This is not a new concept and has been widely explored in meta-learning literature. The main contribution of the paper seems to be in its specific application to text classification models and the use of the task embeddings to generate the Film parameters. \n\nOverall, I found the paper very hard to follow, there are numerous typos and the exposition made it really difficult to extract the key contribution, the motivations for the specific choices, and the key take-aways. The paper seems a lot like an amalgamation of many related concepts, which is not a bad thing but in the current version of the paper there is no clear motivation for the various choices. To provide just one instance, if the main motivation is to use task embeddings then there are many straight-forward ways to incorporate them in existing models, such as in the input to the encoder or the classifier, and carefully considering/evaluating such choices would have been useful in motivating the approach. Another choice made by the authors is to use only the values in FIM corresponding to the adapters. Why is it sufficient to only use these, is there any analysis on the quality of the task embedding when restricting to only the adapters?\nSimilarly, the motivation for using task embeddings to generate Film adapters is not clear. For instance, one can use Film adapters with regular fine-tuning on the few-shot set to learn “task-specific” Film adapters. Note that this baseline is not present in the paper. So how is including the task embedding (which also uses gradient information) helpful over fine-tuning? Motivation as well as empirical results in this direction are lacking. \n\nMoreover, there are several concerns regarding experiments. \n- First, as mentioned above, there are relevant baselines that need to be provided to understand the value of the proposed approach. These also include: ProtoNet with fine-tuning, ProtoNet with adapters fine-tuning, ProtoNet-BN with only Film fine-tuning. Currently, the main baselines for the paper are ProtoNet and ProtoNet-BN, none of which use gradient information like the proposed method. These baselines will help demonstrate the utility of task embeddings over fine-tuning.\n- The authors use datasets from [1]. However, I see there are 17 datasets used in that paper and the authors picked 7 of them. I am not convinced that seq-length is an appropriate reason to drop 10 datasets, for instance, the authors can run the model with a larger sequence length during evaluation. Moreover, there are decisions like only using a subset of classes in one of the datasets (Huffpost), and why this is done is also not clear. As such, due to the use of non-standard datasets, I am not sure how to interpret the results.\n- Ablations are carried on all the downstream tasks used for final evaluation, which makes me think the results are overfit to these tasks. I didn’t find these particularly insightful either. For example, ablations on these choices would have been more helpful: using/not-using adapters or Film adapters, hidden-size of adapters, fine-tuning adapters vs using task embeddings, alternatives to incorporate task embeddings.\n- Some hyper-parameter choices are missing. For example in section 4.2, what are the hyper-parameters of the RNN d()? Similarly, what is the architecture and hyper-parameters of the MLPs a(.) in adaptation network?\n- There is no analysis on what the task embeddings are capturing and why they are useful here. This would have helped motivate the paper quite a bit. \n\nThese concerns together make me hesitant about the value of the task-embedding approach presented here. The authors should consider a thorough re-writing of the paper to provide motivation and to explain their approach clearly, and consider the suggested baselines and improvements to experiments explained above.\n\n[1] Learning to few-shot learn across diverse natural language classification tasks.  \n[2] Meta-dataset: A dataset of datasets for learning to learn from few examples.  \n[3] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes.  \n[4] Film: Visual reasoning with a 382 general conditioning layer.  \n[5] Task2vec: Task embedding for meta-learning.\n\n\nUpdate post rebuttal\n\nThe authors provided a lengthy response with additional experiments. These partly address some concerns but raise a few more. \nWhen using more tasks and relevant baselines as reported here, I can see that the margins of improvements are significantly less than what is reported in the paper. For ex, FT-PN-BN is competitive with the proposed method. Some of the new results are also very puzzling. Like why does prototypical networks after fine-tuning becomes much worse than without fine-tuning. Another related problem I hadn't noticed in the first review was the difference between prototypical and other fine-tuning baselines, this difference seems too large, contrary to previous literature, and warrants more exploration. Comparing some results in the rebuttal and in Table 1, it seems that most of the improvements for the proposed approach are coming from the 3 new introduced datasets. These results are also very surprising. For instance, what is the property of SNIPS dataset on which Leopard and PN suddenly perform at half the accuracy of Grad2Task (21, 28 vs 52)!\n\nI have still raised my score in favor weak accept since the rebuttal did address some concerns. Authors should incorporate these results and the presentation improvements in the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is an appropriate discussion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "oyi2nxParmq", "reviewer": "Reviewer_D6xc", "summary": "The paper presents a novel method for few-shot text classification tasks. The method is built on top of the CNAP  The key contribution is to effectively encode the gradients of networks as the task representation so that the meta-learning stage can use such task representations to improve the few-shot generalization ability. The two key technical contributions are a task embedding network (an RNN-based layer-by-layer gradient encoder via Fisher Information Matrix) and an adaptation network for auto-regressively adapting parameters from the bottom to the top layer.  The implementation is based on BERT-like LMs with Adapter bottleneck modules, which are parameter efficient to compute. \n\nThe authors compared the proposed method with a few meta-learning methods on a diverse set of tasks, and show that their method outperforms other methods for most of the time. The empirical analysis and ablations also provide insights for understanding the proposed method. \n\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall, the work is a nice contribution towards using the gradient information to encode a task instead of only using the input/output text. The gradient information implies more about the network behavior which is closer to the features of a task. The paper is mostly well written with nice figures and tables for delivering important information. I do not have major concerns about the paper.  There are a few minor notes and suggestions below:\n\n- I think in the problem definition, Line 98, it would be better to explicitly define the learning stages and a shared text classification model. The meta-learning stage has the access to the source tasks, and the few-shot generalization stage does not. And there is a shared model after the meta-learning stage. \n\n- In the preamble of Section 4, it would be better to illustrate more the motivation of the design choice and define the \"adaptation parameters\" clearly. It is vague in the current presentation and makes people wonder about the underlying motivations of the design. \n\n- It would be better to visualize the task representation in some way and show the task-to-task distances so that we can inspect if the closer tasks can locate near each other. \n\n- Line 202, BERT-base only allows 128 tokens? I don't think so. It should be 512 if I understand correctly. Please double-check it and correct me if I was wrong about this. \n\n- A few typos: Line 19: \"a intermediate\"; Line 201 BERT-BERT. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "CbZUQemL2s", "reviewer": "Reviewer_RV6z", "summary": "This paper proposed Grad2Task, a model for few-shot text classification tasks. Grad2Task first computes gradients when fine-tuning the base model on a support set; then the task embedding network decodes task representations using these gradients; finally the task representations go through MLP layers and output scaling and shifting parameters, which are plugged into the base model for task-specific adaptation. \n\nThe paper provides comprehensive evaluation on 10 tasks. Grad2Task outperforms strong baselines (Leopard, ProtoNet-BERT) in most settings. Analysis with a toy dataset shows the task representations are informative. Ablation study justified several design choices in Grad2Task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Novelty. The work is the first to explore gradient-based task representations for meta-learning. In addition to combining recent efforts (e.g., Task2Vec, meta-learning), the authors also reduces parameter size using lightweight design (e.g., adapters, shifting and scaling parameters).\n- Detailed ablation study that explains the design choices in detail. The failure cases also brings interesting observations. Future work may stem from these observations.\n- Strong performance when compared to strong baselines.\n\nWeakness:\n- Notation are sometimes confusing. (1) In Eq.3 the function d is overloaded. The function outputs the task embedding in the first part. It outputs RNN hidden representation in the second part. (2) In line 7-8 in Algorithm 1, e_j is a new variable that seems to be newly-introduced. Please also provide hints for the reader to link e_j with p(y|x_j), e.g., refer the reader to Eq. 1.\n\nQuestions:\n- Is there any specific reason that RNN is selected as the task embedding network? \n- How long is the training process? How large is the final model? Some comparison in model size and training efficiency may be included.\n\nMinor Suggestions:\n- Eq. 7. Is the first stage done on the support set or the query set or both?\n- Line 274: “w/ XY”, Table 2 “w/ X&Y”. Please keep consistent.\n- I believe the right side of Fig 1 is actually zooming in the part of [e_l^{task}] —> [Bottleneck Adapter]. Please consider highlighting this information to make the figure more accessible. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide detailed discussion on Grad2Task's limitations in Sec 7. I listed some of my concerns and suggestions in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "2qrxW_qK-Vb", "reviewer": "Reviewer_Ywoc", "summary": "  A new method for few-shot text classification", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Th paper has a good analytical presentation and a set of thorough experiments that demonstrate it is better on the selected datasets. It is not clear how many labeled instances are required, and at which threshold the method degrades. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not convincing; issues raised have to do with text classification n in general, b not so much with the paper", "review_text": "", "extra_scores": {"ethical_concerns": "none ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.33"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed that that the method of encoding gradients to obtain a task embedding and using FILM to adapt the text-classification network to that task. Many of the initial concerns raised primarily by reviewer WPsh were addressed in the rebuttal through additional experiments, leading the reviewer to raise their score by 1. In general, the design choices and claims are well supported by ablation studies and comparisons against various baselines on various datasets. The writing is also structured and of decent quality. That said, the authors need to address the new "puzzling" issues raised by reviewer WPsh's updated review on the fine-tuning and comparison to other methods, especially as they cast doubt on the validity of the reported numbers.
Decision: Accept (Poster)