Few-shot Text Classification with Distributional Signatures
Authors: Yujia Bao
Menghua Wu
Shiyu Chang
Regina Barzilay
Venue:  (2020)
Keywords: text classification
meta learning
few shot learning

Abstract: In this paper, we explore meta-learning for few-shot text classification. Meta-learning has shown strong performance in computer vision, where low-level patterns are transferable across learning tasks. However, directly applying this approach to text is challenging--lexical features highly informative for one task may be insignificant for another. Thus, rather than learning solely from words, our model also leverages their distributional signatures, which encode pertinent word occurrence patterns. Our model is trained within a meta-learning framework to map these signatures into attention scores, which are then used to weight the lexical representations of words. We demonstrate that our model consistently outperforms prototypical networks learned on lexical knowledge (Snell et al., 2017) in both few-shot text classification and relation classification by a significant margin across six benchmark datasets (20.0% on average in 1-shot classification).

Published as a conference paper at ICLR 2020
FEW-SHOT TEXT CLASSIFICATION WITH DISTRIBUTIONAL SIGNATURES
Yujia B Menghua Wu Shiyu Chang', Regina Barzilay TComputer Science and Artificial Intelligence Lab, MIT MMITIIBM Watson Al LAB, IBM Research {yujia, rmwu, reguraegea mit edu, {shiyu. chang) com
ABSTRACT
In this paper, we explore meta-learning for few-shot text classification. Meta- learning has shown strong performance in computer vision, where low-level pat- terns are transferable across learning tasks. However, directly applying this ap- proach to text is mhallenging-lexieee features highly informative for one task may be insignificant for another. Thus, rather than learning solely from words, our model also leverages their distributional signatures, which encode pertinent word occurrence patterns. Our model is trained within meta weea-learning framework to map these signatures into attention scores, which are then used weight the lexi- cal representations of words. We demonstrate that our model consistently outper- forms prototypical networks learned on lexical knowledge (Snell et al.) (017) both few-shot text classification and relation classification by a significant margin across six benchmark datasets (20.0% on average 1-shot classification) [
INTRODUCTION
In computer vision, meta-learning has emerged as a promising methodology for learning in a low- resource regime. Specifically, the goal is to enable algorithm expand to new classes for which only a few training instances are available. These models learn to generalize in these low-resource conditions by recreating such training episodes from the data available. Even in the most extreme low-resource scenario-a single training example per class-this approach yields 99.6% accuracy on the character recognition task (Sung et al. 2018).
Given this strong empirical performance, we are interested in employing meta-lcarning frameworks in NLP. The challenge, however, is the degree of transferability of the underlying representation learned across different classes. In computer vision, low-level patterns (such as edges) and their corresponding representations can be shared across tasks. However, the situation different for language data where most tasks operate at the lexical level. Words that are highly informative for one task may not be relevant for other tasks. Consider, for example, the corpus HuffPost headlines, categorized into 41 classes. D shows that words highly salient for one class do not play significant role in classifying others. Not surprisingly, when meta-learning is applied directly on lexical inputs, its performance drops below simple nearest neighbor classifier. The inability of a traditional meta-learner to zoom-in on important features is further illustrated in Figure 2 when considering the target class fifty (lifestyle for middle-aged), the standard prototypical network (Snell et al. (017) attends to uninformative words like "date," while downplaying highly predictive words such "grandma."
In this paper we demonstrate that despite these variations, we can effectively transfer representations across classes and thereby enable learning low-resource regime. Instead of directly considering words, our method utilizes their distributional signatures, characteristics of the underlying word dis- tributions, which exhibit consistent hehaviour across classification tasks. Within the meta-learning framework, these signatures enable us to transfer attention across tasks, which can consequently be used weight the lexical representations of words. One broadly used example of such distributional
"Equal contribution.
Our code is availahle at http:/::///tttbbb.oooo////a/////

---

Published as a conference paper at ICLR 2020
classes
Prototypical network (seen class) this gorgeous grandma proves beauty has expiration date
welbors
tramp
Prototypical network (unseen class)
pe
this gorgeous grandma proves beauty has no expiration date
Seep
cancer
withis
Our model (unseen class)
uailer
words
ravel
this gorgeous grandma proves beauty has no expiration date
Figure 1: Different classes exhibit different word distributions in HuffPost headlines. We compute the local mutual information (LMI) (Evert 2003) between words and classes. For each class, we include its top LMI-ranked words. Darker colors indicate higher LMI.
Visualization of word importance example from class fifty in HuftPost headlines. Top: fifty is seen during meta-training ical network (Snell et al.) (017)) finds important words. Middle: fifty unavailable during meta- training; fails to generalize. Bottom: Our model identifies key words for unseen classes.
Figure 2:
prototyp-
signatures is tf-idf weighting, which explicitly specifies word importance in terms of its frequency document collection, and its skewness within a specific document.
Building on this idea, we would like to learn to utilize distributional signatures in the context of cross-class transfer. In addition to word frequency. we assess word importance with respect to a specific class. This latter relation cannot be reliably estimated of the target class due the scarcity of labeled data. However, we can obtain noisy estimate of this indicator by utilizing the few provided training examples for the target class, and then further refine this approximation within the meta-learning framework. We note that while the representational power distributional signatures weaker than that of their lexical counterparts, meta knowledge built distributional signatures are better able to generalize.
Our model consists of components. The first is atfention generator, which translates distri- butional signatures into attention scores that reflect word importance for classification. Informed by the attention generator's output, our second component, a ridge regressor, quickly learns make predictions after seeing only few training examples. The attention generator is shared across all episodes, while the ridge regressor trained from scratch for each individual episode. The latter's prediction loss provides supervision for the attention generator. Theoretically, we show that the attention generator robust word-substitution perturbations.
We evaluate our model on five standard text classification datasets (Lang 1995 Lewis et al. 2004 Lewis 1997, He & McAuley 2016) Misra 2018) and one relation classification dataset (Han al,, 2018). Experimental results demonstrate that model delivers significant performance gains over all baselines. For instance, our model outperforms prototypical networks by 20.6% on average in one-shot text classification and 17.3% in one-shot relation classification. In addition, both qualitative and quantitative analyses confirm that our model generates high-quality attention for unseen classes.
2 RELATED WORK
Meta-learning Meta-learning has been shown to be highly effective in computer vision, where low- level features are transferable across classes. Existing approaches include learning metric space
over input features (Koch) 2015) Vinyals et al. 60 al. 2017 Sung et al. 2018), developing
a prior over the optimization procedure (Ravi & Larochelle| 2016 Finn et al. 2017 Nichol &
Schulman] 2018 Antoniou et al. 2018), and exploiting the relations between classes Garcia & Bruna 2017). 7). These methods have been adapted with some success to specific applications in NLP.
including machine translation (Gu et al. 2018). text classification (Yu al., 2018) Geng et al.
2019 Guo et al., 2018)) Jiing et al. 2019) and relation classification (Han et al. 2018). Such models
primarily build meta- meta--knoweddeee lexical representations.
2

---

Published as a conference paper at ICLR 2020
training data: examples from J
training data is used source pool (our extensiion)
poffics
tecc
trcvel
mecio
sparts
bacuty
taste
poltics
tech
travel
medio
sports
bacutyy
fcete
1. sample N clusses
2. retrieve examples from remainnin classes
testing data )ricst
culture
sample
money
poltiis
medio
sports
tech
Eeenutyy
N classes
examples from which the support and query are drown
examples from which the nupport and query are drown
source pool four extension)
tcate
money
a) meta-training
meta-testing
Figure 3: Episode generation. a) Meta-training: First, sample N classes from ) Then, sample the support set and the query set from the N classes. We use examples from the remaining classes to form the source pool. b) Meta-testing: Select N new classes from y and sample the support set and the query set from these N classes. We use all examples from )uimin to form the source pool.
However, as our experiments show, there exist innate differences in transferable knowledge between image data and language data, and lexicon-awand meta-learners fail to generalize on standard multi- class classification datasets.
In this work, we observe that even though salient features in text may not be transferable, their dis- tributional behaviors are alike. Thus, focus on learning the connection between word importance and distributional signatures. As a result, our model can reliably identify important features from novel classes.
Transfer learning Our work is also closely related to transfer learning: we assume access to large number of labeled examples from source classes, and we would like to identify word importance for the target classification task. Current approaches transfer knowledge from the source to the target by either fine-tuning pre-trained encoder (Howard Ruder 2018 Peters et al. 2018 Radford et al.
2018
Bertinetto et al. 2019). or multi-task learning with shared encoder (Collobert & Weston Liu et al., 2015 Luong et al.] 2015 Strubell et al., 2018). Recently, Bao et al. (2018) also
2008
successfully transferred task-specific attention through human rationales.
In contrast to these methods, where the transfer mechanism is pre-designed, we learn to transfer utilize distributional statistics to
based the performance of downstream tasks. Specifically, transfer attention across tasks. We note that while Wei et al. (2017) and [Sn et al. (2018) also learn transfer mechanisms for image recognition, their methods do not directly apply NLP.
3 BACKGROUND
In this section, we first summarize the standard meta-learning framework for few-shot classifica- tion and describe the terminology (Vinyals et al. 2016). Next, we introduce our extensions to the framework. Figure and graapicallly illustrate our framework.
Problem statement Suppose we are given labeled examples from a set of classes trrain Our goal is develop a model that acquires knowledge from these training data, that we can make predictions over new (but related) classes, for which we only have a few annotations. These new classes belong to set of classes Mest disjoint from traiin
Meta-training In meta-learning, we emulate the above testing scenario during meta- meta-training so our model learns to quickly learn from few annotations. To create single training episode, we first sample N classes from ytrain For each of these N classes, we sample K examples as our training data and T. examples as our testing data. We update our model based on loss over these testing data. Figure 4 shows example of an episode. We repeat this procedure to increase the number of training episodes, each of which is constructed over its own set of classes. In literature, the training data of one episode is commonly denoted as the support set, while the corresponding testing data is known as the query set. Given the support set, we refer to the task of making predictions over the query set as N W-way Khohot classification.
Meta-testing After we have finished meta- traininr we apply the same episode-based mechanism to test whether model can indeed adapt quickly to new classes. To create a testing episode,
3

---

Published as a conference paper at ICLR 2020
religion
baauty
games
impact
tech
religion
beauty
games
:
rolgion
beaufy
games
media
Impact
tach
support
query
examples)
examples)
source pool
traditional episode
b) our extension
Figure 4: Single episode with N = 3, K = -, 2. Rectangles denote input examples. The text inside corresponds their labels. An episode contains support set, query set, and source pool.
first sample N new classes from ) Then we sample the support set and the query set from the N classes. We evaluate the average performance on the query set across all testing episodes.
Our extension We observe that even though all examples from ) are accessible throughout meta-training, the standard meta-learning framework (Vinyals et al., (016)) only learns from small subsets of these data per training episode. In contrast, our model leverages distributional statistics over all training examples for more robust inference. To accommodate this adjustment, we augment each episode with a source pool (Figure 4p). During meta- eeta-training (Figure 31), this source pool includes all examples from training classes not selected for the particular episode. During meta- testing (Figure 3), this source pool includes all training examples.
4 METHOD
Overview Our goal is to improve few-shot classification performance by learning high-quality attention from the distributional signatures of the inputs. Given particular episode, we extract relevant statistics from the source pool and the support set. Since these statistics only roughly approximate word importance for classification, we utilize an attention generator to translate them into high-quality attention that operates over words. This generated attention provides guidance for the downstream predictor, ridge regressor, to quickly learn from a few labeled examples
We note that the attention generator is optimized over all training episodes, while the ridge regressor is trained from scratch for each episode. Figure 5 illusttaaees the two components of our model.
Attention generator: This module generates class-specific attention by combining the distributional statistics of the source pool and the support set (Figure (5h). The generated attention provides the ridge regressor an inductive bias on the word importance. We train this module based on feedback from the ridge regressor (Section 4.1).
Ridge regressor: For each episode, this module constructs lexical representations using the attention derived from distributional signatures (Figure(5)), The goal of this module is make predictions over the query set, after learning from the support set (Figure 5 and tt). Its prediction loss is end-to-end differentiable with respect to the attention generator which leads efficient training (Section 4.2).
In our theoretical analysis, we show that the attention generator's outputs are invariant to word- substitution perturbations (Section 4.3).
4.1 ATTENTION GENERATOR
The goal of the attention generator is to assess word importance from the distributional signatures of each input example. There are many choices distributional signatures. Among them, we focus on functions of unigram statistics, which provably robust to word-substitution perturbations (Section 4.3). We utilize the large source pool to inform the model of general word importance and leverage the small support will be used later to construct the input representation for downstream classification.
estimate class-specific word importance. The generated attention
Note this generated attention can be applied to any downstream predictor [Le et al. 2019
Snell et al.
2017
Details from these experiments may be found in the Appendix A.9
4

---

Published as a conference paper at ICLR 2020
b) Ridge regressor: Constructing representstions
d) Ridge regressor: inference on the query set
feal)
(())
Y2
CCE
media
4Q
YQ
ganen
Query
counce pool
MILSTT
beauly han no intton rase
beauty
o6-)
4
W
pupport
Yss
Poppont
Attention Generator
Ridge regressor: training support
Figure 5: Illustration of our model for an episode with N 3, K 1, I. = 2. The attention generator translates the distributional signatures from the source pool and the support set into an attention for each input example (5a). The ridge regressor utilizes the generated attention to weight the lexical representations (5b). It then learns from the support set (5c) and makes predictions over the query set (5d).
well-documented in literature that words which appear frequently are unlikely to be informa- tive (Sparck Jones 1972). Thus, we would like to downweigh frequent words and upweight rare words. To measure general word importance, we select an established approach by /Arora et al. 2016):
8 s(x;)
& + (x4)
where & = 10-3, is the 7thh word of input example and P (1;) is the unigram likelihood of a;; over the source pool.
On the other hand, words that are discriminative in the support set are likely to be discriminative in the query set. Thus, we define the following statistic to reflect class specifie word importance:
t(r;)= = H(P (y xz)))
where the conditional likelihood P ( m;) is estimated over the support set using a regularized linear classifie and H(-)) the entropy operator. We note that t(-) measures the uncertainty of the class label given the word. Thus, words that exhibit skewed distribution will be highly weighted.
Directly applying these statistics may not result in good performance for two reasons: the two statistics contain complementary information, and it is unclear how to combine them; and 2) these statistics are noisy approximations to word importance for classification. To bridge this gap, we concatenate these signatures and employ bi-directional LSTM (Hochreiter & Schmidhuber 1997) to fuse the information across input: = biLSTM((s(z); ((x]]. Finally, we use dot-product attention to predict the attention score of word xx:
exp (v hx)
0r E, exp (xTh))
where h; is the output of the biL.STM at position i and v is learnable vector.
4.2 RIDGE REGRESSOR
Informed by the attention generator, the ridge regressor quickly learns to make predictions after see- ing a few examples. First, for each example a given episode, we construct lexical representation that focuses on important words, as indicated by attention scores. Next, given these lexical repre- sentations, we train the ridge regressor on the support set from scratch. Finally, we make predictions over the query set and use the loss to teach the attention generator to produce better attention.
See Appendix A.1 for details.

---

Published as a conference paper at ICLR 2020
Constructing representations Given that different words exhibit varying levels of importance towards classification, we construct lexical representations that favor pertinent words. Specifically, define the representation of example
<(z) = ====>r aj (xed
where (-ba()) is a pre-trained embedding function that maps a word into R
Training from the support set Given an N-way K-shot classification task, let 4 E RNNKEE be the representation of the support set, obtained from c(-), and Ys E IR- be the one-hot labels. We adopt ridge regression (Bertinetto et al. 2019) to fit the labeled support set for the following reasons: 1) ridge regression admits closed form solution that enables end-to-end differentiation through the model, and 2) with proper regularization ridge regression reduces over-fitting on the small support set. Specifically, we minimize regularized squared loss
RRR (W) $ Ys 1 [W]]
(5)
over the weight matrix W R Here ||-|| denotes the Frobenius norm, and À 0 controls the conditioning of the learned transformation W. The closed-form solution can be obtained as
W $ ( ( -Ys
where / is the identity matrix.
Inference the query set Let S denote the representation of the query Although optimized for regression objective in Eq equation [ the learned transformation has been shown to work well in few-shot classification after calibration step (Bertinetto al., 2019.
Ye abo +
where a R+ and b E are meta parameerssssss learned through meta-training. Finally, we apply a softmax over Yo to obtain the predicted probabilities Pq. Note that this calibration only adjusts the temperature and scale of the softmax; its mode remains unchanged. During meta-training, compute the cross-entropy loss _ between Pq and the labels the query set. Since both 4s and & depend on ((-), C provides supervision for the attention generator.
4.3 THRORETICAL ANALYSIS
Working with distributional signatures brings certified robustmess against input perturbations, For- mally, let (P, P.S. 2) be a single episode. where P is the source pool, S is the support set and Q is the query set. For any input text x e S J Q, the attention generator produces word-level importance given the support set and the source pool:
AttGen( S)P)..
Now consider word-substitution perturbation that replaces word by (w)) If we arbitrar- ily swap words, we may encounter nonsensical outputs, as important words may be substituted by common words, like "a" "the." Thus, consider perturbations that preserve the unigram prob- abilities of words, estimated over the source pool: P (w) = (a(w)) ) for all w E V. this way, important words (for one class) are mapped to similarly important words (perhaps for another class). We & and Q denote the support and query sets after applying o word by word.
Theorem 1. Assume V + V satisfies P (w) = =P (a(w)) for all w. If a is bijection, then for any inpul text S U Q and its perturbution ã E S U Q, the outputs of the attention generator are the same:
AnGen( S, AttGen(ā S,P).
The idea behind this proof is to show that general word importance and class saass-c word impor- tance are invariant to such perturbations. Details may be found in Appendix (A.2
Alternatively, one can view this susstiiutittuinn perturbation as a change to the input distribution.
Let P (x y) be the conditional probability of the input æ given the class label y. Assuming that
"We view this as mapping from the vocabulary V itself.

---

Published as a conference paper at ICLR 2020
is bijection, the conditional after perturbation can be expressed P( y) = P v), where o- (xx) the result of applying a every word in Intuitively, Theorem 1 tells us that if a word w is a discriminative feature for the original classification task, then after the change of the input distribution, the word a (w)) should be discriminative feature for the task. This property makes sense since word-importance for classification should depends only on the relative differences between classes. fact, the theorem holds when the input to the attention generator is any function of unigram counts. We also note that the classification performance on the query set can be different, the pre- trtinrriee embedding function fera(-) generally not invariant under such perturbations.
EXPERIMENTAL SETUP
5.1 DATASETS
We evaluate our approach on five text classification datasets and one relation classification dataset (See Appendix A.4 for more details.)
20 Newsgroups is comprised of informal discourse from discussion forums Documents are organized under 20 topics.
Lang 1995
RCV1 is collection of Reuters newswire articles from 1996 1997 (Lewis et al. 2004 These articles are written in formal speech and labeled with a set of topic codes. We consider 71 second- level topics total class set and discard articles that belong more than one class.
Reuters-21578 collection of shorter Reuters articles from 1987 (Lewis) 1997). We use the standard ApteMod version of the dataset. We discard articles with more than one label and consider 31 classes that have at least 20 articles.
Amazon product data contains customer reviews from 24 product categories (He & McAuley 2016). Our goal is to classify reviews into their respective product categories. Since the original dataset notoriously large (142.8 million reviews), select more tractable subset by sampling 1000 reviews from each category.
HuffPost headlines consists of news headlines published on HuflPost between 2012 and 2018 (Mirra (018). These headlines split among 41 classes. They are shorter and less grammatical than formal sentences.
FewRel is a relation classification dataset developed for few-shot learning (Han et al. 2018). Each example a single sentence, annotated with head entity, tail entity, and their relation. The goal is to predict the correct relation between the head and tail. The public dataset contains 80 relation types.
5.2 BASELINES
We compare our model (denoted as OUR) different combinations of representations and learning algorithms. Details of the baselines may be found in Appendix A.1I
We evaluate three representations. AYG represents each example as the mean of its embeddings. IDF represents each example as the weighted average of its word embeddings, with weights given by inverse document frequency over all training examples. CNN applies convolution over the input words and obtains the representation by max-over-time pooling (Kim 2014).
Representations
Learning algorithms In addition to the ridge regressor (RR) (Bertinetto et al. 2019), we evaluate two standard supervised learning algorithms and two meta-learning algorithms. NN is 1-nearest- neighbor classifier under Euclidean distance. FT pre-trains classifier over all training examples, then finetunes the network using the support set (Chen et al. 2019). MAML meta-learns a prior model parameters, that the model can quickly adapt to new classes (Finn al. 2017). Prototypical network (PROTO) meta-learns a metric space for few-shot classification by minimizing
All processed datasets along with their splits are publicly available.
7

---

Published as a conference paper at ICLR 2020
Method
20 News
Amazon
HuffPost
RCVI
Reuters
FewRel
Average
Rep.
Alg.
shot 5 shot
shot 5 shot
shot 5 shot
shot
shot
shot 5 shot
shot
shot
shot 5 shot
AVG
NN
33.9
45.8
46.7
60.3
31.4
41.5
43.7
60.8
56.5
80.5
47.5
60.6
43.3
58.2
IDF
NN
38.8
51.9
51.4
67.1
31.5
42.3
41.9
58.2
57.8
82.9
46.8
60.6
44.7
60.5
CNN
FT
33.0
47.1
45.7
32.4
44.1
40.3
62.3
70.9
91.0
54.0
71.1
46.0
63.2
36.2
45.4
37.2
51.9
35.6
41.6
28.4
31.2
59.5
68.1
44.0
16.5
40.1
47.4
IDF
PROTO
37.8
46.5
41.9
59.2
34.8
50.2
32.1
35.6
61.0
72.1
43.0
61.9
41.8
CNN
PROTO
29.6
35.0
34.0
44.4
33.4
44.2
28.4
29.3
65.2
74.3
49.7
65.1
40.1
48.7
AVG
MAML
33.7
43.9
39.3
47.2
36.1
49.6
39.9
50.6
54.6
62.5
43.8
57.8
41.2
51.9
IDF
MAML
37.2
48.6
43.6
62.4
38.9
53.7
42.5
54.1
61.5
72.0
48.2
65.8
45.3
59.4
CNN
MAML
28.9
36.7
35.3
43.7
34.1
45.8
39.0
51.1
66.6
85.0
51.7
66.9
42.6
54.9
AVG
RR
37.6
57.2
50.2
72.7
36.3
54.8
48.1
72.6
63.4
90.0
53.2
72.2
48.1
69.9
IDF
RR
44.8
64.3
60.2
79.7
37.6
59.5
48.6
72.8
69.1
93.0
55.6
75.3
52.6
74.1
32.2
44.3
37.3
53.8
37.3
49.9
41.8
59.4
71.4
87.9
56.8
71.8
46.1
61.2
OUR
52.1
68.3
62.6
81.1
43.0
63.5
54.1
75.3
81.8
96.0
67.1
83.5
60.1
78.0
OUR w/o I(-)
50.1
67.5
61.7
80.5
42.0
60.8
51.5
75.1
76.7
93.7
66.9
83.2
58.1
76.8
OUR w/o s(.)
41.9
60.7
51.1
75.3
40.1
60.2
48.5
72.8
78.1
94.8
65.8
82.6
54.2
74.4
OUR w/o biLSTM
50.3
66.9
61.9
80.9
42.2
63.0
51.8
74.1
77.2
95.4
66.4
82.9
58.3
77.2
OUR W EBD
39.7
57.5
56.5
76.3
40.6
58.6
48.6
71.5
81.7
95.8
61.5
80.9
54.8
73.4
Table 1: Results of 5-way 1-shot and 5-way 5-shot classification on six datasets. The bottom four rows present our ablation study. For complete results with standard deviations see Table X and 9 Appendix A.12
in
the Euclidean distance between the centroid of each class and its constituent examples (Snell et al. 2017
5.3 IMPLEMENTATION DETAILS
We use pre-trained fastText embeddings (Joulin et al., 2016) for our model and all baselines. For sentence-level datasets (HuffPost, FewRel), also experiment with pre-trained BERT embed- dings (Devlin et al. 2018) using HuggingFace's codebase (Wolf et al. 2019). For relation classifica- tion Rewlel), augment the input of our attention generator with positional embeddings (Zhang et al. 2017
In the attention generator, output (Srivastava et al. 2014). In the ridge regressor, we optimize meta- meta-parameters À and a in the log space maintain the positivity constraint. All parameters are optimized using Adam with learning rate of 0.001 (Kingma & Ba (2114).
bILSTM with 50 hidden units and apply dropout of 0.1 on the
sample 100 training episodes per epoch. We apply early stopping when the validation loss fails to improve for 20 epochs. We evaluate test performance based on 1000 testing episodes and report the average accuracy over different random seeds.
During meta-training,
RESULTS
We evaluated our model in both 5-way 1-shot and 5-way 5-shot settings. These results are reported in Table Our model consistently achieves the best performance across all datasets. On average, our model improves 5-way 1-shot accuracy by 7.5% and 5-way 5-shot accuracy by 3.9%, against the best baseline for each dataset. When comparing against CNN+PROTO, our model improves by 20.0% on average in 1-shot classification. The empirical results clearly demonstrate that meta-learners privy to lexical information consistently fail, while our model able to generalize past class-spccific vocabulary. Furthermore, Figure 6 illustrates that lexicon-aware meta-learner (CNN+PROTO) is able to overfit the training data faster than our model, but our model more readily generalizes to unseen classes.
We also provide the positional embeddings to the baseline
8

---

Published as a conference paper at ICLR 2020
seen classes
unseen classes
seen classes
unseen classes
1.0
1.0
awe accuracy
avg. accuracy
0.8
Myw Amim hy
0.6
0.6
2000
4000
0
4000 # tasks seen during training
2000
# tasks seen during training
(a) CNN+PROTO
(b) OUR
Figure 6: Learning curve of CNN+PROTO (left) v.s. OUR (right) on Reuters dataset. We plot average 5-way I-shot accuracy over 50 episodes sampled from seen classes (blue) and unseen classes (red). While OUR has weaker representational power, generalizes better to unseen classes.
Method
HuflPost
FewRel
Rep.
Alg.
1 shot
5 shot
shot
shot
AVG
NN
23.23 20.20
34.22 10.28
52.37 20.27
64.85 10.25
IDF
NN
33.49 #0.18
45.71 +0.17
48.65 #0.18
62.35 10.18
CNN
37.30
51.56 +1.28
61.10
80.04 40.69
AVG
PROTO
34.21 10.56
49.77 11.90
50.27 20.98
66.24 4:2.01
PROTO
36.06 20.84
54.58 +0.90
48.23 =0.68
67.82 +0.72
CNN
PROTO
36.17 11.00
50.55 #0.96
57.08 =5.52
75.01 #2.21
AVG
MAML
38.58 1.56
55.32 1:4.2
47.18 =3.44
64.50 1:2.72
IDF
MAMI.
34.22 40.74
56.50 +1.50
50.06 22.88
68.43 42.50
CNN
MAMI
38.39 1166
53.86 20.76
47.68 1.66
71.56 #4.75
AVG
RR
25.34 +0.14
51.52 #:0.44
55.65 #0.27
73.91 4:0.77
IDF
RR
40.38 10.11
61.72 11.03
54.48 20.26
73.48 1:072
CNN
RR
41.37 20.54
53.10 10.76
65.65
25.70
78.65 1:4.24
OUR
42.12 20.15
62.97 10.67
70.08 20.56
88.07 1:0.27
Table 2: 5-way 1-shot and 5-way 5-shot classification on HuffPost and FewRel using BERT.
Ablation study We perform ablation studies on the attention generator. These results are shown at the bottom of Table [ We observe that both statisties s(-) and t(-) contribute the performance, though the former has larger impact. We also note that instead of computing word importance independently for each word using multilayer perceptron (OUR w/o biLSTM), fusing information across the input with an biLSTM improves performance slightly.
Finally, we observe that restricting meta-knowledge distributional signatures is essential to per- formance: when both lexical word embeddings and distributional signatures are fed to the attention generator (OUR EBD), the performance drops consistently.
Contextualized representations For sentence-level datasets (FewRel, HuflPost), we also experi- ment with contextualized representations, given by BERT (Devlin et al. 2018). These results are shown in Table While BERT significantly improves classification performance on FewRel, we ob- serve no performance boost on HuffPost. We postulate that this discrepancy arises because relation classification is highly contextual, while news classification is mostly keyword-based.
We visualize our inrention-weigheee representation ((x) in Figure Compared to directly using general word importance s(x) or class-specifit word importance t(z). ). our method produces better separation, which enables effective learning from few examples. This highlights that the power of our approach lies not in the distributional signatures themselves, but rather in the represen- tations learned on top of them.
Analysis
Figure visualizes the model's input and output the same query example in two testing episodes. The example belongs to the class jobs in the Reuters dataset. First, we observe that our model generates meaningful attention from noisy distributional signatures. Furthermore, the generated

---

Published as a conference paper at ICLR 2020
THE VE
(a) s(-)
(b) t(-)
(c) OUR
Figure 7: PCA visualization of the input representation for the query set of testing episode (N = 5, K 5, L 500) sampled from 20 Newsgroups. Weighted averages of word embeddings given by (a) s(). 2 (b) u(-), and (c) the attention generator meta-trained on a disjoint set of training classes.
Fine-grained classification
Coarse-graines classification
s(x)
finnish Bnemployment december last year compared with 6.8 pet in november and pet in december 1985, the central statistical office said
6.7 pet in finnish inemployment
6.7 pet in december last year compared with 6.8 pet november and pet in december 1985, the
central statistical office said
t(x) finnish mnmmplomennt was 6.7 pet in finnish inemployment was 6.7 pec in
december last year compared with 6.8 pet in november and 6.1 pet in december 1985,, the central statistical office said
december last year compared with 6.8 pel in
november and 6.1 pet in december 1985 the
central statistical office said
OURS finnish unemployment
6.7 pet in december last year compared with 6.8 pet in november and 6.1 pet in december 1985, the central statistical office said
6.7 pet in december last year compared with 6.8 pet november and 6.1 pet in december 1985,, the central statistical office said
finnish unemployment
Figure 8: Attention weights generated by our model are specific to task. We visualize our model's inputs s(z) (top), t(x) (middle), and output (bottom) for one query example from class jobs in Reuters dataset. Word "statistical" downweighed for jobs when compared other economics classes (left), but it becomes important when considering dissimilar classes (right). Fine-grained classes: jobs, retail, industrial production index, wholesale production index, consumer production index. Coarse-grained classes: jobs, cocoa, aluminum, copper, reserves.
attention is task-specific: in the depicted example, if the episode contains other economics-related classes, the word "statistical" downweighed by our model. Conversely, "statistical" upweighted when we compare jobs to other distant classes.
7 CONCLUSION
In this paper, propose novel meta-learning approach that capitalizes on the connection between word importance and distributional signatures to improve few-shot classification. Specifically, we learn attention generator that translates distributional statistics into high-quality attention. This generated attention then provides guidance for fast adaptation to new classification tasks. Exper- imental results both text and relation classification validate that model identifies important words for new classes. The effectiveness our approach demonstrates the promise of meta learning with distributional signatures.
ACKNOWLEDGMENTS
We thank the MIT NLP group and the reviewers for their helpful discussion and comments. This work is supported by MIT-IBM Watson AI Lab and Facebook Research. Any opinions, findings, conclusions, recommendations expressed in this paper those of the authors, and do neces- sarily reflect the views of the funding organizations.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint 1XX:::180000900. 2018.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. simple but tough-to-beat baseline for sentence embeddings. 2016.
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 1903-1913, 2018.
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable clomed-form solvers. In International Conference on Learning Representations, 2019, URL /ettps:/coperzee
net forum di--ml122hctttttttt..
Christopher M Bishop. Pattern recognition and machine learning. Springer Science+ Business Media, 2006.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In International Conference on Learning Representations, 2019.
URL https:://ppeeeeeiiiiieeee net do
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167. ACM, 2008
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Stefan Evert. The statistics of word cooccurrences: word pairs and collocations. 2005.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arX1v:1711.04043, 2017.
Ruiying Geng, Binhua Li, Yonghin Li, Yuxiao Ye, Ping Jian, and Jian Sun. Few-shot text classifica- tion with induction network. arXiv preprint /90iv: /0482, 2019.
Jiatao Gu, Yong Wang. Yun Chen, Victor OK Li, and Kyunghyun Cho. Meta-learning for low- resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3622-3631, 2018.
Jiang Guo, Darsh Shah, and Regina Barzilay. Multi-source domain adaptation with mixture of experts. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4694 2018.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-arr evaluation. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 4803-4809, 2018
Ruining He and Julian McAuley, Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings the 25th international conference world wide web, pp. 507-517. International World Wide Web Conferences Steering Committee, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: 1801 006000646, 2018
11

---

Published as a conference paper at ICLR 2020
Xiang Jiang, Mohammad Havaei, Gabriel Chartrand, Hassan Chouaib, Thomas Vincent, Andrew Jesson, Nicolas Chapados, and Stan Matwin. Attentive agmostti meta-leaming for few-shot text classification, 2019. URLpatps: httpp:://pen.eevee....... iot.://eeee..ntttttoooomoooaaaee
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint 2016.
Yoon Kim. Convolutional neural networks for sentence classification.
arXiv preprint
arXiv:1408.58 5882, 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1012.6442.68000 2014.
Gregory Koch. Siamese neural networks for one-shot image recognition. 2015.
Ken Lang. Newsweeder: learning to filter netnews. In Proceedings of the Twelfth International Conference International Conference Machine Learning, pp. 331-339, Morgan Kaufmann Publishers Inc., 1995.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10657-10665, 2019.
David D. Lewis. Reuters-21577 text categorization test collection, distribution 1.0. 1997.
David D Lewis. Yiming Yang, Tony G Rose, and Fan Li. Revl: A new benchmark collection for text categorization research. Journal of machine learning research, 3(Apr):361-397,2 2004.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03110. 2017.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Yi Wang. Representa- tion learning using multi-task deep neural networks for semantic classification and information retrieval. 2015.
Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence sequence learning. arXiv preprint 1rXhv:1511.06114 2015.
Rishabh Misra. News category dataset, 06 2018.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 2, 2018.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
Alec Radford, Kanthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
2017.
Richard Socher, Danqi Chen, Christopher Manning, and Andrew Ng. Reasoning with neural
tensor networks for knowledge base completion. In Advances in neural information processing systems, pp. 926-934, 2013.
Karen Sparek Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):1 11-21, 1972.
12

---

Published as a conference paper at ICLR 2020
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-199) 1958, 2014.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. Linguistically- informed self-attention for semantic role labeling. arXiv preprint arXiv:/8044 180,008999 2018.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. arXiv preprint /812. (..2..299,, 2018.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, pp. 3630-3638, 2016.
Ying Wei, Yu Zhang, and Qiang Yang. Learning to transfer. arXiv preprint arXiv 10X.::/70.005550 7708.05629 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface' trans- formers: State-of-the-ar art natural language processing. ArXiv, abs/1910.03771, 2019.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning the good, the bad and the ugly, 2017.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. Diverse few-shot text classification with multiple metrics. arXiv preprint 1rXiv:180..05513, 2018.
Ningyu Zhang, Zhanlin Sun, Shumin Deng, Jiaoyan Chen, and Huajun Chen. Improving few- shot text classification via pretrained language representations. arXiv preprint 1908:::900.00000 2019.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Processing, pp. 35-45, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.11655//1/DD7710044 URL https: www ac web org www.orggg..gggg/.n...ooooo -
13

---

Published as a conference paper at ICLR 2020
A SUPPLEMENTAL MATERIAL
A. REGULARIZED LINEAR CLASSIFIER
Given N-way K-shot classification task, the goal of the regularized linear classifier is approx- imate task-specific word importance using the support set.
Let (. Fr} be an input example, and let feea(-) be an embedding function that maps each word 2; into R We compute the representation of z by the average of its embeddings:
(xx)
T Σ (xt)
Since the support set only contains few examples, we adopt a simple linear classifier to reduce overfitting:
p =
www.mao(wWWiiiiiiiooooooooo (x)))
where W E R is the weight matrix to learn. We minimize the cross entropy loss between the prediction and the ground truth label while penalizing the Frobenius norm of W. We stop training once the gradient norm less than 0.1. Finally, given a word I;, we estimate its conditional probability via softmax (W (2:)).
Time efficiency Since the support set very small (less than 25 examples) and the loss function is strongly convex, this linear classifier converges very fast in practice Note that for larger problems, we can speed up computation by formulating this procedure as a regression problem and solving for its closed-form solution (as Section 4.2)
A.2 PROOF OF THEOREM 1
Theorem 1. Assume V + V satisfies P (w) = =P (a(w)) for all w. If a is bijection, then for any input text E S and its perturbation è E E Q, the outputs of the attention generator the same:
AnGen(æ S, P) = AttGen(E S,P).
Proof. It suffices to show that the general word importance s(.) and the class-specific word im- portance t(-) are the same under the perturbation Φ For the former, since e preserves unigram probability estimated over the source pool, we have
E
E
s (xx = E + (xx) & ++ = s(o(x)))
To prove the latter, we need show that the conditional probability P (y x;)) estimated over the support set is also invariant under &. Let #(x; A S) denote the occurrences of the word in examples with class label y over the support set S. Using maximum likelihood estimation, we have
#(o; Ay 5) = #(o(a:) Ay Ay $) Ply Fi. () = Ey Ey # () Ey ((( Av ())
Ply (z))),
where the second equality is derived from the fact that is bijection.
A.3 LEARNING PROCEDURE
Algorithm [ contains the pseudo code for our learning procedure. We apply early stopping when validation loss does not improve for 20 epochs.
A.4 DATASETS
To reliably test our model's ability to generalize across classes, consider two data splitting mech- anisms in our experiments: 1) easy split: we randomly permute all classes and split them into train/val/test; 2) hard splir: select train/val/test based on the class hierarchy such that train
less than second single GeForce GTX TITAN X
14

---

Published as a conference paper at ICLR 2020
Algorithm Meta-training procedure. Nirain = | the number of training classes. N Niain the number of classes of each few-shot task. K, L are the number of support and query examples per target class, respectively. To case notation, we use SAMPLE(S, .N) to denote subset S with N elements, chosen uniformly at random. We use D(V) S D to denote the set of all elements (Ia, yi) for which y: E y.
Input:
Training set = [(11, y1), 1, (12, v2)....) where each yi E)tamm {1, Nanin] Hyperparameters: parameters Φ; cnefficients for losses X, a,
repeat
y SAMPLE()*Mh) .N)
sample target classes b create support pool sample support set and query set
De +- D()nain ())
T 7 - 0,0
y do
T - T SAMMLLE(D((())) K)
+ SAMPLEE (y)) \7 L)
Generate attention scores using T T3,7%, D
Eq. Eq
Construct attention-weigee
representations 4s, Po using o0-) Compute closed-form solution W of ridge regression from X, 4g. and support labels Ys.
Di Eq
Given W, b, compute cross entropy loss _ on the query set do. YQ
Update parameters (attention generator and a, using C
until stopping criterion is met
Dataset
tokens / example
vocab size # examples cls
train cls
# val cls
test cls
Newsgroups
32137
941
RCVI
372
7304
20
37
10
Reuters
168
2234
20
15
5
11
Amazon
140
17062
1000
10
5
9
HuffPost
11
8218
900
20
FewRel
24
16045
700
65
5
Table 3: Dataset statistics. See Appendix A.4 for details.
classes are distant to val and test. We applied the easy split to one sentence-level dataset (Huff- Post) and one document-level dataset (R)uters-21578), Hard split is used for the other four datasets (details below). This setting tests the generalization capacity of the algorithm, following Xian et al. (2017).
20 Newsgroups Each class in 20 Newsgroups belongs to one of six top-level categories, which roughly correspond to computers, recreation, science, polities, religion, and for-sale. partition the set of labels that top-level category spans two splits. Train contains "sci" and "ree," val contains "comp," and test contains all other labels.
The Amazon dataset does not come with predefined top-level categories. To generate hard split, we first apply spectral clustering to classes based on their word distributions. Then we select train/val/test from different clusters.
Amazon
RCV1 We apply the same approach as above.
FewRel While FewRel does not provide higher-level categories, observe that most relations occur between named entities of similar types. Thus, we extract the named entity type of the head and tail for each example using a pretrained spaCy parser For each class, determine the most common head and tail entity types. Test contains all classes for which the most common head entity type WORK
PORK_OF_AA CF-.oOAAARR. ART. Train and validation were arbitrarily split contain the remaining relations.
A.5 OTHER BASELINES
We also consider two other baselines: induction network (Geng et al. 2019) and P-MAML (Zhang et al. 2019). Implementation details may be found in Appendix A.11
Sitts::/spacy/ io/
15

---

Published as a conference paper at ICLR 2020
20 News
Amazon
HuffPstt
RCVI
Reuters
FewRel
Method
1 shot
shot
shot 5 shot
shot
5 shot
shot
5 shot 1 shot
shot
shot 5 shot
INDUCTION NET 27.6
32.1
30.6
37.1
34.9
44.0
32.3
37.3
58.3
66.9
50.4
56.1
P-MAML
47.1
58.4
31.0
51.3
53.0
72.9
OUR
52.1 68.3
62.6
81.1
43.0
63.5
54.1
75.3
81.8
96.0
67.1
83.5
Table Comparison against Induction Net (Geng al., 2019) and P-MAML (Zhang et al. 2019 We run P-MAMI. on text classification datasets with shorter documents, for which it is feasible to finetune BERT (after WordPiece tokenization, documents from "longer" datasets exceed BERT"s max length of 512 tokens).
seen classes
unseen classes
scen classes
unscen classes
1.00
1.00
avg. accuracy
avg. accuracy
0.75
0.75
0.50
0.50
0.25
0.25
2000 # tasks seen during training (a) HuffPost
4000
0
2000
4000
# tasks seen during training
(b) Reuters
Learning curves of INDUCTION NET on Huffpost and Reuters. We plot average 5-way 5-shot accuracy over 50 episodes sampled from seen classes (blue) and unseen classes (red).
Figure 9:
Induction network encodes input examples using a biLSTM with self-attentive pool- ing (Lin al.] 2017). Based on the encoded representation, it then computes prototype for each class through dynamic routing over the support set (Sabour et al. 2017). Finally, it uses neural tensor layer (Socher al. 2013) to predict the relation between each query example and the class prototypes.
P-MAML combines pre-training with MAML. It first finetunes pre-trained BERT repre- sentation on the meta-training data using masked language modeling (Devlin al. 2018). Based on this finetuned representation, it trains first-order MAML Finn et al. (2017 to enable fast adaptation.
Table 4 shows that learning with distributional signatures (OUR) significantly outperforms the two baselines across all datasets. This is not surprising: both baselines build meta-knowledge on top of lexical representations, which may not generalize when the lexical distributions are vastly different between seen classes and unseen classes. Figure 9 shows the learning curve of INDUCTION NET. Similar to Figure [6] we see that INDUCTION NET overfits to meta train classes with poor general- ization meta-test classes. Figure 10 depicts the perplexity of BERT's masked language model objective, during finetuning for P-MAMI. Again, we see that the meta-train and meta-test classes exhibit vast lexical mismatch, as the perplexity improves only slightly on meta-test classes.
A.6 ANALYSIS LEARNED REPRESENTATION
To further understand the rationale behind performance boost, we provide more detailed anal- ysis on our model's empirical behavior.
Visualizing the embedding space We visualize our attention-weghted representation ((x) in 20 Newsgroups (Figure II] and HuffPost (Figure 12). We observe that our model produces bet- ter separation than the unweighted average AVG and directly using the distributional statistics, s(x) or t(x). For instance, in 20 Newsgroups, our model recognizes three clusters: relgionnni,, wen.religion..crisiinn
miikppooiiissmideass and (aalk-ollitics.mis talk, politioc..gnn..
16

---

Published as a conference paper at ICLR 2020
seen classes
unseen classes
seen classes
unseen classes
15
perplexity
perplexity
40
m
10
20
250
500 # steps
750
1000
0
250
500
750
1000
steps
(a) HuffPost
(b) Reuters
Figure 10: Perplexity during BERT language model finetuning in P-MAML. The lexical distributions mismatch significantly between meta-train and meta-test classes.
talk pol lice lics-gum gums
potticc.gue guns
talk relgiinn
religion mine
soc chrisinn
Soc ro wiggon.chrissa
(a) AVG
(b) s-)
poli.polli mile
talk polties
tolk golticc guns
talk pol lics puns telk relgiionmnne soc.reigooochhtttaa
talk malkeeeiioonn..oo mise
soc welgionchrsttion
(c) t(-)
Figure 11: PCA visualization of the input representation for a testing episode in 20 Newsgroups I. 500 (the query set has 500 examples per class). AVG: average word
with N embeddings. s(-): weighted average of word embeddings with weights given by s(.). t(-)) weighted average of word embeddings with weights given by t(-). OUR: weighted average of word embed- dings with weights given by the attention generator meta-trained on disjoint set of training classes.
Cosine similarity to oracle word importance We also quantitatively analyze the generated atten- tion in in 20 Newsgroups and HuffPost (Figure 13
To obtain a more reliable estimate of word importance, we train an oracle model over all examples from the N target classes. This oracle model uses biLSTM to encode each example from its word embeddings. then generates an attention score based on this encoding. In order to estimate the importance of individual unigram, this attention score to weight the "original" word
17

---

Published as a conference paper at ICLR 2020
science word
Word
(a) AVG
(-)
ktyl
cducadion
(c) t--)
(d) OUR
Figure 12: PCA visualization of the input representation for testing episode in HuffPost Headlines with N 5, K = 5, L 500. AVG: average word embeddings. 8(.): weighted average of word embeddings with weights given by s(-). t(-): weighted average of word embeddings with weights given by t(-). OUR: weighted average of word embeddings with weights given by the attention generator meta-trained on a disjoint set of training classes.
embeddings (not the output of the biLSTM). MLP with one hidden layer is used perform N-way classification from the attention-weighted representation.
Compared to both s(-) and t(-), the attention generated by our model is much closer to the oracle's s, which explains our model's 's large performance gains. Note that the attention generator does not see any examples from the target classes during meta-training.
Visualizing the generated attention Figure 14 visualizes the generated attention for testing episode HuffPost. We observe that our model identifies meaningful keywords from the sentence.
A.7 EFFECT OF SOFTMAX CALIBRATION
we use ridge regression with softmax calibration as our downstream predictor due to its efficiency and effectiveness. To study the effect of the softmax calibration, follow BBrtinetto et al. (2019) and compare ridge regression against direct optimization of binary logistic classifiers (LR) with Newton's method (five iterations). Since Newton's method admits closed-form solu- tion at each iteration, the final solution is also end-to-end differentiable. For each N-way K-shot episode, to adapt the binary logistic classifier for multi-class prediction, we train N binary (one-vs- all) classifiers using the support set. The output of the N binary classifiers combined to make the final predictions on the query set (Bishop) 2006 Chapter 4.1.2).
In Section 4.2
The results are shown in Table 5 Overall, RR and LR produce similar results (78.0 vs. 77.1 on 5- way 5-shot, 60.1 vs. 58.5 on 5-way 1-shot, respectively), though RR performs slightly better. When using LR the downstream predictor, learning with distributional signatures improves 5-way 1-shot accuracy by 6.1 % and 5-way 5-shot accuracy by 3.2% over the best baseline. Note that LR requires solving N binary classifications for each episode, which is less efficient than RR.
18

---

Published as a conference paper at ICLR 2020
0.4
0.8
0.74
0.7
0.32
av cosine similarity
0.3
avg cosine similarity
0.6
0.59
0.51
0.21
0.2
0.4
0.15
0.11
0.1
0.2
0
0
AVG s(-) t(-) OUR
s())
I(-)
OUR
Newsgroups
HuffPost Headlines
Figure 13: Average cosine similarity to the oracle word importance over the query set of a testing episode with N 5, N classes. Since examples in HuffPost Headlines are 30 times shorter, the cosine similarities are higher in this corpora. AVG: uniform distribution over the words. s(-): word importance estimated directly by s(.). t(-): word importance estimated directly by t(-). OUR: word importance estimated by the meta-learned attention generator.
500. This oracle is estimated using all labeled examples from the
class
input example
taste
you wo n't even miss the meat with these delicious vegetarian sandwiches
taste
these cookies are spot on copies of the oscars dresses
word news
prime minister saad hariri 's return to lehanon moment of truth new zealand just became the 11th country to send rocket into orbit word news
style
heyonce dressed like the queen she is at the grammys
style
tilda swinton that jacket dress
science
the world of science has a lot to look forward to in 2016
science
dione crosses saturn disk in spectacular new image the global search for education just imagine secretary hargreaves
education
education
thinking at harvard what is the future of learning
Visualization the attention generated by our model on 10 query examples from a 5-way Figure 14:
5-shot testing episode in Huffpost Headlines.
A.8 FINETUNING WORD EMBEDDINGS DURING META-TRAINING
In Table I. we fix word embeddings during meta-trane meta-training for all experiments. Table studies the effect of finetuning word embeddings during meta-training. We observe that when the word embed- dings are finetuned, performance drops for nearly all models.
To intuitively understand this behavior, we compare the meta-train vocabulary with the meta-test vo- cabulary. On Amazon, 15095 of 28591 (52.8%) tokens the meta-train vocabulary are not present in meta-test, and 21049 of 34545 (60.9%) tokens in the meta-test vocabulary not present in meta- train. On Reuters, 3604 of 6372 (56.6%) meta-tran train tokens are not in meta-test, while 2481 of 5249 (47.2%) meta-test tokens not in meta-test. Due to this lexical mismatch, finetuning will destroy the original geometry the pretrained word embeddings, leading to poor generalization on unseen tasks.
19

---

Published as a conference paper at ICLR 2020
Method
20 News
Amazon
HuffPost
RCVI
Reuters
FewRel
Average
Rep.
Alg.
1 shot shot
shot 5 shot
shot 5 shot
shot 5 shot
shot 5 shot 1 shot
shot
shot 5 shot
AVG
RR
37.6
57.2
50.2
72.7
36.3
54.8
48.1
72.6
63.4
90.0
53.2
72.2
48.1
69.9
IDF
RR
44.8
64.3
60.2
79.7
37.6
59.5
48.6
72.8
69.1
93.0
55.6
75.3
52.6
74.1
CNN
RR
32.2
44.3
37.3
53.8
37.3
49.9
41.8
59.4
71.4
87.9
56.8
71.8
46.1
61.2
OUR
RR
52.1
68.3
62.6
81.1
43.0
63.5
54.1
75.3
81.8
96.0
67.1
83.5
60.1
78.0
AVG
LR
37.7
56.8
50.0
71.6
36.3
56.0
47.9
72.5
62.7
89.3
52.4
72.9
47.8
69.9
IDF
LR
46.1
64.2
60.5
79.9
38.7
59.3
47.3
72.5
68.1
93.0
55.5
74.9
52.7
74.0
CNN
LR
32.6
47.5
41.0
58.5
37.6
50.9
44.2
63.6
75.0
90.3
58.7
72.1
48.2
63.8
OUR
I.R
49.7
67.6
61.6
80.7
41.9
61.9
54.4
74.1
79.5
95.6
65.8
82.8
58.8
77.1
Table 5: Results of 5-way I-shot and 5-way 5-shot classification on six datasets using ridge regres- sion vs. logistic regression with Newton's 's method.
Method
5-way 5-shot Reuters
5-way 5-shot Amazon
Rep.
Alg.
Fix embedding Finetune embedding
Fix embedding Finetunc embedding
AVG
PROTO
68.15 +1.53
58.40 #2.84
51.99 #2.72
41.34 #1.61
IDF
PROTO
72.07
12.69
65.93 13.08
59.24 11110
48.37 12.41
CNN
PROTO
74.33 11.86
74.20 12.85
44.49 11.53
44.08 #2.21
AVG
MAML
62.46
1:0.88
50.59 1:1.8
47.22 #:199
30.95 1:108
IDF
MAML
71.96 1..48
68, 68.76 1:139
62.45 1:133
50,63 10.61
CNN
MAML
85.00 4:0.76
84.19 4:0.41
43.70
43.96 #1.19
AVG
RR
90.00 +:0.99
86.87 4:0.60
72.78 40.23
75,65 #0.51
IDF
RR
93.02
93.01 4:0.47
79.78 +0.28
78.73 +0.59
CNN
RR
87.93 +1.49
85.47 4-0.57
53.89 +1.04
52.63 4-2.02
OUR
RR
96.00 +0.27
95.98 4:039
81.16
#0.31
78.94
Table Fixed vs. finetuned word embeddings during meta-training. Since meta-train and meta-test classes exhibit different word distributions, fixing the word embedding yields better generalization. That is, we avoid overfitting to the meta-train vocabulary and destroying the geometry of the pre- trained embeddings.
A.9 DISTRIBUTIONAL SIGNATURES WITH OTHER CLASSIFIERS
In this paper, we demonstrate the advantage of learning meta-knowledge top of distributional signatures. We focus on a ridge regressor (Bertinetto et al.) 2019) as our downstream classifier due its simplicity and effectiveness. However, the idea of learning with distributional signatures is not limited to ridge regression; rather, we can combine distributional signatures with other learning methods such as prototypical networks (Snell al. 2017) and induction networks (Geng al. 2019).
Prototypical networks To augment prototypical network with features learned from distributional signatures, we can construct per-class prototypes based on the attention-weighte representation <(z) (Section 4.2). From Table 7 we see that learning with distributional signatures improves 5- way 1-shot accuracy by 9.9% and 5-way 5-shot accuracy by 16.7%, against the best prototypical network baseline for each dataset.
Induction networks We can also augment induction networks with meta-knowledge learned from distributional signatures. Specifically, we directly feed our aatention-weightee representation ((x) (Section 4.2) to the induction module. To avoid over-fitting
meta-train features, we replace the meta- learnee relation module (see [Geng et al. (2019) for details) by parameter-free nearest neighbour predictor, similar to that of prototypical networks. From Table 7 we again see that learning with distributional signatures improves 5-way 1-shot accuracy by 14.3% and 5-way 5-shot accuracy by 25.1% average.
These empirical results clearly demonstrate the benefit of our approach. However, both prototypical networks and induction networks perform consistently worse than the ridge regressor across all
20

---

Published as a conference paper at ICLR 2020
Method
20 News
Amazon
HuffPost
RCVI
Reuters
FewRel
Average
Rep.
Alg.
shot
shot
shot 5 shot
shot 5 shot
shot
shot
shot 5 shot
shot
shot
shot 5 shot
AVG
PROTO
36.2
45.4
37.2
51.9
35.6
41.6
28.4
31.2
59.5
68.1
44.0
46.5
40.1
47.4
IDF
PROTO
37.8
46.5
41.9
59.2
34.8
50.2
32.1
35.6
61.0
72.1
43.0
61.9
41.8
54.2
CNN
PROTO
35.0
34.0
33.4
44.2
28.4
29.3
65.2
74.3
65.1
40.1
48.7
OUR
PROTO
42.4
59.5
52.6
73.9
40.6
58.9
45.1
63.6
72.2
93.9
57.5
75.3
51.7
70.9
LSTM INDUCT 32.1
30.6
34.9
44.0
37.3
58.3
66.9
56.1
39.0
45.6
OUR
INDUCT 45.4 60.0 56.6
73.5
40.4
60.2
42.3
60.3
74.5
93.1
60.5
77.3
53.3
70.7
Table 7: Performance of prototypical networks and induction networks learned lexical infor- mation vs. distributional signatures (OUR). LSTM with INDUCT the original induction network architecture.
1.0
1.0
0.8
0.8
accuracy
accuracy
0.6
0.6
0.4
0.4
0.2
0.2
25
50
75
100
25
75
100
number training examples
number training examples
(a) 20Newsgroups
(b) Amazon
1.0
1.0
accuracy
accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0
25
50
75
100
25
50
75
100
number of training examples (c) FewRel
number of training examples (d) HuffPost
Figure 15: Learning curve for fully-supervised CNN classifier vs OUR. Blue indicates CNN accu- racy, with standard deviation shaded. Solid horizontal line is OUR'S 5-shot accuracy; dashed line I-shot accuracy. OUR's 5-shot performance competitive, especially when the total number of labeled examples is small.
experiments. Our hypothesis is that the ridge regressor enables task-specific finetuning based on the support set, while the prototypical network and induction network directly compute nearest neighbours from the meta-learned metric space.
A.10 FEW-SHOT LEARNING V.S. SUPERVISED LEARNING
To understand the extent of few-shot learning's 's utility, we compare our results with that of fully- supervised CNN classifier, increase the number of training points. From Figure 15 that our model's 5-shot accuracy is approximately equivalent to a sully-super classifier trained on 50 examples.
21

---

Published as a conference paper at ICLR 2020
A.11 IMPLEMENTATION DETAILS
We detail the implementations of our baselines here. All baseline code are available in our reposi- tory.
CNN For ID convolution, we use filter windows of 3, 4, 5 with 50 feature maps each. We applied ReLU after max-over-time pooling.
Prototypical Network Prototypical network meta-learns a multi-layer perceptron to transform the input representation into an embedding space that is suitable for few-shot classification. If the input representation is learnable (e.g., CNN), the parameters for the input representation are also updated using meta-trainin trattnt.aiiing. the experiments, we use a MLP with one hidden layer and ReLU activation. The dimensions of both the hidden layer and the output layer are 300. We apply dropout with rate 0.1 the hidden layer.
MAML MAML meta-learns an initialization such that the model can quickly adapt to new tasks after a few gradient steps. For prediction on the input representation, we use a MLP with one hidden layer of 300 ReLU units. We apply dropout with rate 0.1 to the hidden layer. During the MAML inner loop (adaptation stage), we perform ten updates with step size 0.1 (we empirically found this outperform one-step MAML). We backpropagate higher order gradients thoughout meta veta-traiining. During the MAML outerloop, we average the gradient across ten sampled tasks and use Adam with learning rate 100 to update the parameters.
Finetune Chen et al. (2019) recently showed that fine-tuning a properly pre-trained classifier can achieve competitive performance when compared with the state-of-the-arr meta-learning. Following their work, we explicitly reduce the intra-class variation during the pre-training stage. Similar to MAML, we use MLP with one hidden layer (300 ReLU units) to make predictions from the input representation (e.g.. CNN). During finetuning stage, we re-train the MLP from scratch and fine-tune the learnable parameters of the input representation. We stop fine-tuning once the gradient norm is A
less than 10
Induction network Induction network consists of three modules: encoder module, induction mod- ule and relation module. The encoder module uses a biLSTM with self-attention to obtain a fix- length representation of each input example. The induction module performs dynamic routing to compute the class-specific prototype. The relation module uses neural tensor layer to predict the relation association between each query example and the prototypes. Following Geng et al. (2019), we set the hidden state size of biLSTM to 256 (128 for each direction) and the attention dimension to 64. The iteration number in dynamic routing is set to 3. The dimension of the neural tensor layer set to 100.
P-MAML P-MAML (Zhang et al. 2019) has two phases: masked language model pretraining (De- vlin al. 2018) and MAML (Finn al. 2017). For pretraining, used Hugging Face's language model finetuning code with default hyperparameters and BERT's pretrained base-uncased model. We applied early stopping when the validation perplexity failed to decrease for 2 epochs. After pre- training, we add a softmax layer on top of the representation of the [CLS] token (Devlin et al. al.] (0188) make predictions. During the MAML inner loop (adaptation stage), we perform ten updates with step size 10-3, Following Zhang et al. (2019), we do not consider higher order gradients. During the MAML outerloop, we average the gradient across ten sampled tasks and use Adam with learning rate 10-5 to update the parameters.
22

---

Published as a conference paper at ICLR 2020
A.12 RESULTS
This section contains experimental results with standard deviations.
Rep. Alg.
20 News
Amazon
HuffPost
RCVI
Reuters
FewRel
AVG NN
33.95 10.33
46.76
10.14
31.45
10.18
43.76
10.14
56.48
10.91
47.58
10.38
IDF NN
38.88 #0.33
51.43
10.15
31.53
10.18
11.96
#0.22
57.76
#0.91
46.84
#0.31
CNN FT
33.00 20.74
45.71
#0.86
32.45
20.54
40.33
11.47
70.89
+4.30
54.08
20.33
AVG PROTO
36.25 10.33
37.26
1:.85
35.68
1:225
28.48
4..96
59.54
#1.48
44.04
#0.80
IDF PROTO
37.86 11.13
41.91
11.17
34.88
20.73
32.14
10.51
61.00
11.23
43.09
11.15
PROTO
29.67 11.02
34.02
+1
33.49
20.79
28.43
10.68
65.22
11.52
49.78
20.22
AVG MAML
33.74 +0.32
39.35
+1.38
36.14
+1.23
39.98
54.55
+1.08
43.83
#2.04
IDF MAML
37.26 1166
43.63
12.42
38.95
10.48
42.58
10.77
61.46
11.50
48.22
11220
CNN MAML
28.98 11.62
35.30
11.04
34.12
#0.86
39.03
20.97
66.62
11.89
51.73
#3.98
AVG RR
37.60 10.10
50.25
10.23
36.33
10.36
48.17
10.17
63.39
10.05
53.25
11.01
IDF RR
44.83 11.07
60.27
1:33
37.68
10.96
48.
65
10.57
69.12
1..87
55.65
1:008
CNN RR
32.25 11.6
37.30
10.80
37.32
11.15
41.81
1.449
71.40
11.6
56.83
:2.30
OUR
52.17
0.65
62.66
10066
43.09
10.16
54.15
1:00
81.81
1:1.1
67.10
1:093
OUR w/o t(-)
50.15 1.62
61.77
10.73
42.09
10.37
51.51
10.75
76.71
11.44
66.93
20.46
OUR w/o s(-)
41.99
10.69
51.12
10.88
40.16
20.23
48.59
20.62
78.15
10.
65.83
20.33
OUR w/o biLSTM
50.35
20.73
61.95
+0.40
42.22
#0.66
51.88
#0.84
77.17
#1.03
66.42
#0.35
OUR W EBD
39.68
22.60
56.48
12.63
40.6
10.84
48.64
64 10.69
81.68
12.21 61.47 1188
Table 8: 5-way s-shot classification. The bottom four rows present our ablation study.
Rep. Alg.
20 News
Amazon
HuffPost
RCV1
Reuters
FewRel
AVG
45.87 #0.39
60.36
20.20
41.55
20.23
60.84
10.19
80.51
20.60
60.67
10.30
IDF NN
51.94 20.20
67.15
20.21
42.35
10.15
58.27
20.23
82.88
20.57
60.62
10.41
CNN FT
47.17 40.98
63.91
#1.20
44.13
20.70
62.34
+0.06
90.95
#3.59
71.19 +0.37
AVG PROTO
45.42 11.36
51.99
=2.72
41.67
11.13
31.22
1.58
68.
11.53
46.55
11.58
IDF PROTO
46.53 11.44
59.24
11.0
50.24
10.94
35.63
10.83
72.07
+2.69
61.99
11.82
CNN PROTO
35.09 20.71
11.99
11.53
44.21
20.58
29.33
20.81
74.33
11.86
65.16 #0.99
AVG MAML
43.92 11.07
47.22
11.99
49.69
10.54
50.69
11.03
62.46
1.058
57 17 1:886
IDF MAML
48.62
62.45
1133
53.70
20.29
54.14
10.72
71.96
1:148
65 80
CNN MAMI
36.79 20.78
43.70
+1.38
45.89
20.53
51.15
10.30
85.00
70.76
66.90
+3.23
AVG RR
57.24 10.19
72.78
54.86
#0.21
72.62
90.00
+0.49
72.22
4.0.66
IDF RR
64.35
40.54
79.78
#0.28
59.56
#1.78
72.85
40.21
93.02
+0.45
75.30
40.34
CNN RR
44.32 20.44
53.89
11.54
49.96
20.20
59.47
20.58
87.93
11.49
71.81
11:25
OUR
68.33
+0.17
81.16
+0.31
63.51
10.10
75.38
+1.12
96.00
+7.27
83.53
+0.27
OUR w/o t(-)
67.59 10.57
80.58
10.19
60.86
10.19
75.12
12
20.90
93.71
20.84
83.20
10.30
OUR w/o s(-)
60.77
#0.52
75.37
20.27
60.29
20.35
72.84
20.23
94.79
20.32
82.62
20.20
OUR w/o biL.TM
66.99
+0.32
80.90
20.36
63.04
20.20
74.19
20.22
95.36
20.30
82.90 20.21
OUR ERD
57.52 #2.39
76.34
20.65 58.58
11.00
71.50 10.28 95. 76 20.53 80.88 1118
Table 9: 5-way 5-shot classification. The bottom four rows present our ablation study.
23

---

Published as a conference paper at ICLR 2020
talk, middts
sci.space
fisc.forsale
talk.polities.misse
comp-graphics
israel
space
sale
president
image
armenian
nasa
00
cramer
graphies
turkish
launch
shipping
mr
jpeg
armenians
orbit
offer
stephanopoulos
images
israeli
shuttle
condition
people
gif
jews
moon
1st price forsale asking comics
government
format
armenia henry
optilink myers clayton
file
arab
earth
people
mission
ftp
jewish solar
gay
color
sci.crypt
comp.windows.x
cong.os.ms-winnooss.mii.
talk.politics.gum.
talk. molk.religion.mis
key clipper
window
gun
god
max
guns
jesus
encryption
motif
windows
fbi firearms
sandvik
chip
server
g9v
christian
keys
widget
b8f
alf batf bible
security file
a86
jehovah
privacy x11
xterm
145
weapons christ
government
pl
people
lord
escrow entry
1d9
waco
kent
des dos
34u
cdt
brian
rec.autos sci.med
comp.sys. wwm..yy..uc.hardwarr
sci.electronics
rec.sport.hockey
medical
mac
circuit
hockey
cars
disease
apple centris quadra Ic monitor duo nubus drive simms
wire
game
engine
msg
ground
team
ford
cancer
wiring
nhl
oil
health
voltage
play
dealer patients
battery
season
callison doctor
copy
games
mustang hiv
amp
25
food
electronies
autos diet
audio
alt.atheism bike
rec.motoncycles comp.sys. wwtp:ssss.im..p.h.aee. hardware rec.sport.uusebul.
soc.religion.chrritiinn
god
sesi
baseball
god
atheism dod
drive
game
church
atheists ride
ide controller
year
jesus
keith bmw
team
christ
livesey
com
card
players
morality
riding
bus
games hit
christians
religion moral
hikes
drives
christian
motorcycle
bios
braves runs
rutgers
islamic
dog
disk
bible
say
rider
pe
pitcher
faith
Table 10: Top 10 LMI-ranked words for each class in 20 Newsgroup. Class names are shown in italic. Different class exhibit different salient features.
24


Reviews:
[{"review_id": "ryeH-6BycH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on applying meta learning approaches to text classification.\n\nThe primary contribution is an attention mechanism based on word statistics --- most importantly word frequency. This novel attention mechanism is motivated by the observation that the base units in text (lexemes) are more likely to have task specific interpretations than lower level patterns in vision. And, while a lexeme based attention mechanism trained on one task may not transfer well to other tasks, a mechanism based on coarser word statistics is less likely to focus in on task specific patterns.\n\nA secondary contribution is the use of ridge regression [1] to perform meta-learning for text classification.\n\nThe paper presents experiments on a number of text classification tasks from the NLP literature. Aside from the new attention mechanism and the use of ridge regression, the proposed approach makes use of FastText word embeddings or BERT sentence representations, depending on the task. The paper demonstrates significant improvements over baselines that use other methods of aggregating word representations. All of baselines were implemented for this paper.\n\nThe idea of using coarse statistical signatures to calculate attention is an interesting one. However, I have concerns about both the clarity of this paper and the lack of clear comparison to previous work.\n\n== Clarity ==\n\nMany of the details of the model and learning approach are vaguely discussed, or relegated to Figures 4 & 5. I think the paper would benefit from a more formal definition of the entire learning procedure.\n\n== Comparison to previous work ==\n\nThis paper seems to be following the standard FewRel experimental setup. Also, the RCV1 experiments seem to follow the [2] which was cited by the in the paper under review. However, it is not clear if the setups are the same or if the numbers are comparable.\n\nI am not sure about the existence of comparable results for the other tasks, but for FewRel at least the baselines presented here significantly underperform other papers' reports of equivalent models.\n\n  - The paper from [3] that introduced FewRel reported 69.2 / 84.8 for CNN based prototypical networks --- far above the 49.8 / 65.2 reported here.\n\n  - [4] found that a BERT model with no FewRel specific training at all achieves 72.9% on the 5way/1shot task. Which is above all of the BERT based models reported in Table 2.\n\nI may be missing something, but if these numbers are actually not comparable then this paper should contain an explanation of how the experimental setup differs. And if the setup is actually the same as previous work, I expect to see a comparison of results.\n\n[1] https://openreview.net/pdf?id=HyxnZh0ct7\n[2] https://openreview.net/forum?id=SyxMWh09KX\n[3] https://www.aclweb.org/anthology/D18-1514/\n[4] https://arxiv.org/abs/1906.03158", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rklewo2-9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: Based on the extensive improvements by the authors, I have updated my rating. However, I still have doubts about the potential of this approach to reach practically useful levels of accuracy.\n\nThis paper introduces a simple method to weight pretrained lexical features for use in meta learning of few-shot text classification. The method boils down to weighting word inut features, in the form of pretrained word-embeddings, by attention computed from inverse document frequency and class local mutual information. The idea is that this measure of feature informativeness transfers between tasks, whereas lexical features themselves are highly task-specific. The approach is well motivated and is empirically shown to outperform existing approaches to few-shot text classification with a significant margin.\n\nWhile the improvement over existing approaches is quite substantial, I believe the paper should not be accepted to ICLR for the following reasons. First, the contribution is quite limited and not particularly novel. While two weight functions are proposed, the majority of improvement comes simply from normalizing IDF with attention. Based on existing work on delexicalized features for NLP tasks such as parsing, this is quite a straightforward extension. Given the limited contribution, a short paper seems a better fit. Second, the approach simply trades variance for bias. This brings us to the question of how likely the approach is to be a building block in bringing us towards a pratically useful few-shot classification method. Given the weak representational power of the model, I believe this is unlikely. I see a situation similar to syntactic parsing for low-resource languages, where a collection of simple techniques similar in spirit to the current approach, like delexicalization, brought results far above the naive baselines, but never approached practically useful results. I think this is a crucial point to address in meta-learning research in general to make sure we’re not just solving a toy problem with tailored heuristics.\n\nAdditional notes:\n\nWhat is the motivation for using a BiLSTM to combine inverse document frequency and inverse class entropy? Is the sequence information at all useful, or would a simple projection and nonlinearity give the same result?\n\nThe theoretical analysis is completely self-evident from the definition of the feature space. Replaing a feature with an equivalent feature of course gives the same result and I don’t see the need to “mathematize” this.\n\nThe effect of approximating logistic regression with linear regression + calibration is not analyzed and it is not clear what the effect of this approximation is in the text classification scenario. I would suggest to compare to differentiating through a direct optimization of the logistic formulation, for example with Newton’s method, or plain SGD, as in Bertinetto et al. (2019).\n\nTable 1. Why not run the attention-based feature aggregator together with all algorithms. The main contribution is at the input representation level, and this should be applicable across algorithms. In fact, if we remove the BiLSTM which seems to have a very small effect the representation function does not contain learnable parameters in itself.\n\nPlease provide the average across datasets in Table 1.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkglFISTYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effects of using function of ngram statistics as feature to generate attention score per word. The attention score is then used as weights to aggregate document embedding by doing a weighted average on word embedding. The output is finally fed into a ridge regressor to do the final predictions on target labels. \n\nMain comments:\nThis paper has a clear motivation and decent experimental results (though some concern on baseline models, see below). The introduction of using distributional signature to derive attention scores seems interesting and a novel contribution. However I was not able to fully understand the intuition behind the benefit of doing attention mechanism on top of ngram statistics (see my question below as well). \nAlso the reference/baseline models used in the experiment might not be strong enough. If you could compare your model with some latest algorithms proposed in the few-shot-learning communities, that would be more convincing as well. \nTo list a few:\n* P-MAML: [Zhang et al., 2019]\n* Induction-Network-Routing: [Geng et al., 2019]\n* ROBUSTTC-FSL [Yu et al., 2018]\n\nI am leaning to give a \"weak reject\" based on my current knowledge and understanding of the paper. But I will be willing to revisit the decision after we get feedback from the author(s). \n\nIn particular, I would be glad if the author could clarify the questions below.\n\n* From table 1, it seems Method IDF+RR is a competitive model. IIUC, the statistics of s(.) is highly correlated with IDF which also indicates general word importance in corpus. My questions are that, \n1) regarding ablation test \"OUR w/o biLSTM\", how is $h$ calculated in this case (without biLSTM)?\n2) since each word is represented based on two statistical number (map function by t(.) and s(.)), can you give any intuitive explanation that why getting attention score from that makes sense?\n3) do you have any experiments using the distributional signature as a common feature in standard text classification problems? In other words, is this method only (significantly) beneficial to few-short-learning? If it is also useful in general text classification task, it would be a good \"plus\" here.\n\n* From table 2, can you explain why CNN+RR benefits a lot from the BERT embedding? Actually it gets more percentage of improvement than the model \"OUR\".\n\n* For all the usages of pre-trained embedding (fasttext or BERT), are you further finetuning the embedding parameters during your training? Or you freeze the embedding parameters?\n\n[Zhang et al., 2019] Ningyu Zhang et al., Improving Few-shot Text Classification via Pretrained Language Representations. arXiv preprint arXiv: 1908.08788\n[Geng et al., 2019] Ruiying Geng, Binhua Li, Yongbin Li, Yuxiao Ye, Ping Jian, and Jian Sun. 2019. Few-shot text classification with induction network. arXiv preprint arXiv:1902.10482.\n [Yu et al., 2018] Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang,\nand Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. arXiv preprint arXiv:1805.07513", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a meta-learning approach for few-shot text classification. The main idea is to use an attention mechanism over the distributional signatures of the inputs to weight word importance. Experiments on text classification datasets show that the proposed method improves over baselines in 1-shot and 5-shot settings.

The paper addresses an important problem of learning from a few labeled examples. The proposed approach makes sense and the results clearly show the strength of the proposed approach.

R1 had some questions regarding the proposed method and experimental details. I believe this have been addressed by the authors in their rebuttal.

R2 suggested that the authors clarified their experimental setup with respect to prior work and improved the clarity of their paper. The authors have made some adjustments based on this feedback, including adding new sections in the appendix.

R3 had concerns regarding the contribution of the approach and whether it trades variance for bias. The authors have addressed most of these concerns and R3 has updated their review accordingly.

I think all the reviewers gave valuable feedbacks that have been incorporated by the authors to improve their paper. While the overall scores remain low, I believe that they would have been increased had R1 and R2 reassessed the revised submission. I recommend to accept this paper.

Decision: Accept (Poster)