Pretrained models can be fine-tuned on natural language processing (NLP) tasks and have achieved significant improvements over training on task-specific annotated data. More recently, a pretraining technique named bidirectional encoder representations from transformers (BERT)  was proposed and has enabled the creation of state-of-the-art models for a wide variety of NLP tasks, including question answering (SQuAD v1.1) and natural language inference, among others. However, there have not been many efforts in exploring pretrained language representations for few-shot text classification. The technical contributions of this work are two-fold: 1) we explore the pretrained model to address the poor generalization capability of text classification, and 2) we propose a meta-learning model based on model-agnostic meta-learning (MAML) which explicitly disentangles the task-agnostic feature learning and task-specific feature learning to demonstrate that the proposed model achieves significant improvement on text classification accuracy on public benchmark datasets. To the best of our knowledge, we are the first to bridge the pretraining strategy with meta-learning methods for few-shot text classification.

Introduction	Deep learning (DL) has achieved great success in many fields such as computer vision, speech recognition, and machine translation  ,  ,  thanks to the advancements in optimization techniques, larger datasets, and streamlined designs of deep neural architectures. However, DL is notorious for requiring large labeled datasets, which limits the scalability of a deep model to new classes owing to the cost of annotation. Humans, however, are readily able to learn and distinguish new classes rapidly with only a few examples. This gap between human and machine learning provides opportunities for DL development and applications. Few-shot learning generally resolves the data deficiency problem by recognizing novel classes from very few labeled examples. This limitation in the size of samples (only one or very few examples) challenges the standard fine-tuning method in DL. Early studies in this field  applied data augmentation and regularization techniques to alleviate the overfitting problem caused by data scarcity but only to a limited extent. Instead, researchers have been inspired by human learning to explore meta-learning  to leverage the distribution over similar tasks.

Language Representation Pretraining	While the pretraining tasks have been designed with particular downstream tasks in mind  , we focus on those training tasks that seek to induce universal representations suitable for downstream few-shot learning tasks. We utilize BERT  as a recent study  has shown its potential to achieve state-of-the-art performance when fine-tuned in NLP tasks. BERT combines both word and sentence representations (via masked language model and next sentence prediction objectives) in a single very large pretrained transformer  . It is adapted to both word- and sentence-level tasks with task-specific layers. We feed the sentence representation into a softmax layer for text classification based on  .

Datasets and Evaluation	We use the multiple tasks with the multi-domain sentiment classification  dataset ARSC. This dataset comprises English reviews for 23 types of products on Amazon. For each product domain, there are three different binary classification tasks. These buckets then form 23 INLINEFORM0 3 = 69 tasks in total. Following  , we select 12 (4 INLINEFORM1 3) tasks from four domains (i.e., Books, DVDs, Electronics, and Kitchen) as the test set, with only five examples as support set for each label in the test set. We thus create 5-shot learning models on this dataset. We evaluate the performance by few-shot classification accuracy following previous studies in few-shot learning  ,  . To evaluate the proposed model objectively with the baselines, note that for ARSC, the support set for testing is fixed by  ; therefore, we need to run the test episode once for each of the target tasks. The mean accuracy from the 12 target tasks is compared to those of the baseline models in accordance with  . We use pretrained BERT-Base for the ARSC dataset.

Model parameters INLINEFORM7 are trained to optimize the performance of INLINEFORM8 on the unseen test examples from INLINEFORM9 across tasks. The meta-objective is as follows:  INLINEFORM0  The goal of MAML is to optimize the model parameters INLINEFORM0 such that the model can learn to adapt to new tasks with parameters via a few gradient steps on the training examples of the new tasks. The model is improved by considering how the test errors on the unseen test data from INLINEFORM1 change with respect to the parameters. The meta-objective across tasks is optimized using stochastic gradient descent (SGD). The model parameters INLINEFORM2 are updated as follows:  INLINEFORM0

Background: Meta-Learning	Our work is built on the recently proposed MAML framework  , which we describe briefly here. MAML aims to learn the learners (for the tasks) and the meta-learner in the few-shot meta-learning setup  ,  ,  . Formally, it considers a model that is represented by a function INLINEFORM0 with parameters INLINEFORM1 . When the model adapts to a new task INLINEFORM2 , the model changes the parameters from one INLINEFORM3 to the next, where a task contains INLINEFORM4 training examples and one or more test examples (K-shot learning). MAML updates the parameters INLINEFORM5 via one or a few iterations of gradient descent based on the training examples of task INLINEFORM6 . For example, for one gradient update,  INLINEFORM0 , where the step size INLINEFORM0 is a hyperparameter; INLINEFORM1 is a loss function that evaluates the error between the prediction INLINEFORM2 and target INLINEFORM3 , where INLINEFORM4 , INLINEFORM5 are input–output pairs sampled from the training examples of task INLINEFORM6 .

[th] [1] P-MAML Algorithm Training Datapoints INLINEFORM0 Construct a task INLINEFORM1 with training examples using a support set INLINEFORM2 and a test example INLINEFORM3 Randomly initialize INLINEFORM4 Pre-train INLINEFORM5 with BERT Denote INLINEFORM6 as distribution over tasks not done Sample batch of tasks INLINEFORM7 : all INLINEFORM8 Evaluate INLINEFORM9 using INLINEFORM10 Compute adapted parameters with gradient descent: INLINEFORM11 Update INLINEFORM12 using each INLINEFORM13 from INLINEFORM14 and INLINEFORM15

Improving Few-shot Text Classification via Pretrained Language Representations.	Text classification tends to be difficult when the data is deficient or when it is required to adapt to unseen classes. In such challenging scenarios, recent studies have often used meta-learning to simulate the few-shot task, thus negating explicit common linguistic features across tasks. Deep language representations have proven to be very effective forms of unsupervised pretraining, yielding contextualized features that capture linguistic properties and benefit downstream natural language understanding tasks. However, the effect of pretrained language representation for few-shot learning on text classification tasks is still not well understood. In this study, we design a few-shot learning model with pretrained language representations and report the empirical results. We show that our approach is not only simple but also produces state-of-the-art performance on a well-studied sentiment classification dataset. It can thus be further suggested that pretraining could be a promising solution for few shot learning of many other NLP tasks. The code and the dataset to replicate the experiments are made available at this https URL.

Conclusion	In this study, we attempt to analyze language representation pretraining for few-shot text classification empirically. We combine the MAML algorithm with the pretraining strategy to disentangle the task-agnostic and task-specific representation learning. Results show that our model outperforms conventional state-of-the-art few-shot text classification models. In the future, we plan to apply our method to other NLP scenarios.

Training Procedure	The training procedure of our approach consists of two parts. Language Representation Pretraining. Given all the training samples, we first utilize pretraining strategies to learn task-agnostic contextualized features that capture linguistic properties to benefit downstream few-shot text classification tasks. Episode-based Meta Training. Given the pretrained language representations, we construct episodes to compute gradients and update the model in each training iteration with MAML.

Contemporary approaches to few-shot learning often decompose the training procedure into an auxiliary meta-learning phase, which includes many sub-tasks, following the principle that the testing and training conditions must match. They extract some transferable knowledge by switching the task from one mini-batch to the next. Moreover, the few-shot model is able to classify data into new classes with just a small labeled support set. Existing approaches for few-shot learning are still plagued by problems, including imposed strong priors  , complex gradient transfer between tasks  , and fine-tuning of the target problem  . The approaches proposed by  and  , which combine non-parametric methods and metric learning, may provide possible solutions to these problems. The non-parametric methods allow novel examples to be rapidly assimilated without suffering from the effects of catastrophic overfitting. Such non-parametric models only need to learn the representation of the samples and the metric measure. Recently, a variety of techniques were proposed for training general-purpose language representation models using an enormous amount of unannotated text, such as ELMo  and generative pretrained transformer (GPT)  .

We believe the performance of our model can be further improved by adopting additional mechanisms like adaptive metrics, which will be part of our future work. Note that, our approach is very simple and independent of the encoder choices, and can, therefore, be easily adapted to fit other encoder architectures for sophisticated NLP tasks.

Discussions	It should be noted that human beings are intelligent to leverage learned knowledge about the world in understanding language.  think human beings have a universal grammar, and our daily language system is only a formal expression of this universal grammar. In other words, there are deep structures related to concepts and superficial structures related to speech and symbols in a language. Moreover, neuroscience research has proposed a prominent idea that language processing may offer such a principle that the brain contains partially separate systems for processing syntax and semantics. The part of the prefrontal cortex responsible for language production, called Broca’s area, is thought to be important for parsing syntactic information and applying selective attention to help a separate comprehension system interpret the semantics  . Our idea for few-shot learning in NLP is somewhat similar to this assumption as the pretraining stage may learn common syntax information across tasks, and the meta-learning stage may learn semantic knowledge, which is task specific.

Episode-Based Meta Training	Given the pretrained language representations, we construct episodes to compute the gradients and update our model in each training iteration. The training episode is formed by randomly selecting a subset of classes from the training set, then choosing a subset of examples within each selected class to act as the support set INLINEFORM0 with a subset of the remaining examples to serve as the query set INLINEFORM1 . Training with such episodes is achieved by feeding the support set INLINEFORM2 to the model and updating its parameters to minimize the loss in the query set INLINEFORM3 . We call this strategy as episode-based meta training. The adaptation of meta-learning using the MAML framework with pretrained language representations is summarized in Algorithm SECREF4 , called P-MAML. The use of episodes makes the training procedure more faithful to the test environment, thereby improving generalization. It is worth noting that there are exponentially many possible meta tasks to train the model with, thus making it difficult to overfit.

Problem definition	Few-shot classification is a task in which a classifier must adapt and accommodate new classes that are not seen in training, given only a few examples of each of these new classes. We have a large labeled training set with a set of defined classes INLINEFORM0 . However, after training, our ultimate goal is to produce classifiers on the testing set with a disjoint set of new classes INLINEFORM1 for which only a small labeled support set will be available. If the support set contains INLINEFORM2 labeled examples for each of the INLINEFORM3 unique classes, the target few-shot problem is called a INLINEFORM4 -way INLINEFORM5 -shot problem. Usually, INLINEFORM6 is a too small sample set to train a supervised classification model. Therefore, we aim to perform meta-learning on the training set in order to extract transferrable knowledge that will allow us to perform better few-shot learning on the support set to classify the test set more successfully.

Evaluation Results	To evaluate the performance of our model, we compared it with various baseline models. The evaluation results are shown in Table TABREF9 : P-MAML is our current approach, Match Network  is a few-shot learning model using metric-based attention method, Prototypical Network  is a deep matrix-based method using sample averages as class prototypes, MAML  is an MAML method that is compatible with any model trained with gradient descent and applicable to a variety of learning problems, Relation Network  is a metric-based few-shot learning model that uses a neural network as the distance measurement and calculate class vectors by summing sample vectors in the support set, ROBUSTTC-FSL  is an approach that combines adaptive metric methods by clustering the tasks, Induction-Network-Routing  is a recent state-of-the-art method which learn generalized classwise representations by combining the dynamic routing algorithm with a typical meta-learning framework. From the results shown in Table TABREF9 , we observe that our approach achieves the best results amongst all meta-learning models. Compared with ROBUSTTC-FSL and Induction-Network-Routing, which adopt several metric methods and dynamic routing algorithms, our approach still provides more advantages.

Ablation Study	To analyze the contributions and effects of language representation pretraining in our approach, we perform ablation tests. GloVe is the method with pretrained GloVe  word embeddings; w/o pretrain is our method without pre-trained embeddings (random initialization). From the evaluation results in Table TABREF11 , we observe the performance drop significantly without pretraining, which proves the effectiveness of explicit common linguistic features learning. We also notice that our model with GloVe does not achieve good performance even compared with the random initialization, which indicates that the poor generalization capability for few-shot text classification.

All model parameters are updated by backpropagation using Adam with a learning rate of 0.01. We regularize our network using dropout with a rate of 0.3 tuned using the development set.

Question: What tasks are explored?
Answer: 69 tasks