Venue: NAACL 2018

# Diverse Few-Shot Text Classification with Multiple Metrics

Mo Yu\({}^{*}\) Xiaoxiao Guo\({}^{*}\) Jinfeng Yi\({}^{*}\) Shiyu Chang Saloni Potdar Yu Cheng Gerald Tesauro Haoyu Wang Bowen Zhou

###### Abstract

We study few-shot learning in natural language domains. Compared to many existing works that apply either metric-based or optimization-based meta-learning to image domain with low inter-task variance, we consider a more realistic setting, where tasks are diverse. However, it imposes tremendous difficulties to existing state-of-the-art metric-based algorithms since a single metric is insufficient to capture complex task variations in natural language domain. To alleviate the problem, we propose an adaptive metric learning approach that automatically determines the best weighted combination from a set of metrics obtained from meta-training tasks for a newly seen few-shot task. Extensive quantitative evaluations on real-world sentiment analysis and dialog intent classification datasets demonstrate that the proposed method performs favorably against state-of-the-art few shot learning algorithms in terms of predictive accuracy. We make our code and data available for further study.1

Footnote 1: [https://github.com/OrowOlverserWeb_Janson](https://github.com/OrowOlverserWeb_Janson)

+
Footnote †: Equal contributions from the corresponding authors: yum@us.ibm.com, xiaoxiao.guo@ibm.com, jinfengy@us.ibm.com.

## 1 Introduction

Few-shot learning (FSL) Miller et al. (2000); Li et al. (2006); Lake et al. (2015) aims to learn classifiers from few examples per class. Recently, deep learning has been successfully exploited for FSL via learning meta-models from a large number of **meta-training tasks**. These meta-models can be then used for rapid-adaptation for the **target/meta-testing tasks** that only have few training examples. Examples of such meta-models include: (1) metric-/similarity-based models, which learn contextual, and task-specific similarity measures Koch (2015); Vinyals et al. (2016); Snell et al. (2017); and (2) optimization-based models, which receive the input of gradients from a FSL task and predict either model parameters or parameter updates Ravi and Larochelle (2017); Munkhdalai and Yu (2017); Finn et al. (2017); Wang et al. (2017).

In the past, FSL has mainly considered image domains, where all tasks are often sampled from one huge collection of data, such as Omniglot Lake et al. (2011) and ImageNet Vinyals et al. (2016), making tasks come from a single domain thus related. Due to such a simplified setting, almost all previous works employ a common meta-model (metric-/optimization-based) for all few-shot tasks. However, this setting is far from the realistic scenarios in many real-world applications of few-shot text classification. For example, on an enterprise AI cloud service, many clients submit various tasks to train text classification models for business-specific purposes. The tasks could be classifying customers' comments or opinions on different products/services, monitoring public reactions to different policy changes, or determining users' intents in different types of personal assistant services. As most of the clients cannot collect enough data, their submitted tasks form a few-shot setting. Also, these tasks are significantly diverse, thus a common metric is insufficient to handle all these tasks.

We consider a more realistic FSL setting in this paper, where tasks are diverse. In such a scenario, the optimal meta-model may vary across tasks. Our solution is based on the metric-learning approach Snell et al. (2017) and the key idea is to maintain multiple metrics for FSL. The meta-learner selects and combines multiple metrics for learning the target task using **task clustering** on the meta-training tasks. During the meta-training, we propose to first partition the meta-training tasks into clusters, making the tasks in each cluster

likely to be related. Then within each cluster, we train a deep embedding function as the metric. This ensures the common metric is only shared across tasks within the same cluster. Further, during meta-testing, each target FSL task is assigned to a task-specific metric, which is a linear combination of the metrics defined by different clusters. In this way, the diverse few-shot tasks can derive different metrics from the previous learning experience.

The key of the proposed FSL framework is the task clustering algorithm. Previous works Kumar and Daume III (2012); Kang et al. (2011); Crammer and Mansour (2012); Barzilai and Crammer (2015) mainly focused on convex objectives, and assumed the number of classes is the same across different tasks (_e.g._ binary classification is often considered). To make task clustering (i) compatible with deep networks and (ii) able to handle tasks with a various number of labels, we propose a **matrix-completion based task clustering** algorithm. The algorithm utilizes task similarity measured by cross-task transfer performance, denoted by matrix \({\bf S}\). The \((i,j)\)-entry of \({\bf S}\) is the estimated accuracy by adapting the learned representations on the \(i\)-th (source) task to the \(j\)-th (target) task. We rely on matrix completion to deal with missing and unreliable entries in \({\bf S}\) and finally apply spectral clustering to generate the task partitions.

To the best of our knowledge, our work is the first one addressing the diverse few-shot learning problem and reporting results on real-world few-shot text classification problems. The experimental results show that the proposed algorithm provides significant gains on few-shot sentiment classification and dialog intent classification tasks. It provides positive feedback on the idea of using multiple meta-models (metrics) to handle diverse FSL tasks, as well as the proposed task clustering algorithm on automatically detecting related tasks.

## 2 Problem Definition

Few-Shot LearningSince we focus on **diverse metric-based FSL**, the problem can be formulated in two stages: (1) **meta-training**, where a set of metrics \(\mathcal{M}=\left\{\Lambda_{1},\cdots,\Lambda_{K}\right\}\) is learned on the **meta-training tasks**\(\mathcal{T}\). Each \(\Lambda_{i}\) maps two input \((x_{1},x_{2})\) to a scalar of similarity score. Here \(\mathcal{T}=\left\{\mathrm{T}_{1},\mathrm{T}_{2},\cdots,\mathrm{T}_{N}\right\}\) is a collection of \(N\) tasks. Here \(K\) is a pre-defined number (usually \(K\ll N\)). Each task \(\mathrm{T}_{i}\) consists of training, validation, and testing set denoted as \(\big{\{}D^{train}_{i},D^{valid}_{i},D^{test}_{i}\big{\}}\), respectively. Note that the definition of \(\mathcal{T}\) is a generalized version of \(\mathcal{D}^{(meta-train)}\) in Ravi and Larochelle (2017), since each task \(\mathrm{T}_{i}\) can be either few-shot (where \(D^{valid}_{i}\) is empty) or regular2. (2) **meta-testing**: the trained metrics in \(\mathcal{M}\) is applied to **meta-testing tasks** denoted as \(\mathcal{T}^{\prime}=\left\{\mathrm{T}^{\prime}{}_{1},\cdots,\mathrm{T}^{ \prime}{}_{N^{\prime}}\right\}\), where each \(\mathrm{T}^{\prime}{}_{i}\) is a few-shot learning task consisting of both training and testing data as \(\big{\{}D^{triain}_{i},D^{ttest}_{i}\big{\}}\). \(D^{triain}_{i}\) is a small labeled set for generating the prediction model \(\mathrm{M}^{\prime}_{i}\) for each \(\mathrm{T}^{\prime}_{i}\). Specifically, \(\mathrm{M}^{\prime}_{i}\)s are kNN-based predictors built upon the metrics in \(\mathcal{M}\). We will detail the construction of \(\mathrm{M}^{\prime}_{i}\) in Section 3, Eq. (6). It is worth mentioning that the definition of \(\mathcal{T}^{\prime}\) is the same as \(\mathcal{D}^{(meta-test)}\) in Ravi and Larochelle (2017). The **performance of few-shot learning** is the macro-average of \(\mathrm{M}^{\prime}_{i}\)'s accuracy on all the testing set \(D^{trest}_{i}\)s.

Footnote 2: For example, the methods in Triantafillou et al. (2017) can be viewed as training meta-models from any sampled batches from one single meta-training dataset.

Our definitions can be easily generalized to other meta-learning approaches Ravi and Larochelle (2017); Finn et al. (2017); Mishra et al. (2017). The motivation of employing multiple metrics is that when the tasks are diverse, one metric model may not be sufficient. Note that previous metric-based FSL methods can be viewed as a special case of our definition where \(\mathcal{M}\) only contains a single \(\Lambda\), as shown in the two base model examples below.

Base Model: Matching NetworksIn this paper we use the metric-based model Matching Network (MNet) Vinyals et al. (2016) as the base metric model. The model (Figure 0(b)) consists of a neural network as the embedding function (**encoder**) and an augmented memory. The encoder, \(f(\cdot)\), maps an input \(x\) to a \(d\)-length vector. The learned metric \(\Lambda\) is thus the similarity between the encoded vectors, \(\Lambda(x_{1},x_{2})=f(x_{1})^{T}f(x_{2})\), i.e. the metric \(\Lambda\) is modeled by the encoder \(f\). The augmented memory stores a support set \(S=\left\{(x_{i},y_{i})\right\}_{i=1}^{|S|}\), where \(x_{i}\) is the supporting instance and \(y_{i}\) is its corresponding label in a one-hot format. The MNet explicitly defines a classifier \(\mathrm{M}\) conditioned on the supporting set \(S\). For any new data \(\hat{x}\), \(\mathrm{M}\) predicts its label via a similarity function \(\alpha(.,.)\) 

between the test instance \(\hat{x}\) and the support set \(S\):

\[y=P(.|\hat{x},S)=\sum_{i=1}^{|S|}\alpha(\hat{x},x_{i};\theta)y_{i}, \tag{1}\]

where we defined \(\alpha(.,.)\) to be a softmax distribution given \(\Lambda(\hat{x},x_{i})\), where \(x_{i}\) is a supporting instance, i.e., \(\alpha(\hat{x},x_{i};\theta)=\nicefrac{{\exp(f(\hat{x})^{T}f(x_{i}))}}{{\sum _{j=1}^{|S|}\exp(f(\hat{x})^{T}f(x_{j}))}}\), where \(\theta\) are the parameters of the encoder \(f\). Thus, \(y\) is a valid distribution over the supporting set's labels \(\{y_{i}\}_{i=1}^{|S|}\). To adapt the MNet to text classification, we choose encoder \(f\) to be a convolutional neural network (CNN) following Kim (2014); Johnson and Zhang (2016). Figure 1 shows the MNet with the CNN architecture. Following Collobert et al. (2011); Kim (2014), the model consists of a convolution layer and a max-pooling operation over the entire sentence.

To train the MNets, we first sample the training dataset \(D\) for task \(T\) from all tasks \(\mathcal{T}\), with notation simplified as \(D\sim\mathcal{T}\). For each class in the sampled dataset \(D\), we sample \(k\) random instances in that class to construct a support set \(S\), and sample a batch of training instances \(B\) as training examples, i.e., \(B,S\sim D\). The training objective is to minimize the prediction error of the training samples given the supporting set (with regard to the encoder parameters \(\theta\)) as follows:

\[\mathop{\mathbb{E}}_{D\sim\mathcal{T}}\Big{[}\mathop{\mathbb{E}}_{ B,S\sim D}\big{[}\sum_{(x,y)\in B}\log(P(y|x,S;\theta))\big{]}\Big{]}. \tag{2}\]

Base Model: Prototypical NetworksPrototypical Network (ProtoNet) Snell et al. (2017) is a variation of Matching Network, which also depends on metric learning but builds the classifier \(\mathrm{M}\) different from Eq. (1):

\[y=P(.|\hat{x},S)=\sum_{i=1}^{L}\alpha(\hat{x},S_{i};\theta)y_{i}. \tag{3}\]

\(L\) is the number of classes and \(S_{i}\)=\(\{x|(x,y)\in S\wedge y\!\!=\!\!y_{i}\}\) is the support set of class \(y_{i}\). \(\alpha(\hat{x},S_{i};\theta)=\nicefrac{{\exp\left(f(\hat{x})^{T}\sum_{x\in S _{i}}f(x)\right)}}{{\sum_{j=1}^{L}\exp\left(f(\hat{x})^{T}\sum_{x^{\prime}\in S _{j}}f(x^{\prime})\right)}}\).

## 3 Methodology

We propose a task-clustering framework to address the diverse few-shot learning problem stated in Section 2. We have the FSL algorithm summarized in Algorithm 1. Figure 2 gives an overview of our idea. The initial step of the algorithm is a novel task clustering algorithm based on matrix completion, which is described in Section 3.1. The few-shot learning method based on task clustering is then introduced in Section 3.2.

### Robust Task Clustering by Matrix Completion

Our task clustering algorithm is shown in Algorithm 2. The algorithm first evaluates the transfer performance by applying a single-task model \(i\) to another task \(j\) (Section 3.1.1), which will result in a (partially observed) cross-task transfer performance matrix **S**. The matrix **S** is then cleaned and completed, giving a symmetry task similarity matrix **Y** for spectral clustering Ng et al. (2002).

#### 3.1.1 Estimation of Cross-Task Transfer Performance

Using single-task models, we can compute performance scores \(s_{ij}\) by adapting each \(\mathrm{M}_{i}\) to each task

Figure 1: The Convolutional Neural Networks (CNN) used in this work: (a) A CNN classifier. The encoder component takes the sentence as input and outputs a fixed-length sentence embedding vector; the classifier component predicts class labels with the sentence embedding. (b) A Matching Network, which only contains an encoder like in (a), and makes prediction via a k-Nearest-Neighbor classifier with the similarity defined by the encoder.

 

\(T_{j}(j\neq i)\). This forms an \(n\times n\) pair-wise classification performance matrix **S**, called the _transfer-performance matrix_. Note that **S** is asymmetric since usually \(\textbf{S}_{ij}\neq\textbf{S}_{ji}\).

```
Input:\(N\) meta-training tasks \(\mathcal{T}\)=\(\{\mathrm{T}_{1},\mathrm{T}_{2},\cdots,\mathrm{T}_{n}\}\); number of clusters \(K\); \(N^{\prime}\) target few-shot meta-testing tasks \(\mathcal{T}^{\prime}\) Output: Meta-model \(\mathcal{M}\) = \(\{C_{1:K}~{}(K\text{ task clusters}),~{}\mathcal{F}=\{f_{1},f_{2},\cdots,f_{K}\}~{}( K\text{ task encoders})\}\) . One classifier \(\operatorname{M}_{i}^{\prime}\) for each target task \(\operatorname{T}_{i}^{\prime}\).
1 Robust Task Clustering:\(C_{1:K}\) = RobustTC(\(\mathcal{T}\),\(K\)) (Algorithm 2) Cluster-Model Training: Train one encoder (multi-task MNet) \(f_{i}\) on each task cluster \(C_{i}\) (Section 3.2.1) Few-Shot Learning on Cluster-models: Train a model \(\operatorname{M}_{trg}\) on task \(\operatorname{T}_{trg}\) with the method in Section 3.2.2.
```

**Algorithm 1**RobustTC-FSL: Task Clustering for Few-Shot Learning

Ideally, the transfer performance could be estimated by training a MNet on task \(i\) and directly evaluating it on task \(j\). However, the limited training data usually lead to generally low transfer performance of single-task MNet. As a result we adopt the following approach to estimate **S**:

We train a CNN classifier (Figure 1(a)) on task \(i\), then take only the encoder \(\operatorname{M}_{i}^{enc}\) from \(\operatorname{M}_{i}\) and freeze it to train a classifier on task \(j\). This gives us a new task \(j\) model, and we test this model on \(D_{j}^{valid}\) to get the accuracy as the transfer-performance \(\mathbf{S}_{ij}\). The score shows how the representations learned on task \(i\) can be adapted to task \(j\), thus indicating the similarity between tasks.

```
Input:A set of \(n\) tasks \(\mathcal{T}=\{\mathrm{T}_{1},\mathrm{T}_{2},\cdots,\mathrm{T}_{n}\}\), number of task clusters \(K\) Output:\(K\) task clusters \(C_{1:K}\)
1 Learning of Single-Task Models: train single-task models \(\operatorname{M}_{i}\) for each task \(\mathrm{T}_{i}\) Evaluation of Transfer-Performance Matrix: get performance matrix **S** (Section 3.1.1) Score Filtering: Filter the uncertain scores in S and construct the symmetric matrix **Y** using Eq. (4) Matrix Completion: Complete the similar matrix **X** from **Y** using Eq. (5) Task Clustering:\(C_{1:K}\)=SpectralClustering(\(\textbf{X},K\))
```

**Algorithm 2**RobustTC: Robust Task Clustering based on Matrix Completion

Remarks: Out-of-Vocabulary ProblemIn text classification tasks, transferring an encoder with fine-tuned word embeddings from one task to another is difficult as there can be a significant difference between the two vocabularies. Hence, while learning the single-task CNN classifiers, we always make the word embeddings fixed.

#### 3.1.2 Task Clustering Method

Directly using the transfer performance for task clustering may suffer from both efficiency and accuracy issues. First, evaluation of all entries in the matrix **S** involves conducting the source-target transfer learning \(O(n^{2})\) times, where \(n\) is the number of meta-training tasks. For a large number of diverse tasks where the \(n\) can be larger than 1,000, evaluation of the full matrix is unacceptable (over 1M entries to evaluate). Second, the estimated cross-task performance (i.e. some \(\textbf{S}_{ij}\) or \(\textbf{S}_{ji}\) scores) is often unreliable due to small data

Figure 2: Overview of the idea of our multi-metric learning approach for few-shot learning. (a) an illustration of the sparse cross-tasks transfer-performance matrix with unobserved entries (white blocks) and unreliable values (top-right and bottom-left corners), where red colors indicate positive transfer and blue colors indicate negative transfer; (b) the constructed binary partially-observed matrix with low-rank constraint for matrix completion and clustering (see Section 3.1 for the details); (c) an encoder trained with the matching network objective Eq. (2) on a task cluster (tasks 1, 2 and 3 in the example).

 

size or label noise. When the number of the uncertain values is large, they can collectively mislead the clustering algorithm to output an incorrect task-partition. To address the aforementioned challenges, we propose a novel task clustering algorithm based on the theory of matrix completion (Candes and Tao, 2010). Specifically, we deal with the huge number of entries by randomly sample task pairs to evaluate the \({\bf S}_{ij}\) and \({\bf S}_{ji}\) scores. Besides, we deal with the unreliable entries and asymmetry issue by keeping only task pairs \((i,j)\) with consistent \({\bf S}_{ij}\) and \({\bf S}_{ji}\) scores. as will be introduced in Eq. (4). Below, we describe our method in detail.

Score FilteringFirst, we use only reliable task pairs to generate a _partially-observed_ similarity matrix \({\bf Y}\). Specifically, if \({\bf S}_{ij}\) and \({\bf S}_{ji}\) are high enough, then it is likely that tasks \(\{i,j\}\) belong to a same cluster and share significant information. Conversely, if \({\bf S}_{ij}\) and \({\bf S}_{ji}\) are low enough, then they tend to belong to different clusters. To this end, we need to design a mechanism to determine if a performance is high or low enough. Since different tasks may vary in difficulty, a fixed threshold is not suitable. Hence, we define a dynamic threshold using the mean and standard deviation of the target task performance, i.e., \(\mu_{j}=\text{mean}({\bf S}_{:j})\) and \(\sigma_{j}=\text{std}({\bf S}_{:j})\), where \({\bf S}_{:j}\) is the \(j\)-th column of \({\bf S}\). We then introduce two positive parameters \(p_{1}\) and \(p_{2}\), and define high and low performance as \({\bf S}_{ij}\) greater than \(\mu_{j}+p_{1}\sigma_{j}\) or lower than \(\mu_{j}-p_{2}\sigma_{j}\), respectively. When both \({\bf S}_{ij}\) and \({\bf S}_{ji}\) are high and low enough, we set their pairwise similarity as \(1\) and \(0\), respectively. Other task pairs are treated as uncertain task pairs and are marked as unobserved, and don't influence our clustering method. This leads to a partially-observed symmetric matrix \({\bf Y}\), i.e.,

\[{\bf Y}_{ij}{=}{\bf Y}_{ji}{=}\left\{\begin{array}{ll}1&\quad\mbox{if ${\bf S}_{ ij}>\mu_{j}+p_{1}\sigma_{j}$}\\ &\quad\mbox{and ${\bf S}_{ji}>\mu_{i}+p_{1}\sigma_{i}$}\\ 0&\quad\mbox{if ${\bf S}_{ij}<\mu_{j}-p_{2}\sigma_{j}$}\\ &\quad\mbox{and ${\bf S}_{ji}<\mu_{i}-p_{2}\sigma_{i}$}\\ \mathrm{unobserved}&\mathrm{otherwise}\end{array}\right. \tag{4}\]

Matrix CompletionGiven the partially observed matrix \({\bf Y}\), we then reconstruct the full similarity matrix \({\bf X}\in\mathbb{R}^{n\times n}\). We first note that the similarity matrix \({\bf X}\) should be of low-rank (proof deferred to appendix). Additionally, since the observed entries of \({\bf Y}\) are generated based on high and low enough performance, it is safe to assume that most observed entries are correct and only a few may be incorrect. Therefore, we introduce a sparse matrix \({\bf E}\) to capture the observed incorrect entries in \({\bf Y}\). Combining the two observations, \({\bf Y}\) can be decomposed into the sum of two matrices \({\bf X}\) and \({\bf E}\), where \({\bf X}\) is a low rank matrix storing similarities between task pairs, and \({\bf E}\) is a sparse matrix that captures the errors in \({\bf Y}\). The matrix completion problem can be cast as the following convex optimization problem:

\[\min_{{\bf X},~{}{\bf E}} \|{\bf X}\|_{*}+\lambda\|{\bf E}\|_{1}\] s.t. \[{\bf P}_{\Omega}({\bf X}+{\bf E})={\bf P}_{\Omega}({\bf Y}),\]

where \(\|\circ\|_{*}\) denotes the matrix nuclear norm, the convex surrogate of rank function. \(\Omega\) is the set of observed entries in \({\bf Y}\), and \({\bf P}_{\Omega}:\mathbb{R}^{n\times n}\mapsto\mathbb{R}^{n\times n}\) is a matrix projection operator defined as

\[[{\bf P}_{\Omega}({\bf A})]_{ij}=\left\{\begin{array}{ll}{\bf A}_{ij}&\quad \mbox{if $(i,j)\in\Omega$}\\ 0&\quad\mbox{otherwise}\end{array}\right.\]

Finally, we apply spectral clustering on the matrix \({\bf X}\) to get the task clusters.

Remark: Sample EfficiencyIn the Appendix A, we show a Theorem 7.1 as well as its proof, implying that under mild conditions, the problem (5) can perfectly recover the underlying similarity matrix \({\bf X}^{*}\) if the number of observed correct entries is at least \(O(n\log^{2}n)\). This theoretical guarantee implies that for a large number \(n\) of training tasks, only a tiny fraction of all task pairs is needed to reliably infer similarities over all task pairs.

### Few-Shot Learning with Task Clusters

#### 3.2.1 Training Cluster Encoders

For each cluster \(C_{k}\), we train a multi-task MNet model (Figure 1(b)) with all tasks in that cluster to encourage parameter sharing. The result, denoted as \(f_{k}\) is called the **cluster-encoder** of cluster \(C_{k}\). The \(k\)-th metric of the cluster is thus \(\Lambda(x_{1},x_{2})=f_{k}(x_{1})^{\mathsf{T}}f_{k}(x_{2})\).

#### 3.2.2 Adapting Multiple Metrics for Few-Shot Learning

To build a predictor \(\operatorname{M}\) with access to only a limited number of training samples, we make the prediction probability by linearly combining prediction from learned cluster-encoders:

\[p(y|x)=\sum_{k}\alpha_{k}P(y|x;f_{k}).\] 

where \(f_{k}\) is the learned (and frozen) encoder of the \(k\)-th cluster, \(\{\alpha_{k}\}^{K}_{k=1}\) are adaptable parameters trained with few-shot training examples. And the predictor \(P(y|x;f_{k})\) from each cluster is

\[P(y=y_{l}|x;f_{k})=\frac{\exp\left\{f_{k}(x_{l})^{\intercal}f_{k}(x)\right\}}{ \sum_{i}\exp\left\{f_{k}(x_{i})^{\intercal}f_{k}(x)\right\}} \tag{7}\]

\(x_{l}\) is the corresponding training sample of label \(y_{l}\).

**Remark: Joint Method versus Pipeline Method** End-to-end joint optimization on training data becomes a popular methodology for deep learning systems, but it is not directly applicable to diverse FSL. One main reason is that deep networks could easily fit any task partitions if we optimize on training loss only, making the learned metrics not generalize, as discussed in Section 6. As a result, this work adopts a pipeline training approach and employing validation sets for task clustering. Combining reinforcement learning with meta-learning could be a potential solution to enable an end-to-end training for future work.

## 4 Tasks and Data Sets

We test our methods by conducting experiments on two text classification data sets. We used NLTK toolkit3 for tokenization. The task are divided into meta-training tasks and meta-testing tasks (target tasks), where the meta-training tasks are used for clustering and cluster-encoder training. The meta-testing tasks are few-shot tasks, which are used for evaluating the method in Eq. (6).

Footnote 3: [http://www.nltk.org/](http://www.nltk.org/)

### Amazon Review Sentiment Classification

First, following Barzilai and Crammer (2015), we construct multiple tasks with the multi-domain sentiment classification Blitzer et al. (2007) data set. The dataset consists of Amazon product reviews for 23 types of products (see Appendix D for the details). For each product domain, we construct three binary classification tasks with different thresholds on the ratings: the tasks consider a review as positive if it belongs to one of the following buckets \(=5\) stars, \(>=4\) stars or \(>=2\) stars.4 These buckets then form the basis of the task-setup, giving us 23 \(\times\) 3=69 tasks in total. For each domain we distribute the reviews uniformly to the 3 tasks. For evaluation, we select 12 (4\(\times\)3) tasks from 4 domains (_Books, DVD, Electronics, Kitchen_) as the meta-testing (target) tasks out of all 23 domains. For the target tasks, we create 5-shot learning problems.

Footnote 4: Data downloaded from [http://www.cs.jhu.edu/~mdredze/datasets/sentiment/](http://www.cs.jhu.edu/~mdredze/datasets/sentiment/), in which the 3-star samples were unavailable due to their ambiguous nature Blitzer et al. (2007).

### Real-World Tasks: User Intent Classification for Dialog System

The second dataset is from an online service which trains and serves intent classification models to various clients. The dataset comprises recorded conversations between human users and dialog systems in various domains, ranging from personal assistant to complex service-ordering or customer-service request scenarios. During classification, intent-labels5 are assigned to user utterances (sentences). We use a total of 175 tasks from different clients, and randomly sample 10 tasks from them as our target tasks. For each meta-training task, we randomly sample 64% data into a training set, 16% into a validation set, and use the rest as the test set. The number of labels for these tasks varies a lot (from 2 to 100, see Appendix D for details), making regular \(k\)-shot settings not essentially limited-resource problems (e.g., 5-shot on 100 classes will give a good amount of 500 training instances). Hence, to adapt this to a FSL scenario, for target tasks we keep one example for each label (one-shot), plus 20 randomly picked labeled examples to create the training data. We believe this is a fairly realistic estimate of labeled examples one client could provide easily.

Footnote 5: In conversational dialog systems, intent-labels are used to guide the dialog-flow.

Remark: Evaluation of the Robustness of Algorithm 2Our matrix-completion method could handle a large number of tasks via task-pair sampling. However, the sizes of tasks in the above two few-shot learning datasets are not too huge, so evaluation of the whole task-similarity matrix is still tractable. In our experiments, the incomplete matrices mainly come from the score-filtering step (see Eq. 4). Thus there is limited randomness involved in the generation of task clusters.

To strengthen the conclusion, we evaluate our algorithm on an additional dataset with a much larger number of tasks. The results are reported in the multi-task learning setting instead of the few-shot learning setting focused in this paper. Therefore we put the results to a non-archive version of 

this paper6 for further reference.

Footnote 5: [https://arxiv.org/pdf/1708.07918.pdf](https://arxiv.org/pdf/1708.07918.pdf)

## 5 Experiments

### Experiment Setup

BaselinesWe compare our method to the following baselines: (1) **Single-task CNN**: training a CNN model for each task individually; (2) **Single-task FastText**: training one FastText model Joulin et al. (2016) with fixed embeddings for each individual task; (3) **Fine-tuned the holistic MTL-CNN**: a standard transfer-learning approach, which trains one MTL-CNN model on all the training tasks offline, then fine-tunes the classifier layer (i.e. \({\rm M}^{(cls)}\) Figure 1(a)) on each target task; (4) **Matching Network**: a metric-learning based few-shot learning model trained on all training tasks; (5) **Prototypical Network**: a variation of matching network with different prediction function as Eq. 3; (6) **Convex combining all single-task models**: training one CNN classifier on each meta-training task individually and taking the encoder, then for each target task training a linear combination of all the above single-task encoders with Eq. (6). This baseline can be viewed as a variation of our method without task clustering. We initialize all models with pre-trained 100-dim Glove embeddings (trained on 6B corpus) Pennington et al. (2014).

Hyper-Parameter TuningIn all experiments, we set both \(p_{1}\) and \(p_{2}\) parameters in (4) to \(0.5\). This strikes a balance between obtaining enough observed entries in **Y**, and ensuring that most of the retained similarities are consistent with the cluster membership. The window/hidden-layer sizes of CNN and the initialization of embeddings (random or pre-trained) are tuned during the cluster-encoder training phase, with the validation sets of meta-training tasks. We have the CNN with window size of 5 and 200 hidden units. The single-metric FSL baselines have 400 hidden units in the CNN encoders. On sentiment classification, all cluster-encoders use random initialized word embeddings for sentiment classification, and use Glove embeddings as initialization for intent classification, which is likely because the training sets of the intent tasks are usually small.

Since all the sentiment classification tasks are binary classification based on our dataset construction. A CNN classifier with binary output layer can be also trained as the cluster-encoder for each task cluster. Therefore we compared CNN classifier, matching network, and prototypical network on Amazon review, and found that CNN classifier performs similarly well as prototypical network. Since some of the Amazon review data is quite large which involves further difficulty on the computation of supporting sets, we finally use binary CNN classifiers as cluster-encoders in all the sentiment classification experiments.

Selection of the learning rate and number of training epochs for FSL settings, i.e., fitting \(\alpha\)s in Eq. (6), is more difficult since there is no validation data in few-shot problems. Thus we pre-select a subset of meta-training tasks as meta-validation tasks and tune the two hyper-parameters on the meta-validation tasks.

### Experimental Results

Table 1 shows the main results on (i) the 12 few-shot product sentiment classification tasks by leveraging the learned knowledge from the 57 previously observed tasks from other product domains; and (ii) the 10 few-shot dialog intent classification tasks by leveraging the 165 previously observed tasks from other clients' data.

Due to the limited training resources, all the supervised-learning baselines perform poorly. The two state-of-the-art metric-based FSL approaches, matching network (4) and prototypical network (5), do not perform better compared to the other baselines, since the single metric is not sufficient for all the diverse tasks. On intent classification where tasks are further diverse, all the single-metric or single-model methods (3-5) perform worse compared to the single-task CNN baseline (1). The convex combination of all the single training task models is the best performing base-line overall. However, on intent classification it only performs on par with the single-task CNN (1), which does not use any meta-learning or transfer learning techniques, mainly for two reasons: (i) with the growth of the number of meta-training tasks, the model parameters grow linearly, making the number of parameters (165 in this case) in Eq.(6) too large for the few-shot tasks to fit; (ii) the meta-training tasks in intent classification usually contain less training data, making the single-task encoders not generalize well.

In contrast, our RobustTC-FSL gives consistently better results compared to all the baselines.

 

It outperforms the baselines in previous work (1-5) by a large margin of more than 6% on the sentiment classification tasks, and more than 3% on the intent classification tasks. It is also significantly better than our proposed baseline (6), showing the advantages of the usage of task clustering.

Adaptive RobustTC-FSLAlthough the RobustTC-FSL improves over baselines on intent classification, the margin is smaller compared to that on sentiment classification, because the intent classification tasks are more diverse in nature. This is also demonstrated by the training accuracy on the target tasks, where several tasks fail to find any cluster that could provide a metric that suits their training examples. To deal with this problem, we propose an improved algorithm to automatically discover whether a target task belongs to none of the task-clusters. If the task doesn't belong to any of the clusters, it cannot benefit from any previous knowledge thus falls back to single-task CNN. The target task is treated as "out-of-clusters" when none of the clusters could achieve higher than 20% accuracy (selected on meta-validation tasks) on its training data. We call this method **Adaptive RobustTC-FSL**, which gives more than 5% performance boost over the best RobustTC-FSL result on intent classification. Note that the adaptive approach makes no difference on the sentiment tasks, because they are more closely related so re-using cluster-encoders always achieves better results compared to single-task CNNs.

### Analysis

Effect of the number of clustersFigure 3 shows the effect of cluster numbers on the two tasks. RobustTC achieves best performance with 5 clusters on sentiment analysis (SA) and 20 clusters on intent classification (Intent). All clustering results significantly outperform the single-metric baselines (#cluster=1 in the figure).

Effect of the clustering algorithmsCompared to previous task clustering algorithms, our RobustTC is the only one that can cluster tasks with varying numbers of class labels (e.g. in intent classification tasks). Moreover, we show that even in the setting of all binary classifications tasks (e.g. the sentiment-analysis tasks) that previous task clustering research work on, our RobustTC is still slightly better for the diverse FSL problems. Figure 3 compares with a state-of-the-art logistic regression based task clustering method (**ASAPMT-LR**) Barzilai and Crammer (2015). Our RobustTC clusters give slightly better FSL performance (e.g. 83.12 vs. 82.65 when #cluster=5).

\begin{table}
\begin{tabular}{l|c c} \hline  & \multicolumn{2}{c}{**Avg Acc**} \\
**Model** & **Sentiment** & **Intent** \\ \hline (1) Single-task CNN w/pre-trained emb & 65.92 & 34.46 \\ (2) Single-task FastText w/pre-trained emb & 63.05 & 23.87 \\ (3) Fine-tuned holistic MTL-CNN & 76.56 & 30.36 \\ (4) Matching Network Vinyals et al. (2016) & 65.73 & 30.42 \\ (5) Prototypical Network Snell et al. (2017) & 68.15 & 31.51 \\ \hline (6) Convex combination of all single-task models & 78.85 & 34.43 \\ \hline \hline
**RobustTC-FSL** & **83.12** & **37.59** \\ \hline
* & **42.97** \\ \hline \end{tabular}
\end{table}
Table 1: Accuracy of FSL on sentiment classification (Sentiment) and dialog intent classification (Intent) tasks. The target tasks of sentiment classification are 5-shot ones; and each intent target task contains one training example per class and 20 random labeled examples.

Figure 3: Effect of clusters. RobustTC-SA and RobustTC-Intent: the performance of our RobustTC clusters on the sentiment and intent classification tasks. ASAP-MT-LR-SA: the state-of-the-art ASAP-MT-LR clusters on the sentiment-analysis tasks (the method is not applicable to the intent-classification tasks).

 

Visualization of Task ClustersThe top rows of Table 2 shows the ten clusters used to generate the sentiment classification results in Figure 3. From the results, we can see that tasks with same thresholds are usually grouped together; and tasks in similar domains also tend to appear in the same clusters, even the thresholds are slightly different (e.g. t2 vs t4 and t4 vs t5).

The bottom of the table shows the weights \(\alpha\)s in Eq.(6) for the target tasks with the largest improvement. It confirms that our RobustTC-FSL algorithm accurately adapts multiple metrics for the target tasks.

## 6 Related Work

Few Shot LearningFSL Li et al. (2006); Miller et al. (2000) aims to learn classifiers for new classes with only a few training examples per class. Bayesian Program Induction Lake et al. (2015) represents concepts as simple programs that best explain observed examples under a Bayesian criterion. Siamese neural networks rank similarity between inputs Koch (2015). Matching Networks Vinyals et al. (2016) map a small labeled support set and an unlabeled example to its label, obviating the need for fine-tuning to adapt to new class types. These approaches essentially learn one metric for all tasks, which is sub-optimal when the tasks are diverse. An LSTM-based meta-learner Ravi and Larochelle (2017) learns the exact optimization algorithm used to train another learner neural-network classifier for the few-shot setting.

Previous FSL research usually adopts the \(k\)-shot, \(N\)-way setting, where all the few-shot tasks have the same number of \(N\) class labels, and each label has \(k\) training instances. Moreover, these few-shot tasks are usually constructed by sampling from one huge dataset, thus all the tasks are guaranteed to be related to each other. However, in real-world applications, the few-shot learning tasks could be diverse: there are different tasks with varying number of class labels and they are not guaranteed to be related to each other. As a result, a single meta-model or metric-model is usually not sufficient to handle all the few-shot tasks.

Task ClusteringPrevious task clustering methods measure the task relationships in terms of similarities among single-task model parameters Kumar and Daume III (2012); Kang et al. (2011); or jointly assign task clusters and train model parameters for each cluster to minimize the overall training loss Crammer and Mansour (2012); Barzilai and Crammer (2015); Murugesan et al. (2017). These methods usually work on convex models but do not fit the deep networks, mainly because of (i) the parameters of deep networks are very high-dimensional and their similarities are not necessarily related to the functional similarities; and (ii) deep networks have flexible representation power so they may overfit to arbitrary cluster assignment if we consider training loss alone. Moreover, these methods require identical class label sets across different tasks, which does not hold in most of the realistic settings.

## 7 Conclusion

We propose a few-shot learning approach for diverse tasks based on task clustering. The proposed method can use multiple metrics, and performs significantly better compared to previous single-metric based methods when the few-shot tasks come from diverse domains. Future work includes generalizing our method to non-NLP problems, as well as applying the task-clustering idea to other few-shot learning frameworks Ravi and Larochelle (2017); Finn et al. (2017); Mishra et al. (2017); Cheng et al. (2017).

\begin{table}
\begin{tabular}{c|l l|l|l|l|l|l|l|l|l} \hline  & **Clab** & **Class** & **Class** & **Class** & **Class** & **Class** & **Class** \\ \hline  & mincutance:t2 & dropout:1.2 & babsy:0.5 & maxmax:0 & upper:1.5 & base:1.4 & cutmax:t.9 & current:5.5 & opt:1.8 & output:t.4 \\  & camera:2 & monotype:t4 & magnitues:t5 & busy:t4 & camera:5 & bump:1.5 & software:2.2 & magnitues:t4 & software:4.5 & twys:t2 \\  & health:2 & absy:2 & spont:5 & passy:5 & opt:1.5 & soft/ece:4 & music:4 & inspyt:4 & \\  & magazines:t2 & cell:2 & wys:5 & health:5 & pseudo:5 & gummet:2 & music:5 & & \\  & office:2 & computer:2 & video:4 & window:5 & system:4 & wildo:4 & & & \\  & outdoor:2 & computer:4 & & & & groovy:4 & & & \\  & sports:t4 & people:4 & & & & & & & \\  & sports:t4 & jewards:5 & & & & & & & \\  & smoke:2 & & & & & & & & \\  & video:2 & & & & & & & & \\ \hline
**all-04** & 0.444 & 0.4416 & 0.4452 & **0.7843** & 0.7979 & 0.7196 & **0.8932** & 0.7363 & 0.7155 & 0.6315 \\
**divd+05** & 0.4411 & 0.2493 & **0.5807** & 0.1686 & -0.0355 & **0.4150** & -0.2603 & -0.0867 & 0.0547 \\
**lidchrone+4** & 0.6823 & 0.7268 & 0.7929 & **1.2669** & **1.1119** & 0.7255 & **1.2196** & 0.7065 & 0.6625 & 1.0945 \\ \hline \end{tabular}
\end{table}
Table 2: Visualization of clusters on the Amazon review domain. The top shows the training tasks assigned to the 10 clusters. Here the number N\(\in\{2,4,5\}\) refers to the threshold of stars for positive reviews. At the bottom we show three tasks with largest improvement from RobustTC-FSL. The top-3 most relevant task clusters (i.e. with highest weights \(\alpha\)s in Eq.6 ) are highlighted with blue bold font.

 



## References

* Barzilai and Crammer (2015) Aviad Barzilai and Koby Crammer. 2015. Convex multi-task learning by clustering. In _AISTATS_.
* Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In _ACL_, volume 7, pages 440-447.
* Candes and Tao (2010) Emmanuel J Candes and Terence Tao. 2010. The power of convex relaxation: Near-optimal matrix completion. _IEEE Transactions on Information Theory_, 56(5):2053-2080.
* Chandrasekaran et al. (2011) Venkat Chandrasekaran, Sujay Sanghavi, Pablo A Parrilo, and Alan S Willsky. 2011. Rank-sparsity incoherence for matrix decomposition. _SIAM Journal on Optimization_, 21(2):572-596.
* Cheng et al. (2017) Yu Cheng, Mo Yu, Xiaoxiao Guo, and Bowen Zhou. 2017. Few-shot learning with meta metric learners. In _NIPS 2017 Workshop on Meta-Learning_.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12(Aug):2493-2537.
* Crammer and Mansour (2012) Koby Crammer and Yishay Mansour. 2012. Learning multiple tasks using shared hypotheses. In _Advances in Neural Information Processing Systems_, pages 1475-1483.
* Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. _arXiv preprint arXiv:1703.03400_.
* Johnson and Zhang (2016) Rie Johnson and Tong Zhang. 2016. Supervised and semi-supervised text categorization using one-hot lstm for region embeddings. _stat_, 1050:7.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. _arXiv preprint arXiv:1607.01759_.
* Kang et al. (2011) Zhuoliang Kang, Kristen Grauman, and Fei Sha. 2011. Learning with whom to share in multi-task feature learning. In _Proceedings of the 28th International Conference on Machine Learning (ICML-11)_, pages 521-528.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _EMNLP_, pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
* Koch (2015) Gregory Koch. 2015. _Siamese neural networks for one-shot image recognition_. Ph.D. thesis, University of Toronto.
* Kumar and Daume III (2012) Abhishek Kumar and Hal Daume III. 2012. Learning task grouping and overlap in multi-task learning. In _Proceedings of the 29th International Conference on Machine Learning (ICML-12)_.
* Lake et al. (2011) Brenden M Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B Tenenbaum. 2011. One shot learning of simple visual concepts. In _CogSci_, volume 172, page 2.
* Lake et al. (2015) Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. _Science_, 350(6266):1332-1338.
* Li et al. (2006) Fei-Fei Li, Rob Fergus, and Pietro Perona. 2006. Oneshot learning of object categories. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 28(4):594-611.
* Miller et al. (2000) Erik G Miller, Nicholas E Matsakis, and Paul A Viola. 2000. Learning from one example through shared densities on transforms. In _Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on_, volume 1, pages 464-471. IEEE.
* Mishra et al. (2017) Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. 2017. A simple neural attentive metalearner. In _NIPS 2017 Workshop on Meta-Learning_.
* Munkhdalai and Yu (2017) Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. _arXiv preprint arXiv:1703.00837_.
* Murugesan et al. (2017) Keerthiam Murugesan, Jaime Carbonell, and Yiming Yang. 2017. Co-clustering for multitask learning. _arXiv preprint arXiv:1703.00994_.
* Ng et al. (2002) Andrew Y Ng, Michael I Jordan, and Yair Weiss. 2002. On spectral clustering: Analysis and an algorithm. In _Advances in neural information processing systems_, pages 849-856.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, volume 14, pages 1532-1543.
* Ravi and Larochelle (2017) Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In _International Conference on Learning Representations_, volume 1, page 6.
* Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. _arXiv preprint arXiv:1703.05175_.
* Triantafillou et al. (2017) Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. 2017. Few-shot learning through an information retrieval lens. In _Advances in Neural Information Processing Systems_, pages 2252-2262.
* Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In _Advances in Neural Information Processing Systems_, pages 3630-3638.
* Wang et al. (2017) Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. 2017. Learning to model the tail. In _Advances in Neural Information Processing Systems 30_, pages 7032-7042