Venue: NAACL 2021

# Knowledge Guided Metric Learning for Few-Shot Text Classification

Dianbo Sui\({}^{1,2}\), Yubo Chen\({}^{1,2}\), Binjie Mao\({}^{1,2}\), Delai Qiu\({}^{3}\), Kang Liu\({}^{1,2}\), Jun Zhao\({}^{1,2}\)

\({}^{1}\) National Laboratory of Pattern Recognition, Institute of Automation, CAS \({}^{2}\) School of Artificial Intelligence, University of Chinese Academy of Sciences \({}^{3}\) Beijing Unisound Information Technology Co., Ltd. {dianbo.sui, yubo.chen, binjie.mao, kliu, jzhao}@nlpr.ia.ac.cn qjudelai@unisound.com

###### Abstract

Humans can distinguish new categories very efficiently with few examples, largely due to the fact that human beings can leverage knowledge obtained from relevant tasks. However, deep learning based text classification model tends to struggle to achieve satisfactory performance when labeled data are scarce. Inspired by human intelligence, we propose to introduce external knowledge into few-shot learning to imitate human knowledge. A novel parameter generator network is investigated to this end, which is able to use the external knowledge to generate different metrics for different tasks. Armed with this network, similar tasks can use similar metrics while different tasks use different metrics. Through experiments, we demonstrate that our method outperforms the SoTA few-shot text classification models.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3266–3271 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Humans are adept at quickly learning from a small number of examples. This motivates research of few-shot learning [45, 8], which aims to recognize novel categories from very few labeled examples.

The key challenge in few-shot learning is to make full use of the limited labeled examples to find the "right" generalizations. Metric-based approaches [49, 28, 29, 30, 21] are effective ways to address this challenge, which learn to represent examples in an appropriate feature space and use a distance metric to predict labels. However, directly employing metric-based approaches in text classification faces a problem that tasks are diverse and significantly different from each other, since words that are highly informative for one task may not be relevant for other tasks [1]. Therefore, a single metric is insufficient to cope with all these tasks in few-shot text classification [49].

To adapt metric learning to significantly diverse tasks, we propose a knowledge guided metric learning method. This method is inspired by the fact that human beings approach diverse tasks armed with knowledge obtained from relevant tasks [13]. We use external knowledge from the knowledge base (KB) to imitate human knowledge, whereas the role of external knowledge has been ignored in previous methods [49, 1, 10]. In detail, we resort to distributed representations of the KB instead of symbolic facts, since symbolic facts face the issues of poor generalization and data sparsity. Based on such KB embeddings, we investigate a novel parameter generator network [1, 17] to generate task-relevant relation network parameters. With these generated parameters, the task-relevant relation network is able to apply diverse metrics to diverse tasks and ensure that similar tasks use similar metrics while different tasks use different metrics.

In summary, the major contributions of this paper are:

* Inspire by human intelligence, we present the first approach that introduces external knowledge into few-shot learning.
* A novel parameter generator network based on external knowledge is proposed to generate diverse metrics for diverse tasks.
* Experimental results on the public dataset show that our model significantly outperforms previous methods.

## 2 Problem Setting

Few-shot classification aims at training a model that can recognize novel classes from very few labeled examples. The training and testing of the model are conducted on two datasets (training set and test set) with no overlapped classes. At

the training and test stages, the labeled examples are called the support set, which serves as a meta-training set and the meta-testing examples are called the query set. If the support set contains \(N\) labeled examples for each of \(C\) unique classes, the few-shot problem is called \(C\)-way \(N\)-shot. To guarantee a good generalization performance at test time, the training and evaluation of the model are accomplished by episodically sampling the support set and the query set [25]. More concretely, in each meta-training iteration, an episode is formed by randomly selecting \(C\) classes from the training set with \(N\) labeled examples for each of the \(C\) classes to serve as the support set \({\cal S}=\left\{(x_{i},y_{i})\right\}^{C\times N}_{i=1}\), as well as a fraction of the remainder of those \(C\) classes' examples to act as the query set \({\cal Q}=\left\{(x_{i},y_{i})\right\}^{C\times N+m}_{i=C\times N+1}\), where \(x_{i}\) and \(y_{i}\in\left\{1,...,C\right\}\) are the sentence and its label, and \(m\) is the number of query samples. The model is trained on the support set \({\cal S}\) to minimize the loss of its predictions over the query set \({\cal Q}\). This training procedure is iteratively carried out episode by episode until convergence.

## 3 Methodology

### Sentence Embedding Network

In this network, a pre-trained BERT [40] encoder is used to model sentences. Given an input text \(x_{i}=([\mathrm{CLS}],w_{1},w_{2},...,w_{T},[\mathrm{SEP}])\) as input, the output of BERT encoder is denoted as \(\textbf{H}(x_{i})\in\mathbb{R}^{(T+2)\times d_{1}}\), where \(d_{1}\) is the output dimension of the BERT encoder. We use the first token of the sequence (classification token) as the sentence representation, which is denote as \(\textbf{h}(x_{i})\).

In meta-learning, the representation of each class is the mean vector of the embedded sentences belonging to its class,

\[\mathbf{c}_{z}=\frac{1}{|{\cal S}_{z}|}\sum_{(x_{i},y_{i})\in{\cal S}_{z}} \textbf{h}(x_{i})\in\mathbb{R}^{d_{1}} \tag{1}\]

where \({\cal S}_{z}\) denotes the set of sentences labeled with class z. Following Sung et al. (2018), we use concatenation operator to combine the query representation \(\textbf{h}(x_{j})\) with the class representation \(\textbf{c}_{z}\).

\[\begin{split}&\textbf{p}_{z,j}=\operatorname*{concatenation}( \mathbf{c}_{z},\textbf{h}(x_{j}))\in\mathbb{R}^{2d_{1}}\\ \end{split} \tag{2}\]

### Knowledge Guided Relation Network

This module takes combined representation (shown in Equation 2) and the knowledge of the support set as input, and produces a scalar in range of \(0\) to \(1\) representing the similarity between the query sentence and the class representation, which is called relation score. Compared with the original relation network [69], we decompose the relation network into two parts, task-agnostic relation network and task-relevant relation network, in order to serve two purposes. Task agnostic relation network models a basic metric function, while task-relevant relation network adapts to diverse tasks.

Task-Agnostic Relation NetworkThe task-agnostic relation network uses a learned unified metric for all tasks, which is the same with the original relation network [69]. With this unified metric, \(C\) task-agnostic relation scores

Figure 1: The main architecture for a \(C\)-way \(N\)-shot (\(C\) = \(3\), \(N\) = \(2\)) problem with one query example.

 

are generated for modeling the relation between one query input \(x_{j}\) and the class representation \(c_{z}\),

\[r^{agn}_{s,j}=RN^{sgn}(\textbf{p}_{z,j}|\boldsymbol{\theta}^{sgn})\in\mathbb{R}, \quad z=1,2,...,C \tag{3}\]

where \(RN^{agn}\) denotes task-agnostic relation network and \(\boldsymbol{\theta}^{agn}\) are learnable parameters.

Task-Relevant Relation NetworkThe task-relevant relation network is able to apply diverse metrics for diverse tasks armed with external knowledge. In detail, for each support set \(\mathcal{S}\) (\(\mathcal{S}\) contains \(C\times N\) labeled sentences), we retrieve a set of potentially relevant KB concepts \(K(\mathcal{S})\), where each concept \(k_{i}\) is associated with KB embedding \(\mathbf{e}_{i}\in\mathbb{R}^{d_{2}}\). (we will describe these processes in the following section). We average over these KB embeddings element by element to form the knowledge representation of this support set.

\[\mathbf{k}_{S}=\frac{1}{|K(\mathcal{S})|}\sum_{k_{i}\in K(\mathcal{S})} \mathbf{e}_{i}\in\mathbb{R}^{d_{2}} \tag{4}\]

Then we use this knowledge representation to generate task-relevant relation network parameters,

\[\boldsymbol{\theta}^{rel}=\mathbf{M}\cdot\mathbf{k}_{S}\in\mathbb{R}^{d_{3}} \tag{5}\]

where \(\mathbf{M}\in\mathbb{R}^{d_{3}\times d_{2}}\) are learnable parameters and \(d_{3}\) denotes the number of parameters of the task-relevant relation network.

With these generated parameters, we use the task-relevant network to generate \(C\) task-relevant relation scores \(r^{rel}_{z,j}\) for the relation between one query input \(x_{j}\) and the class representation \(c_{z}\),

\[r^{rel}_{z,j}=RN^{rel}(\textbf{p}_{z,j}|\boldsymbol{\theta}^{rel})\in\mathbb{ R},\quad z=1,2,...,C \tag{6}\]

where \(RN^{rel}\) denotes task-relevant relation network. Finally, relation score is defined as:

\[r_{z,j}=Signoid(r^{agn}_{z,j}+r^{rel}_{z,j}) \tag{7}\]

where a sigmoid function is used to keep the score in a reasonable range. Following Sung et al. (2018), the network architecture of relation networks is two full-connected layers and mean square error (MSE) loss is used to train the model. The relation score is regressed to the ground truth: the matched pairs have similarity \(1\) and the mismatched pairs have similarity \(0\).

\[L=\sum_{z=1}^{C}\sum_{j=1}^{|\mathcal{Q}|}(r_{z,j}-\textbf{1}(y_{j}==z)) \tag{8}\]

### Knowledge Embedding and Retrieval

We use NELL Carlson et al. (2010) as the KB, stored as (subject, relation, object) triples, where each triple is a fact indicating a specific relation between subject and object, e.g., (Intel, competes with, Nvidia).

Knowledge EmbeddingSince symbolic facts suffer from poor generalization and data sparsity, we resort to distributed a representation of triples. In detail, given any triple \((s,r,o)\), vector embeddings of the subject \(s\), the relation \(r\) and the object \(o\) are learned jointly such that the validity of the triple can be measured in the real number space. We adopt the BILINEAR model Yang et al. (2015) to measure the validity of triples:

\[f(s,r,o)=\textbf{s}^{T}diag(\textbf{r})\textbf{o}\in\mathbb{R} \tag{9}\]

where \(\textbf{s},\textbf{r},\textbf{o}\in\mathbb{R}^{d_{2}}\) are the embeddings associated with \(s\), \(r\), \(o\), respectively, and \(diag(\textbf{r})\) is a diagonal matrix with the main diagonal given by the relation embedding **r**. To learn these vector embeddings, a margin-based ranking loss is designed, where triples in the KB are adopted to be positive and negative triples are constructed by corrupting either subjects or objects.

Knowledge RetrievalInspired by the previous studies Yang and Mitchell (2017); Yang et al. (2019), exact string matching Charras and Lecroq (2004) is used to recognize entity mentions from a given passage and link recognized entity mentions to subjects in KB. Then, we collect the corresponding objects (concepts) as candidates. After this retrieval process, we obtain a set of potentially relevant KB concepts, where each KB concept is associated with a KB embedding.

## 4 Experiment

### Dataset

Our model is evaluated on the widely used ARSC Blitzer et al. (2007) dataset, which comprises reviews for 23 types of products on Amazon. For each product domain, there are three different binary classification tasks. These buckets form 69 tasks in total. Following Yu et al. (2018), we select 12 tasks from four domains Books, DVDs, Electronics, and Kitchen) as testing set, with only 5 examples as support set for each class 



### Implementation Details

In our experiments, we use hugginface's implementation1 of BERT (base version) and initialize parameters of the BERT encoding layer with pre-trained models officially released by Google2. To represent knowledge in NELL Carlson et al. (2010), BILIN-EAR model Yang et al. (2015) is implemented with the open-source framework OpenKE Han et al. (2018) to obtain the embedding of entities and relations. The size of embeddings of entities and relations is set to 100. To train our model, We use Adam optimizer Kingma and Ba (2014) with a learning rate of 0.00001. All experiments are run with an NVIDIA GeForce RTX 2080 Ti.

Footnote 1: [https://huggingface.co/transformers](https://huggingface.co/transformers)

Footnote 2: [https://github.com/google-research/bert](https://github.com/google-research/bert)

### Experiment Results

Baseline.We compare our method to the following baselines: (1) **Match Network** is a metric-based attention method for few-shot learning; (2) **Prototypical Network** is a metric-based method that uses sample averages as class prototypes; (3) **MAML** is an optimization-based method through learning to learn with gradients; (4) **Relation Network** is a metric-based method that leverages two full-connected layers as the distance metric and sums up sample vectors in the support set as class vectors; (5) **Graph Network** is a graph-based model that implements a task-driven message passing algorithm on the sample-wise level; (6) **ROBUSTTC-FSL** is an approach that combines adaptive metric methods by clustering the tasks; (7) **Induction Network** is a metric-based method by using dynamic routing to learn class-wise representations.

Analysis.Experiment results on ARSC are presented in Table 1. We observe that our method achieves the best results amongst all meta-learning models. Both Induction Network and Relation Network use a single metric to measure the similarity. Compared with these methods, we attribute the improvements of our model to the fact that our model can adapt to diverse tasks with diverse metrics. Compared with ROBUSTTC-FSL, our model leverages knowledge to get implicit task clusters and is trained in an end-to-end manner, which can mitigate error propagation.

### Effectiveness of Introducing Knowledge

To analyze the contributions and effects of external knowledge in our approach, we perform some ablation and replacement studies, which are shown in Table 2. **Ablation** means that we delete the task-relevant relation network and the model is reduced to the original BERT-based relation network. We observe that ablation degrades performance. To exclude the factor of reduction in the number of parameters, we conduct a **replacement** experiment, in which we replace the task-relevant relation network with a task-agnostic relation network. We find that increasing the number of parameters can slightly improve performance, but there is still a big gap between our model. Therefore, we conclude that the effectiveness of our model is credited with introducing external knowledge rather than increasing the number of model parameters.

### Different Strategies of Introducing Knowledge

To analyze different strategies of introducing knowledge in few-shot learning, we remove the task-relevant relation network, and replace the BERT encoder in our method with **KT-NET** encoder Yang et al. (2019) and **K-BERT** encoder Liu et al. (2019). In the KT-NET encoder, an attention mechanism is used to adaptively fuse selected knowledge with BERT. In the K-BERT encoder, a knowledge-rich sentence tree is the input of the model. These methods both introduce knowledge

\begin{table}
\begin{tabular}{c c} \hline \hline Model & Mean Acc \\ \hline Matching Network Vinyals et al. (2016) & 65.73 \\ Prototypical Network Snell et al. (2017) & 68.15 \\ MAML Finn et al. (2017) & 78.33 \\ Graph Network Garcia and Bruna (2017) & 82.61 \\ Relation Network Sung et al. (2018) & 83.07 \\ ROBUSTTC-FSL Yu et al. (2018) & 83.12 \\ Induction Network Geng et al. (2019) & 85.63 \\
**Ours** & **87.93** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of mean accuracy (%) on ARSC dataset. The scores of baselines are taken from Geng et al. (2019).

\begin{table}
\begin{tabular}{c c} \hline \hline Model & Mean Acc \\ \hline Ours & 87.93 \\ Ablation & 86.09 (\(\downarrow\) 1.84) \\ Replacement & 86.40 (\(\downarrow\) 1.53) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation and replacement studies of our model on ARSC dataset.



at the representation level 3, while our method injects knowledge at the task level. The result is shown in Table 3. Combined Table 2 and Table 3, we find that (1) introducing knowledge can improve the performance of few-shot text classification; (2) it is more effective to introduce knowledge at the task level rather than at the representation level.

Footnote 3: Knowledge is used to enhance the sentence representation.

## 5 Conclusion

Inspired by human intelligence, we introduce external knowledge into few-shot learning. A parameter generator network is investigated to this end, which can use external knowledge to generate relation network parameters. With these parameters, the relation network can handle diverse tasks with diverse metric. Through various experiments, we demonstrate the effectiveness of our model.

## Acknowledgments

We thank the anonymous reviewers for their insightful comments. We also thank Yushan Xie and Zhixing Tian for helpful suggestions. This work is supported by the National Key RD Program of China (Grant No. 2020AAA0106400), the National Natural Science Foundation of China (Grant No. 61976211 and Grant No. 61806201) and the Key Research Program of the Chinese Academy of Sciences (Grant No. ZDBS-SSW-JSC006).

## References

* Bao et al. (2019) Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2019. Few-shot text classification with distributional signatures. _arXiv preprint arXiv:1908.06039_.
* Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood boom-boxes and blenders: Domain adaptation for sentiment classification. In _Proceedings of the 45th annual meeting of the association of computational linguistics_.
* Carlson et al. (2010) Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, and Tom M Mitchell. 2010. Toward an architecture for never-ending language learning. In _Twenty-Fourth AAAI Conference on Artificial Intelligence_.
* Charras and Lecroq (2004) Christian Charras and Thierry Lecroq. 2004. Handbook of exact string matching algorithms. In _Citeseer_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In _Proceedings of the 34th International Conference on Machine Learning_.
* Garcia and Bruna (2017) Victor Garcia and Joan Bruna. 2017. Few-shot learning with graph neural networks. _arXiv preprint arXiv:1711.04043_.
* Geng et al. (2020) Ruying Geng, Binhua Li, Yongbin Li, Jian Sun, and Xiaodan Zhu. 2020. Dynamic memory induction networks for few-shot text classification. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Geng et al. (2019) Ruying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_.
* Ha et al. (2016) David Ha, Andrew Dai, and Quoc V Le. 2016. Hypernetworks. _arXiv preprint arXiv:1609.09106_.
* Han et al. (2018) Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. OpenKE: An open toolkit for knowledge embedding. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
* Jia et al. (2016) Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. 2016. Dynamic filter networks. In _Advances in Neural Information Processing Systems_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. _Behavioral and brain sciences_.
* Li et al. (2019) Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. 2019. Revisiting local descriptor based image-to-class measure for few-shot learning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
* Lu et al. (2019)

\begin{table}
\begin{tabular}{c c} \hline \hline Model & Mean Acc \\ \hline Ours & 87.93 \\ K-BERT [Liu et al., 2019] & 86.42 (\(\downarrow\) 1.51) \\ KT-NET [Yang et al., 2019] & 86.28 (\(\downarrow\) 1.65) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Different strategies of introducing knowledge on ARSC dataset.

 

Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019. K-bert: Enabling language representation with knowledge graph. _arXiv_, pages arXiv-1909.
* Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In _Advances in Neural Information Processing Systems_.
* Sung et al. (2018) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
* Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In _Advances in neural information processing systems_.
* Yang et al. (2019) An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li. 2019. Enhancing pre-trained language representations with rich knowledge for machine reading comprehension. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
* Yang and Mitchell (2017) Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in lstms for improving machine reading. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_.
* Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In _International Conference on Learning Representations_.
* Yu et al. (2018) Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Zhang et al. (2020) Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. 2020. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_