Venue: NAACL 2019

# Continual Learning for Sentence Representations Using Conceptors

Tianlin Liu

Department of Computer Science and

Electrical Engineering

Jacobs University Bremen

28759 Bremen, Germany

t.liu@jacobs-university.de &Lyle Ungar and Joao Sedoc

Department of Computer and

Information Science

University of Pennsylvania

Philadelphia, PA 19104

{ungar, joao}@cis.upenn.edu

###### Abstract

Distributed representations of sentences have become ubiquitous in natural language processing tasks. In this paper, we consider a continual learning scenario for sentence representations: Given a sequence of corpora, we aim to optimize the sentence encoder with respect to the new corpus while maintaining its accuracy on the old corpora. To address this problem, we propose to initialize sentence encoders with the help of corpus-independent features, and then sequentially update sentence encoders using Boolean operations of _concept_ matrices to learn corpus-dependent features. We evaluate our approach on semantic textual similarity tasks and show that our proposed sentence encoder can continually learn features from new corpora while retaining its competence on previously encountered corpora.

## 1 Introduction

Distributed representations of sentences are essential for a wide variety of natural language processing (NLP) tasks. Although recently proposed sentence encoders have achieved remarkable results (e.g., Yin and Schutze (2015); Arora et al. (2017); Cer et al. (2018); Pagliardini et al. (2018)), most, if not all, of them are trained on _a priori_ fixed corpora. However, in open-domain NLP systems such as conversational agents, oftentimes we are facing a dynamic environment, where training data are accumulated sequentially over time and the distributions of training data vary with respect to external input Lee (2017); Mathur and Singh (2018). To effectively use sentence encoders in such systems, we propose to consider the following _continual sentence representation learning task_: Given a sequence of corpora, we aim to train sentence encoders such that they can continually learn features from new corpora while retaining strong performance on previously encountered corpora.

Toward addressing the continual sentence representation learning task, we propose a simple sentence encoder that is based on the summation and linear transform of a sequence of word vectors aided by matrix conceptors. Conceptors have their origin in reservoir computing Jaeger (2014) and recently have been used to perform continual learning in deep neural networks He and Jaeger (2018). Here we employ Boolean operations of conceptor matrices to update sentence encoders over time to meet the following desiderata:

1. _Zero-shot learning_. The initialized sentence encoder (no training corpus used) can effectively produce sentence embeddings.
2. _Resistant to catastrophic forgetting_. When the sentence encoder is adapted on a new training corpus, it retains strong performances on old ones.

The rest of the paper is organized as follows. We first briefly review a family of linear sentence encoders. Then we explain how to build upon such sentence encoders for continual sentence representation learning tasks, which lead to our proposed algorithm. Finally, we demonstrate the effectiveness of the proposed method using semantic textual similarity tasks.1

Footnote 1: Our codes are available on GitHub [https://github.com/liqutianlin0121/contSentEmbed](https://github.com/liqutianlin0121/contSentEmbed)

NotationWe assume each word \(w\) from a vocabulary set \(V\) has a real-valued word vector \(v_{w}\in\mathbb{R}^{n}\). Let \(p(w)\) be the monogram probability of a word \(w\). A corpus \(D\) is a collection of sentences, where each sentence \(s\in D\) is a multiset of words (word order is ignored here). For a collection of vectors \(Y=\{y_{i}\}_{i\in I}\), w

for \(i\) in an index set \(I\) with cardinality \(|I|\), we let \([y_{i}]_{i\in I}\in\mathbb{R}^{l\times|I|}\) be a matrix whose columns are vectors \(y_{1},\cdots,y_{|I|}\). An identity matrix is denoted by **I**.

## 2 Linear sentence encoders

We briefly overview "linear sentence encoders" that are based on linear algebraic operations over a sequence of word vectors. Among different linear sentence encoders, the smoothed inverse frequency (SIF) approach Arora et al. (2017) is a prominent example - it outperforms many neural-network based sentence encoders on a battery of NLP tasks Arora et al. (2017).

Derived from a generative model for sentences, the SIF encoder (presented in Algorithm 1) transforms a sequence of word vectors into a sentence vector with three steps. First, for each sentence in the training corpus, SIF computes a weighted average of word vectors (line 1-3 of Algorithm 1); next, it estimates a "common discourse direction" of the training corpus (line 4 of Algorithm 1); thirdly, for each sentence in the testing corpus, it calculates the weighted average of the word vectors and projects the averaged result away from the learned common discourse direction (line 5-8 of Algorithm 1). Note that this 3-step paradigm is slightly more general than the original one presented in Arora et al. (2017), where the training and the testing corpus is assumed to be the same.

```
Input : A training corpus \(D\); a testing corpus \(G\); parameter \(a\), monogram probabilities \(\{p(w)\}_{w\in V}\) of words
1forentence \(s\in D\)do
2\(q_{s}\leftarrow\frac{1}{|s|}\sum\limits_{w\in s}\frac{a}{p(w)+a}v_{w}\)
3 end while
4Let \(u\) be the first singular vector of \([q_{s}]_{s\in D}\).
5forsentence \(s\in G\)do
6\(q_{s}\leftarrow\frac{1}{|s|}\sum\limits_{w\in s}\frac{a}{p(w)+a}v_{w}\)
7\(f^{\text{SIF}}_{s}\gets q_{s}-uu^{T}q_{s}\).
8 end for Output :\(\{f^{\text{SIF}}_{s}\}_{s\in G}\)
```

**Algorithm 1**SIF sentence encoder.

Building upon SIP, recent studies have proposed further improved sentence encoders Khodak et al. (2018); Pagliardini et al. (2018); Yang et al. (2018). These algorithms roughly share the core procedures of SIF, albeit using more refined methods (e.g., softly remove more than one common discourse direction).

## 3 Continual learning for linear sentence encoders

In this section, we consider how to design a linear sentence encoder for continual sentence representation learning. We observe that common discourse directions used by SIF-like encoders are estimated from the training corpus. However, incrementally estimating common discourse directions in continual sentence representation learning tasks might not be optimal. For example, consider that we are sequentially given training corpora of tweets and news article. When the first tweets corpus is presented, we can train a SIF sentence encoder using tweets. When the second news article corpus is given, however, we will face a problem on how to exploit the newly given corpus for improving the _trained_ sentence encoder. A straightforward solution is to first combine the tweets and news article corpora and then train a new encoder from scratch using the combined corpus. However, this paradigm is not efficient or effective. It is not efficient in the sense that we will need to re-train the encoder from scratch every time a new corpus is added. Furthermore, it is not effective in the sense that the common direction estimated from scratch reflects a compromise between tweets and news articles, which might not be optimal for either of the standone corpus. Indeed, it is possible that larger corpora will swamp smaller ones.

To make the common discourse learned from one corpus more generalizable to another, we propose to use the conceptor matrix Jaeger (2017) to characterize and update the common discourse features in a sequence of training corpora.

### Matrix conceptors

In this section, we briefly introduce matrix concepts, drawing heavily on Jaeger (2017); He and Jaeger (2018); Liu et al. (2019). Consider a set of vectors \(\{x_{1},\cdots,x_{n}\}\), \(x_{i}\in\mathbb{R}^{N}\) for all \(i\in\{1,\cdots,n\}\). A conceptor matrix is a regularized identity map that minimizes

\[\frac{1}{n}\sum\limits_{i=1}^{n}\left\lVert x_{i}-Cx_{i}\right\rVert_{2}^{2}+ \alpha^{-2}\|C\|_{\text{F}}^{2}. \tag{1}\]

where \(\|\cdot\|_{\text{F}}\) is the Frobenius norm and \(\alpha^{-2}\) is a scalar parameter called _apperture_. It can be shown 

that \(C\) has a closed form solution:

\[C=\frac{1}{n}XX^{\top}(\frac{1}{n}XX^{\top}+\alpha^{-2}I)^{-1}, \tag{2}\]

where \(X=[x_{i}]_{i\in\{1,\cdots,n\}}\) is a data collection matrix whose columns are vectors from \(\{x_{1},\cdots,x_{n}\}\). In intuitive terms, \(C\) is a soft projection matrix on the linear subspace where the typical components of \(x_{i}\) samples lie. For convenience in notation, we may write \(C(X,\alpha)\) to stress the dependence on \(X\) and \(\alpha\).

Conceptors are subject to most laws of Boolean logic such as NOT \(\neg\), AND \(\wedge\) and OR \(\vee\). For two receptors \(C\) and \(B\), we define the following operations:

\[\neg C:= \mathbf{I}-C, \tag{3}\] \[C\wedge B:= (C^{-1}+B^{-1}-\mathbf{I})^{-1}\] (4) \[C\lor B:= -(\neg C\land\neg B) \tag{5}\]

Among these Boolean operations, the OR operation \(\vee\) is particularly relevant for our continual sentence representation learning task. It can be shown that \(C\lor B\) is the concept computed from the union of the two sets of sample points from which \(C\) and \(B\) are computed. Note that, however, to calculate \(C\lor B\), we only need to know two matrices \(C\) and \(B\) and do not have to access to the two sets of sample points from which \(C\) and \(B\) are computed.

### Using conceptors to continually learn sentence representations

We now show how to sequentially characterize and update the common discourse of corpora using the Boolean operation of conceptors. Suppose that we are sequentially given \(M\) training corpora \(D^{1},\cdots,D^{M}\), presented one after another. Without using any training corpus, we first initialize a conceptor which characterizes the corpus-independent common discourse features. More concretely, we compute \(C^{0}\coloneqq C([v_{w}]_{w\in Z},\alpha)\), where \([v_{w}]_{w\in Z}\) is a matrix of column-wisely stacked word vectors of words from a stop word list \(Z\) and \(\alpha\) is a hyper-parameter. After initialization, for each new training corpus \(D^{i}\) (\(i=1,\cdots,M\)) coming in, we compute a new conceptor \(C^{\text{temp}}\coloneqq C([q_{s}]_{s\in D^{i}},\alpha)\) to characterize the common discourse features of corpus \(D^{i}\), where those \(q_{s}\) are defined in the SIF Algorithm 1. We can then use Boolean operations of concepts to compute \(C^{i}\coloneqq C^{\text{temp}}\lor C^{i-1}\), which characterizes common discourse features from the new corpus as well as the old corpora. After all \(M\) corpora are presented, we follow the SIF paradigm and use \(C^{M}\) to remove common discourse features from (potentially unseen) sentences. The above outlined conceptor-aided (CA) continual sentence representation learning method is presented in Algorithm 2.

```
Input : A sequence of \(M\) training corpora \(\mathcal{D}=\{D^{1},\cdots,D^{M}\}\); a testing corpus \(G\); hyper-parameters \(a\) and \(\alpha\); word probabilities \(\{p(w)\}_{w\in V}\); stop word list \(Z\).
1\(C^{0}\gets C([v_{w}]_{w\in Z},\alpha)\) . forcprus index \(i=1,\cdots,M\)doforsentence \(s\in D^{i}\)do
2\(q_{s}\leftarrow\frac{1}{|s|}\sum\limits_{w\in s}\frac{a}{p(w)+a}v_{w}\)
3 end for
4\(C^{\text{temp}}\gets C([q_{s}]_{s\in D^{i}},\alpha)\)
5\(C^{i}\gets C^{\text{temp}}\lor C^{i-1}\)
6 end for
7for\(s\in G\)do
8\(q_{s}\leftarrow\frac{1}{|s|}\sum\limits_{w\in s}\frac{a}{p(w)+a}v_{w}\)
9\(f_{s}^{\text{CA}}\gets q_{s}-C^{M}q_{s}\)
10 end for Output :\(\{f_{s}^{\text{CA}}\}_{s\in G}\)
```

**Algorithm 2**CA sentence encoder.

A simple modification of Algorithm 2 yields a "zero-shot" sentence encoder that requires only pre-trained word embeddings and no training corpus: we can simply skip those corpus-dependent steps (line 2-8) and use \(C^{0}\) in place of \(C^{M}\) in line 11 in Algorithm 2 to embed sentences. This method will be referred to as "zero-shot CA."

## 4 Experiment

We evaluated our approach for continual sentence representation learning using semantic textual similarity (STS) datasets Agirre et al. (2012, 2013, 2014, 2015, 2016). The evaluation criterion for such datasets is the Pearson correlation coefficient (PCC) between the predicted sentence similarities and the ground-truth sentence similarities. We split these datasets into five corpora by their genre: news, captions, wordnet, forems, tweets (for details see appendix). Throughout this section, we use publicly available 300 

-dimensional GloVe vectors (trained on the 840 billion token Common Crawl) Pennington et al. (2014). Additional experiments with Word2Vec Mikolov et al. (2013), Fasttext Bojanowski et al. (2017), Paragram-SL-999 Wieting et al. (2015) are in the appendix.

We use a standard continual learning experiment setup (cf. Zenke et al. (2017), section 5.1)) as follows. We sequentially present the five training datasets in the order2 of news, captions, wordnet, forums, and tweets, to train sentence encoders. Whenever a new training corpus is presented, we train a SIF encoder from scratch3 (by combining all available training corpora which have been already presented) and then test it on each corpus. At the same time, we incrementally adapt a CA encoder4 using the newly presented corpus and test it on each corpus. The lines of each panel of Figure 1 show the test results of SIF and CA on each testing corpus (specified as the panel subtitle) as a function of the number of training corpora used (the first \(n\) corpora of news, captions, wordnet, forums, and tweets for this experiment). To give a concrete example, consider the blue line in the first panel of Figure 1. This line shows the test PCC scores (\(y\)-axis) of SIF encoder on the news corpus when the number of training corpora increases (\(x\)-axis). Specifically, the left-most blue dot indicates the test result of SIF encoder on news corpus when trained on news corpus itself (that is, the first training corpus is used); the second point indicates the test results of SIF encoder on news corpus when trained on news and captions corpora (i.e., the first _two_ training corpora are used); the third point indicates the test results of SIF encoder on news corpus when trained on news, captions, and wordnet corpora (that is, the first _three_ training corpora are used), so on and so forth. The dash-lines in panels show the results of a corpus-specialized SIF, which is trained and tested on the same corpus, i.e., as done in Arora et al. (2017), section 4.1). We see that the PCC results of CA are better and more "forgetting-resistant" than train-from-scratch SIF throughout the time course where more training data are incorporated. Consider, for example, the test result of news corpus (first panel) again. As more and more training corpora are used, the performance of train-from-scratch SIF drops with a noticeable slope; by contrast, the performance CA drops only slightly.

Footnote 2: The order can be arbitrary. Here we ordered the corpora from the one with the largest size (news) to the smallest size (tweets). The results from reversely ordered corpora are reported in the appendix.

Footnote 3: We use \(a=0.001\) as in Arora et al. (2017). The word frequencies are available at the GitHub repository of SIF.

Footnote 4: We used hyper-parameter \(\alpha=1\). Other parameters are set to be the same as SIF.

As remarked in the section 3.2, with a simple modification of CA, we can perform zero-shot sentence representation learning without using any training corpus. The zero-shot learning results are

Figure 1: PCC results of STS datasets. Each panel shows the PCC results of a testing corpus (specified as a subtitle) as a function of increasing numbers of training corpora used. The setup of this experiment mimics Zenke et al. (2017, section 5.1).

 

presented in Table 1, together with the time-course averaged results of CA and train-from-scratch SIF (i.e., the averaged values of those CA or SIF scores in each panel of Figure 1). We see that the averaged results of our CA method performs the best among these three methods. Somewhat surprisingly, the results yielded by zero-shot CA are better than the averaged results of train-from-scratch SIF in most of the cases.

We defer additional experiments to the appendix, where we compared CA against more baseline methods and use different word vectors other than GloVe to carry out the experiments.

## 5 Conclusions and future work

In this paper, we formulated a continual sentence representation learning task: Given a consecutive sequence of corpora presented in a time-course manner, how can we extract useful sentence-level features from new corpora while retaining those from previously seen corpora? We identified that the existing linear sentence encoders usually fall short at solving this task as they leverage on "common discourse" statistics estimated based on a priori fixed corpora. We proposed two sentence encoders (CA encoder and zero-shot CA encoder) and demonstrate their the effectiveness at the continual sentence representation learning task using STS datasets.

As the first paper considering continual sentence representation learning task, this work has been limited in a few ways - it remains for future work to address these limitations. First, it is worthwhile to incorporate more benchmarks such as GLUE Wang et al. (2019) and SentEval Conneau and Kiela (2018) into the continual sentence representation task. Second, this work only considers the case of linear sentence encoder, but future research can attempt to devise (potentially more powerful) non-linear sentence encoders to address the same task. Thirdly, the proposed CA encoder operates at a corpus level, which might be a limitation if boundaries of training corpora are ill-defined. As a future direction, we expect to lift this assumption, for example, by updating the common direction statistics at a sentence level using Autoconceptors Jaeger (2014); section 3.14). Finally, the continual learning based sentence encoders should be applied to downstream applications in areas such as open domain NLP systems.

## Acknowledgements

The authors thank anonymous reviewers for their helpful feedback. This work was partially supported by Joao Sedoc's Microsoft Research Dissertation Grant.

## References

* Agirre et al. (2015) E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Maritxalar, R. Mihalcea, G. Rigaua, L. Uriaa, and J. Wiebe. 2015. Semeval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In _Proceedings of the 9th international workshop on semantic evaluation_, pages 252-263.
* Agirre et al. (2014) E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agire, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In _Proceedings of the 8th international workshop on semantic evaluation_, pages 81-91.
* Agirre et al. (2016) E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, pages 497-511, San Diego, California.
* Agirre et al. (2013) E. Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre, and W. Guo. 2013. Sem 2013 shared task: Semantic textual similarity. In _Second Joint Conference on Lexical and Computational Semantics_, volume 1, pages 32-43.
* Agirre et al. (2012) E. Agirre, M. Diab, D. Cer, and A. Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In _Proceedings of the First Joint Conference on Lexical and Computational Semantics_, SemEval '12, pages 385-393, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Arora et al. (2017) S. Arora, Y. Liang, and T. Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In _International Conference on Learning Representations_.
* Bojanowski et al. (2017) P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Cer et al. (2018) D. Cer, Y. Yang, S. Kong, N. Hua, N. Limtiaco, R. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, et al. 2018. Universal sentence encoder. _arXiv preprint arXiv:1803.11175_.
* Conneau and Kiela (2018) A. Conneau and D. Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. _arXiv preprint arXiv:1803.05449_ 

X. He and H. Jaeger (2018)Overcoming catastrophic interference using concept-aided backpropagation. In International Conference on Learning Representations, Cited by: SS4.
* H. Jaeger (2014)Controlling recurrent neural networks by conceptors. Technical report Jacobs University Bremen. Cited by: SS3.
* H. Jaeger (2017)Using concepts to manage neural long-term memories for temporal patterns. Journal of Machine Learning Research18 (13), pp. 1-43. Cited by: SS5.
* M. Khodak, N. Saunshi, Y. Liang, T. Ma, B. Stewart, and S. Arora (2018)A la carte embedding: cheap but effective induction of semantic feature vectors. In Proceedings of ACL, Cited by: SS6.
* Redmond. Cited by: SS1, SS2.
* T. Liu, L. Ungar, and J. Sedoc (2019)Unsupervised post-processing of word vectors via concept negation. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-2019), Honolulu, Cited by: SS1, SS2.
* V. Mathur and A. Singh (2018)The rapidly changing landscape of conversational agents. Cited by: SS1, SS2.
* T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean (2013)Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, D. R. Ho et al. (Eds.), Cham, pp. 3111-3119. External Links: Link, Document Cited by: SS1, SS2.
* M. Pagliardini, P. Gupta, and M. Jaggi (2018)Unsupervised learning of sentence embeddings using compositional n-gram features. In Proceedings of the NAACL 2018, Cited by: SS1, SS2.
* J. Pennington, R. Socher, and C. D. Manning (2014)Glove: global vectors for word representation. In Proceedings of EMNLP, pp. 1532-1543. Cited by: SS1, SS2.
* A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2019)GLUE: a multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, Cited by: SS1, SS2.
* J. Wieting, M. Bansal, K. Gimpel, K. Livescu, and D. Roth (2015)From paraphrase database to compositional paraphrase model and back. Transactions of the Association for Computational Linguistics3, pp. 345-358. Cited by: SS1, SS2.
* Z. Yang, C. Zhu, and W. Chen (2018)Zero-training sentence embedding via orthogonal basis. Cited by: SS1, SS2.
* W. Yin and H. Schutze (2015)Convolutional neural network for paraphrase identification. In Proceedings of the NAACL HLT 2015, pp. 901-911. Cited by: SS1, SS2.
* F. Zenke, B. Poole, and S. Ganguli (2017)Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70 of Proceedings of Machine Learning Research, pp. 3987-3995. External Links: Link, Document Cited by: SS1, SS2.