Title: LIFELONG WORD EMBEDDING VIA META-LEARNING

Abstract: Learning high-quality word embeddings is of significant importance in achieving better performance in many down-stream learning tasks. On one hand, traditional word embeddings are trained on a large scale corpus for general-purpose tasks, which are often sub-optimal for many domain-specific tasks. On the other hand, many domain-specific tasks do not have a large enough domain corpus to obtain high-quality embeddings. We observe that domains are not isolated and a small domain corpus can leverage the learned knowledge from many past domains to augment that corpus in order to generate high-quality embeddings. In this paper, we formulate the learning of word embeddings as a lifelong learning process. Given knowledge learned from many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner, where the metalearner is able to provide word context similarity information at the domain-level. Experimental results demonstrate that the proposed method can effectively learn new domain embeddings from a small corpus and past domain knowledges 1 . We also demonstrate that general-purpose embeddings trained from a large scale corpus are sub-optimal in domain-specific tasks.

1. INTRODUCTION:

Learning word embeddings (Mnih & Hinton (2007) ; Turian et al. (2010) ; Mikolov et al. (2013a; b; c) ; Pennington et al. (2014) ) has received a significant amount of attention due to its high performance on many down-stream learning tasks. Word embeddings have been shown effective in NLP tasks such as named entity recognition (Sienčnik (2015) ), sentiment analysis (Maas et al. (2011) ) and syntactic parsing (Durrett & Klein (2015) ). Such embeddings are shown to effectively capture syntactic and semantic level information associated with a given word (Mikolov et al. (2013a) ).

The "secret sauce" of training word embedding is to turn a large scale in-domain corpus into billions of training examples. There are two common assumptions for training word embeddings: 1) the training corpus is largely available and bigger than the training data of the potential downstream learning tasks; and 2) the topic of the training corpus is closely related to the topic of the down-stream learning tasks. However, real-world learning tasks often do not meet one of these assumptions. For example, a domain-specific corpus that is closely related to a down-stream learning task may often be of limited size. If we lump different domain corpora together and train general-purpose embeddings over a large scale corpus (e.g., GloVe embeddings (Pennington et al. (2014) ) are trained from the corpus Common Crawl, which covers almost any topic on the web), the performance of such embeddings on many domain-specific tasks is sub-optimal (we show this in Section 6). A possible explanation is that although many domain words share similar meanings with the same out-of-domain words, with no in-domain awareness, dumping many out-of-domain co-occurrences as training examples may bias in-domain embeddings. (e.g., if the domain is about food, then an out-of-domain "python" as a programming language can bias "java", while the indomain word "chocolate" is more likely to help).

To solve the problem of the limited domain corpus, one possible solution is to use transfer learning (Pan & Yang (2010) ) for training domain-specific embeddings (Bollegala et al. (2015) ; Yang et al. (2017) ). However, these methods just manage to leverage out-of-domain embeddings trained from a large scale corpus to help limited in-domain corpus. The very in-domain corpus is never expanded. Also, one common assumption of these works is that a pair of similar source domain and target domain is manually identified in advance. In reality, given many domains, manually catching useful information in so many domains are very hard. In contrast, we humans learn the meaning of a word more smartly. We accumulate different domain contexts for the same word. When a new learning task comes, we may quickly identify the new domain contexts and borrow the word meanings from existing domain contexts. This is where lifelong learning comes to the rescue. Lifelong machine learning (LML) is a continual learning paradigm that retains the knowledge learned in past tasks 1, . . . , n, and uses it to help learning the new task n + 1 (Thrun (1996); Silver et al. (2013) ; Chen & Liu (2016) ). In the setting of word embedding: we assume that the learning system has seen n domain corpora: (D 1 , . . . , D n ), when a new domain corpus D n+1 comes by demands from that domain's potential down-stream learning tasks, the learning system can automatically generate word embeddings for the n + 1-th domain by effectively leveraging useful past domain knowledge.

The main challenges of this task are 2 fold. 1) How to identify useful past domain knowledge to train the embeddings for the new domain. 2) How to automatically identify such kind of information, without help from human beings. To tackle these challenges, the system has to learn how to identify similar words in other domains for a given word in a new domain. This, in general, belongs to metalearning (Vilalta & Drissi (2002) ; Peng et al. (2002) ). Here we do not focus on specific embedding learning but focus on learning how to characterize corpora of different domains for embedding purpose.

The main contributions of this paper can be summarized as follows: 1) we propose the problem of lifelong word embedding, which may benefit many down-stream learning tasks. We are not aware of any existing work on word embedding using lifelong learning 2) we propose a lifelong embedding learning method, which leverages meta-learning to aggregate useful knowledge from past domain corpora to generate embeddings for the new domain.

2. RELATED WORKS:

Learning word embeddings has been studied for a long time (Mnih & Hinton (2007) ). Many earlier methods employ complex neural network architectures (Collobert & Weston (2008) ; Mikolov et al. (2013c) ). Recently, a simple and effective unsupervised model called skip-gram (Mikolov et al. (2013b; c) ) was proposed to turn plain text corpus into large-scale training examples without any human annotation. It uses the current word to predict the surrounding words in a context window by maximizing the likelihood of the predictions. The learned parameters for each word are then the embeddings of that word. Although such embeddings can be trained in large scale and easily obtained online (Pennington et al. (2014) ; Bojanowski et al. (2016) ), they are sub-optimal for many domain-specific tasks (Bollegala et al. (2015) ; Yang et al. (2017) ). Domain corpus also suffers from limited size to train high-quality embeddings.

Our work is most related to Lifelong Machine Learning (LML)) (or lifelong learning). Much of the work on LML focused on supervised learning (Thrun (1996) ; Silver et al. (2013) ; Ruvolo & Eaton (2013) ; Chen & Liu (2016) ) Recent years, several works have also been done in the unsupervised setting, mainly on topic modeling (Chen & Liu (2014) ), information extraction (Mitchell et al. (2015) ) and graph labeling (Shu et al. (2016) ). However, we are not aware of any existing research that has been done on using lifelong learning for word embedding. LML is related to transfer learning and multi-task learning (Pan & Yang (2010)), which have been leveraged in word embeddings (Bollegala et al. (2015) ; Yang et al. (2017) ). However, LML is different from transfer learning (see the survey book from Chen & Liu (2016) ). Given many domains with uncertain relevance for the new domain, the lack of guidance on which kind of information is worth learning from the past domains is a problem. And there's no good measure of similarity of two words in different domains.

The proposed method leverages meta-learning (Vilalta & Drissi (2002) ), which is to perform machine learning on learning tasks. Recently, meta-learning (or learning to learn) has been used to learn parameters of an optimizer (Andrychowicz et al. (2016) ), to learn neural architectures (Fernando et al. (2017) ). We leverage a meta-learner to accumulate knowledge during lifelong learning. 

3. MODEL OVERVIEW:

The overall lifelong learning process is depicted in Figure 1 . Given a series of domain corpora D n = d 1 , d 2 , . . . , d n , the lifelong learning system first learns a meta-model (learner) on domainlevel word context similarity from the first m domain corpus. As more domains arrive, the system accumulates the knowledge of domain corpora. When a new domain D n+1 comes, the system uses the meta-learner to catch past domain knowledge that is useful and related to the new domain D n+1 as augmented knowledge. With the augmented knowledge, the word embedding learning process is performed and the resulting embeddings are used for further down-stream learning tasks. The metalearner here plays a central role in automatically identifying useful knowledge from past domains to help the new domain. By using a pairwise network, the meta-learner finds words in the past domains that are similar to the new domain. Then the co-occurrence knowledge of those similar words from the past domain is used together with the new domain corpus to train the new domain embeddings.

4. META-LEARNER:

In this subsection, we describe how a meta-learner can help to identify similar words from many past domain corpora. When it comes to borrowing knowledge from past domains, the first problem is what to borrow. Although binary cross-domain embeddings are studied in Bollegala et al. (2015) ; Yang et al. (2017) , they mostly assume that a relevant domain is already identified and shared words between two domains have similar meanings. In reality, given a wide spectrum of domains, borrowing knowledge from a non-relevant domain may not be helpful or even harmful to word embeddings (we show this Section 6). The meaning of one word in one domain may be quite different from the same word in another. For example, the word "java" in the programming context is different from the restaurant context. Borrowing the knowledge from a restaurant corpus can be harmful to the representations of "java" in a programming context.

4.1. TRAINING EXAMPLES:

On top of learning embeddings for specific domains, we build a meta-learner to learn a general word context similarity from the first m domains, where m n. In practice, if n is small, m domains can simply be sampled from n domains. Here since our experiments are conducted on hundreds of domains, we hold-out m domains to train the meta-learner. The expected input to the meta-learner is a pair of the same word from similar ("java" from two corpora of the restaurant domain) or different domains (e.g. "java" from the restaurant domain or the programming domain). The output of the meta-learner is whether two words are from the same domain or not. We first characterize words in domain corpora. Given a specific word in a domain, we choose its co-occurrence counts with f frequent words within a context window (like word2vec) as the discrete features (a sparse vector of length f ) of a word in that domain. This is inspired by the fact that a good dictionary (e.g. Longman dictionary) uses only a few thousand words to explain all other words. We denote the selected the top f frequent words over m domains as V wf . Then given a domain corpus D i , we sample l subcorpora D i,j ∼ P (D i ) by selecting a fixed amount of chunks in D i . A chunk can be a sentence or a document in D i . We randomly select a fixed amount of chunks because the word features built from the sub-corpus are expected to be on the same scale. Then we randomly select a subset of words from top f words as training example words V meta . These training example words are the same in all domains D 1:m . We use these words in V meta as co-occurrence features and build features u w i,j,k for the word w k ∈ V meta on the j-th sub-corpus of the i-th domain. We build word features for all m domain sub-corpora D 1:m,1:l .

Finally, a pairwise meta-learner is trained on pairs of word features drawn from different domain sub-corpora for the same word. Given a word w k ∈ V meta , a pair of word features (u w i,j,k , u w i,j ,k ), where j = j , forms a postive example; whereas (u w i,j,k , u w i ,j ,k ) with i = i (j and j can be equal or not) forms a negative example.

The m domains are split into disjoint m t training domains, validation domains, and testing domains. So both the validation and testing examples are unseen examples during training. We enforce such isolation and wish the trained meta-learner can be more generally applied to the rest n -m domains.

4.2. PAIRWISE NETWORK:

We train a simple but efficient neural network to learn pairwise domain-level word context similarity. The idea of making such a network small but high-throughput is crucial in lifelong settings. This is because the meta-learner is heavily used in the later lifelong learning process. Given so many domains with so many words asking for detecting similarity, a small pairwise network with fewer parameters is desirable to save more memory being used for high-throughput inference.

The proposed pairwise network contains only one shared fully-connected layer (normalized by the co-occurrence feature) to learn continuous features from co-occurrence (discrete) features, a cosine function to learn similarity and a sigmoid layer to generate predictions like linear regression. The network is parameterized as follows:

EQUATION

where | • | 1 is the l1-norm, W s and b are weights and σ(•) is the sigmoid function. Cosine similarity is defined as Cosine(x, y)

= x•y ||x||2•||y||2

. Most trainable weights of this simple network reside in W 1 , which learn continuous features on the f words. These weights can also be interpreted as an embedding matrix for the f words. These f word embeddings serve as general word embeddings to explain domain-specific words. We train the meta-learner over a hold-out domain set as the base meta-learner M . Then we fine-tune the meta-learner based on new domain corpus, as described in the next section.

5. LIFELONG LEARNING PROCESS:

The previous section ends up with a base meta-learner M . In this section, we describe how the lifelong learning system works based on M , n -m domains, and the new domain corpus D n+1 .

5.1. WORD CONTEXT RETRIEVAL:

Assume the lifelong learning system has seen n domain corpora. The system stores knowledge into a knowledge base K. The knowledge base K contains a base meta-learner M trained over the first m domains, fine-tuned meta-learners M m+1:n and knowledge over past n -m domains. The knowledge includes the vocabulary of word features V wf , n -m domain corpora D m:n , vocabularies on n -m domains V m:n , and word features on those vocabularies E m:n . The word features E m:n are computed from one sample from each domain corpus.

Given a new domain corpus D n+1 , the lifelong learning system first fine-tunes the base meta-learner M . This ends with a fine-tuned meta-learner M n+1 for this new domain. The tuning process makes the meta-learner more suitable for the new domain to retrieve past knowledge. Tuning examples are sampled similarly as the training examples of base meta-learner, except that negative examples are sampled between D n+1 and D m+1:n . Then the lifelong learning system retrieves similar in-domain word context information as augmented knowledge, which is used in embedding training in the next subsection.

The retrieval process is described in Algorithm 1. Firstly, line 1-2 build word features for the new domain corpus. These two operations are already done when preparing fine-tuning data for the meta-learner. Here we just mention them for storing knowledge purpose in line 13. Line 3 retrieves past domain knowledge, which is the reversed process similar to line 13 for the new domain. Line 4 defines the variable that stores useful past knowledge. Line 5-12 retrieves relevant words from past domains and store them in A. More importantly, the fine-tuned meta-learner at line 9 finds similar words from past domains. Then we only keep similar words with a probability higher than a threshold delta at line 10. This threshold controls the quality of the accumulated words O. Line 11 retrieves co-occurrence information about the set of words O from past domain D i as augmented knowledge. Such augmented knowledge is more close to the co-occurrence information in the new domain D n+1 . ScanCooccurrence(D i , O) can be viewed as scanning the past domain corpus and grab the word in O with its context words like word2vec, or as retrieving rows of co-occurrence counts from the co-occurrence matrix of the domain D i . Line 13 simply stores the knowledge of the new domain for further use.

Algorithm 1: Lifelong domain-level word context retrieval Input : a knowledge base K containing knowledge over past (n -m) domains, a new domain corpus D n+1 , and a fine-tuned meta-learner M n+1 . Output: a word co-occurrence set A, where each element is a 2-tuple (w t , w c ), representing useful knowledge from past domains.

V n+1 ← BuildVocab(D n+1 ) E n+1 ← BuildWordFeature(D n+1 , K.V wf ) (D m:n , V m:n , E m:n , M m:n ) ← K m:n A ← ∅ for (D i , V i , E i ) ∈ D m:n , V m:n , E m:n do O ← V i ∩ V n+1 F i ← RetrieveWordFeature(E i , O) F n+1 ← RetrieveWordFeature(E n+1 , O) S ← M n+1 .inference(F i , F n+1 ) O ← {o|o ∈ O and S[o] ≥ δ} A ← A ∪ ScanCooccurrence(D i , O) end K n+1 ← (D n+1 , V n+1 , E n+1 , M n+1 ) return A

5.2. LIFELONG WORD EMBEDDING:

In this subsection, we first describe the skip-gram model introduced by Mikolov et al. (2013b) in the context of a new domain in the lifelong setting. Given a new domain corpus D n+1 with a vocabulary V n+1 , the goal of the skip-gram model is to learn a vector representation for each word w ∈ V n+1 in that domain. Assume the domain corpus is represented as a sequence of words D n+1 = (w 1 , . . . , w T ), the objective of the skip-gram model is to maximize the following log-likelihood:

EQUATION

where C t is the set of indices of words surrounding word w t in a fixed context window; N t is a set of indices of words (negative samples) drawn from the vocabulary V n+1 for the t-th word; u and v represent word vectors (or embeddings) we are trying to learn. The goal of skip-gram is to independently predict the presence (or absence) of context words w c given the word w t . When the size T of the corpus is extremely large, the skip-gram model, in fact, can be fed with billions of training examples. So the vector of a word can be trained to have a good representation of the similarity with the word's context words.

However, depending on the specific down-stream tasks, many domain corpora may not have a large scale corpus. And a random sequence of words drawn from other domains may not truly reflect the distribution P (w c |w t ) in domain D n+1 . This is where the previously computed augmented word co-occurrence A come to rescue. Assume our lifelong learning system has seen m domains to build the meta-learner M and n -m domains to build the knowledge into K. Given a new domain corpus D n+1 , we first perform Algorithm 1 to obtain the augmented word co-occurrence from past domains A. Then this co-occurence information A is integrated into the objective function of skip-gram as following:

EQUATION

where w c is a random word drawn from the vocabulary. We use the default hyperparameters of skip-gram model (Mikolov et al. (2013b) ). Note that in the skip-gram model as we scan through the corpus w t can also be w c 's context word. But in the augmented information here, we do not allow such bi-directional co-occurrence happen since w t may not be a useful context word for the word w c in the (n + 1)-th domain.

6. EXPERIMENTAL RESULTS:

We present extensive evaluations to assess the effectiveness of our approach. Following the suggestions of Nayak et al. (2016) ; Faruqui et al. (2016) , we leverage the learned word embeddings as continuous features in several domain-specific down-stream tasks, including document classification, aspect extraction, and sentiment classification. We do not evaluate the learned embeddings directly as in traditional word embedding papers (Mikolov et al. (2013b) ; Pennington et al. ( 2014)) because domain-specific dictionaries of similar / non-similar words are in general not available.

6.1. DATASETS:

We use the Amazon Review datasets He & McAuley (2016) as a huge collection of multiple-domain corpus. We consider each second-level category (the first level is department) as a domain and aggregate all reviews under each category as one domain corpus. This ends up with a rather diverse domain collection. Due to limited computing resources, we limit each domain corpus up to 60 MB. We randomly select 3 domains ("Computer Components", "Cats Supply" and "Kitchen Storage and Organization") as new domains for down-stream tasks on product type classification and sentiment classification. Then we deliberately pick the "Laptops" domain as the new domain for aspect extraction task since the annotation is on Laptop reviews. Each new domain corpus is cut to 10 MB and 30 MB in order to test the practical performance of a small new domain. We randomly select 56 (m) domains to train and evaluate the meta-learner. Lastly, three random collections of 50, 100 and 200 (n -m) domains corpora are used as past domains.

6.2. EVALUATION OF META-LEARNER:

We split the 56 domains as 39 (m t ) domains for training, 5 domains for validation and 12 domains for testing. So the validation and testing domain corpora have no overlapping with the training domain corpora. This leads to a more general base meta-learner for many unseen new domains. 1 . Finally, we empirically set delta = 0.7 as the threshold.

6.3. DOWN-STREAM TASKS:

We use 3 down-stream tasks to evaluate the effectiveness of our approach. For each task, we leverage an embedding layer to store the pre-trained embeddings. We choose our embedding dimensions as 300, which is the same size as many pre-trained embeddings (GloVec.800B (Pennington et al. (2014) ) or fastText Wiki English (Bojanowski et al. (2016) )). We freeze the embedding layers during training, so the result is less affected by the rest of the model and the training data. To make the performance of all tasks relatively consistent, we leverage the same Bi-LSTM model (Hochreiter & Schmidhuber (1997) ) on top of the embedding layer to learn task-specific features from different embeddings. The input size of Bi-LSTM is the same as the embedding layer and the output size is 128. All tasks leverage many-to-one Bi-LSTMs for classification purpose except aspect extraction, which uses many-to-many Bi-LSTM for sequence labeling. In the end, a fully-connected layer and a softmax activation are applied after Bi-LSTM, with the output size specific to each task.

6.4. BASELINES:

No Embedding (NE): We randomly initialize the word vectors and train the word embedding layer during the training process of each down-stream task. Note that only in this baseline do we allow embeddings trainable.

fastText: This is the lower-cased embeddings pre-trained from English Wikipedia using fastText (Bojanowski et al. (2016) ). We lower the cases of all corpora of down-stream tasks to match the words in this embedding. Note that although the corpus of Wikipedia contains a wide spectrum of domains covering almost everything of human knowledge, the amount of corpus for a specific domain (e.g, a product) may not be large enough. The total amount of Wikipedia is just several billions of tokens, which is on the same scale as Amazon Review datasets (8 billion tokens).

GoogleNews: This is the pre-trained embeddings using word2vec2 based on part of the Google News datasets, which contains 100 billion words.

GloVe.Twitter.27B: This embedding is pre-trained using GloVe (Pennington et al. (2014) ) based on Tweets, which have 27 billion words. Note this embedding is lower-cased and has 200 dimensions.

GloVe.6B: This is the lower-cased embeddings pre-trained from Wikipedia and Gigaword 5, which has 6 billions of tokens.

GloVe.840B: This is the cased embeddings pre-trained from Common Crawl, which has 840 billions of tokens. This embedding corpus is the largest one among all embeddings. It contains almost all web pages available before 2015. We show that although GloVe.840B is general enough on almost any task, its performance is sub-optimal on many domain tasks. New Domain 10M (ND 10M): This is a baseline embedding pre-trained only from the new domain 10 MB corpus. We show that the embeddings trained from a small corpus alone are not good enough.

New Domain 30M (ND 30M): Then we increase the size of the new domain corpus to 30 MB to see the difference affected by the corpus size.

200 Domains + New Domain 30M (200D + ND 30M): Another straightforward embedding is to concatenate the corpora from all past domains and the new domain together to train a mixed embedding. We use this baseline to show that unselected past domain corpora may reduce the performance of down-stream tasks. How to smartly adapt the embeddings into a closer context is crucial.

Lifelong Past Domains + New Domain (LL [P]D + ND [X]M) : This is different versions of our proposed method. For example, we use LL 200D + ND 30M to donate embeddings trained from a 30MB new domain corpus and 200 past domains.

6.5. PRODUCT TYPE CLASSIFICATION:

This task is to classify a review into a product type (leaf-level category in Amazon product category system). There are many product types under each domain (2nd-level category). We use the randomly selected 3 domains as the new domains to form 3 multi-class classification sub-tasks. These domains are: Computer Components, Kitchen Storage and Organization and Cats Supplies. For each sub-task, we randomly draw 1200 reviews for each product type. We drop classes with less than 1200 reviews. This ends up with 13, 17 and 11 classes for Computer Components, Kitchen Storage and Organization and Cats Supplies, respectively. For each sub-task, we keep 10000 reviews as the testing data (to make the result more accurate) and split the rest as 7:1 for training and validation data, respectively. All sub-tasks are evaluated on accuracy. We train and evaluate each sub-task on each baseline 10 times (with different initialization) and average the results.

From Aspect extraction is an important task in sentiment analysis (Liu (2012; 2015) ). We use the dataset from SemEval-2014 Task 4: Aspect-based sentiment analysis (Pontiki et al. (2014) ) as a downstream new domain task. This dataset contains human annotated Laptop aspects and their polarities. It has 3045 training examples and 800 testing examples. We use the Laptop domain corpus from the Amazon Review Dataset as the new domain corpus to train the lifelong embedding. We leverage the original evaluation script to report precision, recall, and F1-score. Again, we average 10 runs of the results.

From Table 3 , we can see that aspect extraction is quite different from product type classification. Again, our lifelong embedding performs best. Surprisingly, the performance of 200D + ND 30M is very good. This indicates aspect extraction requires both good general embedding and domainspecific embeddings. For example, good representations of general words can help to identify nearby aspects and good aspects words can also help.

6.7. SENTIMENT CLASSIFICATION:

We select 6000 4-rating reviews as positive reviews and 6000 2-rating reviews as negative reviews from 3 domains used in product type classification to form 3 sentiment classification sub-tasks. Again, to ensure enough number of valid digits in the results, we use 10000 out of 12000 reviews for testing. The results are averaged over 10 runs.

From Table 4 , we can see the performance of most domain-specific baselines is very close (We omit the minor differences between different sizes of past domains). Sentiment classification, in general, requires polarity words to determine the sentiment of a document. Pre-trained general embeddings may introduce non-polarity information into the embeddings. When domain corpus is leveraged, the difference is small. This is close to our previous experience. A possible explanation is that sentiment classification relies on sentiment words like "good" or "bad". However, those words have similar context words, e.g., "This phone is good." and "This phone is bad.". So the co-occurrence-based training corpus of embedding is not good for learning the embeddings of sentiment words.

6.8. FUSING WITH THE OUT-OF-DOMAIN WORLD:

Although most cross-domain embedding papers focus on leveraging different existing pre-trained embeddings, we focus on expanding the domain-specific corpus. We believe if we can expand the domain-specific training corpus on a much larger scale (like breaking the training corpus of GloVe.860B into many domains), the performance of the proposed method is much better. However, our focus does not forbid our method from leveraging existing cross-domain transfer learning method (? Bollegala et al. (2015) ; Yang et al. (2017) ). A simple way of leveraging existing embeddings in these papers is to concatenate existing pre-trained embeddings with domain-specific embeddings. To demonstrate our method further improves the domain-specific parts of the downstream tasks, we further evaluate two methods: (1) GloVe.840B&ND 30M, which concatenates new domain alone embeddings with GloVe.860B; (2) GloVe.840B&LL 200D + ND 30M, which concatenates our lifelong embeddings with GloVe.860B.

As shown in Table 5 , concatenating embeddings improve the performance a lot. Our method further improves the domain-specific parts of the embeddings. While existing cross-domain embedding methods can only use the 30 MB corpus of the new domain, our method allows those methods to further leverage the expanded corpus.

7. CONCLUSIONS:

In this paper, we formulate a lifelong word embedding learning process. Given many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner. The meta-learner is able to provide word context similarity information on domain-level. Such information can help to accumulate new domain-specific training corpus in order to get better embedding. Experimental results show that the proposed method is effective in learning new domain embeddings from a small corpus and past domain knowledge.

:

We will release the code after final revisions.

https://code.google.com/archive/p/word2vec/

Figures and Tables:

Figure 1: Lifelong word embedding learning process.

F1 score of fine-tuning on base meta-learner ) sub-corpora over the set of reviews from each domain and limit the size of the sub-corpora to 10 MB. We select top 5000 words as word features (f ). We randomly select 500 words (|V meta | = 500) from each domain and ignore words with zero counts on co-occurrence to obtain pairwise examples. This ends up with 80484 training examples, 6234 validation examples, and 20740 testing examples. The f1-score of meta-learner is 81%.We further fine-tune the meta-learner for each new domain. We sample 3000 words from each new domain, which ends with slightly fewer than 6000 samples after ignoring zero co-occurrences. We select 3500 examples for training, 500 examples for validation and 2000 examples for testing. The testing f1-score is shown in Table

Table content: <table><tr><td/><td colspan="4">Cmptr. Components Kitch. Storage &amp; Org. Cats Supples Laptops</td></tr><tr><td>10MB</td><td>0.832</td><td>0.841</td><td>0.856</td><td>0.817</td></tr><tr><td>30MB</td><td>0.847</td><td>0.859</td><td>0.876</td><td>0.854</td></tr><tr><td>We sample 2 (l</td><td/><td/><td/><td/></tr></table>

Accuracy of different embeddings on product type classification tasks (numbers in parenthesis indicates number of classes)

Table content: <table><tr><td/><td colspan="3">Cmptr. Cmpnts. (13) Kitch. Strg. &amp; Org. (17) Cats Supples (11)</td></tr><tr><td>NE</td><td>0.596</td><td>0.653</td><td>0.696</td></tr><tr><td>fastText</td><td>0.705</td><td>0.717</td><td>0.809</td></tr><tr><td>GoogleNews</td><td>0.76</td><td>0.722</td><td>0.814</td></tr><tr><td>GloVe.Twitter.27B</td><td>0.696</td><td>0.707</td><td>0.80</td></tr><tr><td>GloVe.6B</td><td>0.701</td><td>0.725</td><td>0.823</td></tr><tr><td>GloVe.840B</td><td>0.803</td><td>0.758</td><td>0.855</td></tr><tr><td>ND 10M</td><td>0.77011</td><td>0.74905</td><td>0.85</td></tr><tr><td>ND 30M</td><td>0.794</td><td>0.766</td><td>0.87</td></tr><tr><td>200D + ND 30M</td><td>0.793</td><td>0.759</td><td>0.859</td></tr><tr><td>LL 200D + ND 10M</td><td>0.791</td><td>0.761</td><td>0.872</td></tr><tr><td>LL 50D + ND 30M</td><td>0.795</td><td>0.768</td><td>0.868</td></tr><tr><td>LL 100D + ND 30M</td><td>0.803</td><td>0.773</td><td>0.874</td></tr><tr><td>LL 200D + ND 30M</td><td>0.809</td><td>0.775</td><td>0.883</td></tr></table>

Table2, we can see that the performance of different classification tasks varies a lot. This is mostly caused by the number of classes in each sub-task. The lifelong embeddings LL 200D + ND 30M performs best. The maximum train corpus of this method is in total just 80 MB (30 MB new domain + 50 MB from 200 past domains). The difference in the numbers of LL past domains indicates more past domains have better results. Surprisingly, the GloVe.840B trained on 840 billions of tokens does not perform well enough compared to the limited new domain corpora. But it performs very well on Computer Components, which means this domain is relatively general.

Table content: <table/>

Performance of different embeddings on aspect extraction task

Table content: <table><tr><td/><td colspan="3">Precision Recall F1-Score</td></tr><tr><td>NE</td><td>0.596</td><td>0.493</td><td>0.54</td></tr><tr><td>fastText</td><td>0.655</td><td>0.47</td><td>0.547</td></tr><tr><td>GoogleNews</td><td>0.7</td><td>0.638</td><td>0.667</td></tr><tr><td>GloVe.Twitter.27B</td><td>0.642</td><td>0.468</td><td>0.541</td></tr><tr><td>GloVe.6B</td><td>0.68</td><td>0.505</td><td>0.579</td></tr><tr><td>GloVe.840B</td><td>0.722</td><td>0.6406</td><td>0.679</td></tr><tr><td>ND 10M</td><td>0.663</td><td>0.57</td><td>0.613</td></tr><tr><td>ND 30M</td><td>0.713</td><td>0.62</td><td>0.663</td></tr><tr><td>200D + ND 30M</td><td>0.731</td><td>0.65</td><td>0.688</td></tr><tr><td>LL 200D + ND 10M</td><td>0.724</td><td>0.636</td><td>0.677</td></tr><tr><td>LL 50D + ND 30M</td><td>0.736</td><td>0.637</td><td>0.683</td></tr><tr><td>LL 100D + ND 30M</td><td>0.723</td><td>0.65</td><td>0.685</td></tr><tr><td>LL 200D + ND 30M</td><td>0.734</td><td>0.659</td><td>0.694</td></tr></table>

Accuracy of different embeddings on sentiment classification task

Table content: <table><tr><td/><td colspan="3">Cmptr. Cmpnts. Kitch. Strg. &amp; Org. Cats Supples</td></tr><tr><td>NE</td><td>0.777</td><td>0.764</td><td>0.67</td></tr><tr><td>GoogleNews</td><td>0.847</td><td>0.815</td><td>0.732</td></tr><tr><td>GloVe.Twitter.27B</td><td>0.776</td><td>0.813</td><td>0.727</td></tr><tr><td>GloVe.840B</td><td>0.877</td><td>0.859</td><td>0.779</td></tr><tr><td>ND 10M</td><td>0.885</td><td>0.849</td><td>0.795</td></tr><tr><td>ND 30M</td><td>0.889</td><td>0.867</td><td>0.806</td></tr><tr><td>200D + ND 30M</td><td>0.886</td><td>0.87</td><td>0.807</td></tr><tr><td>LL 200D + ND 10M</td><td>0.882</td><td>0.85</td><td>0.773</td></tr><tr><td>LL 200D + ND 30M</td><td>0.891</td><td>0.872</td><td>0.808</td></tr><tr><td>6.6 ASPECT EXTRACTION</td><td/><td/><td/></tr></table>

Performance of Concatenation with GloVe.860B

Table content: <table><tr><td>CC</td><td>KSO</td><td>CS</td><td>Laptops (f1)</td></tr></table>

Peer review (iclr2018):
 This paper presents a lifelong learning method for learning word embeddings. Given a new domain of interest, the method leverages previously seen domains in order to hopefully generate better embeddings compared to ones computed over just the new domain, or standard pre-trained embeddings. The general problem space here -- how to leverage embeddings across several domains in order to improve performance in a given domain -- is important and relevant to ICLR. However, this submission needs to be improved in terms of clarity and its experiments. In terms of clarity, the paper has a large number of typos (I list a few at the end of this review) and more significantly, at several points in the paper is hard to tell what exactly was done and why. When presenting algorithms, starting with an English description of the high-level goal and steps of the algorithm would be helpful. What are the inputs and outputs of the meta-learner, and how will it be used to obtain embeddings for the new domain? The paper states the purpose of the meta learning is "to learn a general word context similarity from the first m domains", but I was never sure what this meant. Further, some of the paper*s pseudocode includes unexplained steps like "invert by domain index" and "scanco-occurrence". In terms of the experiments, the paper is missing some important baselines that would help us understand how well the approach works. First, besides the GloVe common crawl embeddings used here, there are several other embedding sets (including the other GloVe embeddings released along with the ones used here, and the Google News word2vec embeddings) that should be considered. Also, the paper never considers concatenations of large pre-trained embedding sets with each other and/or with the new domain corpus -- such concatenations often give a big boost to accuracy, see : "Think Globally, Embed Locally—Locally Linear Meta-embedding of Words", Bollegala et al., 2017 https://arxiv.org/pdf/1709.06671.pdf That paper is not peer reviewed to my knowledge so it is not necessary to compare against the new methods introduced there, but their baselines of concatenation of pre-trained embedding sets should be compared against in the submission. Beyond trying other embeddings, the paper should also compare against simpler combination approaches, including simpler variants of its own approach. What if we just selected the one past domain that was most similar to the new domain, by some measure? And how does the performance of the technique depend on the setting of m? Investigating some of these questions would help us understand how well the approach works and in which settings. Minor: Second paragraph, GloVec should be GloVe "given many domains with uncertain noise for the new domain" -- not clear what "uncertain noise" means, perhaps "uncertain relevance" would be more clear The text refers to a Figure 3 which does not exist, probably means Figure 2. I didn*t understand the need for both figures, Figure 1 is almost contained within Figure 2 When m is introduced, it would help to say that m < n and justify why dividing the n domains into two chunks (of m and n-m domains) is necessary. "from the first m domain corpus" -> "from the first m domains"? "may not helpful" -> "may not be helpful" "vocabularie" -> "vocabulary" "system first retrieval" -> "system first retrieves" COMMENTS ON REVISIONS: I appreciate the authors including the new experiments against concatenation baselines. The concatenation does fairly comparably to LL in Tables 3&4. LL wins by a bit more in Table 2. Given these somewhat close/inconsistent wins, it would help the paper to include an explanation of why and under what conditions the LL approach will outperform concatenation.