Title: TRANSFERRING SLU MODELS IN NOVEL DOMAINS Anonymous authors Paper under double-blind review

Abstract: Spoken language understanding (SLU) is a critical component in building dialogue systems. When building models for novel natural language domains, a major challenge is the lack of data in the new domains, no matter whether the data is annotated or not. Recognizing and annotating "intent" and "slot" of natural languages is a time-consuming process. Therefore, spoken language understanding in low resource domains remains a crucial problem to address. In this paper, we address this problem by proposing a transfer-learning method, whereby a SLU model is transferred to a novel but data-poor domain via a deep neural network framework. We also introduce meta-learning in our work to bridge the semantic relations between seen and unseen data, allowing new intents to be recognized and new slots to be filled with much lower new training effort. We show the performance improvement with extensive experimental results for spoken language understanding in low resource domains. We show that our method can also handle novel intent recognition and slot-filling tasks. Our methodology provides a feasible solution for alleviating data shortages in spoken language understanding.

1. INTRODUCTION:

The recent surge of artificial intelligence motivates the technical and applicable exploration of novel human-computer interactions. Spoken dialogue systems are widely studied and used in various mobile devices. Well-known commercial applications driven by dialogue systems include intelligent personal assistants and intelligent robots for customer services. As more and more products and scenarios integrate dialogue systems in intelligent services, the ability for model adaptation in dialogue systems is critically needed. Many state-of-the art dialogue systems follow a learning pipeline that includes components such as spoken language understanding (SLU), dialogue management (DM) and spoken language generation or retrieval (SLG). Since both dialogue management and spoken language generation systems rely on knowledge or information learned from spoken language understanding, research and industry community have paid much attention to the SLU area. In this paper, we focus on the problem of data shortage for SLU in a new domain.

Spoken language understanding typically defines and represents user utterances in terms of semantic frames comprised of domains, intents and slots (Tur & Mori, 2011) . A spoken language understanding task involves classifying the domain, detecting the intent of user utterance and identifying token sequences corresponding to slot values in the semantic frame. A learning model is often designed to handle the above sub-tasks separately and sometimes simultaneously. Deep learning models are the cutting-edge techniques for achieving impressive performance in spoken language understanding compared with conventional machine learning models.

However, deep neural-network models often require the collection and manual annotation of data, which is a time and labor intensive process. These problems pose a major challenge to building high-quality learning models in new domains. To address this problem, we propose to perform domain adaption of SLU models by transferring the SLU knowledge and model parameters learned in an auxiliary domain where a model is already built. A related problem is that only few examples of some output intent and slot classes are available during the training phrase in a new domain, because training sets for a new domain are small. To deal with this limitation, we propose a novel few shot learning model in a meta learning paradigm, we call it the Transfer Semantic Similarity Model (TSSM). The TSSM achieves few-shot learning by exploiting the semantic embeddings of both seen and unseen intent and slots.

In summary, the challenges of spoken language understanding include: (1) Expanding SLUs to novel domains is necessary as the development of artificial intelligent products extends to more areas in our society. But it is difficult to get sufficient data or spoken language sentence in novel domains.

(2) Without sufficient labeled data, deep learning methods are difficult to train and thus it is hard to obtain satisfactory performance in a new domain. (3) Unseen intents and slots exist because small annotated training set might not cover all intents and slots. Thus, the generalization ability of the models is crucial for dealing with unseen classes.

Our contributions are summarized as follows:

• We design a novel and efficient SLU model, the TSSM. It incorporates transfer learning and meta learning for SLU tasks in low resource domains.

• The proposed TSSM can improve model performance with only a small amount of training data in new domains and it can handle unseen classes. The experiments illustrate that our proposed methods outperform the state-of-the-art baselines in many experimental domains.

• Our work can treat intent and slot detection tasks as a structured multi-objective optimization problem, which further improves the accuracy of learning tasks in new domains.

2. RELATED WORK:

Spoken language understanding is usually defined as a supervised learning problem, involving conventional machine learning models (Young, 2002; Hahn et al., 2011; Wang et al., 2005) and deep learning models (Mesnil et al., 2015; Kurata et al., 2016; Sarikaya et al., 2011) on massive amount of annotated training data. Although unsupervised learning and semi-supervised learning based approaches have been proposed as well (Chen et al., 2013; 2015b; a) , deep learning approaches were shown to outperform most others. Besides the requirement of a large amount of annotated data being available, the intents and slots are usually predefined. As a result, they are inflexible to expand to new domains. Some researchers have paid attention to alleviate this limitation. An example is the work by Korpusik et al. , which retrains models to cover new intents and associated slots while redesigning a semantic schema (Korpusik et al., 2014) .

There have also been some attempts to learn shared latent semantic representations or model parameters for multiple domains via transfer learning. A survey for transfer learning with applications is Pan & Yang (2010) . In general, the goal of transfer learning is to improve performance in novel domains that only has small data. In these target domains, it is critical to reduce the model re-training efforts by bridging the knowledge from source tasks to the new target tasks. In natural language processing (NLP), researchers have studied cross-domain NLP problems but have mainly focused in cross-lingual problem settings (Mou et al., 2016; Buys & Botha, 2016; Kim et al., 2017a; b) .

In Hakkani-Tr et al. (2016) , both multi-task and transfer learning approaches are proposed to learn shared implicit feature representation across various domains for spoken language understanding. Researchers in Amazon have shown in a recent study that DNN-based natural language engines can re-use the existing models through transfer learning (Goyal et al., 2018) .

When the training data is missing some task labels, one can exploit meta learning to apply few-shot learning to help with the knowledge transfer Fei-Fei et al. (2006) ; Palatucci et al. (2009) ; Vinyals et al. (2016) ; Ravi & Larochelle (2018) . For example, Vinyals et al. (2016) learns a network that maps a small amount of labeled support set and unlabeled examples to its labels, saving the effort in fine-tuning to new class types. Besides image object-recognition tasks, in SLU some experiments have been conducted. The objective of Ferreira et al. (2015) is to predict the semantic tag sequences of a user query without using any target-domain user utterances and thus in-context semantic tags.

The research builds a statistical model to predict the SLU for unseen data in Yazdani & Henderson (2015) ; Chen et al. (2016) . An action-matching algorithm is proposed by Zhao & Eskénazi (2018) to learn a cross-domain embedding space that models the semantics of dialog responses, which, in turn, lets a neural dialog generation model generalize to new domains. Recent publication by Google (Bapna et al., 2017) explores semi-supervised slot-filling based on deep learning that can use the slot labels without the need for any labeled or unlabeled data in domain examples or the need for any explicit schema alignment, to quickly bootstrap the model in a new domain.

3. OUR PROPOSED APPROACH FOR MODEL TRANSFER:

In this section, we introduce our transfer learning approach to address the cross-domain SLU problem. We consider a source domain D s and a target domain D t , where our goal is to learn a model in the target domain. Typically, we have a large annotated dataset for D s , while only a few labeled data for D t . The tasks in both D s and D t are of the same types, namely they are both slot filling and intent detection tasks. However, D s and D t have different intent label spaces T s and T t , because the target domain might have new intent values (slot faces the same situation).

The number of possible intents and slots grows rapidly as we get into new domains with new intents and slots. One natural approach in solving the learning problem is to train one binary classifier for each possible label, and decide whether or not to include the classification model in the output. However, this would require training a large number of classifiers. It also would be impossible to generalize to target domains that include intents and slots that do not show up in the training set, since there will not be any parameter sharing among the classifiers. Instead of encoding slot names and intent names as discrete features in the target domain, we encode them using a continuous representation. To be specific, we learn a representation of slot names, slot values and intent names from their constituent words using semantic networks. We then check to see if these representations match the representations of utterances.

The architecture of our model is illustrated in Fig. 1 . The model includes the following major components.

(1) a word hashing layer to convert one-hot word representations into embedding vectors, (2) a bi-directional Long Short Term Memory networks(LSTMs) (Hochreiter & Schmidhuber, 1997) layer to capture features from word embeddings, (3) a intent specific bi-LSTMs layer combined with an semantic network to detect intent, (4) a slot specific bi-LSTMs layer powered with an attention component and semantic network to conduct slot filling. We first pretrain the model in a source domain with adequate annotated data. We then wish to allow the model to extend to new slot names, slot values and intent names in the target domain with little annotated data. Let w = (w 0 , w 1 , w 2 , . . . , w T +1 ) represent the input word sequence with w 0 and w T +1 being the beginning-of-sequence ( bos ) and end-of-sequence ( eos ) tokens, respectively. T is the number of words. Let s = (s 0 , s 1 , . . . , s T ) be the slot label sequence, where s 0 is a padded slot label that maps to the beginning-of-sequence token bos and s i ∈ S. Let τ ∈ T be the intent class. S and T are the slot and intent label space, respectively. Let sn = (sn 1 , sn 2 , . . . , sn m ) represent the m slot names. sn i = (w 1 , . . . , w mi ) is a vector of the words for the i-th slot name, where m i is its number of words. Each slot s i has a list of slot values sv i = (sv i,1 , sv i,2 , . . . , sv i,ni ), where n i is the number of slot values for the i-th slot. A slot value is allowed to be a phrase like 'fried chicken' for slot 'Food'. Let sv i,j = (w 1 , . . . , w ni,j ) be the words of the j-th slot value for i-th slot and n i,j is the number of words for this slot value.

3.1. BASE NETWORKS:

We use a Pretrained DNN model as our baseline model (Goyal et al., 2018) . The model has three layers of bi-LSTMs. The first layer is the common bi-LSTMs layer, which learns task related representations from input utterances. The learned representations are taken as input to the upper two bi-LSTMs layers, which are optimized separately for the slot-filling task and intent-detection task.

Our model differs from the baseline model in three aspects. First, Goyal et al. ( 2018) treated intents and slots as discrete symbols and train classifiers to make predictions on these symbols. Such an approach limits the knowledge transfer between two domains as the classifier layers (affine transform leading to the softmax) need be re-initialized when we transfer the model via fine-tuning in new domains, where the output labels are different. In our model, we encode intents and slots as continuous representations via their constituent words. The classification problems are transformed to semantic similarity problems and the whole model parameters could be transferred to new domains.

The second difference is the usage of gazetteers (lists of slot values for a slot). Goyal et al. ( 2018) used gazetteer features as an additional input. Such features are binary indicators of the presence of an n-gram in a gazetteer. In our model, we used an attention network (Section 3.3) to encode external slot values into a semantic vector that represents the slot from the value perspective, which suits our semantic framework naturally.

Finally, there are no connections between the upper two layers in the baseline by Goyal et al. (2018) . However, we believe that the output of intent specific bi-LSTMs layer could benefit the slot detection task. As one of our contributions, in our network, we concatenate the output of common bi-LSTMs layer and intent bi-LSTMs layer to feed to the slot bi-LSTMs layer.

For the intent specific bi-LSTMs layer, we concatenate the last hidden state of each direction to acquire the global representation of the whole utterance

h intent = [ -→ h intent T +1 , ← - h intent 0 ]

. Such an operation converts utterances with variable lengths into a fixed-length vector, with which the information throughout the entire utterance can be captured. We have also tried other methods like take Mean-Pooling or Max-Pooling of bi-LSTM hidden states to acquire h intent but get similar results.

3.2. SEMANTIC NETWORKS:

We wish to build a representation-learning mechanism for the SLU task that can generalize to unseen words and labels. For slot names, slot values and intent names, we use the same semantic network and transform them to fixed-length representation vectors. Take slot name sn i = (w 1 , w 2 , . . . , w mi ) as an example, the semantic network first extracts the word embedding E(w i ) for each word. E(w i ) is then fed to a feed forward layer to get the non-linear semantic features. The last is a Mean-Pooling layer, which applies the mean operations over each dimension of semantic features across all words. The output is treated as the semantic representation for sn i ,

EQUATION

where W slot transf rom and b slot transf rom are the learned linear projection matrix and bias for slot name semantic network respectively. In the same way, we could get the semantic representation r slot value i,j

for slot value sv i,j and r intent name i for the intent name τ i . More sophisticated vectorspace semantic representations of the slots/intents are an area for future work, but we believe the simple Mean-Pooling would be a proper choice here since intent names, slot names and slot values are typically very short and composed by entity names.

3.3. SLOT VALUE ATTENTION NETWORKS:

With semantic networks, we formulate a semantic representation r slot value i,j of each slot value sv i,j from the i-th candidate slot s i . The whole slot value representation list is r slot value i,1 , . . . , r slot value i,ni

. At each decoding step t, the slot bi-LSTMs layer output state h slot t is connected to each slot's slot values' representations to calculate the similarity scores. Take slot s i as an example, we compute an attention weight α t,i,j for slot value sv i,j via a bi-linear operator, which reflects how relevant or important slot value sv i,j is to the current slot hidden states h slot t , αt,i,j ∝ exp((r slot value i,j

) Wvh slot t ) (2)

where W v is a parameter matrix. The overall slot value representation for slot s i would be

r slot values t,i = n i j αt,i,jr slot value i,j

.

(3)

3.4. TRAINING PROCEDURE:

The source domain data containing utterances and associated intents/slots is used to train the model. The idea behind this model is to learn the representation for both utterances and intents/slots such that utterances with the same intents/slots are close to each other in the continuous semantic space.

We define a semantic score between an utterance and an intent/slot using the similarity between their representation embeddings h and r by Sim(h, r) = h W s r. W s is the corresponding similarity matrix for slot or intent task.

The posterior probability of a possible intent given an utterance is computed based on the semantic score through a softmax function

EQUATION

where τ i is an intent candidate. Similarly, we can get the posterior probability of a possible slot at each time. Since we have the representations of both slot name and slot value, we first add the semantic scores from slot name and slot value and then feed to the softmax function to get the overall slot posterior probability

EQUATION

EQUATION

where s i is an slot candidate. For model training, we maximize the likelihood of the correctly associated intents/intents given all training utterances. The parameters W of the model are optimized by an joint negative log likelihood objective:

L = - l k=1 log P τ = τi|h intent w k , w k - l k=1 T k t=0 log P st = si|h slot t,w k , w k (7)

where l is the total number of training utterances, w k is the k-th input utterance. The overall objective function combines the intent classification task and slot filling task objectives. We observe that this multitask architecture achieves better results than separately training intent and slot models.

The model is optimized using mini-batch stochastic gradient descent(SGD) (Huang et al., 2013) .

3.5. TRANSFER TO A NEW DOMAIN:

We train our TSSM using labeled data from source domain D s . This stage is called pretraining, where the networks learn to extract semantic relationships between utterance representations and intent/slot representations. The model is able to adapt to new domains, since the seen and unseen intents/slots representations are in the same semantic space, use the same shared composition semantic network with the same unsupervised word embeddings as input.

After the pretraining, we fine-tune our model on the target domain D t with less annotated data and with unseen intents/slots. The model for the new domain D t are the same as that for D s and are initialized by the pretrained network parameters. We tried fine-tune different components of the model. We fix some parts and fine-tune on the other parts. After experiments on several combinations of components, we found that fine-tune the whole model gives us the best performance.

For zero-shot learning where no fine-tune data is available, in order to predict new intents/slots on new utterances, the TSSM calculates semantic similarities. We transform each input utterance w into a vector h intent using the base network, and then estimate its semantic similarity with all intents including seen and unseen intents. The vector representations for new intents can be generated from the trained semantic network by feeding the word embedding vectors of new intents as the input. For the utterance w, the estimated semantic score of the i-th intent is defined as Sim(h intent , r intent i ) in Equation 3.4. Then predicted intent for each utterance is decided according to the estimated semantic scores. Predicting slot sequences is similar to predicting intents except that the calculation is conducted step by step at each position.

4. EXPERIMENTS AND RESULTS:

In this experiment, we use five popular public dialogue datasets from five domains and three datasets obtained from practical applications. The CamHotel dialogue dataset (Hotel) (Wen et al., 2015) records some dialogues about booking a hotel room in Cambridge. The CamRestaurant dialogue dataset (Restaurant) (Wen et al., 2016c; a) contains dialogue records about booking a restaurant in Cambridge. The Laptop and TV dialogue dataset (Wen et al., 2016b ) are dialogues about buying Laptops and TVs. The Airline Travel Information System dataset (Atis) (Price, 1990 ) contains dialogues about airline travel, such as the flight time, the destination, etc. Weilidai, Huobijijin and Weizhongyouzhe are three financial products from WeBank, an internet bank in China. While Weilidai is very popular and its online customer service accumulated a lot of queries, Huobijijin and Weizhongyouzhe are newly developed products and have only few queries. We would like to pretrain on Weilidai queries and fine-tune on Huobijijin and Weizhongyouzhe queries to improve the online customer service's experience of the last two products.

In order to thoroughly evaluate the transfer learning ability on more diversified domains, we make five different pairs of source and target domains, as shown in table 1. In case when we have multiple source domains, all source domains will be combined into a large source domain. For each pair of the source and target domain pair, we first pretrain the model on the source domain. Then we finetune the model on the target domain training data, finally we evaluate the fine-tuned model on the target domain test set. We use accuracy to evaluate the intent classification task and we use accuracy and F1 score to evaluate the slot filling task. Apart from the Pretrained DNN (Goyal et al., 2018) , we also compare TSSM to Max Entropy model and Conditional Random Field model (CRF). In order to evaluate the transfer learning performance of the proposed model, we compare the proposed model with the baselines when we have different number of target domain training data. After pretraining in the source domain, we fine-tune the model on different percentages of the target domain data, and we evaluate the performance of the models on the test set. We use 50% of the target domain data as test set, and we vary the percentage of train set used from 5% to 46%.

The results are shown in Figure 2 . As a higher percentage of target domain data is used in finetuning, the performances of all models improve. Generally, transfer learning models perform better than non-transfer models, which implies that transfer learning can help improve the performance in new domain. The proposed TSSM model outperforms the baseline model significantly in almost all cases. In some cases, the proposed TSSM is not better than DNN. This might because our model has more parameters than the baseline DNN model and in some cases simper model performs better.

Compared with intent classification task, the slot filling task benefit more from transferring from the source domain. This is because the intent has larger domain differences compared with the slots. For example, the intent in the hotel domain contains verbs such as "inform" "inform no match" while the intent in the Atis domain contains nouns such as "flight" "aircraft". While the slots in all domains mostly contain nouns. The improvement of transferring from Atis to Hotel dataset is worse than the other source and target pairs. We think this is because the intent and slot difference between "Atis" and 'Hotel' domains are bigger than the domains difference of other tasks.

4.2. RESULTS UNDER DIFFERENT NUMBERS OF NEW LABELS IN THE TARGET DOMAIN:

In order to evaluate the generalization ability of the TSSM model, we test the model on previously unseen labels. More specifically, we remove zero, one and two labels and their corresponding training instances from the target domain training set. Then we evaluate TSSM on the target domain In each figure, X-axis is the percentage of data used in the target domain, yaxis is the performance (higher is better). The three columns corresponds to intent accuracy (left column), slot accuracy (middle column) and slot f1 score (right column). The five rows corresponds to different source and target domain pairs. From the top to the bottom are "Restaurant, Laptop, TV, Atis" to "Hotel" "Restaurant, Hotel, Laptop, TV" to "Atis" "Atis" to "Hotel" "Weilidai" to "Weizhongyouzhe" and "Weilidai" to "Huobijijin".

test set which has full lists of labels. The results are shown in Figure 3 . Here we use 10% of all target domain data as the training set, when the percentage of target domain training data increases to 20% or 30%, the results are similar. As the number of unseen label increases in the target domain test set, the performance of all model drops significantly. Transfer learning methods again out-performs none-transfer learning baselines, which demonstrate that transferring knowledge from source domain is generally beneficial to the target domain.

As the number of unseen label increases, the performance drop of the TSSM model is significantly slower than the other baselines models. Compared with a softmax classification layer, the semantic similarity module works better since it can still compute semantic similarity for unseen labels. With a pretrained word embeddings, the TSSM can theoretically compute semantic similarity for arbitrary label as long as the label has a word embedding. The experiments demonstrate that the TSSM model can effectively deal with cold start labels that is not seen in the target domain training data.

4.3. ZERO-SHOT TRANSFER LEARNING:

In the extreme case, there could be no training instances in the target domain at all. In order to test the proposed TSSM model on such extreme case, we report the performances of the TSSM model when there is no target domain training instances. The result is shown in Table 2 . The random guess performance is 0.05 for intent classification accuracy, 0.02 for slot accuracy and 0.02 for slot f1.

The TSSM have an performance much better than random guess. In such case, both the Pretrained DNN, the Max Entropy model and the CRF model could not work, since all of them require target domain training instances to initialize their softmax layer. One of the improvements we have made is to use the intent bi-LSTMs output states as input features to the slot bi-LSTMs, we call this mechanism "intent2slot". To evaluate the effectiveness of "intent2slot" mechanism, we compare the TSSM with its ablation variant without "intent2slot", and the result is shown in Table 3 . The results demonstrate that the "intent2slot" mechanism helps to improve intent classification performance. The parameters in intent bi-LSTMs layer might have obtained additional supervision from the upper slot filling task. The improvements for transferring from Restaurant, Laptop, TV, Atis to Hotel is not obvious and the reason needs further experiments. 

5. CONCLUSIONS:

In this paper, we introduced a deep semantic similarity network to transfer from domains with sufficient labeled data to low resource domains. The experiments illustrated that our proposed methods outperform the state-of-the-art baselines in several experimental settings. In the future, we will consider different regularization choices to make our model more robust in changing data situations. We will also consider different data set types for dialog systems.

:



Figures and Tables:

Figure 1: TSSM architecture for joint SLU. The orange lines are inputs for semantic similarity network and the green lines represent the inputs for attention network.

Figure 2: Performance comparison under different percentage of training data used in the target domain fine-tuning.In each figure, X-axis is the percentage of data used in the target domain, yaxis is the performance (higher is better). The three columns corresponds to intent accuracy (left column), slot accuracy (middle column) and slot f1 score (right column). The five rows corresponds to different source and target domain pairs. From the top to the bottom are "Restaurant, Laptop, TV, Atis" to "Hotel" "Restaurant, Hotel, Laptop, TV" to "Atis" "Atis" to "Hotel" "Weilidai" to "Weizhongyouzhe" and "Weilidai" to "Huobijijin".

Figure3: Performance vs the number of unseen intent/slot in the target domain test set, where 10% of the total target domain data is used. In each figure, X-axis is the number of unseen intent/slots in the target domain test set, y-axis is the performance (higher is better). Table 2: The zeros-shot performance of TSSM Source and Target Intent Accuracy Slot Accuracy Slot F1 nohotel2hotel 0.787 0.871 0.870 atis2hotel 0.478 0.756 0.759 noatis2atis 0.220 0.617 0.616

Data Sizes of the source and target domains.

Table content: <table><tr><td>Source Domain</td><td>Target Domain</td><td colspan="2"># Source Sentence # Target Sentence</td></tr><tr><td>Restaurant, Laptop, TV, Atis</td><td>Hotel</td><td>48566</td><td>2168</td></tr><tr><td colspan="2">Restaurant, Hotel, Laptop, TV Atis</td><td>45756</td><td>4978</td></tr><tr><td>Atis</td><td>Hotel</td><td>4978</td><td>2168</td></tr><tr><td>Weilidai</td><td>Huobijijin</td><td>5730</td><td>927</td></tr><tr><td>Weilidai</td><td colspan="2">Weizhongyouzhe 5730</td><td>1053</td></tr><tr><td colspan="3">4.1 RESULTS UNDER DIFFERENT SIZES IN TARGET-DOMAIN DATA SETS</td><td/></tr></table>

The zeros-shot performance of TSSM Source and Target Intent Accuracy Slot Accuracy

Table content: <table><tr><td>Slot F1</td></tr></table>

The TSSM with/without "intent2slot"

Table content: <table><tr><td colspan="3">Source and Target Intent Accuracy</td><td colspan="2">Slot Accuracy</td><td>Slot F1</td><td/></tr><tr><td/><td colspan="6">TSSM-abl TSSM TSSM-abl TSSM TSSM-abl TSSM</td></tr><tr><td>nohotel2hotel</td><td>0.949</td><td>0.952</td><td>0.924</td><td>0.926</td><td>0.938</td><td>0.935</td></tr><tr><td>atis2hotel</td><td>0.945</td><td>0.952</td><td>0.937</td><td>0.938</td><td>0.945</td><td>0.945</td></tr><tr><td>noatis2atis</td><td>0.807</td><td>0.840</td><td>0.891</td><td>0.896</td><td>0.899</td><td>0.888</td></tr></table>

Peer review (iclr2019):
 Summary: The authors present a network which facilitates cross-domain learning for SLU tasks where the the goal is to resolve intents and slots given input utterances. At a high level, the authors argue that by fine-tuning a pre-trained version of the network on a small set of examples from a target-domain they can more effectively learn the target domain than without transfer learning. Feedback: * An overall difficulty with the paper is that it is hard to distinguish the authors* contributions from previous works. For example, in Section 3.1, the authors take the model of Goyal et al. as a starting point but explain only briefly one difference (contatenating hidden layers). In Section 3.2 the contributions becomes even harder to disentangle. For example, how does this section relate to other word-embeddings papers cited in this section? Is the proposed method a combination of previous works, and if not, what are the core new ideas? * Some sections are ad-hoc and should be justified/explained better. For example, the objective, which ultimately determines the trained model behaviour uses a product of experts formulation, yet the authors do not discuss this. Similarly, the overarching message, that by fine-tuning a suitable model initialisation using small amounts of data from the target domain is fairly weak as the authors do not detail exactly how the model is fine-tuned. Presumably, given only a small number of examples, this fine-tuning runs the risk of overfitting, unless some form of regularisation is applied, but this is not discussed. * Lastly, there are some curious dips in the plots (e.g., Figure 2 bottom left, Figure 3 top left, bottom left), which deserve more explanation. Additionally, the evaluation section could be improved if the scores were to show error-bars. Minor: All plots should be modified so they are readable in grey-scale.