Gated Convolutional Neural Networks for Domain Adaptation

Abstract: Domain Adaptation explores the idea of how to maximize performance on a target domain, distinct from source domain, upon which the classifier was trained. This idea has been explored for the task of sentiment analysis extensively. The training of reviews pertaining to one domain and evaluation on another domain is widely studied for modeling a domain independent algorithm. This further helps in understanding correlation between domains. In this paper, we show that Gated Convolutional Neural Networks (GCN) perform effectively at learning sentiment analysis in a manner where domain dependant knowledge is filtered out using its gates. We perform our experiments on multiple gate architectures: Gated Tanh ReLU Unit (GTRU), Gated Tanh Unit (GTU) and Gated Linear Unit (GLU). Extensive experimentation on two standard datasets relevant to the task, reveal that training with Gated Convolutional Neural Networks give significantly better performance on target domains than regular convolution and recurrent based architectures. While complex architectures like attention, filter domain specific knowledge as well, their complexity order is remarkably high as compared to gated architectures. GCNs rely on convolution hence gaining an upper hand through parallelization.

With the advancement in technology and invention of modern web applications like Facebook and Twitter, users started expressing their opinions and ideologies at a scale unseen before. The growth of e-commerce companies like Amazon, Walmart have created a revolutionary impact in the field of consumer business. People buy products online through these companies and write reviews for their products. These consumer reviews act as a bridge between consumers and companies. Through these reviews, companies polish the quality of their services. Sentiment Classification (SC) is one of the major applications of Natural Language Processing (NLP) which aims to find the polarity of text. In the early stages BIBREF0 of text classification, sentiment classification was performed using traditional feature selection techniques like Bag-of-Words (BoW) BIBREF1 or TF-IDF. These features were further used to train machine learning classifiers like Naive Bayes (NB) BIBREF2 and Support Vector Machines (SVM) BIBREF3 . They are shown to act as strong baselines for text classification BIBREF4 . However, these models ignore word level semantic knowledge and sequential nature of text. Neural networks were proposed to learn distributed representations of words BIBREF5 . Skip-gram and CBOW architectures BIBREF6 were introduced to learn high quality word representations which constituted a major breakthrough in NLP. Several neural network architectures like recursive neural networks BIBREF7 and convolutional neural networks BIBREF8 achieved excellent results in text classification. Recurrent neural networks which were proposed for dealing sequential inputs suffer from vanishing BIBREF9 and exploding gradient problems BIBREF10 . To overcome this problem, Long Short Term Memory (LSTM) was introduced BIBREF11 .
All these architectures have been successful in performing sentiment classification for a specific domain utilizing large amounts of labelled data. However, there exists insufficient labelled data for a target domain of interest. Therefore, Domain Adaptation (DA) exploits knowledge from a relevant domain with abundant labeled data to perform sentiment classification on an unseen target domain. However, expressions of sentiment vary in each domain. For example, in $\textit {Books}$ domain, words $\textit {thoughtful}$ and $\textit {comprehensive}$ are used to express sentiment whereas $\textit {cheap}$ and $\textit {costly}$ are used in $\textit {Electronics}$ domain. Hence, models should generalize well for all domains. Several methods have been introduced for performing Domain Adaptation. Blitzer BIBREF12 proposed Structural Correspondence Learning (SCL) which relies on pivot features between source and target domains. Pan BIBREF13 performed Domain Adaptation using Spectral Feature Alignment (SFA) that aligns features across different domains. Glorot BIBREF14 proposed Stacked Denoising Autoencoder (SDA) that learns generalized feature representations across domains. Zheng BIBREF15 proposed end-to-end adversarial network for Domain Adaptation. Qi BIBREF16 proposed a memory network for Domain Adaptation. Zheng BIBREF17 proposed a Hierarchical transfer network relying on attention for Domain Adaptation.
However, all the above architectures use a different sub-network altogether to incorporate domain agnostic knowledge and is combined with main network in the final layers. This makes these architectures computationally intensive. To address this issue, we propose a Gated Convolutional Neural Network (GCN) model that learns domain agnostic knowledge using gated mechanism BIBREF18 . Convolution layers learns the higher level representations for source domain and gated layer selects domain agnostic representations. Unlike other models, GCN doesn't rely on a special sub-network for learning domain agnostic representations. As, gated mechanism is applied on Convolution layers, GCN is computationally efficient.
Traditionally methods for tackling Domain Adaptation are lexicon based. Blitzer BIBREF19 used a pivot method to select features that occur frequently in both domains. It assumes that the selected pivot features can reliably represent the source domain. The pivots are selected using mutual information between selected features and the source domain labels. SFA BIBREF13 method argues that pivot features selected from source domain cannot attest a representation of target domain. Hence, SFA tries to exploit the relationship between domain-specific and domain independent words via simultaneously co-clustering them in a common latent space. SDA BIBREF14 performs Domain Adaptation by learning intermediate representations through auto-encoders. Yu BIBREF20 used two auxiliary tasks to help induce sentence embeddings that work well across different domains. These embeddings are trained using Convolutional Neural Networks (CNN).
Gated convolutional neural networks have achieved state-of-art results in language modelling BIBREF18 . Since then, they have been used in different areas of natural language processing (NLP) like sentence similarity BIBREF21 and aspect based sentiment analysis BIBREF22 .
In this section, we introduce a model based on Gated Convolutional Neural Networks for Domain Adaptation. We present the problem definition of Domain Adaptation, followed by the architecture of the proposed model.
Given a source domain $D_{S}$ represented as $D_{S}$ = { $(x_{s_{1}},y_{s_{1}})$ , $(x_{s_{2}},y_{s_{2}})$ .... $(x_{s_{n}},y_{s_{n}})$ } where $x_{s_{i}} \in \mathbb {R}$ represents the vector of $i^{th}$ source text and $y_{s_{i}}$ represents the corresponding source domain label. Let $T_{S}$ represent the task in source domain. Given a target domain $D_{T}$ represented as $D_{S}$0 = { $D_{S}$1 , $D_{S}$2 .... $D_{S}$3 }, where $D_{S}$4 represents the vector of $D_{S}$5 target text and $D_{S}$6 represents corresponding target domain label. Let $D_{S}$7 represent the task in target domain. Domain Adaptation (DA) is defined by the target predictive function $D_{S}$8 calculated using the knowledge of $D_{S}$9 and $(x_{s_{1}},y_{s_{1}})$0 where $(x_{s_{1}},y_{s_{1}})$1 but $(x_{s_{1}},y_{s_{1}})$2 . It is imperative to note that the domains are different but only a single task. In this paper, the task is sentiment classification.
The proposed model architecture is shown in the Figure 1 . Recurrent Neural Networks like LSTM, GRU update their weights at every timestep sequentially and hence lack parallelization over inputs in training. In case of attention based models, the attention layer has to wait for outputs from all timesteps. Hence, these models fail to take the advantage of parallelism either. Since, proposed model is based on convolution layers and gated mechanism, it can be parallelized efficiently. The convolution layers learn higher level representations for the source domain. The gated mechanism learn the domain agnostic representations. They together control the information that has to flow through further fully connected output layer after max pooling.
Let $I$ denote the input sentence represented as $I$ = { $w_{1}$ $w_{2}$ $w_{3}$ ... $w_{N}$ } where $w_{i}$ represents the $i_{th}$ word in $I$ and $N$ is the maximum sentence length considered. Let $I$0 be the vocabulary size for each dataset and $I$1 denote the word embedding matrix where each $I$2 is a $I$3 dimensional vector. Input sentences whose length is less than $I$4 are padded with 0s to reach maximum sentence length. Words absent in the pretrained word embeddings are initialized to 0s. Therefore each input sentence $I$5 is converted to $I$6 dimensional vector. Convolution operation is applied on $I$7 with kernel $I$8 . The convolution operation is one-dimensional, applied with a fixed window size across words. We consider kernel size of 3,4 and 5. The weight initialization of these kernels is done using glorot uniform BIBREF23 . Each kernel is a feature detector which extracts patterns from n-grams. After convolution we obtain a new feature map $I$9 = [ $w_{1}$0 ] for each kernel $w_{1}$1 . 
$$C_{i} = f(P_{i:i+h} \ast W_{a} + b_{a})$$   (Eq. 5) 
where $f$ represents the activation function in convolution layer. The gated mechanism is applied on each convolution layer. Each gated layer learns to filter domain agnostic representations for every time step $i$ . 
$$S_{i} = g(P_{i:i+h} \ast W_{s} + b_{s})$$   (Eq. 6) 
where $g$ is the activation function used in gated convolution layer. The outputs from convolution layer and gated convolution layer are element wise multiplied to compute a new feature representation $G_{i}$ 
$$G_{i} = C_{i} \times S_{i}$$   (Eq. 7) 
Maxpooling operation is applied across each filter in this new feature representation to get the most important features BIBREF8 . As shown in Figure 1 the outputs from maxpooling layer across all filters are concatenated. The concatenated layer is fully connected to output layer. Sigmoid is used as the activation function in the output layer.
Gating mechanisms have been effective in Recurrent Neural Networks like GRU and LSTM. They control the information flow through their recurrent cells. In case of GCN, these gated units control the domain information that flows to pooling layers. The model must be robust to change in domain knowledge and should be able to generalize well across different domains. We use the gated mechanisms Gated Tanh Unit (GTU) and Gated Linear Unit (GLU) and Gated Tanh ReLU Unit (GTRU) BIBREF22 in proposed model. The gated architectures are shown in figure 2 . The outputs from Gated Tanh Unit is calculated as $tanh(P \ast W + c) \times \sigma (P \ast V + c)$ . In case of Gated Linear Unit, it is calculated as $(P \ast W + c) \times \sigma (P \ast V + c)$ where $tanh$ and $\sigma $ denotes Tanh and Sigmoid activation functions respectively. In case of Gated Tanh ReLU Unit, output is calculated as $tanh(P \ast W + c) \times relu(P \ast V + c)$ 
Multi Domain Dataset BIBREF19 is a short dataset with reviews from distinct domains namely Books(B), DVD(D), Electronics(E) and Kitchen(K). Each domain consists of 2000 reviews equally divided among positive and negative sentiment. We consider 1280 reviews for training, 320 reviews for validation and 400 reviews for testing from each domain.
Amazon Reviews Dataset BIBREF24 is a large dataset with millions of reviews from different product categories. For our experiments, we consider a subset of 20000 reviews from the domains Cell Phones and Accessories(C), Clothing and Shoes(S), Home and Kitchen(H) and Tools and Home Improvement(T). Out of 20000 reviews, 10000 are positive and 10000 are negative. We use 12800 reviews for training, 3200 reviews for validation and 4000 reviews for testing from each domain.
To evaluate the performance of proposed model, we consider various baselines like traditional lexicon approaches, CNN models without gating mechanisms and LSTM models.
Bag-of-words (BoW) is one of the strongest baselines in text classification BIBREF4 . We consider all the words as features with a minimum frequency of 5. These features are trained using Logistic Regression (LR).
TF-IDF is a feature selection technique built upon Bag-of-Words. We consider all the words with a minimum frequency of 5. The features selected are trained using Logistic Regression (LR).
Paragraph2vec or doc2vec BIBREF25 is a strong and popularly used baseline for text classification. Paragraph2Vec represents each sentence or paragraph in the form of a distributed representation. We trained our own doc2vec model using DBOW model. The paragraph vectors obtained are trained using Feed Forward Neural Network (FNN).
To show the effectiveness of gated layer, we consider a CNN model which does not contain gated layers. Hence, we consider Static CNN model, a popular CNN architecture proposed in Kim BIBREF8 as a baseline.
Wang BIBREF26 proposed a combination of Convolutional and Recurrent Neural Network for sentiment Analysis of short texts. This model takes the advantages of features learned by CNN and long-distance dependencies learned by RNN. It achieved remarkable results on benchmark datasets. We report the results using code published by the authors.
We offer a comparison with LSTM model with a single hidden layer. This model is trained with equivalent experimental settings as proposed model.
In this baseline, attention mechanism BIBREF27 is applied on the top of LSTM outputs across different timesteps.
All the models are experimented with approximately matching number of parameters for a solid comparison using a Tesla K80 GPU.
Input Each word in the input sentence is converted to a 300 dimensional vector using GloVe pretrained vectors BIBREF28 . A maximum sentence length 100 is considered for all the datasets. Sentences with length less than 100 are padded with 0s.
Architecture details: The model is implemented using keras. We considered 100 convolution filters for each of the kernels of sizes 3,4 and 5. To get the same sentence length after convolution operation zero padding is done on the input.
Training Each sentence or paragraph is converted to lower case. Stopword removal is not done. A vocabulary size of 20000 is considered for all the datasets. We apply a dropout layer BIBREF29 with a probability of 0.5, on the embedding layer and probability 0.2, on the dense layer that connects the output layer. Adadelta BIBREF30 is used as the optimizer for training with gradient descent updates. Batch-size of 16 is taken for MDD and 50 for ARD. The model is trained for 50 epochs. We employ an early stopping mechanism based on validation loss for a patience of 10 epochs. The models are trained on source domain and tested on unseen target domain in all experiments.
The performance of all models on MDD is shown in Tables 2 and 3 while for ARD, in Tables 4 and 5 . All values are shown in accuracy percentage. Furthermore time complexity of each model is presented in Table 1 .
We find that gated architectures vastly outperform non gated CNN model. The effectiveness of gated architectures rely on the idea of training a gate with sole purpose of identifying a weightage. In the task of sentiment analysis this weightage corresponds to what weights will lead to a decrement in final loss or in other words, most accurate prediction of sentiment. In doing so, the gate architecture learns which words or n-grams contribute to the sentiment the most, these words or n-grams often co-relate with domain independent words. On the other hand the gate gives less weightage to n-grams which are largely either specific to domain or function word chunks which contribute negligible to the overall sentiment. This is what makes gated architectures effective at Domain Adaptation.
In Figure 3 , we have illustrated the visualization of convolution outputs(kernel size = 3) from the sigmoid gate in GLU across domains. As the kernel size is 3, each row in the output corresponds to a trigram from input sentence. This heat map visualizes values of all 100 filters and their average for every input trigram. These examples demonstrate what the convolution gate learns. Trigrams with domain independent but heavy polarity like “_ _ good” and “_ costly would” have higher weightage. Meanwhile, Trigrams with domain specific terms like “quality functional case” and “sell entire kitchen” get some of the least weights. In Figure 3 (b) example, the trigram “would have to” just consists of function words, hence gets the least weight. While “sell entire kitchen” gets more weight comparatively. This might be because while function words are merely grammatical units which contribute minimal to overall sentiment, domain specific terms like “sell” may contain sentiment level knowledge only relevant within the domain. In such a case it is possible that the filters effectively propagate sentiment level knowledge from domain specific terms as well.
We see that gated architectures almost always outperform recurrent, attention and linear models BoW, TFIDF, PV. This is largely because while training and testing on same domains, these models especially recurrent and attention based may perform better. However, for Domain Adaptation, as they lack gated structure which is trained in parallel to learn importance, their performance on target domain is poor as compared to gated architectures. As gated architectures are based on convolutions, they exploit parallelization to give significant boost in time complexity as compared to other models. This is depicted in Table 1 .
While the gated architectures outperform other baselines, within them as well we make observations. Gated Linear Unit (GLU) performs the best often over other gated architectures. In case of GTU, outputs from Sigmoid and Tanh are multiplied together, this may result in small gradients, and hence resulting in the vanishing gradient problem. However, this will not be the in the case of GLU, as the activation is linear. In case of GTRU, outputs from Tanh and ReLU are multiplied. In ReLU, because of absence of negative activations, corresponding Tanh outputs will be completely ignored, resulting in loss of some domain independent knowledge.
In this paper, we proposed Gated Convolutional Neural Network(GCN) model for Domain Adaptation in Sentiment Analysis. We show that gates in GCN, filter out domain dependant knowledge, hence performing better at an unseen target domain. Our experiments reveal that gated architectures outperform other popular recurrent and non-gated architectures. Furthermore, because these architectures rely on convolutions, they take advantage of parellalization, vastly reducing time complexity.

Figure and table captions:
Fig. 1: Architecture of the proposed model
Fig. 2: Variations in gates of the proposed GCN architecture.
Table 1: Average training time for all the models on ARD
Table 2: Accuracy scores on Multi Domain Dataset.
Table 3: Accuracy scores on Multi Domain Dataset.
Table 4: Accuracy scores on Amazon Reviews Dataset.
Table 5: Accuracy scores on Amazon Reviews Dataset.
Fig. 3: Visualizing outputs from gated convolutions (filter size = 3) of GLU for example sentences, darker indicates higher weightage

Questions about this paper:
For the purposes of this paper, how is something determined to be domain specific knowledge?

Answers:
reviews under distinct product categories are considered specific domain knowledge