Enhancing Sentence Relation Modeling with Auxiliary Character-level Embedding

Abstract: Neural network based approaches for sentence relation modeling automatically generate hidden matching features from raw sentence pairs. However, the quality of matching feature representation may not be satisfied due to complex semantic relations such as entailment or contradiction. To address this challenge, we propose a new deep neural network architecture that jointly leverage pre-trained word embedding and auxiliary character embedding to learn sentence meanings. The two kinds of word sequence representations as inputs into multi-layer bidirectional LSTM to learn enhanced sentence representation. After that, we construct matching features followed by another temporal CNN to learn high-level hidden matching feature representations. Experimental results demonstrate that our approach consistently outperforms the existing methods on standard evaluation datasets.

Traditional approaches BIBREF0 , BIBREF1 , BIBREF2 for sentence relation modeling tasks such as paraphrase identification, question answering, recognized textual entailment and semantic textual similarity prediction usually build the supervised model using a variety of hand crafted features. Hundreds of features generated at different linguistic levels are exploited to boost classification. With the success of deep learning, there has been much interest in applying deep neural network based techniques to further improve the prediction performances BIBREF3 , BIBREF4 , BIBREF5 .
A key component of deep neural network is word embedding which serve as an lookup table to get word representations. From low level NLP tasks such as language modeling, POS tagging, name entity recognition, and semantic role labeling BIBREF6 , BIBREF7 , to high level tasks such as machine translation, information retrieval and semantic analysis BIBREF8 , BIBREF9 , BIBREF10 . Deep word representation learning has demonstrated its importance for these tasks. All the tasks get performance improvement via further learning either word level representations or sentence level representations. On the other hand, some researchers have found character-level convolutional networks BIBREF11 , BIBREF12 are useful in extracting information from raw signals for the task such as language modeling or text classification.
In this work, we focus on deep neural network based sentence relation modeling tasks. We explore treating each sentence as a kind of raw signal at character level, and applying temporal (one-dimensional) Convolution Neural Network (CNN) BIBREF6 , Highway Multilayer Perceptron (HMLP) and multi-layer bidirectional LSTM (Long Short Term Memory) BIBREF13 to learn sentence representations. We propose a new deep neural network architecture that jointly leverage pre-trained word embedding and character embedding to represent the meaning sentences. More specifically, our new approach first generates two kinds of word sequence representations. One kind of sequence representations are the composition of pre-trained word vectors. The other kind of sequence representation comprise word vectors that generating from character-level convolutional network. We then inject the two sequence representations into bidirectional LSTM, which means forward directional LSTM accept pre-trained word embedding output and backward directional LSTM accept auxiliary character CNN embedding output. The final sentence representation is the concatenation of the two direction. After that, we construct matching features followed by another temporal CNN to learn high-level hidden matching feature representations. Figure FIGREF1 shows the neural network architecture for general sentence relation modeling.
Our model shows that when trained on small size datasets, combining pre-trained word embeddings with auxiliary character-level embedding can improve the sentence representation. Word embeddings can help capturing general word semantic meanings, whereas char-level embedding can help modeling task specific word meanings. Note that auxiliary character-level embedding based sentence representation do not require the knowledge of words or even syntactic structure of a language. The enhanced sentence representation generated by multi-layer bidirectional LSTM will encapsulate the character and word levels informations. Furthermore, it may enhance matching features that generated by computing similarity measures on sentence pairs. Quantitative evaluations on standard dataset demonstrate the effectiveness and advantages of our method.
Besides pre-trained word vectors, we are also interested in generating word vectors from characters. To achieve that, we leverage deep convolutional neural network(ConvNets). The model accepts a sequence of encoded characters as input. The encoding si done by prescribing an alphabet of size INLINEFORM0 for the input language, and then quantize each character using one-hot encoding. Then, the sequence of characters is transformed to a sequence of such INLINEFORM1 sized vectors with fixed length INLINEFORM2 . Any character exceeding length INLINEFORM3 is ignored, and any characters that are not in the alphabet are quantized as all-zero vectors. The alphabet used in our model consists of 36 characters, including 26 english letters and 10 digits. Below, we will introduce character-level temporal convolution neural network.
Temporal Convolution applies one-dimensional convolution over an input sequence. The one-dimensional convolution is an operation between a vector of weights INLINEFORM0 and a vector of inputs viewed as a sequence INLINEFORM1 . The vector INLINEFORM2 is the filter of the convolution. Concretely, we think of INLINEFORM3 as the input token and INLINEFORM4 as a single feature value associated with the INLINEFORM5 -th character in this token. The idea behind the one-dimensional convolution is to take the dot product of the vector INLINEFORM6 with each INLINEFORM7 -gram in the token INLINEFORM8 to obtain another sequence INLINEFORM9 : DISPLAYFORM0 
Usually, INLINEFORM0 is not a single value, but a INLINEFORM1 -dimensional vector so that INLINEFORM2 . There exist two types of 1d convolution operations. One is called Time Delay Neural Networks (TDNNs). The other one was introduced by BIBREF6 . In TDNN, weights INLINEFORM3 form a matrix. Each row of INLINEFORM4 is convolved with the corresponding row of INLINEFORM5 . In BIBREF6 architecture, a sequence of length INLINEFORM6 is represented as: DISPLAYFORM0 
where INLINEFORM0 is the concatenation operation. In general, let INLINEFORM1 refer to the concatenation of characters INLINEFORM2 . A convolution operation involves a filter INLINEFORM3 , which is applied to a window of INLINEFORM4 characters to produce the new feature. For example, a feature INLINEFORM5 is generated from a window of characters INLINEFORM6 by: DISPLAYFORM0 
Here INLINEFORM0 is a bias term and INLINEFORM1 is a non-linear function such as the thresholding function INLINEFORM2 . This filter is applied to each possible window of characters in the sequence INLINEFORM3 to produce a feature map: DISPLAYFORM0 
with INLINEFORM0 .
On top of convolutional neural network layers, we build another Highway Multilayer Perceptron (HMLP) layer to further enhance character-level word embeddings. Conventional MLP applies an affine transformation followed by a nonlinearity to obtain a new set of features: DISPLAYFORM0 
One layer of a highway network does the following: DISPLAYFORM0 
where INLINEFORM0 is a nonlinearity, INLINEFORM1 is called as the transform gate, and INLINEFORM2 is called as the carry gate. Similar to the memory cells in LSTM networks, highway layers allow adaptively carrying some dimensions of the input directly to the input for training deep networks.
Now that we have two kinds of word sequence representations. One kind of sequence representations are the composition of pre-trained word vectors. The other kind of sequence representation comprise word vectors that generating from character-level convolutional network. We can inject the two sequence representations into bidirectional LSTM to learn sentence representation. More specifically, forward directional LSTM accept pre-trained word embedding output and backward directional LSTM accept character CNN embedding output. The final sentence representation is the concatenation of the two direction.
Recurrent neural networks (RNNs) are capable of modeling sequences of varying lengths via the recursive application of a transition function on a hidden state. For example, at each time step INLINEFORM0 , an RNN takes the input vector INLINEFORM1 and the hidden state vector INLINEFORM2 , then applies affine transformation followed by an element-wise nonlinearity such as hyperbolic tangent function to produce the next hidden state vector INLINEFORM3 : DISPLAYFORM0 
A major issue of RNNs using these transition functions is that it is difficult to learn long-range dependencies during training step because the components of the gradient vector can grow or decay exponentially BIBREF14 .
The LSTM architecture BIBREF15 addresses the problem of learning long range dependencies by introducing a memory cell that is able to preserve state over long periods of time. Concretely, at each time step INLINEFORM0 , the LSTM unit can be defined as a collection of vectors in INLINEFORM1 : an input gate INLINEFORM2 , a forget gate INLINEFORM3 , an output gate INLINEFORM4 , a memory cell INLINEFORM5 and a hidden state INLINEFORM6 . We refer to INLINEFORM7 as the memory dimensionality of the LSTM. One step of an LSTM takes as input INLINEFORM8 , INLINEFORM9 , INLINEFORM10 and produces INLINEFORM11 , INLINEFORM12 via the following transition equations: DISPLAYFORM0 
where INLINEFORM0 and INLINEFORM1 are the element-wise sigmoid and hyperbolic tangent functions, INLINEFORM2 is the element-wise multiplication operator.
One shortcoming of conventional RNNs is that they are only able to make use of previous context. In text entailment, the decision is made after the whole sentence pair is digested. Therefore, exploring future context would be better for sequence meaning representation. Bidirectional RNNs architecture BIBREF13 proposed a solution of making prediction based on future words. At each time step INLINEFORM0 , the model maintains two hidden states, one for the left-to-right propagation INLINEFORM1 and the other for the right-to-left propagation INLINEFORM2 . The hidden state of the Bidirectional LSTM is the concatenation of the forward and backward hidden states. The following equations illustrate the main ideas: DISPLAYFORM0 
Deep RNNs can be created by stacking multiple RNN hidden layer on top of each other, with the output sequence of one layer forming the input sequence for the next. Assuming the same hidden layer function is used for all INLINEFORM0 layers in the stack, the hidden vectors INLINEFORM1 are iteratively computed from INLINEFORM2 to INLINEFORM3 and INLINEFORM4 to INLINEFORM5 : DISPLAYFORM0 
Multilayer bidirectional RNNs can be implemented by replacing each hidden vector INLINEFORM0 with the forward and backward vectors INLINEFORM1 and INLINEFORM2 , and ensuring that every hidden layer receives input from both the forward and backward layers at the level below. Furthermore, we can apply LSTM memory cell to hidden layers to construct multilayer bidirectional LSTM.
Finally, we can concatenate sequence hidden matrix INLINEFORM0 and reversed sequence hidden matrix INLINEFORM1 to form the sentence representation. We refer to INLINEFORM2 is the number of layers, INLINEFORM3 as the memory dimensionality of the LSTM. In the next section, we will use the two matrixs to generate matching feature planes via linear algebra operations.
Inspired by BIBREF10 , we apply element-wise merge to first sentence matrix INLINEFORM0 and second sentence matrix INLINEFORM1 . Similar to previous method, we can define two simple matching feature planes (FPs) with below equations: DISPLAYFORM0 
where INLINEFORM0 is the element-wise multiplication. The INLINEFORM1 measure can be interpreted as an element-wise comparison of the signs of the input representations. The INLINEFORM2 measure can be interpreted as the distance between the input representations.
In addition to the above measures, we also found the following feature plane can improve the performance: DISPLAYFORM0 
In INLINEFORM0 , the INLINEFORM1 means one-dimensional convolution. Join mean concatenate the two representation. The intuition behind INLINEFORM2 is let the one-dimensional convolution preserves the common information between sentence pairs.
Recall that the multi-layer bidirectional LSTM generates sentence representation matrix INLINEFORM0 by concatenating sentence hidden matrix INLINEFORM1 and reversed sentence hidden matrix INLINEFORM2 . Then we conduct element-wise merge to form feature plane INLINEFORM3 . Therefore, the final input into temporal convolution layer is a 3D tensor INLINEFORM4 , where INLINEFORM5 is the number of matching feature plane, INLINEFORM6 is the number of layers, INLINEFORM7 as the memory dimensionality of the LSTM. Note that the 3D tensor convolutional layer input INLINEFORM8 can be viewed as an image where each feature plane is a channel. In computer vision and image processing communities, the spatial 2D convolution is often used over an input image composed of several input planes. In experiment section, we will compare 2D convolution with 1D convolution. In order to facilitate temporal convolution, we need reshape INLINEFORM9 to 2D tensor.
The matching feature planes can be viewed as channels of images in image processing. In our scenario, these feature planes hold the matching information. We will use temporal convolutional neural network to learn hidden matching features. The mechanism of temporal CNN here is the same as character-level temporal CNN. However, the kernels are totally different.
It's quite important to design a good topology for CNN to learn hidden features from heterogeneous feature planes. After several experiments, we found two topological graphs can be deployed in the architecture. Figure FIGREF20 and Figure FIGREF20 show the two CNN graphs. In Topology i@, we stack temporal convolution with kernel width as 1 and tanh activation on top of each feature plane. After that, we deploy another temporal convolution and tanh activation operation with kernel width as 2. In Topology ii@, however, we first stack temporal convolution and tanh activation with kernel width as 2. Then we deploy another temporal convolution and tanh activation operation with kernel width as 1. Experiment results demonstrate that the Topology i@ is slightly better than the Topology ii@. This conclusion is reasonable. The feature planes are heterogeneous. After conducting convolution and tanh activation transformation, it makes sense to compare values across different feature planes.
We selected two related sentence relation modeling tasks: semantic relatedness task, which measures the degree of semantic relatedness of a sentence pair by assigning a relatedness score ranging from 1 (completely unrelated) to 5 ( very related); and textual entailment task, which determines whether the truth of a text entails the truth of another text called hypothesis. We use standard SICK (Sentences Involving Compositional Knowledge) dataset for evaluation. It consists of about 10,000 English sentence pairs annotated for relatedness in meaning and entailment.
We first initialize our word representations using publicly available 300-dimensional Glove word vectors . LSTM memory dimension is 100, the number of layers is 2. On the other hand, for CharCNN model we use threshold activation function on top of each temporal convolution and max pooling pairs . The CharCNN input frame size equals alphabet size, output frame size is 100. The maximum sentence length is 37. The kernel width of each temporal convolution is set to 3, the step is 1, the hidden units of HighwayMLP is 50. Training is done through stochastic gradient descent over shuffled mini-batches with the AdaGrad update rule BIBREF16 . The learning rate is set to 0.05. The mini-batch size is 25. The model parameters were regularized with a per-minibatch L2 regularization strength of INLINEFORM0 . Note that word embeddings were fixed during training.
The task of semantic relatedness prediction tries to measure the degree of semantic relatedness of a sentence pair by assigning a relatedness score ranging from 1 (completely unrelated) to 5 (very related). More formally, given a sentence pair, we wish to predict a real-valued similarity score in a range of INLINEFORM0 , where INLINEFORM1 is an integer. The sequence INLINEFORM2 is the ordinal scale of similarity, where higher scores indicate greater degrees of similarity. We can predict the similarity score INLINEFORM3 by predicting the probability that the learned hidden representation INLINEFORM4 belongs to the ordinal scale. This is done by projecting an input representation onto a set of hyperplanes, each of which corresponds to a class. The distance from the input to a hyperplane reflects the probability that the input will located in corresponding scale.
Mathematically, the similarity score INLINEFORM0 can be written as: DISPLAYFORM0 
where INLINEFORM0 and the weight matrix INLINEFORM1 and INLINEFORM2 are parameters.
In order to introduce the task objective function, we define a sparse target distribution INLINEFORM0 that satisfies INLINEFORM1 : DISPLAYFORM0 
where INLINEFORM0 . The objective function then can be defined as the regularized KL-divergence between INLINEFORM1 and INLINEFORM2 : DISPLAYFORM0 
where INLINEFORM0 is the number of training pairs and the superscript INLINEFORM1 indicates the INLINEFORM2 -th sentence pair BIBREF10 .
Referring to textual entailment recognition task, we want to maximize the likelihood of the correct class. This is equivalent to minimizing the negative log-likelihood (NLL). More specifically, the label INLINEFORM0 given the inputs INLINEFORM1 is predicted by a softmax classifier that takes the hidden state INLINEFORM2 at the node as input: DISPLAYFORM0 
After that, the objective function is the negative log-likelihood of the true class labels INLINEFORM0 : DISPLAYFORM0 
where INLINEFORM0 is the number of training pairs and the superscript INLINEFORM1 indicates the INLINEFORM2 th sentence pair.
Table TABREF31 and TABREF32 show the Pearson correlation and accuracy comparison results of semantic relatedness and text entailment tasks. We can see that combining CharCNN with multi-layer bidirectional LSTM yields better performance compared with other traditional machine learning methods such as SVM and MaxEnt approach BIBREF17 , BIBREF0 that served with many handcraft features. Note that our method doesn't need extra handcrafted feature extraction procedure. Also our method doesn't leverage external linguistic resources such as wordnet or parsing which get best results in BIBREF10 . More importantly, both task prediction results close to the state-of-the-art results. It proved that our approaches successfully simultaneously predict heterogeneous tasks. Note that for semantic relatedness task, the latest research BIBREF10 proposed a tree-structure based LSTM, the Pearson correlation score of their system can reach 0.863. Compared with their approach, our method didn't use dependency parsing and can be used to predict tasks contains multiple languages.
We hope to point out that we implemented the method in BIBREF10 , but the results are not as good as our method. Here we use the results reported in their paper. Based on our experiments, we believe the method in BIBREF10 is very sensitive to the initializations, thus it may not achieve the good performance in different settings. However, our method is pretty stable which may benefit from the joint tasks training.
In this experiment, we will compare tree LSTM with sequential LSTM. A limitation of the sequence LSTM architectures is that they only allow for strictly sequential information propagation. However, tree LSTMs allow richer network topologies where each LSTM unit is able to incorporate information from multiple child units. As in standard LSTM units, each Tree-LSTM unit (indexed by INLINEFORM0 ) contains input and output gates INLINEFORM1 and INLINEFORM2 , a memory cell INLINEFORM3 and hidden state INLINEFORM4 . The difference between the standard LSTM unit and tree LSTM units is that gating vectors and memory cell updates are dependent on the states of possibly many child units. Additionally, instead of a single forget gate, the tree LSTM unit contains one forget gate INLINEFORM5 for each child INLINEFORM6 . This allows the tree LSTM unit to selectively incorporate information from each child.
We use dependency tree child-sum tree LSTM proposed by BIBREF10 as our baseline. Given a tree, let INLINEFORM0 denote the set of children of node INLINEFORM1 . The child-sum tree LSTM transition equations are the following: DISPLAYFORM0 
Table TABREF35 show the comparisons between tree and sequential based methods. We can see that, if we don't deploy CNN, simple Tree LSTM yields better result than traditional LSTM, but worse than Bidirectional LSTM. This is reasonable due to the fact that Bidirectional LSTM can enhance sentence representation by concatenating forward and backward representations. We found that adding CNN layer will decrease the accuracy in this scenario. Because when feeding into CNN, we have to reshape the feature planes otherwise convolution will not work. For example, we set convolution kernel width as 2, the input 2D tensor will have the shape lager than 2. To boost performance with CNN, we need more matching features. We found Multi-layer Bidirectional LSTM can incorporate more features and achieve best performance compared with single-layer Bidirectional LSTM.
Existing neural sentence models mainly fall into two groups: convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In regular 1D CNNs BIBREF6 , BIBREF8 , BIBREF19 , a fixed-size window slides over time (successive words in sequence) to extract local features of a sentence; then they pool these features to a vector, usually taking the maximum value in each dimension, for supervised learning. The convolutional unit, when combined with max-pooling, can act as the compositional operator with local selection mechanism as in the recursive autoencoder BIBREF3 . However, semantically related words that are not in one filter can't be captured effectively by this shallow architecture. BIBREF20 built deep convolutional models so that local features can mix at high-level layers. However, deep convolutional models may result in worse performance BIBREF19 .
On the other hand, RNN can take advantage of the parsing or dependency tree of sentence structure information BIBREF3 , BIBREF21 . BIBREF4 used dependency-tree recursive neural network to map text descriptions to quiz answers. Each node in the tree is represented as a vector; information is propagated recursively along the tree by some elaborate semantic composition. One major drawback of RNNs is the long propagation path of information near leaf nodes. As gradient may vanish when propagated through a deep path, such long dependency buries illuminating information under a complicated neural architecture, leading to the difficulty of training. To address this issue, BIBREF10 proposed a Tree-Structured Long Short-Term Memory Networks. This motivates us to investigate multi-layer bidirectional LSTM that directly models sentence meanings without parsing for RTE task.
In this paper, we propose a new deep neural network architecture that jointly leverage pre-trained word embedding and character embedding to learn sentence meanings. Our new approach first generates two kinds of word sequence representations as inputs into bidirectional LSTM to learn sentence representation. After that, we construct matching features followed by another temporal CNN to learn high-level hidden matching feature representations. Our model shows that combining pre-trained word embeddings with auxiliary character-level embedding can improve the sentence representation. The enhanced sentence representation generated by multi-layer bidirectional LSTM will encapsulate the character and word levels informations. Furthermore, it may enhance matching features that generated by computing similarity measures on sentence pairs. Experimental results on benchmark datasets demonstrate that our new framework achieved the state-of-the-art performance compared with other deep neural networks based approaches.

Figure and table captions:
Figure 1: Neural Network Architecture for Deep Matching Feature Learning. M-BLSTM is Multilayer Bidirectional LSTM. Orange color represents sequence representations that concatenating pre-trained word vectors. Purple color represents sequence representation concatenating word vectors that generating from character-level convolutional network and HMLP.
Figure 2: CNN Topology I Figure 3: CNN Topology II
Table 1: Semantic Relatedness Task Comparison.
Table 2: Textual Entailment Task Comparison.
Table 3: Results of Tree LSTM vs Sequence LSTM on auxiliary char embedding.

Questions about this paper:
By how much do they outperform existing methods?

Answers:
Best proposed result had 0.851 and  0.842 compared to best previous result of 0.828 and 0.846 on person correlation and accuracy respectively.