Abstract: Neural network based approaches for sentence relation modeling automatically generate hidden matching features from raw sentence pairs. However, the quality of matching feature representation may not be satisfied due to complex semantic relations such as entailment or contradiction. To address this challenge, we propose a new deep neural network architecture that jointly leverage pre-trained word embedding and auxiliary character embedding to learn sentence meanings. The two kinds of word sequence representations as inputs into multi-layer bidirectional LSTM to learn enhanced sentence representation. After that, we construct matching features followed by another temporal CNN to learn high-level hidden matching feature representations. Experimental results demonstrate that our approach consistently outperforms the existing methods on standard evaluation datasets.

Introduction: Traditional approaches BIBREF0 , BIBREF1 , BIBREF2 for sentence relation modeling tasks such as paraphrase identification, question answering, recognized textual entailment and semantic textual similarity prediction usually build the supervised model using a variety of hand crafted features. Hundreds of features generated at different linguistic levels are exploited to boost classification. With the success of deep learning, there has been much interest in applying deep neural network based techniques to further improve the prediction performances BIBREF3 , BIBREF4 , BIBREF5 . A key component of deep neural network is word embedding which serve as an lookup table to get word representations. From low level NLP tasks such as language modeling, POS tagging, name entity recognition, and semantic role labeling BIBREF6 , BIBREF7 , to high level tasks such as machine translation, information retrieval and semantic analysis BIBREF8 , BIBREF9 , BIBREF10 . Deep word representation learning has demonstrated its importance for these tasks. All the tasks get performance improvement via further learning either word level representations or sentence level representations. On the other hand, some researchers have found character-level convolutional networks BIBREF11 , BIBREF12 are useful in extracting information from raw signals for the task such as language modeling or text classification. In this work, we focus on deep neural network based sentence relation modeling tasks. We explore treating each sentence as a kind of raw signal at character level, and applying temporal (one-dimensional) Convolution Neural Network (CNN) BIBREF6 , Highway Multilayer Perceptron (HMLP) and multi-layer bidirectional LSTM (Long Short Term Memory) BIBREF13 to learn sentence representations. We propose a new deep neural network architecture that jointly leverage pre-trained word embedding and character embedding to represent the meaning sentences. More specifically, our new approach first generates two kinds of word sequence representations. One kind of sequence representations are the composition of pre-trained word vectors. The other kind of sequence representation comprise word vectors that generating from character-level convolutional network. We then inject the two sequence representations into bidirectional LSTM, which means forward directional LSTM accept pre-trained word embedding output and backward directional LSTM accept auxiliary character CNN embedding output. The final sentence representation is the concatenation of the two direction. After that, we construct matching features followed by another temporal CNN to learn high-level hidden matching feature representations. Figure FIGREF1 shows the neural network architecture for general sentence relation modeling. Our model shows that when trained on small size datasets, combining pre-trained word embeddings with auxiliary character-level embedding can improve the sentence representation. Word embeddings can help capturing general word semantic meanings, whereas char-level embedding can help modeling task specific word meanings. Note that auxiliary character-level embedding based sentence representation do not require the knowledge of words or even syntactic structure of a language. The enhanced sentence representation generated by multi-layer bidirectional LSTM will encapsulate the character and word levels informations. Furthermore, it may enhance matching features that generated by computing similarity measures on sentence pairs. Quantitative evaluations on standard dataset demonstrate the effectiveness and advantages of our method.

Question: By how much do they outperform existing methods?
Answer: Best proposed result had 0.851 and  0.842 compared to best previous result of 0.828 and 0.846 on person correlation and accuracy respectively.