Question: According to authors' findings, when trained on small size datasets, what can be done to improve the sentence representation?
Evidence: Traditional approaches BIBREF0 , BIBREF1 , BIBREF2 for sentence relation modeling tasks such as paraphrase identification, question answering, recognized textual entailment and semantic textual similarity prediction usually build the supervised model using a variety of hand crafted features. Hundreds of features generated at different linguistic levels are exploited to boost classification. With the success of deep learning, there has been much interest in applying deep neural network based techniques to further improve the prediction performances BIBREF3 , BIBREF4 , BIBREF5 .
A key component of deep neural network is word embedding which serve as an lookup table to get word representations. From low level NLP tasks such as language modeling, POS tagging, name entity recognition, and semantic role labeling BIBREF6 , BIBREF7 , to high level tasks such as machine translation, information retrieval and semantic analysis BIBREF8 , BIBREF9 , BIBREF10 . Deep word representation learning has demonstrated its importance for these tasks. All the tasks get performance improvement via further learning either word level representations or sentence level representations. On the other hand, some researchers have found character-level convolutional networks BIBREF11 , BIBREF12 are useful in extracting information from raw signals for the task such as language modeling or text classification.
In this work, we focus on deep neural network based sentence relation modeling tasks. We explore treating each sentence as a kind of raw signal at character level, and applying temporal (one-dimensional) Convolution Neural Network (CNN) BIBREF6 , Highway Multilayer Perceptron (HMLP) and multi-layer bidirectional LSTM (Long Short Term Memory) BIBREF13 to learn sentence representations. We propose a new deep neural network architecture that jointly leverage pre-trained word embedding and character embedding to represent the meaning sentences. More specifically, our new approach first generates two kinds of word sequence representations. One kind of sequence representations are the composition of pre-trained word vectors. The other kind of sequence representation comprise word vectors that generating from character-level convolutional network. We then inject the two sequence representations into bidirectional LSTM, which means forward directional LSTM accept pre-trained word embedding output and backward directional LSTM accept auxiliary character CNN embedding output. The final sentence representation is the concatenation of the two direction. After that, we construct matching features followed by another temporal CNN to learn high-level hidden matching feature representations. Figure FIGREF1 shows the neural network architecture for general sentence relation modeling.
Our model shows that when trained on small size datasets, combining pre-trained word embeddings with auxiliary character-level embedding can improve the sentence representation. Word embeddings can help capturing general word semantic meanings, whereas char-level embedding can help modeling task specific word meanings. Note that auxiliary character-level embedding based sentence representation do not require the knowledge of words or even syntactic structure of a language. The enhanced sentence representation generated by multi-layer bidirectional LSTM will encapsulate the character and word levels informations. Furthermore, it may enhance matching features that generated by computing similarity measures on sentence pairs. Quantitative evaluations on standard dataset demonstrate the effectiveness and advantages of our method.
Our model shows that when trained on small size datasets, combining pre-trained word embeddings with auxiliary character-level embedding can improve the sentence representation.
Answer: Combining pre-trained word embeddings with auxiliary character CNN embedding output.