Trends in Semantic Parsing — Part 1
By: Desh Raj
Tags: Machine Learning, Lstm, Deep Learning, Semantic Parsing, Naturallanguageprocessing

Learning sentence embeddings using deep neural models

Vector semantics have been used extensively in all NLP tasks, especially after word embeddings (Word2Vec, GloVe) were found to represent the synonymy-antonymy relations well in real space.

Similar to word embeddings, we can try to obtain dense vectors to represent a sentence, and then find some way to obtain the formal representation from it. Ivan Titov (University of Edinburgh) has recently proposed a couple of models which use LSTMs [1] and Graph CNNs [2] for dependency-based SRL task.

I will first explain the task. We work on datasets where the predicates are marked in the sentence, and the objective is to identify and label the arguments corresponding to each predicate. For instance, given the sentence “Mary eats an apple,” and the predicate marked as EATS, we need to label the words ‘Mary,’ ‘an,’ and ‘apple’ as agent, NULL, and theme, respectively. Also, since a single sentence may contain multiple predicates, the same word may get different labels for each predicate. Essentially, if we repeat the process once for each predicate, out task effectively reduces to a sequence labeling problem.

LSTM-based approach [1]: LSTMs (which are a type of RNNs that can preserve memory) have been used to model sequences since they were first introduced. In the first model, the sequence labeling is performed as follows.

Vectors are obtained from each word by concatenating pre-trained embeddings (Word2Vec), random embeddings, and randomly initialized POS embeddings.

The word vector also contains a 1-bit flag to mark whether it is the predicate in that particular training instance. This is done to ensure that the network treats each predicate differently.

These are fed into a bi-LSTM layer to obtain the word’s context in the sentence.

Finally, to label any word, we take the dot product of its hidden state with the predicate’s hidden state and obtain a softmax classifier over it as follows.

Further, we can have the weight matrix parametrized on the role label r as:

where the vectors in the dot product correspond to randomly initialized embeddings for the predicate lemma and the role, respectively.

GCN-based approach [2]: In a second model, Graph Convolutional Networks (GCNs) have been used to represent the dependency tree for the sentence. In a very crude sense, a GCN input layer encodes the sentence into an mXn matrix based on its dependency tree, such that each of the n nodes of the tree is represented as an m-dimensional vector. Once such a matrix has been obtained, we can perform convolutions on it.

It is then evident that a one-layer GCN can capture information only about its immediate neighbor. By stacking GCN layers, one can incorporate higher degree neighborhoods. (Please look at Fig. 3 of the paper [2] for the model diagram.)

GCNs and LSTMs are complementary. Why? LSTMs capture long-term dependencies well but are not able to represent syntax effectively. On the other hand, GCNs are built directly on top of a syntactic-dependency tree so they capture syntax well, but due to the limitation of fixed-size convolutions, the range of dependency is limited. Therefore, using a GCN layer on top of the hidden states obtained from a bi-LSTM layer would theoretically capture the best of both worlds. This hypothesis has also been corroborated through experimental results.

Encoder-decoder model [3]: In this paper, the task is broadened into formal representation rather than SRL. If we consider the formal representation as a different language, this is similar to a machine translation problem, since both the natural as well as formal representations mean the same. As such, it might be interesting to apply models used for MT to semantic parsing. [3] does exactly this.

An encoder converts the input sequence to a vector representation and a decoder obtains the target sequence from this vector.