NLP Lecture 12 @ CMU — A Watch👓 & Read Treat🍨
By: Jeevesh Juneja
Tags: Machine Learning, NLP, Parser, Lstm, Trees

Generating Incremental Trees — Part 1 : by Graham Neubig

In this lecture, we learn how to adapt the structure of outputs of a neural net to predict some tree structure over an input sequence.

What kind of tree structure over the sequence ?

First, note that the words in a sentence, are related to other words in the sentence. Two words can be related , if one is the subject of the other, or one is the object of another, or in any other way. These relations usually end up making trees over the input sentence.

2 types of tree structure on the input sequence

But why do this ? What use is the structure over a sentence? Does it contain any more information than the original sentence?

Yes, it does. For example, the structure on the above sentence helps one determine whether, ‘I’ saw a girl who was carrying a telescope, or whether ‘I’ used a telescope to see a girl.

But it may not necessarily contain, more information, always. Sometimes, the structure is completely determined by the input sentence. But, still we might need it. In particular, once you have some pre-defined relations between parts of a sentence, and have a structure(with respect to these pre-defined relations) over the sentence, you can easily extract information from the sentence.

[[Think : Why not use BERT ? (Hint : BERT doesn’t do good, when you try to keep coherence, for long discourses)]]

TGrep is a tool you can use, to search through trees, for particular type of relations, or combinations thereof.

Shift Reduce Dependency Parsing

We figure out a clever way of generating trees incrementally, using the same kind of actions at each time step.

Arc-Standard Shift Reduce Parsing

Note how in the above algorithm, we have the same choices at each time step. And how even though that is the case, we can still generate any possible tree over a sentence.

Now, the question is what should we base our choice on ?

We should certainly send in the words in the stack and those in queue as input to our neural net, but should also send the tree structure that we built till now.. but how to do that? The input to a NN is generally of fixed size, & even if it is recurrent, how to send in the relations that we have made till now ?

Chen and Manning in 2014, chose to send in these features :

Top 3 words in buffer are the ones that are closest to being processed next. The children and grandchildren of words on stack, don’t reside on the stack. Words are removed from the stack, as soon as they get a parent.

But doing this kind of feature extraction doesn’t suit an ML engineer, when you have NNs that can predict these.

Just use RNNs to extract features.

Notice how the above is just a simple RNN, taking words one be one, just drawn differently. The parameters W can be different based on whether input word is Noun etc. or may even depend on the type of phrase the words captured till a point form.

We can also use LSTMs above, in which case, the forget gate values will correspond to ‘ignoring’ children. Like in above sentence ‘this’ doesn’t matter.. it could have been “I hate that movie”.

But, still how to use this in parsing?

We can use this for all words in the stack and for all the words in the buffer, surely. But how to do the above for the tree we have made till a point? Notice how the tree was made by following a sequence of steps; so we can just send in that sequence, as the representation of the complete tree formed till now!