NLP Lecture 13 by Graham Neubig
By: Jeevesh Juneja
Tags: Machine Learning, Generative Model, NLP, Parsing, Trees

Parsing with Dynamic Programming — by Graham Neubig

Previous Post : Lecture 12

Prerequisites : Context Free Grammars, Chomsky Normal Form, CKY Algorithm. You can read about them from here.

In the following, “grammar” refers to CFG. CKY algorithm is also referred to as CYK algorithm, sometimes.

In this lecture, we will explore further, the ways of generating trees over(or “parsing”) our input sentence. In particular, we would like to make use of models, which instead of making tree one action at a time(and hence making locally optimal choices), make the tree by choosing multiple actions/whole tree at a time(and hence making globally optimal choices). [ c.f. Lecture 11 ].

As a first step, we will view our problem of generating trees from the view point of the grammar of the language under consideration. We know that a grammar consists of production rules, and any string in the language, is obtained as a result of repeated application of those rules. Each sequence of application of those rules, corresponds to a derivation tree, that encapsulates how the string was formed from the grammar.

Our aim, now, is to find out the derivation tree/sequence of production rules that is most likely to produce the given string. This would correspond to a global solution.

The solution space, of all trees, has some structure, and we could use this, to efficiently traverse the solution space, akin to Lecture 11. The CKY algorithm does exactly that. It provides a way to list all the trees that could have generated the string, and iterate over them. But, it can only be applied grammars converted to CNF; so we must convert our grammar to CNF, first.

As seen in Lecture 11 , we would need two things :

A way to calculate score of a tree, as a function of the scores of various production rules used to generate it. [Similar to calculating score of a whole sequence of POS tags based on the score of POS tag(\psi) at each time-step] A way to sum/max over the scores of all trees, using DP. [Similar to summing/maxing over all possible POS sequences, using DP]

Think : What would the transition and emission probabilities of Lecture 11 transform into ?

They would just be replaced by a single probability, corresponding to the application of a particular production rule, at a time step. The Neural Network is expected to predict one logit for each valid production rule in the grammar. Where, a “valid” production rules are the ones whose input symbol, is the one whose sub-tree is being currently generated.

The score of the tree can be calculated recursively by calculating scores corresponding to each sub-tree, and summing them with the score of the production rule being applied at current step. This solves (1.) above.

To solve (2.), we use the CKY algorithm, building sequences of production rules for smaller to bigger units of the string, sequentially. Define a “split”(|) of a string as a partition of the string into exactly two parts.

For example, “My dear wife,| come, sit by my side” is a split of a whole string. Each possible partition corresponds to a split of the root node, which denotes the whole sentence.

We use the relation that all the trees generated over a string can be enumerated by listing “all combinations of all the trees that can be generated on each of the two parts in a partition”, for each possible partition of the string.

The same recursion is valid for summing and max operation.

Exercise 1: Try to come up with a variant of CKY algorithm, if the grammar were not in CNF form. Would the recursion still hold ? [[Hint : Note, how each partition must correspond to application of one valid production rule, in the recursion above.]]

Exercise 2: What would be easier for Neural Net to learn from: a grammar in CNF form, or a grammar in natural form ? Why ?

Extension to translation

The above approach can be used for translating sentences. We would need to make a new grammar, based on production rules for translations, and the rules of target language.

For example, in case of translating from French to English, production rules will be of form :

Translations of individual words into source language (dog -> chien). General Translation rules ( X ate Y -> X a mangé Y ) Grammatical rules of target language( VP -> VBD, NP ; NP -> dog)

The aim now is to find the most likely tree that could generate the source sentence, given this grammar. Note that, since there are no production rules that will allow you to generate target language words from the start token(S), all source sentences must be generated via translating from target language. In fact, the first rules to be considered in the CKY algorithm, will correspond to rules of type 1. above.

Exercise 3 : Can you think of any limitations of using trees and a grammar as above for translation ? [[ Hint : Do natural languages, actually have a “context free” grammar? ]]