Title: Learning Graphical State Transitions
Authors: Daniel D. Johnson
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Supervised Learning
Structured prediction

Abstract: Graph-structured data is important in modeling relationships between multiple entities, and can be used to represent states of the world as well as many data structures. Li et al. (2016) describe a model known as a Gated Graph Sequence Neural Network (GGS-NN) that produces sequences from graph-structured input. In this work I introduce the Gated Graph Transformer Neural Network (GGT-NN), an extension of GGS-NNs that uses graph-structured data as an intermediate representation. The model can learn to construct and modify graphs in sophisticated ways based on textual input, and also to use the graphs to produce a variety of outputs. For example, the model successfully learns to solve almost all of the bAbI tasks (Weston et al., 2016), and also discovers the rules governing graphical formulations of a simple cellular automaton and a family of Turing machines.

TL;DR: I introduce a set of differentiable graph transformations, and use them to build a model with a graphical internal state that can extract structured data from text and use it to answer queries.

Review discussion:
Dear authors,
First, I would like to say that the topic of the article is clearly interesting, but the paper is very hard to follow since the writing is a little bit rough. The paper could clearly benefit to some additional figures for example when presenting the model, and to separate the presentation of the different allowed operations with the “technical” computations.
My first question is about the positioning of the paper and the links between your approach and other approaches like Relational Neural networks (or lifed relational neural networks)  for example or even differentiable neural computers. Particularly, having a discussion about the expressiveness of the different approaches would be very nice (if feasuible)
Could you please discuss a little bit more the difference/advantages/drawbacks of your approach with existing works ?
My second question concerns the way the structure is defined in the experimental part (Algorithm 1). It is not clear to me if other structures could be proposed. Could you explain a little bit more this point ? And particularly the advantage of the chosen structure w.r.t other possibilities and for the particular tasks studied here ?
Thank you for your valuable input. The next revision of the paper will include an additional section comparing this model to existing works, as well as a summary and clearer explanation of the transformation operations.
Overall, the primary strengths of this model relative to other models are:
1) the input to the network does not need to be structured. This is in contrast to GGS-NNs or other existing graph-based approaches, which require input in graphical format, or to lifted relational networks, which require input in terms of predicate-based facts and hand-specified rules.
2) the internal state of the network is an explicit graph, and can potentially be extracted, analyzed, or used for downstream applications. This is in contrast to the internal state of other architectures with differentiable internal state, such as memory networks or differentiable neural computers: although they can store graphical relationships implicitly, they do not have an explicit graphical structure. (Note that differentiable neural computers do have connections based on the temporal access patterns of the data, but these do not necessarily correspond to edges between nodes in the underlying structure of the data.)
The structure described in Algorithm 1 utilizes all of the graph transformations in a particular order that is well-suited to a question-answering task. This is also one of the most powerful structures that I considered.
However, subsets of the graph transformations could be used to construct a model for a different type of task.
For instance, the node addition transformation can be removed if the set of nodes is fixed, and the query processing steps can be removed if the desired output is a graph. Appendix B also shows how the structure can be modified to answer the query based on a sequence of graphs, rather than a single graph.
Dear Author,
Algorithm 1 applies to a single dialog with K sentences: the graph represents a memory of the dialog that is later used to answer a query.  As the graph starts from 0 at step 1, shall I assume that all the trainable parameters of the graph (U and W)  are only learnt from one sample dialog, basically only learnt to perform the inference? If this is true, what are the trainable parameters that encode knowledge from other dialogs? The parameters used to extract representation i(k)? If I am wrong, how are the U and W parameters passed from previous sample dialogs?
Thank you for your comments.
The parameters of the model are not learned from only one dialog.
Instead, they are learned based on a large corpus of data. U and W are not the trainable parameters of the graph; those are the trainable parameters associated with a single GRU cell, described by Cho et al. (2014). In Algorithm 1, each graph transformation (indicated with a capital T) has its own set of trainable parameters, based on its internal structure (described in detail in Appendix B).
Additionally, the GRU layer used to produce i(k) also has trainable parameters.
It is true that each sample dialog corresponds to an independent graph structure, which starts as the empty graph for each dialog.
However, the parameters used by each graph are shared between all dialogs, and determine how the graph is updated and how information flows between nodes. The graph structure does not represent the network itself, so there are no parameters actually attached to the graph.
Instead, the parameters of the model are used to modify the graph.
Thus when running on different dialogs, the same parameters are used, but end up producing a different graphical structure.
It can be useful to think of the graph not as an actual part of the model, but instead as a type of memory. The model itself uses its parameters to read and modify the data stored in the graph and change the graph's structure. This is similar to how a Differentiable Neural Computer, for instance, can use its read-write heads to read and modify the data stored in memory.
I am not confident that I understand your question about learning "to perform the inference". In all cases, the parameters are learned to maximize the probability that the graph is built correctly and that the correct answer is chosen for the query. At inference time, the model is then used to build a graph and produce an answer to a query.
However, even so, the parameters are trained across all dialogs.
If I have misunderstood your questions or if this answer does not fully address them, please let me know.
This helps a lot!
So I assume that Algorithm 1 iteratively constructs  GGT-NNs but the parameters U and W are external shared variables. I would be curious what software you use. Such a network would take a lot of time to compile w/Theano, and you have to redo it every time.
Some comments in the paper  about how your network representation of memory compares to "memory networks" would be nice.
Inference is data inference given a model and some input data. In some type of architecture, like CRF, inference can be as costly as learning.
I have revised the paper since my original submission. The current revision includes a comparison of GGT-NNs to other network architectures, including memory networks, in section 6.
I use Theano to compile the network, but I only have to compile the network once, so it is quite fast. The graph structure is encoded as a set of matrices (depicted in Figure 1) so the same Theano function can be used for multiple sample dialogs.
Based on the reviewer comments, I have uploaded a new revision of the paper, with the following changes:
- Added figures depicting the differentiable graph format and each of the graph transformations
- Added a section comparing the GGT-NN model to existing works
- Included baselines for the bAbI tasks, with discussion of the relative performance of the GGT-NN model
- Clarified the feasibility of alternative network configurations (relative to Algorithm 1)
- Moved implementation details of the graph transformations into the appendix
- Clearly separated and simplified the background information section
- Clarified the training procedure for the GGT-NN model
Thank you very much for your feedback.
The paper proposes an extension of the Gated Graph Sequence Neural Network by including in this model the ability to produce complex graph transformations. The underlying idea is to propose a method that will be able build/modify a graph-structure as an internal representation for solving a problem, and particularly for solving question-answering problems in this paper. The author proposes 5 different possible differentiable transformations that will be learned on a training set, typically in a supervised fashion where the state of the graph is given at each timestep. A particular occurence of the model is presented that takes a sequence as an input a iteratively update an internal graph state to a final prediction, and which can be applied for solving QA tasks (e.g BaBi) with interesting results.
The approach  in this paper is really interesting since the proposed model is able to maintain a representation of its current state as a complex graph, but still keeping the property of being differentiable and thus easily learnable through gradient-descent techniques. It can be seen as a succesfull attempt to mix continuous and symbolic representations. It moreover seems more general that the recent attempts made to add some 'symbolic' stuffs in differentiable models (Memory networks, NTM, etc. ) since the shape of the state is not fixed here and can evolve.
My main concerns is about the way the model is trained i. e by providing the state of the graph at each timestep which can be done for particular tasks (e. g Babi) only, and cannot be the solution for more complex problems. My other concern is about the whole content of the paper that would perhaps best fit a journal format and not a conference format, making the article still difficult to read due to its density.
Thank you very much for your review and comments.
Future work on training the GGT-NN model may enable it to solve complex problems where full graph information is not available. Potential avenues for exploration include providing graph states for a subset of the training data (suggested below in the comment by Daniel Tarlow) and introducing additional regularization to allow the network to be trained end-to-end without any graph information.
This paper proposes learning on the fly to represent a dialog as a graph (which acts as the memory), and is first demonstrated on the bAbI tasks. Graph learning is part of the inference process, though there is long term representation learning to learn graph transformation parameters and the encoding of sentences as input to the graph. This seems to be the first implementation of a differentiable memory as graph: it is much more complex than previous approaches like memory networks without significant gain in performance in bAbI tasks, but it is still very preliminary work, and the representation of memory as a graph seems much more powerful than a stack. Clarity is a major issue, but from an initial version that was constructive and better read by a computer than a human, the author proposed a hugely improved later version. This original, technically accurate (within what I understood) and thought provoking paper is worth publishing.
The preliminary results do not tell us yet if the highly complex graph-based differentiable memory has more learning or generalization capacity than other approaches. The performance on the bAbI task is comparable to the best memory networks, but still worse than more traditional rule induction (see http://www.public.asu.edu/~cbaral/papers/aaai2016-sub.pdf). This is still clearly promising.
The sequence of transformation in algorithm 1 looks sensible, though the authors do not discuss any other operation ordering.
In particular, it is not clear to me that you need the node state update step T_h if you have the direct reference update step T_h,direct.
It is striking that the only trick that is essential for proper performance is the ‘direct reference’ , which actually has nothing to do with the graph building process, but is rather an attention mechanism for the graph input: attention is focused on words that are relevant to the node type rather than the whole sentence.
So the question “how useful are all these graph operations” remain. A much simpler version of a similar trick may have been proposed in the context of memory networks, also for ICLR'17 (see match type in "LEARNING END-TO-END GOAL-ORIENTED DIALOG" by Bordes et al)
The authors also mention the time and size needed to train the model: is the issue arising for learning, inference or both? A description of the actual implementation would help  (no pointer to open source code is provide). The author mentions Theano in one of my questions: how are the transformations compiled in advance as units? How is the gradient back-propagated through the graph is this one is only described at runtime?
Typo: in the appendices B.2 and B.2.1, the right side of the equation that applies the update gate has h’_nu while it should be h_nu.
In the references, the author could mention the pioneering work  of Lee Giles on representing graphs with  RNNs.
Revision: I have improved my rating for the following reasons:
- Pointers to an highly readable and well structured Theano source is provided.
- The delta improvement of the paper has been impressive over the review process, and I am confident this will be an impactful paper.
- Much simpler alternatives approaches such as Memory Networks seem to be plateauing for problems such as dialog modeling, we need alternatives.
- The architecture is this work is still too complex, but this is often as we start with DNNs, and then find simplifications that actually improve performance
Thank you very much for your review and comments.
Although the GGT-NN model was evaluated on the bAbI tasks, which involve representing a dialog, it can also be used for non-dialog tasks. The cellular automaton and Turing machine tasks described in section 5.2 are two examples of tasks where the output is a graph instead of a textual answer. I also briefly describe alternate operation configurations in Section 4.2 and Appendix D.
The direct reference update step (T_h,direct) is based on the presence of input words that correspond directly to specific node types. If none of those words appear in a particular input sentence, then the direct reference update will contain no information. In all of the bAbI tasks, some entity is directly described in each sentence, so it is true that in this case the node state update step (T_h) may not be necessary.
However, the node state update step is necessary whenever an input sentence does not reference any node type. As one example, this occurs with the "run" and "simulate" commands for the cellular automaton and Turing machine tasks, described in Appendix C.
The importance of direct reference on GGT-NN performance is certainly very interesting. For some tasks, a simpler method such as "match type" (as described in "LEARNING END-TO-END GOAL-ORIENTED DIALOG" by Bordes et al., submitted to ICLR'17) combined with memory networks may be sufficient.
However, other tasks seem to benefit greatly from a graph representation.
For instance, bAbI task 19 requires finding a path between a source and a destination, and was easily solved using the GGT-NN model.
Although direct reference (and possibly match type) can identify the source and the destination, the intermediate nodes are not mentioned in the query, so it is the graph operations that enable the path between them to be found. The graph operations are also a natural fit for tasks that require graph-structured output, which seem difficult to formulate using simpler models.
The time and size issue described in the paper arises both during learning and in inference, and is caused primarily by the overhead of storing all possible edge connections. During inference the usage could be potentially reduced with sparse matrix operations, but this has not yet been explored.
The source code is available at https://github.com/hexahedria/gated-graph-transformer-network (and a link has been added in the most recent revision of the paper). In essence, all of the transformations are implemented as functions that take in Theano tensors and return new Theano tensors. These transformations are combined into a Theano computation graph that implements Algorithm 1 and also computes the loss between the final output and the target. The gradient is then back-propagated using the built-in Theano automatic differentiation functions.
Thank you for pointing out the typo in appendices B.2 and B.2.1; it has been fixed in the newest revision of the paper.
The work of Lee Giles is indeed relevant and interesting. I have added a reference to and brief discussion of "Learning and extracting finite state automata with second-order recurrent neural networks" by Giles et al. (1992) since this seemed most relevant to the GGT-NN model.
The main contribution of this paper seems to be an introduction of a set of differential graph transformations which will allow you to learn graph->graph classification tasks using gradient descent. This maps naturally to a task of learning a cellular automaton represented as sequence of graphs. In that task, the graph of nodes grows at each iteration, with nodes pointing to neighbors and special nodes 0/1 representing the values. Proposed architecture allows one to learn this sequence of graphs, although in the experiment, this task (Rule 30) was far from solved.
This idea is combined with ideas from previous papers (GGS-NN) to allow the model to produce textual output rather than graph output, and use graphs as intermediate representation, which allows it to beat state of the art on BaBi tasks.
Thank you very much for your review and comments.
For the Rule 30 task, the model was trained on examples that only included seven steps of the automaton, and was then evaluated on three variants of the task: one with seven steps, one with 20 steps, and one with 30 steps.
Although it is true that the model was unable to generalize perfectly to inputs longer than those seen in training data, the model did achieve 100% accuracy on inputs of the same length as it was trained on (7 steps). It is likely that the model could be trained to correctly produce longer sequences if it was trained with longer examples.
The paper has been updated with the following changes:
- Fixed typo in the equations given in B.2 and B.2.1.
- Added a reference to work done by Giles et al. in Section 6.
- Clarified the operation of direct reference in Sections 3 and 4.
- Added a link to the source code for the model (in Appendix B).
- Minor wording changes in the Abstract, Introduction, and Sections 3 and 4.

Meta-review:
role: PC
data: comment: The idea of building a graph-based differentiable memory is very good. The proposed approach is quite complex, but it is likely to lead to future developments and extensions. The paper has been much improved since the original submission. The results could be strengthened, with more comparisons to existing results on bAbI and baselines on the experiments here. Exploring how it performs with less supervision, and different types of supervision, from entirely labeled graphs versus just node labels, would be valuable.

Decision: Accept (Oral)