Title: Tree-structured decoding with doubly-recurrent neural networks
Authors: David Alvarez-Melis
Tommi S. Jaakkola
Venue: ICLR 2017
Keywords: Natural language processing
Supervised Learning
Structured prediction

Abstract: We propose a neural network architecture for generating tree-structured objects from encoded representations. The core of the method is a doubly-recurrent neural network that models separately the width and depth recurrences across the tree, and combines them inside each cell to generate an output. The topology of the tree is explicitly modeled, allowing the network to predict both content and topology of the tree when decoding. That is, given only an encoded vector representation, the network is able to simultaneously generate a tree from it and predict labels for the nodes. We test this architecture in an encoder-decoder framework, where we train a network to encode a sentence as a vector, and then generate a tree structure from it. The experimental results show the effectiveness of this architecture at recovering latent tree structure in sequences and at mapping sentences to simple functional programs.

TL;DR: A new architecture for generating tree-structured objects from encoded representations, which models separately the width and depth recurrences across the tree and predicts both content and topology.

Review discussion:
There are two reasons of not using EOS token in a tree decoder (P-4), while I am less convinced why the second is crucial. If it is crucial, do you think it will also help in a sequential decoder, as the argument on P-5 also holds for a sequential decoder -- *rationale behind this is that the label of a node will likely be influenced not only by its context, but also by the type of node*.
On P-8, it is claimed that LSTM-DRNN achieves a Macro F1 score of 51% over argument nodes. Can more results be shown for comparison between different models over argument nodes?
In Eqn-8, on the left-hand side, is the \hat{x}_i actually \hat{x} as i is the subscript of vertices? I assume the loss function is the summation of loss value at all the nodes.
1. Our explanation for why the second reason is important is the following. In our model the stop decisions are parametrically separate from tokens. This offers a more direct way of controlling the topology without interference from changing token parameters, and allows to make the label decision be made after (and dependent on) the node type.
While we have not examined the effect in great detail, the increase in performance we observed over token-based topology prediction (even for trees where adding padding increases size only modestly) suggests it is indeed an important factor.
As you point out, the argument above holds for sequences too, so we expect that the effect may be similar.
However, since all but the last node in a sequence are of the same type (non-terminal), the effect will most likely be less pronounced.
While this would be interesting to study, it falls outside the scope of this paper.
2. Neither of the two sets of methods we compare against released code (Dong and Lapata did, but not for the IFTTT task).
For this reason, we quote F1 accuracy results directly from those papers, while we are not able to obtain finer grained statistics (such as argument accuracy) for those. We can add the corresponding result for our other model, and we have contacted the authors of the other methods to ask for access to their results. If we are able to get these, we will add the corresponding results for their methods too.
3. Yes, that’s a typo, thanks for pointing it out.
The encoder part of the model is still a seq RNN which encodes a string representing the tree structure?  The string itself already contains some (not all) topological information of this tree. Is that correct?
For the experiments shown here, the encoder is always a seq RNN (though a tree-structured encoder could be used as well, if the structure of the input is known).
The string fed into this encoder does indeed have some implicit information about topology (e.g. whether two words are contiguous impacts the probability of them being parent-child and sibling-sibling). The task of the decoder is thus to infer the tree-structure from this noisy signal.
Do you have any experiments to measure the impact of beam size on the performance? I'm curious to see how accurate the greedy decoding would be.
We don't, at least not yet. Unfortunately, doing beam search over trees in this framework poses various modeling challenges, which we have not yet fully resolved.
However, the results in the paper show that even greedy (beam size 1) decoding does a decent job. We conjecture this is because trees entail shorter dependencies between nodes than flat sequences, so greedy decoding is less detrimental than for sequence decoders.
In addition, the tasks presented here might be more amenable to greedy decoding than, say, machine translation with long sentences.
Authors' response well answered my questions. Thanks.
Evaluation not changed.
This paper proposes a neural model for generating tree structure output from scratch. The model does 1) separate the recurrence between depths and siblings; 2) separate the topology and label generation, and outperforms previous methods on a benchmark IFTTT dataset. Compared to previous tree-decoding methods, the model avoids manually annotating subtrees with special tokens, and thus is a very good alternative to such problems. The paper does solid experiments on one synthetic dataset, and outperforms alternative methods on one real-world IFTTT dataset.
There are couple of interesting results in the paper that I believe is worth further investigation.
Firstly, on the synthetic dataset, the precision drops rapidly with the number of nodes. Is it because that the vector representation of the sequential encoder fails to provide sufficient information of long sequences, such that the tree decoder can not do a good job? Or is it because that such tree decoder is not tolerant to the long sequence input, i.e., large tree structure? I believe that it is important to understand this before a better model can be developed.
For example, if it is the fault of encoder, maybe an attention layer can be added, as in a seq-to-seq model, to preserve more information of the input sequence.
Moreover, besides only showing how the precision changes with the number of nodes in the tree, it might be interesting to investigate how it goes with 1) number of depths; 2) number of widths; 3) symmetricity; etc.
Moreover, as greedy search is used in decoding, it might be interesting to see how it helps, if it does, to use beam-search in tree decoding.
On the IFTTT dataset, listing more statistics about this dataset might be helpful for better understanding the difficulty of this task. How deep are the trees? How large are the vocabularies on both language and program sides?
The paper is well written, except for minor typo as mentioned in my pre-review questions.
In general, I believe this is a solid paper, and more can be explored in this direction.
So I tend to accept it.
Thank you for your comments and observations. We will add additional statistics on the results in the synthetic dataset and of the IFTTT task and revise the manuscript soon with this information.
Regarding your question, precision drops as tree size grows likely because of both of the factors you mention: more information has to be encoded in a fixed size vector, and in a larger tree there is more opportunity for information degradation while decoding. We’ve observed that output tree depth has the strongest effect in performance degradation.
A possible way to further investigate which of these two components (encoder or decoder) has a greater impact in performance degradation would be to compare against a sequence to sequence encoder-decoder, and measure the effect of input length in that case. This, however, still has the issue that an encoder might be good at generating representations to be used as starting states for sequence decoding, but not as good when generating hidden states to be used to generate a tree.
Thus, really isolating the effect of encoder and decoder in this phenomenon is quite challenging.
Regarding using attention and beam search, we agree these are interesting directions and we will leave them for future work.
You mentioned ``technical challenges'' of using beam search in tree decoding. Could you be more specific about the challenges? Are they mathematical challenges or implementation difficulty? Why can't it be achieved by using same techniques as in sequential beam-search?
In details, you start with the root, and after first spanning, you get K possibilities of next step. You feed them to a priority-queue, and span each item further in the order of cost/heuristics. In this way, isn't each item in the queue in fact a partially-finished tree?
Then you can stop the decoding when you have K completed trees? Am I over-simplifying the problem?
It will be great if you could explain this in more details. Thanks.
Again, thank you for your feedback. We have uploaded a revised version of the paper that addresses most of your comments. Please note in particular:
* Additional statistics for performance degradation in the synthetic task as a function of depth and width (page 7).
* Additional details about the characteristics and difficulty of the IFTTT task (Appendix, pages 14-15): vocabulary sizes, tree depth, etc.
* A full additional task: machine translation, with some interesting and promising results (Section 4.3).
The challenges are in terms of implementation. If the tree structure was given, we agree that beam search would proceed in the way you describe, which in essence is very similar to sequence beam search, following some predefined tree traversal path.
However, when the actual tree structure has to be constructed from scratch during the search, things get significantly more complicated. Suppose we want to perform beam search with beam size K and vocabulary of size V. There are two main differences with the sequence case:
* In sequence beam search, where to “branch out" at time t is unambiguous: the next step to be added is always appended at the end of previous (length  t-1) sequences. In the tree case, however, one can branch out from any of the current leaves of a tree.
* In sequence beam search, one must select the top K among K*V options, all of these having one of K possible prefixes (of length t-1), and all of them being topologically equivalent (length t sequences starting from the left boundary of the sentence). In the tree case, a tree only shares prefix with other trees that have followed the exact same branching decisions.
So, to select the surviving K trees, should we consider only trees with the same prefix (i.e. having a width-k beam per prefix) or all trees, even if not directly comparable ( ~ global beam search)?
What this boils down to is that tree-generation beam search requires some sort of “tree of decisions”, where every state can be thought of as (T_p, m,  t_a, t_f, L):
* T_P:  a tree prefix T_p encompassing all previous decisions (tree structure and node labels)
* m: a node to search over, selected among unprocessed leaves of T_p
* t_a: boolean ancestral branching variable for node m
* t_f:  idem, fraternal
* L: a label for node M
Starting from the prefix common to all trees (the root), every decision (m,t_a,t_f,L) leads to a new state, and there are O(n)*2*2*V such possible decisions (where n is prefix tree size), so this meta-tree grows fast.
While this is certainly doable, it requires some careful bookkeeping that we have not managed to implement yet.
As a compromise between this and myopic node-level argmax decoding (which we were doing before), we have opted for a strategy of sampling K trees independently, scoring them, and keeping the highest scored. This is how we generated the translations in the MT task.
Thank you for the question, beam search is definitely a key point that we’ll have to address in the near future to improve the quality of generation.
Thanks for the clarification of the technical difficulty. It makes a lot of sense, though I wonder if it can (or even should) be simplified from a probabilistic view.
(This discussion is only for research purpose, and I do not expect beam search to be added in the current paper.
That is, my final evaluation will be made based on current paper quality.)
As you explain, the state is a tuple of (T_p, m,  t_a, t_f, L):
* T_P:  a tree prefix T_p encompassing all previous decisions (tree structure and node labels)
* m: a node to search over, selected among unprocessed leaves of T_p
* t_a: boolean ancestral branching variable for node m
* t_f:  idem, fraternal
* L: a label for node M
Given T_p and M, the t_a, t_f and L becomes independent, and their joint probability is:
P(t_a, t_f, L | T_p, M)=P(t_a | T_p, M) * P(t_f | T_p, M) * P(L | T_p, M), which gives
log P(t_a, t_f, L) = log P(t_a) + log P(t_f) + log P(L).
This way, instead of separating the decisions, it is natural to sort the top K tuples (t_a, t_f, L) based on the total cost log P(t_a, t_f, L).
Each of these top K tuples yields a new partial-tree.
There should be other alternatives to simplify the decisions in principle.
The paper propose DRNN as a neural decoder for tree structures. I like the model architecture since it has two clear improvements over traditional approaches — (1) the information flows in two directions, both from the parent and from siblings, which is desirable in tree structures (2) the model use a probability distribution to model the tree boundary (i.e. the last sibling or the leaf). This avoids the use of special ending symbols which is larger in size and putting more things to learn for the parameters (shared with other symbols).
The authors test the DRNN using the tasks of recovering the synthetic trees and recovering functional programs. The model did better than traditional methods like seq2seq models.
I think the recovering synthetic tree task is not very satisfying for two reasons — (1) the surface form itself already containing some of the topological information which makes the task easier than it should be (2) as we can see from figure 3, when the number of nodes grows (even to a number not very large), the performance of the model drops dramatically, I am not sure if a simple baseline only captures the topological information in the surface string would be much worse than this. And DRNN in this case, seems can’t show its full potentials since the length of the information flow in the model won’t be very long.
I think the experiments are interesting.
But I think there are some other tasks which are more difficult and the tree structure information are more important in such tasks.
For example, we have the seq2seq parsing model (Vinyals et al, 2014), is it possible to use the DRNN proposed here on the decoder side? I think tasks like this can show more potentials of the DRNN and can be very convincing that model architectures like this are better than traditional alternatives.
Thank you for your comments.
Regarding the difficulty of the synthetic task, the surface form indeed contains some topological information, otherwise the task would be unrealistic.
But this does not necessarily mean the task is easy. Given that two topological decisions must be made for every node, there are is a combinatorial number of possible output trees for a fixed size sentence, even if always processing sentences in a breadth-first left-to-right manner.
For example, using lambda-logical notation, the surface form “A B C D” can lead to the trees: “A B C D”, “A (B C D)”, “A (B) C (D)”, “A (B (C (D)))”, etc. There are roughly O(n^2) such possibilities if we knew the node labels.
But in our setting the decoder is not given the labels, it generates them, so there’s an exponential (in n) number of possible outputs.
So, even if there is some topological information in the surface form, recovering the correct tree directly from it without learning some general properties about feasible output trees would be very hard. The task of the DRNN decoder is thus to learn these properties during training, similar to how a seq-to-seq model must implicitly learn a language model on the decoder side, to be able to generate sentences not only utilizing the information provided by the input, but also generating feasible outputs as per previously seen data.
We agree that in its current form, it’s not easy to appreciate the difficulty of the task. To address this and your point (2), we are implementing a simple baseline to give the reader more information to understand the complexity of the task and the performance of our method. We will add these results as they become available.
Regarding additional tasks, as we mention to another reviewer, we are setting up an additional (significantly harder) task that will evaluate the performance of our method when dealing with larger, more diverse trees. We hope to have the results ready before the review discussion deadline.
Again, thank you for your feedback. We have uploaded a revised version of the paper that address your comments.
In particular:
* We added results for an additional (much more difficult) task: machine translation.
Although the setting is quite simplified (small training dataset, little hyper-parameter tuning), the proposed method shows some interesting properties that its sequential counterparts lack.
Yet to add but coming soon:
* A baseline for the synthetic task, which puts more clearly in evidence the difficulty of this seemingly naive task. We will update the document with these results soon.
We now include results in the synthetic dataset for a (strong) baseline, a PCFG parser. Please see latest revision for details.
Thanks
This paper proposes a variant of a recurrent neural network that has two orthogonal temporal dimensions that can be used as a decoder to generate tree structures (including the topology) in an encoder-decoder setting. The architecture is well motivated and I can see several applications (in addition to what's presented in the paper) that need to generate tree structures given an unstructured data.
One weakness of the paper is the limitation of experiments. IFTTT dataset seems to be an interesting appropriate application, and there is also a synthetic dataset, however it would be more interesting to see more natural language applications with syntactic tree structures.
Still, I consider the experiments sufficient as a first step to showcase a novel architecture.
A strength is that the authors experiment with different design decisions when building the topology predictor components of the architecture, about when / how to decide to terminate, as opposed to making a single arbitrary choice.
I see future applications of this architecture and it seems to have interesting directions for future work so I suggest its acceptance as a conference contribution.
Thank you for your comments. We chose the IFTTT task because of two main reasons.
First, we wanted a task where the input was sequential, so that we could use a vanilla RNN as an encoder, thus minimizing the variability due to design choices on the encoder side.
Second, we wanted a task where the actual tree was a crucial part of the output (as opposed to just being a latent structure to generate the nodes), so that a regular sequence-to-sequence model could not be directly used.
Though we think the IFTTT task was reasonable for this experimental setting, we agree that there are other interesting applications of the proposed model.
To expand the experimental framework, we are working on an additional NLP task that will more clearly highlight the advantage of a DRNNs over sequence decoders, and which we’re aiming to have ready before the end of the review discussion period.
Thank you again for your comments. We have uploaded a revised version of the paper that address the limitation of the experiments (among other things). We added a toy machine translation task, which albeit simplified, highlights key advantages of using a method that can make use of rich language structure during decoding, which we achieve by training DRNNs on dependency-parsed data.

Meta-review:
role: PC
data: comment: The paper introduces a new model for generating trees decorated with node embeddings. Interestingly the authors do not assume that even leaf nodes in the tree are known a-priori. There has been very little work on this setting, and, the problem is quite important and general. Though the experiments are somewhat limited, reviewers generally believe that they are sufficient to show that the approach holds a promise.
 
 + an important and under-explored setting
 + novel model
 + well written
 
 - experimentation could be stronger (but seems sufficient -- both on real and artificial data)

Decision: Accept (Poster)