Title: Structured Attention Networks
Authors: Yoon Kim
Carl Denton
Luong Hoang
Alexander M. Rush
Venue: ICLR 2017

Abstract: Attention networks have proven to be an effective approach for embedding categorical inference within a deep neural network. However, for many tasks we may want to model richer structural dependencies without abandoning end-to-end training. In this work, we experiment with incorporating richer structural distributions, encoded using graphical models, within deep networks. We show that these structured attention networks are simple extensions of the basic attention procedure, and that they allow for extending attention  beyond the standard soft-selection approach, such as attending to partial segmentations or to subtrees. We experiment with two different classes of structured attention networks: a linear-chain conditional random field and a graph-based parsing model, and describe how these models can be practically implemented as neural network layers. Experiments show that this approach is effective for incorporating structural biases, and structured attention networks outperform baseline attention models on a variety of synthetic and real tasks: tree transduction, neural machine translation, question answering, and natural language inference. We further find that models trained in this way learn interesting unsupervised hidden representations that generalize simple attention.

TL;DR: Use a graphical model as a hidden layer to perform attention over latent structures

Review discussion:
In this framework, in eq1, \sum_{i=1}^n p(z=i|x,q) equals 1. In eq2, in my understanding, \sum_{i=1}^n p(z_i=1) is not guaranteed to be 1.
Therefore, the final representation computed from the attention can sometimes be as large as n times the vector representations of x.
When the sentence is very long and the attention wants to selected many things, will this be a problem? The vector computed from the attention can be sometimes too large?
Thank you for your question.
Indeed, the sum is not guaranteed to be 1, and for the Sigmoid attention we found this to be ok, but for structured attention we found training to be unstable due to the issue you pointed out. We therefore had to normalize the marginals. We mention this in the appendix (page 14: "The marginals are globally normalized for additional stability"), but we agree that this is an important detail. We will move this to the main part.
Hope this clarifies things, thanks!
In Section 3.2, Example 2: when the model is used for Syntactic Tree Selection, the constraints such as ensuring projectivity appear to require constraints that involve all of the z_ij variables. In this case, isn't the CRF clique structure fully connected? Is it still efficient to perform inference in these cases where you have larger cliques in the CRF?
Yes, the CRF is fully connected.
While it is true that (in general) fully connected means that inference is intractable, there are many special cases where inference is tractable/efficient.
Specifically, in this case we can use the inside-outside algorithm on the data structures of Eisner (1996) to obtain the marginal probability of each word's parent in O(n^3) time (with respect to the sentence length). Forward/backward pass through this inference algorithm is shown in appendix B.
Also, while we've focused on models that admit exact (and tractable) inference, our approach is applicable to approximate inference methods such as loopy belief propagation (since each step of belief propagation is differentiable).
For example, differentiating through approximate inference was explored in Gormley (2015), in the context of minimum risk training of higher-order dependency parsers
Thanks!
Why is the root symbol necessary in the tree transduction problem in 4.1?
In Fig 3 (right) the soft alignments seem wrong -- the row for the closing parenthesis ")" after "0 13" seem to be aligned with the wrong closing parenthesis (despite the claim in the caption).
Right--the root symbol is not necessary for 4.1, but we kept it for consistency with 4.3. There is no difference in performance when the root symbol is not used.
Regarding Fig 3, we agree that the alignment of ")" after "0 13" is incorrect. Our claim was that closing parentheses generally have more weight on opening parentheses (not necessarily one particular/correct opening parenthesis).
But we agree that the claim could be misleading/confusing--we will correct.
Thanks!
The authors propose to extend the “standard” attention mechanism, by extending it to consider a distribution over latent structures (e.g., alignments, syntactic parse trees, etc.). These latent variables are modeled as a graphical model with potentials derived from a neural network.
The paper is well-written and clear to understand. The proposed methods are evaluated on various problems, and in each case the “structured attention” models outperform baseline models (either one without attention, or using simple attention). For the two real-world tasks, the improvements obtained from the proposed approach are relatively small compared to the “simple” attention models, but the techniques are nonetheless interesting.
Main comments:
1. In the Japanese-English Machine Translation example, the relative difference in performance between the Sigmoid attention model, and the Structured attention model appears to be relatively small. In this case, I’m curious if the authors analyzed the attention alignments to determine whether the structured models resulted in better alignments.
In other words, if ground-truth alignments are available for the dataset, or if they can be human-annotated for some test examples, it would be interesting to measure the quality of the alignments in addition to the BLEU metric.
2. In the final experiment on natural language inference, I thought it was a bit surprising that using pretrained syntactic attention layers did not appear to improve model performance, but instead appear to degrade performance. I was curious if the authors have any hypotheses for why this is the case?
Minor comments:
1. Typographical error: Equation 1: “p(z | x, q” → “p(z | x, q)”
2. Section 3.3: “Past work has demonstrated that the techniques necessary for this approach, … ” →  “Past work has demonstrated the techniques necessary for this approach, … ”
Thank you for the comments.
- Checking the alignments seems like a great idea. Obtaining ground-truth alignments, as noted, could be tricky--we could potentially use an external alignment model and treat that as the ground truth (as in Luong et al. 2015).  We spot-checked a couple of translations ourselves using Google translate, and we further had a native Japanese speaker annotate the example sentence shown in the figure. The alignments from the sigmoid/structured attention model largely agreed with the hand annotation. This was not the case for the simple (softmax) model.
- We were also surprised to find that the pretrained syntactic attention model does not outperform the non-pretrained model for natural language inference. It is possible that the pretrained attention was too strict for this task. We hope to explore this in the future.
This is a solid paper that proposes to endow attention mechanisms with structure (the attention posterior probabilities becoming structured latent variables). Experiments are shown with segmental atention (as in semi-Markov models) and syntactic attention (as in projective dependency parsing), both in a synthetic task (tree transduction) and real world tasks (neural machine translation and natural language inference). There is a small gain in using structured attention over simple attention in the latter tasks. A clear accept.
The paper is very clear, the approach is novel and interesting, and the experiments seem to give a good proof of concept.
However, the use of structured attention in neural MT seems doesn't seem to be fully exploited here: segmental attention could be a way of approaching neural phrase-based MT, and syntactic attention offers a way of incorporating latent syntax in MT -- these seem very promising directions.
In particular it would be interesting to try to add some (semi-)supervision on these attention mechanisms (e.g. posterior marginals computed by an external parser) to see if that helps learning the attention components of the network, or at least help initializing them.
This seems to be the first interesting use of the backprop of forward-backward/inside-outside (Stoyanov et al. 2011). As stated in sec 3.3., for general probabilistic models the forward step over structured attention corresponds to the computation of first-order moments (posterior marginals) while the backprop step corresponds to second-order moments (gradients of marginals wrt log-potentials, i.e., Hessian of log-partition function). This extends the applicability of the proposed approach to arbitrary graphical models where these quantities can be computed efficiently. E.g. is there a generalized matrix-tree formula that allows to do backprop for non-projective syntax? On the negative side, I suspect the need for second-order statistics may bring some numerical instability in some problems, caused by the use of the signed log-space field. Was this seen in practice?
Minor comments/typos:
- last paragraph of sec 1: "standard attention attention"
- third paragraph of sec 3.2: "the on log-potentials"
- sec 4.1, Results: "... as it has no information about the source ordering" -- what do you mean here?
Thank you for the comments.
- Right, structured attention could be a way of doing neural phrase-based MT (i.e. segmentation attention at the word-level) and neural syntax-based MT (e.g. attend to word + soft parent during decoding). We are exploring this currently.
- Generalizing to other probabilistic models via the matrix-tree formulation is a very interesting direction of future research!
- We found that working in the log semi-field was enough to mitigate numerical issues.
- Sec 4.1 "... as it has no information about the source ordering": For the synthetic experiments, we do not employ an LSTM on the encoder side, and hence the baseline model (without any attention) is just a bag-of-words.
This is a very nice paper. The writing of the paper is clear. It starts from the traditional attention mechanism case. By interpreting the attention variable z as a distribution conditioned on the input x and query q, the proposed method naturally treat them as latent variables in graphical models. The potentials are computed using the neural network.
Under this view, the paper shows traditional dependencies between variables (i.e. structures) can be modeled explicitly into attentions. This enables the use of classical graphical models such as CRF and semi-markov CRF in the attention mechanism to capture the dependencies naturally inherit in the linguistic structures.
The experiments of the paper prove the usefulness of the model in various level — seq2seq and tree structure etc. I think it’s solid and the experiments are carefully done. It also includes careful engineering such as normalizing the marginals in the model.
In sum, I think this is a solid contribution and the approach will benefit the research in other problems.

Meta-review:
role: PC
data: comment: The area chair shares the reviewer's opinion and thinks that this is a very solid paper that deserves to be presented at ICLR. The idea is novel, well described and backed up by solid experiments (that show some empirical gains).

Decision: Accept (Poster)