Tree-Structured Attention with Hierarchical Accumulation

Abstract: Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks. However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently. On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efficiently as the Transformer. In this paper, we attempt to bridge this gap with Hierarchical Accumulation to encode parse tree structures into self-attention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German task. It also yields improvements over Transformer and Tree-LSTM on three text classification tasks. We further demonstrate that using hierarchical priors can compensate for data shortage, and that our model prefers phrase-level attentions over token-level attentions.

Reviews and author responses:

This paper extends transformers by enabling them to incorporate hierarchical structures like constituency trees structured attention with hierarchical accumulation. In particular, they modified the architecture of transformers such that they can learn phrase-level attention scores, and use them in the final assigned task of text classification or language translation. 
They also explored a couple of variants of their proposed architecture: Hierarchical embeddings and Subnet masking which helped them outperform SOTA methods including Tree-LSTM (similar to this paper in principle, except that it was designed for LSTM-based models and not transformers). 

The paper is well written and well-augmented with supportive figures. Particularly, Figure1 was very helpful in understanding all the complexities of the proposed model. Further, the authors justify the utility of the proposed approach covering different aspects of evaluation including comparative studies with baselines, ablation studies, phrase vs token-level attentions, training-time analysis. 

A limitation of this work is high inference cost. As the results indicate, parsing trees from text is the most costly step in the entire framework, and consequently, the inference time of proposed model will still be much higher than transformers. Hence, this work might still not be applicable to low-latency constraint scenarios. 


Other Comments: 
1) I did not fully understand why it would be better to mask out the non descendants in subnet masking approach. Why shouldn't a phrase node seek attention from tokens outside its scope? Probably the answer lies in the the way these trees are constructed. Nevertheless, it would be useful to provide some intuition with examples to motivate subtree masking. 

2) In Equation(5), the subscripts "i-1" should be replaced with "i"? Otherwise it will be sensitive to ordering of non-terminal nodes in N, and also Figure 1 wouldn't make sense.


Summary: This paper describes an attention-based method to encode trees with constant parallel-time complexity maintaining scalability, uses this model to encode constituency parses of input sentences for the tasks of machine translation and text classification, and shows improved accuracy/bleu over models that do not encode the parses. 

Strengths:

The method proposed by the authors is scalable despite encoding tree structures. The models give considerable improvements on various machine translation datasets and the authors also show that the model is more sample efficient. I appreciate the charts showing training and inference time with sentence length, and the tables showing the ablations and attention distributions. 

Weaknesses

The authors do not use pre-trained embeddings for any of the classification models, but using these embeddings boost performance to much more than what they authors have achieved here. My main question is, if we use pre-trained embeddings, do encoding constituency parses add anything over and above them? I would like to see this method improve results over some state of the art classification models and not just over the tree-LSTM. In other words, how much classification performance does this method yield over current SOTA models, because the SST results are quite a bit below the current SOTA.

The authors achieve an accuracy of 98.2 on the IMDB dataset. Is this actually the case or is this a bug? Even models using BERT barely achieve an accuracy of 96% (http://nlpprogress.com/english/sentiment_analysis.html). Or am I looking at the wrong dataset here? Can the authors clarify?


This paper proposes a novel mechanism to leverage additional tree-structure information into the transformer. The proposed method consist three main component: a hierarchical accumulation strategy to aggregate the leaf node embedding information to the non-terminal nodes; a hierarchical embedding which is akin to positional embedding, however encapsulating the leaf-to-node and relative position information; a sub-tree masking strategy to filter out irrelevant information. Although the formulations seem a little bit overcomplicated/cumbersome to me, I find the figures are really helpful to understand the gists, which are intuitive and straightforward. 

Strengths:

* Intuitive and novel ideas. The tree structure information is incorporated into the transformer in a sophisticated and elegant manner, with only constant time-complexity overhead. 

* Strong empirical results and well-defined ablation study. Good analysis on the performance vs dataset size and time complexity

Cons:

* Heavy notations which could be more concise. 

* Implementing such an architecture without source code could be difficult.

Questions:

* Comparing with the naive transformer architecture (e.g. base model), how many additional parameters are there in your model? I see that for fair comparison, the authors use the same base transformer architecture, however it would still be very helpful if they can provide the statistics of the number of parameters for the proposed model and the compared baselines. 

* Are all the baselines using the same parser (CoreNLP)? If not, would the difference of parsing trees be a confounding factor?





Meta-review: This paper incorporates tree-structured information about a sentence into how transformers process it. Results are improved. The paper is clear. Reviewers liked it. Clear accept.

Decision: accept-poster