Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel mechanism to leverage additional tree-structure information into the transformer. The proposed method consist three main component: a hierarchical accumulation strategy to aggregate the leaf node embedding information to the non-terminal nodes; a hierarchical embedding which is akin to positional embedding, however encapsulating the leaf-to-node and relative position information; a sub-tree masking strategy to filter out irrelevant information. Although the formulations seem a little bit overcomplicated/cumbersome to me, I find the figures are really helpful to understand the gists, which are intuitive and straightforward. 

Strengths:

* Intuitive and novel ideas. The tree structure information is incorporated into the transformer in a sophisticated and elegant manner, with only constant time-complexity overhead. 

* Strong empirical results and well-defined ablation study. Good analysis on the performance vs dataset size and time complexity

Cons:

* Heavy notations which could be more concise. 

* Implementing such an architecture without source code could be difficult.

Questions:

* Comparing with the naive transformer architecture (e.g. base model), how many additional parameters are there in your model? I see that for fair comparison, the authors use the same base transformer architecture, however it would still be very helpful if they can provide the statistics of the number of parameters for the proposed model and the compared baselines. 

* Are all the baselines using the same parser (CoreNLP)? If not, would the difference of parsing trees be a confounding factor?




review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: This paper describes an attention-based method to encode trees with constant parallel-time complexity maintaining scalability, uses this model to encode constituency parses of input sentences for the tasks of machine translation and text classification, and shows improved accuracy/bleu over models that do not encode the parses. 

Strengths:

The method proposed by the authors is scalable despite encoding tree structures. The models give considerable improvements on various machine translation datasets and the authors also show that the model is more sample efficient. I appreciate the charts showing training and inference time with sentence length, and the tables showing the ablations and attention distributions. 

Weaknesses

The authors do not use pre-trained embeddings for any of the classification models, but using these embeddings boost performance to much more than what they authors have achieved here. My main question is, if we use pre-trained embeddings, do encoding constituency parses add anything over and above them? I would like to see this method improve results over some state of the art classification models and not just over the tree-LSTM. In other words, how much classification performance does this method yield over current SOTA models, because the SST results are quite a bit below the current SOTA.

The authors achieve an accuracy of 98.2 on the IMDB dataset. Is this actually the case or is this a bug? Even models using BERT barely achieve an accuracy of 96% (http://nlpprogress.com/english/sentiment_analysis.html). Or am I looking at the wrong dataset here? Can the authors clarify?

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper extends transformers by enabling them to incorporate hierarchical structures like constituency trees structured attention with hierarchical accumulation. In particular, they modified the architecture of transformers such that they can learn phrase-level attention scores, and use them in the final assigned task of text classification or language translation. 
They also explored a couple of variants of their proposed architecture: Hierarchical embeddings and Subnet masking which helped them outperform SOTA methods including Tree-LSTM (similar to this paper in principle, except that it was designed for LSTM-based models and not transformers). 

The paper is well written and well-augmented with supportive figures. Particularly, Figure1 was very helpful in understanding all the complexities of the proposed model. Further, the authors justify the utility of the proposed approach covering different aspects of evaluation including comparative studies with baselines, ablation studies, phrase vs token-level attentions, training-time analysis. 

A limitation of this work is high inference cost. As the results indicate, parsing trees from text is the most costly step in the entire framework, and consequently, the inference time of proposed model will still be much higher than transformers. Hence, this work might still not be applicable to low-latency constraint scenarios. 


Other Comments: 
1) I did not fully understand why it would be better to mask out the non descendants in subnet masking approach. Why shouldn't a phrase node seek attention from tokens outside its scope? Probably the answer lies in the the way these trees are constructed. Nevertheless, it would be useful to provide some intuition with examples to motivate subtree masking. 

2) In Equation(5), the subscripts "i-1" should be replaced with "i"? Otherwise it will be sensitive to ordering of non-terminal nodes in N, and also Figure 1 wouldn't make sense.

review_title: Official Blind Review #2

Meta-review: This paper incorporates tree-structured information about a sentence into how transformers process it. Results are improved. The paper is clear. Reviewers liked it. Clear accept.
Decision: Accept (Poster)