Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: This paper presents a Transformer-based model for aspect-based sentiment analysis, intended to support the unsupervised induction of constituents within the Transformer forward pass. Their evaluations demonstrate that their model can match (and in some cases improve upon) models which depend upon explicit dependency parse information in the input, and reliably exceed parse-free models.  

I strongly vote for rejection, largely on grounds of quality, elaborated below.  

Pros: The paper begins with an interesting idea and implementation, and the results support the claim that their architecture may replicate some of the contribution of dependency parse information.

Cons: The presentation is unclear on the concept of "constituent" and the motivation of the model. The later iterations of the model become rather complicated and don't seem well-motivated. The qualitative evaluations don't strongly support the claims of the paper.  

Quality

1. The design of the model and the use of the word "constituent" seems conceptually problematic to me. What do you take the word "constituent" to mean for your motivation and model design? It seems to me that it might be sufficient to call the ConsTrans model a "spatial attention smoothing" model. Why isn't this a sufficient description? What does the concept "constituent" add?This question is relevant because the later model iterations are developed on further syntactic ideas (typed/labeled relations, syntactic "distance"). But if the model doesn't have a necessary syntactic framing, it's not clear these are the correct model improvements to consider.
2. The qualitative evaluations are far too light, testing only a small amount of the model performance.
  a. Grammar induction: In particular, I would appreciate a far more in-depth evaluation of the inferred constituent structures. How do they compare to gold and silver dependency parses of within-domain sentences? The current evaluation checks for model inferences on just one short span of text (the aspect term). This is probably one of the easiest terms for the model to recognize as a constituent, too, since the aspect terms are known to be constituents and have verbatim copies in the input sentence. The current evaluation is also only performed on Twitter17 --- why?
  b. Interpreting learnt relation labels: I found this evaluation extremely confusing, involving an ad-hoc dependency parsing algorithm built upon an a posteriori fact discovered in model analysis (that relation embedding L2 norm indicates inverse syntactic distance). The resulting parse in Figure 5 is almost entirely incorrect and commits many basic mistakes (for example, not linking the determiner "the" with its immediately adjacent noun). The claim about linking adjectives and nouns is not particularly interesting to me, since this is far less ambitious than the motivation of the model --- if it were, the model could have been quite a bit simpler, I think.
3. Significance results are given (thanks!) but with a strangely high significance threshold (0.15 at one point and 0.2 at another). This is not a reasonable significance threshold in my view.  

Clarity

Some minor comments:

1. The constituent derivation algorithm is not clear. Eqn 3 suggests that constituent probabilities are a function of token pairs, but Algorithm 1 line 11 suggests that they can be indexed by a single token position. Is something missing from the algorithm presentation?
2. Eqn 3 first condition should be i <= j, I think.
3. Figure 4 is not a complete sentence. There's no verb. There's also no clear sentiment (aspect-based or otherwise) that we could talk about for this sentence. A different sentence could serve as both a motivating example for constituency and as a more revealing example of the model's syntactic knowledge.

## Post-rebuttal response

I have read the other reviews and the authors' responses, and do not wish to change my review. The proposed model seems quite complex with somewhat unclear conceptual motivations, and does not clearly demonstrate impressive performance gains despite the complexity. I would suggest that the authors attempt to change one of these things in a later paper, either by revisiting the model design, or task choice and evaluation (to better motivate the model).
review_title: Overly complex model, conceptually unclear, and not well evaluated

final_score: rating: 3: Clear rejection
review_content: This paper presents a constituent-based transformer model with auxiliary relation embeddings and labels to enhance the performance of aspect-based sentiment analysis. The constituent-based transformer modifies the original transformer by re-weighting the attention weights with constituent-based similarities. The auxiliary relation predictions helps to differentiate the relations that are useful for sentiment prediction. 

In general, this paper is readable and has a clear motivation. The strengths include:
1. The idea of modifying the basic transformer to a constituent-based transformer by incorporating constituent similarities without any supervision is interesting. 
2. The authors propose three model variations progressively, namely ConsTrans, RelConsTrans and RelConsTransLG to clearly demonstrate the motivation and effect of each variation.

However, the paper still lack the following aspects:
1. Some parts of the description is unclear with details missing. From (9), the generated $l_{ij}$ is a continuous value, how do you make it a relation label? Why do you take the L2 norm on $r_{ij}$ in (10)? What is the intuition of such computation to obtain the MSE? What is $L(\hat{y},y)$ in (11) and how to compute it? If $y$ is the sentiment label of the aspects, how it could be used to update the label generator $l_{ij}$? What is the intuition of training $\theta_{main}$ and $\theta_{aux}$ using separate datasets? Could you show what is the difference in terms of the performance?
2. Given the complexity of the model, the improvement compared to the baseline models are relatively trivial. And it is somewhat insufficient to limit the application only to aspect-based sentiment analysis. The contribution is thus limited.
3. How do you sample the meta-train set? Is the result over one meta-train set or averaged over different meta-train set?
4. Minor points: caption is missing for figure 2. The first row in (3) should be $i\leq j$.


review_title: Interesting modification on transformer, but limited applications and improvements

final_score: rating: 4: Ok but not good enough - rejection
review_content: According to two phenomena in Transformer's pre-trained language model: 
1. Tokens with similar self-attention distribution tend to be distributed in the same constituent; 
2. The L2 distance between token representations reveals the degree of syntactic dependency; 

an aspect-based sentimental analysis system is designed which does not rely on explicit syntactic information. It is not a surprising point to combine the so-called syntactic information learning with sentential analysis. Although the model has achieved comparable results with systems using explicit syntax, the meta-learning is somewhat overclaimed.

There are some defects in this work:
1. Over dependence on the pre-trained language model: Although the author claims that explicit syntactic information is not used, the system relies on the syntactic information contained in the pre-trained language models such as BERT to promote the training of ConsTrans, RelConsTrans, and RelConsTransLG. I doubt whether the training of the model can converge without the pre-trained language model only relying on the randomly initialized transformer and still achieve comparable accuracy with systems with explicit syntax? What is the performance of using a weaker pre-trained language model like Distibert? How about the performance in stronger pre-trained models like RoBERTa?
2. Model size and inference performance: the extra transformer layers added makes the result not directly comparable with that of BERT. Therefore, the author needs to report the effect of the BERT baseline under the same parameter scale. In addition, to what extent will the introduction of the constituent probability scorer and relation label generator affect the inference performance of the model? For this simple task of sentiment analysis, do we need such a complex model design and training method design after the already huge pre-trained language model such as BERT?
3. Inappropriate description: 
(1) In RelConsTrans, the proposed model encodes the syntactic relation as a feature for constituent scoring based on the role of syntactic relation for sentiment analysis. But according to Eq. 6 and 7, the so-called relation (that could not be determined) act as a bias in scoring function, I think it just provides a more feature source (dependency syntax) for scoring. 
(2) In RelConsTransLG, the r_{i,j} and l_{i,j} is similar. I think the only conclusion that can be drawn is that the model is further enhanced with the features of dependency syntax recently, which has little relationship with meta-learning. 

Generally speaking, this work can be regarded as integrating the implicit constituent and dependency syntactic features from the pre-trained language models into the training in the sentient analysis, so that explicit syntax is no longer needed. Appendix A.6 and A.7 also confirm this point. Although it has a positive effect on the need to remove explicit syntax, its reliance on pre-trained language models limits the extensibility of this approach and is likely to be ineffective on traditional unpretrained LSTM, CNN, and Transformer models. In addition, since the writing is pretended to make the model close to a concept of meta-learning, it ignores the important points that should be clarified.
review_title: This paper studies the model design of removing explicit syntax from sentiment analysis task, but it also has some limitations.

Meta-review: The paper proposes a constituent-based transformer for aspect-based sentiment analysis. The approach allows conducting aspect-based sentiment analysis to leverage the syntactic information without pre-specified dependency parse trees. 

Overall, the idea is interesting. However, all the reviewers shared the following concerns: 

- Paper descriptions of methodology and experiments are not clear and require significant rewriting and reorganization. 
- The proposed approach is not well-justified by the empirical study presented in the paper. Especially, a more detailed ablation study is required to justify the design. 

We would suggest the authors addressing the feedback from the reviewers to improve the paper. 
Decision: Reject