Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work addresses the problem of aspect-based sentiment analysis *with* rationale extraction that can serve as an explanation (interpretation) of the decision.
Compared to previous work this paper models the problem as multi-aspect classification, therefore having one model for all aspects instead of one for each aspect. This is obviously useful, but in addition to having a smaller total model they show that they also have (slightly) better prediction AND AT THE SAME TIME "better" explanations (which is always hard to evaluate).

This is a useful paper, and I can clearly see it used. However, it might not get many ICLR attendants excited.

I found the paper very hard to read, having to re-read many parts to understand exactly what is the goal of each section. The row names of the table 1-3 do not correspond to the names in the text which makes the understanding even harder. Also, I think that some plot that shows accuracy vs "interpretability", with model size as a third dimension (eg surface of the bubble) would convey the take-home of the paper much clearer than the many tables with small differences between them.

A minor comment: the SVM has a huge parameter space, which is weird. Are you using *all* bigrams? A fairer comparison would be to use more n-grams, but keep only the most frequent ones.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a neural model for multi-aspect sentiment of reviews. Apart from sentiment classification, the architecture is able to return a mask for each aspect highlighting the sequences of words used to make the sentiment prediction for that aspect. This improves the interpretability of the model. The aspects are fixed and the training set requires a multi-labeled dataset with one binary positive/negative label per aspect.

The network architecture is mostly standard: a text review is turned into a list of word embeddings which are fed through a trainable masker outputting one mask per aspect, essentially "selecting" a different set of words (embeddings) per aspect. From here on, a standard encoder can be used (the authors end up using a CNN with max-pooling but LSTM or other could be used), followed by a classification step (standard MLP) outputting a sentiment score per aspect. A couple tricks are used to make the masks converge to something meaningful by adding regularization terms to control the number of selected words and encouraging consecutive words to be selected.

I'm leaning toward rejection partly due to the limited novelty in the approach but mainly due to the experimental evaluation section which requires improvements.
- High aspect correlation baseline: the average aspect correlation across reviews in the training set is high (that is, if one aspect is positive, most aspects will be and viceversa). A natural baseline to include is the optimal "no-aspect" decision boundary: in other words, if for each review we were given the majority rating across aspects (is this review mostly positive or negative) and were able to predict this always correctly, what would the macro-aspect F1 score be if we were to simply predict that label to all aspect of that review?
- Please quantify how much does your model rely on this aspect correlation. For example, if you were to sort the reviews in the validation set by their aspect correlation, and chart the F1 score as a function of the prefix of this sorted dataset what would we see? This would be especially telling when comparing the different baselines and classifiers again each other in this chart.
- SVM: you mentioned not running it on the larger datasets due to lengthy training time. I would suggest including NB-SVM implemented with logistic regression rather than actual SVM since it's always a great baseline and fast to compute (in "Baselines and Bigrams: Simple, Good Sentiment and Topic Classification").
- The final model that ends up giving the best results is really a 2-step pipeline (rather than end-to-end trainable) and requires training twice: once to learn the masks and once to actually train the rest of the network after the word embeddings are extended with the masks.

 










review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper is mostly concerned with multi-aspect sentiment classification.

Multi-aspect sentiment analysis is an old problem (> 10 years), that usually operates on datasets with multidimensional sentiment labels (price, cleanliness, etc.). This paper has some variation on this old problem in the sense that it's more focused on generating explanations / justifications rather than purely on sentiment analysis.

Although an interesting variant on a (somewhat niche) existing problem, I do feel that some aspects of this paper are a little dated -- claiming that the method is more useful for interpretability / justification doesn't seem so convincing these days unless you can make a stronger argument as to how it could actually be used to generate justifications that would be shown to users. I don't really see that in the current version. At the moment the output of the system appears to be attention weights over several aspects, showing which words in a review focus on which aspect. But this doesn't seem so different from how older methods (e.g. Bag-of-Words based techniques for multi-aspect sentiment classification), though their labels were possibly at the level of sentences rather than reviews.

The actual method itself seems appropriate, and is a nice update on previous techniques for multi-aspect sentiment analysis.

Perhaps I missed some detail explaining why such a comparison is impossible, but there seems to be no direct comparison against traditional techniques for multi-aspect sentiment analysis. E.g. aren't even traditional models (like say Titov & McDonald) comparable to this? I understand the goal is somewhat different, but the input/output modality seems similar and such a comparison could be helpful.

Overall, I think these days the standard is quite high for papers that make claims about generating convincing justifications/interpretations/explanations. There's little by way of human evaluation (there's a bit in the appendix, but it's not really focused on whether or not the model is useful for the claimed purpose of "explanation").

More positively, the authors have done *a lot* by way of experiments, in terms of ablations, and showing quantitative performance from different dimensions. The appendix is detailed and shows many examples. Overall I feel somewhat borderline about this paper, because there's a lot here, but I feel some of these key components mentioned above are missing.
review_title: Official Blind Review #1

Meta-review: This paper proposes a neural network model for predicting multi-aspect sentiment and generating masks that can justify the predictions. The positive aspects of the paper include improved results over the state-of-the-art.

Reviewers found the technical novelty limited, and the experiments short of being fully convincing. After the author rebuttal, there were discussions between the reviewers and the AC, and the reviewers still thought the paper is not fully convincing given these limitations.

I thank the authors for their submission and detailed responses to the reviewers and hope to see this research in a future venue.
Decision: Reject