Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigate incorporating entity abstraction to transformer language models for text reasoning tasks. The paper proposes different methods to inject entity abstraction information into transformer LMs and experiments on a synthetic dataset show that the proposed method helps compositional generalization. However, experiments on two realistic datasets show that the proposed method fail to effectively improve performance.
main_review: Strengths:
1. The paper is generally well written and easy to follow. The idea of using entity abstraction is simple to implement and can be widely applied if works.
2. The experimental results on CLUTRR is promising and show improved performance on compositional generalization, which is interesting.

Weaknesses:
1. The idea of using entity abstraction is straightforward and not very novel as doing POS tagging for input is widely used in many NLP models, especially in traditional NLP pipelines.
2. While the performance on synthetic datasets is promising, on realistic datasets the proposed method fails to improve the performance. Therefore the effectiveness of the method is not fully supported by the experiments.
summary_of_the_review: This paper proposed a simple method which incorporating entity abstraction to transformer language models. The method is simple while somewhat trivial, and the novelty/technical contribution is not very significant. The experimental results on synthetic dataset is good but not as well on realistic datasets. Therefore I believe this paper is below the bar.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studied the utility of incorporating entity type abstractions into pre-trained Transformers and its effectiveness on three tasks which require logical reasoning: 1) compositional language understanding with text-based relational
reasoning 2) multi-hop question answering 3) conversational question answering. Empirical result shows that the proposed method significantly improves the synthetic compositional language understanding task while only marginally improves the other two tasks in natural languages.
main_review: Strengths
- The paper is clearly written and easy to understand.
- The improvement on the synthetic logical reasoning setting (CLUTRR) is significant.
- The conclusion of the paper provides some insights on the capacity of T5-small model to understand entity abstraction.

Weakness
- The paper only shows end-to-end performance on the three tasks without deeper analysis on why entity abstraction does not always improve the performance. Like the authors pointed out, the current conclusion could be due to the noisy entity type predictions or the fact that the tasks do not require entity abstraction. Without the deeper analysis, readers cannot generalize the conclusion beyond the three datasets mentioned in this paper to other unexplored downstream tasks.
- Only one pretrained transformer model t5-small is tested, therefore the empirical findings of this paper cannot be applied to other models, which usually contain much more parameters and perform better on various downstream tasks.
summary_of_the_review: Technically, the paper presents novel approaches to incorporate entity type abstraction with transformer language model. As the related work section discussed, there are a lot of other related works also incorporating knowledge from named entities. To me, using only the entity type is marginally novel. Also, the empirical findings lack deeper analysis and were merely conducted on a single and toy-sized pretrained language model. For empirical results, the only significant improvement is on a synthetic dataset while the gain on real-world datasets is also marginal. Therefore, the significance of the paper's contribution is limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper empirically explored three different ways to add abstraction, additional input embeddings, a separate sequence to encode, and an auxiliary prediction task. Experiments on CLUTRR, HotpotQA, and CoQA show that the models with abstract entity knowledge perform slightly better than without it.
main_review: strengths:

1. This paper is in a good shape for reading, easy following. 

2. The experiment part gives convincing numbers on performance boosting on three NLP tasks, CLUTRR, HotpotQA, and CoQA.

weaknesses:

1. The experiment part lacks analysis on the model which could provide more intuitions on how the model works. More analytic experiments will bring more interesting findings of the proposed model. For instance, we could dive deep into why the model variants perform differently between CLUTRR and HotpotQA & CoQA.


summary_of_the_review: Overall, this is a solid empirical paper with convincing experimental results. It will be of great help for revealing the intuition of the model if introducing more analytic experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the effect of incorporating entity type abstraction into pre-trained Transformers. To achieve that, the authors have tried five different architectures to build the abstraction aware model. The proposed model is tested on three NLP datasets for reasoning. Empirical results show that entity type abstraction is beneficial in formally defined logical reasoning environments with simple language. While for QA datasets with more natural language, the baseline is already very strong and the improvement of incorporating abstraction is minor.
main_review: **Strengths**:
\
Overall a good paper. It is well presented, and the experimental results are solid.

**Weaknesses**:
\
My major concern is about the experiments. Specifically,
1. There is very minor boost in performance on QA datasets. The goal of the proposed model should be improving real-world data, instead of synthetic data (CLUTTR in this paper). Because improving numbers on synthetic data could be hacky, and not scalable to practical applications.
2. The performance of baselines are far below the current top performers. I understand that the paper is not interested in achieving new state-of-the-art, and the test set used might be different. However, in that case you can not prove that the proposed methods are still beneficial on a model that achieves way better baseline results. An analogy here is that something working under low-resource setting is not necessarily effective when there is adequate access to data.
3. The method of incorporating abstraction is not universal. We’ve seen different methods work for different datasets. If there is a new reasoning dataset, which architecture in Table 1 should people use? Testing all five is very inefficient.

**Style, Typos etc.**
1. In Section 4,1, it is mentioned that “the inverse relation of “father” can be “son” or “daughter”, we accept both”. What does “accept both” mean here? “Ann is the daughter of Brett” and “Ann is the son of Brett” can not be both correct. Right?
2. Also in Section 4.1, “We generated 390,000 examples that were roughly split 70/30 between training and testing”. But in Section 4.1.1, the size of test set becomes 10,000. Which one is correct?
3. The lvl.2 output of CLUTTR in Appendix B does not use the answer template specified in Section 4.
4. TACRED has been mentioned multiple times in the Appendix. How is TACRED used in this work?
summary_of_the_review: This paper tries to address an important problem, but the experiments conducted show that the proposed methods are not scalable to more real-world datasets. I do not recommend acceptance of the paper in its current form.
review_title: null

Meta-review: This is a clearly written paper about integration of entity abstraction to the transformer based language modeling methods for language processing tasks that require reasoning (this is clarified by the authors later as tasks that require linger chains of reasoning) and have shown results on CLUTTR, HotpotQA, and CoQA. The reviewers seem to agree on two issues: First, it is not clear why the proposed idea does not result in a lot of improvement, except the synthetic CLUTTR. Authors provided additional experimental results on yet another dataset. Second, the paper would benefit from a detailed analysis of the experimental results, for example, why don't abstractions help on all datasets.
Decision: Reject