Towards Abstractive Knowledge Representations in LLMs
Authors: Gaurav Arora, Shreya Jain, Vaibhav Saxena, Srujana Merugu


Reviewer paper summaries: The study investigated LLMs' ability in multilingual reasoning based on two proposed aspects: Reasoning by "Equivalence" and Reasoning by "Inheritance".

- Reasoning by Equivalence: The paper assesses LLMs' ability to reason across languages by evaluating if the models can provide consistent answers to the same questions posed in different languages. The study introduces novel tasks and benchmark datasets for evaluating this ability and finds significant inconsistencies in the answers provided by SOTA LLMs across languages.

- Reasoning by Inheritance: The paper also explores how well LLMs can handle inheritance relationships, which are fundamental for deductive and inductive reasoning. The findings reveal that LLMs frequently violate inheritance constraints, particularly in Hindi, with violation rates up to 37.2%.

- Compositional Representations: The authors also propose a novel method of constructing compositional representations to address the investigated issues, where tokens are represented as a composition of equivalent tokens across languages. This approach aims to enhance knowledge sharing and reduce conflicts.
Summary of strengths: 1. The study provides detailed analyese on multilingual reasoning, and offers interesting finds of the crosslingual discrepancy in reasoning performances from LLMs. The identification of significant inconsistencies in reasoning across languages and the frequent violation of inheritance constraints highlight important areas for improvement in LLM development.
2. The study also introduced new evaluation benchmarks for reasoning by equivalence and inheritance across six languages. The two aspects of reasoning ability provides insights for multilingual research.
3. The paper presents a novel method for constructing compositional representations, which shows promising results in reducing conflicts and improving knowledge sharing across languages.
Summary of weaknesses: 1. **Overclaiming:** Reasoning is a broad concept of ability that can be characterized in multiple ways and categorized into diverse types (e.g., inductive/deductive reasoning, numerical/logical/commonsense/spatial reasoning). Therefore,  there is a lack of a well-established theoretical and empirical background for the proposal of the two concepts (Reasoning by "Equivalence" and Reasoning by "Inheritance") *across languages* to encompass every aspect of reasoning, which the authors seemingly intend to suggest when describing their motivation (e.g., in Introduction). In the early sections, the authors describe reasoning in a very general framework, but their experiments focused only on multilingual settings of factual questions on well known named entities.

2. **Potential Performance Drop:** The CoRe augments the attention mechanism with proximal token representations selected across languages. It is reported that this reduces the cross-lingual conflict rate. However, I am concerned if this modification downgrades the model's original performance in the reasoning task and other domains, considering the delicacy of the transformer attention mechanism in pretrained LLMs. It looks like the conflict rate between languages is the only metric that the paper focuses on, while the goal of multilingual reasoning research should be achieving a robustly powerful reasoning ability across languages. There lacks additional experimental results and discussions regarding this, which harms the reliability and robustness of the method.
Comments, suggestions and typos: 1. The title and some of the claims in the paper (which stress reasoning in general) are overly broad compared to what the experiments truly focused on (reasoning across languages).
2. I am not entirely sure how the step 1 of CoRe is implemented: how are top-n proximal tokens *across each language* selected based on compatibility for a given token? Do you select the proximal tokens for every token across all languages in the entire vocabulary? What is the size of the smaller candidate pool of tokens per language that is chosen? I would appreciate it if more details about the method and experiments are included in the main paper or Appendix.
3. The authors could consider adding figures in PDF format to ensure higher clarity.
Overall assessment scores: 3