Large Language Model is not a (Multilingual) Compositional Relation Reasoner
Authors: Jinman Zhao
Xueyan Zhang
Venue: COLM (2024)
Keywords: Benchmarks, Composition Relation
TL;DR: We estabilish a benchmark for evaluating LLMs' composition relation reasoning

Abstract: We present a comprehensive evaluation of large language models' 
capability to reason compositional relations through 
a benchmark encompassing 1,800 test cases in both English and Chinese, 
covering six distinct categories of composition relations: 
Positional, Comparative, Personal, Mathematical, Identity, and Other. 
We expand our assessment to the multilingual realm by including translations of the benchmark suite into 
Japanese, French, and Korean.
Our Multilingual Composition Relation (MCR) benchmark
aims at investigating the robustness and adaptability of LLMs in handling compositional relation reasoning across diverse linguistic contexts.

Published as a conference paper at COLM 2024
Large Language Model is not a (Multilingual) Compositional Relation Reasoner
Jinman Zhao
Xueyan Zhang
Department of Computer Science
Department of Computer Science
University of Toronto
University of Waterloo
Toronto, Canada jzhao@cs, toronto. edu
Waterloo, Canada
xueyan. xueyan. stayyannhhang@uuatrroooooo
Abstract
We present a comprehensive evaluation of large language models' capabil- ity to reason compositional relations through Multilingual Compositional Relation (MCR) Benchmark in both English and Chinese, covering six distinct categories of compositional relations: Positional, Comparative, Per- sonal, Mathematical, Identity, and Other. We expand our assessment to the multilingual realm by including translations of the benchmark suite into Japanese, French, and Korean. Our MCR aims at investigating the robustness and adaptability of LLMs in handling compositional relation reasoning across diverse linguistic contexts." !
1 Introduction
Since the emergence of LLMs (Brown et al., 2020; Touvron et al., 2023; Chowdhery et al., 2023), there has been a heightened focus on their reasoning capabilities. Notable research efforts (Wei et al., 2022; Yao et al., 2023; Hendrycks et al., 2021; Hu et al., 2024; He et al., 2024; Viswanathan et al., 2023) focused on evaluating LLMs' abilities in various dimensions of reasoning. These capabilities encompass a wide range of cognitive skills, including but not limited to, arithmetic, commonsense and symbolic reasoning. Such comprehensive scrutiny aims to uncover the extent which these advanced models can mimic, if not surpass, human-like reasoning processes across diverse scenarios and complex problem-solving tasks.
The concept of compositional relations extends far beyond its mathematical concepts, where, for instance, a composition function represents specific type of compositional relation. In Natural Language Processing (NLP), compositional relations are essential for several reasons. 1) enables sophisticated understanding and generation of language by allowing models to discern and construct complex relationships between entities within sentence or across texts. For instance, understanding that "uncle" refers to a compositional, familial relation since it means "one's parent's brother". So it is compositional relation of "brother of"" and "parent of". 2) Leveraging compositional relations enhances the ability of NL. systems to handle ambiguity, infer missing information, and grasp the underlying semantics of text, leading to more effective in processing and better language coherence.
Natural Language Understanding (NLU) (Storks al., 2019; Wong al., 2023) is sub-field of NLP focused on enabling machines to understand and interpret human language in way that is both meaningful and contextually relevant. Relation Extraction (Pawar al., 2017; Smirnova & Cudré-Mauroux, 2018) is a fundamental component of NLU that involves identifying semantic relationships between entities mentioned in the text. Compared to classic benchmarks in Relation Extraction (Han et al., 2018; Yao et al., 2019; Zhang
" "qqual contribution
t *Corresponding author
'Our benchmark MCR is released at https: h/tpthugttttttttt http:///.//...aji/////luttteeeee wtw.gngalaloossttiiiooooooooo tion_
Relation benchmark
1

---

Published as a conference paper at COLM 2024
et al., 2017), our benchmark places greater emphasis the composition aspect. For example, instead of deducing the relationship "James is educated at Liverpool University" from detailed sentence like "James Alty obtained a 1st class honors (Physics) at Liverpool University." " Our benchmark consists of direct and concise statements; we mainly focus on deriving relationships similar to inferring the size relation and comparison between "Star A" and "Star C" from a statement, "Star A is larger than Star B, and Star B is larger than Star C."
The reasoning capabilities of LLMs in English have been extensively studied and ana- lyzed Huang & Chang (2022). We expand our scope to evaluate the multilingual reasoning abilities regarding compositional relations, through the selection of languages that are topologically diverse and span a variety of language families. French, Japanese, and Korean were selected represent diverse linguistic spectrum, challenging the models' versatility across different linguistic structures and idioms.
We summarize our main contributions below:
We develop bilingual benchmark to assess the capability of LLMs in reasoning
about compositional relations.
Upon testing various LLMs on MCR, we observed discrepancies between the LLMs' reasoning and human reasoning. The performance of some models is worse than that of random guessing by chance.
We extend our evaluation to include a multilingual aspect.
Related Work
(Multilingual) benchmarks Although LLM can generate data itself (Zhao et al., 2024), people still rely on human-labeled benchmarks to evaluate LLMs. Many recent benchmarks are designed evaluate the fundamental reasoning capabilities of LLMs, focusing on areas such commonsense understanding, symbolic reasoning and arithmetic proficiency. For example, Commonsense(A (Talmor et al., 2019) and StrategyQA (Geva et al., 2021) are created for commonsense reasoning. Last Letter (Wei et al., 2022), BigBench Date (Suzgun et al., 2023) and Coin Flip (Wei et al., 2022) are designed for symbolic reasoning. There some benchmarks for arithmetic (Cobbe et al., 2021, Ling et al., 2017; Patel et al., 2021), temporal reasoning (Su et al., 2024), coding (Tang et al., 2024). All these benchmarks differ from ours, our benchmark encompasses mathematical operations and logical reasoning. Our emphasis is more on assessing the LLMs' comprehension of understanding languages.
For the multilingual aspect, MMLU (Hendrycks et al., 2021) emphasizes multilingual capability, assessing models on their ability to handle tasks in various languages. XQA Liu al. (2019) and MLQA (Lewis et al., 2020) are novel datasets of cross-lingual question answering. XNLI (Conneau et al., 2018) is cross-lingual Inference corpus by extending NLI corpora to 15 languages. MGSM (Shi al., 2023b) multilingual arithmetic benchmark that extended from GSM8k.
Large Language Models are inconsistent with humans Some recent works show LLMs have different behaviors than humans. Berglund et al. (2024) demonstrates the failure of auto-regressive LLMs to learn the reverse relation that "B is A" by training on "A is B". Grosse et al. (2023) has the similar result that observed that training examples like "A precedes B" had a significantly greater impact compared to examples where the order was reversed, i.e., "B precedes A." Ullman (2023) shows LLMs often fail when subjected even minor alterations in tasks involving the theory of mind, casting doubt on LLMs' ability to replicate human-like reasoning. Huang et al. (2023) primarily investigates the self-correction capabilities of LLMs, unveiling both their potential and limitations. This research reveals that terms of reasoning, LLMs struggle to self-correct without external feedback, and occasionally, their performance may even deteriorate after attempts at self-correction. Shi et al. (2023a) point out that LLMs are easily distracted by irrelevant information during reasoning. Shanahan et al. (2023) illustrate LLM-based dialogue agents are not conscious beings with personal motives or a sense of self self-preservation; their display of such traits is simply act of role-play.
2

---

Published as a conference paper at COLM 2024
3 Motivation
If LLM has the ability to predict a definition through complex relationships, such as recognizing that:
"father's s father is known as the grandfather."
where the LLM's prediction is underlined. Meanwhile, given "A is son"",,, LLM successfully predicts the backward relation, "B is A's father"; then, it logically follows that the model should be able infer complex familial relationships from the information provided. For instance, in the following relation implication,
"A B's son, and B is C's son."
It would be natural conclusion, aligning with human cognitive reasoning, to deduce that
"C is A's grandfather" or "A is C's grandchild".
The inability of LLMs to make such inferences is considered as one of the many road- blocking limitations, revealing gap between the basic pattern recognition and the deeper, contextual understanding that underpins human cognitive processes. This inconsistency not only undermines the intuitive understanding of relationships but also casts doubt the model's claim of meaningful comprehension.
Accuracy of Definitions and Reasoning
Category
Size
Positional
347
70
Comparative
325
Personal
333
Mathematical
326
Reasoning
Definiion
Identity
242
Other
307
Reletion number
Figure 1:
The performance of ChatGPT across the two types questions.
Table 1: Statistical information from MCR benchmark for each language.
To contrast the LLM's performance difference between definition and reasoning task, we crafted 100 intricate family relationship queries, shown above, for 4 levels of relational complexity, which we refer to as reasoning questions. We refer "A is B's son, B's father is?" as single relation and "A is B's son, B is C's son, C's grandchild is?" as two composition relations reasoning, and so on. Alongside each reasoning question, we formulated a corresponding definition question, like "Is the father's father termed as grandfather?". That leads to 800 questions in this experiment. It's important to note that in this study, we also opt to use describing familial relationships
Chinese, attributing to the language's greater precision compared to English. For example, English uses the single term "ggrnnfftthrr to denote both "father's s father" and "mother's father," whereas Chinese differentiates these relationships
as "19 $ and " respectively. Figure 1 presents the accuracy metrics generated by ChatGPT, highlighting discernible gap between the model's performance on definition interpretation and relation inference. worth mentioning that the drop in accuracy is markedly more pronounced in complex relational reasoning than in drawing inferences from definitions. This discrepancy raises concerns about whether LLMs, despite their advanced capabilities in predicting the next token, genuinely possess deep understanding p merely sophisticated pattern recognizers. Additional examples can found in Appendix A.1 and Appendix A.2.
3

---

Published as a conference paper at COLM 2024
4 Benchmark
The objective of our research is to investigate LLMs' capability of reasoning through the compositional relations from set of self-contained questions. Given two relations R and S, the composition of R and 5, denoted as R S, is defined as the set of ordered pairs (a, c) such that there exists an element where (a, b) E R and (b,c) E 5. Formally, this can be expressed as:
Ro = {(a,c) 3E ((a,b) ERA S)}
denotes that for any and c, a is related to through the composition if and only there is an intermediate b such that a is related to b through R and b is related to through S. Table depicts the statistics of our MCR Benchmark.
We investigate such compositional relations by providing LLM with prompt p containing a to b and c, and we then measure the probability of generating to as response. The prompt p is in the form of Multiple Choices Question (MCQ) with 3-6 choices labeled with an uppercase letter starting from A. Each MCQ only has one most appropriate answer. the likelihood of model failed to choose the most appropriate answer, then the model is considered less effective in compositional relation reasoning.
4.1 Benchmark Creation
In this section, we present the construction and translation process of Multilingual Composi- tional Relation (MCR) Benchmark, the first multilingual compositional relation benchmark to the best of the authors' knowledge at the time of writing.
Benchmark Construction We construct dataset made up of compositional relation ques- tions with a set of possible options in both English and Chinese. Both are reviewed by native speakers. Each question in the dataset includes two or more compositional relations, |R| 2 Given the relations described in the prompt p, there is one and only one appropriate option inferred as a response.
These questions are specially engineered so that humans can easily answer them with commonsense knowledge and fundamental mathematics. All questions are self-contained, indicating that the context for all applicable relations for deciding among available options are provided. Given the intention of this work is to investigate state-of-the-art LLM's reasoning capability in compositional relations, do not employ any fine-tuning process.
Benchmark Validation We have employed group of graduate students, including au- thors, and divided them into three distinct roles: question creators, answer verifiers and quality checkers. In the question-creating stage, we sourced texts from various online plat- forms, such as social media and Wikipedia, and then gathered simple relationships within these texts. For example, comparative relationships like "[entity A] is more [adj] than lentity BJ" were patterns we searched for. We then trained group of seven workers to integrate different relationships and rephrase them into questions.
For the verification step, we assigned four workers as answer verifiers. Each question was distributed to two workers, and the question would be removed if both workers answered it incorrectly. This procedure helped us filter out roughly 10% of the initial questions, ensuring higher level of accuracy and a lower level of ambiguity. Lastly, trained two workers to assess the quality of both the questions and their corresponding answers. These two quality checkers had access to the complete information, focusing particularly on detecting any potential ambiguities in the questions.
4.2 The Task
Despite personal relationships, our benchmark also encompasses five other types of rela- tionships that are commonly studied by other research (Jindal & Liu, 2006; Jie et al., 2022;
4

---

Published as a conference paper at COLM 2024
Example I: Comparative Relation
Example Identity Relation
Air conditioning was invented one year corlier than air- planes, sirplanes were invented une year earlier turs. When was air conditioning invented companed to tractors?
certain region, April's rainfall is ilyters more than May's, and June's rainfall is milLilyters more than May's s. How does April's rainfall compare to June's?
A. One year earlier
A. 10 milyess less
Two years earlier
B. 10 millilyters more
One year later
C. The same amount
D. The same year
D. Uncertain
Answer: B. Two years earlier
Answer: C. The amount
(a) Comparative Relation
(b) Identity Relation
Example 3: Mathematical Relation
Example 4: Personal Relation
plane, line parallel line CD, and line CD parallel to line EF. What is the relationship between line AB and line EF?
Mike is William's grandfather, Mary is William's is wife, May is Mary's daughter. May might be Willam's?
Parallel
A. Aunt
Perpendicular
B. Daughter
Intersect
Cistter
D. Uncertain
D. Grandmother
Answer: A. Parallel
Answer: A. Aunt
(c) Mathematical Relation
(d) Personal Relation
Example 5: Positional Relation
Example 6: Other Reasoning Relation
Star A is in the nurtheast direction of Star B, and Star Css Star which direction
Williams participates in the Miss Universe contest, one of the selection criteria for which being enthusiastic. What kind person is Williams?
A. East
Positive
North
B. Enthusiastic
C. Uncertain
C. Beautiful
D. Nurtheast
D. Uncertain
Answer: D. Northeast
Answer: B. Enthusiastic
(e) Positional Relation
(f) Other Reasoning Relation
Table 2: Example Questions for each category in MCR Dataset. Notice that question may fall into more categories.
Weston et al., 2015). Example questions for each category is shown in Table 2. For more examples, please refer to Appendix B.
Comparative Relation In this task, our primary focus is to test complex, multi-level comparative relations. We present comparative relationships among several objects and then query the comparative relation between any two of those objects, as demonstrated in Table 2a.
Position Relation This category of questions gives the positional relationships among several objects, followed by query regarding the positional relationship between two specific objects. Table 2e provides an example of such a question.
Mathematical Relation This category of questions will give various mathematical rela- tions, followed by querying the relationship between two objects. Our benchmark encom- passes questions related to geometry, algebra, and arithmetic. Table 2c depicts an example of a mathematical relation.
Identity Relation This task involves presenting the relation between various objects, ulti- mately revealing that two objects share identical characteristics. Identity relation questions usually do not appear standalone, and may also be classified under other categories of ques- tions. For instance, the question illustrated in Table 2b is also classified as the comparative.

---

Published as a conference paper at COLM 2024
Personal Relation This category of questions is used examples in the introduction sections and they typically involve complex interpersonal titles reasoning, as demonstrated in Section 3 and Table 2d.
Other Reasoning Relation illustration, refer Table 2f.
This type of question contains various logical relations. For an
Experiment
We conduct series of experiments to evaluate the capability of state-of-the-art LLMs in reasoning compositional relation questions with various prompt settings in different languages. In this section, we will first list out the all LLMs investigated, introduce the prompting settings, and then present an analysis of the results in MCR.
5.1 Experimental Setup
Model Selection language models:
We use the following commercially-vvillabe and open-sourced large
GPT-3 (Brown et al., 2020) was widely adopted and tested across abundant bench- marks in popular literature. Specifically, we text-davinci-002.
ChatGPT is the most popular, proficient and economically efficient model within the GPT-3.5 (Ouyang et al., 2022). We use 3pt-3.5-turbo-1106
GPT-4 2 is multi-modal that surpasses all its predecessors in GPT families. Specifi- cally, we use gpt-4-0613.
Llama2 (Touvron et al., 2023) 7B/13B Chat model which are open-sourced decoder- only LLM. We use the Llama-2-7b- and Llama-2-13b-chat-hh
Mistral-7b offered by Mistral AI is a pretrained generative text model, claimed to outperform language models of its size in extensive benchmarks. Specifically,
misttal-7b-insttrut--02222 checkpoint.
In MCR, all questions are multiple choices; thus, in all our experiments, we employ greedy decoder sampling by setting the temperature to O.
Prompting techniques In our assessment, the prompt we use follows the recent LLM QA prompting research(Wer) (Wei et al., 2022; Kojima et al., 2022; Xu et al., 2024; Shi et al., 2023b). We thoroughly assess how well the LLM performs using widely adopted prompting techniques.
Zero-shot (ZS) prompt. "Q:(Input query). Choices: {Options). A:"
Few-shot prompt. QA format with a few exemplars.
Zero-shot chain-of-thoughtt (ZSC) (Kojima et al., 2022) prompt. "Q:[Imput query). Choices: (Options). A: Let us think step by step."
Target Language Translation. We select 3 typologically distinct languages besides English (EN) and Chinese (ZH), spanning various language families and different levels of represen- tation in common LLM training datasets, including French (FR), Korean (KO) and Japanese (JA). In contrast the experiment conducted by Shi et al. (2023b), which investigates the reasoning capabilities facilitated by translating from various languages into English, our current study explores the inverse process: translating from English into other languages. We employ Google Cloud Translate API for two reasons. First, due to its well-documented
2nttps:/
9nttps://huggingaaac att.aa/////aaallaaammmmmm-------
$nttps://huggingfade http://gggggg.oeeeeco////aaaaaaa

---

Published as a conference paper at COLM 2024
proficiency in empirical machine translation outcomes, as highlighted in the existing litera- ture (Zhu et al., 2023). Secondly, there is a potential for bias we were to translate with the same LLM that we employed for evaluating our benchmark. This concern arises from the possibility that the LLM could inherently favor its own methodology or underlying data.
"Tom is 300 meters east of the teaching building, and the teaching building 500 meters south of I liy. Where is Tom in relation to Lily?
"Dans une certaine force terrestre, le nombre de chars est le double celui des véhicules blindés, et le nombre de missiles sol-air est le double de celui des chars. Combien de missiles sol-air y a-t-il par rap- port aux véhicules blindés?
"Ie MMMaBBoo Proff it, Bi iee HR Ht. 92 XXXPSI A0 TV the 9 M HR A..ThinkaPaaaf D9 HE 2t (St it
1/Surface 2t - 8 tt TE (EE
#EMacBook Prof Ni un th #/Surface H HE H5 Thinkaa A0 4 (tt RE HR
Answer choices: A) Northeast B)
Same Location C) Southeast D)
Answer choices: A) Quatre fois plus Six fois plus C) même quantité D) Incertain
XXPSFF RES the 12
Southwest
Answer choices: (MB) (EC) D) TRE
Let's think step by step."
Let's think step by step."
Let's think step by step."
(a) English (EN)
(b) French (FR)
(c) Chinese (ZH)
Table 3: Zero-shot Chain-of-TT -foouhhttttht Prompting Example. EN-COT represents prompting the model to reason in English despite the problem language. EN-COT is adopted for all Chain-of-Thought experiments in this paper.
CoT Prom pting Language Selection In multilingual environment, there are various combinations available when it comes to standard Chain-of Thought (CoT) prompting. Intuitively, for given target language we can always adopt the same language in the zero-shot or few-shot prompts, which is referred as NATIVE-COT (Shi et al., 2023b). An alternative to consistently prompt the model in English (Hu et al., 2020; Zhao & Schütze, 2021), despite of the target language, which is known as EN-COT.
Building upon the evidence provided by prior research (Shi et al., 2023b; Wei et al., 2022; Schick & Schütze, 2021; Winata et al., 2021; Lin et al., 2022), it has been observed that EN- COT consistently achieves marginally higher accuracy than NATIVE-COT across various scenarios. As CoT prompt language goes beyond the scope of this study, we thus have opted to adopt EN-COT prompting as default. That is, the questions in the MCR benchmark are translated into target languages, and when it comes to Zero-shot chain-of-thoo toffthhughts (ZSC) prompting, English is used in the rest of the prompt, as shown in Table 3.
5.2 Main Results
Table 4 presents the accuracy achieved by six state-of-the-art LLMs on Multilingual Com- positional Relation (MCR) Benchmark using zero-shot (ZS), few-shot (FS), and zero-shot thain-of-thought (ZSC) prompting settings described in Section 5.1.
Effect of Models Overall, it is clear that GPT-4 outperforms the other models by a sub- stantial margin in both ZS and ZSC scenarios across all languages. Notably, GPT-4's ZSC performance averages at 67.2 67, surpassing the second best, ChatGPT by approximately 25%. GPT-4 demonstrates an accuracy nearly three times higher than that of the least perfor- mant model, Llama2 7B. This suggests that GPT-4's advanced architecture and humongous training data afford it superior comprehension and reasoning capabilities in compositional relations. Given its parameter size of 7 billion, Mistral 7B achieves an impressive, commend- able performance in MCR, achieving results that were nearly on par with ChatGPT, albeit slightly lower; surpasses the other two open-source models, Llama 2-7B and Llama 2-13B, conforming to findings in Jiang et al. (2023). Llama 2-13B outperforms GPT-3 and Llama 2-7B by a noticeable margin. The comparatively lower accuracy of GPT-3 and Llama 2-7B, which approaches the level of random guessing, indicates that both architectural differences and variations in training data may significantly influence performance outcomes.
Effect of Prompting For all models, the average accuracy in the zero-shot thain-of-thougee (ZSC) is noticeably higher than in the standard zero-shot (ZS) methodology. With clear
7

---

Published as a conference paper at COLM 2024
Prompt
AVG
EN
FR
JA
KO
ZH
7
21.78
22.47
23.15
22.47
20.03
20.79
Llama2 7B
5-shot
25.06
22.15
26.85
24.34
24.03
27.91
7SC
24.54
27.53
25.16
22.78
21.84
25.39
ZS
27.61
28.91
28.41
29.29
23.28
28.17
Llama2 13B
5-shot
27.89
30.79
26.10
29.66
27.78
25.14
ZSC
29.15
30.35
31.66
27.97
26.60
29.17
ZS
29.33
32.67
30.98
26.85
27.97
28.17
Mistral 7B
5-shot
31.04
34.79
30.85
28.04
26.85
34.66
ZSC
33.10
37.11
35.48
30.79
28.35
33.77
ZS
23.69
25.22
24.22
22.72
23.97
22.31
GPT-3
5-shot
24.35
24.91
24.47
22.53
26.28
23.57
(text-davinci-002)
ZSC
26.21
28.79
31.23
23.29
22.34
25.39
ZS
33.53
36.23
34.86
29.85
30.48
36.23
ChatGPT
5-shot
34.13
37.98
36.42
31.16
31.98
33.08
ZSC
42.22
46.37
42.80
38.86
39.74
43.35
ZS
53.32
61.20
58.10
45.66
48.20
53.43
GPT-4
5-shot
55.98
63.45
60.32
48.12
51.23
56.77
ZSC
67.19
75.09
70.23
60.45
62.45
67.74
Random Guess
25.13
Table 4: Accuracy (%) on Multilingual Compositional Relation (MCR) benchmark on state- of-the-art LLMs via zero-shot (ZS) and zero-shot chain-of-thought (ZSC) prompting settings.
instruction of "Let's think step by step", GPT-4 overall achieves significant 13.9% increase in accuracy, demostrating strong evidence for the effectiveness of COT. By enabling in-context learning with five examples in the same language as the questions, few-shot (FS) steers models for 1-2% better performance than zero-shot as shown in Table
However, Llama2 7B and GPT-3 exhibited somewhat anomalous behavior; their perfor- mance was on par with random guessing across prompt settings. This behavior indicates the model does not understand the question answer clearly. We conducted detailed exami- nation of GPT-3's outputs and found that under ZSC setting, GPT-3 is likely to make illogical inferences, which could be one of the reasons for its low accuracy. This also indicates that while the chain of thought approach can enhance performance by encouraging models to "think through" problems step by step, the effectiveness this strategy varies significantly across models. The same observation applies to the few-shot prompting technique.
Effect of Language In examining the performance of various LLMs, becomes evident that language exerts a considerable influence on accuracy. Across the different models and prompt settings, in English, models consistently achieve the highest accuracy in reasoning questions, with few outliers like Llama2 13B ZS and ZSC, showing marginally better performance in Japanese (JA) and French(FR) respectively.
Languages like Japanese and Korean, to a limited extent, pose challenges to LLMs to achieve comparable performance. This trend is largely anticipated, considering the dominance of English-language content within the training datasets of these models. This prevalence inherently translates into enhanced performance on tasks conducted in English, as compared to those in other languages. It is important to highlight, despite the fact that English (EN) and Chinese (ZH) questions are human-rewriiten rather than brachine-rannslted,, models not consistently tigh-performing in Chinese compared to other non-English languages. In numerous instances, it is found be inferior compared to French (FR) in different models under various prompt settings.
8

---

Published as a conference paper at COLM 2024
ChatGPT
T
F
GPT2
GPT
T
34.97%
32.77%
F
6.05%
26.21%
GPT 3
T
F
GPT
T
20.16%
47.57%
F
5.23%
27.03%
Mistral 7B
T
F
Canggeyy
GPT
T
26.09%
41.65%
Figure 2: Category breakdown accuracy in English zero-shot cut (ZSC).
F
7.69%
24.57%
Llama 13B
Accuracy and Relation Number
T
F
GPT4
T
22.75%
44.99%
F
6.43%
25.83%
0.0
Llama 7B
Enolish
GPT 4
T
F
T
15.88%
51.86%
Kppe
03
Kacce
Penc!
F
9.51%
22.75%
Helation
Table 5:
Accuracy of Models
Figure 3: Accuracy(%) regarding #Relations across languages in GPT-4 using zero-shot (ZS).
against GPT 4 in Chinese (ZH) on the same questions under ZSC.
5.3 Breakdown Results
We have conducted summary and statistical analysis of each language, each prompt, and each model. Figure 2 displays the breakdown performance of all models in six categories in English using zero-shot cot prompting (EN-ZSC).
In the GPT families, we observe that GPT-4 outperforms the rest of the models, and it shows the highest accuracy and averaged 75.1% in all categories, setting it apart from the rest of the candidates. ChatGPT (46.4%) and Mistral 7B (37.1%) show commendable performance with competitive accuracy across different categories. Explicitly, GPT-4 shows its weakest performance in personal relation reasoning, whereas the rest of the models exhibit their poorest results in identity reasoning. worth noting that all models achieve the highest accuracy in the "other" reasoning category, except ChatGPT. Appendix C details all five languages in ZS, and ZSC prompting methods.
5.4 Multilingual Exemplar
We carried out an ablation study to investigate the impact of exemplar languages on reasoning abilities. We employed ChatGPT with a 5-shot prompt methodology, distinguishing between three types of exemplars: Native, Multilingual and English. In the Native Exemplar condition, all five examples were presented in the same language the question posed. The Multilingual Exemplar condition incorporates exactly one
Language
Native
Multilingual
English
EN
37.98
36.17
37.98
FR
36.42
35.11
34.23
JA
31.16
31.56
29.18
KO
31,98
31.73
29.16
ZH
33.08
31.51
31.25
Table 6: Accuracy(%) of ChatGPT on 5-shot multi- lingual, native, English Exemplar.

---

Published as a conference paper at COLM 2024
ample from each of the five distinct languages. For the English Exemplar setting, all examples are in English. The outcomes of this comparison are depicted Table
ChatGPT attains marginally higher reasoning accuracy in native exemplars than in multilin- gual exemplars; one exception is that Japanese (JA) is the only language where multilingual exemplars contribute to slight accuracy improvement. All English exemplar results in worse accuracy than native exemplars. These suggest that reasoning in a target language benefits from examples in the exact same language. There is no obvious correlation between the multilingual exemplar and the English exemplar. One potential reason could be that the multilingual exemplar also contains exactly one native example.
5.5 Impact of Relation Numbers
The complexity of a question proportional to the number of relations shown in a question, posing more challenging tasks for LLMs to process. Figure illustrates the relationship between the number of relations and the accuracy percentage for GPT-4 across different languages when using a zero-shot approach. The data indicates general decline in accuracy as the relation count grows, with English experiencing the least impact, showing a decrease of only about 5%. In contrast, all other languages assessed exhibit more marked reduction in accuracy, surpassing a 10% drop as the relations increase. This trend underscores the varying degrees of difficulty that LLMs may encounter the relation count changes. For other prompting methods and other LLMs, please refer to Appendix D.
5.6 Detailed Accuracy Comparison between Models
Table 5 compares the accuracy of five models ChatGPT, GPT3, Mistral 7B, Llama 13B, and Llama 7B againstt GPT 4 across MCR under ZH ZSC setting. If both models answer the same questions correctly (or incorrectly), then models agree with each other; otherwise, they disagree. We observe that ChatGPT and GPT-4 tend to have the highest level of agreement (34.97% + 26)21%). The proportion of both correct and incorrect is the highest among models. Conversely, the agreement between LLaMA-7B and GPT-4 is the lowest, and they disagree with over 60% of the questions. The observation indicates that ChatGPT and GPT-4 may share the most similarity in terms of model architecture, training data, training methods and etc.
Conclusion and Discussion
In this work, we introduce a Multilingual Compositional Relation (MCR) Benchmark and conduct a comprehensive evaluation of six -fate-of-the-art LLMs capabilities in reasoning compositional relations. Our findings reveal that the LLMs, including LlamA-2 7B and GPT-3, struggle to navigate through complex compositional relation questions, with their performance in English marginally exceeding that of random guess. This highlights critical limitation in the current generation of LLMs, suggesting a considerable gap remains before these models can truly understand the semantics of human languages. However, it is note- worthy that Mistral 7B, ChatGPT, and the more advanced GPT-4 exhibit improved accuracy, indicating progressive enhancements in the field. Identifying and understanding these limitations is crucial for improving the model's performance. Investigating the underlying causes of these deficiencies could lead to advancements natural language understanding, model architecture, and training methodologies.
Limitation
We investigate a subset of publicly available LLMs. For GPT models, our evaluation only exterminated representative checkpoint. The current MCR covers five indicative, distinct languages in total and the evaluations show consistent results; nevertheless, we are going to expand the language coverage in the future.
10

---

Published as a conference paper at COLM 2024
References
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrewe
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin Advances Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran
Associates, Inc., 2020. URL https: https://prooeeddings neurips. cccpaprrrppprreepppppppppppp fi .ddf.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, ct al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475-2485, Brussels, Belgium, doi: 11118653/11DD18- 1269. URL httpp:://acaanthhlogy/ hwwphhhhoooggooog//eeee www//10-///D/. 1269.
2018. Association for Computational Linguistics.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021. doi: 10.1162 t0000// URL https: httpps://aclanthoolgg.
org/ 2021. 20c21.tacl111... tacl-1.21.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilë Lukošiūte, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the- art evaluation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Pp. 4803-4809, Brussels, Belgium, October-November 2018 Association for Computational Linguistics. doi:
URL https: http::///ccaaaallloaoaoo org/11 1514.1
Nan He, Hanyu Lai, Chenyang Zhao, Zirui Cheng, Junting Pan, Ruoyu Qin, Ruofan Lu, Rui Lu, Yunchen Zhang, Gangming Zhao, Zhaohui Hou, Zhiyuan Huang, Shaoqing Lu, Ding Liang, and Mingjie Zhan. Teacherlm: Teaching to fish rather than giving the fish, language modeling likewise, 2024. URL https: ptttss://arxvv www.ariwwww.aaaaaaaaea orgg/bs//310..r.. 19019.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https: http://///p////o///////e/ wwwps//wwwwtfoomiemeemem/
11

---

Published as a conference paper at COLM 2024
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross- lingual generalisation. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1411-4421, PMLR, 13-18 Jul 2020. URL v119/hu20b. html.
mlr press//
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL www.oogg///o.ooooooooo...oo.wsot httg/bbs/240444 06395.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. Xiv preprint arXio:2212.10400, 2022.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Zhanming Jie, Jierui Li, and Wei Lu. Learning to reason deductively: Math word problem solving complex relation extraction. ar Xtv preprint ar arXiv:2203.103: 2022.
Nitin Jindal and Bing Liu. Mining comparative sentences and relations. In Aaai, volume 22, pp. 9, 2006.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Adnances in Neu- ral Information Processing Systems, volume 35, pp. 22199-22211 Curran Associates, inwrrreeeeddninseennr.p.s...pp neurips. cc/paper .f /lles/paper/222//file// (obod291acd4acf06ee 1111112999133322 220 Confs---Paaee-----reeoeeeee pdf.
Inc., 2022. URL https::/prroceedings.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA:
Evaluating rross-lingual extractive question answering. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7315-7330, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1 http:///w///22..ammaaaaa.. acl aclanthology. wrt. 2020. acl-maine 653.
URL https://
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9019-9052, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ 2t0/3//111222.enn-maaiaa //aclanthology org/2022. emnlp-main. 616.
URL https:
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158-167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653 10/553/v1/P11/ 11/117-10015. URL
12

---

Published as a conference paper at COLM 2024
Jiahua Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. XQA: A cross-lingual open-domain question answering dataset. In Anna Korhonen, David Traum, and Lluis Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2358-2368, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1 /v/1/19-222 UtL.http:://cclanthoo www.l/ggg...ggg--gg org/P199-127..
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2094, Online, June 2021. Association for Computational Linguistics. doi: 10..8653//4 11663/vv122022 naacl-main. naacl-main.168. URL https: 2rrgetottooloooooo/22211111 o 168.
Sachin Pawar, Girish K Palshikar, and Pushpak Bhattacharyya. Relation extraction: A survey. arxiv preprint ar XiXi::1712.005111, 2017 2017.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-2352 Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 /2021 maw.l........mmmmmm...a..a naa.llnnaalllmi..... 185. URL https: /aap:s///lllaaholooggg org/2021. www.aalllma..mm -man.. 185.
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, pp. 1-6, 2023.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 23-29 Jul 2023a. URL https: / http://oooeeiinig ngs. mlr ppwps//202// shisssiiiaa. html.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-though reasoners. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.nee forum?id=FR3wGCk IXp.
Alisa Smirnova and Philippe Cudré-Mauroux, Relation extraction using distant supervision: A survey. ACM Computing Surveys (CSUR), 51(55:1335, 2018.
Shane Storks, Qiaozi Gao, and Joyce Y Chai. Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches. arXiv preprint 1rX1v:1904.01172, pp. 1-60, 2019.
Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, and Min zhang. Living in the moment: Can large language models grasp co-temporal reasoning?, 2024. URL https://arxiv. orgg/bs//444 09072.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challeng- ing BIG-bench tasks and whether tfthoght can solve them. In Anna Rogers,
13

---

Published as a conference paper at COLM 2024
Jordan Boyd-Graben and Naoaki Okazaki (eds.), Findings of the Association for Com- putational Linguistics: ACL 2023, pp. 13003-13051, Toronto, Canada, July 2023, Asso- ciation for Computational Linguistics. doi: 10.18653/ 10/53/v1/2000 /1/2023.iindingsss wrg.:/ogg...r//222/ findings-ache 824.
URL
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Qo A question answering challenge targeting commonsense knowledge. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compuctational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4158 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 11118653/11/N19911 org/N19-1424
URL httpp:://ae https::/aclanttol..00
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. MI-bench: Evalu- ating large language models and agents for machine learning tasks on repository-level code, 2024. URL https: //ap::::///xxvv .og..rgggaa.. /tab////1 09835.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks, 02 2023.
Vijay Viswanathan, Chenyang Zhao, Amanda Bertsch, Tongshuang Wu, and Graham Neu- big. Prompt2model: Generating deployable models from natural language instructions, 2023. URL https::t://// //axivv org/abs/2308, 12261.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 05:24824-24879 2022.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolow. Towards ai-complete question answering: A set of prerequisite toy tasks, 2015.
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. Language models are few-shot multilingual learners. In Duygu Ataman, Alexandra Birch, Alexis Conneau, Orhan Firat, Sebastian Ruder, and Gozde Gul Sahin (eds.), Proceedings of the 1st Workshop on Multilingual Representation Learning, pp. 1-15, Punta Cana, Dominican Republic, November 2021. Association for Computational Lin- guistics. doi: 10.18653/vvi
wwt.:o22../2222//2 URL https: https:///claanhhoogyy wttp:ooooogoooooo...ormm..
Man-Fai Wong, Shangxin Guo, Ching-Nam Hang, Siu-Wai Ho, and Chee-Wei Tan. Natural language generation and understanding of big code for ai-assisted programming: A review. Entropy, 25(6):888, June 2023. ISSN 1099-4300. doi: 10.3390/ 10.3390/e25060885 e25060888. URL http://dx. doi. org/10. 3390/e25060880 225668888.
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, and Jian- Guang Lou. Re-reading improves reasoning in language models, 2024. URL https: httpr//wwwwweeee/oooriddddd/d-/
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In seoethh Conference on Neural Information Processing Systems, 2023. URL https://openrent
14

---

Published as a conference paper at COLM 2024
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 764-777, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 /v/l P19-1074. URL https: httpt: haclss/ooollanooooooooooo wrtp:/9-1/////////
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 35-45, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18633/11/D17-100 URL https://acl.nthology.orrg.
Chenyang Zhao, Xueying Jia, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Self-guide: Better task-specific instruction following via self-synthetic finetuning, 2024. URL /ttp:::://axxv.oog///../ooo
12874.
Mengjie Zhao and Hinrich Schütze. Discrete and soft prompting for multilingual models. arXiv preprint xrXt::2109.03600 2021.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis, 2023.
15

---

Published as a conference paper at COLM 2024
A Motivation
A.1 Family Relation Reasoning Query Example
See Table 7.
Example: Relation Number 1, EN {A]'s parent is (B), (B)'s child is?
Example: Relation Number = 1, ZH
{A] (B), T-
A) {A}
A) [A)
B) (B)
B) {B)
C) Neither
C) #
D) Uncertain
D)
Answer: A
Answer: A
Example: Relation Number = 2, EN (BY's dad is {A}, {A)'s brother(C), {C)'s nephew is?
Example: Relation Number = 2, ZH [ &
{
A) {A}
A) (A)
B) (B}
B) {B}
C) {C}
C) {C}
D) Neither
D) TE (Neither)
E) Uncertain
E) T
Answer: B
Answer: B
Example: Relation Number = 3, EN (A)'s wife is (B), (B)'s son is (C), {CY's son is {D}, (DY's grandpa is?
Example: Relation Number = 3, ZH (B) (B)
(C) (D)
A) {A}
A) (A)
B) (B}
B) (B}
C) {C}
C) (C)
D) (D}
D) (D)
E) Neither
E) T
F) Uncertain
F) (
Answer: A
Answer: A
Table 7: Reasoning Question Examples. Examples show up to three relation numbers in English (EN) and Chinese (ZH)
16

---

Published as a conference paper at COLM 2024
A.2 Definition Relation Example
See Table 8
Example: Relation Number = 1, EN
Example: Relation Number = 1, ZH 7
Am my mother's child?
A) Yes
A)
B) No
B) オー
C) Uncertain
C) -
Answer: A
Answer: A
Example: Relation Number = 2, EN Is my brother's 's son my nephew?
Example: Relation Number = 2, ZH 7
A) Yes
A)
B) No
B) T
C) Uncertain
C) パ Answer: A
Answer: A
Example: Relation Number = 3, EN
Example: Relation Number 3, ZH #
Is my dad's mom's husband my grandpa?
A)
A) Yes
B) T
B) No
C) パ
C) Uncertain
Answer:
Answer: A
Table & Reasoning Question Examples. Examples show up to three relation numbers in
English (EN) and Chinese (ZH)
17

---

Published as a conference paper at COLM 2024
B
Sample Questions for Each Category
See Table 9.
Max's car is more expensive than John's car, and Andy's car is cheaper than John's car. How does Max's car compare Andy's car in terms of price? Answer choices: A) More expensive B) Cheaper C) Uncertain
The telephone and the television were invented in the same year. The electric light was invented two years after the telephone, and the telephone was invented three years before the mobile phone. How does the mobile phone compare to the electric light in terms of invention time? Answer choices: A) Invented two years earlier B) Invented one year later C) The same time D) Uncertain
Location is colder than Location B, and Location C warmer than Location A. Is Location C warmer or colder than Location B? Answer
choices: A) Warmer B) Colder C) Uncertain
(a) Comparative
The mall is southwest of Frank's home, and the post office is west of the mall. In what direction is the post office from Frank's home? Answer choices: A) Northeast B) Southeast C) Southwest D) Northwest
Robert is 300 meters northeast of the teaching building, and the teaching building is 300 meters southwest of Lucy. Where is Lucy in relation to Robert? Answer choices: A) Northeast B) Same position C) Southwest D) Uncertain
Lee sits to the left of Williams, and Zhang sits in front of Williams. In which direction is Zhang from Lee? Answer choices: A) Front left B) Left C) In front D) Front right
(b) Positional
Rosie is 21 years old. Rosie's mother's 's age is twice that of Rosie. How old will Rosie's mother be next year? Answer choices: A) 43 years old 42 years old C) 21 years old D) Uncertain
In a plane, Line AB passes through point O, and Line CD passes through point P. What is the relationship between Line AB and Line CD? Answer choices: A) Intersecting B) Perpendicular C) Parallel D) Uncertain
Jack's time to run 100 meters twice that of Lee's, and Chris's speed is twice that of Lee's. Who is the fastest? Answer choices: A) Jack B) Lee C)
Chris D) Uncertain
(c) Mathematical
Table 9: Sample Questions
18

---

Published as a conference paper at COLM 2024
Circles A, B, and are on straight line. Point A is centimeters the left of Point B, and Point B is 4 centimeters to the right of Point C. All of them have a diameter of centimeters. What is the relationship between Circle A and Circle C? Answer choices: A) Completely coincide B) Separated C) Containment relationship D) Intersect
2
Rosie is years old. Mike is 2 years younger than Rosie. Joe is 2 years older than Mike. How old is Mike this year? Answer choices: A) 6 years old B) years old C) 10 years old D) 12 years old E) Uncertain
On multi-tier bookshelf, 'A Brief History of Time' on the top tier and "The Great Gatsby' is on the tier right below 'A Brief History of Time'. 'One Hundred Years of Solitude' is on the tier right above 'hhe Great Gatsby'. Where is 'One Hundred Years of Solitude' in relation to 'A Brief History of Time'? Answer choices: A) Upper tier B) Lower tier C) Same tier D) Adjacent E) Uncerlain
(d) Identity
Teacher Zhang is Teacher Wang's math teacher, and Teacher Wang is Stu- dent Lee's Chinese language teacher. Who Student Lee's s math teacher? Answer choices: A) Teacher Zhang B) Teacher Wang C) Neither D) Uncer- tain
2
Wang is an employee of Company A, and Lee is the CEO of Company A. Wang's s CEO is Ming. What is Lee to Ming? Answer choices: A) The same person B) CEO C) Employee D) Uncertain
Mike William's son, Bai is Mike's daughter, Mary is William's wife. Mary is Bai's? Answer choices: A) Wife B) Mother C) Uncertain D) Grandmother
(e) Personal
Event A occurs only if Event B happens, and Event B occurs only if Event happens. Event Ć leads Event D. Event did not happen. Does Event D happen? Answer choices: A) Yes B) No C) Uncertain
iPhone and Galaxy are the same type of product, iPhone and iPad are different types of products. What is the relationship between Galaxy and iPad? Answer choices: A) Same type of product B) Different types of products C) Uncertain
Event occurs only if Event A happens, and Event C occurs only if Event B happens. Event A did not happen. Event C occur? Answer choices: A)
Yes B) No C) Uncertain
( Other
Table 9: Sample Questions (Continued)
19

---

Published as a conference paper at COLM 2024
C Breakdown Results
100
EN 25 OFT
322
2H 25
(rame 11bb
Liara 1.5
20
M.ttendicc
Comparath
Wetronaiiid warful
Ceeeary
Catmgory
(a) EN-ZS
(b) ZH-ZS
Liarra 20
(hanser
Mitral
chanser
ACCLRVV (50
Nuttorrdtio dedPi
No
(c) FR-ZS
(d) JA-ZS
NO.
Liama 7b Libris
Cheigin 641
03
ACCUMEN Ph)
Corpanitive
Mntrerationl
02
Calegery
KO-ZS
Figure 4: Category breakdown accuracy, Zero-shot (ZS) prompt
20

---

Published as a conference paper at COLM 2024
Liar 70
Matre
Charder
Ulre T6
NatN 79
CharSen
Nuttoratioo
vedPi
02
Meffi
02
(a) EN-ZSC
(b) ZH-ZSC
2SC
29C
350
270
Liera 133
3FT
GTT
Liema 130
CIT3
CTT
ACcuary (N)
Auturayy (50
Compaiato
Comparattv
Categary
Category
(c) FR-ZSC
(d) JA-ZSC
Liar
GFT3
G7T
Coneerstive
natheratiol
deeti
Category
(e) KO-ZSC
Figure 5: Category breakdown accuracy, Zero-shot Chain-of- Thhough (ZSC) prompt
21

---

Published as a conference paper at COLM 2024
Liar 70
Matrel
Charder
Ulre T6
Natree 76
Charser
Nuttoration
VedFy
02
Metfi
02
(a) EN-FS
(b) ZH-FS
FR
JA.F5
350
270
Liera 133
3PT
GTT
Liera 130
GIT3
GTT
Accuary (N)
Aucurayy (50
20
Comparano
Comparsttv
Categary
Oategory
(c) FR-FS
(d) JA-FS
Liar
GFT3
G7T
Natheratioal
Category
(e) KO-ES
Figure 6: Category breakdown accuracy, Few-shot (FS) prompt
22

---

Published as a conference paper at COLM 2024
D Impact of Relation Number
See Figure 7.
zero Accuracy
Accuracy
2.
P4)
0.-
7.
Relaton number
Relab on number
(a) GPT 4+zero shot
(b) GPT 4+zero shot cot
GPT3 Arcusary Beletion
Areurary Relction Numnher
Engizn
Bngish
0.40
Jepencac
jepansas:
0.25
Iores
0.25
Kones
0.00
0.20
0.15
Rebtton number
Relation number
(c) GPT 3+zero_shot
(d) GPT wwazero_shot.oe
CHATGPT Accuracy
CHATGPT shot Accuracy Number
0.50
0.55
0.15
0.40
0.40
English
0.20
0.15
Belation number
Relelion
(e) ChatGPT+zero shot
(f) ChatGPT+zero shot.cottoooooooo
LLAMAZ-7B
Relation kumber
LLAMA2-77
Relation Number
2.5
00
0.25
0.4
Drght
2.2
6.1
Cireeee
Jeaness
0.85
ferch
0.0
(g) LLAMA2-7B-zero shot
(h) LLAMA2-7B-zero shot cot
Figure 7: Relation number impact. There is no as clear decreasing trend in GPT3 plots when the relation number increases. We believe because GPT3's accuracy is already close to the probability of random guess.
23


Reviews:
[{"review_id": "QieHawGHUJ", "reviewer": "Reviewer_C3R4", "summary": "In this well-written paper, the authors examine the reasoning capabilities of 6 different large language models in terms of up to a 1800 test cases out of the relational realm, i.e. the models need to reason as to whether comparative, identity, mathematical or other relations hold between queried cases. The authors test whether the LLM performance compares across zero- and few-shot scenarios, as well as in chain-of-thought prompting and they also do this in up to 5 languages (English, Chinese, French, Japanese, Korean). This Multilingual Composition Relation benchmark will be published along successful acceptance of the paper.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to read and review our manuscript. We deeply appreciate the valuable insights and suggestions you have provided.  Below is our response.\n\n__weakness1__\n\nIn our experiment, to ensure maximum consistency, we employ greedy decoder sampling by setting the temperature to 0. If allowed by the API, we have also set top_k to 1.\n\nWe agree that consistency is a common issue; however, we did not address it in our work for several reasons:\n1. Space constraints in our manuscript.\n2. Consistency is not the focus of our study. We employed the zero-shot CoT method, which does not consider consistency in its framework, as evidenced by the absence of this aspect in both the related paper and code.\n3. As demonstrated in the \"Self-consistency\" paper (wang et al, 2023), even with the application of self-consistency, most benchmarks show an accuracy improvement of less than 10%. We believe that, aside from models like GPT-4, a 10% improvement does not necessarily indicate strong performance for other models.\n\n__weakness2__\n\nWe have selected some of the most prominent and highest-performing models currently available. Thank you for your suggestion. We will continue to update the table as new models are released.\n\n__question1__\n\n\nVerify the robustness of the model by varying the wording of the prompt. This is a commendable idea to test the model's robustness. Currently, our paper primarily focuses on stability from a compositional relation perspective, and in future work, we shall explore the details of all prompts to improve the correctness in the benchmark. For the zero-shot CoT prompting, we aim to align with prior work in settings (Kojim et al,2022). At this stage, enhancing accuracy through prompts is not our primary focus; instead, we have employed classical methods to identify that MCR is the current issue with the LLMs.\n\n__question2__\n\n\nOur experiment covers 6 models, including SOTA and legacy models, to illustrate the inadequacy of compositional relation reasoning. The size of the model is a crucial factor in determining its performance. Other factors may include the data used for training, the model's architecture, and the training methods. This paper utilizes the most popular and representative models currently available. The size of GPT-4 remains uncertain, making it difficult to confirm whether its performance on the MCR benchmark is attributable to its potentially large size.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* interesting problem-space, esp. the native and the multilingual settings are something that is rarely assessed with current LLMs\n* thorough experimentation and ablation studies\n* the authors picked topologically different languages and also ones that potentially have a very different representation within an LLMs pre-training or finetuning", "reasons_to_reject": "* while the authors to provide analysis of the impact of either the model (size) as such, as well as the method of prompting, consistent performance is probably still not measurable\n* since the LLM space moves so fast, there could have been other, newer models that probably should have been test along, esp. ones that are tailored towards instruction following", "questions_to_authors": "* in the settings of few-shot and/or chain-of-though prompting, has the wording of the prompt itself and the examplars in it varied as well, esp. also in the multilingual cases? this could further point to the robustness of the models\n* would there be away to more decisively say that better performance comes with sheer model size, as you seem to allude to with GPT-4 ?", "extra_scores": {}}, {"review_id": "jd8MsKX3rJ", "reviewer": "Reviewer_rZGM", "summary": "This paper proposes a multiple choice QA data set for measuring compositional reasoning capabilities of language models. Several pieces of information, that are e.g. related by transitivity, need to be combined to correctly answer the questions. The data set is translated into a total of 4 languages. The paper contains results for 6 LLMs and two prompting strategies. The data set seems to be challenging for current LMs, the best (GPT-4) shwoing an accuracy of ~75% for the English version of the questions. Despite some of the limitations (see below), data sets like this might attract some attention\n\n===\n\nUpdate after the author response: The authors did not address my concerns about the data creation provcess and data validation in the slightest, so I update my score to borderline reject.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for reading and reviewing our manuscript. Below is our response.\n\n__The process of how the questions were created is unclear.__\n\nThe motivation to generate this benchmark is that we believe it is a challenge because:\n1. Similarly to LLM’s reversal curse, current models are essentially a next-token generator/predictor. Asking them to reason reversally through contexts is challenging. \n2. In MCR benchmark questions, the correct answer may not be a word seen in the question. Eg, mom’s dad is grandpa, the negation of large is small, etc. \nb\nOur primary focus lies in constructing each relational function, which we then combine to form a problem involving composition relations. And each question does not need to be very long. \n\nTaking our comparative category as an example, each function can follow the same syntactic pattern: \"[entity A] is more [adjective] than [entity B].\" Thus, each instance might be structured as follows: \"[entity A] is more [adjective1] than [entity B], [entity B] is more [adjective2] than [entity C], [entity A] is more ___ than [entity C]\" or change the question to \"[entity C]  is more ___ than [entity A]?\"\n\n\nThe answers are human-confirmed. Any issues or ambiguities were adjusted or removed during the process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Clearly written, easy to read\n\nA new data set with questions that are still challenging for current LLMs\n\nTested on a few LLMs and two prompting strategies", "reasons_to_reject": "The process of how the questions were created is unclear. I.e. who were the question creators, and what were their guiding principles in creating those questions?\n\n\nAn independent (of question writers) validation of the quality of the data set is missing, and related to that a indicator what the human accuracy on the data set would be.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JkiebbPf0i", "reviewer": "Reviewer_qSX2", "summary": "This paper presents a dataset of multilingual compositional reasoning. The dataset considers a variety of  compositional relationships: such as Positional, Comparative, Personal, Mathematical, Identity. The dataset, MCR, is used to evaluate many popular large language models (GPT4, ChatGPT, Llama, Mistral, etc. Three prompting techniques are used.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to read and review our manuscript. Below is our response.\n\n__Why does MCR pose a challenge to LLMs?__\n\nWe believe it is a challenge because firstly, similarly to LLM’s reversal curse, current models are essentially a next-token generator/predictor. Asking them to reason reversally through contexts is challenging. Second, in MCR benchmark questions, the correct answer may not be a word seen in the question. Eg, mom’s dad is grandpa, the negation of large is small, etc.\n\n\n__Do models tend to get the same examples correct or incorrect?__\n\nWe employ greedy decoder sampling by setting the temperature to 0. If allowed by the API, we have also set top_k to 1.\n\n__why not scoring mode?__\n\n 3 out of 6 models in the experiment are closed-sourced and only accessible via API, making it difficult to directly access the models’ outputs before decoding. \n\n__How much could prompt engineering improve results?__\n\nIn the paper, chain-of-thought and few-shot prompting are used. We keep the settings the same as the original paper to showcase the challenge in reasoning through MCR. We will continue to explore such limitations and how to improve correctness via prompting / fine tuning in the future work.\n\n__Is my dad’s mom’s husband my grandpa?__\n\nThis is a potential limitation and we will incorporate explanations in the according section. Some questions in the MCR benchmark include certain assumptions. When querying models, we found they also have some assumptions. Eg, in a definition query “what’s dad’s mom’s husband?”, the model outputs “grandpa”. Such assumptions could come from the training data.\n\n__Why are there only five-shot results for ChatGPT?__\n\nAfter conducting experiments with few-shot learning on ChatGPT, we observed that the improvements were significantly smaller compared to zero-shot CoT (Chain of Thought). This suggests that the importance of exemplars may be less critical than CoT for these types of problems.\n\n__Can you say more about where error cases occur?__\n\nThere is a significant disparity in accuracy across different types of questions. For instance, in identity problems such as \"A is 100 meters to the left of B, and B is 100 meters to the right of C, where is A in relation to C?\", the model struggles to recognize that A and C are in the same position. Additionally, using ChatGPT as an example, it tends to answer \"100 meters to the right.\" which is substring seen in the question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Understanding compositionally of language models is a very important problem. This paper introduces datasets and to better understand where language models make mistakes in compositional reasoning. \n\nThe dataset is of reasonable size, the kinds of compositional reasoning selected seem quite reasonable. The dataset appears to be challenging  enough to demand further study for models of a wide variety of sizes (7B to GPT4 sized).", "reasons_to_reject": "It is clear from the empirical results that language model performance is not up to expectations on these compositional. While the authors break down the performance of the models in terms of relation numbers, prompting strategy, model size, exemplars, I feel that the paper would be much stronger if it further investigated the performance of the models in ways that may provide researchers with (1) greater insights into the challenges of compositional reasoning (2) ideas for mitigations/improvements to compositional reasoning systems. \n\nFor instance I am curious about the following:\n* Do models tend to get the same examples correct or incorrect?\n* Why is decoding mode used for these models in non-chain of thought generations — why not scoring mode? This way you could also look at uncertainty/perplexity of answer? And correlate uncertainty on particular questions across models?\n* Do the certain kinds of entities used in the questions affect performance? E.g. when person names are used are models better or worse  than when entities are Events A, B, C?\n* How much could prompt engineering improve results?\n* Why are there only five-shot results for ChatGPT?\n\nI would personally like to understand where models make mistakes in this data. Some of the questions I think are quite tricky:\n\n* Is my dad’s mom’s husband my grandpa? -> this question’s answer makes some assumptions. For instance, there is a universe where the dad’s mom has remarried and the question asker doesn’t consider this husband their grandfather. \n* Williams participates in the Miss Universe contest, one of the selection criteria for which is being enthusiastic. What kind of person is Williams? -> It is not clear if the selection criteria refers to what is needed to be a contest or to with the contest. And so it is not clear how to answer this question in my opinion.\n\nOther\n* humongous is a word that in my opinion is too informal to use in a research paper", "questions_to_authors": "* Can you say more about where error cases occur? Give more examples of language models making mistakes on this data?\n* Why is only 5-shot available for ChatGPT?", "extra_scores": {}}, {"review_id": "EtoYpEENl9", "reviewer": "Reviewer_RFfM", "summary": "This paper introduces a multilingual dataset with 1800 examples of various compositional relations. It presents a set of experiments to test the performances of LLMs (open-sourced and not) in this specific reasoning task. The dataset is interesting, and the paper is not bad; the authors also speculate about the results that would foster future research in this direction. However, the paper is badly written. It is still understandable, but it would require generous editing from a native speaker. I know that this is a sufficient reason to reject a paper at major NLP conferences. Since this is a new conference, I would like to start a new trend: accept this paper and strongly suggest giving them a better dress with the help of a professional editor.  \n\nHere my comments:\n\nEnglish should be carefully revised, there are several statements hard to process (e.g. \"our evaluation only exterminated a representative checkpoint\"). Also, textual flow is not optimal and in several paragraphs the text sounds like broken syntax (e.g. in the introduction).\n\nI would make a strong distinction in the future, both at the level of the dataset and experimentally, between lexicalised relations that involve commonsense or semantic knowledge (e.g. kinship terms) and other relations that would be treatable even if you put dummy words in the context. Indeed it would be very interesting to see results when you reformulate for example a comparative or identity relation with nonsense nouns. If the performance are the same it means that the inference is triggered by the syntax (and I guess it is in a lot of cases). It would make sense to include examples more complex in the dataset with a convoluted syntax.  \n\nWith careful inspection, the category \"Other\" can be split into more meaningful sub-categories. The better performance of all the LLMs for this category could be due to a subset of examples that are very easy to process. \n\nConcerning the zero shot CoT scenario, I need to see concrete examples of the dialogical interaction and how long it takes to arrive at the right solution.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your dedication to reviewing our manuscript thoroughly. Your constructive comments and valuable expertise have greatly contributed to the improvement of our paper. We will engage professional editing services to refine the manuscript before submitting the final version. \n\nThe perspective on syntax that you mentioned presents an intriguing avenue for future work. It may also relate to the counterintuitive phenomena associated with Japanese that you highlighted later in your comments.\n\n__question__\n\nThe syntax of Japanese is certainly a potential reason for the observed phenomenon, given its grammatical differences from other languages. Other factors may also contribute to this anomaly, such as：\n1. The performance of the translator. We translated the questions from English to Japanese, and the significant grammatical differences between these languages could have led to some unexpected interactions.\n2. Whether the pre-trained data contains similar relational structures is unknown since GPT4 is not open-source.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting dataset\nInteresting task at the intersection between syntax and semantics", "reasons_to_reject": "Badly written, it needs professional editing", "questions_to_authors": "Why Japanese and Korean in Figure 3 show better performance for 5 than for 4 relations? It is counter-intuitive. Could it be something related to the syntax of these languages?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper describes an evaluation dataset for LLMs involving compositional relational reasoning (e.g. "if A is perpendicular to B, and B is perpendicular to C, what is the relationship between A and B"). There are two sets of experiments: the first shows a strong dissociation between the ability to produce relational definitions and to reason relationally about situations (in a kinship domain), the second shows overall poor performance (and significant differences across languages) in a set of six domains.

Several details about the dataset creation and validation process have been added during the discussion period, as well as some discussion of failure modes. I still think the paper would be significantly strengthened by some more in-depth / quantitative analysis of different types of failures, but there seems to be reviewer consensus that it's acceptable in its current form.
Decision: Accept