Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism
Authors: Mengyu Ye
Tatsuki Kuribayashi
Jun Suzuki
Goro Kobayashi
Hiroaki Funayama
Venue: EMNLP 2023 Main (2023)
Keywords: Negation
Prompting
Reasoning
Language model
Model Analysis
Probing
Chain-of-thought
TL;DR: This work analyzed reasoning ability of large language models(LLMs) against lexical negation using step-by-step reasoning and shows that LLMs still struggled with addressing such negation.

Abstract: Large language models (LLMs) take advantage of step-by-step reasoning instructions, e.g., chain-of-thought (CoT) prompting. 
Building on this, their ability to perform CoT-style reasoning robustly is of interest from a probing perspective. 
In this study, we inspect the step-by-step reasoning ability of LLMs with a focus on negation, which is a core linguistic phenomenon that is difficult to process. 
In particular, we introduce several controlled settings (e.g., reasoning in case of fictional entities) to evaluate the logical reasoning abilities of the models. We observed that dozens of modern LLMs were not robust against lexical negation (e.g., plausible$\rightarrow$implausible) when performing CoT-style reasoning, and the results highlight unique limitations in each LLM family.

Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism
Mengyu Ye
Tatsuki Kuribayashi Jun Suzuki!.
Goro Kobayashi
Hiroaki Funayama
'Tohoku University
MBBZAA *RIKEN
vve.menguu.St s1, goro. koba, h. funa jun. www.suuutuuokkuuuuu jp.
tatsuki. suurbbaysshii www.aaaaasaaaaaai....... @mbzuai ac. ae
Abstract
Is the following sentence
Think step-by-step
May is turboglide player. Stepover happens in turboglide.
Large language models (LLMs) take advan- tage of step-by-step reasoning instructions, e.g. chain-of-thought (CoT) prompting. Building this, their ability perform CoT--ttyle soning robustly is of interest from a probing perspective. In this study, we inspect the step- by-step reasoning ability of LLMs with fo- cus on negation, which is a core linguistic phe- nomenon that is difficult process. partic- ular, we introduce several controlled settings (e.g., reasoning on fictional entities) to evalu- ate the logical reasoning abilities of the models. We observed that dozens of modern LLMs were not robust against lexical negation (e.g.. plausi- ble me-riimplausible) when performing CoT--style reasoning, and the results highlight unique lim- itations in each LLM family.
plausible?
yes
May did stepover.
Is the following sentence implausible?
X
LLM
yes
May did stepover.
The answer is
Figure 1: Overview of our experiments conducted to evaluate each model's reasoning ability against lexical negation. The model must answer no the latter ques- tion about the implausibility of the sentence based on the valid logical flow. Here, to evaluate the robust logical skill separately, the controlled reasoning chain is given, and the model must derive the answer based solely on validity of the logical flow without commonsense knowledge due to fictional entities, e.g., turboglide.
https: https///ggtuub..ooommmmgss8s com/nuy0862//// stepyystep-reasonnng-----eee
Our controlled experiments using dozens of LLMs, including GPT-4 (OpenAI, 2023), demon- strate that such models deteriorate in performance substantially when processing questions involving words with just a negative prefix, e.g., implausible, unreasonable (Figure 1). addition, the results show that each LLM family has its unique biases against lexical negation, which suggests that dif- ferent LLM training settings produce substantial differences under certain conditions, and the prob- lems to be addressed are model-dependent. These issues clarify the exact weakness of modern LLMs.
1 Introduction
Few-shot learning (Brown al., 2020) has led to a remarkable performance in large language models (LLMs). In particular, instructions to generate a reasoning process along with the answer, i.e., chain- of-thought (CoT) prompting (Wei et al., 2022; Ko- jima et al.. 2022), have improved the performance of LLMs. Building this, the ability of LLMs to perform CoT-style reasoning robustly is of interest from the probing perspective- how correctly these models perform step-by-step reasoning?: however, to the best of our knowledge, deeper analyses have yet to be explored fully. To address this question, this study investigates the step-by-step reasoning ability of LLMSs with special focus on robustness against (lexical) negation. Historically, negation has been challenging for neural models (Socher al., 2013; Kassner and Schütze, 2020), and de- termining whether the step-by-step reasoning of LLMs overcomes this limitation is important in the natural language processing (NLP) community.
Reasoning Against Lexical Negation
Given a chain of the reasoning process, we ex- pect the LLM to derive a logically valid conclusion even when the problem involves lexical negation (Figure 1). In Scction 2.1, introduce the task format, and Section 2.2 elaborates on the controlled task settings elucidate the abilities of the mod- els. Note that our task similar to CoT reasoning: however, we provide the models with predefined reasoning chains to facilitate controlled analyses.

---

Setting
Few-shot exemplars
Target example
fails at this setting
Is sentence "A does B" plausible? is C player. B happens in C/X. So the answer is yes/no.
sentence "D does E" plausible? D a player. E happens in F/Y.
CoT-style reasoning fails.
BASE
So the answer is
Is a sentence "A does 8" plausible? is a C player. B happens in C/X. So the answer is yes/no.
Is a sentence "a does S" plausible? is player. 3 happens in 7/x. So the answer is
Reasoning cannot be abstracted to fictional texts.
FIC
Abstract CoT-sttylc reasoning is only achieved on the affirmative domain.
Is sentence "A does 8 implausible? is C player. B happens in C/X. So the answer ycs/no.
sentence "a does J implausible? is player. 3 happens in 7/x- So the answer
FICNEG
a sentence "A does 8" plausible? FICNEG-O is a C player. happens in C/X.
a sentence does f implausible? is a player. happens in 7/x-
Model cannot handle domain shift terms negation.
the answer yes/no.
So the answer
Table I: General task format in each setting. Few-shot exemplars are first shown to a model, and then the model
answers to the target example given its question and reasoning chain. Symbols, e.g., and a, are replaced with certain real or fictional entities in the actual input. The REAL setting indicates that the entity choices reflect the factual reality, and FIc. indicates that the entity choices do not reflect factual reality, c.g., Is " "Judy Tate was safe at first." plausible? Judy Tate is turboglide player. Getring at first happens in turboglide. So the answer is yes. Refer to Appendix A for the exact input.
2.1 Format: Syllogism
into the task gradually (see the examples shown in Table I).
We evaluated the LLMs' ability to judge the va- lidity of particular types of syllogisms. Here, we utilized three settings to ensure the robustness of the results (Section 3); however, we consider the following SPORTS TASK (SP) format exam- ple to explain the settings. The base format of the syllogism is as follows:
BASE setting: In this setting, premises and con- clusions are aligned with the fact, e.g., Messi did a stepoverr is plausible; however, Messi performed a triple axel is implausible.
Fictional setting (FIC): We do not focus on de- riving an answer directly based on the model's 's knowledge without considering the reasoning chain. To eliminate such solution from the BASE setting and ablate the effect of factuality, we replace the PERSON and SPORT entities with fictional entities (see Appendix C. for details about fictional en- tities), where the correct conclusion can only be derived from the premise information in given reasoning chain. Note that these fictional entities are also used in subsequent settings.
Premisel: PERSON is a SPORT player:
Premise2: ACTION happens in the SPORT.
Conclusion:
PERSON does ACTION.
The above syllogism converted into instances, as shown in Table 1, comprising a question about the validity of particular conclusion (Is a sen- tence...lluusible") chain of the reasoning pro- cess (premises), and yes/no answer part. In the experiments, few-shot exemplars (Table column 2) were first input to a model, and then the model completes the answer for the target example (_ in Table 1, column 3) with yes/no. Here, the correct answer depends on whether the SPORTS entities mentioned in the chain (premises and 2) are the same. The exact input to the models is described in Appendix A.
In-domain negation setting (FICNEG): With this setting, test the model's robustness against lexical negation. Here, first design an in-domain setting where both few-shot exemplars and a tar- get example involve lexical negation. Specifically, we turn the original question into one that in- volves a word with a negative prefix, e.g., plau- sible-timpluusible (see Appendix B for the word list). 2 Thus, the correct answer to the question
2.2 Controlled Task Modification
To analyze how the models struggle with negation, we introduce presumably challenging properties
Testing negation in syntax, e.g., not, is another direction: however, this incurs additional difficulties. e.g., the scope of negation. We adopted the lexical negation as an initial investigation
'Strictly speaking. the answer should also be unknown when the two sports differ. In our experiments, our prompts explicate to answer no in such cases

---

should be flipped from yes/no no/yes.'
employed 10 variants of the base words and their corresponding negated expressions, c.g., plausi- ble/implausible, reusonable/unreasonadae Aver- age and standard deviation scores across these runs were reported.
Out-domain negation setting (FICNEG-O): We design an out-domain setting where few-shot exemplars do not involve lexical negation, but the target example has. If model fails at only this setting, this implies that the model overfits to the domain of the few-shot exemplars in terms of lex- ical negatio.. In addition, FIC and FICNEG-O O differ only in terms of the existence of the negation in the target example (this point can also be isolated by comparing these results).
We tested 14 common LLMs, e.g.. GPT- 4 and 3.5 (OpenAl, 2023), four LLaMA vari- ants (Touvron et al., 2023). Vicuna (Zheng et al., 2023), five OPT varisnts (Zhang et al., 2022), BLOOM (Scao et al., 2022), and BOOMZ (Muen- nighoff et al., 2022). Additional LLMs are tested in Appendices E and D. including the Alpaca (Taori et al., 2023), OPT-IML (Iyer et al., 2022), GPT- NeoXT (Together Computer, 2023), resulting in a total of 31 L.LMs (see Appendix E for more model details).
Models:
3 Experimental Settings
Task: In addition to the SPORTS TASK (SP) de- scribed in Section 2.1, we also introduce several different task formats. One is the OCCUPATION TASK (OC), where the underlying reasoning is sim- ilar the SP task, but the vocabulary and wordings are different:
Three exemplars given to the model along with general task instructions, e.g.. Let's think step by step (Appendix A). Note that the exemplars have at least one yes and one no an- swer. We also examined different exemplar orders, yielding consistent results independent of the ex- emplar orders (Appendix F). Here, the answer with the higher probability between yes and no in the model's outputs for the target example is consid- ered the answer. See Appendix E.1 for additional technical details.
Inference:
Premisel: PERSON is TITLE.
Premise2: OCCUPATION is described as TITLE.
Conclusion: PERSON is a OCCUPATION.
We also introduce the WRIGHT TRANSITION
TASK (WT), where the transitivity between the two propositions targeted:
Premisel: ANIMAL1 is heavier than ANIMAL2.
Premise2: ANIMAL2 is heavier than ANIMAL3.
Conclusion:
1NIMAL1 is heavier than 3.IMAL
To evaluate the LLMs, the accuracy of each model's binary answers measured (see Appendix G for the Fi-score results). In addition, to quantify the output bias, we also calculated a no-ratio, i.e., how many out of 1,000 instances the models answered no. Note that the chance rates of the accuracy and the expected no-ratio are 0.5 because the dataset is balanced in terms of the gold answer distribution Appendix C.3).
Metries:
All of these syllogisms are extended to the four different levels (BASE, FIC, FICNEG, and FICNEG- O) described in Section 2.2. See Appendix C for setting details.
Data: For the SP task, we collected 1,000 in- stances from the unddrstiddn task in the BIG-Bench dataset (Srivastava et al., 2022) for the BASE setting. 4 we also manually created the OC and WT instances to be similar to the SP instances.? We then modified these instances to create more challenging versions (Frc, FICNEG, and FICNEG- O). To enhance the generality of our findings, we
Results, Analyses, and Discussions
Tables 2 and 3 show the average and standard deviation of accuracy and no-ratio of cach model in the SP, OC, and WT tasks.
"We also adopt setting involving real entities and negation (NEG) in Appendix D. The results are generally competitive slightly better than those in the FICNEG setting
Consistent degradation against negation: We found that all models demonstrated performance degradation with the FICNEG-O setting; however, the GPT-4 model performed above chance (Ta- ble 2). In other words, the considered LLMs failed to address lexical negation in CoT-style reason- ing. We also observed a notable trend whereby the LLMs preferred to answer no regardless of the gold
'Note that SP task was originally intended evaluate the commonsense knowledge about sports. contrast, we to assess pure reasoning ability by providing the necessary facts to derive a conclusion a reasoning chain.
'While we created 1,000 instances for the OC task, 100 instances were created for the WT task since this task is re- garded as supplementary one; nevertheless, quite similar results to the other tasks were obtained.

---

SPORTS Task
OCCUPATION Task
WFIGHT TRANS, Task
Model
BASE
Fic
FICNEG
FICNEG-O
BASE
FIC
FICNEG
FICNEG-O
BASE
FIC
FICNEG
FicNEG-O
GPT-4
99.0104
56.71446
92.31333
66.61159
9822-02
76.5281
90.2441
75.71120
1000100
886.136
9811533
12.8-12
GPT:3.5
99.7+0.1
59,8+11.5
72.8447
36.6+555
97.1-03
58.8+133
58.4+25
39.9-21
73.5+455
68.1+100
63.6-85
35.3-102
L.LMMA-5B
99.8
89.012.9
90.74:10
12.
10000000
100.00:1
99.910.1
15.5263
10000000
10000000
66.02101
4
LLaMA-30B
99.8.:0.2
84.9:39
99.04.0.5
4.9.66.1
100.0=00
99.9
87.9422
18.5
99.34:08
89.3453
82
1.
LLaMA 13B
98.9+004
77.1+2.2
50.7++1.4
23.1+884
99.9-0.1
72.0=555
91.1+448
43.6=111
83.7+58
91.4663
82.2-80
46.0+000
LL-MA-7B 7B
93.7+11.4
63.6++4.8
58.6+55.0
49,5++0.0
68.0-18
59.7#220
53.24-20
46.2=00
68.24588
60.244.11
57.24111
46.0+00
Vicuna- 13B
98.4+-0.2
7..3++
83.4++3.7
21.6+74
99.8-0.1
72.5+
74.0+66.7
24.6==56
70.7+-4.0
84.8+-5.2
93-4-26
40.6=12.4
OPT- 175B
9..5
59.7:45.2
62.9-12.8
14.5
92.8-16
92.3
14.2
460000
804
18.0
53.2-76
11.6
OPT-6688
91.71:2.3
85.3.14.11
35.8.17.2
37.4:1299
88.8
99.61:044
36.91:102
35.3111.4
86.31.666
69.9155
43.2-27
46.010.0
3
72.5+336
51.4++0.7
47.8418
49.2++00
59.5-13
54.1+003
38.6+3.7
46.2+00
54.0+0000
54.0+0.0
44.7-37
46.0+00
OPT-13B
13.
72.7:46.3
49.5-4277
49,2++00
61.8-27
58.84227
32.4+:10.6
46.2+000
77.3=11.00
78.3+...1
46.0=00
OPT.67B
85.9+008
76.5+
46.7+13
45.8
71.4=13
86.3=22
26.3+5.1
46:2=00
15.1
54.2404
46.0=0
46.0+0.0
BLOOM
99.2
89.2
50.5
49 49.4
100.0=0.1
94.1=117
53.8+000
46.0=033
87.6268
83.745.1
50.9-68
1.
BLO0MZ
91.4113.0
50.5.100
410.2
48.9113
92.018
55.71:06
45.01006
46.21:1
54.1103
540100
4600-0
46.4110
Table 2: Average and standard deviation of models' accuracies for each setting the SPORTS TASK, OCCUPATION TASK and WEIGHT TRANS. TASK (scores are multiplied by 100).
SPORTS Task
OCCUPATION Task
WEIGHT TRANS, Task
Model
BaSE
Fic
FicNen
FICNEG-O
0
BASE
Fic
FicNBo
FICNEG-O O
BASE
Fic
PicNeo
FicNeo O
GPT-4
50.910.5
938147
41.9134
77.6180
53.2107
77.218.1
365242
50.1164
$4.1100
65.D111
44.1:005
82
GPT:3.5
50.7+00.
9..7++15
22.344477
87.1+35
5588+0
95.0+1.3
4.6+25
93.74-2.1
80.3+0.5
70,9+122
73.2+09
70.9+20
LLaMA-65B
50.2:20.0
39.5:22.0
58.7431
13:24:66
53.0200
$3.910.1
4622:0
69.3263
$4.0:0000
$4.0:00
80.0:110
97.6204
LLaMA-308
50.6.10.2
35.41339
50.11.077
50.2133
53.01000
53.910..1
34.11228
67.1113
53.310.11
43.3.10.5
58.0.108
98 1
LLMMA-13B
50.740.4
48.4+6.2
95.2422
68.8+122
53.1+01
81.8+55.
43.8+93
97.4+1.1
70.3+006
62.4+006
62.8+110
100.0+0.0
1LaMA-78
1..6
86.914
10.71:55
100.0:000
85
94.112.0
3.1237
100.0:0.0
85.8106
93.8.10.4
88.6113
1000100
Vicuma-138 13B
50.6-1-.44
37.0
50,1
37.74-126
53.2+-11
20.24-7
76.546688
83.3
69.2-10.5
42.0-10.4
88.6
OPT-175B
1..9
10.2:15.2
14.3
95.0
52.1
47.24.5.4
41.6:1322
99.8.1022
34.4
12.0:1122
33.2444
1.66
OPT-668
47.2.14.8
38.416.11
3641222
18.31:288
63.9227
54.3104
17.01104
19.11
64.310.9
83.7100
91.8144
1000100
30T:30B
77.5++3.6
99.4++0.7
93.7444.4
100.0++0.0
93.0+113
99.7+003
90.5448
100.0+0.0
100.0+00.0
100.0++0.0
44.3+22
100.0+0.0
OPT:13R
55.0:23.0
30.4:99.
13.5
100.0:000
91.22:7
96.02227
15.4
100.0:0.0
34.1+14
45.1
100.0:000
100.0200
OPT.6.7B
53.4+
31.0++9.
97.3+
96.6+b6.7
77.5+27
64.6++4.
60.3+1799
100.0+0.0
98.9++0.1
99.8++0.0
100.0+00.
100.0+00
BLOOM
50.1+0.2
61.3++2.7
0.0+0.0
99.9++0.2
53.0=0.
1..3+
99.8++03
64.2+009
41.7+1.1
93
2+05
BLO0MZ
58.3122
100.0:10.0
99.910.2
99.4
60.6220
98.
19.6
98.8200
99.9.1022
99.9-100
100.0:100
100.0.:00
97
Table 3: Average and standard deviation of models' no-ratio in the model outputs for each setting in the SPORTS TASK OCCUPATION TASK and WEIGHT TRANS. TASK (scores are multiplied by 100).
answer in the FICNEG-O setting (Table 3). Note that LLMs with accuracy rates of approximately 50% tended to continuously respond with no (or yes). This finding was particularly noticeable with the FICNEG-O setting where the LLMs that exhib- ited higher accuracy were those that constantly an- swered no (with the exception of the GPT-4 model). These indicate that models do not randomly behave but exhibit some systematic error patterns. Such consistent degradation also observed in case of BASE-+FIC, which suggests that CoT-sttle prompt- ing is supported by factors aligning with factuality along with the (insufficient) pure reasoning ability of the model.
Differences across model families: Interest- ingly, we also found that different LLM families struggled under different settings (the green to pur- ple patterns in Table 2). For example, the LLaMA models performed well with the FICNEG task but not the OPT models (Table 2). In particular, al- though the GPT-3.5, OPT-175B, and BLOOM(Z) models have approximately the same scale of pa- rameters, they exhibited contrastive trends. Similar trends were also observed for the no-ratio case. For example, with the FICNEG and FICNEG-O. the GPT 3.5, LLaMA-7B, and BLOOM models demonstrated extreme statistics approaching or 100, and their behavior flipped completely due to the different types of prompting between the FIC-

---

100
of LLMs. The closest work to ours reported an inverse scaling law of LLMs' performance against negated prompts (Joel et al., 2023). addition, we further elucidated the exact limitations and inter- model differences under controlled task settings.
Mean Accuracy (%)
80
BASE
60
FIC
FICNEG FICNEG-O
40
Step-by-step reasoning: Generating an infer- ence process with neural models has received in- creasing attention in terms of both performance improvement and model explainability (Ling et al., 2017; Sun et al., 2019; Rajani et al., 2019; Shwartz et al., 2020; Madaan al., 2021; Gu et al., 2022; Aoki al., 2023). Recently, the instruction make LLMs generate intermediate reasoning steps (i.e., CoT prompting) has led to improvements model performance (Wei et al., 2022). In this study, we attempted to elucidate the LLM's reasoning ability implicitly assumed in the CoT-sttyle prompt- ing and clarify that this success does not entail the LLMs' robust logical reasoning abilities (par- ticularly against lexical negation). Note that the deterioration in the fictional settings also elcidate that LLMs work well only in the frequent domain in the training data (McCoy et al., 2023).
20
10
20
30
40
50
60
LLaMA Model Size (Billion)
Figure 2: Relationship between model size (X-axis) and performance (y-axis) of LLaMA models for each task. Each point indicates the average accuracy of the corre- sponding model and setting. The colored area indicates the standard deviation.
NEG and FICNEG-O O tasks. The performance dif- ference between, for example, LLaMA-65B and OPT-66B also demonstrates that some factors other than parameter size induce substantial gap in per- formance toward certain linguistic phenomena.
Scaling law breaks: Scaling law in LLMs has generally been reported (Gordon et al., 2021; Ivgi et al., 2022); however, the improvement over the model scale broke, specifically in the FICNEG-O O setting, which confirms that our introduced task is challenging. Figure 2 shows this tendency for the LLaMA models.
Logical reasoning with LLMs and artificially controlled experiments: Integrating logical rea- soning ability into neural models a pivotal goal in the artificial intelligence field (Marcus, 2003). With this aim, enclosing the models' exact weakness with artificially controlled data has been actively conducted in our field (Betz al., 2021; Clark al., 2020; Lu al., 2021; Kudo et al., 2023); we show the peculiar case that just the flip of one word (adding nation prefix) causes drastic effeets for modern LLMs.
In summary, generally, we found that including lexical negation in the tasks caused a drastic perfor- mance reduction for the compared LLMs. The re- sults of the controlled experiments further revealed that different LLMS exhibited substantially differ- ent limitations and biases. Notably, we further tested the robustness of our findings with differ- ent prompt configurations and obtained consistent results (Appendix F).
Conclusions
In this study, have investigated the ability of LLMs derive valid conclusions given a reason- ing chain with a (lexical) negation, a historically tough phenomenon for neural models. The results of multi-difficulty controlled experiments revealed that LLMs with CoT-style prompting struggled to address negation; a simple flip of one word (e.g., plausible- satiible--imphnnn implausible) has significantly hurted their performance. In addition, we have found con- sistent, systematic failure patterns unique in each LLM family. For example, some models always answered no to different question settings. the future, we plan to analyze the model's internal and explore the source of this weakness.
5 Related Work
Negation and neural models: Negation is a core operation in natural language and logic, and pre- vious studies have investigated and attempted to improve neural models in terms of addressing nega- tion (Socher et al., 2013; Warstadt et al., 2019; Kim al., 2019; Kassner and Schütze, 2020: Ettinger, 2020; Hossain et al., 2020; Hosseini et al., 2021; Truong et al., 2023). We align such challenges in the context of CoT-style prompting and the scaling

---

Limitations
Acknowledgements
We would like express our gratitude to the mem- bers of the Tohoku NLP Group for their insightful comments. And special thanks Keisuke Sak- aguchi for his valuable suggestions on how to im- prove clarity in several aspects. This work was supported by the JSPS KAKENHI Grant Num- ber JP21H04901, JP22121492; JST Moonshot R&D Grant Number JPMJMS201 (fundamental research); and JST SPRING Grant Number JP- MJSP2114.
First, although we considered up to 31 LLMs, sev- eral other LLMs cannot be evaluated due to com- putational limitations, e.g., PaLM-540B (Chowd- hery al., 2022) and PaLM2 (Anil et al., 2023). Thus, evaluating the performance of these mod- els is left to future work. Second, in terms of the generality of the obtained results, the exam- ined prompt variations were limited, although we did examine prompts with different formats and orders (Appendix F). Third, in the current study, we adopted somewhat peculiar setting where the ofaiin-o-rrasoming process is given from the per- spective of the original CoT setting. Therefore, exploring the limitations in the inference based on the reasoning chain generated by the model will be an interesting direction from a practical perspective. Fourth, our analysis was limited to manavior-based probing: however, there are other paradigms to investigate (Lasri et al., 2022). In particularly, in- specting the inner workings of the models would be important understand the mechanism of the model's failure. However, this was difficult because some model parameters were not open, and the vast number of layers//ncad//paameters in large models made it difficult to track the precise patterns of the inner workings of the model. Finally, this study only considered lexical negation in English and was further confined to specific task formats and a certain type of syllogism. Therefore, extending the experimental scope will help further elucidate the exact limitations of the models.
References
Rohan Anil. Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Sia- mak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. PaLM 2 Technical Report. arXiv preprint, cs.CCL23005 10403v1.
Yoichi Aoki, Keito Kudo, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi. and Kentaro Inui. 2023, Empirical investigation of neural symbolic reasoning strategies. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1154-1162, Dubrovmik, Croatia, Association for Computational Linguisties.
Gregor Betz, Christian Voigt. and Kyle Richardson. 2021. Critical Thinking for Language Models. Procedingss of the 14th International Conference on Computational Semantics (ICCS), pages 63-75.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language Models Few-Shot Learners. In Advances in Neural Information Processing Systems 33 (NeurIPS), pages 1877-1901.
Ethics Statement
Our findings demonstrate that LLMs struggle to address lexical negation under step-by-step CoT- style reasoning settings. This problem is generally related to the problem of hallucinations LLMs.
We hope that our findings help to understand this issue by highlighting their exact weakness against negation.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. 2022. PaLM: Scaling Lan- guage Modeling with Pathways. arXiv preprint, es. C1,22204..2223 lv5..
The synthetic dataset utilized in the current study was created using automatic rules; thus, there were ethical concerns regarding human workers or annotators during the dataset creation processes. In addition, the entity distribution of the dataset is fully balanced, and most of them are fictional, and there are no intended biases, c.g., the stereotypical relationship between gender and occupation.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as Soft Reasoners over Language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), pages 3882-3890.

---

Tim Dettmers, Mike Lewis. Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(; 8-bit Matrix Multi- plication for Transformers at Scale. arXiv preprint, cs.LG/2208.0733999
of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7811-7818
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bow- man, and Ellie Pavlick. 2019, Probing What Differ- ent NLP Tasks Teach Machines about Function Word Comprchension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 235-2499
Allyson Ettinger. 2020. What BERT Is Not: Lessons from New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics (TACL), 8:34-48.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. 2021. Data and Parameter Scaling Laws for Neural Ma- chine Translation. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 5915-5922.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
Yuling Gu, Bhavana Dalvi, and Peter Clark. 2022. DREAM: Improving Situational QA by First Elab- orating the Situation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1115- 1127.
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, and Kentaro Inui. 2023. Do deep neural networks capture compositionality in arithmetic reasoning? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guisties, pages 1351-1362, Dubrovnik. Croatia, As- sociation for Computational Linguistics.
Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An Analysis of Natural Language Inference Benchmarks through the Lens of Negation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 9106-9118.
Karim Lasri, Tiaoo Pimentel, Alessandro Lenei, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the Usage of Grammatical Number. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (ACL), pages 0818-8831.
Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R De- Hjelm, Alessandro Sordoni, and Aaron Courville. 2021. Understanding by Understanding Not: Model- ing Negation in I Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies (NAACL-HLT), pages 1311-
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program Induction by Rationale Genera- tion: Learning to Solve and Explain Algebraic Word Problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (ACL), pages 158-167.
Maor Ivgi, Yair Carmon, and Jonathan Berant, 2022. Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experi- ments. In Findings of the Association for Computa- lional Linguistics: EMNLP 2022, pages 7354-7371.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neuro- Logic Decoding: (Un)supervised Neural Text Gener- ation with Predicate Logic Constraints. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (NAACL- HLT), pages
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster. Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang. Christo- pher Dewan, Asli Celikyilmaz, Luke Zeutlemoyer, and Ves Stoyanov. 2022. OPT-IML: scaling language model instruction meta learning through the lens of generalization. arXiv preprint, 22.C12222221122 1201
Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021. Think about it! Improving defeasible reasoning by first mod- cling the question scenario. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), P), pages 6291 6291-66 1-6300.
Jang Joel, Ye Seonghyeon, and Minjoon Seo. 2023. Can large language models truly understand prompts? a
case study with negated prompts. In Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop (TLANLP), pages 52-62.
Gary F Marcus. 2003. The Algebruic Mind: Integrating Connectionism and Cognitive Science. MIT press.
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. 2023. Em- bers of autoregression: Understanding large language models through the problem they are trained to solve.
Nora Kassner and Hinrich Schütze. 2020. Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly. In Proceedings

---

Niklas Muennighoff. Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai- ley Schoelkopf. Xiangru Tang, Dragomir Radev, Alham Fikri Aji. Khalid Almubarak, Samuel Al- banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint, cs.CL/2211 1.12261..1762.
2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (NAACI-HLLT), pages 2633-2643.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. 2023. Stanford alpaca: An mnstruction-folloving llama model. https: gi thub. ctt. http//ttts--abbb lati/ssu-lab/ss httm///attsullbbtttffrd.aaaaaaa stanford_alpaca.
OpenAL. 2023. GPT-4 Technical Report. arXiv preprint, cs.CL/2303.08774v3.
Together Computer. 2023.
OpenChatKit: An Open Toolkit and Base Model for Dialogue-style Appli- cations. ver. 0.15. GitHub repository.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Informa- tion Processing Systems 35 (NeurlPSS. pages 27730- 27744.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez. Armand Joulin. Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint, cs As.CCL/232.111 13377v...
Thinh Hung Truong, Timothy Baldwin, Karin Ver- spoor, and Trevor Cohn. 2023. Language mod- els are not naysayers: An analysis of language models negation benchmarks. arXiv preprint, cs. (LL/2306.881891/
Nazneen Fatema Rajani, Bryan McCann. Caiming Xiong, and Richard Socher. 2019. Explain Yourself! Leveraging Language Models for Commonsense Rea- soning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4932-4942.
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Ha- gen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang. Jason Phang. Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019. Investigat- ing BERT's Knowledge of Language: Five Anal- ysis Methods with NPIs. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing DEMNLP-IJCNLLS pages 2877-2887.
Teven Le Scao, Angela Fan, Christopher Akiki, BI- lie Pavlick, Suzana Ilie, Daniel Hesslow, Ro- man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Lan- guage Model. arXiv preprint, es. CL.C221..05100vr
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised Commonsense Question Answering with Self-Talk In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615-4629.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2022. tnain-of-Though Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35 (NewrIPS), pages 24824-24837.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1644 Seattle, Washington, USA. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alnsoo, et al. 2022. Beyond the Im- itation Game: Quantifying and extrapolating the capabilities of language models. arXiv preprint, cs.CL/2206.046152.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Sridhar,
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving Machine Reading Comprehension with General Reading Strategies. In Pruceedings of the

---

Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-tree trained Transformer Language Models. arXiv preprint, cs. 1LL/2205.01068v4.
Base words
Negated words
plausible
implausible
believable
unbelievable
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li. Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. 2023. Judg- ing -lm-as-a-judge with mt-bench and chatbot arena. arXiv preprint, es. 23L/2306.05685v.
reasonable
unreasonable
thinkable
unthinkable
probable
improbable
imaginable
unimaginable
convincing
unconvincing
A Prompt Examples
conceivable
inconceivable
feasible
unfeasible
Tables 16, 17, 18, and 19 show examples of the ex- act model input at each setting of our experiments.
credible
uncredible
Table 4: Full list of base words and lexically negated words used in the experiments.
B Lexical Negation
We created instances involving lexical negation by replacing an adjective in question with one with a negative prefix. For example, the question Is the proposition "Messi did a stepover" plausible? is converted to Is the proposition "Messi did a stepover" implausible? ? Specifically, we used the terms listed in Table 4 to achieve this conversion. Note that the original SP task only adopts the word plausible. Here, enhanced the diversity of the prompts ensure the generality of our findings. The lexical negation list was created as follows: (i) GPT-4 was employed to generate nine synonyms of the word plausible, and then (ii) we manually added proper negation prefixes to each synonym to form the lexically negated term.
1,217 fictional animals (see Table 5 for specific ex- amples). In terms of people's names, we initially collected 50 typical male and female first names from the"Name Corpus: List of Male, Female, and Pet names" ? which is available in the CMU Arti- ficial Intelligence Repository. We then randomly collected 100 family names from the "Telligent- CommunitySample 0.1.1" dataset, 8 which is ac- cessible via the PowerShell Gallery. Finally, we created a list of 100 fictional names for each sport by randomly combining the first and last names (ex- amples for the SPORTS task are shown in Table 6). We also used the weight data of Mammals to generate the gold label in the (non-fictional) setting of the WEIGHT TRANS, task.
C Task Details
C.2 Task Formats
Here, we describe the task settings in detail. To ensure the robustness of our findings, we conduct additional experiments on two tasks (in addition to the SP task), i.e., the OCCUPATION (OC) and weight transitivity (WEIGHTTRAN) TRANS:; WT) tasks. The results across the tasks support the overall con- clusion derived in the main part of this paper (See Appendix D for additional information).
SPORTS Task: See Section 2 and Table 1.
OCCUPATION Task: The task format in the FICNEG-O setting is described as follows:
Few-shot exemplar:
Q: Is a sentence "PERSON is TITLE" plausible? A: PERSON is OCCUPATION1. Only OCCUPATION1/2 are TITLE. So the answer is yes/no.
C.1 Fictional Names//nformation
In the fictional settings, used fictional enti- ties in all tasks. Here, we used GPT-4 to gener- ate the names of the fictional sports, occupations, and animals. We used five fictional sports (i.e.. (hydrosprint, aeropaddleball, gravitydodge, tur- boglide and titantumble)), five fictional occupa- tions (i.c., hydropurator, sonotextilist, chronoar- chaeor, quantumbotanialist, and psycostylist) and
Target example:
Q: a sentence "PERSON a TITLE" implausi-
hhttps://ww.cck cs. wtt.ccc..u.uiia.. wmm..ed///roopp//l/ttii//ras/ nlp/ po/ppoor///aame//eeee html
" "https://ww/r powershe
Te TelligentCommuni Lyg.genttoounnnnnaammlll.l...l S "Mammals ordered by their weight: https:
'In considcration of readability, this is presented on the
//thewessi www.eeerytttingcc.o.o. wwtryyttin.com/aaann adult adult-weisd wet.eeett.htt html
last page of the paper.

---

Fictional Sports
Fictional Occupations
Fictional Animals
Hydrosprint
Hydropurator
Flickerbeast
Striped Quillaphant
Whiskerfluff
Aeropaddleball Sonotextilist
Quokkalinga
Pinnapartholanka
Glidefin Skyweasel
Gravitydodge
Chronoarchaeor
Prismazebra
Pangolirex
Fawnimouse
Turboglide Titantumble
Quantumbotamialist
WaveSkitterer
Shadow Glidehopper
Nimbuswolftail
Psycostylist
Fluffentinger
Glimmerhorn Crestail
Grizmalian Whiskerlop
Table 5: Examples of fictional sports, occupations and animals we used.
Hydrosprint
Aeropaddleball
Gravitydodge
Turboglide
Titantumble
Tilda Pruift
Phoebe Richardson
Hussein Whiffield
Rob Hancock
Vita Ehmore
Sansone Brady
Michel Allen
Larisa Keller
Chas Morrow
Jay Fowler
Judy Tate
Alyssa Melntyre
Wilburn Anderson
Sonja Fletcher
Stefan Camp
Petrina Norman
Dosi Sykes
Ernesto Hall
textitKaleb Graham
Malcoln Pearson
Rutherford Lucas
Fruncisco McCoy
Douggie Barbour
Garwin Shields
Cassi Cooke
Way Franklin
Lorain Reid
Celia Jain
Gunter Payne
Linnet Page
Jannel Stanton
Neda Rose
Raynard Kemp
Elliott Blum
Myrilla Anderson
Law
Sonni Burnett
Gregor Neill
Hailey Hatcher
Tobye Washington
Owen McGee
Agathe Frederick
Carlton Morris
Cornelius McCarthy
Granville White
Kalvin Barr
Darrick Rogers
Katti Davies
Purker Baxter
Corny Reid
Table 6: Examples of fiction person names we used for each fictional sport.
ble?
cluded 496 yes labels and 504 labels, and the FiC dataset included 495 yes labels and 505 no labels. The FICNEG dataset included 504 yes la- bels and 496 no labels, and the FICNEG-O dataset included 505 yes labels and 495 no labels.
A: PERSON is OCCUPATION1. Only OCCUPATION1/2 are TITLE. So the answer is
Put simply, the underlying reasoning flow is sim- ilar to that of the SP task; however, here, the entities (i.e., the occupation and property names) differ.
D Full Results
All results for the SP, OC, and WT tasks are shown in Table 9, 10, and 11, respectively Note that the WT experiment was conducted at a 1/10 scale (1,000 instances-100 seed instancesx 10 negated words) as a supplementary experiment.
WEIGHT TRANS. Task: The task format in the FICNEG-O setting is as follows:
Few-shot exemplar:
Is sentence 1ANIMAL1 is heavier than ANIMAL 2 plausible?
We also examined the textseNeg setting, where real (not fictional) entities used; however, the question involved negation as intermediate set- ting between the BASE and FICNEG settings. The performance all models is shown Table 8. As can be seen, the results are generally competitive or slightly better than those obtained with the FICNEG setting. In other words, the model cannot handle negation in natural text, and abstract reasoning over negation is even more difficult.
1NIMAL1/2 is heavier than ANIMAL3. ANIMAL3 is heavier than ANIMAL2/1. So the answer is yes/no.
Target example: Is sentence 1ANIMAL1 is heavier than ANIMAL2" implausible?
ANIMAL1/2 is heavier than 3NIMAL3. ANIMAL3 heavier than ANIMAL2/1. So the answer is
Here, the transitivity of reasoning (A>B, B>C, then A>C) targeted.
E Models
In this study, we evaluated the 31 models listed Table 15. 11 For the GPT-4 (gpt-4-0314) (OpenAI, 2023), and GPT-3.5 models (i.e., text-davinci-002.
C.3 Answer Distribution
Essentially, the yes:o ratio of the gold labels was approximately 1:1. Strictly speaking, the distribu- tion differed slightly from 1:1 due to the random seed used the dataset creation process. For ex- ample, for the SPORTS task, the BASE dataset in-
IIn consideration of readability, these tables are presented after several tables Presented after several tables demonstrating supplemental results.

---

lext-davinci-003 (Ouyang al., 2022), and gpt- 3.5-turbo-0301). the experiments were conducted on June 2023 utilizing OpenAl's API. Note that gpt-4-0314 and gpt-3.5-turbo--e t-tuubbo-030 will be phased out the near future.
Model
Accuracy
No-ratio
GPT4
99.8 +0.3
52.1 1
GPT-3.5-turbo
90.5 12.4
61.6 11.7
text-davinci-003
99.5 +0.4
$2.0 1.6
text-davinci-002
99.3 10.3
$2.9 11.8
LLaMA-65B
100.0 10.0
$2.1 11.7
The experiments for the other (non- -penOO models were conducted using Huggingface Trans- formers (Wolf et al., 2020) with the 8-bit op- tion (Dettmers al., 2022). For the LLaMA (Tou- vron al., 2023) models, we received the model weights from the LLAMA Release Team on May 25, 2023. addition, we recovered the Vicuna and Alpaca (Taori et al., 2023) models based on the provided LLaMA weights. For the OPT models ranging from 1.3 B to 66B, OPT-IMI models, and OPT-IML-MaX models (lyer et al., 2022), employed the models available from the Huggingface Community Model Hub We re- ceived the model weight for the OPT-175B (Zhang et al., 2022) model from Meta on May 28, 2022. We also used the BLOOM (Scao et al., 2022), BLOOMZ. (Muennighoff et al., 2022), and NeoXT- Chat-Base-200 20B (Together Computer, 2023) models available from the Huggingface Community Model Hub.
LLaMA-30B
99.5 10.5
51.6 11.5
LLaMA-13B
95.3 13.3
50.0 16.1
LLaMA-7B
85.1 13.1
62.3 18.3
Vicuna- 13B
93.7 15.3
47.2 18.3
Vicuna-77 7B
93.4 1:3.7
48.5 17.8
Alpaca-7B
89.5
52.5 +3.6
OPT-175B
65.6 +14.4
17.7
+12.7
OPT-66B
93.0 +335
$2.0 +3.0
ОPT-30B
58.0 13.2
87.8
10.66
OРT-13B
64.1 +4.9
71.8 #7.6
OPT-6.7B
71.8 14.1
47.1 15.5
OPT-2.7B
55.7 +2.5
83.3
118.6
OPT-1.3B
67.2 15.5
55.6 121.33
OPT-IML-MaX-300
84.7 12.3
46.6 112.3
OPT-ML--Max-L3B
62.1 16.2
76.6 112.3
OPT-IML-30B
80.4 1444
39.0 111.9
OPT-IML-1.33
65.3 +5.4
67.4
12.2
BLOOM
94.5
46.7 +3.7
BLOOM-7.IB
58.7
35,0 18.3
BLOOM-3B
56.4 10.7
93.8 13.9
BLOOM-1.7
50.2
12.6
BLOOMZ
76.0 #:4.4
70.5
19.9
BLOOMZ-7.ID
54.0 12.8
98.1
11.4
BLOOMZ-3B
52.7 4:2.3
99.4
10.6
E.1 Model Settings During Generation
BLOOMZ-I.7B
52.3 11.8
99.8
10.3
To ensure that the models only output yes or no, we applied some changes during the answer gener- ation process. Specifically, for the OpenAl models, we introduced an equal logit bias to yes and no using the provided logit_bil option, while ting tempreture = 0.0, max_ soaa_ttkessss = 1. For the other non-OpenAl models, we manually ascer- tained the logit of yes and no, ultimately using the greater of the two as the model's final response under the same settings as the OpenAl models, in which temperature 0.0, max_new new tokens 1.
NeoXT ReoXT-Chaat-a Chat-Base- B0s
73.4 19.7
78.5 19.3
Table 7: Average and standard deviation of models' accuracies and the no-ratio of the model outputs in Fic- tional Prompt.
entities Thus, we conducted supplementary exper- iments in which the few-shot exemplars pertained to fictional entities. These experiments were im- plemented under the FIC setting, and the results are presented in Table 7, where the values are the averages from the SP and OC tasks.
F Robustness over Different Prompts
We explored the influence of the prompt format in both few-shot exemplars and target examples. Here, we used the following for- mat questions with the gold labels designated no. (Note, the format with gold labels of yes were unaltered.) A corresponding example shown as follows:
Prompt format:
To ensure the robustness of our results across dif- ferent settings, we conducted supplementary ex- periments to investigate both prompt order and for- mat. These experiments were conducted using the SP and OC tasks. Note that these supplementary experiments were conducted at 1/10 scale (1,000 1000 seed instancesx 10 negated words).
a sentence "PERSON does ACTION" plausible?
PERSON is SPORTS palyer. ACTION happens/does not happen in SPORTS. So the answer is yes/no.
Fictional prompt: The few-shot exemplars in the main experiments consistently involved real

---

Model
Accuracy
No-ratio
was yes, no, yes; thus, we conducted supplemental experiments where the gold label sequences were altered to yes, yes, no and no, yes, yes. The results of the prompt order experiments are shown in Ta- ble 13, which shows the averages from the SP and OC tasks.
GPT-4
91.1 18.8
56.1 =7.8
GPT-3,5-turbo
86.1 1334
40.2 -9.1
text-davinci-000
99.6 +0.5
48.6 1.11
lext-davinci-002
100.0 10.0
48.3
1.3
LLaMA-65B
99.8 10.2
48,2 1.3
LLaMA-30B
96.6 12.0
44,9 330
LLaMA-13B
83.3 19.7
62.2
10.8
LLaMA-7B
70.0 48.5
30.0
992
G F1 Score
Vicuna- 13B
97.4 11.7
11.
Certain models (e.g.. BLOOMZ family and OPT family in Table 9) predominantly registered an ac- curacy of approximately 50% by consistently re- sponding with no (or yes), Note that this pattern was particularly evident for the FICNEG-O setting, with the GPT-4 model being significant outlier. To highlight these models, provided the macro- averaged Fl-scores in Table 14.
Vicuna-77 7B
82.3 = 14.6
31.1
164
Alpaca-7B
55.4 15.3
71.0 99.4
DPT-175B
61.7 11.2
31.1
+10.7
OPT-66B
31.8 +5.7
38.9
14.4
OPT-30B
42.3 14.4
93.2 5500
OPT-13B
47.2 437
63.5
13.
OPT-6.7B
45.2 15.8
71.42200
OPT-2.7B
45.8 44.4
35.3
OPT-1.3B
48.4 11.3
99.2 1.2
OPT-MML-MaX-301
42,3 = 10.5
14.2
112.6
OPT-MML-MaX-133B
48.3 11.4
100.0 0.2
OPT-IML-305
20.8 11.8
44.7 19.2
OPT-IML-L.3B
48.2 1.5
99.7 0.4
BLOOM
63.7 +6.0
16.1 7777
BLOOM-7.IB
55.1 12.2
BLOOM-3B
11.7
0.5
BLOOM-1.7B
48.4 11.3
0.1
BLOOMZ
17.5 +6.5
65.3
10.00
BLOOMZ-7.IB
48.1 11.1
99.8 -02
BLOOMZ-3B
48.0 +1.5
99.5 0.44
BLOOMZ-1.7B
48.3 11.3
100.0 0000
NeoXT-Chat Baaattt-aee-201
49.3 +1.9 96.0 3.4
Table Average and standard deviation of model accu- racies and no-ratio for the NEG setting (i.e., real entities and questions with negation).
Is a sentence "PERSON does ACTION" im- plausible?
PERSON is SPORTS palyer.
ACTION happens/does not happen in SPORTS. So the answer is no/yes.
Compared to the original format (Section 2), premise 2 changes. Here, the task is not to identify the consistency of sports/occupation name; how- ever, the conclusion depends on the existence of does not.
The results are shown in Table 12. Note that both the accuracy and no ratio values are the averages obtained from the SP and OC tasks.
Prompt order: We investigated the impact of the prompt order with a specific focus on the position the no label in the three exemplars. The order the three exemplars in the main experiments

---

Accuracy
No-ratio
Model
BASE
FIC
FICNEG
FICNEG-O O
BASE
FIc
FICNEG
FICNEG O
GPT-4
99.0=+0.4
56.7+4.6
92. 3.3
66.6 15.9
50.9+0.5
93.8+4.77
41.9+3.4
77.6+8.0
GPT-3,5-turbo
99.710.11
1.5
72.814.7
36,613.5
50.7100.1
9..7
22.314.77
87.1
text-davinci-003
99.910.1
81.611..8
87.213.2
28.0119.6
50.4100.1
40.4113.3
51.2.1.7
64.8.18.2
(ext-davinci-022
100.010.0
74.4.2..
91.112.3
49.1
50.410.00
76.012.7
40.6.1.22
99.8.10.3
LLaMA-65B
99.8+0.0
89.0:12.9
90.7+3.0
12.8=16.5
50.2+00.0
39.542.9
58.7+3.1
73.2 16.6
LLaMA-30B
99.810.2
84.9.339
99.010.5
4.916.11
$0.610.2
35.413.9
50.1
50.2.13.3
LLaMA-13B
98.9-1.4
77.1=2.2
50.7=144
23.148.4
50.7+0.4
48.4-6.2
95.2=22.6
62.2
LLaMA-7B
93.711.4
63.6.14.8
58.615.0
49.510.00
56.611.5
86.914.8
10.7
100.010.00
Vicuna-138
98.410.2
77.3111.
8.741.7
21.617.4
$0.610.44
37.0
50.119.3
32.7
Vicuna-7B
98.310.3
93.0.12.6
58.516.8
29.71:20.2
50.3
57.442.6
8.1.6.9
52.0-125.9
Alpaca-7B
91.41.77
83.3.14.9
48.613.4
49.511.7
43.812.6
67.015.0
85.618,2
98.2.13.0
OPT- DPT-175B
9.5
59.7:5.2
62.9
10.6
46.9
10.245.2
12.3
10.0
OPT-66B
91.7+2.3
85.3++4.1
35.8+7.2
37.4+12.9
47.2+4.8
38.4+6.1
36.4+22.88
88.3+12.8
OPT-30B
72.5
51.4:00.7
47.8118
49.21:000
77.543.6
99 99.77
93.71...4
100.010.00
OPT-13B
73.3
72.716.3
49.512.7
49.2110.0
55,013.0
30.4
74.5113.6
100.010.00
OPT-6.7B
85.910.8
76.5118.0
46.713.4
45.81666
53,412.4
31.019.4
97.313.5
96.6.1.77
OPT-2.7B
75.2+5.4
54.4.5.44
35.516.0
39.919.5
42.2111.8
7.947.4
36.2:122.0
80.5
OPT-1.3B
70.7+4.8
56.6++4.6
49.2+0.0
47.7+4.77
72.0+8.8
20.1+15.2
100.0=0.0
96.4+11.4
OPT-IML-Max-30B
96.110.4
80.8
51.8116.
46.515.55
51.411..
69.5114.8
10.31:1.7
73.2-12.9
OPT-IML-Max-133B
57.711.5
51.711.
49.5
49.5
92.311.9
98.811.4
100.010.0
100.010.0
OPT-ML-33B
94.810.7
83.315..8
52.218.9
46.31..88
49. 49.1
66. 16.00
50.2115.2
91.713.77
OPT-IML-1.3B
58.110.8
52.911.4
49.5
49.510.0
89.412.0
97.611.4
100.010.00
100.010.0
BLOOM
99.210.1
89.2.12.7
50.510.0
49.410.2
$0.110.2
61.31127
0.0.1.00
99.910.2
BLOOM-7.IB
68.3+0.9
10.9+1100
50.5+0.0
43.6+9.8
55.4+6699
1.4
0.0:
73.4=39.4
BLOOM-3B
51.410.6
50.510.0
51.510.7
49.510.0
99.010.66
100.0.0000
2.4.2.1
100.0.1..0
BLOOM-1.7B
50.7+0.9
49.5=10.0
49.5+0.0
49.5+0000
2.34:2.4
0.0:10.0
100.0=0..0
100.0+1..0
BLOOMZ
91.412.0
50.510000
49.410.2
48.911.3
58.312.4
100.010.00
99,910.2
99.411.3
BLOOMZ-7.1B
52.811.0
50.5+10.0
49.5
49.510.00
97.5111.1
100.010.00
100.010..0
100.011..0
BLOOMZ-3B
50.710.2
50.5+0.00
49.510.0
49.510.00
99.610.2
100.010.00
100.0.1..0 100.010.0
BLOOMZ-1.7B
50.6+0.3
50.5+0.0
49.5+0.0
49.5+0.00
99.8+0.33
100.0+0.0
100.0=0.00
100.0+00.0
NeuXT-Chat-Base-200
77.5133.
52.4.188
49.510.1
49.5L10.0
72.9.13.8
98.11188
100.010.1
100.010.00
Table 9: Average and standard deviation of model accuracies and the no-ratio of the model outputs at each setting for the SPORTS TASK.

---

Accuracy
No-ratio
Model
BASE
FIC
FICNEG
FICNEG-O
BASE
FIC
FICNEG
FICNEG O
GPT-4
98.2+-0.2
76.5+8.1
90.244.
75.7+12.0
53.2+0.7
77.2+8..
3..5+4.2
50.1+6.4
GPT-3.5-turbo
97.110.7
58.8111.3
58,412.5
39.912.1
$5.810.8
9.3
2.6.
93.712.
text-davinci-003
99.910.0
73.312.4
60.612.6
25.914.3
53.110.0
80.312244
6.912.7
77.911.7
lext-davinci-002
100.0.0.0
63.713.0
62.914.2
49.316.3
53.010.0
90.113.00
9.114.2
95.7.1..0
LLaMA-65B
100.0:0.00
100.0:0.11
99.9
15.5
53.0+0.0
53.9+0.1
06.2:10.1
69.3
LLaMA-30B
100.0.0.00
99.910.1
87.912.8
18.5
$3.010.0
53.910.1
34.1
67.4111.3
LLaMA-13B
99.9--.11
72.0+5.55
91.14-88
43.6
53.1+0.1
81.8
43.8-9..3
97.4
LLaMA-7B
68.011.8
59.71.20
53.212.00
46.2.1.00
85.011.8
94.112.00
3.113.7
100.010.00
Vicuna-I
99.810.11
72.513.3
74.016.7
24.6.15.6
53.211.11
81.313.3
20.2:1..7
76.516.8
Vicuna-7B
93.5.1.5
64.4122.5
$3.810.00
38.315.2
59.542.55
89.4
0.0.1..0
22.2-126.1
Alpaca-7B
83.412.9
83,414.5
39.216.00
43.511.7
69.612.9
70.0
78.019.7
97.2.177
OPT-175B
9.68+116
92.3
30.2114.8
46.0#0.2
$2.1
47.245.4
13.6
99.8100.2 2
OPT-66B
88.8+2.6
09.6+0.4
36.9+10.2
35.3+11.4
63.9+2.7
54.3+0.4
89.1+11.4 17.0
OPT-30B
1..
54.110.3
38.6133.7
46.210.0
93.0
99.710.33
90.511..
100.010.00
OPT-13B
61.812.7
58.812.7
32.4110.6
46.210.0
91.212.7
95.012.7
75.4118.4
100.010.0
OPT-6.7B
7..411.3
86.312.7
26.315.1
46.210.0
77.512.7
64.614.0
60.3111.9
100.010.00
OPT-2.7B
53.4-0.3
55.010.9
45.117.3
46.210.0
99.610.4
98.811.0
10.719.4
100.010.00
OPT-1.3B
1.2
6..9+6.2
46.1+0.1
46.2+0.0
95.6+115
86.8+6.3
99.9+0.1
100.0+0.00
OPT-IML-MuX-308
87.211.5
74.211.77
43.5113.1
37.5
64.1
79.611.7
11.31:4.8
42.8
OPT-MMLMaxx-13BB
73.913.4
71.216.11
46.210.00
46.210.0
67.915.5
82.616.1
100.010.00
100.0.0.00
OPT-MLL-0B
89.311.0
75.113.8
20.4115.4
42.0.13.3
$3.712.3
77.614.4
39.11:20.0
95.8133.3
OPT-IML-1.3B
77.114.7
78.516.7
46.210.0
46.0.0.33
64.216.7
74.816.99
100.010.00
99.8.10.3
BLOOM
100.0.0.1
94.111.7
$3,810.00
46.0.1..3
$3.010..1
$9.311..
0.0.0.00
99.8.10.3
BLOOM-7.IB
72.3+0.8
3.3
$3.8+0.22
47.0+1..4
61.3+4..1
61.9+10.0
0.2::0.4
90.0=31.6
BLOOM-3B
53.210.1
$3.810.0
$3.810.0
46.210.00
99.810.1
100.010.0
0.0.1.00
100.0.10.0
BLOOM-1.7B
52.743.3
46.310.2
46.240.00
46.2-1..0
9.811..1
0.14:04
100.0=0..0
100.0+10.0
BLOOMZ
92.011.8
55.710.6
45.0110.6
46.210.1
60.6 0
98.110.6
98.810.6
99.910.2
BLOOMZ-7.1B
5..311.2
54,4:10.5
46.210.0
46.210.0
9..6
99.4
100.010.00
100.010.0
BLOOMZ-3B
54.410.3
54.110.3
46.210.0
46.210.0
98.410..4
98.81004
100.010.00
100.0.0.00
BLO0MZ-1.7B
57.6+22.1
53.8+0.0
46.2+0.0
46.2+0.0
94.4+22.7
100.0+0.00
100.0=0.00
100.0+00.0
SAAXT-Chat-Base-220
66.916.2
55,81222
46.210.00
46.2-10.0
86.116.2
98.012.2
100.010.0
100.010.00
Table 10: Average and standard deviation of model accuracies and the no-ratio of the model outputs at each setting for the OCCUPATION TASK.

---

Accuracy
No-ratio
Model
BASE
FIC
FICNEG
FICNBG
BASE
FIC
FICNEG
FICNEG-O
GPT-4
100.0+F0.0
88.6 13.6
98. 15.3
12.8+12.1
54.0+0.0
6..0+1.4
44.1=0.5
32.8+199
GPT-3.5-turb
73.514.5
10.0
63.618.5
35.3110.2
80.310.5
70.91122
73.2=0.9
70.912.0
text-davinci-003
99.31222
95.814.55
94.215.4
16.2112.7
53.310.2
49.810.4
44.8-0.6
47.411..
text-davinci-002
96.312299
88.818.7
99.3100.5
46.0.10.0
57.710.3
65.2.0.9
45.3-0.0
100.0.10.0
LLaMA-65B
100.0::0.0
100.0+0.0
60.0
43.6
$4.0
54.0+0.0
80.0=100
97.6:t0.4
LLaMA-30B
99.3100.8
89.3
88.018.2
44.311.5
53.3.0.1
43.310.5
58.0-0.8
98.310..1
LLaMA-13B
83.7.5..8
91416.33
82.2+8.99
46.0+1.00
70.340.6
62.440.6
62.8=1.0
100.0:10.0
LLaMA-7B
68.2.15.8
60.214.11
17.2111.6
46.010.00
85.810.6
93.8.0.4
88.6-1.2
100.0.10.0
Vicuna- 13B
70.714..
5.2.2
93.4112.
40.6.112.4
83310.4
69.2.0.35
42.0-0.4
88 88.6.10.9
Vicuna-7B
90.1133.4
89.0
79.943.9
43.615.7
62.710.4
45.810.35
0.73
97.6.10.6
Alpaca-7B
72.117.6
62.914.9
46.010.0
46.0.1..0
81910.88
90.7.00.5
100.0.0.0
100.0.10.0
OPT-175B
80.4115.2
58.0111.6
$3.247.6
41.6
1..4
12.0+112
95.6 4322=4.2
OPT-66B
86.3++6.6
69.9+5.5
43.242.7
46.0+0.0
64.3+0.9
83.7+0.6
91.8-1.4
100.0+0.0
OPT-30B
54,0:10.0
54.010.0
44.74:3.7
46.010.0
100.0
100.010.0
44.3=2.6
100.0110.0
OPT-13B
77.3111.0
78.318.1
46.010.0
46.010.0
34.111.4
45.111.9
100.0-0.0
100.010.00
OPT-6.7B
55.111.1
54.210.4
46.010.0
46.010.00
98,910.1
99.810.00
100.0-0.0
100.0.1..0
OPT-2.7B
65.816.7
71.517.9
53.815.1
46.010..0
26.811.4
52.912.0
33.0_3.5
100.0.10.0
OPT-1.3B
62.0+6.4
59.1+6.0
46.0+0.0
46.0+-.00
26.0++211
26.5+
100.0-0.0
100.0+0.0
OPT-MML-MaX-30B
97.7100.7
93.313.2
25.211183
33.24110.3
56.110.1
60.7
31.2=118
71.241.
OPT-MLL-Max-133B
85.218.0
80.0.10.4
46.010.0
46.010.0
53.811.5
71.211.
100.0
100.0.10.0
OPT-IML-30B
92.513.55
99.2111.0
28.1110.4
38.0:110.9
46.510.4
$3.610.11
30.5
90.0
OPT-IML-1.3B
46.411.3
48.317.3
19.3
40.7
0.6.0..2
2.310.7
93.2-112
90.9112.0
BLOOM
87.616.88
83.718.1
80.911.88
45.411..6
64.2.0.99
41,711.11
93.9-1.0
98.2.10.5
BLOOM-7.1B
46.0:t0.0
46.0=0.0
$4.0++0.0
47.4#3.6
0.0.E..0
0.0+0.00
0.6=0.2
69.8+t:.6
BLOOM-3B
$4.0.10.0
54.010.0
$4.010.0
46.010.0
100.010.0
100.0.0.00
0.0
100.0.10.0
BLOOM-1.7B
46.0:10.0
46.0-0.00
46.0+1..0
46.0+10.0
0.0:10.0
0.0:0.00
100.0=0.0
100.0:10.0
BLOOMZ
$4.110.3
54.010.0
46.010.0
46.411.0
99.9110.0
100.010.0
100.0-0.0
92.4111.7
BLOOMZ-7.1B
$4.0.1.00
54.010.0
46.010.0
46.010.00
100.010.0
100.010.0
100.0=0.0
100.0.10.0
BLOOMZ-3B
59.813.0
54.510.5
46.010.0
46.010..0
94.210..3
99.510.1
100.0-0.0
100.0.10.0
BLOOMZ-1.78
54.0++0.0
54.0+0.0
46.0+0.0
46.0+-000
100.0+0.0
100.0+0.0
100.0=0.0
100.0+0.00
NeoXT-Chat-Bas--0B
89.118.3
68.0111.4
46.0
46.2.10.6
64.5.10.9
84.611.4
100.0=0.0
99.8
Table II: Average and standard deviation of model accuracies and the no-ratio of the model outputs at each setting for the WEIGHT TRANS. TASK.

---

Accuracy
No-ratio
Model
BASE
Fic
FICNEG
FICNEG-O
BASE
Fic
FICNEG
FICNEG-O
GPT-
92.817.0
66.9115.6
94.112.1
82.8118.1
48.710.3
80.011.7
47.610.3
46.910.9
GPT-3.5-turbo
99.540.7
59.9+3.6
71.4+10.8
40.045.3
54.020.5
88.6+0.5
22.941.2
87.5
LLaMA-65B
100.010.00
100.0.0.00
100.010.0
6.616.5
53.510.5
48.510.2
51,51002
55.110.8
LLaMA-30B
100.0.0.00
100.0:0.00
97.2133.5
16.8
53.510.5
48.510.2
48.7100.5
36.71114
LLaMA-13B
100.0:10.0
98.0112.6
99.2.1..3
16.1115.1
53.510.5
50.510.4
50.810.1
64.511.6
LLaMA-7B
91.5++8.9
88.3+11.7
84.6+8.1
46.5+5.5
62.0+
60,2+1.3
36.2+0.9
95.0++0.7
Vieuna-
100.0:0.00
90.717.1
95.2144.2
19.6
53,5
57.8
46.7
53.0
OPT-1757
98.5+1.8
100.0+0.0
91.8+6.9
34.2+17.8
55.1+0.6
48.5+0.2
4..8+0.6
82.7+1.9
OPT-66B
96.51:399
100.0:1..0
25.8111.8
16.9
57.010.8
48.510.2
2.
6..411..7
ОPT-30B
89.517.2
74.5113.9
49.912.8
$1.511.5
64.011.
74.0.1..5
97.110.1
100.0100.0
OPT-13B
83.6110.6
83.9115.2
63.4.120.0
$1.51155
70.0.1.5
64.51117
67.3118
100.0.10.0
OPT-6.7B
93.7.16.5
99.0.11.0
50.113.4
41.611144
59.811.1
49.5
98.610..2
90.2
BLOOM
100.0.1..0
99.4.10.9
51.715.1
43.618.1
$3.510.5
49.1
3.310.4
92.110.8
BLOOMZ
96.0:E..3
50.3
51.2:3.2
64.0
57.540.5
98.210.1
95.9110.4
86.9111.4
Table 12: Average and standard deviation of model accuracies and the no-ratio at each setting for different Prompt format.

---

Accuracy
No-ratio
Model
BASE
Fic
FICNEG
FICNEG-O
BASE
Fic
FICNEG
FICNEG-O
GPT-4 (yes-no-yes)
98.1 1.3
13.8
90.915.0
69.8
54.310.5
84.2.1..3
42.410.5
63.0111.
GPT- (no-yes-ee)
98.0=1.0
15.6
87.9+5.2
66.6
54,510.4
78.4+14
39.4=0.5
6.
GPT-4 (yes-yes-no)
98.0-1.2
71.0-14.1
96,412.2
72.0117.6
54.210.4
77.511.3
49.610.4
51111.7
GPT-3.5-turbo (ees-no-yes)
98.6.1..7
55.112.88
66.488.4
41.91444
54.610.6
93.410.4
17.911.
90.410.6
GPT-3.5-turbo (n0-yes-yes)
98.0-1.2
54.913.5
71.017.1
41.41-1.3
55.510.77
93.510.5
22.8-0.9
90.0.10.6
GPT-3.5-turbo (yes-yes-no)
99.2.1..8
60.4.15.1
75.1
34.718.2
53.110.4
88.110.6
34.910.9
83.2.1..9
LLaMA-65B (yes-no-yes)
1.0.0=0.0
95.3:15.4
96.1+4.7
21.7213.7
53.5+0.3
43.8
55.3=0.6
1.
LLaMA-65B (no-yes-yes)
100.0.0.00
91.114.0
91.2.14.1
39.319.4
53.510.5
$1.911.1
$2.011.1
87.8.10
LLaMA-650 (yes-yes-no)
99.7=0055
90. 10..
90.1
9.2112.1
53.240.3
38.611.2
61.41111
56.211.2
LLaMA-30B (yes-no-yes)
0.0
94.5++6.0
93.7+6.0
14.3+14.0
53.5+0.5
43.0+0.7
45.7=
56.8+1.4
LLaMA-30B (no-yes-yes)
97.2-1.00
93.7155.2
89.115.1
14.1:15.2
56.2.10.5
44.710.7
42,9-0.9
1.
LLaMA-30B (yes-yes-no)
98.6-0.5
83.7-16.8
87.8111.6
21.9117.2
52.110.3
32.211.8
61.31.55
7.7
LLaMA-13B (yes-no-yes)
99.0
74.317.2
19.7
36.8113.3
53.310.5
66.411,66
63.612.3
83.7.1.77
1LBMA-13B (no-yes-yes)
91.2=3.22
54.0+3.9
3..943.4
48,414.6
62.3+0.5
93.2
9..0
9..8+0.55
LLaMA-13B (yes-yes-no)
98.511.5
86.617.2
712117.3
27.9117.3
52.010.66
54.311.5
73,412.2
69.0.1..7
LLaMA-7B (yes-no-yes)
82.9
58.0:E5.4
54.816.4
51,511.3
71221.
90.5
7.5
100.0:00.0
LLaMA-7B (no-yes-yes)
7.99-6.9
50.512.7
39.019.4
51.511.5
78.611.1
98.010.2
77.912.1
100.0.10.0
LLaMA-7B (yes-yes-no)
96.312.2
82.6.7.22
40.219.5
50.91222
54.910.99
65.910.8
7..2.1.66
99 99.411.22
Vicuna- 13B (yes-no-yes)
98.7=0.9
75.1
10.
24.0#8.2
52.8+0.6
62.242211
1.8
$7.7+:2.2
Vicuna-13B (no-yes-yes)
96.0.2.2
68.2-13.9
66.3112.0
29.7110.2
56.610.7
78.5
68.31221 32.312.8
Vicuna- 13B (yes-yes-no)
1.3
79.916.3
79.7
27.848.3
52.210.6
58.411.8
33.9=1.3
$9.5::2.2
OPT- 175B (yes-no-yes)
95.5-2.9
80.0=15.6
41.2+14.0
49.0+8.0
52.6+0.7
30.2+1.99
20.5+1.9
97.5++088
OPT-175B (no-yes-yes)
91.2-18.1
90.6.1..1
41.4111.0
49.217.6
56.610.99
50.5
10.7+1.5
9..8
OPT-175B (yes-yes-no)
93.3-1.7
87.7-13.4
37.1115.1
44.4112.8
47.410.66
36.211.5
29.5-3.1
9.
OPT-66B (yes-no-yes)
3.2
93.3116.6
33.718.2
40.2111.8
59,911.1
44.910.7
1..0
88.711.2
OPT-66B (no-yes-yes)
83.3=8.1
80.8
10.2
50.7
6..41.6
61.7
5..1
99.2++0.2
OPT-66B (yes-yes-no)
86.8.5.6
89.916.0
18.9114.8
47.916.6
65.711.1
58.410.7
58.811.7
96.4.10.7
OPT-30B (yes-no-yes)
68.916.8
48.911..
46.0
$1,51113
84.6E1.1
99.610.1
93.1
100.0:00.0
OPT-30B (no-yes-yes)
61.115.5
48.811..
50.912.2
$1,5113
92.510.1
99.710.1
99.510.1
100.0.10.0
OPT-30B (yes-yes-no)
57.112.6
48.5:1..
38.61114.6
51,51115
96.410.3
100.0:0.00
85.0.1..6
100.0:1.00
OPT-13B (yes-no-yes)
70.0=6.7
66.2+12.8
44.2+11.66
5..5+1.3
75.0+2.0
65.
56.3=2.4
100.0:0..0
OPT-13B (no-yes-yes)
73.8.3.4
77.0.1..1
51.2.11.4
48.717.3
70.0.1.77
47.6.1..8
99.710.1
97.0.1..8
OPT-13B (yes-yes-no)
71.2+7.7
63.8=14.1
29.3+11.0
51.5
74.2+2.0
82.8
68.5=1.5
100.0+0.00
OPT-6.7B (yes-no-yes)
61.0-6.3
80.9+4.44
40.2+12.7
49.5+4.99
66.8+1.4
49.6+2.00
78.9=2.4
98.0++0.5
OPT-6.7B (no-yes-yes)
7..2-8.5
67.7
20.9
160.7
50.6:1.88
18.2:1..6
99.
1.4
OPT-6.7B (yes-yes-no)
68.2-6.2
62.317.6
32.117.5
51.511.5
84.2.0.33
86.210.9
45.212.6
100.0.10.0
BLOOM (yes-no-yes)
99.2-0.9
87.8.3.11
48.5
$1,51113
54.210.4
60.7
0.0
100.0.10.0
BLOOM (no-yes-yes)
96.9=1.8
75.8+4.0
46.5+2.66
4..6+5.5
56.6+0.6
02.8
2.0-0.3
89.9 1.7
BLOOM (yes-yes-no)
99.1
91.015.1
48.3
49.113.4
52.6.0.66
57.610.4
0.6.0.11
96.2.10.7
BLOOMZ syes-no-yee yes)
90.13332
49.1
50.8+1.9
51.211.5
62.6Ht0.3
99.4
99.4=0.0
99 99.6:t0.1
BLOOMZ (no-yes-yes)
49.0.12.1 86.9-2.5
50.8
51.4113
66.2.1..3
99,510.1
99.0 99.210.0
BLOOMZ (yes-yes-no)
1.9
19.1
49.1
48.81334
65.611.66
99.5
97.6-0.2
94.2-10.8
Table 13: Average and standard deviation of model accuracies and the no-ratio of the model outputs at each setting for different Prompt orders.

---

SPORTS Task
OCCUPATION Task
WEIGHT TRANS. Task
Model
BASE
Fic
FICNEG FICNEG-O
BASE
FicNEG FICNEG-O
BASE
FICNEG FICNEG-O
GPT-4
0.99=00
0.46+0.1
0.92+000
0.6
0.98
0.73+0.1
0.38204
0.7620.1
1.0+00
0.87002
0.98+0.11
0.75=002
GPT--.5-turb
100:000
0.521000
0.7.1..1
027:000
0.9710.0
0.46:1000
0.52102
0.2910.0
0.7.10.1
0.65101
0.62.1.11
0.31
text-davinci-0033
1:0:00
0.81200
0.872000
0
10000
0.7.200
0.49201
0.2120.0
0:99:000
0.96200
0.9420.1
0.13:01
lext-lavinci-1022
1,0000
0.73+a00
0.91400
0.33-00
10400
0.55:40.1
0.53:4011
0.38440.1
0:96:100
0.88::011
0.994:0.
0.32+00
LLaMA-65B
100100
0.891000
0.91
0.17
104000
1.0100
1.0.000
0.1310.0
1.0100
100000
0.63
0.31:00
LLaMA-308
100:00
0.851000
0.991000
001
101000
1.0100
0.87.:00
0.1610.1
0.991000
0.892:1
0.88101
0.311:00
LLaMA-130
0.99=000
0.77+00.
0.38 +.0
0.2-01
10+00
0.67+00.1
0.91+00
0,3+000
0.82++0.1
0.91+01
0.82+0.11
0
L.aMA-7B
0:94000
0.57
0.5.2..1
0:33
0.63-20.0
0.48:20.0
0.37200
0.3220.0
0.62:20.1
0.48
0.494022
0.32:00
Vicuna-138
098.100
0.77..00
1:.0
0.19:00
0.73.1
0.2.1.00
0.66:10.1
0.841:1
0.911000
0.32.00.1
Vicuna 7B
0.98=00
0.93+0..1
0.49++0.1
0:26=02
0.93+0.01
0.56+000
0.35+0.0
0,28+0.0
0.9+000
0.89+00
0.8+0.00
0.3+00
Alpacn-78
091
0.8320
0..
0:34
0:821000
08220.
0.35201
0.310
0.67:2011
0.58201
0.32200
0.32.00
OPT-175B
0:9
0.521a.1
0.58.202
0.3.0
0,93100
0.92.00
0.28
0.31
0.79:103
0.51200
0.4g0.1
0.29:001
OPT-66B
091:100
0.851..0
0.31
0.27:111
0.8810000
1.0.000
0.27
0.26.10.1
0.85.10.11
0.64::011
0.3240.0
0.32:000
OPT-308
0.71:100
0.351000
0.361000
0.33.00
0,4910000
0.36:10.
0.291:00
0.3210.0
0.35.100
0.352:00
0.4110.0
0.32.:00
OPT-
0,73=u0
0.71+0011
0.45+000
0.33-00
0.52++0.0
0.46++0.1
0,27+0011
0.32+0.0
0.76++0.3
0.77+001
0.32+000
0
OPT-6.78
0.86:200
0.75
0.322200
0:31
0.68
0.86-200
0.24:000
0.3220.0
0.37:200
0.35200
0.32.00
0.32200
OPT-2.7B
0.75=a.1
0.44
0.31+0.0
0.3=00
0.36++0.0
0.38+000
0.32=000
0.32+0.0
0.64+a0.1
0.69
0.47+0.1
0.32=00
OPT-1.38
0.68:1011
0.511:011
0.331000
0:33200
0.441.0.0
0.6:10.1
0:322:00
0.3210.0
0.58.101
0.562:01
0.32.0..
0.52:000
OPT-IMI-MMxx33
0:96200
0.810
0.4110.11
0.42-01
0.87100
0.71:000
0.320
0.3120.1
0:981000
0.93000
0.2.0.11
0.29:001
OPT-MMI-Max-L388
0.48=00
0.36
0.33+00
0.33-00
0.73+
0.66-20.1
0.32
0.32+0.0
0.84
0.77
0.32+0.0
0.32=00
OPT-IML-30BB
095-00
0.83+o.1
0.51+
0.35=00
0.89
0.72+00
16
0.3+0.0
0.92+00
0.99=00
0.23
0.29
OPI-IML-1.38
0.39100
0.3310.
0:33.00
0.7610.1
0.76.10.1
0.321000
0.3210.0
0.33100
0.35101
0,2810.1
0.291011
BLOOM
0.99-00
0.89+0u0
0.34+000
0.33-00
L0+00
0.94+000
0.35+00
0.32+0.0
0.87+00.1
0.83+001
0.41+0.1
0.32+00
11.00M-7.1R
0068
0.372o0
0.342000
0:31200
0.721
0.75.200
0.35.00
0.3210.0
0.322000
0.32200
0.35
0.31:00
BLOOMB3B
0.36=00
0.34+00
0.37+00.
0:33=00
0.35
0.35+000
0.35=000
0.32+0.0
0.35+000
0.35=000
0.35
0.32=00
BL00M-1.78
0:36:00
0.331000
0.331000
0333000
0.451.01
0.32.000
0:322:00
0.3210.0
0.321000
0:321000
0.32.1..
0.52:000
BLOOMZ
091
0.341:00
0.33100
0333200
0.921000
0.39.000
0:312:00
0.3210.0
0.35100
0:35200
0.32.0.
0.35.00
BL.0OMZ-7.18
0.39=00
0.34
0.33
0.33-00
0.44
0.36+000
0.32g00
0.32+0.0
0.35+00
0.35
0.32
0.32=00
BLOOMZ-3B
0.341000
0,341000
0.33
0.33000
0,38100
0.37.1000
0.321000
0,3210,0
0.47.10.1
0.361:00
0,32.1000
0.32.000
BLOOMZ 1.7B
0.34=a0
0.34+00
0.33++0.0
0.33-00
0.45+0.0
0.35+000
0.32+00
0.32+0,0
0.35+000
0.35+00
0.32+0.0
0.32+00
MOXXT-Cha--Bee-211
0.76
0.38
0.33:10.0
033
0.619.1
0.39.00
0.322000
0.3210.0
0.881001
0.620
0.32g00
0.32.00
Table 14: Models' average-macro Fl score at each settings.

---

Model Name
Details
URL
GPT-4
OpenAT's API call, specifically gpt-4-0314
netps: //patt///...m....ooo openol. apt rnl-rrefenenee/he
GPT-3.5-turbo
OpenAT's API call, specifically gpt-3.5-turh-0300
hetps: /ttp////p.oo...ooo/oo openai con/doex/
apl-referensee referenee/chtt
lext-davinci-003
OpenAl's API call, specifically text-davinci-003
hetps: poop://plctee /tppttromm....cmmoomo apenal. con/ con/docs/ api- rewereeeeeeeeeeee completions
lext-davinci-002
OpenAl's API call, specifically text-davinci-002
https: /tppttoom. openal. con/doce api wwwrreereeeeeeee complettions
LLaMA-65B
Original model weight provided by LLAMA Release Team
/tggtt/ coe/ Tacebookrr esear esearch/12a
LLaMA-30B
Original model weight provided by LLAMA Release Team
https.//githal wht./ghtttt.oooooooo cowr aecesooereeerrchhlee...
LLaMA-13B
Original model weight provided by LLAMA Release Team Original model weight provided by I.LAMA Release Team
hetps: thub, cow/ Facebookr esear esearch/11.a ch/1lass
LLaMA-7B
hesps: iigithib hids cone Facebonke eoner cl/ll wis
Vicuna-Iie
Recovered model weight based LLaMA
hetps: /chpg: lnng co/Tesys/
vicumn- 1ib ta
Vicuna-7B
Recovered model weight based on LLaMA
hetps:/ httpss//huggingth wag.ggeeeggee.g...ooooooo por Imsys/ vicura- delta-v).
Alpaca-7B
Recovered model weight based LLaMA
Stcps:/hhuggrgge grace. tatsu- alpasa- 76-wdiff
OPT-175B
Original model weights provided by Meta
hetps://githil /tgithub.b... Facebooke esear
retaseq/tiiob
OPT-66B
Available Huggingface
hetps: //hug///ggggggg
Fareaacc////tt-eee /opt--éhh
OPT-30B
Available Huggingface
hetps: //hugg//agiiie iing gface. ood ('acabock//ppt-aaat
OPT-13B
Available Huggingface
hetper /hugg ingfaree on Fambbock/opt /opt-/h
OP
Available Huggingface
hetps: whwggggggggaoooeeo oof wwt.://wwoaoooooooeeoo Facebockk//ptt- 0.
OPT-2.7B
Available Huggingface
hetper /hugg ingfare, on Farnbook/opp -.
OPT-1.3B
Available Huggingface
hetps:// htw.ggggggaacccccc col wtwctbock//ppt-11111 1.pt-1...
OPT-IML-Max-300
Available Huggingface
hetpe: /hugg ingface, Faceboook/ rax-
OPT-IML-Max-30B
Available Huggingface
hetps: /thag inggaco. wt.faaabook///////// opt
OPT-IML-30B
Available Huggingface
//hug inggaiingaase Facibook/ opt-
OPT-IML-30B
Available Huggingface
hetps: /hogggiggggggte wwt.acccooo.../ogooooooooo 3b
BLO0M
Available Huggingface
heape: /hhgg ingiininn
BL00M-7.1B
Available Huggingface
BLOOM-3B
Available Huggingface
hetps:/ /hugg ainge grace. bugge wew.eennnonoo-....eeeeoooooooooe
BLO0M-1.7B
Available Huggingface
hetps: hhagggngg wag...ccc.....oooo...oo.oc.ec. wtt.cc.coo..sccieeeeeee oegccinnee
BLOOMZ
Available Huggingface
B1.00MZ-7.1B
Available Huggingface
heapee innffrre ienowe
BLOOMZ-3B
Available Huggingface
hetps: /hugg gfacc.cccc....oooo..o....cce co/bogscience bloow-33
BL.0MMZ-1.78
Available Huggingface
hetper ingginaaace bilbti iencee bicome- 1b7
NeoXT-Chas Base-20B
Available Huggingface
hetps: whwgggigggacce hwwp://w...cogttteeooooeeeeeee Chat
Table 15: Full model list and details.

---

Few-shot exemplars
Determine whether an artificially constructed sentence relating to sports is plausible or not.
Q: Is the following sentence plausible? "Bam Adehayo scored reverse layup in the Western Conference Finals."
A: Let's think step by step.
Bam Adcbayo is an American basketball player. Scoring a reverse layup happens in hasketball. So the answer is yes.
Q: Is the following sentence plausible? "Santi Cazorla scored touchdown."
A: Let's think step by step.
Santi Cazorla is soccer player. Touchdown happens in football. So the answer is no.
Q: Is the following sentence plausible? "DeMar DeRozan was called for the goaltend."
A: Let's think step by step.
DeMar DeRozan is American baskethall player. Gnaltending happens in baskethall. So the answer is yes.
Target example
Q: Is following sentence plausible? "Raisel Iglesias was safe first." A: Let's think step by step. Raisel Iglesias is a baschall player. Getting out at first happens basehall. So the answer
Table 16: Example of model input for BASE setting, i.e., entity choices are realistic, neither few-shot exemplars nor the target example has lexical negation.
Few-shot exemplars
Determine whether artificially constructed sentence relating to fiction sports plausible
Q: following sentence plausible? "Bam Adebayo scored a reverse layup the Western Conference Finals." "
Let's think step by step.
Bam Adebayo is an American basketball player. Scoring a reverse layup happens in basketball. answer yes.
Q: following sentence plausible? "Santi Cazorla scored a touchdown."
A: Let's think step by step.
Santi Cazorla is soccer player. 'Tucchdown happens football, So answer is no.
Q: following sentence plausible? "DeMar DeRozan was called for the goaltend."
A: Let's think step by step.
DeMar DeRozan is an American basketball player. Goaltending happens in baskethall. So answer is yes.
Target example
Q: Is the following sentence plausible? "Harrison Bullock was safe first."
A: Let's think step by step. Harrison Bullock turboglide player. Getting first happens in turboglide. So the answer
Table 17: Example of model input for fictional setting (FIc), i.e., entity choices are fictional, neither few-shot exemplars nor the target example has lexical negation.

---

Few-shot exemplars
Determine whether an artificially constructed sentence relating to fiction sports is implausible or not.
Q: Is the following sentence implausible? "Bam Adebayo scored a reverse layup in the Western Conference Finals."
A: Let's think step by step.
Bam Adebayo is an American baskethall player. Scoring a reverse layup happens in baskethall. So the answer is no.
Q: Is the following sentence implausible? "Santi Cazorla scored a touchdown."
A: Let's think step by step.
Santi Cazorla is soccer player. Touchdown happens in football. So the answer is yes.
Q: Is the following sentence implausible? "DeMar DeRozan was called for the goaltend."
A: Let's think step by step.
DeMar DeRozan is an American baskethall player. Gnaltending happens in baskethall. So the answer is
Target example
Q: following sentence implausible? "Harrison Bullock was safe first."
A: Let's think step by step.
Harrison Bullock is turboglide player. Getting out at first happens in turboglide. So the answer
Table 18: Example of model input for in-domain negation setting (FICNEG), i.e., entity choices are fictional, both few-shot exemplars and target example have lexical negation.
Few-shot exemplars
Determine whether artificially constructed sentence relating to fiction sports plausible
Q: following sentence plausible? "Bam Adebayo scored a reverse layup the Western Conference Finals." "
Let's think step by step.
Bam Adebayo is an American basketball player. Scoring a reverse layup happens in basketball. answer yes.
Q: following sentence plausible? "Santi Cazorla scored a touchdown."
A: Let's think step by step.
Santi Cazorla is soccer player. 'Tucchdown happens football, So answer is no.
Q: following sentence plausible? "DeMar DeRozan was called for the goaltend."
A: Let's think step by step.
DeMar DeRozan is an American basketball player. Goaltending happens in baskethall. So answer is yes.
Target example
Q: Is the following sentence implausible? "Harrison Bullock was safe first." A: Let's think step by step. Harrison Bullock turboglide player. Getting first happens in turboglide. So the answer
Table Example model input for out-domain negation setting (FICNEG-O) i.e., entity choices are fictional. only the target example has lexical negation.


Reviews:
[{"review_id": "TazKdYklPO", "reviewer": "Reviewer_YrJj", "summary": "", "questions": "A: How does performance in the Fic setting change if you provide fictional few-shot exemplars instead of real ones?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your invaluable review!\n\n**Question A: What would happen if you provided fictional few-shot examples in the FIC setting?**\n\nWe appreciate your insightful query. We are willing to share the additional experimental results obtained with fictional prompts in the camera-ready version. We believed that using real-entity prompts would be a suitable setting as a first step, considering the practical usage of LLMs (i.e., people don’t query LLMs with fictional prompt), but again your suggestion is insightful for scientific analysis.\n\nWe recognize that the descriptions of our controlled task in Table 1 are incorrect. Specifically, under the FIC, FICNEG, FICNEG-O settings, the variables should be those representing real ones (e.g., a, b, etc.) instead of the current ones (e.g., $\\alpha$, $\\beta$, etc.); that is, your understanding and the examples shown in Tables 14–18 are correct (few-shot examples have real entities). We’ll correct the table and apologize for the potential confusion.\n\n**Mismatched caption in the appendix (Tables 5 and 6)**\n\nWe will ensure to make the corrections in the camera-ready version. We’re sorry for the mismatched caption of Tables 5 and 6 in Appendix E.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The evaluation is thorough and covers a wide range of models, giving readers a full picture beyond just \"LLMs fail to do X\".\n\nThe experiment is well-designed to help disentangle some of the factors contributing to CoT's effectiveness.", "reasons_to_reject": "I did not notice any methodological flaws that would be cause for rejection.", "questions_to_authors": "A: How does performance in the Fic setting change if you provide fictional few-shot exemplars instead of real ones?", "paper_topic_and_main_contributions": "This paper conducts a controlled assessment of large language models' robustness to negation when instructed to evaluate syllogisms involving fictional referents. The authors evaluate 31 models (results for 14 of which are presented in the main text), finding that model performance consistently degrades when fictional referents are used, and when negation in instructions is not matched between demonstrations and the target example. These findings support the conclusion that models lack enough \"pure reasoning\" ability to faithfully carry out chain-of-thought reasoning in general, and that CoT performance instead relies on a combination of factors beyond just the structure of the reasoning chains.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The caption of Table 5 in the appendix is duplicated from Table 6."}, "extra_text": {}}, {"review_id": "uF7bgCFBdE", "reviewer": "Reviewer_a7Sc", "summary": "", "questions": "Question A: The authors “used ten variants of negated words, e.g., implausible, unreasonable” (lines 119-121). To be clear, were the base (i.e., non-negated) words also used?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments!\n\n**The averaged results in the main text could mask differences in model behavior between the different domains.**\n\nWe appreciate your perceptive feedback. We fully agree that averaging behaviors across both sports and occupation domains (Tables 2 and 3) may result in an inadequate comprehension of the varying behaviors between these domains although we separately showed the per-task results in Appendix E. Following your suggestion, we will divide Tables 2 and 3 into two separate tables each and position them suitably in the camera-ready version.\n\n**Question A: Were the non-negated words also used in the experiments?**\n\nYes, we did utilize all 10 non-negated words (e.g., plausible, reasonable) shown in Table 4 in our experiments, although it wasn't clearly mentioned in the description of our BASE setting (lines 114-116). We'll ensure this is clarified in the camera-ready version.\n\n**Additional metrics other than accuracy would sharply penalize those models that always answer “no” (or “yes”).**\n\nWe agree with your suggestion that offering additional metrics like F1 scores would facilitate a better understanding of the various trends between different LLMs. We will incorporate F1 scores and an expanded discussion in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is very well-written and easy to read overall. The methods are sound, and the authors conducted an impressive number of auxiliary experiments, on various domains, models, and prompt styles, to ensure the robustness of the results. The results themselves, particularly that different model families break at different points, are very interesting and show the importance of differences in language models beyond parameter size.", "reasons_to_reject": "The results shown in the main text (i.e., Tables 2 and 3) average together results from both the sports and occupation domains, which can mask differences in model behavior between the different domains. For example, in Tables 8 and 9, it seems that most OPT models in the FIC setting tend to answer “yes” in the sports task and “no” in the occupation task, while one might conclude from Table 3 that they tend to answer with a mix of “yes” and “no”. I would suggest using one domain for Tables 2 and 3 and referring to the Appendix for the others.", "questions_to_authors": "Question A: The authors “used ten variants of negated words, e.g., implausible, unreasonable” (lines 119-121). To be clear, were the base (i.e., non-negated) words also used?", "paper_topic_and_main_contributions": "This paper studies whether large language models are able to correctly answer questions requiring step-by-step reasoning, and whether these abilities are robust in the presence of lexical negation in the prompts. The authors sample instances from the BIG-Bench sports-understanding task (and generate similar instances in the occupation and weight-transitivity domains), and modify them with fictional entities and negated words to create four increasingly difficult tasks. Their results show decreasing performance as the tasks become harder, with different model families beginning to struggle at different points, and no performance improvement (or even a performance decrease) with increasing model size.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Looking at Tables 2 and 3, the most striking thing I notice is that on the FicNeg-O task, with the notable exception of GPT-4, the best-performing models are precisely those that always answer “no”, and therefore get ~50% accuracy. The authors may want to consider (additionally) presenting metrics other than accuracy; for example, the minimum of the F1 scores between the two classes (or even the macro-averaged F1 score) will sharply penalize those models that always answer “no” (or “yes”).\n\nThe authors note in Appendix C that they experiment with a weight transitivity task; however, this is not mentioned in the main text—it should be (not everyone will read the appendix, and the authors should make sure the readers know about all the work they have done!)."}}, {"review_id": "u87ikXcoEg", "reviewer": "Reviewer_RddC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\n**Reasons To Reject: Failures of LLMs at various tasks is likely to happen.It is interesting that the paper here tries to overcome this by trying to understand the decompose the task, to understand the exact source of the failure.**\n\nWe’re struggling with comprehending the Reasons to Reject you provided fully. Per our understanding, you state that pinpointing the specific source of LLMs' failures by decomposing the task is an interesting direction, and this is exactly what our study does (see Table 1). If this is the case, we view it as a positive comment, and there seems to be a disparity between the soundness score (3) and your reason to reject. The other two reviewers have given positive reviews and scores (soundness: 4 and 5) that align with their positive reviews. Therefore, we kindly request that you reconsider your score so it aligns with your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper highlights a basic failure of even some of the most apt modern LLMs.", "reasons_to_reject": "Failures of LLMs at various tasks is likely to happen. It is interesting that the paper here tries to overcome a very general \"there is problem\" alter by decomposing the task, to understand the exact source of the failure.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper investigates the ability of modern LLMs to perform syllogisms involving morphologically negative words, with Chain of Thoughts systems.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this is a solid paper with well-structured and detailed experiments, as well as clear articulation of the findings. They also found that the results around different LLMs being differenly successful in dealing with negation to be interesting. The work is timely as it helps us better understand today's popular LMs with impressive (but error-prone) chain-of-thought abilities.

The reviewers pointed out some ways to improve presentation (e.g., splitting Tables 2 and 3), which the authors acknowledged and agree to.
Decision: Accept-Main