Empower Nested Boolean Logic via Self-Supervised Curriculum Learning
Authors: Hongqiu Wu
Linfeng Liu
hai zhao
Min Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: nested boolean logic
curriculum learning
TL;DR: A new self-supervised learning to enhance logical learning process.

Abstract: Beyond the great cognitive powers showcased by language models, it is crucial to scrutinize whether their reasoning capabilities stem from strong generalization or merely exposure to relevant data. As opposed to constructing increasingly complex logic, this paper probes into the boolean logic, the root capability of a logical reasoner. We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested boolean logic, a task that humans can handle with ease. To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method Curriculum Logical Reasoning (Clr), where we augment the training data with nested boolean logic chain step-by-step, and program the training from simpler logical patterns gradually to harder ones. This new training paradigm allows language models to effectively generalize to much harder and longer-hop logic, which can hardly be learned through naive training. Furthermore, we show that boolean logic is a great foundation for improving the subsequent general logical tasks.

Empower Nested Boolean Logic via Self-Supervised Curriculum Learning
Hongqiu Wu and Linfeng Liu and Hai Zhao 1.2" and Min Zhang
'Department of Computer Science and Engineering, Shanghai Jiao Tong University
KKey Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China School of Computer Science and Technology, Soochow University, Suzhou, China HHarbin Institute of Technology, Shenzhen, China hinng/iiii,,nneeegniueeeeeeeeeee edu. cn, zwaohaiaccaaaae sjtu. edu.cn,
mi minzhang@suda edu. cn
Abstract
Reading Comprehension
Algebra
Vel joan kows Jurest to accout huw quicklyy eucapes 1t Waljom
An secopiaze fliee along the tese miden aggars at
Beyond the great cognitive powers showcased by language models, it is crucial to scrutinize whether their reasoning capabilities stem from strong generalization or merely exposure to rel- evant data. As opposed to constructing increas- ingly complex logic, this paper probes into the boolean logic, the root capability of a logical reasoner. We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested boolean logic, task that hu- mans can handle with ease. To empower lan- guage models with this fundamental capabil- ity, this paper proposes new self-supervised learning method Curriculum Logical Reason- ing (CLR), where we augment the training data with nested boolean logic chain hee--by-step and program the training from simpler logi- cal patterns gradually to harder ones. This new training paradigm allows language models to effectively generalize to much harder and longer-hop logie, which can hardly be learned through naive training. Furthermore, we show that boolean logic is a great foundation for im- proving the subsequent general logical tasks!
the help of file hindee ookss us itew le is later proved pouness joun returns to hin to
102 za/he.
Find the arerage speed of the plane sccund the siald?
clother and hidee bebind the dace whee Jasaet. comes looking for him.
Aterage speed = 151.5 An/hr
Game of
Faljeaze de a...
Dee 4 sumbace ta 15, 131
Open-domain QA
bastc arichaacte operacsone (--9))
Way Gons ungel
(13- (110-4)
One theorry
Emur....
Eas!!
La reit fo. Ln not
3 ia nos h- sa nos
16. Trre or valonr
True cz Esise?
Nested Boolcan Logic
Figure 1: While language models capable of han- dling a range of complex logical tasks, they do not perform well on more basic nested boolean logic.
arithmetical reasoning (Ling al., 2017).
While this is charming, these over- over- parameterizeed language models are shown to be good at exploiting superficial statistical cues achieve decent scores on end tasks (Zhou et al., 2021; Sanyal et al., 2022a;
1 Introduction
Wu et al., 2023b). Early on BERT, it found that simply by adding a "not" to the claims, BERT would be fooled into a random selector (Niven and Kao, 2019). It is time to go back and scrutinize whether the state-of-the- the- PLMs master solid log- ical capability, as truly powerful logical reasoners.
Artificial intelligence has made a giant leap from perception to cogmition, with powerful pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Clark et al., 2020; Raf- fel al., 2020; Brown al., 2020; He al., 2021b), large language models (LLMs) (Chung et al., 2022: Chowdhery et al., 2022: OpenAI, 2023) demon- strating human-level comprehension and reasoning powers on a series of challenging tasks like com- monsense reasoning (Zellers et al., 2019), open- domain -anwwriinn (Mihaylov et al., 2018), "Corresponding author: This paper was partially supported by Joint Research Project of Yangtze River Delta Science and Technology Innovation Community (No. 2022CSJGG1400.
Rather than creating even more complex logic, this paper concentrates on the root level of logi- cal reasoning- boolean logic, in Figure 1. Any logic can be reduced to a combination of multiple boolean operations, including negation -, intersec- tion A, and union V. In this paper, we introduce a new probing method to quantify the boolean logi- cal reasoning of a language model, fine-grained to different levels of logical difficulty.
htttps: http://ggthhbhhooo//e. wwt.:///////gaaaaeeeeeeeee com/giingaaa//oollilll
However, our results show that none of PLMs

---

Complex Lagk: 2...
Boolcan Logie:
Nested Boolcan: <(~A)4)
The earth
Original sample
-X V
So: The earth is flat.
Convert context-question
Is So true or false?
So: The earth is flat.
Add nested boolean logic
S1: So is a false statement.
NOT only
S2: Si is false statement.
Boolean Logic:
Nested Bookean:
Complex Logle:
Ss: S2 is true statement.
-A
Coniiner
((AA)V)
Cantior S8
Is Sy true or false?
So: The carth is flat.
Add nested boolean logic NOT & AND & OR
S1: So is a false statement.
S2: S1 is a false statement.
Ss: Either S2 S1 is a true statement.
/ Ss: Both S2 and S1 are true statements.
Figure 2: Overview of Curriculum Logical Reasoning.
Is Sa true or false?
Table 1: Method augment arbitrary samples with nested boolean logic.
possess the necessary proficiency to tackle the multiple nesting of (multi-nestedd) simple boolean operations, even the tate-offthheart models like DeBERTa-V3 (He et al., 2021a) and ChatGPT (OpenAI, 2023). Faced with more than three nested boolean operations, they quickly degenerate into random selector. even with the thain-of-though prompt (Wei al., 2022; Zhang et al., 2022b). Con- versely, this task is very simple for humans, pared other more general reasoning tasks. This raises a shadow over their generalizability ac- quired from large amount of training.
0. Given arbitrary input sample suppose that the model accurately predicts pa pa(x) y. We now define an operation 8 on x, which can be regarded as a transformation on the text, denoted as 8
2.1 From Simple Boolean Logic to Nested Boolean Logic
We concentrate on the logical operation, which specifically manipulates the underlying logical chain by transformation on the text. We present a new form of logical operation that corresponds to only boolean operators, i.e. intersection A, union V, and negation We might concentrate on the simplest negation first.
To empower the language models with such a fundamental capability in nested boolean logic, we propose new self-supervised training paradigm, Curriculum Logical Reasoning (CLR), inspired by curriculum learning (Bengio al., 2009). Con- cretely, we construct the nested boolean logic step- by-step from simple to hard on top of the original training samples in a self sulf-super manner (De- vlin et al., 2019). The model is encouraged to start with learning simple logical patterns and then move forward to hard ones gradually, rather than learning hard logic with single leap. We find that recalling simpler logic while learning harder logic can result in better outcome. Our experiments demonstrate that CLR significantly enhances the logical learn- ing process. Excitingly, pre-learning boolean logic acts as a great foundation step to further enhance the subsequent logical end tasks, like ReClor and DREAM. Figure 2 illustrates CLR very lively.
Suppose that the input statement x entails fact f. which can be either a true fact or a false fact, represented by yo- The logical process can be for- mulated as = yo. where = refers to "implies that" and yo {0, 1} (0 for True and for False). We illustrate a toy example our logical op- eration in Table 1. First, the model is required to discriminate whether the stated fact in x is true or false. It states false fact "the earth is flat", so yo (False). Next, transfer it to context- question template and denote the context Sp. It is still binary classification and the answer for it is limited True or False. This template can be applied to arbitrary tasks. For instance, a sentiment analysis sentence "cold movic" can be rewritten to a statement like "cold movie expresses a positive movie watching".
2 Introducing Nested Boolean Logic
This section presents method to introduce multi- nested boolean logic existing data.
Our idea is craft series of statements after Sg- Each statement asserts the truth falsity of the previous statement, which is uniformly chosen. We denote such a statement as boolean statement, and
We first present the notations. Let x denote the input text, with its ground truth y, and pa denote the classifier (e.g. a language model) with parameters

---

ask the model discriminate the final statement. For instance, yo and S asserts So false, so y1 should be negated, 1/1 0. After deduction, we can obtain y/3 1.
Thee consequence is that they can get the final sult without correctly classifying the original fact. Eq. and exclude this potential threat and fo- cus entirely on the model's capability in handling nested boolean logic. In other words, if the model reasons from a misclassified fact, its final result can be noisy, misleading the analysis.
Logically, the assertion of "true" results in no change of the current logic and the assertion of "false" results in negation. å can be nested for k times without affecting the fact in x:
Hence, we are interested in two metrics:
Clean accuracy (clean%): It refers to the gen- eral accuracy score.
II
6 x yk
Boolean accuracy (boolean %): It refers to the accuracy only calculated those samples where the model accurately discriminates the original fact, as represented in Eq. 3 and 4. This can only be calculated on augmented data.
where Yi denotes each intermediate answer after times of boolean statements and yk denotes the eventual answer. We denote Eq. as multi-nested boolean logic.
Obtaining final ys is free of external annotation, as in self-supervised learning, by programming the following recursion:
3 Benchmark
To benchmark the multi- boolean logic, we construct a new dataset in this paper and following experiments are based on this. As apart from other datasets, it composed of series of subsets, rep- resenting different levels of logical complexity. We will release this benchmark for future research.
9i 6; asserts false
y/
-
(2)
yi-1 & asserts truc
Such multi-nested boolean logic poses little chal- lenge to humans. We hopefully assume that strong language model can tackle that as well.
3.1 Data Collection
We generalize the negation operation to other boolean operations as in the bottom of Table 1. Concretely, we uniformly choose one statement from S to Sy and uppend it with either "and" or "or" chosen uniformly.
We collect the raw data from SciTail (Khot et al., 2018), a scientific text entailment dataset with a premise and a hypothesis for each sample, which is labeled as entail or nol entail. We join the premise and hypothesis together to make them "fact", with the entailed pair labeled True and not entailed one labeled as False. Some samples are shown in Appendix A. Eventually, we get 6,000 raw samples and randomly sample 1,000 of them as the test set with the rest as the training set.
2.2 Quantify Boolean Logic
We probe the mastery in nested boolean logic of language model by measuring its performance against our boolean statements. An ideal logical reasoner supposed to make clear logical transi- tions between truth and falsity. We are particularly interested in this situation: the model accurately discriminates the original fact, while falters in delivering the correct answer subsequent to k boolean statements. This can be formulated as:
On top of the raw data, we convert it to the context-question format and then impose boolean statements to generate the adversarial set, which means that the resultant samples are likely to fool the model (Zellers al., 2018. 2019). Specifically, we uniformly choose a value k from some range and insert & boolean statements following the origi- nal sample. The range of bounds the minimal and maximal nesting of boolean logic on each sample, and larger value of & suggests more nesting on the logic chain. For instance, the samples in Table 1 correspond to k 0 and h = 3 (see Appendix A), We denote this benchmark as BoolKill, in which each sample a logic chain started with a potential fact and followed by a series of boolean statements. is worth noting that BoolKill is a group of sets
Po I
8; x
+ yk
(3)
where pa satisfies:
Pe (x) yo.
(4)
Deep neural models are good at exploiting super- ficial features rather than delving into the entire semantics (Wu et al., 2023a; Sanyal et al., 2022a).

---

DeBERTa-bas
DeBERTa-large
GPT2-1.56
each model on un, u1, u2, and ug (ūg), indicat- ing the number of nested boolean operations.
96.4
98.1
96.0
96.7
97.8
96.8
We evaluate two statc-of-the-ar PLMs:
DeBERTa-Ve (He et al., 202 of the
Table Performances on raw data and its templated up-
strongest BERT-style language models;
ChatGPT (OpenAI, 2023): the strongest large
for different levels of logical difficulty, and each level has its own training and test set. We use the
language model, as powerful zero-shot learner.
ChatGPT shows an impressive ability to follow human instructions and we directly evaluate it on the test sets For DeBERTa, we first fine-tune it on the ug training set and evaluate it on the ug test, NOT: We curve the results in Figure 3. We find that each model exhibits high performance U1, suggesting their proficiency in tackling sin- gle boolean logic. DeBERTa performs better than ChatGPT, probably due task specific fine-tuning. However, as the nesting increases, each model suf- fers from a notable decline regardless of size. For instance in (a), starting from 1/2 in which the sam- ples are suffixed by only two boolean statements, DeBERTa-bar falls to 53.8% while DeBERTa- large falls to 65.4%. From ug, strong as DeBERTa- large, it leans to random selector, whose accuracy gets close to 50%. Similar situations can be seen on ChatGPT, while its degradation is more gentle. It suggests that even state-of-the-art models possess a critical limitation in the basic nested boolean logic, only able to handle up to three nested operations. This is far below humans' level.
following notations to spot them:
row: the raw data in which each sample is
statement of a fact;
ug: the clean set in which each raw sample is only transferred to context-question template, with semanties unchanged;
Ug the adversarial set constructed on top of uo in which each sample is suffixed by k boolean statements;
uky the adversarial set in which each sam- ple is suffixed by kn ~ l2 boolean statements; ūkz u is negation-only, and we use ū to distinguish the adversarial set additionally con- taining AND and OR.
3.2 Data Bias
The first thing verify is whether uo is semanti- cally equivalent raw. From Table 2, we find that each model achieves very close performances on raw and ug, suggesting that the context-question template does not induce bias to the original data.
AND & OR: From (b), it is counter-intuitive that DeBERTa performs better on sets additionally including AND and OR. We conjecture that the model utilizes the inherent bias that AND + False and OR = True in majority of cases. Such a shortcut is particularly useful when k is small. Interestingly from (d), well-trained ChatGPT appears not to use this, and its performance drops even faster on ū. Therefore, we foeus û and û with large k in the following experiments.
The average sentence length will vary due to the boolean statements on raw data, which grows linearly from 36 to 88, from i1 to üg. The overall statistics of BoolKill are Appendix A.
To minimize the bias between subsets, we keep the ratio of positive and negative samples to 1:1 in all subsets. Additionally, BoolKill semi- annotated dataset, comprising auman-annotated facts and synthetic boolean statements. The lat- ter introduces several figh-frequency words like "true", "false", and "statement", which may induce large bias if these words do not occur in balance in data. For instance, the model may make the de- cision based the relative number of "true" and "false" in the sentence. Hence, also keep the occurrence of "true" and "false" the same for both positive and negative samples all subsets.
Chain-of-Thought (CoT) (Wei et al., 2022: Zhang et al., 2022b; Yao et al., 2023) is proven to be an effective prompt method to amplify the reasoning ability of LLMs, with asking them to offer the procedure while performing the reasoning. From Figure (c) and (d), we find that ChatGPT performs better with the assistance of COT. How- ever, we raise a criticism in the paper: does CoT promote logical reasoning? Indeed, our study show that CoT may bring new logical concern. We will
3.3 Evaluation Results
We report the thorough results on each level of logi- cal difficulty on BoolKill. We sequentially evaluate
We use the API from openai. The backbone model is 3.5-bb5-.

---

Boolan (%)
-hage
Boolean (%) 8.
-aage
Boolan (%)
wi CbT
Ucolean (%)
1 5
wi CoT
Number of seited
Number of nested
Number of nested
Number of nested
(a) DeBERTa-V3 V3 (ug)
(b) DeBERTa-V3 (ix)
(c) ChatGPT (ur)
(d) ChatGPT (ūs))
Figure 3: Boolean accuracy of different models with increasing numbers of nested boolean operations (ug/ūg).
further discuss in Sec. 1.
samples in the new turn of training benefits the eventual performance, which potentially reminds the model of what it learns previously. Our ultimate goal is that the model can gradually learn to tackle more complex logic that it has not before.
4
Empower Nested Boolean Logic
We present superviee learning manner.
4.1 Self-Supervised Learning
Empirical Results
The straightforward method is to fine-tune the model on BoolKill. The concept behind is to se- quentially introduce boolean statements top of some corpus and let the model learn tackle multi- nested boolean logic self self-superviselly.
As opposed to the prior section, where we evaluate the model on each level of logical difficulty, in this section, we evaluate each model on BoolKill alternative. These sets
W1-1, U5-8; and U5-8 cover the range from k 1 to 8. un-4 is a simpler one and Ug and ūş-s appear be highly challenging, since we previously show that state-of- the-art PLMs are almost powerless for the nested boolean logic beyond 1/..
However, we find language models struggle to fit the samples in BoolKill when the potential logic within the data is too hard, and still be random selector. It indicates that naive training is not the best therapy to learn complex logical patterns.
We experiment on DeBERTa- V3-Ta- and DeBERTa-V3-large Each model is trained for 3,000 steps with a batch size of 16 and learning rate of 2e-5 le-5 for the base / large one.
4.2 Curriculum Logical Reasoning
Inspired by Curriculum Learning (Bengio et al., 2009), where the machine learning model is en- couraged to learn the task starting with easier sam- ples and ending with harder ones, we propose Cur- riculum Logical Reasoning (CI.)) to enhance the process of learning logical reasoning.
To verify CLR, we report two experiments. the first experiment, we compare different training settings and evaluate the models on BoolKill. In the second, we leverage the boolean logic in BoolKill to benefit other general logical tasks.
There is a natural match between curriculum learning and logical philosophy, because the logic chain is a step-by-step progression from single to complex. CLR means that, rather than learning hard logic from scratch, the model starts with learning simpler logic, e.g. single boolean logic, and then moves forward harder logic gradually, e.g. multi- nested boolean logic.
5.1 Nested Boolean Logic
The results across various BoolKill sets sum- marized in Table We find that naively training the model on ug-s only produces random accuracy scores on all three test sets, even on two simpler ones ug and W1-44 While on uo-1 and ug-44 we find that DeBERTa-V3-large can achieve better out- comes on simpler u1-4 compared to DeBERTa-V3- base. It suggests that larger model possibly has greater learning ability to handle more nested boolean operations, but it is still very hard even for strong DeBERTa-V3-lare logical patterns in U5 within single leap.
We show concrete instance. We start to train the model on No-1, which solely includes single boolean operations. Next, train such model on u0-2, which further includes two-nested boolean operations. This gradual progression continues until the model is trained on 00-4- The above procedure can denoted as up~1 + u0-2 + Uo-3 - 4p We find that reusing the easier
learn very difficult
However, CLR brings significant performance

---

Ug
Wind
MG-88 boolean%
Ms--8
ReClor
DREAM
clean
sp 58.2
79.9
DeBLRTa- V3-base
DeBERTa -V3-base
uo
sp
59.0
80.2
U5-08
50.2
46.6
49.6
51.2
up-11 sp
61.6 13.4
82.0 -2.1
NAIVE
100+4
96.0
71.5
53.6
53.4
10011 110-2
sp
62.6 14.4
82.8 -2.9
Ug
94.6
72.2
52.9
56.0
DeBERTa
71.4
90.4
00
96.2
64.6
49.0
50.5
-V3-large
10
110-22
ep
74.8 13.4
92.5 -2.1
CLR
Ug
0g--
ug-2
96.1
96.4
96.3
96.8
94.6*
89.6
79.2
57.6 T
70.0 T
56.7
66.8
60.7
-7h (LoRA)
LLaMA2
00-2
6p
55.4
61.6 pa.
86.9
85.1
0g-44
95.8
97.4
77.5
73.0
1g-11
DeBERTa- V3 large
DeBERTa
Mo-1
96.6
55,2
48.6
51.3
50.8
-V3-buse
sp M0-1
L1.3
97.8 .0.3
NAIVE
96.4
97.9
61.9
54.7
ug--4
96.1
77.6
51.7
60.8
Table 4: Results on general MRC tasks. "sp" refers to the task-specific training and we evaluate the model
Ug
97.7
70.2
52.7
48.7
1g--2
98.0
87.0
60.7 T
55.2
on the corresponding test set.
CLR
+ Ug-vi
97.7
98.5
71.1 T
59.3
Ug
97.6
99.4
84.3 T
68.2
0g-4
97.3
99.5
81.9
82.3
(Yu et al., 2020), reasoning-required MRC col- lected from graduate admission exams; DREAM (Sun et al., 2019), dialoguc-based MRC. Con- cretely, we first train DeBERTa- V3 BoolKill an initialization and then fine-tune on the task- specific data of ReClor and DREAM.
Table 3: Results on BoolKill, comparing CLR with naive training. We use "->" to denote the curriculum setting we perform, where model inherits the trained weights from the last level. We highlight the step-by- step performance gains CLR brings with "p"
The results are shown in Table 4. We find that learning boolean logic acts as a nice initialization for the subsequent reasoning tasks on both ReClor and DREAM. For instance, initializing with u0-1 improves SaBERTa-V3-base by 3,4% compared to naive fine-tuning on ReClor, and u6-1 - uo-2 further improves by 4.4%. It is worth noting that to alone does not provide any useful signals (59.0% on ReClor and 80.2% on DREAM), suggesting that it is the boolean logic that we add into the data that enhances the eventual logical performance.
boosts on every model and every test set. Its ad- vantages are especially significant on harder u5-8 and u5-8. For instance on DeBERTa-V3-large achieves an impressive boolean accuracy of 84.3% on U5-8 and 82.3% on u5-8. uplifting naive train- ing by about 30%, also kceping high clean accu- racy of 97.6% and 97.3% on Up- It is worth noting that the model has ever seen the hard samples in Uz-8 and ūة-g during training, and Ct.R effectively generalizes the model to unseen logical patterns. Additionally, all models consistently maintain a strong accuracy on Mo throughout the process of CLR, suggesting that they learn to discriminate the original facts and tackle boolean logic simultane- ously. As a contrast, naive self-supervised training leads to inferior up results.
As contrast, we first train the model on task- specific data and then fine-tune it on boolean logic. We find that more complex logic in ReClor or DREAM does not enable the model to perform any better on u0~1 or even harms it, confirming our initial idea, that the model may ignore the basic logic during training, even if it appears to handle more complex problems sometimes.
Moreover, we find that each level of curriculum brings a considerable improvement to the model. For instance, the performance of DeBERTa- V3- base has outperformed all naive baselines when it just completes the second level of training on uo-2-
is the generic form of CLR to pre-learn boolean logic and then learn complex logic.
5.3 Ablation Study
5.2 Boolean Benefits Complex Logic
The ablation study is made under negation-only sets. We first discuss the composition of levels to make up the curriculum perform CLR. We remove some levels from the full curriculum setting up~1 + ul-2 + up-3 + u0 Additionally, we include another strong baseline by merging all the
Boolean logic acts the atomic component of logic. Our intuition is that it can solidify more general end tasks that require complex logical rea- soning. We conduct validation on two machine reading comprehension (MRC) datasets: ReClor

---

No clean%
01-4
MG-8
Mi-4
Ma-1
boolear%
clean% %
boolean's %
NAI
Mo-1,".
up-4
95.5
95.8
66.5
GPT2-1.55
CLR
H0-l
96.3
96.8
79.2
NAI
dgwll Mgwd
93.8
99.2
65.6
NAI
CLR
Ug-1 - M0-
M0-1; M0-3
95.8
95.4
92.7
86,6
66.4
55.7
CLR
Ug.a
M0w2
00-1
94.1
95.6
94.4
74.0
98.6
84.6
52.8
71.2
55.8
NAT
MO-2, NO-4
95.8
60.1
51.1
94.3
99.9 10.7
79.4 +13.8
CLR
ug-22 - M0w4
90.8
82.9
55.8
OPT-7b (LoRA)
NAI
Ugh1, Ng.
+ 95.9
M0-1
96.2
95.6
64.6
95.6
89.9
49.0
65.6
57.7
NAI
(g-11
M0-4
Md-1
93.3
94.3
68.7
98.0
54,2
63.8
CLR
+ 95.3
94.3
64.8
CLR
udw2
94.3
78.2
53.0
+ 95.6
96.5
72.2
U0-3
94.7
97.9
64.5
Ug
95.5
98.8 10.8
69.4 +5.6
Table 5: Ablation study on DeBERTa-V3-baal We omit the notations of dy-2 and Ug-3 in "...".
LLaMA2-7b (LoRA)
NAI
M0wd
97.3
99.4
67.9
M0
96.3
64.3
48.3
CLR
Ug-2
97.6
86.2
51.8
u0-3
97.7
98.6
67.8
training sets together, c.g. U0-1, 40-22 40~33 U0-4 and performing naive training. The difference is that CLR strategically samples the training data from easy ones hard ones rather than uniformly. The results are summarized in Table 5. We find that any leap from the full curriculum can result in a notable performance drop, highlighting the importance of a complete and gradual progression of logical learning. Interestingly, we also find that learning from simpler M0-11 -> M0-3 achieves a better outcome compared to harder up-2 u0-44
97.6
99.9
10.3
75.9 ts.u
Table 6: Results of LLMs, including the efficient fine- tuning method (LoRA).
method (LoRA) (Hu et al., 2022).
From Table 6, we find that CLR works very well on GPT2-1.5b, achieving a boolean accuracy of 79.4% on Us-8, outperforming naive training by a notable margin of 13.8%. However, larger-scaled OPT-7b does not yield better results as expected. Specifically, it achieves comparable results on sim- pler #1~4, while greatly lags behind much smaller GPT2-1.5b harder 05-88 We conjecture that parameter efficient fine-tuning might compromise the acquisition of complex reasoning capability, e.g. multi-nested boolcan logic, leading to a non- negligible performance drop.
Next, we discuss the composition of samples for each level. We remove the simpler samples that belong to the prior level (to-1 + U2 - Uy + u4) and see whether the model would forget what it has learned before as result. From Table 5, we find that the removal process gives comparable results ug and t1-44 However, when it comes harder Ug--8 it leads to performance drop of 6%. These findings underscore the importance of reusing simpler samples when stepping forward to the new level, especially when evaluating on harder or even unseen data like U5-8-
Further Discussion
6.1 Chain-of-Thoughe
We discuss CoT in more detail, and this part is par- ticularly geared to the current LLMs. It has been shown that when being asked to give the procedure, the model can perform the reasoning more pre- cisely. In the prior section, we show that CoT can assist ChatGPT in achieving better performance BoolKill. We notice that the intermediate thinking procedure exposed by CoT is equally important. A ideal reasoner can not only make the final answer but also reasonable intermediate results.
5.4 Fine-tuning Large Language Models
We also evaluate our method on LLMs. However, fine-tuning LLMs requires a huge amount of re- sources. As a compromise, recent studies propose several efficient fine-tuning methods that only up- date small ratio of parameters within LLMs. We experiment on three models, GPT2-1.5b (Brown al., 2020), OPT-7b (Zhang al., 2022a), and LLaMA2-7b (Touvron et al., 2023). They both be- long to the decoder-only architecture as ChatGPT. We fine-tune GPT2-1.5b with full parameters and fine-tune the 7b models with the low rank adaption
However, we find that ChatGPT leans to fall into inconsistent deduction when giving the intermedi- ate results, as we illustrate in Table 7.
For [a], we can first obtain that S3 is false from

---

the previous statements since So is true. Hence. the deduction of the model up this step cor- rect. In the next step, however, the model draws an incorrect conclusion based on the fact that S3 is false, that is S4 is true. Indeed, 54 should be false since it doesn't I match S3, incurring wrong final answer. Similar cases can be found in [b]. These cases indicate that when making longer reasoning, ChatGPT can fall into mistake in some logical step, even though each step very easy when cutting individually.
True/False
DeBEKTa-V3-Ban DeBERIa-V V3-ias--
sp 51.4
58.2
boolean sp
57.8
62.6
DeBERTa-V3 -l-lage
sp
67.1
71.4
DeBERTa- V--laage boolean
ap
73.4
74.8
Table 8: Results on true-or-false questions ReClor.
with nested boolean logic showcase a significant improvement, achieving 6.4% and 6.3% points of gain respectively.
7 Related Work
[a] Query
S0: A crust is a portion of a world.
The study of boolean operations is the fundamen- tal requirement for series of challenging tasks, e.g. arithmetical reasoning (Ling et al., 2017), com- monsense reasoning (Zellers et al., 2019), read- ing comprehension (Yang et al., 2018), dialogue comprehension (Sun et al., 2019). We concentrate on the multi-nested boolean logic by augmenting the text with boolean statements. Previous stud- ies analyze more general logical reasoning, e.g. RICA (Zhou et al., 2021), RobustLR (Sanyal al., 2022a), FaiRR (Sanyal al., 2022b), by logical paraphrase or contrast sets.
S1: SO is a false statement.
S2: S1 is a false statement.
S3: $2 is a false statement.
S4: S3 is a true statement.
Is S4 true or false? Let's think step by step.
Answer
$3: S2 is fälse statement. It contradicts the prior assessment $2 true, $3 is false.
$4: S3 is true statement.
It contradicts the prior assessment S3 false, so S4 is true.
1 Query
S0: Humans sometimes eat clouds.
S1: S0 is a true statement.
Self-supervised learning methods typically gen- erate learnable inputs on top of unlabeled corpora, e.g. by masking (Devlin et al., 2019), insertion (Wu et al., 2022), sentence reordering (Lan et al., 2020), contrastive learning (Gao et al., 2021), while our method by imposing series of sentences to the suffix, actually generating learnable logic. We in- troduce curriculum learning (Bengio et al., 2009), which allows the model to learn step by step to fur- ther facilitate self-supervised learning. Curriculum learning is under-discussed in context of language processing (Xu et al., 2020; Lee et al., 2022).
S2: S1 is false statement.
S3: $2 is truc statement.
$4: S3 is true statement.
S5: $4 is false statement.
Is $5 true false? Give the procedure.
[6] Answer
$1: S0 is a true statement. (Truc assuming S0 truc)
S2: S1 is a false statement, (False because S1 is true)
S3: $2 is a true statement. (True because $2 is false)
S4: S3 is a true statement. (False because $3 true)
Therefore, S5 is true based on the evaluation.
Table 7: ChatGPT case study. S4 in [a] should be false.
While deep neural networks are capable of han- dling very complex tasks, in reality they lean to ploit spurious cues (Goodfellow et al., 2015; Madry et al., 2018; Wu al., 2023a), and can be powerless to very simple perturbations as a consequence. Our work discloses that language models are poorly skilled at basic boolean logic. In parallel, studies show that language models can be easily fooled by some naive patterns within the text, e.g. lexi- cal overlap (McCoy et al., 2019; Wu al., 2023c), entity boundary ( (Yang et al., 2023), word order (Zhang et al., 2019).
6.2
True False
We take a further look at true-or-false questions, a specific and common question type in MRC and logical end tasks. Specifically, we filter out the sam- ples with questions that contain keywords "true" or "false". In ReClor, there are 173 such samples out of the 500 in its development set. The evaluation results on true-or-false questions are shown in Ta- ble 8. We find that both DeBERTa models strug- gle with seemingly simple true-or-false questions, showing lower accuracy compared to the overall performance. However, the models pre-learned
We also release challenging benchmark to eval- uate boolean logical reasoning. There are a series of work focusing on constructing challenging logic,

---

e.g. ReClor (Yu et al., 2020), HotporQA (Yang et al., 2018), ANLI (Nie et al., 2020).
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mad- eling with pathways. CORR, abs/2204.02311.
8 Conclusion
This paper provides a quantified analysis on the multi-nested boolean logic. We flag the deficiency in the hate-of-the-ane language models in terms of such basic capability, which will inevitably cause pitfalls in dealing with more complex reasoning tasks. For this, we propose Curriculum Logical Reasoning, a new self-supervised learning method to empower language models with foundational logical capability. We also show that our idea can act as a cornerstone learning method for general logical reasoning.
Limitations
We cannot exhaust all the arrangements of cur- riculum to perform CLR, which could potentially achieve even better performances. We have dis- cussed the potential risk of chain-of-though as sec- ondary contribution of our work, which will be interesting to study in the future. Our method to introduce nested boolean logic is general, while our experiments are based on one source. Another option is to collect data from more general corpus specific domains of interest, which is promising. Eventually, we do not have enough resources to run large language models above 7b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi. Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetune language models. CoRR, abs/2210. 11416.
Kevin Clark, Minh-Thang Thang Luong, Quoe V. Le, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
References
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Con- ference on Machine Learning, ICML 2009, Montreal, Quebec. Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 41-48, ACM.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre- pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Lunguage Tech- nologies, NAACI-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers). pages 4171-4186, Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NewrIPS 2020. December 6-12, 2020, virtual.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 6894- 6910. Association for Computational Linguistics.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adver- sarial examples. In 3rd International Conference on

---

Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OvenReview.net.
Pengcheng Hc, Jianfeng Gao, and Weizhu Chen. 2021a. 3ebertav3: Improving deberta using electra-style pre- training with gradient-disentangled emhedding shar- ing. CoRR, abs/211 11.11..000 1.09543.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3428-3444 Association for Computational Linguis- ties.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 202 Deberta: decodding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? A new dataset for open book question an- swering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2381-2391. Association for Computational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Sym- posium on Educationnl Advances in Artificial Intel- ligence (FAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5189 0197. AAAI Press.
question answering.
Yixin Nic, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1885-4901. Association for Computational Linguisties.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, 2020. ALBERT: A lite BERT for sulf-super learning of language representations. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Timothy Niven and Hung- Yu Kao. 2019. Probing neu- ral network comprehension of natural language ar- guments. Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Vol- ume I: Long Papers, pages for Computational Linguistics.
Association
Mingyu Lee, Jun-Hyung Park, Junho Kim, Kang-Min Kim, and Kenn Lee. 2022. Efficient pre-training of masked language model via concept-based curricu- lum masking. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLPP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pages 7417-7427. Association for Computational Linguistics.
OpenAl. 2023. GPT-4 technical report. CORR, abs/2303.08774.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:67
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30- August 4, Volume 1: Long Papers, pages 158-167, Association for Computational Linguistics.
Soumya Sanyal, Zeyi Liao, and Xiang Ren. 2022a. Ro- bustir: A diagnostic benchmark for evaluating logical robustness deductive reasoners. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 9614 11416963. Association for Computational Linguis- tics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Soumya Sanyal, Harman Singh, and Xiang Ren. 2022b. Fairr: Faithful and robust deductive reasoning over natural language. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2022, Dublin,
Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 11692.

---

Ireland, May 22-27, 2022. pages 1075-1093, Associ- ation for Computational Linguistics.
Hongqiu Wu, Shaohua Zhang, Yuchen Zhang, and Hai Zhao. 2023c. Rethinking masked language model- ing for chinese spelling correction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pages 10743-10756. Association for Computational Lin- guistics.
Kai Sun, Dian Yu. Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A chal- lenge dataset and models for dialogue-based reading comprehension. Trans. Assoc. Comput. Linguistics, 7:217-231.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie. Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.0925
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understand- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4095-6104 Association for Computational Linguisties.
Yifei Yang, Hongqiu Wu, and Hai Zhao. 2023, Attack named entity recognition by entity boundary interfer- ence. CoRR, abs/2305.00225.
Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpoiqa: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369. Association for Computational Linguistics.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran. Thomas L. Griffiths, Yuan Cao. and Karthik Narasimban. 2023. Tree of thoughts: Deliberate problem solving with large language models. CORR, Ahs/2305.10601.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain-of-though prompt- ing elicits reasoning in large language models. In NeurlPS.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset re- quiring logical reasoning. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Hongqiu Wu, Ruixue Ding, Hai Zhao Boli Chen, Pengjun Xie, Fei Huang, and Min Zhang. 2022. Forg- ing multiple training objectives for pre-trained lan- guage models via meta-learning. In Findings of the Association for Computational Linguistics: EMNIP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 6454 6454-6466. Association for Com- putational Linguisties.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale ad adversarial dataset for grounded commonsense inference. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgiun, October 31 November 4, 2018 pages 104. As- sociation for Computational I Linguistics.
Hongqiu WU, Ruixue Ding, Hai Zhao, Pengjun Xie, Fei Huang, and Min Zhang. 2023a. Adversarial self-attention for language understanding. In Thirty- Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 13727-13735. AAAI Press.
Seventh AAAI Conference
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791-4800. Association for Computational Linguis- tics.
Hongqiu Wu, Yongxiang Liu, Hanwen Shi, Hai Zhao, and Min Zhang. 2023b. Toward adversarial training on contextualized language representation. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin,

---

Todor Mihaylow, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022a. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01068.
A BoolKill
Task
PN ratio
Size
Length
Vocab
BoolKill
binary
6000
36~-8
14315
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: paraphrase adversaries from word scrambling. Proceedings of 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Téchnologies, NAACL-HLI' 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 1298- 1308. Association for Computational Linguistics.
Table 9: Statistics of BoolKill.
The planet Mercury the closest planets the Sun. [Hypothesis] Mereury is closest the sun. [Label] Entail
[Premise]
SeiTail
SO: planet Mercury the closest
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompting in large language models. CoRR, abs/2210.03493..
Context- question (k=0)
planets the Sun.
So, Mercury is closest the sun.
SO true or false? [abell]
Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho. Jay Pujara, and Xiang Ren. 2021. RICA: evaluating robust inference capabilities based commonsense axioms. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November. 2021. pages 7560-7579. Association for Computa- tional Linguistics.
SO: The planet Mercury the closest of
the planets the Sun.
BoolKill (k=1)
So, Mercury is closest the
S1 false statement.
Is S1 false?
[Label] False
S0: The planet Mercury is the closest of the plancts the Sun.
So, Mercury is closest to the sun.
BoolKill (k=3)
S1 is false statement.
S2 is true statement.
S3 is false statement.
Is S3 true false?
[Label] True
Table 10: Illustration of some samples from Biol


Reviews:
[{"review_id": "JG3wsuZbmF", "reviewer": "Reviewer_ZqCq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n**R1:** The proposed training process is complex\n\nPlease allow us to clarify that our proposed **Curriculum Logical Reasoning (CLR) is a simple and easy-to-use training method.**\nFollowing our method, the nested boolean logic data can be obtained on top of almost any types of sentences (e.g. NLI, sentiment analysis), without additional annotation. Then, we train the model from easy level to hard level. The difficulty of all samples can be easily distinguished by the number of nesting.\n\nWe will also share our data to further facilitate this process.\n\n**R2:** Potential trade-offs between boolean pre-training and general pre-training\n\nFirst, we agree that it would be interesting to study the combination of logical pre-training and general pre-training in future study. Thank you for your comments!\n\nHowever, we will answer your question from two perspectives.\n\n1. **CLR is a continual training process after general pre-training like MLM**. Its purpose is to enhance the logical capability of LMs (e.g. performing better on logical end tasks). **Respectfully, the trade-off between the subsequent logical capability and general capability is not a critical issue,** to the best of our knowledge.\n\nIndeed, we acknowledge that any specialized training may compromise the general capability of an LM, **while this gap is not within the scope of our paper.**\n\n2. In addition, CLR is better suited for the transfer learning in a multi-task setting as in T5. We assume that the ability to handle multiple tasks is also a general capability. In this case, logical reasoning is a crucial sub-task or capability. **Considering that our method greatly benefits tasks like ReClor and DREAM, it is highly promising that it will not compromise other tasks to a large extent.**\n\nAgain, thank you for your question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-structured and provides a clear and thorough investigation into the logical reasoning capabilities of pre-trained language models. The proposed Clr method is a promising approach to enhance the logical reasoning abilities of language models, and the experimental results provide strong evidence to support this claim. Overall, the paper offers valuable insights into the limitations of pre-trained language models and proposes a new training method to address this issue. Future research can build on this work to further explore the potential of this approach in enhancing the logical reasoning capabilities of language models.", "reasons_to_reject": "The proposed training process is complex, and it is worth considering how to integrate it seamlessly with the general pre-training process without compromising the model's ability to learn general capabilities. The paper provides valuable insights into enhancing the logical reasoning capabilities of language models, but the potential impact on the model's ability to learn general capabilities should be further explored. Future research can investigate how to balance the proposed training method with the general pre-training process to ensure that the model can learn both fundamental logical capabilities and general capabilities. Overall, the paper offers a promising approach to enhance the logical reasoning abilities of language models, but further research is needed to address the potential trade-offs between specialized and general capabilities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates whether pre-trained language models possess true logical reasoning capabilities and finds that they struggle with multi-nested boolean logic tasks. To address this issue, the authors propose a new self-supervised learning method called Clr, which gradually increases the complexity of logical patterns to enable language models to effectively generalize to much harder and longer-hop logic. The paper provides valuable insights into the limitations of pre-trained language models and proposes a new training method to empower language models with fundamental logical capabilities.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bUCPtWvWox", "reviewer": "Reviewer_hbvC", "summary": "", "questions": "- I was wondering whether you have analyzed the complex tasks as to how much of the reasoning capabilities from Boolkill are required in their question types and whether those are actually the questions that are getting answered correctly after using the pretrained models. (If the model struggled with negation before, did the pertaining help with that?!)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and appreciation of our work.\n\n**Q1:** Whether we have analyzed the question type\n\nThank you for your great question!\nWe take a closer look at ReClor. We filter out the samples with the questions that contain the keywords \"true\" or \"false\".\nThere are 173 such samples out of the 500 in its dev set.\n\nThe results are below: (\"sp\" refers to task-specific training data)\n\n| | | True/False | All |\n| :---- | :----: | :----: | :----: |\n| DeBERTa-base | sp | 51.4 | 58.2 |\n| DeBERTa-base | boolean + sp | **57.8** | 62.6 |\n| DeBERTa-large | sp | 67.1 | 71.4 |\n| DeBERTa-large | boolean + sp | **73.4** | 74.8 |\n\n**It suggests that pre-learning nested boolean logic greatly enhances the performance of the model on \"True/False\"-type questions.**\n\nWe promise to add this part of the experiment in our new revision. Thank you for your comments!\n\n\nLine 231 typo\n\nThank you for your correction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow despite the included technical terms, the entangled process, and multi-aspect contributions.\n- The results showcased in the paper strongly suggest their approach's effectiveness.", "reasons_to_reject": " - I do not find any reason to reject!", "questions_to_authors": "- I was wondering whether you have analyzed the complex tasks as to how much of the reasoning capabilities from Boolkill are required in their question types and whether those are actually the questions that are getting answered correctly after using the pretrained models. (If the model struggled with negation before, did the pertaining help with that?!)", "paper_topic_and_main_contributions": "This paper has two interesting sections, where it first evaluates the ability of language models to do boolean reasoning (not, and, or) via a newly created benchmark. Then, they propose a Curriculum-based training mechanism based on their proposed benchmark, which can enhance the ability of the same evaluated models on the reasoning tasks. They further showcase that pretraining on their dataset with their proposed mechanism can even enhance the ability of LMs on harder reasoning tasks. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 231: I think it should be k boolean statements rather than `x`. "}, "extra_text": {}}, {"review_id": "TJWD20UdEC", "reviewer": "Reviewer_sk6i", "summary": "", "questions": "1. Why do the authors choose OPT-7B instead of LLaMA-7B for the experiments? \n2. Could you explain a bit more about the following conclusion in Section 5.3: \"These findings underscore the importance of reusing simpler samples when stepping forward to the new level, especially when evaluating on harder or even unseen data like u5∼8.\"?\n3. Could you verify whether the conclusion \"Learning boolean logic serves as a foundational step that further improves subsequent tasks like ReClor and DREAM.\" holds for large language models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n**Q1 & 3**: Results on LLaMA-7b\n\nThank you for your suggestion. We have made the experiments on the latest LLaMA 2 model. Please see the results below, which highly support the conclusions in our paper.\n\n| | | $u_0$ | $u_{1\\sim4}$ | $u_{5\\sim8}$ |\n| :---- | ----: | :----: | :----: | :---- |\n| OPT-7b (LoRA) | |\n| Naïve | $u_{0\\sim1}$, $u_{0\\sim2}$, $u_{0\\sim3}$, $u_{0\\sim4}$ | 94.3 | 98.0 | 63.8 |\n| CLR | $u_{0\\sim1}$ | 93.3 | 68.7 | 54.2 |\n| | -> $u_{0\\sim2}$ | 94.3 | 78.2 | 53.0 |\n| | -> $u_{0\\sim3}$ | 94.7 | 97.9 | 64.5 |\n| | -> $u_{0\\sim4}$ | **95.5** | **98.8** | **69.4**  $\\uparrow5.6$ |\n| LLaMA2-7b (LoRA) | |\n| Naïve |  $u_{0\\sim1}$, $u_{0\\sim2}$, $u_{0\\sim3}$, $u_{0\\sim4}$ | 97.3 | 99.4 | 67.9 |\n| CLR | $u_{0\\sim1}$ | 96.3 | 64.3 | 48.3 |\n| | -> $u_{0\\sim2}$ | 97.6 | 86.2 | 51.8 |\n| | -> $u_{0\\sim3}$ | **97.7** | 98.9 | 67.8 |\n| | -> $u_{0\\sim4}$ | 97.6 | **99.9** | **74.9** $\\uparrow7.0$ |\n\nFurthermore, pre-learning nested boolean logic benefits both subsequent DREAM and ReClor on LLaMA2-7b.\n\n| | | ReClor | DREAM |\n| :---- | :----: | :----: | :----: |\n| DeBERTa-large | sp | 71.4 | 90.4 |\n| DeBERTa-large | boolean + sp | 74.8 | 92.5 |\n| LLaMA2-7b (LoRA) | sp | 55.4 | 85.1 |\n| LLaMA2-7b (LoRA) | boolean + sp | **61.6** | **86.9** |\n\nWe will add these results in our new revision and hope these supplementary results can further enhance the soundness of our paper. Thank you.\n\n**Q2:** Explain more on \"These findings underscore the importance of reusing simpler samples…\"\n\nWe find that it is important to remind the model of what it previously learned when training on new samples. The improvement is especially significant when evaluating the model on harder logics. For instance, when the model is trained on $u_3$ (3-nested boolean logic), we incorporate $u_1$ and $u_2$ samples together into the training. The resultant model generalizes better to harder and unseen $u_{5\\sim8}$. **Briefly, this strategy mitigates the forgetfulness in model training when performing CLR.**\n\n\n**R1:** Encompassing numerous aspects simultaneously affects the cohesion of the paper\n\n**We respectfully disagree with your opinion that our paper is not well-organized due to numerous aspects included, while our paper is recognized well-structured and easy to follow by multiple reviewers (e.g. ZqCq and hbvC).** Of course, we will keep trying our best to make our paper clear to all readers.\n\nPlease allow us to re-clarify the contributions of our paper. **Our contributions are multi-aspect.**\n\n1.\tWe introduce a new benchmark and offer a comprehensive analysis on boolean logical capability of PLMs/LLMs.\n\n2.\tWe propose a new self-supervised learning method to enhance logical learning.\n\n3.\tOur method further enhances the logical capability on general logical tasks.\n\n4.\tAdditional insights are provided, e.g. on CoT.\n\n**All these aspects above are very closely related to each other,** especially every posterior aspect can be right built on the basis of prior one, **which are tied to be a comprehensive entirety by nature,** serving for the core theme of our paper, logical reasoning.\n\n**We are glad to see your acknowledgement on the comprehensiveness of our work. If such acknowledgement makes sense, then there is no way such comprehensiveness making our work not sound.**\n\nSurely our work cannot be perfect, thanks for your comments to let our work better.\n\n**R2:** Related work\n\n**Due to our multi-aspect contributions, our related work should also involve multiple aspects of work, which is what we have done in the paper**, e.g. logical reasoning, self-supervised learning, curriculum learning, some benchmarks.\n\n**R3:** Small scale experiment in additional exploration of in-context learning on ChatGPT\n\n**To reduce the variance of the results, we perform three rounds of the same experiment on ChatGPT.** We conjecture that CLR facilitates in-context learning by empowering the model with more structured and diverse samples, e.g. easy to difficult, compared to all difficult.\n\nHowever, this is our additional exploration of CLR, **which is not the main contribution of our work.** While we realize that this part of the study is not complete, **our primary intention is to shed light to future research,** e.g. the better way to organize the samples for in-context learning.\n\n**We have already discussed this point in our Limitations section.** Thank you for your question.\n\n**R4:** Is GPT2-1.5b a large language model\n\nTypically, LLMs $\\ge$ 7b, thus we experiment on OPT-7b in the paper.  **Our motivation to include GPT2-1.5b in our experiment is to compare the difference between LoRA and full-parameter fine-tuning,** since we do not have enough resources to conduct full-parameter fine-tuning on models above 7b.\n\n**Surely we have reported the LLaMA2-7b (LoRA) results as a supplementary for our experiment.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author built the BoolKill benchmark for the evaluation of Nested Boolean Logic and conducted comprehensive experiments to show the improvement across different sizes of language models. \n2. The author found that Learning boolean logic serves as a foundational step that further improves subsequent tasks like ReClor and DREAM based on the experiments on DeBERTa.", "reasons_to_reject": "1. The authors of this paper made an earnest attempt to encompass numerous aspects simultaneously. However, the overall cohesion of the paper lacks a distinct logical progression. I would strongly advise the authors to reconsider the organization of their presentation.\n2. In need of improvement is the section dedicated to related work (Section 7), which appears somewhat disorganized. I recommend that the authors consider revising Section 7 to enhance its logical coherence. \n3. In Section 6.2, the authors' evaluation was based on only 10 samples. This limited sample size may have resulted in biased findings due to the small scale of the experiment.\n4. I am concerned about 5.4 Fine-tuning Large Language Models regarding the model size. Is GPT2-1.5B qualified for LLMs?", "questions_to_authors": "1. Why do the authors choose OPT-7B instead of LLaMA-7B for the experiments? \n2. Could you explain a bit more about the following conclusion in Section 5.3: \"These findings underscore the importance of reusing simpler samples when stepping forward to the new level, especially when evaluating on harder or even unseen data like u5∼8.\"?\n3. Could you verify whether the conclusion \"Learning boolean logic serves as a foundational step that further improves subsequent tasks like ReClor and DREAM.\" holds for large language models?", "paper_topic_and_main_contributions": "**Paper Topic:**\nThe paper focuses on probing the logical reasoning capabilities of pre-trained language models, particularly their understanding and handling of multi-nested boolean logic. The study investigates the apparent gap between language models and human capabilities in reasoning and proposes a method to enhance the models' logical proficiency.\n\n**Main Contributions:**\n1. **Analysis of Existing Models**: The paper discovers that existing pre-trained language models, even large ones, behave like random selectors when faced with multi-nested boolean logic, which humans can handle with ease.\n\n2. **Introduction of Curriculum Logical Reasoning (CLR)**: A new self-supervised learning method called Curriculum Logical Reasoning (CLR) is proposed. CLR augments training data with multi-nested boolean logical chains step-by-step, programming the training with simpler logical patterns before gradually moving to harder ones. This enables the model to learn much more complex and longer logic chains, which would be challenging through naive training.\n\n3. **Enhanced Performance through CLR**: Experiments demonstrate that CLR significantly enhances the logical learning process in language models, allowing them to handle more complex reasoning tasks.\n\n5. **Additional Findings**: The paper also presents several additional insights:\n   - Learning harder logic while recalling simpler logic leads to better outcomes.\n   - A chain-of-thought approach exposes basic logical flaws but can improve final answers.\n   - Progressively offering samples from simple to hard stabilizes the performance of in-context learning.\n\nOverall, the paper offers a comprehensive analysis of the limitations in current language models' reasoning capabilities and provides a new approach to improving them through a graduated and systematic learning paradigm. I will consider to raise scores if the authors address my concerns.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N.A.", "Typos_Grammar_Style_and_Presentation_Improvements": "Certain sentences appear unnatural, such as the following example: \"When being asked to give the procedure, the model can perform more precisely the tasks that require logical reasoning.\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the logical reasoning capabilities (or lack of thereof) of pre-trained language models, specifically in their comprehension and handling of multi-nested boolean logic. It addresses the disparity between language models and human reasoning skills and introduces a method to improve the models' logical proficiency. The study introduces a new benchmark and a curriculum-learning approach based on it. Reviewers found the work well-structured, clear and thorough, and having strong and comprehensive experimental results. Reviewers raised some minor issues concerning how catastrophic forgetting of the vanilla language modelling task could impact the performance of resulting models, and also the size of LLMs used. This issues were addressed during the rebuttals.
Decision: Accept-Main