NAIL: A Challenging Benchmark for Na\"ive Logical Reasoning
Authors: Xinbo Zhang
Changzhi Sun
Yue Zhang
Lei Li
Hao Zhou
Venue: ICLR 2022 Submitted (2022)
Keywords: Logical Reasoning
Benchmark

Abstract: Logical reasoning over natural text is an important capability towards human level intelligence.
Existing datasets are either limited and inadequate to train and evaluate logical reasoning capability (e.g., LogiQA and ReClor),
or not oriented for logical reasoning (e.g., SQuAD and HotpotQA).
In this paper, we focus on a specific category of logical reasoning, named \emph{\mytask}, and propose a new large scale benchmark, named \mydata, targeted for learning and evaluating models' capabilities towards \mytask.
 \mydata is source from  standardized exams such as Chinese National Civil Servants Examination and Law School Admission Test.
Furthermore, to collect more data, we propose to imitate the example of standardized exams rather than designing them from scratch.
\mydata is available in both Chinese and English containing a total of $10,296 * 2$ instances.
Empirical results show that current state-of-the-art neural models struggle on \mydata with very poor accuracy (the best result is 30.10\% for \mydata and 36.15\% for Chinese \mydata), while human experts can perform nearly 100\% accuracy.
Further results indicate that human imitations can significantly help models learn logic from natural text.

Under review as a conference paper at ICLR 2022
NAIL: A CHALLENGING BENCHMARK FOR NAÏVE LOGICAL REASONING
Anonymous authors Paper under doublc-blind review
ABSTRACT
Logical reasoning over natural text is an important capability towards human level intelligence. Existing datasets are either limited and inadequate to train and evaluate logical reasoning capability (c.g.. LogiQA and ReClor), not oriented for logical reasoning (c.g., SQuAD and HotpotQA). In this paper, we focus on a specific category of logical reasoning. named naive logical reasoning, and propose new large-scale benchmark, named NAIL, aiming to help models learn and evaluate naive logical reasoning capability. NAIL sourced from standardized exams such as Chinese National Civil Servants Examination and Law School Admission Test, Furthermore, collect more data, we propose to imitate examples of standardized exams rather than designing them from scratch. NAIL is available in both Chinese and English containing total of 10, 296 instances. Empirical results show that state-of-the-ar neural models struggle on NAIL with very poor accuracy (the best result is 30.15 1% for NAIL and 36.2% for Chinese NAII), while human experts can perform 100% accuracy. Further results indicate that human imitations can significantly help models learn naïve logical reasoning ability.
INTRODUCTION
Current deep models have achieved near human-level performance on many tasks in NLP (Devlin) et al. 2019; Liu et al. 2019), and more often than not, superficial knowledge suffices to solve the problems. To move towards human intelligence, we need equip the models with logical reasoning capabilities (e.g., ability to draw logical conclusions from given statements), which is also a long sought-after goal of Al (Newell & Simon, 1956 McCarthy et al. 1960). One related task is natural language inference whose goal is to assign the logical relationships (contrad itd.dddddddddddeeo neutra ral and antaaii Iment) to sentence pairs. To push the development of models in logical reasoning, the researchers have focused on more challenging reading comprehension tasks, which often require more complex reasoning as well as longer input. However, most existing reading comprehension datasets are not oriented for the logical reasoning (e.g.. SQuAD and HotpotQA), with the exception of LogiQA (Liu et al. 2020), ReClor (Yu et al., 2020 and LR-LSAT (Wang et al. 2021).
Above datasets (LogiQA, ReClor and LR-LSAT) are limited and inadequate to train and evaluate logical reasoning capability. The reason that all of the three datasets involve diverse types of logical reasoning, such drawing an alternate conclusion the argument, finding necessary/sufficien assumptions, whether statements strengthen/weet weaken the argument or explain/resolve the situation. Mixing multiple types of logical reasoning may pose the following challenges. a). From the perspec- tive of human cognition, different types of logical reasoning correspond to different problem-solving ideas. But in practice we usually train a model on the whole dataset with the same idea, which makes it more limited for models to learn different logical reasoning capability. b). From the perspective of machine learning, if there are many reasoning types mixed in a dataset, then there will be less data for each reasoning type, which is inadequate to train and evaluate logical reasoning capability (demonstrated in our experiments). Furthermore, when the model does not work, it is difficult to determine which reasoning type is the bottleneck (no reasoning type annotation in the dataset), which may hinder the design of better models.
To tackle the challenges, focus on more fine- ine-grained type of logical reasoning, named naïve logical reasoning (Section[)). which is to infer the logical conclusion from statements that describe triples (subject, predicate, object) and the relationships among them. A typical example of naïve

---

Under review as a conference paper at ICLR 2022
22 a " Mo "Clé "D [There four pretty girls, nameely A, B, C, and D, all of whom love to dress beautifully. One day, four girls get dressed in different colors: yellow, green, blue, and red. When asked about the color, said that B did not dress in yellow; B said that not dress in green; said that did not dress and she the only who told the truth.)
Paragraph: A, B,C, D 31 B * RIœT 13,9, 2
*
Aid "B + "BU
"Ditt #.00.
C= A 1
blue; said that among B, and
girl dressed
P Color of clothes
Q: An A X
A & A1 told the truth, then which the
following statements
correct?)
A. CR) 22 @.
DM
90 is dressed in green, and D dressed red.)
B. A
red.) (B dressed blue, and C dressed in red.)
[A dressed in green,
C. B
D. D
A #
(D dressed green, and A dress in red.)
A truth
8 told the truth
told the truth
Color clothes
Color of clothes
P
Color of clothes
S
S
Y
G
R
O
Y
G
B
G
B
R
A
0
1
A
0
A
0
1
0
B
0
0
0/1
1
0
B
0
0
0
C
0/1
0
C
1
0
0
D
0
0
D
0 0
1
Figure 1: An example of NAIL requiring naive logical reasoning. Highlighted option is the correct answer. Tables depict the reasoning process from the human perspective. indicates uncertain state. 0 and indicate false and true respectively. Red grid with "0/1" means there is conflict.
logical reasoning is shown in Figure [ answer this query, we need to iteratively derive conclusions according to the conditions, and stop the searching branch if conflict occurs. Specifically, in the initial table, we only know that "D not green" from "D told the truth". Assume that "A told the truth", and infer that "A is green, C is non- non-green". Next derive "C is green" from "C told the lie". Then is derived as both green and non-green, thus causing a conflict. Similar processes for assuming "B told the truth" and "C told the truth". It takes extensive training and practice for human brains to cope with such complex logical reasoning.
Inspired by the datasets extracted from standardized examinations (Lai al. 2017 Clark et al. 2018 Liu et al. 2020), we build a new large-scale benchmark, NAIL, by selecting naïve logical reasoning examples from standardized exams such the National Civil Servants Examination of China and Law School Admission Test. However, such examples are limited in their number, as it takes efforts for human experts design these questions. To collect more data, we propose imitate examples of standardized exams rather than designing them from scratch. Unlike simple data augmentation (e.g.. substitution, paraphrasing), human imitation aims at getting more diverse examples while maintaining the underlying logic (Figure D. NAIL. is available in both Chinese and English, containing total of 10, 296 instances.
Empirical results show that state-of-the-art neural models struggle NAIL with very poor accuracy (the best result is 30. % for NAIL and 36. 2% for Chinese NAIL), while human experts can perform 100% accuracy. Further results indicate that human imitations can significantly help models learn naïve logical ability from natural text.
NAÏVE LOGICAL REASONING
The naïve logical reasoning is a more fine- fine-grained type of logical reasoning. Formally, we give the definition as follows.
The subject a described resource, usually an entity, such as person or location. The predicate indicates an altribute of the subject or indicates some kind of relationship between the subject and the object. When denoting an astribute, the object is the attribute value, usually a literal value, e.g., (Jacket, Color, Red), otherwise the object is entity, e.g., (Beijing, Capital, China).
Definition 1.
Definition 2. Assuming the set of subjects S, the set of predicates P. the set of objects O and several statements, each statement describes triple (subject, predicate, object) or some logical relationship
2

---

Under review as a conference paper at ICLR 2022
between triples. The naive logical reasoning is the process of reasoning from these statements to reach a logical conclusion.
In this work, we explore naive logical reasoning in the form of reading comprehension. A typical example and detailed reasoning process is shown in Figure Similar to the format of multiple-choice reading comprehension, it contains context, a query and four options with only correct answer. To solve the problem, the model needs to understand the logical connections between the subjects, predicates and objects, and then derive valid option.
3 NAIL: DATA COLLECTION AND ANALYSIS
NATI, is a carefully designed benchmark for naive logical reasoning similar to the format of multiple- choice reading comprehension. Inspired by the datasets extracted from standardized examinations (Lai et al. 2017 Clark et al. 2018 Liu et al. 2020), we first collect a small amount of examples from examinations, denoted as NAIL-E 1Section3.11 Then we propose to imitate examples of NAIL-E to collect more data, denoted as NAIL-I (Section Finally we provide a detailed analysis of the proposed NAIL (Section B.3).
3.1 COLLECTION FROM EXAMINATION
We searched for such examples widely from two different types of public examinations: Chinese National Civil Servants Examination (CNCSE) and Law School Admission Test (LSAT). CNCSE CNCSSE a once-a-year competitive examination in China, and there are overall 120-140 examples per exam per year. But only 1-4 examples belong to the scope of naïve logical reasoning, which is also the most difficult type of problems for candidates within the given 120 minutes. And the LSAT is standardized test for prospective law school candidates in the United States, Canada, and growing number of other countries. Logical Reasoning multiple-choice section of LSAT, containing 24-26 questions under the limitation of 35 minutes, where about 2-4 problems fall into naive logical reasoning category.
We artificially selected examples that belong to the naïve logical reasoning category from the above two examinations. Finally we obtained 488 examples from the last 25 years of CNCSE and LSAT in the last 30 years, denoted NAIL-E. These two exams are from countries with different native languages. The former is expressed in Chinese, and the latter is expressed in English. And note that there are slight difference in language style between them. And for diversity and fairness, in later step we get all examples in both Chinese and English through translation.
3.2 COLLECTION FROM IMITATION
After collecting small amount of NAIL-E, we expect to expand the number of the dataset. Designing examples from scratch requires a huge effort from human experts. One simple solution is data augmentation, which artificially scales up data by creating modified data from existing data, such as word/sentence shuffling, word replacement and syntactic variation. However, the data augmented in this way highly correlated with the original data, and the model easily captures these semantic surface correlations, which makes it limited to train and evaluate logical reasoning capability. To alleviate this limitation, we propose to imitate examples NAIL-E, to create more examples with a diverse semantic surface while keeping the underlying logic of the original example. Furthermore, in the process of human imitation, we design strict strategies to control the quality of imitation.
Imitation Example See Figure the context of an example from NAIL-E consists of sentences logical template (see "backbone template"). The example focus on the description of scenario: pi sicking Pri noce Cha rming (m,1), which involves five entities: Iii_Na (81). Wang_Wei (Pt), Wu_Gang (p2),
and a query (split by blank lines), each of which can be represented
Li-Qiang (pa), Liu_Dawei (pt), and three noun/adjective propertics: tall (a1), handsome (a2), a_PhD (az). An accepted imitation needs to keep original underlying logic but have seman- tically different groundings, i.e., to describe a completely different scenario. Imitation 1 and
'The term naive refers to the fact that this logical reasoning process human in this task is spontancous, intuitive and unsystematic.
https:/ www.. lsac. org/
cal reasoni ig
3

---

Under review as a conference paper at ICLR 2022
Backbone Template
Original Example
Imitation 1
Imitation 2
Imitation 3
G- m, #1' F9-EE ds, dg. and at.
35th--199 11 SUFF
19 H, 457 SI
#HI. *T 1 M:
*HI. *AT, 81 e2- M=. (Bun Yi's Prince
*HE #18. 4 th- #44. (Sun YI's Prince
#HE: 88t. # #1 #-
(sy's is my is supposed to be: By dg. and az.)
SE.8. (Xiao Li's ideal gift
(Li Ne's Prince Charming tall,
Charming supposed be: handsome, and PhD.)
Charming is supposed to rich, house, and has car.)
supposed to be: expensive, veluable to collect, and nice-looking,)
supposed handsome, and a Phi.)
S SRI 4: 317 +. 24 # -it 1 & 10
SUREE. R..
$
SU3 RR. *%. 3:4R
BD) (p Jte4
1. (t
10:99
# S 97 ft.
ft.
(She knows type oftr
(She knows men. Wang Wel, Wu Gang. Qlang and Liu Dawei, and only one meets all her requirements.)
(She knows
(She knows men, Qu Hong, Mel Zhen, Jl Fan,
(She knows gifts, bracelets, watches, bangles, and rings, and only one meets all her requirements.)
Pt. Ps,Pe only meets s
Heng. Mel Zhen. Fan,
and Zhang Lei, and only one maats all her requirements.)
and Zhang Lei, and only one moots all her requirements.)
raquirements.)
(.B) (p # A 3 (HM)) it 01 472 (MM) I dz. #1 (#)) Mag. (Among
(14442. 16 T-28 1
(1) 41 1: 11.
(1) 40. #1. 88 1441114 (Among the 4 mon, are rich, 2 car house.)
(1) 43 1 15.
(Among the men, 3 are are PhDs and handsome.)
(Among the 4 mon, 3 are tall, Phis is handsome.)
th (Among the 4 gifts. 3
type_oftnn.
are
ai, 2 at, and is az-)
nice-looking and valuable to collect:)
40 p:t (paadd pyare both a..)
(2) A 35 #E (Qu Hong and Mai Zhan are both PhDs.)
(2) M # (Qu
(2) AU 2 2t.
(Wang Wel and Wu Gang
Heng and Mei Zhen both have car.)
(Bracelets and watches are both nice-looking))
are both PhDs.)
Pa P. aTa at
(3) #t H (Zhang Lei and Fan are
(3) # W 1981. (Zhang Lei and Ji Fan are of the same wealth.)
(3) A Witttee H9H (Rings and bangles are of the same price.)
(peand of the attributo of..))
(Liu Dawei and Li Qiang
are of the same height.)
the same height.)
ps p. 11 #1
(4) #
(4) #6 A5
(4) # 4 to (Either Fan Hong is rich.)
(4) 7F T 13 #t0 (Elther Bangles bracelats is expensive.)
(Either P. P. a.))
(Either Qlang
TJ. (Either JI Fan Heng tall.)
Wang Wei tall.)
# i
10 + *++
it 3: #f *+++
if 9 8:44 *++ meets requirements of Sun Yi?)
in / R+ (Which meets requirements of Xiao Li?)
#1
(Who meots all the requirements of s, ?)
(Who meets
(Who meets all the
requirements of Li Na?)
requirements of Sun Yi?)
A. p.
A. #btiiiu Dawei)
A. KZZhnn Lei)
A. Z.Sakn Lei)
A. ARiidk(Rings)
2. Qiang)
B. $ N(JI Fan)
B. # N1JJ Fan)
B. (Banglas)
GPz D.P1
C. (Wu Gang)
R (Mei Zhen)
66-6% Zhen)
(Wachee)
D. (Wang Wel)
D. M (Qu Heng)
D. (uu Hong)
D. ((raccellsss))
Figure 2: An original example from CNCSE and its three imitations, which share the same backbone template. Imitation subject-level. Imitation 2 subject-and--objcctleedd And Imitation is subject-and annd-dedicate-aeee object-level.
are unqualified imitations. Since Imitation only conducts subject-level imitations (S imitations), iee, only superficially substitute the five entities ($1:Li _Na->Sun_YY Pt:Wang_Wee Beeeiii-__ennneee M.sWu.Gann-Mei.2.. ..heei,,enn,pa P3:Li Qiang->Ji_Faae itation 2 conducts subject-and-objjettleed imitations (SO imitations), i.e., not only substituting the entities, but also altering the corresponding objects (property values). The difference between Imi- tation 2 and the original example is much greater than that between Imitation and and the raw problem, however, Imitation 2 is still not an ideal imitation. Furthermore, Imitation 3 changes the scenario m into pi ck i ng i ideal gift. conducts -nd--ttddidd--tee--aa- -bjeee imitations (SPO imitations). Note that all of the three imitations share the same logic templates. And P2 is the answer for all of the original example and above three imitations.
Further, Im-
ft. Imitation 3 is an expected imitative writing, which
Imitation Process We first select a group of people from variety of occupations: professional editors, legal practitioners, in-service civil servants and college students (from different majors). Empirically. people in these occupations have strong logical and verbal skills. These people are asked to conduct SPO imitations based on original examples from NAIL-E, and reasonable imitations should meet two requirements: logic invariance and semuntic diversity. We trained these candidate people how to conduct SPO imitations such as Figure 2 official imitation phase, in which process we eliminated some people of poor quality. Finally we employed 82 qualified people] to imitatively construct problems based on NAIL-E, and they are paid Averagely, it costs a trained person about 3-4 minutes to finish an imitation:
We then conducted a trial phase before the
RMBY2.8 per imitation dl starting from coming up a scenario, then replacing the subjects, predicates, and objects of the raw
1 Consssstnng of 42 native Chinese speakers and 40 native English speakers. Native speakers of language will be assigned imitation tasks expressed hy that language.
'A part-time employee can produce 15-20 imitations per hour, where he/she can get RMBY42-RMBBSS while the local minimum wage is RMB423 per hour.
4

---

Under review as a conference paper at ICLR 2022
problem with those icenario-related while keeping invariant logic, and finally smoothing the new sentence with some transition words if necessary. For each original example in NAIL-E, we expect at least 20 imitations (except for extremely difficult cases). Overall we use 813 paid work hours in total to build the NAII.-I
Imitation Quality Control We adopt following strategies to ensure the quality of imitations:
I. As mentioned above, we conducted trial phase before the official imitation phase. In this phase, we asked them to imitate a small number of problems. Although we do not necessarily need them to write the backbone template, we will check the logic and give feedback to help them understand the task. This process was iterated for three rounds. Only those who passed the trial phase could participate in the official imitation.
2. During the official imitation, we set up an online chat room to communicate with employees and answer their questions timely.
3. To embrace semantic diversity, each original example is shown to at least 5 people, that is, one can only conduct imitations based on one original example. People who are assigned to the same example imitate independently without interference from each other, to ensure varied inspiration for imitation.
4. To ensure logic invariance, we adopt double-checking strategy:
Cross Checking: Everyday, for each employee, we sample 5 imitations from all of his/her daily imitations. And the sampled imitation is assigned to other 2 employees for cross-checking, The imitation will be qualified only if they both approved. and the criterion for approval is that the imitation share the same logics with the original example. If any one of the imitations produced by one employee fail, then all imitations of that employee for that day will be returned to re-check and repair.
To further ensure that the underlying logic do not deviate during imitation, we introduce another team of experts to solve the imitative examples. The team consists of 20 experienced experts. 10 of them speaks Chinese as native language and have passed CNCSE, while the other 10 speaks English and have passed LSAT.. Fl Each imitative example was presented to 3 experts randomly, who are allowed to select one and only choice from "A", "B", "C", "D", otherwise, "UNABLE_TE SUNABLE.TO_ANSERR
Post Checking:
bugs exist in the example, causing no correct choice or multiple correct choices. As long as one of the 3 experts pointed out "UNABLE. TT_TT____ TO_ANSWER", then the imitator of this problem should recheck the logic, until each of these 3 experts could give a choice from "A", "B", "C",""" Note that in the post checking process, we broke up the imitations together and shuffle randomly, otherwise if a person is faced with imitations from same original example, he/she is prone to give a shorteut option with speculation.
Translation Quality Control After collecting high-quality mono-collections, we first adopted Google Translation to translate Chinese/English collections into another language, and then employed 10 professional bilingual experts in Chinese and English for manual correction. Bilingual experts were asked to pay attention to logie-invariance and faithfulness during translation. Next, to ensure translation quality, we also adopted the post checking strategy. That is, we asked the 20 human experts mentioned above to solve the translated examples. Each translated example was presented to 3 experts randomly. Since human experts excel in solving naive logical reasoning problems, (i.c. achieve 100% accuracy on mono-collections) if any expert made a mistake on a translated sample or pointed out "UNABLE TO ANSWER", the translated instance is sent back to the bilingual experts for revision. Finally, we asked 50 native speakers to read through all paragraphs of the translation parts in NAIL and mark "0"/"1" for each, where "1" stands for translated sample is idiomatic, and "O" otherwise. Then for all samples marked with "0" (about 20%), the bilingual experts and native speakers will work together to polish them and conform the target language norms.
Human Evaluation As mentioned above, an imitation is finally regarded qualified only if the sampled 3 experts could all solve the example. For any original example in NAIL-F,, we also ask 3 experts in the team to solve it. Since the gold answers to examples in NAIL-E are provided in public by the examination committee, and corresponding imitations in NAIL-I share the same answer with the original example. Therefore, we calculated the mean accuracy of these three submissions on the overall NAIL, denoted as the performance of human experts. To better demonstrate the
$ Experts who are native English speakers will be assigned English problems and experts who are native Chinese speakers will be assigned Chinese prohlems.
5

---

Under review as a conference paper at ICLR 2022
Paragraph Query Answers
Reasoning Type
Paragraph: 28t XIEN TH19E, TE ASHINA.
CIGANTTA " 1 7 MAAIESCC CITIBAA IA 11 11-2 J i&, "R "e.eeinntt mf
KKN. s1-ut #2AATIIR 48 CiMiiEae VM &e
mAlll 50il Ei 8580. #ftttt 12 4 Uff A.N.E.. (Three cites, namely City A B, and C, took part in competiton the tile " dvilied city, four were making their precictions, Judge said that ether A or would win. Judge that if and C
"Ri "Bitt HaB. Ciitiitt A FiBRR..
imposablle would
win, Judie
that
could not
won, would only judge
Real/Fake Mixture (36.03%)
Judge
prodicted correctly)
a: theee k "C"M"""" # (aakill A2? (Mtich city won the life?)
A. AllfCCity
B. Brt/City
CiTyCyyy CJ
Arh B.HHiChhhCCyABBB
Paragraph: F.. 2.80% ( -2 -LAAL F. A498t: I RINTM ARSNAAAFFF AFFF wum A
province, and province. New following information giver: oleer than one who comes from Hubel; A one who comes from Henan are different in their age; the one who comes from Henen is younger than B.)
places: Shandong province,
Q 19/E3 (Which of the following statements can be inferred from the pessage?)
Ordering (20.40%)
A. IF A-M-slll: XJ/A not from Hubei.)
A sitaat (yThe
from Honan younger than
REN Eall A Arehe ane from Henan is older
Shandong
NI1E AAHEA 12 The one from Hubel is the youngest
Paragraph: 1p. Z. REE. W WARRRRA Iff:. BA 50 ER: T. Z.7-Ai
different jobs (Beling, Narjing. Changdu) in different dities (a doctor, an actor: and tnachar), and now the following information is given. A docs not work in Beijing, B does not work Nanjing: The one who works in Beijing not teacher, The one who works Nanjing is dodtor, is not an actor.)
(tllat 2 A1PnE 150WEEL Z.E-BR R. (A,
Matching (29.26)
Q 822, Z. 2LRS I.?? (So, where
and work respectivel?))
N. 2.J. Maninng Chengdu Beijing)
Beijing and Nanjing)
C.
W: R. A20t Nariin Beiling and Chargsuu)
D.
M4 RUJ (Chengdu, Narjing and Beijing)
Paragraph: 8ili
90, 44 Witlele M.A Rrt 10 Aull.
Tr AA 4P Aii.. frL2"
4:30 AL 7 95 R. 1. " A A:73. (An elementary school in City has complicated Hukou (a system household registration in Chinal lseues. students grade registered in the City Some the students in migrant workers, who registered
grade registered
City The
students grade
Set Operation (14.31%)
Q: stdl.. in Me (Which of the following statements can be inferred from the material?)
A. FF -e: 22 8 1-8 12 I Weal students grade 2 are not chidren of migrant workers)
B. TREeR F-38 f 7338 (some migrant workers' children are grade
wes "
dectug (some
registered in chidrren migrant workers)
grade 3)
We 979% 8-2 RFC T. (some students grade
Figure 3: The percentage and representative examples of each naive logical reasoning type.
difficulty of NAIL,, we also selected another team consisting of 20 first-year college students. Same as above, an example is shown to 3 students randomly and they have to give the answer independently. We calculated the overall mean accuracy as the performance of human baseline. The evaluation of human baseline is paid separately. Each person can receive RMBY1.5 for answering each example, which generally cost the person 2-3 minutes.
Since we hire part-time people to write imitatively rather than using existing crowd-sourcins platforms, we build our own website, where the quality and overall progress can be viewed at any time.
3.3 DATA ANALYSIS
As mentioned above, NAIL-E and NAIL-I together compose NAIL. NAIL-E, NAIL-I and NAIL are divided into training set, validation set and test set respectively. The overall statistics of NAIL summarized in Table It is worth noting that an original example in NAIL-E and its corresponding imitations will only be in the same data split. We analyze and manually annotate the fine-grained types of examples in the NAIL-E and group them into 4 categories including real/fake mixture, ordering, matching and set operation, whose percentages and representative examples are shown in Figure Each of these types of examples requires naïve logical reasoning.
Train(NAIL)
Dev(NAIL)
Test(NAIL)
NAIL-E
NAIL-I
NAIL-E
NAIL-I
NAIL-E
NAIL-I
# Example
292
5906
97
1904
99
1998
# Ave./Max. of paragraph
68.7 155
70.2/ 188
67.3/
70.3/210
67.0 149
73.3 185
Ave./Max. of query
8.1/74
6.9/81
9,3/71
9.3 134
7.9/92
7
# Ave./Max. of option
9,2/
9.7 / 130
9.1
8.5/134
8,6/
7.7 1.7
Table
Data split and corresponding statistics.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
We adopt the accuracy as the evaluation metric. Simple rule-based methods and strong neural- based methods are included as our baseline, Rule-based methods involve text matching and sliding window. Neural-based methods include BERT (Devlin et al., (019) and RoBERTa (Liu et al. 2019). Detailed descriptions (c.g., hyper- parrm-aarmmeerss) of the baseline models and the results on the validation set are listed in the Appendix A
NAIL
Chinese NAIL
4.1 RESULTS AND ANALYSIS
Model
Input
Dev
Test
Dev
Test
The performance of all baselines on
Random
(C. Q. A)
25.0
25.0
25.0
25.0
the NAIL is presented in Table 2 In particular, the human baseline is 71.3 and 76.4 on the test set of NAIL and
(Q, A)
23.1
24.6
23.3
24.9
Word Matching
(C, Q. A)
21.6
25.8
22.2
24.2
Chinese NAIL separately, while the human expert is 100.0 since ambigu- ous examples are not included in the dataset. In comparison, all of the baselines perform much worse than humans, demonstrating that these methods are very limited in naïve log- ical reasoning. In addition, the results are relatively similar on the English and Chinese datasets. The rule-based approaches achieve an accuracy of 25.8 and 24.2, which is similar to random guess, indicating that word correlation can not be used to help im- prove performance. Pre-trained mod- els have relatively good performance with about 4-10 points improvement compared to the rule-based approaches, showing that pre-trained models have a certain degree of commonsense and logical reasoning capabilities (Huang et al. (2019)). However, the best result by RoBERTa LARGE is 36.2 on the testing data of Chinese NAIL, which is still far below the human performance. This indicates that the knowledge in the pre-trained model is quite weak in logical reasoning.
Sliding Window
(Q, A)
23.8
24.2
24.1
24.7
(C, Q.
21.9
22.1
21.6
22.5
(A)
26.8
26.7
27.2
26.7
BERT LARGE
(Q, A)
27.3
27.1
27.4
26.8
(C, Q, A)
29.0
29.4
29.3
27.7
(A)
27.3
26.5
29.4
27.7
RoBERTa LARGE
(Q, A)
27.8
27.9
32.6
30.5
(C, Q, A)
34.6
30.1
37.3
36.2
Human baseline (C. Q. A)
70.1
71.3
75.5
76.4
Human expert
(C, Q. A)
100.0
100.0
100.0
100.0
Table 2: Main results on NAIL. (accuracy%). The column Input means whether to input context (C). query (Q) and answer options (A).
Different Input Settings We conduct experiments with different input settings. The setting of questions and answer options (Q. A) does not lead to significant improvements compared to the input setting of only answer options (A). One likely reason is that the queries usually do not provide much information, e.g., According to this, it can be deduced that? Further adding context yields a noticeable boost, showing that the context is highly informative.
Transfer Learning We conduct set of transfer learning experiments to understand the degree of overlap in terms of necessary knowledge for solving problems in our dataset and existing datasets.
What are the results if pre-trained model is first trained existing reading comprehension datasets, and then fine-tuned on NAIL? Table 3 shows the results where LogiQA, ReClor and RACE are adopted. [ Overall, we observe that when LogiQA, ReClor or RACE is regarded extra training resource for RoBERTa, the performance on NAIL will increase (30.1 11..11133/33....... 3333333331..0 since models can learn some degree of general reasoning ability when learning other comprehension tasks. While interestingly, if models are trained only on LogiQA, ReClor, or RACE, and then zero-shot to directly test on NAII., the performance is poor and close to random guess. And as for RACE, the zero-shot performance on NAII (22.0) is even far below than that of RoBERTa (23.4). We attribute this to the
'We also try seq based models (i.e. T5 (Raffel et al. 2019)) and well-designed models for other related
tasks (i.e.. DAGN (Huang al. 2021 for LogiQA and ReClor, HGN (Fang et al. 2020) for HotpotQA (Yang)
al. 2018). and QDGAT (Chen et al. 2020) for DROP (Dua et al. 2019) on our English version NAIL.
Details and results are shown in Appendix H
Forr fair comparison, in all hrnsshhencharks experiments in this paper, removed samples in the training
set that duplicates of those in the set.
7

---

Under review as a conference paper at ICLR 2022
Zero shoo--hhufneed Results
Evaluate on
NAIL
NAIL-E
NAIL-I
Logian
Train
RACE
Ø
23,4
20.8
23.5
LogiQA
25.4
39.6
24.7
ReClor
24.7
19.8
25.0
RACE
22.0
29.2
21.6
NAIL
30.1
37.4
29.7
LogiQA-NAN
34.5
38.5
34.3
ReClor-NAA
33.6
33.3
33.4
Leat-ane
16-1
RACE -NAIL
31.0
32.3
31.0
Training data
Figure Transfer learning results when evaluating on the test split of LogiQA, ReClor and RACE. Ø is the zero-shot performance of RoBERTa. is placcholder for these three datasets.
Table 3: Transfer learning results when evaluating the test split of NAIL, NAIL-E, NAIL-L RACE NAIL denotes finetuned RACE first and then finetuned on NAIL.
fact that, naïve logical reasoning is typical category of reasoning which should be significantly distinguished from RACE. Fine-tuning on RACE makes the parameters fit the RACE data, which can lead to side effects on NAIL. We believe that completely different categories of reasoning deserve to be explored separately since they may have different forms strategies of reasoning, especially when training data is insufficient.
What are the results if NAIL is used as extra training resource for existing reading comprehension tasks? Figure shows answers to this question. Generally, can draw the conclusion that using NAIL as pre-training step can significantly improve the supervised- learnning performance for other tasks, such LogiQA (35.3-3669 for test), ReClor (62.6 464.4) and RACE (83.2-885.2 indicates that NAIL can bring naïve logical reasoning ability to the model, which is a basic reasoning ability and can be reflected into other comprehension tasks. An interesting thing that for zero-shot evaluating on RACE, NAIL seems to have side effect in the pretraining process (22.6<27.8) 8). This is because similar reason as mentioned above, that NAIL and RACE consist of completely different categories of examples. Results of more datasets are shown in Appendix D.2
This
Fine-grained Types We further analyze the model performance with respect to different types of naive logical reasoning (Figure 5). We find that language models perform well set operation problems, while struggle on matching and ordering. We think that language models can provide good representation of set object, even if models do not really reason derived from the context.
BERT
RoBERTb
RoBERTa
RoBERT
421
213
252
2EA
Realfake
Matching operation
Realioake
Ordeling
Realfake
Set
Figure 5: Accuracy against reasoning types evaluated on different testing sets.
Error Analysis We further perform detailed analysis of human- errors and model errors, while from Table 2 we can observe that human baseline on NAIL (around 70%) is much lower than that of human expert (100%), and RoBERTa performs much worse (above 30%). We measure the accuracy against several factors on the test set of NAIL: number of sentences in the backbone template of the context, number of possible worlds. and context length.
See Figure 6 a), as the number of sentences the backbone template decreases, the accuracy rate of human baseline increases significantll, and when reduced and below, human baseline does
report result on validation set ReClor, since gold answers test are not acquired. 9hhe number of possible worlds is denoted as, the number of subjects x the number of predicates x the number of objects.
8

---

Under review as a conference paper at ICLR 2022
haliilarrge rids
100
100
1700
Figure 6: (a). X-axis: the number of sentences in the backbone template. The height of bars: Number of samples the test set grouped by X-axis. The number bars: The accuracy against X-axis. (b). Accuracy human baseline and RoBERTa against the number of possible worlds in the context. b)#k" in the x-axis denotes there are k samples in the test set where a S number_ of possible.uorlds-// worlds<b. (c). Number of mistakes made by human baseline and RoBERTa model against various context length.
not make any mistakes, while performance of RoBERTa does not show any trends, even when the number equals to I, the model still makes mistakes in large percentage of cases. See Figure X b)), as the number of possible worlds in the context increases, the accuracy of human baseline tends to decrease significantly. This is because the more possible worlds that need to be considered, the more judgments humans need to perform, and humans tend to overlook certain conflics,, which leads to wrong decisions for problems. However there is significant correlation between the performance of the model and number of possible worlds. This indicates that the model does not really filter and judge the possible worlds. See Figure the model makes mistakes variety of context lengths, while humans perform perfectly for problems with context length < 73.
5 RELATED WORK
There are abundant datasets in reading comprehension, which could facilitate the development of the field. MCTest (Richardson et al. 2013) is a multiple-choice reading comprehension dataset that contains 500 fictional stories and 2k questions. Rajpurkar et al. (2016) proposes the first large-scale reading comprehension dataset SQuAD (100k+ questions), where the answer to each question is a span of text from the passage. Recently more datasets requiring more complicated reasoning types introduced, such as multi-document (Joshi et al. 2017: Dunn et al. 2017). multi-hop reasoning (Yang al., 2018: Welbl et al. 2018 Talmor& Berant [2018). numerical discrete reasoning (Dua et al. 2019) and commonsense reasoning (Mihaylov et al. 2018 Zhang et al. 2018 Huang et al. 2019a). However, these datasets cannot test the logical reasoning ability of the models. To fill this gap, LogiQA (Liu et al. 2020), ReClor (Yu et al. 2020) and LR-LSAT WWan et al. 2021) was proposed.
LogiQA is collected from National Civil Servants Examination. ReClor and LR-LSAT are extracted from Law School Admission Test. These datasets require logical reasoning to answer the questions.
However, from human experience, there are different forms of reasoning strategies answering different logical questions. We believe that completely different categories of logical reasoning deserve to be explored separately (Rudinger et al. 22020). Different from these datasets, we inductively define typical class of logical reasoning, named naïve logical reasoning, and then create new benchmark targeting the task, named NAIL. Compared with LogiQA, Reclor and LR-LSAT, NAIL focus the more fine-graine ine-ggaieed logical reasoning type (naïve logical reasoning).
There have been many datasets extracted from human examinations, such as LogiQA and ReClor mentioned above. Besides, RACE dataset (Lai et al. 2017) is collected from English exams for middle and high school Chinese students. ARC dataset (Clark et al., 2018) consists of 7,787 science exam questions drawn from variety of sources. DREAM (Sun al.] 2019) is dialogue-based multiple- choice reading comprehension dataset collected from English as a Foreign Language examinations which contains 197 questions for 6,444 dialogues.
CONCLUSION
We introduce a more fine- -en-ggrained logical reasoning, naive logical reasoning, and We propose a new large-scale benchmark, NAIL, aiming to help models leam and evaluate naive logical reasoning capability. NAIL is sourced from standardized exams and human imitation. Preliminary results show that there is still long way to go equip deep models with true logical reasoning capability.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kazziorsi, Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367, 2019.
Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, and Wei Chu. Question directed graph attention network for numerical reasoning text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6759-6768, 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint 1r03::1803.000
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese bert. arXiv preprint 1906::9900.8//// 2019.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2368-2378, 2019.
Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqu: A new q&a dataset augmented with context from search engine. arXiv preprint arXiv: 2017.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. Hierarchical graph network for multi-hop question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8823-8838, 2020.
Lifu Huang. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pp. 2391-2401, Hong Kong, China, November 2019a. Association for Computational Linguistics., doi: 10.18653/v1//D11-1 1243. URL https
www www.eb..lb..... org/ www.://www.oog//tthoooooooo http:///ahhhooooy9/-1111
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2391-2401, 2019b.
Yinya Huang, Meng Fang, Yu Cao, Liwei Wang, and Xiaodan Liang. Dagn: Discourse-awarc graph network for logical reasoning. arXiv preprint arXiv:2103. 14349, 2021.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1601- 1611, 2017.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18 URL https 7/ wwwww.o. ac web. www..ooggg/athhooogo///---/---1
10

---

Under review as a conference paper at ICLR 2022
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. 2020.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
John McCarthy et al. Programs with common sense. RLE and MIT computation center, 1960.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.0277 2018.
Allen Newell and Herbert Simon. The logic theory machine-a complex information processing system. IRE Transactions on information theory, 2(3): 1956.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine I Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v//11- 1264. CRI.https:///t aww.ww.cllll........ org/
anthology/DD15 1264
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. Metest: challenge dataset for the open-domain machine comprehension of text. In Proceedingy of the 2013 conference on empirical methods in natural language processing, pp. 193-203, 2013.
Rachel Rudinger, Vered Shwartz, Jena D Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A Smith, and Yejin Choi. Thinking like skeptic: Defeasible inference in natural language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Findings, pp. 4661-4675, 2020.
Processing:
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. Dream: A challenge data set and models for dialogue- based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231 2019.
Alon Talmor and Jonathan Berant. The web knowledge-bris base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 641-651,
2018.
Siyuan Wang. Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From Isat: The progress and challenges of complex reasoning. arXiv preprint arXiv: arXiv:2 2021.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL htt.:httt:////
ac web. www.wwee........eoo www.antoooolyyyooooo emn wewnlp-ddmmae dep-s.mos.. 6).
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2380, 2018
11

---

Under review as a conference paper at ICLR 2022
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: 1810. 12885, 2018.
12

---

Under review as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS OF BASELINES
Rule-based Methods Following LogiQA (Liu et al. 2020), we adopt two simple methods based on text matching. Specifically, word matching is to measure the degree of unigram overlap between the candidate answer and the given paragraph-query pair; sliding window takes into account the n-gram when calculating the matching score.
Neural-based Methods Pre-trained neural models, such as BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019). have achieved impressive performance reading comprehension. The input to the pre-trained model is the concatenation of the paragraph, the query and the candidate answer, separated by [SEP] tokens, denoted as: [CLS], paragraph, [SEP]. query, [SEP]. answer. After the encoding of the pre-trained model, the representation of [CLS]] token followed by an multiple
layer perceptron (MLP) is used for scoring.
We re-implement the rule-based methods following LogiQA (Liu et al. 2020). For pre-trained models, we modify the code of Transformers of HuggingFace (Wolf al. 2020) implement them on NAIL. We take the off-the-shelf model BERT and RoBERTa for NAIL, and Chinese BERT and Chinese RoBERTa (Cui al. 2019) for Chinese NAIL. All hyper-paramettrs are selected by the model performance on the development sets.
B MORE STRONG BASELINES
We also try seq2seq based pre-tee models (i.e. T5 (Raffel et al. 2019)) and well-designed models for other related tasks (i.e. DAGN (Huang et al. (221)) for LogiQA and Reclor, HGN (Fang et al. 2020) for HotpotQA (Yang et al.) 2018), and QDGAT (Chen al. 2020) for DROP (Dua al. 2019 our English version of NAIL.
problem is regarded as four samples to T5 during training. The input of sample is: con- text+query+one of the four choices. And we let the model generate the label for the sample: if the choice the correct answer, then generate "Truc", otherwise generate "False". And during inference, for a problem, we choose the option with the highest probability of generating "True" among the four options as the prediction. We set énput papputtmrr.... www.mmoleeee len 512 and finetuned the off-the-shelf T5-base model on the training set of NAIL, results shown in Table
LogiQA and ReClor are two logical benchmarks introduced in the text of the paper to test models' various reasoning abilities. HotpotQA is multiple-choice QA dataset featuring natural, multi-hop questions, with strong supervision for supporting facts to test models' multi-hop factual reasoning ability. DROP a 96k-question benchmark to models' numerical discrete reasoning ability over paragraphs (such as addition, counting, or sorting).
We choose three state-of-the-ard and open-sourced models designed for the above benchmarks: DAGN for LogiQA and Reclor, HGN for HotpotQA, and QDGAT for DROP respectively. We reproduced these three models with the code released by authors and NAIL training data, Results on NAIL are shown as Table
NAIL
Model
Encoder
Dev
Test
RoBERTa TARGR
RoBERTa TARER
34.6
30.1
RoBERTa BASE
RoBERTa B&SE
27.6
26.4
T5
T5 BASE
27.1
26.3
DAGN
RoBERTa LARCE
36.0
32.3
HGN QDGAT
RoBERTa LARGE
30.1
28.7
RoBERTa LARGE
28.3
28.7
Table 4: Results on NAIL (accuracy%)..
10DANN: http://gtttttb.oomm////eeemo/a/t QDGAT: emn p2pp02200g0a.....
HGN: Htt wtt.://wwwwowwww////oo../aaa
13

---

Under review as a conference paper at ICLR 2022
We observe that DAGN achieves the best results on NAIL, since ReClor and LogiQA much closer to NAIL, the well-designed discourse-awar graph network DAGN is more applicable to NAIL. However there is still very much space for models' improvement compared to humans. HGN a hierarchical graph network for multi-hop question answering, which is designed to aggregate heterogeneous information, and QDGAT is question directed graph attention network, which is dedicated to identifying numbers and the computation between them, which are both of minimal help to NAIL. T5 is very powerful generative model, but it seems that T5 lacks the naive reasoning capability as well. Due to computational resource limitation, we did not try T5 model with larger number parameters.
C THE EFFECT OF NAIL,, NAIL-E AND NAIL-I
To investigate the effect of NAIL, NAIL-E and NAIL-I, we train RoBERTa LARGe model on NAIL, NAIL-E and -A separately and evaluate on the corresponding set of all tests (Table When trained on NAIL-II and evaluated on NAIL-E,, the model achieves that human imitations can significantly help models learn logic from natural text. When trained on NAIL-E and evaluated on NAIL-E, the model achieves the worst accuracy 22.9%. The possible reason that the amount of data of NAIL-E is too small to train deep model. Corresponding results on the validation set are shown Table O
best accuracy 38.5%, showing
Trained
NAIL.
NAIL-E
NAIL-I
Evaluated on 4
NAIL NAIL-E
30.1
26.4
33.9
37.4
22.9
38.5
NAIL-I
29.7
26.6
33.7
Table 5: Performance of RoBERTa on different training sets and test sets.
Trained
NAIL
NAIL-I NAIL-E
Evaluated on
NAIL
34.6
24.9
32.0
NAIL-E
43.3
37.4
33.0
NAIL-I
34.2
24.3
31.9
Table 6: Performance of RoBERTa on different training sets and validation sets.
D TRANSFER LEARNING
D.I OTHER DATASETS AS EXTRA TRAINING RESOURCE
What are the results if pre-trained model is first trained existing reading comprehension datasets, and then fine-tuned on NAIL? Table 3 shows the results on the test set of NAIL,NAIL-E, NAIL-I where LogiQA, ReClor IT and RACE are adopted. And here in Table 7 we give the results on the validation splits of NAIL-E, NAIL-I.
D.2 NAIL AS EXTRA TRAINING RESOURCE
What are the results if NAIL is used as extra training resource for existing reading comprehension tasks? Using NAIL extra training resource, we conduct plenty of experiments on existing benchmarks. Besides LogiQA, ReClor and RACE (see in Figure more results are shown in
11 For fair comparison, in all cross- bosssbenncharks experiments in this paper, we removed samples in the training set that are duplicates of those in the test set. For example, when using ReClor or LogiQA as extra training resource when testing on NAIL, remove problems in the training set of ReClor or LogiQA A if they appear in the test of NAI...
14

---

Under review as a conference paper at ICLR 2022
Evaluate on
NAIL.
NAIL--E
NAIL-I
Train on 4
Ø
23.6
24.2
23.6
LogiQA
33.0
45.1
32.4
ReClor RACE
28.4
25.3
28.5
26.4
27.5
26.4
NAIL
34.6
43.3
34.2
LogiQA NAIL
37.9
47.3
37.5
ReClor- NAIL
35.2
39.6
35.9
RACE--NAIL
35.4
36.3
35.4
Table 7: Transfer learning results when evaluating the validation split of NAIL, NAIL-E, NAIL-I (accuracy%).
Table 9
We adopted MathQA (Amini et al. 2019). and HotpotQA (Yang et al. 2018). For Math A, include a prediction head for multiple choice based on the RoBER TaaERTa-lasee Ta-le model, and we include prediction head for question answering. For MathQA, we concat the
for HotpotQA 12 problem and annotatedd input.
Evaluate on +
MathQA
Evaluate on +
HotpoiQA
Train on +
Test
Train on
Ans F1
MathQA
39.8
HotpotQA
69.8
NAIL + MathQA
41.2
NAIL + HotpotQA
72.6
Table 8:
Evaluating on MathQA after
Table 9:
Evaluating HotpotQA after
RoBERTa-large pretrained on MathQA/NAIL training set.
RoBERTa-large pretrained on HotpotQA/NAIL training set.
Experimental results show that, we use NAIL as extra training resource, the supervised-learninn results all of these six datasets: LogiQA, ReClor, RACE, MathQA, HotpotQA will improve. This indicates that equipping models with naïve logical reasoning ability can help solve math word problems (MathQA), improve multi-hop factual reasoning skills (HotpotQA), solve various logical reasoning problems (LogiQA and ReClor), and enhance general understanding and reading comprehension capability (RACE). In the future, we will on more known datasets, to verify that naïve logical reasoning is basic capability and is helpful for other tasks.
E ANALYSIS OF FINE-GRAINED TYPES
In Section 4 analyze the model performance with respect to different types of naive logical reasoning (Figure 5) Figure & shows the accuracy against fine-grained reasoning types on the test set of NAIL, NAIL-E, NAIL-I when training on NAIL, NAIL-E, NAIL-I respectively. And Figure 7 shows the corresponding results on the validation set.
From above figures we can find that language models perform well on operation problems, while struggle on matching and ordering. We think that language models can provide good representation of set object, even models do not really reason derived from the context.
F DISCUSSIONS ABOUT FUTURE DIRECTIONS
We have some ideas for future directions for models solve NAIL.
The first point is to identify deterministic information or information with low uncertainty. Generally, we need to detect the atomic information that can be used directly without reasoning, which is the key to solving the problem. Different conditions should not be considered with equal priority,
12 Due computational resource limitations, only trained epachs for experiments involved HotpotQA.
15

---

Under review as a conference paper at ICLR 2022
233
Jid
11.
213
Thanesd MJ. Evokuaed MAA.
Thired NAR Eviraled SaL.
Trired SALE Exsladed SALI
$2.
314 344
SEA
Thikee ai MAC stad HLL
Thanee KALL Eask NA..
Talkel WA BAL
251
302
Figure 7: Accuracy against the reasoning types based on different training and development set.
Le., the certain condition start from, the fewer possible worlds that the condition yields. Specifically, for real/fake mixture problems, we also need to promptly identify two atomic statements that yield contradictory, that is, the two statements cannot be both true (there must be one false) or both false (there must be one true), which is often the breakthrough in solving real/fake mixture problems.
The second is to detect informative subjects/objectts Generally, the more frequently a subject/object mentioned in context, the more relevant information carries. Tables and bi(ttri)partite diagrams can help to clarify the correspondence between given subjects and objects. For example, in the case of Figure T tables play an effective role in solving the problems. And as for the third case in Figure 3] which is categorized as matching problem, a tri- tri-partite diagram could help, that is, three disjoint and independent sets U, V and W represent [A,B.C], (Beijing, Nanjing, Chengdu J. (a doctor, an actor, and teacher) respectively, an edge connecting vertex in one set to one in another set denotes "is_a" predicate.
The third direction is allowing models to better utilize elimination. On the one hand, models perform forward elimination, i.c., according to the context, each time we draw a definite conclusion, retain options that logically consistent and exclude those that do not fit the derived conclusion. On the other hand, models can also perform backward elimination. For example, when sometimes it is not easy draw exact inferences directly from the context, then options into the context. substituting an option creates a contradiction within the context, then the option should be excluded. Forward and backward elimination facilitate the model in arriving the correct answer.
substitute the
16

---

Under review as a conference paper at ICLR 2022
Evaluated
Tranedd
Trined Evalusted
RoBERTa
RoBERTa
ROBERT
18%
28.1
345
ReaVfake
Matching operation
Realfake
operation
Reatfake
Set
Thansd en MMk- Dribanad en NA.
Thired on 14.4. EL
Trired MABE Oveusted MI
311
217
351
JLA
Thired MALA
Thined on NAL.
Teined Ealluttd
31.
201
Figure 8: Accuracy against the reasoning types based different training and testing set.
17


Reviews:
[{"review_id": "XHOcb4Ovis", "reviewer": "Reviewer_QPXV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The proposed dataset is carefully designed and sourced where most of the questions are collected by imitating actual examples in exams. All examples are validated by human experts.\n- The collected questions are difficult for BERT-large and RoBERTa-large which show lower accuracies than the human baseline of college students.\n\nWeaknesses:\n\n- The authors test simple rule-based models and vanilla BERT-large/RoBERTa-large models, failing to imply a future direction for developing systems that perform naive logical reasoning. Ideally, the authors could (modify and) test graph neural network-based systems that are proposed for datasets that require symbolic reasoning (e.g., HotpotQA and DROP).\n- It is a bit obvious that systems pre-trained on ReClor, LogiQA, and RACE do not perform well on NAIL because, as the author mention in Section 4.2, the nature of questions are completely different. Rather, I would like to see it used as an extra training resource for existing multiple-choice datasets including these three datasets. I agree with the authors that the naive logical reasoning needs to be explored as an independent research topic, but I think the authors need to prove that the proposed dataset is useful in some sense in relation to other NLU tasks, datasets, and systems.\n- Likewise, the main claim \"human imitations can significantly help models learn logic from natural text\" is supported only by the experiments on the actual examples contained in NAIL (NAIL-E). The authors prove that the models trained on the imitation examples can perform specific types of reasoning contained in the actual examples, but do not assess their ability of more general logical reasoning such as in ReClor and LogiQA.\n- I'm not sure if the authors can use \"naive\" in the sense that is referred to in the context of Naive Set Theory by Halmos (1970). I guess the notion \"naive\" in Naive Set Theory means that the theory is not described on the top of axioms. Does this imply \"spontaneous, intuitive, and unsystematic\"?\n- There are grammatical errors or typos in the paper. It should be proofread.\n\nQuestions and feedback:\n\n- It is unclear how the authors translate Chinese and English examples. In particular, if it is done after the expert validation, translation errors might affect the human baseline accuracy. Are there any quality controls over the translation?\n- Human baseline (around 70%) looks a bit lower than that of experts (100%). Is there an error analysis about them? Is that by careless mistakes or by being truly difficult for college students?\n- The second paragraph of Section 5 sounds interesting but has only one sentence. Could the authors flesh it out?\n- In the description of the post-checking process, what does \"each of experts has been selected through CNCSE\" mean?\n- Please replace the alias tokens in the Open Review abstract with actual terms.", "summary_of_the_paper": "This paper proposes a new dataset that tests the capability of logical reasoning through multiple-choice reading comprehension. Questions are collected from human exams (Chinese National Civil Servants Examination in Chinese and Law School Admission Test in English) and are augmented by imitating the actual exam examples. The main focus of the questions is understanding of the relationship between subject-predicate-object triples, involving four reasoning types: factuality, ordering, matching, and set operation. In the imitation process, the authors hire people from different backgrounds who are asked to write examples that are logically invariant but semantically different from a given actual example (82 people, 813 paid work hours with fair payment,  and at least 20 imitations per actual example). All collected examples are manually validated by experts and answered by college students for measuring human baseline. Experiments with rule-based (word matching and sliding window) and neural network-based (BERT and RoBERTa) models show that the machine performance (30.10 and 36.15 for the English and Chinese sets respectively) is largely lower than the human baseline (71.31 and 76.38), which indicates that these systems struggle to solve the naive logical reasoning questions.", "summary_of_the_review": "The proposed dataset is carefully designed and may be useful for pushing the boundary of assessing the logical reasoning capability of systems through the reading comprehension task, but experiments are insufficient for empirically showing the usefulness of the dataset.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LVtdrKhz8QW", "reviewer": "Reviewer_kWbB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Naive logical reasoning (NAIL) is quite difficult even for humans: even well trained human experts cannot agree on all the answers (such ambiguous answers were left out from the final dataset), and human non-experts score much lower than experts. Therefore, it is expected that the task will be extremely difficult for language models, and the gap between human and system performance is indeed prominent, and calls for future developments in improving logical ability of large neural models.\n\nAs said, the main contributions of the paper are: 1) creation of the dataset for the NAIL task, 2) showing that some pretrained language models are not as good as humans in solving this task. However, I believe that the paper still does not paint the whole picture and should bring much more detailed analyses on the current gaps, why they happen, and how we can address those gaps.\n\nI am not fully convinced about motivation to have this sort of logical reasoning embedded in the model - these NAIL tasks seem more like 'enigmatic' puzzles, and a stronger motivation is needed to justify why pretrained language models should even need the ability to solve such puzzles: can they be fully trained (with larger datasets) to do well on the puzzles? Another question is, where I expected a deeper discussion: do we need AGI to solve puzzles like the one presented in NAIL? Is it a data problem or truly a human-like reasoning problem? In other words, with even larger models (GPT-3, T5) and more training data, can we expect the models to close this gap to human (non-expert) performance? \n\nOverall, the paper is quite thin on additional insightful experiments and further analyses, going beyond reporting the dataset creation protocol and benchmarking a subset of models on the data, and does not read as a complete piece of work.\n\nWhile a direct link is not apparent at first sight, this work also reminds me very much of the ACL 2020 work of Sahin et al. (https://aclanthology.org/2020.acl-main.115/) - they also showed how pretrained models do lack the skill of iterative reasoning upon knowledge. In other words, supported by a slightly different iterative logical reasoning task than NAIL, they reach exactly the same conclusion. One could read that this lack of ability has been shown in the previous paper, and this work just restates with another similar task, which also diminish novelty of this work a bit.\n\nGiven that imitation examples rely on the same backbone template, and the only variation is actually in the semantics of the examples (i.e., variation is at word-level or 'discourse'-level), I would like to understand more why such imitation examples do help increase the scores: is it only because of a higher semantic coverage, or is it about some spurious artefacts that the model learns by seeing more examples of the same backbone type (e.g., some hidden correlation learned by the model). Would it be possible to create imitation examples automatically and augment training data with such examples? How would the models then behave?\n\nOnly performance of BERT and RoBERTa is attested in the task - I would very much suggest to try out other language models (e.g., T5) and think of possible (auxiliary/intermediate) tasks which could prepare off-the-shelf models to become better at this type of task.", "summary_of_the_paper": "This paper proposes a new challenge for large pretrained models (which can of course extend to any other model claiming to store language knowledge): naive logical reasoning. The main contribution of the paper is the construction of a dataset in Chinese and English (termed NAIL) with items for training and evaluating such naive logical reasoning, where the task in the dataset is cast as a standard multi-choice (4-choice in this case) question, with only one correct answer. The main empirical result of the paper consists in showing that pretrained language models, even when trained on naive logical reasoning examples, lag far behind human performance.", "summary_of_the_review": "Similar to some prior work, the paper exposes the inability of (a subset of) pretrained language models (BERT, RoBERTa) to do complex logical reasoning, showcasing that their performance is much lower than what humans score. This is done through the introduction of a new task called naive logical reasoning. Overall, while the NAIL dataset can be quite helpful in guiding future advances on equipping language models with knowledge and methods required to solve such logical 'puzzles', the paper should have done a much better work on the aspects of motivation, discussion, analyses, and higher-level implications of the main results. I feel that more work is needed (see the main review).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L1sXtJ4ZF3F", "reviewer": "Reviewer_VxU4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A really interesting work. A newly introduced and challenging datasets which will be happily received by the scientific community.\nThe paper is well organized and written. All the decisions taken by the authors are properly described and justified.\nMy only concern on a paper of this quality is the lack of experimentation using NAIL as a pre training step for other logical reasoning datasets. If learning the logic of this dataset bring naive reasoning still into a model, that should be reflected into the performance of other logical datasets like LogiQA, ReClor, RACE, but also in CLUTRR, MAthQA, AQuA, HotpotQA among others ...\n", "summary_of_the_paper": "This work introduces NAIL, a bilingual (English and Chinese) a benchmark for naive logical reasoning, inspired in the kind of questions, involving this aspect contained in standardized exams such as Chinese\nNational Civil Servants Examination (CNCSE) and Law School Admission Test (LSAT).\nTwo different sets are collected in this work. NAIL-E, which are actual examples from both CNCSE and LSAT, translated to English and Chinese respectively, and NAIL-I, a data augmentation approach that uses the structure from existing examples and replaces the three aspects of the problem, subject, predicated and objects in order to create new instances.\nAn qualitative analysis is carried out to reach the conclusion that those three replacings are necessary in order to create new, and more challenging instances.\nThe resulting datasets NAIL-E and NAIL-I are finally splitted into train, dev and test, and examined to test their quality.\nRandom baseline, word match, sliding window, BERT and Roberta (as well as human evaluations carried out by average and expert annotators) are carried out showing the quality of the resulting datasets and the challeding there are for current NLP models.\nExtra experimentation is done to test their transfer learning skills among the full dataset, its split NAIL-E and NAIL-I and using LogiQA, ReClor, RACE as pretriaining steps.\n", "summary_of_the_review": "Please provide a short summary justifying your recommendation of the paper.\nGood paper\nVery useful dataset\nAll reasoning steps in the development are properly justified\nExperimentation is reasonable and accurate\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H2uij283uiA", "reviewer": "Reviewer_xN3f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors manually constructed a dataset that aims to train and evaluate the model’s capabilities in naïve logical reasoning, which consists of 10,296 instances that were extracted or re-written form Chinese National Civil Servants Examination (CNCSE) and Law School Admission Test (LSAT) by humans. \n\nStrengths:\n\nA dataset was built to train and evaluate model’s capabilities in naïve logical reasoning, which consists of 10,296 instances.\n\nWeaknesses:\n\nAll the instances in the dataset were selected or written by humans, and there is no valuable or useful method proposed in the dataset-construction process.\nIt is not clear how they guarantee the quality of translation. There are even many grammatical errors in the examples given in this paper. For example, “Selecting Prince Charming”, “Xiao Li’s ideal gift has following characters”, etc.\nIn my opinion, it seems that the main difficulty of answering the questions in the constructed dataset lies in semantic parsing rather than logical reasoning. If the texts are parsed precisely, the models might achieve high accuracy with an appropriate solving (or searching) technique.\n", "summary_of_the_paper": "The authors manually constructed a dataset that aims to train and evaluate model’s capabilities in naïve logical reasoning. The instances were extracted from Chinese National Civil Servants Examination (CNCSE) and Law School Admission Test (LSAT) and those instances were re-written by humans to increase the size of the dataset.\n\n", "summary_of_the_review": "The authors manually constructed a dataset that aims to train and evaluate the model’s capabilities in naïve logical reasoning, which consists of 10,296 instances. The instances were extracted from Chinese National Civil Servants Examination (CNCSE) and Law School Admission Test (LSAT) and those instances were re-written by humans to increase the size of the dataset. However, all the instances in the dataset were selected or written by humans, and there is no valuable or useful method proposed in the dataset-construction process. Besides, it is not clear how they guarantee the quality of translation. There are even many grammatical errors in the examples given in this paper. For example, “Selecting Prince Charming”, “Xiao Li’s ideal gift has following characters”, etc. It seems that the main difficulty of answering the questions in the constructed dataset lies in semantic parsing rather than logical reasoning.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a dataset, based on preexisting standardized tests, of elimination/grid-completion-style logical reasoning puzzles expressed in text; available in both Chinese and English (with some of the text coming from semi-automatic translation). The early pretrained MLMs BERT and RoBERTa perform poorly. 

This paper is solidly borderline. Reviewers had some concerns about the motivation and novelty the work, but I think that there is a plausible enough story for where this data will have value that I'm not comfortable rejecting it only on this basis. More worryingly, the initial submission had some fairly serious writing quality and clarity issues, which impacted both the paper *and* the data. It seems like the authors made significant progress on this in the revision and engaged substantially during the discussion, but reviewers were not fully satisfied that the paper was up to ICLR standards, either as an initial submission or after revisions. This is a small detail, but it's a bad sign for the carefulness of the work that the OpenReview abstract is still unreadable, even after a request from a reviewer.
Decision: Reject