NAIL: A Challenging Benchmark for Na\"ive Logical Reasoning

1 INTRODUCTION . Current deep models have achieved near human-level performance on many tasks in NLP ( Devlin et al. , 2019 ; Liu et al. , 2019 ) , and more often than not , superficial knowledge suffices to solve the problems . To move towards human intelligence , we need to equip the models with logical reasoning capabilities ( e.g. , ability to draw logical conclusions from given statements ) , which is also a long sought-after goal of AI ( Newell & Simon , 1956 ; McCarthy et al. , 1960 ) . One related task is natural language inference whose goal is to assign the logical relationships ( contradicted , neutral and entailment ) to sentence pairs . To push the development of models in logical reasoning , the researchers have focused on more challenging reading comprehension tasks , which often require more complex reasoning as well as longer input . However , most existing reading comprehension datasets are not oriented for the logical reasoning ( e.g. , SQuAD and HotpotQA ) , with the exception of LogiQA ( Liu et al. , 2020 ) , ReClor ( Yu et al. , 2020 ) and LR-LSAT ( Wang et al. , 2021 ) . Above datasets ( LogiQA , ReClor and LR-LSAT ) are limited and inadequate to train and evaluate logical reasoning capability . The reason is that all of the three datasets involve diverse types of logical reasoning , such as drawing an alternate conclusion to the argument , finding necessary/sufficient assumptions , whether statements strengthen/weaken the argument or explain/resolve the situation . Mixing multiple types of logical reasoning may pose the following challenges . a ) . From the perspective of human cognition , different types of logical reasoning correspond to different problem-solving ideas . But in practice we usually train a model on the whole dataset with the same idea , which makes it more limited for models to learn different logical reasoning capability . b ) . From the perspective of machine learning , if there are many reasoning types mixed in a dataset , then there will be less data for each reasoning type , which is inadequate to train and evaluate logical reasoning capability ( demonstrated in our experiments ) . Furthermore , when the model does not work , it is difficult to determine which reasoning type is the bottleneck ( no reasoning type annotation in the dataset ) , which may hinder the design of better models . To tackle the challenges , we focus on a more fine-grained type of logical reasoning , named naı̈ve logical reasoning ( Section 2 ) , which is to infer the logical conclusion from statements that describe triples ( subject , predicate , object ) and the relationships among them . A typical example of naı̈ve Paragraph : A，B，C，D为四位漂亮女生。她们喜欢穿漂亮衣服。某天，她们穿的衣服颜色各不相同，有黄色，绿色，蓝色 和红色四种，在问到她们各自衣服的颜色时，A说： “ B的衣服不是黄色的。 ” B说： “ C的衣服不是绿色的。 ” C说： “ D的衣服不 是蓝色的。 ” D说： “ A，B，C三人中有一个人的衣服是绿色的，而且只有这个人说的是实话。 ” ( There are four pretty girls , namely A , B , C , and D , all of whom love to dress beautifully . One day , four girls get dressed in different colors : yellow , green , blue , and red . When asked about the color , A said that B did not dress in yellow ; B said that C did not dress in green ; C said that D did not dress in blue ; D said that among A , B , and C , only one girl dressed in green and she was the only one who told the truth . ) logical reasoning is shown in Figure 1 . To answer this query , we need to iteratively derive conclusions according to the conditions , and stop the searching branch if a conflict occurs . Specifically , in the initial table , we only know that “ D is not green ” from “ D told the truth ” . Assume that “ A told the truth ” , and infer that “ A is green , C is non-green ” . Next derive “ C is green ” from “ C told the lie ” . Then C is derived as both green and non-green , thus causing a conflict . Similar processes for assuming “ B told the truth ” and “ C told the truth ” . It takes extensive training and practice for human brains to cope with such complex logical reasoning . Inspired by the datasets extracted from standardized examinations ( Lai et al. , 2017 ; Clark et al. , 2018 ; Liu et al. , 2020 ) , we build a new large-scale benchmark , NAIL , by selecting naı̈ve logical reasoning examples from standardized exams such as the National Civil Servants Examination of China and Law School Admission Test . However , such examples are limited in their number , as it takes efforts for human experts to design these questions . To collect more data , we propose to imitate examples of standardized exams rather than designing them from scratch . Unlike simple data augmentation ( e.g. , substitution , paraphrasing ) , human imitation aims at getting more diverse examples while maintaining the underlying logic ( Figure 1 ) . NAIL is available in both Chinese and English , containing a total of 10 , 296 ∗ 2 instances . Empirical results show that state-of-the-art neural models struggle on NAIL with very poor accuracy ( the best result is 30.1 % for NAIL and 36.2 % for Chinese NAIL ) , while human experts can perform 100 % accuracy . Further results indicate that human imitations can significantly help models learn naı̈ve logical ability from natural text . 2 NAÏVE LOGICAL REASONING . The naı̈ve logical reasoning is a more fine-grained type of logical reasoning . Formally , we give the definition as follows . Definition 1 . The subject is a described resource , usually an entity , such as person or location . The predicate indicates an attribute of the subject or indicates some kind of relationship between the subject and the object . When denoting an attribute , the object is the attribute value , usually a literal value , e.g. , ( Jacket , Color , Red ) , otherwise the object is an entity , e.g. , ( Beijing , Capital , China ) . Definition 2 . Assuming the set of subjects S , the set of predicates P , the set of objects O and several statements , each statement describes a triple ( subject , predicate , object ) or some logical relationship between triples . The naive logical reasoning is the process of reasoning from these statements to reach a logical conclusion . 1 In this work , we explore naı̈ve logical reasoning in the form of reading comprehension . A typical example and detailed reasoning process is shown in Figure 1 . Similar to the format of multiple-choice reading comprehension , it contains a context , a query and four options with only one correct answer . To solve the problem , the model needs to understand the logical connections between the subjects , predicates and objects , and then derive a valid option . 3 NAIL : DATA COLLECTION AND ANALYSIS . NAIL is a carefully designed benchmark for naı̈ve logical reasoning similar to the format of multiplechoice reading comprehension . Inspired by the datasets extracted from standardized examinations ( Lai et al. , 2017 ; Clark et al. , 2018 ; Liu et al. , 2020 ) , we first collect a small amount of examples from examinations , denoted as NAIL-E ( Section 3.1 ) . Then we propose to imitate examples of NAIL-E to collect more data , denoted as NAIL-I ( Section 3.2 ) . Finally we provide a detailed analysis of the proposed NAIL ( Section 3.3 ) . 3.1 COLLECTION FROM EXAMINATION . We searched for such examples widely from two different types of public examinations : Chinese National Civil Servants Examination ( CNCSE ) and Law School Admission Test ( LSAT ) . 2 CNCSE is a once-a-year competitive examination in China , and there are overall 120-140 examples per exam per year . But only 1-4 examples belong to the scope of naı̈ve logical reasoning , which is also the most difficult type of problems for candidates within the given 120 minutes . And the LSAT is a standardized test for prospective law school candidates in the United States , Canada , and a growing number of other countries . Logical Reasoning is a multiple-choice section of LSAT , containing 24-26 questions under the limitation of 35 minutes , where about 2-4 problems fall into naı̈ve logical reasoning category . We artificially selected examples that belong to the naı̈ve logical reasoning category from the above two examinations . Finally we obtained 488 examples from the last 25 years of CNCSE and LSAT in the last 30 years , denoted as NAIL-E . These two exams are from countries with different native languages . The former is expressed in Chinese , and the latter is expressed in English . And note that there are slight difference in language style between them . And for diversity and fairness , in later step we get all examples in both Chinese and English through translation . 3.2 COLLECTION FROM IMITATION . After collecting a small amount of NAIL-E , we expect to expand the number of the dataset . Designing examples from scratch requires a huge effort from human experts . One simple solution is data augmentation , which artificially scales up data by creating modified data from existing data , such as word/sentence shuffling , word replacement and syntactic variation . However , the data augmented in this way is highly correlated with the original data , and the model easily captures these semantic surface correlations , which makes it limited to train and evaluate logical reasoning capability . To alleviate this limitation , we propose to imitate examples of NAIL-E , to create more examples with a diverse semantic surface while keeping the underlying logic of the original example . Furthermore , in the process of human imitation , we design strict strategies to control the quality of imitation . Imitation Example See Figure 2 , the context of an example from NAIL-E consists of 6 sentences and a query ( split by blank lines ) , each of which can be represented as a logical template ( see “ backbone template ” ) . The example focus on the description of a scenario : picking Prince Charming ( m1 ) , which involves five entities : Li Na ( s1 ) , Wang Wei ( p1 ) , Wu Gang ( p2 ) , Li Qiang ( p3 ) , Liu Dawei ( p4 ) , and three noun/adjective properties : tall ( a1 ) , handsome ( a2 ) , a PhD ( a3 ) . An accepted imitation needs to keep original underlying logic but have semantically different groundings , i.e. , to describe a completely different scenario . Imitation 1 and 2 1The term naive refers to the fact that this logical reasoning process of human in this task is spontaneous , intuitive and unsystematic . 2https : //www.lsac.org/lsat/taking-lsat/test-format/logical-reasoning are unqualified imitations . Since Imitation 1 only conducts subject-level imitations ( S imitations ) , i.e. , only superficially substitute the five entities ( s1 : Li Na→Sun Yi , p1 : Wang Wei→Qu Heng , p2 : Wu Gang→Mei Zhen , p3 : Li Qiang→Ji Fan , p4 : Liu Dawei→Zhang Lei ) . Further , Imitation 2 conducts subject-and-object-level imitations ( SO imitations ) , i.e. , not only substituting the entities , but also altering the corresponding objects ( property values ) . ( a1 : tall→rich , a2 : handsome→having a house , a3 : a PhD→having a car ) . The difference between Imitation 2 and the original example is much greater than that between Imitation 1 and and the raw problem , however , Imitation 2 is still not an ideal imitation . Furthermore , Imitation 3 changes the scenario m1 into picking ideal gift . Imitation 3 is an expected imitative writing , which conducts subject-and-predicate-and-object-level imitations ( SPO imitations ) . Note that all of the three imitations share the same logic templates . And p2 is the answer for all of the original example and above three imitations . Imitation Process We first select a group of people from a variety of occupations : professional editors , legal practitioners , in-service civil servants and college students ( from different majors ) . Empirically , people in these occupations have strong logical and verbal skills . These people are asked to conduct SPO imitations based on original examples from NAIL-E , and reasonable imitations should meet two requirements : logic invariance and semantic diversity . We trained these candidate people how to conduct SPO imitations such as Figure 2 . We then conducted a trial phase before the official imitation phase , in which process we eliminated some people of poor quality . Finally we employed 82 qualified people3 to imitatively construct problems based on NAIL-E , and they are paid RMB¥2.8 per imitation4 . Averagely , it costs a trained person about 3-4 minutes to finish an imitation : starting from coming up a scenario , then replacing the subjects , predicates , and objects of the raw 3Consisting of 42 native Chinese speakers and 40 native English speakers . Native speakers of a language will be assigned imitation tasks expressed by that language . 4A part-time employee can produce 15-20 imitations per hour , where he/she can get RMB¥42-RMB¥56 , while the local minimum wage is RMB¥23 per hour . problem with those scenario-related while keeping invariant logic , and finally smoothing the new sentence with some transition words if necessary . For each original example in NAIL-E , we expect at least 20 imitations ( except for extremely difficult cases ) . Overall we use 813 paid work hours in total to build the NAIL-I . Imitation Quality Control We adopt following strategies to ensure the quality of imitations : 1 . As mentioned above , we conducted a trial phase before the official imitation phase . In this phase , we asked them to imitate a small number of problems . Although we do not necessarily need them to write the backbone template , we will check the logic and give feedback to help them understand the task . This process was iterated for three rounds . Only those who passed the trial phase could participate in the official imitation . 2 . During the official imitation , we set up an online chat room to communicate with employees and answer their questions timely . 3 . To embrace semantic diversity , each original example is shown to at least 5 people , that is , one can only conduct 4 imitations based on one original example . People who are assigned to the same example imitate independently without interference from each other , to ensure varied inspiration for imitation . 4 . To ensure logic invariance , we adopt a double-checking strategy : • Cross Checking : Everyday , for each employee , we sample 5 imitations from all of his/her daily imitations . And the sampled imitation is assigned to other 2 employees for cross-checking . The imitation will be qualified only if they both approved , and the criterion for approval is that the imitation share the same logics with the original example . If any one of the 5 imitations produced by one employee fail , then all imitations of that employee for that day will be returned to re-check and repair . • Post Checking : To further ensure that the underlying logic do not deviate during imitation , we introduce another team of experts to solve the imitative examples . The team consists of 20 experienced experts . 10 of them speaks Chinese as native language and have passed CNCSE , while the other 10 speaks English and have passed LSAT . 5 Each imitative example was presented to 3 experts randomly , who are allowed to select one and only choice from “ A ” , “ B ” , “ C ” , “ D ” , otherwise , “ UNABLE TO ANSWER ” if bugs exist in the example , causing no correct choice or multiple correct choices . As long as one of the 3 experts pointed out “ UNABLE TO ANSWER ” , then the imitator of this problem should recheck the logic , until each of these 3 experts could give a choice from “ A ” , “ B ” , “ C ” , “ D ” . Note that in the post checking process , we broke up the imitations together and shuffle randomly , otherwise if a person is faced with imitations from same original example , he/she is prone to give a shortcut option with speculation . Translation Quality Control After collecting high-quality mono-collections , we first adopted Google Translation to translate Chinese/English collections into another language , and then employed 10 professional bilingual experts in Chinese and English for manual correction . Bilingual experts were asked to pay attention to logic-invariance and faithfulness during translation . Next , to ensure translation quality , we also adopted the post checking strategy . That is , we asked the 20 human experts mentioned above to solve the translated examples . Each translated example was presented to 3 experts randomly . Since human experts excel in solving naı̈ve logical reasoning problems , ( i.e . achieve 100 % accuracy on mono-collections ) , if any expert made a mistake on a translated sample or pointed out “ UNABLE TO ANSWER ” , the translated instance is sent back to the bilingual experts for revision . Finally , we asked 50 native speakers to read through all paragraphs of the translation parts in NAIL and mark “ 0 ” / “ 1 ” for each , where “ 1 ” stands for a translated sample is idiomatic , and “ 0 ” otherwise . Then for all samples marked with “ 0 ” ( about 20 % ) , the bilingual experts and native speakers will work together to polish them and conform to the target language norms . Human Evaluation As mentioned above , an imitation is finally regarded as qualified only if the sampled 3 experts could all solve the example . For any original example in NAIL-E , we also ask 3 experts in the team to solve it . Since the gold answers to examples in NAIL-E are provided in public by the examination committee , and corresponding imitations in NAIL-I share the same answer with the original example . Therefore , we calculated the mean accuracy of these three submissions on the overall NAIL , denoted as the performance of human experts . To better demonstrate the 5Experts who are native English speakers will be assigned English problems and experts who are native Chinese speakers will be assigned Chinese problems . difficulty of NAIL , we also selected another team consisting of 20 first-year college students . Same as above , an example is shown to 3 students randomly and they have to give the answer independently . We calculated the overall mean accuracy as the performance of human baseline.The evaluation of human baseline is paid separately . Each person can receive RMB¥1.5 for answering each example , which generally cost the person 2-3 minutes . Since we hire part-time people to write imitatively rather than using existing crowd-sourcing platforms , we build our own website , where the quality and overall progress can be viewed at any time .

Summary: This paper proposes a new dataset that tests the capability of logical reasoning through multiple-choice reading comprehension. Questions are collected from human exams (Chinese National Civil Servants Examination in Chinese and Law School Admission Test in English) and are augmented by imitating the actual exam examples. The main focus of the questions is understanding of the relationship between subject-predicate-object triples, involving four reasoning types: factuality, ordering, matching, and set operation. In the imitation process, the authors hire people from different backgrounds who are asked to write examples that are logically invariant but semantically different from a given actual example (82 people, 813 paid work hours with fair payment,  and at least 20 imitations per actual example). All collected examples are manually validated by experts and answered by college students for measuring human baseline. Experiments with rule-based (word matching and sliding window) and neural network-based (BERT and RoBERTa) models show that the machine performance (30.10 and 36.15 for the English and Chinese sets respectively) is largely lower than the human baseline (71.31 and 76.38), which indicates that these systems struggle to solve the naive logical reasoning questions.