Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning
Year: 2023

Abstract:
Syllogistic reasoning, a typical form of deductive reasoning, is a critical capability widely required in natural language understanding tasks, such as text entailment and question answering. To better facilitate research on syllogistic reasoning, we develop a benchmark called SYLLOBASE that differs from existing syllogistic datasets in three aspects: (1) Covering a complete taxonomy of syllogism reasoning patterns; (2) Containing both automatically and manually constructed samples; and (3) Involving both the generation and understanding tasks. We automatically construct 50k template-based syllogism samples by mining syllogism patterns from Wikidata and ConceptNet. To improve our dataset’s naturalness and challenge, we apply GPT-3 to paraphrase the templatebased data and further manually rewrite 1,000 samples as the test set. State-of-the-art pretrained language models can achieve the best generation ROUGE-L of 38.72 by T5 and the best multi-choice accuracy of 72.77% by RoBERTa on SYLLOBASE, which indicates the great challenge of learning diverse syllogistic reasoning types on SYLLOBASE. Our datasets are released at https://github.com/ casually-PYlearner/SYLLOBASE.

heading: 1 Introduction
text: Reasoning, as a typical way for human beings to obtain new knowledge and understand the world, is also an ultimate goal of artificial intelligence (Newell and Simon, 1956; Lenat et al., 1990). Reasoning skills, i.e., examine, analyze, and critically evaluate arguments as they occur in ordinary language, have been required by many natural language processing tasks, such as machine reading comprehension (Liu et al., 2020; Yu et al., 2020), open-domain question answering (Kwiatkowski et al., 2019; Huang et al., 2019), and text gener-
*Corresponding author.
ation (Dinan et al., 2019).1 According to different mental processes, reasoning can be categorized as deductive, inductive, abductive, etc (Copi et al., 2016). In Piaget’s theory of cognitive development (Huitt and Hummel, 2003), these logical reasoning processes are necessary to manipulate information, which is required to use language and acquire knowledge. Therefore, the study of logical reasoning is worthy of our attention because it is so prevalent and essential in our daily lives.
In this study, we focus on syllogism, which is a typical form of reasoning and has been studied for a long time (it was initially defined in Aristotle’s logical treatises Organon, composed around 350 BCE). As shown in Table 1, a syllogism often contains two premises and a conclusion, where the conclusion can be inferred based on the given premises through a deductive reasoning process.2 Though reasoning-required tasks (such as question answering) have been widely studied, the thorough study to test the deductive reasoning capabilities of a model or system is rare. In the study of syllogism, there are only a few datasets, and they have several limitations: (1) They focus merely on categorical syllogism (shown in Figure 1) (Dames et al., 2020; Dong et al., 2020; Aghahadi and Talebpour, 2022). Even though it is the most common type, syllogisms come in a variety of forms. They involve different reasoning processes and are also beneficial. (2) Some datasets (Dames et al., 2020; Dong et al., 2020) are not in natural language, which are difficult to adapt to inference requirements in real natural language scenarios. (3) More severely, all of them have less than 10k samples, which are insufficient for training deep neural networks.
To support further study on syllogistic reasoning, in this work, we build a new natural language
1The definition of logical reasoning, https: //www.lsac.org/lsat/taking-lsat/test-format/ logical-reasoning.
2There can also be three or more premises. More details are given in Section 3.2.4.
2347
benchmark—SYLLOBASE, which has the following features (some examples are shown in Table 2): First, it is a more complete benchmark that covers five types of syllogisms. Therefore, it can support more fine-grained research on certain types, their interrelationships, and their combined effect on other tasks. Second, all premises and conclusions are written in natural language. It more closely resembles real-world application settings in which natural language descriptions rather than categorized inputs are provided. In addition, the power of large-scale pre-trained language models can also be harnessed effectively. Third, with our proposed automatic construction process, we collect a large number of samples (50k in total). They can support the training of deep neural networks. In order to validate the performance on actual human syllogism, we also manually annotate 1,000 samples as the test set. This test set may also be used independently to assess the reasoning capability of models in a zero-/few-shot manner. Finally, to promote a more comprehensive investigation of syllogistic reasoning, we organize both a generation and an understanding task.
The experimental results indicate that there is a great deal of room for improvement in the syllogistic reasoning capabilities of existing models. Our additional experiments demonstrate the efficacy of transferring knowledge learned from our automatically constructed syllogism to actual human reasoning.

heading: 2 Background and Related Work

heading: 2.1 Syllogism
text: Syllogism is a common form of deductive reasoning. Basic syllogism can be categorized as categorical syllogism, hypothetical syllogism, and disjunctive syllogism. They can be further combined into polysyllogisms. In this section, we use the most common categorical syllogism to introduce the term and structure of syllogism. Other types of syllogism will be introduced in Section 3.
The left side of Figure 1 shows a well-known categorical syllogism about “Socrates is mortal”. We can see a categorical syllogism usually contains two premises and a conclusion. A common term (e.g., “human”) links two premises, and the premises respectively define the relationship between “human” and “mortal” or “Socrates”. The reasoning process is to draw a conclusion based on the premises. A syllogism can also be described by a pattern, as shown in the middle side of Figure 1.

heading: 2.2 Related Work
text: Syllogistic Reasoning Dataset Several syllogistic reasoning datasets have been introduced to promote the development of this field. CCOBRA (Dames et al., 2020) is a dataset with around 10k triplets (major premise, minor premise, conclusion). The task is formed as a single-choice question, and the ground-truth conclusion is shuffled with several distractors. ENN (Dong et al., 2020) is another similar dataset, but the syllogism is constructed from WordNet (Miller, 1995). SylloFigure (Peng et al., 2020) and Avicenna (Aghahadi
and Talebpour, 2022) are two natural language textbased syllogism reasoning datasets, but they are designed for different tasks. SylloFigure annotates the data in SNLI (Bowman et al., 2015), restores the missing premise, and transforms each syllogism into a specific figure.3 The target is to predict the correct figure type of a syllogism. Avicenna is a crowdsourcing dataset, and the syllogism is extracted from various sources, such as books and news articles. These syllogisms are used for both natural language generation and inference tasks.
Different from existing datasets that focus only on categorical syllogism, our SYLLOBASE covers more types and patterns of syllogism and is significantly larger than existing datasets. More detailed comparisons are shown in Table 1.
Logic Reasoning in NLP There are several tasks and datasets related to logical reasoning in NLP. The task of natural language inference (NLI) (Bos and Markert, 2005; Dagan et al., 2005; MacCartney and Manning, 2009; Bowman et al., 2015; Williams et al., 2018), also known as recognizing textual entailment, requires model to classify the relationship types (i.e., contradicted, neutral, and entailment) between a pair of sentences. However, this task only focuses on sentence-level logical reasoning, and the relationships are constrained to only a few types. Another NLP task related to logical reasoning is machine reading comprehension (MRC). There are several MRC datasets designed specifically for logical reasoning, such as LogiQA (Liu et al., 2020) and ReClor (Yu et al., 2020). A paragraph and a corresponding question are given, and the model is asked to select a correct answer from four options. This task requires models to conduct paragraph-level reasoning, which is much more difficult than NLI.
The above logic reasoning NLP tasks attempt to improve models’ general logic reasoning capability, but they pay little attention to different types of reasoning processes, such as deductive reasoning or
3Figures in syllogism, https://en.wikipedia.org/ wiki/Syllogism.
inductive reasoning. In this work, we study a specific form of deductive reasoning, i.e., syllogism. We hope our benchmark can support more in-depth studies on the reasoning process.

heading: 3 Data Construction
text: Our target is to develop a large-scale benchmark and support research on several typical kinds of syllogistic reasoning. It is straightforward to collect data through human annotation, as most existing datasets have explored (Dames et al., 2020; Aghahadi and Talebpour, 2022). However, this method is impracticable for obtaining large-scale data due to the high cost of human annotation. Therefore, we propose constructing a dataset automatically from existing knowledge bases and man-
ually rewriting 1,000 samples as the test set.

heading: 3.1 Data Source
text: Inspired by existing studies (Dong et al., 2020) that collect data from knowledge bases, we choose Wikidata (Vrandecic and Krötzsch, 2014) and ConceptNet (Speer et al., 2017) as our data sources because they contain large-scale high-quality entities and relations. Wikidata is an open-source knowledge base, serving as a central storage for all structured data from Wikimedia projects. The data model of Wikidata typically consists of two components: items and properties. Items represent things in human knowledge. Each item corresponds to a identifiable concept or object, or to an instance of a concept or object. We use entities in the top nine categories, including human, taxon, administrative territorial, architectural structure, occurrence, chemical compound, film, thoroughfare, and astronomical object.4 Then, we use the relationship of instance of, subclass of, and part of to extract triplets. ConceptNet is another open-source semantic network. It contains a large number of knowledge graphs that connect words and phrases of natural language with labeled edges (relations). Its knowledge is collected from many sources, where two entities are connected by a closed class of selected relations such as IsA, UsedFor, and CapableOf. We use ConceptNet to extract the descriptive attributes of the entities obtained from Wikidata. By this means, we can obtain another group of triplets, which are also used for constructing syllogism.

heading: 3.2 Data Processing
text: In this section, we introduce the construction process of five types of syllogism data, respectively. Some examples are shown in Table 2.

heading: 3.2.1 Categorical Syllogism
text: As shown in Table 1, a categorical syllogism is composed of a major premise, a minor premise, and a corresponding conclusion. We first construct premises and then use them to infer the conclusion and form syllogisms.
The premise in a categorical syllogism can be summarized as four propositions according to different quantifiers and copulas:
(1) All S are P ; (2) No S are P ; (3) Some S are P ; (4) Some S are not P ;
4The full list and the statistics are available at: https: //www.wikidata.org/wiki/Wikidata:Statistics.
where S and P are two entities. With different combinations of the four propositions, categorical syllogisms can be categorized into 24 valid patterns. The first part of Table 2 shows an example of Dimatis syllogism, which is one of the valid patterns.5 To construct premises, we use the extracted triplets from Wikidata and ConceptNet. To obtain a proposition which contains negative relationship, we can use the Antonym and DistinctFrom relationship in ConceptNet to construct it. Taking the triplets (chemical compound, subclass of, pure substance) and (chemical compound, Antonym, mixture) as an example, we have:
(1) All chemical compounds are pure substances;
(2) No chemical compounds are mixture;
(3) Some pure substances are chemical compounds;
(4) Some pure substances are not mixture.
By this means, we can obtain various premises, which will be used for constructing syllogisms.
Considering the example in Table 2, which is a Dimatis syllogism, we first sample a triplet (carbon dioxide, IsA, chemical compound). Then, we use the middle term chemical compound to sample another triplet (chemical compound, subclass of, pure substance), which forms the minor premise. Finally, we can generate a conclusion based on the pattern definition. All other different patterns of syllogisms can be constructed in a similar way.

heading: 3.2.2 Hypothetical Syllogism
text: Similar to categorical syllogism, a hypothetical syllogism has two premises and a conclusion. The difference is that the premises have one or more hypothetical propositions. A hypothetical Syllogism has three valid patterns (the full list is in Appendix A), and we use five relations (i.e., Causes, HasSubevent, HasPrerequisite, MotivatedByGoal, and CausesDesire) in ConceptNet to construct hypothetical propositions.
The following pattern is used as an example to illustrate the data construction process:
Premise 1: If P is true, then Q is true. Premise 2: If Q is true, then R is true. Conclusion: If P is true, then R is true.
Specifically, we extract a triplet pair where the tail entity of one triplet is the head entity of another triplet, e.g., (success, CausesDesire, celebrate) and (celebrate, CausesDesire, have a party). This triplet pair can construct premises as success makes
5Other patterns can be referred to in Appendix A.
you want to celebrate, and celebration makes you want to have a party. Then, we can build a hypothetical syllogism according to the pattern, and the corresponding conclusion is success makes you want to have a party. Hypothetical syllogism with other patterns can be constructed in a similar way.

heading: 3.2.3 Disjunctive Syllogism
text: A disjunctive syllogism has two premises: One of them is a compound proposition, which tells that at least one proposition is true; The other premise tells that one proposition in the former premise is false. Then, we can infer another proposition in the former premise is true. For example, if P and Q are two propositions, a disjunctive syllogism can be described as:
Premise 1: P is true or Q is true; Premise 2: P is not true; Conclusion: Q is true.
According to whether the two propositions can be both true, a disjunctive syllogism can be categorized as compatible or incompatible.
We use ten relations in ConceptNet to construct disjunctive syllogism, where eight of them (such as PartOF and HasA) are used for compatible disjunctive syllogism, and the rest two (i.e., Antonym and DistinctFrom) are used for incompatible disjunctive syllogism (all relations we used are listed in Appendix B). Here, we use the incompatible disjunctive syllogism as an example to illustrate the construction process.
We first sample a triplet for an entity, such as (newspapers, CapableOf, come weekly) and (newspapers, CapableOf, come daily). Then, we can construct a premise as newspapers can come weekly or come daily. Next, we obtain another premise, such as some newspapers cannot come weekly. Finally, we can have the conclusion as some newspapers come daily. In this way, we can automatically construct various disjunctive syllogisms based on the triplets in ConceptNet.

heading: 3.2.4 Polysyllogism
text: A polysyllogism is a combination of a series of syllogisms. It usually contains three or more premises and a conclusion. We construct polysyllogisms based on categorical syllogisms, and the construction process can be summarized as the following steps:
(1) We sample a categorical syllogism from our categorical syllogism repository (built in Section 3.2.1).
(2) According to the form of the conclusion, we can get its predicate term and subject term.
(3) We use these terms to traverse the repository and select a premise/conclusion that contains them.
(4) We use the conclusion obtained in the second step and the selected premise/conclusion in the third step as two new premises. Then, we can infer the conclusion and check if the generated syllogism follows a valid pattern.
(5) Repeat the above process, and we can obtain a series of syllogisms.
(6) We use both premises in the first syllogism and the minor premise in all other syllogisms as the premises of the polysyllogism. The conclusion is obtained from the last syllogism’s conclusion. By this means, we can construct a polysyllogism.
We provide an example in the fourth row of Table 2 to illustrate the construction process.

heading: 3.2.5 Complex Syllogism
text: In addition to constructing the previous four types of syllogism, we investigate another new type of syllogism, which is called complex syllogism. A complex syllogism contains two premises and a conclusion, and the premises and conclusion are compound propositions, which contain one or more logical connectives (i.e., not, and, or, and if-then). These logical connectives significantly increase the difficulty of the syllogism. An example of a complex syllogism is shown in the last row of Table 2. The construction steps can be summarized as:
(1) We randomly sample a pattern from hypothetical and disjunctive syllogism as a basic pattern.
(2) We replace the simple propositions in the basic pattern (such as P , Q, and R) by a compound proposition with the logical connectives not, and, and or, (e.g., not P, P or Q, and P and Q).
(3) After the replacement, we can infer the conclusion (according to the pattern we derived, as shown in Appendix A) and construct a complex syllogism.
Rule of Replacement To replace a simple proposition by a compound proposition, we use the Synonyms relation in ConceptNet. For example, considering the proposition something that might happen as a consequence of eating ice cream is pleasure, we use the synonym of the entity ice cream, i.e., cone, and construct a compound proposition as something that might happen as a consequence of eating ice cream and cone is pleasure.

heading: 3.3 Rewriting
text: With the above process, we obtain a large number of syllogisms. However, these syllogisms are constructed based on predefined patterns, which have fixed structures and may contain grammar faults. In our preliminary study, we find that models trained on such pattern-based data have a poor robustness, potentially because the models are overfitting to the patterns rather than learning the real reasoning process. To alleviate this problem, we apply GPT3 (Brown et al., 2020) for rewriting, which has been shown to be effective (Ding et al., 2022). Specifically, we use a prompt with some human-rewritten examples to ask GPT-3 to change the expression of the syllogism but keep its original meaning and pattern. The generated results have good quality in fluency, diversity, and logic, which are suitable for training models (some examples are shown in the bottom of Figure 1, and the detailed process is described in Appendix C).
Furthermore, to test the models’ performance on (real) syllogisms and facilitate future in-depth research, we manually rewrite 1,000 samples from our collected data as a test set. The rewriting process includes filtering the noise, correcting the grammar faults, and paraphrasing (details process is described in Appendix D). Our experiments (see Section 4.4) will show that the test data are very challenging, whereas training on our automatically collected data is still effective.
As yet, we have obtained 50k samples by GPT-3 rewriting, which are used for training and validation, and 1k samples by further human annotation, which are used for testing. All of them are equally distributed over the five types.

heading: 4 Experiments

heading: 4.1 Task Formalization
text: Based on our collected data, we design two tasks:
Conclusion Generation It is a natural language generation task. The model should generate the correct conclusion based on two given premises. Premises and conclusions are natural language text, which can be represented as sequences of tokens. Formally, given two premises P1 = {wP11 , · · · , wP1m } and P2 = {wP21 , · · · , wP2n }, the model is asked to generate the conclusion C = {wC1 , · · · , wCl }, where w is a token. Similar to other text generation tasks, the generation probability of the conclusion is determined
by the product of the probability of each word, which can be described as: P (C|P1, P2) =∏
P (wCi |wC<i, [P1;P2]), where [;] is concatenation operation. More premises can be handled by concatenating all of them as a long sequence.
Conclusion Selection It is a natural language understanding task. The model is asked to select a correct conclusion from four options, where three of them are distractors. Detailed construction process is given in Appendix F. With the above notations of premises and conclusion, we can define the conclusion selection task as:
S(Ci, [P1;P2]) = exp(M(Ci, [P1;P2]))∑4 j=1 exp(M(Cj , [P1;P2])) ,
where S(Ci, [P1;P2]) is the predicted probability of Ci as a correct conclusion, and M(·, ·) is the output logit of the model.
The statistics of our dataset for both tasks are given in Appendix G.

heading: 4.2 Baseline and Evaluation Metrics
text: We compare the performance of several models. For the conclusion generation task, we consider Transformer (Vaswani et al., 2017) and several pretrained models, including GPT-2 (Radford et al., 2019), T5 (Raffel et al., 2020), and BART (Lewis et al., 2020). For the conclusion selection task, we employ BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), and ELECTRA (Clark et al., 2020) as baseline methods. For all pre-trained models, we use the base version.
As for evaluation metrics, following previous studies (Aghahadi and Talebpour, 2022), we use ROUGE-1/2/L (Lin, 2004), BLEU-1/2 (Papineni et al., 2002), and BERT-Score (Zhang et al., 2020) to evaluate the performance of the conclusion generation task. ROUGE and BLEU are commonly used metrics for text generation, and they measure the n-grams overlap between the generated text and the ground-truth text. BERT-Score is a recently proposed model-based metric. It leverages the pre-trained contextual embeddings from BERT and matches words in generated and ground-truth texts by cosine similarity. For the conclusion selection task, we use Accuracy to evaluate the models’ performance. The implementation details are provided in Appendix H.

heading: 4.3 Experimental Results
text: The results of all models on the conclusion generation task are shown in Table 3, while those on the conclusion selection task are reported in Table 4.
For the conclusion generation task, we can see that the overall performance in terms of wordoverlap metrics (such as ROUGE and BLEU) is poor. Given that conclusions are often brief (11.84 tokens on average), these results show that the task is fairly challenging. In contrast, the BERT-Score is high, indicating that models are able to generate some semantically correct contents but cannot organize them into a reasonable conclusion. Furthermore, the pre-trained language models perform significantly better than the vanilla Transformer. We attribute this to the natural language nature of our dataset, and these results suggest that our dataset can help future research on leveraging pre-trained language models to generate logically reasonable texts. Finally, we notice that the performance on the human-written test set and the automatically generated validation set (in Table 15) is close, reflecting the good quality of GPT-3 rewriting.
For the conclusion selection task, the overall accuracy is around 70%, showing a significant deviation from perfection. In Table 4, the model for a single type of syllogism is trained solely on the corresponding type of data. Therefore, the result
of type “All” is not the average result of the five types of syllogisms. We notice that almost all results for ELECTRA are highest, but it has only 70.89 for the type “ALL”. We speculate the reason is that the ELECTRA model is not robust when trained with mixed data, and the data in different types of syllogism might confuse it. Intriguingly, the performance on categorical syllogisms is extremely bad. A potential reason is that this type of syllogisms contains more patterns (e.g., categorical syllogisms have 24 valid patterns). As a comparison, the performance on hypothetical syllogisms is significantly higher since there are only three patterns. We also notice that the performance on polysyllogisms is higher than that on categorical syllogisms, despite the fact that the former is derived from the latter. We speculate the reason is that the polysyllogisms have more abundant information in premises (i.e., multiple premises), which is helpful for pre-trained language models to conduct reasoning.

heading: 4.4 Further Analysis
text: We also explore the following research questions. To save space, we report the results of the conclusion generation task, while similar trends can be observed on the conclusion selection task, which is shown in Appendix. Effect of Automatically Constructed Data In our benchmark, the training data are automatically constructed from knowledge bases, while the test data are human annotated.6 To reveal the relationship between them, we conduct an additional experiment: we split the test set as new training, vali-
6We also perform a human evaluation on 100 automatically constructed samples (20 for each type of syllogisms). About 73% samples are grammatically perfect and logically correct. More details can be referred to at Appendix E.
dation, and test sets with a ratio of 8:1:1 (i.e., 800, 100, and 100 samples respectively). Then, we train models on the new training data and test their performance on the new test data. As a comparison, we also train models that have been pre-trained on the original training data (automatically constructed). The results are illustrated in Table 5.
It is clear to see that training on automatically constructed data is beneficial for learning manually rewritten data. This is due to the fact that the original dataset is large and contains sufficient training signals. This also validates the benefit of our dataset—the knowledge acquired from large-scale data can be transferred to more difficult problems. Transfer Learning SYLLOBASE supports study on five types of syllogisms. We explore their internal relationships through a transfer learning experiment. Besides, we also investigate if the knowledge learned on SYLLOBASE can improve other syllogism datasets (e.g., Avicenna). The results are shown in Table 6. In this experiment, we first train a BART model on one dataset (denoted as “pretraining”), then further train it on another dataset (denoted as “fine-tuning”) and report the results.
In the first group of experiments (the first two
rows), we can see learning categorical syllogisms contributes less to learning hypothetical and disjunctive syllogisms. This confirms our concern that merely studying categorical syllogisms is not enough, and it proves our contribution to syllogism study. In terms of the results in rows (3)-(9), we can generally conclude that learning basic syllogisms is beneficial for learning combined syllogisms, and vice versa. One exception is the result in the row (9), and it indicates that the knowledge learned from the complex syllogisms does not help for learning hypothetical syllogisms. We speculate the reasons are: (a) complex syllogisms have significantly more patterns than hypothetical syllogisms (42 vs. 3), and (b) the premise/conclusion of complex syllogisms is too complicated to form effective knowledge for hypothetical syllogisms. Finally, comparing the results in the row (15) and (16), we can see models trained on SYLLOBASE have good generalizability on other syllogism datasets, demonstrating once again the value of our SYLLOBASE on general syllogism research.
Effect of Context in Premises Existing machine reading comprehension datasets often provide a paragraph for reasoning. Inspired by these tasks, we expand the premises in our generated syllogisms by adding more informative context so as to validate the models’ capability of extracting effective clues and inferring conclusions. Specifically, for each premise in the manually rewritten dataset, we ask the annotators to further collect some relevant information through search engines and add it as the context. After this step, both premises are hidden in paragraphs, which makes it more difficult to infer a correct conclusion (as shown in Table 13). Results of both tasks shown in Table 7 indicate: (1) Existing models are still far from tackling reasoning problems in real life; and (2) Extracting clues (such as premises in our case) before reasoning is a promising solution for reasoning tasks, which could be explored in the future.
Appendix I shows a case study with some modelgenerated conclusions of syllogisms.

heading: 5 Conclusion
text: In this work, we built a large-scale benchmark for natural language syllogistic reasoning. It covers five types of syllogism. The data were automatically constructed from knowledge bases by our proposed construction methods. To evaluate the models’ performance on real human syllogism, we manually rewrite 1,000 samples as the test set. Experiments show that syllogistic reasoning is a very challenging task for existing pre-trained language models. Moreover, our further study indicates that existing models are even farther from tackling syllogistic reasoning in real scenarios.
Ethical Statement
This work constructs a new benchmark for syllogistic reasoning. The main dataset is automatically constructed using entities and their relations from Wikidata and ConceptNet. The construction template is predefined and manually reviewed, so the ethical concerns are avoided. For the human rewriting process, we hire five annotators and require them to avoid any social bias and privacy issues in the rewritten material. The results are randomly shuffled and sent back to them for an ethical review. We pay them roughly $15 per hour for annotation.
Limitations
We build a new benchmark for syllogistic reasoning. The limitations are mainly in the experiments part: (1) Due to the limited human resources, our test set is quite small, which may not support training large models directly. (2) We evaluate all models by comparing their predictions with the groundtruth conclusions, but human performance is not evaluated. As a benchmark, it may be better to provide human performance and show the performance gap of existing models. (3) We have not tested the performance of pre-trained models in terms of logical correctness. This kind of automatic metrics has been rarely studied, which can be a potential direction of our future work.

heading: A Patterns in Syllogism
text: We list all valid patterns in categorical (shown in Table 9), hypothetical (shown in Table 10), and complex syllogisms (shown in Table 11).

heading: B Relations from Wikidata and ConceptNet
text: We list all relations that are used for constructing syllogisms in Table 12. For Wikidata, we use 16 relations, which are all used for constructing categorical syllogisms. As for ConceptNet, we use 15 relations, and they are used for constructing categorical, hypothetical, and disjunctive syllogisms.

heading: C GPT-3 Rewriting
text: GPT-3 is a well-known pre-trained language model, which has demonstrated impressive few-shot performance on a wide range of natural language processing (NLP) tasks. Recently, researchers has tried to use GPT-3 to annotate data for NLP tasks (Ding et al., 2022). Inspired by this, we choose GPT-3 to complete the rewriting task. In our case, we use a prompt to ask GPT-3 to change the expression of the syllogism but keep its original meaning and pattern. We also append some humanrewritten examples in the prompt as few-shot input. The generated results have good quality in fluency, diversity, and logic, which are suitable for training
models. The prompts used for rewriting are listed in Table 16-20.

heading: D Human Rewriting
text: First, 500 samples are randomly collected from each type of syllogism, respectively. Then, we examine the semantics and filter out illogical syllogisms. Next, for the remaining ones, we correct the grammatical problems (if any). Finally, for each premise/conclusion, the language is painstakingly paraphrased. The paraphrasing process is illustrated in Algorithm 1, and an example is given in Table 8. After rewriting, the sample is more diverse, fluent, and closer to real human language.

heading: E Annotation of Automatic Data
text: To evaluate the quality of our automatically generated data, we conduct a human annotation for 100 random samples (20 for each type of syllogisms). The annotators are asked to label whether the samples have grammatical faults and incorrect logic. The overall accuracy is 73%. Concretely, the accuracy is 70%, 90%, 70%, 65%, and 70% for categorical syllogisms, hypothetical syllogisms, disjunctive syllogisms, polysyllogisms, and complex syllogisms, respectively. This result reflects: (1) Our automatic data have fairly good quality. Our experiments in Section 4.4 also validates this. (2) The polysyllogism is hard to construct as it concerns multiple syllogisms.

heading: F Distractor Construction in Conclusion Selection Task
text: In the conclusion selection task (introduced in Section 4.1), we mix the correct conclusion with three distractors. Basically, these distractors are generated from the ground-truth conclusion by changing its quantifier, adding negative words, or exchanging its subject and object. Specifically, for different kinds of syllogisms, we show the distractor generation process by some examples.
Categorical Syllogism For a syllogism as follows:
Premise 1: All m are p. Premise 2: All s are m. Conclusion: All s are p.
We can generate distractors of the conclusion as:
(1) Some s are p. (modify quantifiers) (2) All s are not p. (add negative words) (3) All p are s. (exchange subjects and predicates) (4) Some p are not s. (others)
Hypothetical Syllogism For a syllogism as follows:
Premise 1: If P is true, then Q is true. Premise 2: If Q is true, then R is true. Conclusion: If P is true, then R is true.
We can generate distractors of the conclusion as:
(1) If R is true, then P is true. (exchange propositions) (2) If Q is true, then P is true. (exchange propositions) (3) If R is true, then Q is true. (exchange propositions) (4) P is true. (remove a proposition) (5) Q is true. (remove a proposition) (6) R is true. (remove a proposition) (7) If P is true, then R is not true.
(add negative words)
Disjunctive Syllogism For a syllogism as follows:
Premise 1: P is true or Q is true; Premise 2: P is not true; Conclusion: Q is true.
We can generate distractors of the conclusion as:
(1) Q is not true. (add negative words) (2) P is true. (change a proposition) (3) P is true or Q is not true. (add a proposition)
Polysyllogism Syllogism This kind of syllogism is built on several categorical syllogisms. Therefore, we can use the same distractor construction method as categorical syllogisms.
Complex Syllogism This kind of syllogism is constructed by adding one or model logical connectives to the original premises and conclusions. Therefore, to generate the distractors, we can (1) add or remove the negative connective (i.e., not)
from the original proposition; or (2) replace the connectives in the original proposition by others (e.g., and → or). For example, given a syllogism as follows:
Premise 1: If P is true or if Q is true, then R is true; Premise 2: If R is true, then S is true; Conclusion: If P is true or if Q is true,
then S is true.
We can generate distractors of the conclusion as:
(1) If P is true or if Q is true, then S is not true. (add negative words) (2) If P is true or if S is true, then Q is true. (change a proposition) (3) If P is true and if S is true, then Q is true. (change the logical connective words)

heading: G Dataset Statistics
text: The statistics of our SYLLOBASE is given in Table 14.
H Implementation Details
We use PyTorch (Paszke et al., 2019) and Transformers (Wolf et al., 2019) to implement all models. They are trained on 8 Tesla V100 GPUs with 32GB memory. All hyperparameters (e.g., learning rate) are tuned according to the performance (BLEU1/Accuracy) on the validation set.
In the conclusion generation task, for the decoder-only model GPT-2, the major premise and minor premise are concatenated as a long sequence and fed into the model (decoder) to generate
the conclusion. For the encoder-decoder structure (Transformer, T5, and BART), the two premises are concatenated and input to the encoder, while the conclusion is input to the decoder and used for generation. The maximum generation length is set as 128. The training batch size is set as 32. The AdamW (Loshchilov and Hutter, 2019) optimizer is applied with a learning rate of 5e-5. The learning rate decay mechanism is applied. All models are trained by 10 epochs, and the total training time is around 1.22 hours.
In the conclusion selection task, we concatenate two premises as one sequence, use the conclusion as another sequence, and transform them into the text-pair input format, which is commonly supported by pre-trained language models. For example, the input for BERT is: X = [CLS]P1P2[SEP]C[SEP]. The representation of [CLS] is used for option selection. The maximum sequence length is set as 256. The training batch size is set as 64. A learning rate of 2e-5 with decay mechanism is used. The optimizer is also AdamW. All models are trained by ten epochs, and the total training time is around 3.29 hours.

heading: I Case Study
text: We show some results of BART in conclusion generation task to make a case study. We have listed a good case and a bad case for each type of syllogism. They are shown in Table 21. We can see: (1) The model can generate conclusions that are different from the ground-truth but are also correct in logic. This indicates that pre-trained language models can indeed learn some logic reasoning skills from syllogisms rather than merely “remembering” some fixed patterns. (2) Syllogistic reasoning is still difficult for existing models, and the errors stem from several different aspects. As shown in the hypothetical syllogism, the model generates a semantically correct conclusion, but it is irrelevant to the premises. This problem is identified as “hallucination” of pre-trained language models (Nie et al., 2019), i.e., the model cannot decide whether to generate a conclusion based on its learned parameters or the given context. We believe our dataset can contribute to the study of hallucinations in logical reasoning. As for the last case, the model generates a conclusion opposite to the ground-truth. This indicates that existing models may need additional reasoning modules to conduct complex reasoning problems.
Rewrite the following sentences to standard English. Keep the meaning and pattern of the original sentences, but change the expression of the sentences.
pattern: All m are p. Some s are m. [Therefore], some s are p. original sentences: All sugar are carbohydrate. Some decay teeth are sugar. [Therefore], some decay teeth are carbohydrate. rewritten sentences: Sugars are carbohydrates. Somethings that decay your teeth are sugary foods and drinks. [Therefore], carbohydrate eating can sometimes promote tooth decay. pattern: Some p are m. All m are s. [Therefore], some s are p. original sentences: Some visual art are art of painting. All art of painting are activity. [Therefore], some activity are visual art. rewritten sentences: The visual arts are art forms that create works that are primarily visual in nature, such as painting. Painting is the practice of applying paint, pigment, color or other medium to a solid surface. [Therefore], creativily activties are used to develop new artistic works, such as visual art. pattern: No p is m. All m are s. [Therefore], some s are not p. original sentences: No animal is mineral. All mineral are solid. [Therefore], some solid are not animal. rewritten sentences: Evidently, animal and vegetables are living, minerals not. A mineral is a naturally occurring inorganic solid. [Therefore], some substances are solids and are not living beings. pattern: No p is m. All s are m. [Therefore], No s is p. original sentences: No animal is plant. All rose are plant. [Therefore], no rose is animal. rewritten sentences: Traditionally, Animals cannot produce their own energy which not like plants. A rose is a woody perennial flowering plant of the genus Rosa. [Therefore], roses and animals are extremely different species in nature. pattern: No m is p. All s are m. [Therefore], some s are not p. original sentences: No art is clumsiness. All sculpture are art. [Therefore], Some sculpture are not clumsiness. rewritten sentences: Clumsiness is the lack of gracefulness or skill, whereas Art encompasses a diverse range of skill and techniques. Sculptures are artworks crafted with various media and materials. [Therefore], Sculpture makers are often talented at expressing creativity and not clumsy.
Table 17: GPT-3 rewriting prompts for hypothetical syllogisms.
Rewrite the following sentences to standard English. Keep the meaning and pattern of the original sentences, but change the expression of the sentences.
pattern: If P is true, then Q is true. If Q is true, then R is true. [Therefore], if P is true, then R is true. original sentences: Something you might do while dating is kiss. Something that might happen when you kiss someone is they smile. [Therefore], something that might happen when you dating is they smile. rewritten sentences: When you are dating your beloved, you might have a sweet kiss. When you kiss your partner, you may find yourself smiling. [Therefore], when you are dating, you may find that you always have a smile on your face. pattern: If P is true, then Q is true. If Q is true, then R is true. [Therefore], if P is true, then R is true. original sentences: The effect of diminishing your own hunger is eating. The effect of eating is a full stomach. [Therefore], the effect of diminishing your own hunger is a full stomach. rewritten sentences: We are all aware that in order to reduce our hunger, we must consume food. Having a belly stuffed with comforting food can feel like a warm hug from the inside. [Therefore], We may feel full after we have satisfied our appetite. pattern: If P is true, then Q is true. If Q is true, then R is true. [Therefore], if P is true, then R is true. original sentences: Because you want to enjoy yourself, you would listen to music. Because you want to listen to music, you would hear singing. [Therefore], because you want to enjoy yourself, you would hear singing. rewritten sentences: If you want to enjoy yourself after a long day of work, you may listen to music. You want to hear your favorite musician sing because you appreciate music. [Therefore], because you want to have fun, you want to hear some singing. pattern: If P is true, then Q is true. If Q is true, then R is true. [Therefore], if P is true, then R is true. original sentences: attending a lecture requires you to listen. If you want to listen then you should not talk so much yourself. [Therefore], If you want to attending a lecture then you should not talk so much yourself. rewritten sentences: If you are in a lecture, you should focus your attention on listening and be mindful of disrupting the session due to speaking. If you want to devote your time to following the lecture without distractions, you must be aware of reducing your own babble. [Therefore], If you desire to remain in the lecture, it is key to dial down your chatter.
Table 18: GPT-3 rewriting prompts for disjunctive syllogisms.
Rewrite the following sentences to standard English. Keep the meaning and pattern of the original sentences, but change the expression of the sentences.
pattern: P is true or Q is true. P is not true. [Therefore], Q is true. original sentences: Is the meal hot or cool. The meal are not hot. [Therefore], the meal are cool. rewritten sentences: The meal is warm or cold when the man gets home from work. The food is not warm when the man stays late at work. [Therefore], the meal is cold when the man comes home late. pattern: P is true or Q is true. P is not true. [Therefore], Q is true. original sentences: The ocean is gas or liquid. The ocean is not gas. [Therefore], the ocean is liquid. rewritten sentences: The ocean can exist in either liquid or gaseous form. The ocean is not gaseous. [Therefore], oceans do not exist in a gaseous condition, as far as we know. pattern: P is true or Q is true. P is not true. [Therefore], Q is true. original sentences: Memories are good or sad. Memories are not good. [Therefore], memories are sad. rewritten sentences: People like being engrossed in memories, whether good or sad. Old memories are not always pleasant. [Therefore], memories of the past may cause sadness. pattern: P is true or Q is true. P is not true. [Therefore], Q is true. original sentences: You can use an audience to performing in front of or boost your ego. You can not use an audience to boost your ego. [Therefore], you can use an audience to performing in front of. rewritten sentences: When you’re in front of an audience, you can put on a show or increase your self-esteem. You cannot exaggerate your ego in front of an audience. [Therefore], you can give a performance in front of an audience. pattern: P is true or Q is true. P is not true, [Therefore], Q is true. original sentences: My flowers are ugly or pretty. My flowers are not ugly. [Therefore], My flowers are pretty. rewritten sentences: The blooms in my garden are either comely or unappealing. The blooms in my garden are not unsightly. Therefore, These flowers are indeed attractive.
Table 19: GPT-3 rewriting prompts for polysyllogisms.
Rewrite the following sentences to standard English. Keep the meaning of the original sentences, but change the expression of the sentences.
original sentences: No hypothesis is fact. Some proposition are hypothesis. Some proposition are not fact. All proposition are abstract object. [Therefore], some abstract object are not fact. rewritten sentences: A hypothesis is a proposed explanation that differs from fact. Some propositions are hypotheses. Some propositions are proven not to be facts. Every proposition is an abstract object. [Therefore], some abstract objects do not exist as facts. original sentences: Applied science is science. No Science is art. Human science is science. Some Behavioral genetics are not human science. Behaviour genetics is psychology. Genetics is biology. [Therefore], some applied science are not biology. rewritten sentences: Applied science is science in every sense of the word. Science and art are two distinct forms of scholarship. Human science is a branch of science. Behavioral genetics does not involve any human science. Behavioral genetics is a branch of psychology. Genetics is the study of biology. [Therefore], applied science encompasses more than just biology. original sentences: All feline are animal. no plant are animal. All flowering plants are plants. All tiger are genus Panthera. [Therefore], no Panthera are flowering plants. rewritten sentences: A feline is an animal belonging to the cat family. There are many obvious differences between plants and animals. Flowering plants are plants that produce flowers and fruits. The tiger is a member of the genus Panthera. [Therefore], Panthera is different from flowering plants. original sentences: All medication are drug. All hormone are medication. All plant hormone are hormone. Some plant hormone are gibberellins. All drug are useful. All gibberellins are carboxylic acid. [Therefore], Some carboxylic acid are useful. rewritten sentences: A medication is a type of drug. Hormones are a type of medication. Plant hormones are a subset of hormones. Gibberellins are one type of plant hormone. All drugs have some sort of usefulness. Gibberellins are carboxylic acids. [Therefore], some carboxylic acids can be useful.
Table 20: GPT-3 rewriting prompts for complex syllogisms.
Rewrite the following sentences to standard English. Keep the meaning of the original sentences, but change the expression of the sentences.
original sentences: If you want to eat then you should open the refrigerator and open the chiller. It is eat, and it is open the refrigerator. [Therefore], it is open the chiller. rewritten sentences: If you want to eat something after working out, open the refrigerator and make sure the chiller is in good working order. When you are hungry and want to eat anything, you may open the refrigerator to find some food. [Therefore], you must ensure that the chiller is operational. original sentences: My toes are warmth and affectionateness, or My toes are cold. My toes are not warmth or not affectionateness. [Therefore], my toes are cold. rewritten sentences: The temperature here varies greatly, my toes can be warm and friendly, or they might be freezing. Because of the low temperature, my toes are not warm or loving. [Therefore], my toes are freezing. original sentences: Neon lights are dark, or Neon lights are bright and burnished. Neon lights are not dark, and Neon lights are burnished. [Therefore], Neon lights are bright. rewritten sentences: Neon lights might be dim at times, but most of the time they are dazzling and burnished at night. Neon lights are not dim, and they shine burnished at night. [Therefore], the neon lights are really bright at night. original sentences: Something you might do while not getting exercise is not get tired. Sometimes you don’t getting exercise. [Therefore], You don’t get tired. rewritten sentences: If you do not exercise, you might remain energetic. When you don’t workout occasionally, you will not become exhausted. [Therefore], If you are not exercising you will not get tired.
Table 21: Examples of conclusions generated by BART. The good results and bad results are in different colors.
Categorical Syllogism Premise 1: Carbon dioxide is a chemical compound. Premise 2: Chemical compounds are considered pure substances. Conclusion: Pure substances include carbon dioxide. Prediction: Some completely natural substances include carbon dioxide.
Categorical Syllogism Premise 1: All machines are not human. Premise 2: Every truck crane is a type of apparatus. Conclusion: All truck cranes are not human. Prediction: Some apparatus are not truck cranes.
Hypothetical Syllogism Premise 1: What you might do in class is sleep. Premise 2: A dream is something that can happen to you while you retire for the night. Conclusion: You’re most likely daydreaming in class. Prediction: When you attend class, you may have a dream.
Hypothetical Syllogism Premise 1: When you make progress in your project, you may want to celebrate. Premise 2: Having a party is a good choice if you want to celebrate. Conclusion: You may want to have a party if you achieve great progress in your project. Prediction: Celebrating has the effect of bringing people together.
Disjunctive Syllogism Premise 1: Newspapers are generally published daily or weekly. Premise 2: Some newspapers are not published weekly. Conclusion: Some newspapers are daily newspapers. Prediction: Some newspapers are generally published daily.
Disjunctive Syllogism Premise 1: There is still controversy about whether there are ghosts in the world. Premise 2: The man is an atheist. Conclusion: The man thinks that there is no ghost in the world. Prediction: There are ghosts in the world.
Polysyllogism Premise 1: Some movies are not cartoon movies. Premise 2: Science fiction animations belong to animated films. Premise 3: Remake films are also films. Conclusion: Some remakes are out of scope of science fiction cartoons. Prediction: It exists some remakes are out of scope of science fiction cartoon.
Polysyllogism Premise 1: All Pierce Brosnan are human. Premise 2: No human is a beast. Premise 3: Some Pierce Brosnan are not machines. Conclusion: Some machines are not beasts. Prediction: Some Pierce Brosnan are not beasts.
Complex Syllogism Premise 1: Fencing requires wearing a protective mask and gloves. Premise 2: The woman is fencing and wearing a wiremesh mask. Conclusion: The woman is also wearing gloves. Prediction: The woman may be wearing gloves.
Complex Syllogism Premise 1: If Jack has computer skills and programming knowledge, he could write programs. Premise 2: Jack cannot write computer programs, but he can use computers. Conclusion: Jack does not have programming knowledge. Prediction: He can write computer programs.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitation (Page 9).

heading: 3 A2. Did you discuss any potential risks of your work?
text: We have an ethical statement section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction (Page 1).
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Footnotes in Section 2.2 and Section 3.1 and the References section.

heading: B1. Did you cite the creators of artifacts you used?

heading: No response.
text: 7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No, because the licenses are well known, which allow use of the artifacts in work like ours.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No, the use of existing artifacts is only for research, not for commercial use.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3 Data Construction (Page 3).

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We note the language and domains in section 3.1 Data Source (Page 3).
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 Data Construction (Page 3) and Appendix G Data Statistics (Page 15).
C 3 Did you run computational experiments?

heading: Appendix H Implementation Details (Page 15).

heading: C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix H Implementation Details (Page 15).

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.3 Experimental Results.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We used the transformers package for training baseline models and the ROUGE, BLEU, BERT-Score packages for evaluation. Since we only used the common functions/interfaces well known in the NLP, we did not discuss the details.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Appendix D Human Rewriting.
text: 7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No, the human annotators are required to just rewrite the automatic samples, which is unnecessary to give a instruction.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section Ethical Statement.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No, the human annotators are required to just rewrite the automatic data, which is unnecessary to give a instruction.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Our benchmark has no social impacts and we just use some open knowledge bases, like ConceptNet and Wikidata. There is no need to get the approval of an ethics review board.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No, we required the annotators to avoid any social bias and privacy issues in the rewritten material, which is discussed in the Section Ethical Statement.

heading: Future Work
text: This kind of automatic metrics has been rarely studied, which can be a potential direction of our future work