Venue: NAACL 2024

# A Closer Look at the Self-Verification Abilities of

Large Language Models in Logical Reasoning

Ruixin Hong1,2; Hongming Zhang2, Xinyu Pang1, Dong Yu2, Changshui Zhang1

###### Abstract

Logical reasoning has been an ongoing pursuit in the field of AI. Despite significant advancements made by large language models (LLMs), they still struggle with complex logical reasoning problems. To enhance reasoning performance, one promising direction is scalable oversight, which requires LLMs to identify their own errors and then improve by themselves. Various self-verification methods have been proposed in pursuit of this goal. Nevertheless, whether existing models understand their own errors well is still under investigation. In this paper, we take a closer look at the self-verification abilities of LLMs in the context of logical reasoning, focusing on their ability to identify logical fallacies accurately. We introduce a dataset, Fallacies, containing 232 types of reasoning fallacies categorized in a hierarchical taxonomy. By conducting exhaustive experiments on Fallacies, we obtain comprehensive and detailed analyses of a series of models on their verification abilities. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods. Drawing from these observations, we offer suggestions for future research and practical applications of self-verification methods.1

Footnote 1: Data is available at [https://github.com/Raising-hrx/FALLACIES](https://github.com/Raising-hrx/FALLACIES).

1Beijing National Research Center for Information Science and Technology (BNRist);

1Department of Automation, Tsinghua University, Beijing, P.R.China

2Tencent AI Lab, Seattle hrx20@mails.tsinghua.edu.cn, hongmzhang@tencent.com,

## 1 Introduction

Logical reasoning is not only a crucial aspect of human intelligence but also one of the long-term pursuits of artificial intelligence McCarthy (1989). It is indispensable in intelligent systems, enabling problem-solving, decision-making, and critical thinking. Large language models (LLMs) have recently achieved remarkable advancements in a wide range of tasks OpenAI (2023). Being prompted appropriately, LLMs exhibit the emergent ability to "reason" step by step like humans Wei et al. (2022a, b). Nonetheless, increasing research suggests that LLMs struggle with intricate logical reasoning problems, occasionally producing unfaithful reasoning steps fraught with logical fallacies Arkoudas (2023); Liu et al. (2023); Pan et al. (2023); Hong et al. (2023).

To tackle this issue, a prevalent and promising approach is scalable oversight Bowman et al. (2022), where the LLMs could be boosted based on their own evaluation signals Leike and Sutskever (2023). In this regard, various strategies of self-verification using LLMs are proposed to enhance reasoning performance Ling et al. (2023); Weng et al. (2022); Xie et al. (2023); Madaan et al. (2023); Miao et al. (2023). As shown in Figure 1, the LLMs first generate the reasoning process and then self-verify their own output. The verification results are then used to refine the output or further improve the models. Their _assumption_ is that LLMs can reliably identify fallacious reasoning steps.

Though empirical evidence demonstrates their

Figure 1: The self-verification approach requires LLMs to identify the fallacious steps in their own reasoning process. However, LLMs might be susceptible to certain types of fallacies and fail to identify them, leading to the potential failure of self-verification.



preliminary success, a thorough and comprehensive evaluation of the underlying assumption remains unexplored. First, these efforts typically use the performance of the final task (e.g., answer accuracy) to illustrate their effectiveness. However, this is a proxy metric that does not directly reflect the ability of LLMs to identify logical fallacies. LLMs might possibly arrive at the correct answer despite the existence of fallacious intermediate steps Wei et al. (2022b); Creswell et al. (2023); Lanham et al. (2023). Second, they are usually only concerned with _whether or not_ the reasoning step is fallacious, rather than _what type of_ fallacy exists in the step, which could be informative for the self-improvement of LLMs. Such an oversimplification precludes a definitive analysis of the LLMs' ability to identify different types of fallacies.

In this paper, we provide a comprehensive evaluation of the verification abilities of LLMs in logical reasoning. Specifically, we collect a dataset, Fallacies, containing 4,640 reasoning steps for 232 types of fallacies. We evaluate whether LLMs can distinguish between correct and fallacious reasoning steps. Such directed evaluation can provide a more accurate reflection of the verification abilities of LLMs, as it steers away from proxy metrics and delves straight into the actual performance in identifying fallacious steps. Compared to previous datasets, our dataset features more types of fallacies, a larger scale, more fine-grained reasoning, and explicit premises and conclusions. Furthermore, we adopt a hierarchical taxonomy of fallacies, which divides fallacies into two main categories and nine subcategories, allowing for a more systematic approach toward analyzing the verification abilities of LLMs across varying aspects.

We conduct exhaustive experiments on a range of LLMs. First, experimental results show that most LLMs struggle with accurately identifying the fallacious steps. Most LLMs only achieved an overall accuracy rate of less than 80%, suggesting that LLMs could lack sufficient logical verification abilities. Thus, we should be more cautious about the self-verification methods of LLMs. Second, the performance of LLMs can be remarkably imbalanced in different types of fallacies. Most LLMs perform much worse at identifying fallacies related to logical structure than those related to content, pointing toward key directions for improving LLMs' verification and reasoning abilities. Third, we also find that LLMs encounter challenges when it comes to classifying different types of fallacies. Presenting LLMs with the definitions of fallacies does not appear to improve their ability to recognize fallacies. This raises a call for further research to delve into the underlying mechanisms through which LLMs understand reasoning and fallacies. In summary, we present a comprehensive evaluation of the verification abilities of LLMs and highlight their limitations in identifying fallacies, urging the community to apply the self-verification methods with caution.

## 2 Related Work

### Language Models for Logical Reasoning

Compared with traditional symbol-based logic reasoning systems, using language models to directly reason over natural language is a more flexible and popular approach Yu et al. (2023); Yang et al. (2023b). Many efforts have been devoted to improving the logical reasoning abilities of language models from different perspectives, including finetuning methods Clark et al. (2020); Dalvi et al. (2021), pre-training methods Pi et al. (2022); Jiao et al. (2022), and modular methods Hong et al. (2022); Yang et al. (2022). As the model scale increases, specially designed prompts (e.g., Chain-of-Thought prompts Wei et al. (2022b)) can elicit the step-by-step reasoning abilities of LLMs, achieving remarkable improvement in multiple reasoning tasks Chu et al. (2023). However, some studies find that LLMs still struggle with complex logical reasoning problems Arkoudas (2023) and could be susceptible to logical fallacies Payandeh et al. (2023). There is still a lack of comprehensive research to investigate the understanding of LLMs on different logical fallacies.

### Self-Verification with Large Language Models

A prevalent method to enhance the capacity of LLMs is through learning and correction via high-quality verification feedback. This feedback proves instrumental in various aspects of model optimization. For instance, it can be used to fine-tune the models Ouyang et al. (2022); Scheurer et al. (2023); Chen et al. (2023a); Zelikman et al. (2022); Wang et al. (2023); Bai et al. (2022); Huang et al. (2022). In addition, verification feedback can be utilized to re-rank the model outputs Lightman et al. (2023); Weng et al. (2022); He et al. (2023); Ni et al. (2023); Ling et al. (2023). It can also be em 

ployed to refine the outputs (Madaan et al., 2023; Shinn et al., 2023; Chen et al., 2023b) and guide the generation process of the models (Yao et al., 2023; Xie et al., 2023). A straightforward approach to collecting feedback is to collect directly from humans (Ouyang et al., 2022; Fernandes et al., 2023), but this could prove to be costly and unable to provide instant feedback. Alternatively, collecting feedback from external tools (Gou et al., 2023; Chern et al., 2023) or metrics (Jung et al., 2022) could be more feasible but limited to specific tasks. Thus, some researchers turn to use large language models themselves to provide verification feedback (Ling et al., 2023; Weng et al., 2022; Xie et al., 2023; Madaan et al., 2023; Miao et al., 2023), which is more scalable. Nonetheless, recent papers (Huang et al., 2023; Valmeekam et al., 2023) have raised doubts about the self-verification abilities of the LLMs. For instance, Huang et al. (2023) find that LLMs struggle to self-correct their responses without external feedback. However, they still leave some open questions for subsequent research, such as what exactly the performance of LLMs is to verify a single reasoning step. In this paper, we delve deeply into this subject, critically examining the verification abilities of LLMs from the perspective of logical reasoning.

## 3 Fallacies

This section outlines the design principles and process of constructing our dataset Fallacies.

### Design Principles

**Covering more comprehensive error types:** We propose to evaluate the verification abilities of LLMs on a wider range of types of errors. Relying solely on a single existing dataset of logi

\begin{table}
\begin{tabular}{l c} \hline \hline
**Category** & **\# Fallacy** \\ \hline
**Formal** & **24** \\ proposition (prop.) & 6 \\ quantification (quant.) & 6 \\ syllogism (sy1.) & 8 \\ probability (prob.) & 4 \\ \hline
**Informal** & **208** \\ ambiguity (amb.) & 15 \\ inconsistency (incon.) & 3 \\ irrelevance (irrel.) & 78 \\ insufficiency (insuf.) & 58 \\ inappropriate presumption (inappr.) & 54 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Distribution of 232 fallacies in Fallacies.

Figure 2: The hierarchical taxonomy of fallacies. For each sub-category, we present its definition and an example of a fallacy within the sub-category. We use square brackets to indicate the premises and conclusions.

 

cal reasoning might pose a challenge in drawing comprehensive conclusions, as it may struggle to cover the types of errors that can occur in realistic scenarios. For example, synthetic datasets (e.g., ProofWriter Tafjord et al. (2021)) are generated from fixed logic templates and vocabularies, which could leave out the reasoning errors caused by verbal ambiguity.

**Hierarchical fallacy taxonomy:** We propose to categorize reasoning errors at a fine-grained, hierarchical level. This approach enables a more thorough assessment of the performance of LLMs across various types of fallacies and offers a more comprehensive perspective on their effectiveness. Meanwhile, we ensure that each reasoning step exclusively pertains to a single type of fallacy to prevent different types of fallacies from potentially confounding one another.

**Clarifying the premises and conclusions:** The third principle is to indicate the premises and conclusions of the reasoning step explicitly. Reasoning is the process of concluding based on known information, and the validity of reasoning cannot be properly judged without providing premise information. Factors like linguistic omissions and implied background knowledge can complicate reasoning in natural language. Consequently, in our dataset, we strive to ensure that each step is a complete unit of reasoning that contains enough information to verify its correctness.

### Taxonomy of Fallacy

A fallacy is an error in reasoning Bennett (2012); Jevons (1872). It focuses more on whether the conclusion of an argument can be logically deduced from the premises rather than on factual errors or other aspects Lau (2011). Identifying logical fallacies is essential for a judicious reasoning system. Without the ability to identify logical fallacies, a reasoning system may lack essential critical thinking skills, leaving it susceptible to illogical arguments and deliberate manipulation.

Fallacies can be classified into two primary categories: **Formal Fallacies**, errors due to invalid logical structures or inference patterns, and **Informal Fallacies**, errors due to the content of premises and conclusions. Based on specific causes of error, we divide each main category into several subcategories. Each subcategory contains several finest-grained fallacies, divided based on more detailed causes of errors. Figure 2 demonstrates our hierarchical taxonomy of fallacies. We derive this taxonomy by integrating multiple scholarly resources Fieser and Dowden (2011); Magnus (2005); Rescher and Schagrin (2023).

### Data Collection

We first collect 232 types of fallacies from classic academic books Bennett (2012); Fieser and Dowden (2011). The authors of these books carefully collect these fallacies from many available academic resources (including peer-reviewed journals, encyclopedias, and books), covering a substantial portion of common errors in logical reasoning. We collect the definition of each fallacy, i.e., an article containing the description and examples of that fallacy. These definitions are used in academic books to help humans understand the meaning of fallacies. Figure 3 in the Appendix shows an example of the definition. We then assign these 232 fallacies to appropriate categories according to the taxonomy in Sec. 3.2. Table 1 shows the distribution of fallacies. Detailed categorizations and descriptions of these fallacies are in Appendix Table 11.

Then, we collect fallacious and correct reasoning steps. Considering that directly creating fallacious steps from scratch can be challenging, we adopt a strategy in which powerful LLMs first generate candidates, and then we let human experts revise them. Specifically, we ask GPT-4 OpenAI (2023) to generate fallacious steps based on the collected definitions of fallacies. To generate diverse steps covering a wider range of domains, we explicitly require the model to generate around a theme, a randomly sampled noun from ConceptNet Speer et al. (2017). Detailed prompt can be found in Appendix A. Subsequently, human experts carefully proofread and refine these candidates to ensure they fall within the corresponding fallacies. Meanwhile, we require experts to make each step a single inference Dagan et al. (2013), rather than complex reasoning that involves multiple intermediate steps.

We require the experts to fix the fallacious steps to collect the correct contrastive steps. The experts make as few modifications as possible to turn the fallacious steps into correct ones, which do not contain any reasoning errors. For instance, the fallacious step of the fallacy of _Affirming the Consequent_ in Figure 2 can be fixed into _"Since [if it is sunny today, then I'll go to hike] and [It is sunny today], therefore, [I'll go to hike today]."_ More contrastive samples can be found in Table 4 

In this way, we obtain ten fallacious and ten correct steps for each of the 232 types of fallacies. To check data quality, we ask three additional experts to re-annotate 50 randomly sampled steps. They annotate each step as a correct step, a fallacious step that belongs to the corresponding fallacy, or a fallacious step that does not belong to the corresponding fallacy. Their average agreement with the labels achieves 0.856 (Cohen's Kappa), indicating the high quality of our data.

### Comparison with Existing Dataset

We compare our dataset and existing fallacy-related datasets in Table 2. Fallacies holds significant advantages across multiple dimensions. It encompasses a broader spectrum of fallacy types, presenting a fine-grained and hierarchical taxonomy of fallacies. Additionally, Fallacies stands out for its clarity and subtlety, avoiding ambiguous judgments about the correctness of reasoning. We make explicit the premises and conclusions of each reasoning step and the granularity of reasoning in Fallacies is finer. In contrast to the most recent dataset Jin et al. (2022), which contained only fallacious reasoning, our dataset includes both correct and fallacious reasoning steps.

## 4 Experiments

### Models and Prompts

We test a range of common LLMs, including GPT-4 (OpenAI, 2023), GPT-3.5 Peng et al. (2023), Llama2 Touvron et al. (2023), Vicuna Zheng et al. (2023), WizardLM Xu et al. (2023), FlanT5 Chung et al. (2022), Falcon Almazrouei et al. (2023), Baichuan2 Yang et al. (2023), ChatGLM Du et al. (2022), and InternLM Team (2023). We use the default generation parameters in the models' configuration files.2 Details about the models (e.g., the version numbers) can be found in Table 7 in the Appendix. We evaluate the models using the same simple prompt in the zero-shot setting. For the prompt selection, we followed the experience of previous work Liu et al. (2023b) and carefully designed and experimented with several different styles of prompts. In the end, we chose a prompt that worked well for all models as our prompt. Appendix B presents a detailed ablation analysis of prompt selection, as well as experiments under few-shot setting.

Footnote 2: For GPT-4 and GPT-3.5, we set the temperature parameter to 0. Experiments were conducted mainly in Nov. 2023.

### Metrics

We evaluate LLMs on 4,640 steps in Fallacies. We calculate the accuracy for each type of fallacy separately. The accuracy of a higher-level category is the (macro) average of the accuracies of its subcategories. Ultimately, we take the average accuracy on formal and informal categories as the overall accuracy.

### Can LLMs accurately identify fallacious steps?

Table 3 shows the accuracies of the different models for identifying fallacious steps on Fallacies. Based on the experimental results, we have the following observations.

**Identifying fallacious steps is still challenging for LLMs.** Most LLMs struggle with accurately identifying the fallacious steps. As shown in Table 3, the performance of most LLMs in this binary classification task ranges from 60% to 80%, indicating the complexity of this task. The best result is achieved by GPT-4, which reaches an overall average accuracy of 87.7%. However, this may still fall short of guaranteeing the validity of the self-verification approach since it is only the performance of identifying single-step fallacies. For a long argument comprising multiple reasoning steps, the overall verification performance for the

\begin{table}
\begin{tabular}{l c c c c c c} \hline \multirow{2}{*}{**Dataset**} & **Number** & **Number** & ** Taxonomy** & **Granularity** & **Explicit Premises** & **Identifying Fallacy** \\  & **of Fallacies** & **of Steps** & **of Fallacy** & **of Reasoning** & **and Conclusions** & **from Correct Reasoning** \\ \hline Stab and Gurevych (2017) & 1 & 1,029 & No & Coarse & No & **Yes** \\ Habernal et al. (2018) & 1 & 2,085 & No & Coarse & No & **Yes** \\ Jin et al. (2022) & 13 & 2,449 & Coarse & Coarse & No & No \\ \hline
**Fallacies (Ours)** & **232** & **4,640** & **Fine \& Hierarchical** & **Fine** & **Yes** & **Yes** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of Fallacies with existing fallacy-related datasets.

 

entire argument could be the product of the verification performances of each individual step. Consequently, the overall verification performance of the argument might decrease exponentially with the number of steps in it. These results suggest that we need further research on existing self-verification methods to understand how they work and under what situations they can provide correct verification feedback.

**Formal fallacy is more difficult than informal fallacy for LLMs.** The performance of most LLMs on formal fallacies is much lower than on informal fallacies. For example, GPT-3.5 achieves 87.9% accuracy on formal fallacies, while it achieves only 74.1% accuracy on informal fallacies. As stated in Sec. 3.2, formal fallacies are more related to the logical structure of reasoning and require a greater emphasis on the understanding and utilization of formal logic. On the other hand, informal fallacies focus more on the content and semantics of the reasoning and may involve factors such as linguistic expression, semantic understanding, and semantic relevance. Therefore, the differences in the performance of LLMs across different types of fallacies may stem from their ability to understand logical structures and semantic meanings, i.e., the models present some challenges in dealing with the logical structures under the natural language, whereas they perform better in dealing with issues related to content and semantics. Taking a more fine-grained view, among formal fallacies, most models typically perform well on syllogism falacies and poorly on proposition and probability fallacies. For informal fallacies, most models perform worse on ambiguity fallacies and have closer performance on the remaining four sub-categories.

**The performance of the same model on different types of fallacies can be remarkably im 

balanced.** For example, the model Qwen-14B achieves an impressive 91.7% accuracy on the inconsistency fallacies but drops to a mere 67.5% on probability fallacies. This highlights a key observation that models may have superior verification abilities on some types of fallacy but are not necessarily equally well adapted to other types of fallacy. Such imbalance performance could be particularly important for practical applications, as different types of fallacies are not always evenly distributed in a given dataset or scenario, and certain types of fallacies might be more frequent. Therefore, we should not rely on a particular dataset when using or researching self-verification methods. Instead, we need to comprehensively consider the performance of the methods in dealing with different fallacies and scenarios.

**GPT-4 achieves superior performance, particularly in identifying formal fallacies.** Compared to the other models, GPT-4 achieves the best results in overall average accuracy. This gap is insignificant on informal fallacies, where models such as WizardLM-13B, Vicuna-13B, and Qwen-14B have comparable or even better average accuracies than GPT-4 on informal fallacies. However, on formal fallacies, GPT-4's accuracy is 13.5% higher than the second-best model (88.2% for GPT-4 compared to 74.7% for the second-best model Qwen-14B). The results suggest that GPT-4 demonstrates superior abilities in identifying fallacies related to logical structures than other LLMs. However, there is still room to improve the performance of GPT-4. Table 4 demonstrates the prediction results of GPT-4 in some cases. It can be seen that GPT-4 can also fail on some challenging samples.

### Can LLMs distinguish types of logical fallacies?

In addition to whether LLMs can identify between correct and incorrect, we are also interested in whether LLMs can distinguish between different types of fallacies. Classifying types of fallacies requires the model to understand not only the pattern of errors in reasoning but also where the errors occur and why they may occur, which requires a higher level of reasoning ability. Given an error reasoning step, we require the model to recognize the error pattern within it and classify it as one of the 232 fallacies. We conduct experiments on the 2,320 fallacy steps in Fallacies and calculate the macro average accuracy. Previous work has also explored this task and named it "logical fallacy

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c|c} \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c|}{**Formal**} & \multicolumn{4}{c|}{**Informal**} \\  & **prop.** & **quant.** & **syL.** & **prob.** & **Avg.** & **amb.** & **incon.** & **irrel.** & **insuf.** & **inappr.** & **Avg.** & **Avg.** \\ \hline Random & – 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 & 0.4 \\ \hline Flan-T5-Large & 15.0 & 1.7 & 0.0 & 0.0 & 4.2 & 0.0 & 0.0 & 7.7 & 4.7 & 3.9 & 3.2 & 3.7 \\ Flan-T5-xl & 21.7 & 0.0 & 0.0 & 5.0 & 6.7 & 0.0 & 0.0 & 16.4 & 10.7 & 7.8 & 7.0 & 6.8 \\ Flan-T5-xcl & 28.3 & 26.7 & 0.0 & 22.5 & 19.4 & 2.7 & **16.7** & 8.5 & 9.5 & 7.4 & 8.9 & 14.2 \\ Llama2-7B & 16.7 & 0.0 & 0.0 & 0.0 & 4.2 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.2 & 2.2 \\ Llamaa-13B & 8.3 & 0.0 & 0.0 & 0.0 & 2.1 & 0.0 & 0.0 & 0.0 & 0.0 & 0.2 & 0.0 & 1.1 \\ Llama2-7OB & 30.0 & 0.0 & 0.0 & 0.0 & 7.5 & 4.7 & 0.0 & 14.6 & 8.1 & 10.0 & 7.5 & 7.5 \\ S Baichuan2-7B & 0.0 & 0.0 & 1.2 & 0.0 & 0.3 & 0.0 & 0.0 & 3.5 & 0.9 & 3.0 & 1.5 & 0.9 \\ Baichuan2-13B & 0.0 & 0.0 & 0.0 & 2.5 & 0.6 & 0.0 & 0.0 & 13.7 & 3.1 & 4.6 & 4.3 & 2.5 \\ ChatidM-6B & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ ChatGLMLQ-6B & 0.0 & 0.0 & 0.0 & 2.5 & 0.6 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.2 & 0.4 \\ InterLM-7B & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.3 & 0.0 & 0.4 & 0.1 & 0.1 \\ InternLM-20B & 3.3 & 1.7 & 2.5 & 12.5 & 5.0 & 0.7 & 0.0 & 4.9 & 1.0 & 2.2 & 1.8 & 3.4 \\ Falcon-7B & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.1 & 1.0 & 0.0 & 0.2 & 0.1 \\ Wiizandt-13B & 15.0 & 0.0 & 0.0 & 7.5 & 5.6 & 0.0 & 0.0 & 1.0 & 0.5 & 0.2 & 0.3 & 3.0 \\ Vicuna-7B & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.5 & 0.0 & 0.0 & 0.1 & 0.1 \\ Vicuna-13B & 31.7 & 0.0 & 0.0 & 0.0 & 7.9 & 0.0 & 0.0 & 2.2 & 0.0 & 0.6 & 0.5 & 4.2 \\ Qwen-14B & 10.0 & 0.0 & 0.0 & 20.0 & 7.5 & 0.0 & 0.0 & 5.5 & 2.1 & 0.6 & 1.6 & 4.6 \\ GPT-3.5 & 40.0 & 20.0 & 3.8 & 22.5 & 21.6 & 16.0 & 0.0 & 22.7 & 16.4 & 9.6 & 12.9 & 17.3 \\ GPT-4 & **58.3** & **31.7** & **20.0** & **55.0** & **41.2** & **40.0** & **16.7** & **31.9** & **27.2** & **27.8** & **28.7** & **35.0** \\ \hline \end{tabular}
\end{table}
Table 5: Accuracy results (%) on classifying the fallacy types of fallacious steps.

 

detection" (Jin et al., 2022). However, they only classify over 13 types of fallacies, whereas our task requires classifying over 232 types. Our task is more challenging and allows for fine-grained and hierarchical analysis. We evaluate the model using the same prompt in a zero-shot setting.

Table 5 shows the results. First, this task is very challenging for the existing LLMs. The models' performances are poor, with less than 10% overall accuracy, except for three models, GPT-4, GPT-3.5, and Flan-T5-xxl. For example, Vicuna-13B can achieve 78.1% accuracy on identifying fallacious steps, but only 4.2% accuracy on this task. Among all the models, GPT-4 performs the best, achieving an overall accuracy of 35.0%. This indicates that GPT-4 can recognize and classify the reasoning error patterns to a certain degree, showing a stronger reasoning ability than other models. Nevertheless, there is still substantial room for improvement. Further research may be required to achieve higher accuracy and enhance reasoning ability.

It is worth noting that on this task, the models typically perform better on formal than informal fallacies. This is inconsistent with the observation in identifying fallacious steps (Table 3). When identifying fallacious steps, models typically perform worse on formal fallacies and better on informal fallacies. There could be various reasons for this inconsistency. One possible explanation is that the models might just know the names of the fallacies rather than having an in-depth understanding of what these fallacies are. In determining whether there is an error in reasoning, the models might not be relying directly on their understanding of reasoning or fallacies but on some other abilities, which thus contributes to this inconsistency.

### Can LLMs understand fallacies better from their definitions?

We test whether LLMs can perform better in identifying fallacious steps given the fallacy definition. Specifically, for each step in Fallacies, we add the name and definition of its corresponding fallacy to the prompt of LLMs in advance. We then ask the model to determine whether the reasoning step is correct as before. As stated in Sec. 3.3, the definitions of fallacies are gathered from academic sources. An example of the definition can be found in Figure 3 in the Appendix.

We can observe a surprising trend by analyzing the results in Table 6. When definitions of corresponding fallacies are provided in advance, most models' performance decreases rather than improves. For instance, the overall accuracy of Vicuna-13B decreases from 78.1% to 64.6%, with a 13.5% decrease. These results suggest that providing definitions may hurt models' performance.

The reasons for this phenomenon deserve further exploration. One possible reason is that, in pre-trained data, the definitions and the fallacies themselves may not co-occur frequently, resulting in a mismatch with the current setting. Moreover, the mechanism by which the models judge fallacies has not yet been fully clarified. In this case, even if the definitions are provided, the models fail to improve performance. Instead, the prompt becomes complex with the addition of definitions, possibly interfering with their decision-making process. These observations inspire us that more intensive research is called to understand what are the mechanisms by which LLMs understand the reasoning and fallacies.

\begin{table}
\begin{tabular}{l|c c|c} \hline \hline
**Model** & **Formal** & **Informal** & **Avg.** \\ \hline Flan-T5-Large & 62.3(+1.3) & 65.5(-8.1) & 63.9(-3.4) \\ Flan-T5-xl & 56.6 (-11.3) & 67.1 (-7.0) & 61.8 (-9.2) \\ Flan-T5-xxl & 66.2 (+2.3) & 73.1 (+0.1) & 69.7 (+1.3) \\ Llama2-7B & 50.9(-10.4) & 52.0(-19.8) & 51.5(-15.0) \\ Llama2-13B & 57.9 (-2.0) & 53.2 (-10.0) & 55.6 (-5.9) \\ Lama2-70B & 54.0(-19.3) & 56.9 (-31.3) & 55.4 (-2.9) \\ Baichuan2-7B & 52.8 (+0.1) & 58.0(+5.7) & 55.4 (+2.9) \\ Baichuan2-13B & 50.1 (-17.9) & 52.7 (-28.2) & 51.4 (-23.0) \\ ChatGLM-6B & 54.6 (-1.2) & 58.0 (+3.6) & 56.3 (+1.2) \\ ChatGLM2-6B & 58.0 (-3.7) & 61.9 (-13.3) & 60.0 (-8.4) \\ InternLM-7B & 55.1 (-5.5) & 59.4 (-16.0) & 57.2 (-10.8) \\ InterLm-20B & 59.2 (-4.4) & 68.6 (-8.9) & 63.9 (-6.7) \\ Falcon-7B & 40.2 (-2.8) & 46.9 (-23.3) & 43.5 (-13.1) \\ WizardML-13B & 74.2 (+6.7) & 82.9 (-3.6) & 78.5 (+1.5) \\ Vicuna-7B & 70.4 (+2.0) & 78.0 (-4.2) & 74.2 (-1.1) \\ Vicuna-13B & 61.6 (-8.2) & 67.7 (-18.7) & 64.6 (-13.5) \\ Qven-14B & 71.1 (-3.6) & 79.2 (-9.0) & 75.2 (-6.3) \\ GPT-3.5 & 75.1 (+1.0) & 75.7 (-12.2) & 75.4 (-5.6) \\ GPT-4 & 91.2 (+3.0) & 84.9 (-2.3) & 88.0 (+9.3) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Accuracy results (%) of identifying fallacious steps given the definitions of corresponding fallacies. We present the performance variations in the case with the definitions in parentheses compared to the case without definitions.

 

Conclusion

In this paper, we take a closer look at the verification abilities of LLMs in logical reasoning. We collect a dataset containing 232 fallacies and propose a hierarchical taxonomy of fallacies. Our main experimental finding is that most LLMs still struggle to identify fallacies in logical reasoning accurately. This implies that it may be overly optimistic to expect LLMs to be able to inherently identify errors and conduct self-verification reasoning, at least with respect to the current state of technology. Therefore, researchers and practitioners should be more cautious in using self-verification methods. We call for more research to explore the potentials and limitations of self-verification methods to steer LLMs towards improved accuracy and reliability.

## Acknowledgements

We appreciate the anonymous reviewers for their insightful comments. This work is supported by the Natural Science Foundation of China (NSFC No. 62176132) and the Guoqiang Institute of Tsinghua University (2020GQG0005).

## Limitations

In this paper, we present an intensive investigation and evaluation of the verification abilities of Large Language Models (LLMs) for logical reasoning. Although we have reached some findings, we are also aware that there is still some room for improvement and future research areas worth exploring.

Firstly, we conduct experiments only on the most common LLMs. Such a limitation comes from two main reasons: one is due to the limitation of computational resources of our research team; and the other is the barrier of access to certain closed-source models. This results in our inability to perform detailed experiments on all types and all scales of models. Thus, our results may not fully reflect the abilities of all LLMs. In future studies, it is worthwhile to turn our attention to more types and scales of models to provide a more comprehensive evaluation of their abilities.

Second, our study focused mainly on the aspect of logical reasoning. Reasoning in real-world applications often encompasses other types of reasoning, such as numerical reasoning. It would be interesting to extend our research to more types of reasoning. By doing so, we can reveal the boundaries of the abilities of LLMs in these areas, which can enhance our deeper understanding of the performance of LLMs. Meanwhile, it would also help us to figure out how we can improve the accuracy and robustness of the reasoning of LLMs.

This article follows the ACL Code of Ethics. To the best of our knowledge, our work is foundational research, and we do not find obvious risks related to malicious harmful effects, environmental impact, fairness considerations, or privacy considerations.

## References

* Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Bapitste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance.
* Arkoudas [2023] Konstantine Arkoudas. 2023. GPT-4 can't reason. _CoRR_, abs/2308.03762.
* Bai et al. [2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. _CoRR_, abs/2309.16609.
* Bai et al. [2023] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kannile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringert, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Connerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional AI: harmlessness from AI feedback. _CoRR_, abs/2212.08073.
* Bennett [2012] Bo Bennett. 2012. _Logically fallacious: the ultimate collection of over 300 logical fallacies (Academic Edition)_. eBookIt. com.
* Bowman et al. [2012] Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamille Lukosite, Amanda Askell, Andy Jones, Anna Chen, 

Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zae Hatfield-Dodds, Ben Mann, and Jared Kaplan. 2022. Measuring progress on scalable oversight for large language models. _CoRR_, abs/2211.03540.
* Chen et al. (2023a) Angelica Chen, Jeremy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. 2023a. Improving code generation by training with natural language feedback. _CoRR_, abs/2303.16749.
* Chen et al. (2023b) Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2023b. Iterative translation refinement with large language models. _CoRR_, abs/2306.03856.
* A tool augmented framework for multi-task and multi-domain scenarios. _CoRR_, abs/2307.13528.
* Chu et al. (2023) Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weilha Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. A survey of chain of thought reasoning: Advances, frontiers and future. _arXiv preprint arXiv:2309.15402_.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Akanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, HongKun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. _CoRR_, abs/2210.11416.
* Clark et al. (2020) Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_, pages 3882-3890. ijcai.org.
* Creswell et al. (2023) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023. Selection-inference: Exploiting large language models for interpretable logical reasoning. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. _Recognizing Textual Entailment: Models and Applications_. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
* Dalvi et al. (2021) Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entailment trees. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 7358-7370. Association for Computational Linguistics.
* Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: general language model pretraining with autoregressive blank infilling. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 320-335. Association for Computational Linguistics.
* Fernandes et al. (2023) Patrick Fernandes, Aman Madau, Emmy Liu, Antonio Farinhas, Pedro Henrique Martins, Amanda Bertsch, Jose G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and Andre F. T. Martins. 2023. Bridging the gap: A survey on integrating (human) feedback for natural language generation. _CoRR_, abs/2305.00955.
* Fieser and Dowden (2011) James Fieser and Bradley Dowden. 2011. Internet encyclopedia of philosophy: Fallacies.
* Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeynu Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. CRITIC: large language models can self-correct with tool-interactive critiquing. _CoRR_, abs/2305.11738.
* Haberal et al. (2018) Ivan Haberal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. Before name-calling: Dynamics and triggers of ad hominem fallacies in web argumentation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 386-396, New Orleans, Louisiana. Association for Computational Linguistics.
* He et al. (2023) Hangfeng He, Hongming Zhang, and Dan Roth. 2023. Rethinking with retrieval: Faithful large language model inference. _CoRR_, abs/2301.00303.
* Hong et al. (2022) Ruixin Hong, Hongming Zhang, Xinntong Yu, and Changshui Zhang. 2022. METGEN: A module-based entailment tree generation framework for answer explanation. In _Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022_, pages 1887-1905. Association for Computational Linguistics.
* Hong et al. (2023) Ruixin Hong, Hongming Zhang, Hong Zhao, Dong Yu, and Changshui Zhang. 2023. Faithful question answering with monte-carlo planning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 

3944-3965. Association for Computational Linguistics.
* Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. _CoRR_, abs/2210.11610.
* Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaxiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. _arXiv preprint arXiv:2310.01798_.
* Jevons (1872) William Stanley Jevons. 1872. _Elementary lessons in logic: Deductive and inductive: With copious questions and examples, and a vocabulary of logical terms_. Macmillan.
* Jiao et al. (2022) Fangkai Jiao, Yangyang Guo, Xuemeng Song, and Liqiang Nie. 2022. Merit: Meta-path guided contrastive learning for logical reasoning. In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 3496-3509. Association for Computational Linguistics.
* Jin et al. (2022) Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 7180-7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Jung et al. (2022) Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maicuito prompting: Logically consistent reasoning with recursive explanations. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 1266-1279. Association for Computational Linguistics.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In _NeurIPS_.
* Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kermion, Kamile Lukostie, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. 2023. Measuring faithfulness in chain-of-thought reasoning. _CoRR_, abs/2307.13702.
* Lau (2011) Joe YF Lau. 2011. _An introduction to critical thinking and creativity: Think more, think better_. John Wiley & Sons.
* Leike and Sutskever (2023) Jan Leike and Ilya Sutskever. 2023. Introducing superalignment. _OpenAI_.
* Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. _CoRR_, abs/2305.20050.
* Ling et al. (2023) Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. 2023. Deductive verification of chain-of-thought reasoning. _CoRR_, abs/2306.03872.
* Liu et al. (2023a) Hannepa Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023a. Evaluating the logical reasoning ability of chatgpt and GPT-4. _CoRR_, abs/2304.03439.
* Liu et al. (2023b) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. _ACM Comput. Surv._, 55(9):195:1-195:35.
* Madau et al. (2023) Aman Madau, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shirmai Prabhumoye, Yiming Yang, Sean Welleock, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. _CoRR_, abs/2303.17651.
* Magnus (2005) PD Magnus. 2005. forall x: An introduction to formal logic.
* McCarthy (1989) John McCarthy. 1989. Artificial intelligence, logic and formalizing common sense. _Philosophical Logic and Artificial Intelligence_, pages 161-190.
* Miao et al. (2023) Ning Miao, Yee Whye Teh, and Tom Rainforth. 2023. Selfcheck: Using lllms to zero-shot check their own step-by-step reasoning. _CoRR_, abs/2308.00436.
* Ni et al. (2023) Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-Tau Yih, Sida I. Wang, and Xi Victoria Lin. 2023. LEVER: learning to verify language-to-code generation with execution. In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_, volume 202 of _Proceedings of Machine Learning Research_, pages 26106-26128. PMLR.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _CoRR_, abs/2303.08774.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions within human feedback. In _NeurIPS_ 

* Pan et al. (2023) Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. _CoRR_, abs/2305.12295.
* Payandeh et al. (2023) Amirreza Payandeh, Dan Pluth, Jordan Hosier, Xuesu Xiao, and Vijay K. Gurbani. 2023. How susceptible are llms to logical fallacies? _CoRR_, abs/2308.09853.
* Peng et al. (2023) Andrew Peng, Michael Wu, John Allard, Logan Kilpatrick, and Steven Heidel. 2023. Gpt-3.5 turbo fine-tuning and api updates. [https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates](https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates).
* Pi et al. (2022) Xinyu Pi, Wanjun Zhong, Yan Gao, Nan Duan, and Jian-Guang Lou. 2022. Logigan: Learning logical reasoning via adversarial pre-training. In _NeurIPS_.
* Rescher and Schagrin (2023) Nicholas Rescher and Morton L. Schagrin. 2023. Fallacy. Encyclopedia Britannica.
* Scheurer et al. (2023) Jeremy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training language models with language feedback at scale. _CoRR_, abs/2303.16755.
* Shinn et al. (2023) Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning.
* Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_, pages 4444-4451. AAAI Press.
* Stab and Gurevych (2017) Christian Stab and Iryna Gurevych. 2017. Recognizing insufficiently supported arguments in argumentative essays. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 980-990, Valencia, Spain. Association for Computational Linguistics.
* Tafjord et al. (2021) Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. Proofwriter: Generating implications, proofs, and abductive statements over natural language. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_, volume ACL/IJCNLP 2021 of _Findings of ACL_, pages 3621-3634. Association for Computational Linguistics.
* Team (2023) InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InterlnLM](https://github.com/InternLM/InterlnLM).
* Touvron et al. (2021) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhahiri, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian CantonFerrer, Moya Chen, Guillem Cucurull, David Esiboglu, Jude Fernandes, Jeremy Fu, Wenyiu Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inanch Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Kouva, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Axavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rasli Rungta, Kaylan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaqqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciolom. 2023. Llama 2: Open foundation and finetuned chat models. _CoRR_, abs/2307.09288.
* Valmeekam et al. (2023) Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023. Can large language models really improve by self-critiquuing their own plans? _CoRR_, abs/2310.08118.
* Wang et al. (2023) Yizhong Wang, Yeganek Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 13484-13508. Association for Computational Linguistics.
* Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. _Trans. Mach. Learn. Res._, 2022.
* Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In _NeurIPS_.
* Weng et al. (2022) Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. 2022. Large language models are reasoners with self-verification. _CoRR_, abs/2212.09561.
* Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023. Decomposition enhances reasoning via self-evaluation guided decoding. _CoRR_, abs/2305.00633.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Davin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. _CoRR_, abs/2304.12244.
* Yang et al. (2019) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, 

Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Hoze Sun, Hongda Zhang, Hui Liu, Jiamie Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yunpeng Zhang, Zenan Zhou, and Zhiying Wu. 2023a. Baichuan 2: Open large-scale language models. _CoRR_, abs/2309.10305.
* Yang et al. (2022) Kaiyu Yang, Jia Deng, and Danqi Chen. 2022. Generating natural language proofs with verifier-guided search. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 89-105. Association for Computational Linguistics.
* Yang et al. (2023b) Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. 2023b. Logical reasoning over natural language as knowledge representation: A survey. _CoRR_, abs/2303.12023.
* Yao et al. (2023) Shunyau Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. _CoRR_, abs/2305.10601.
* Yu et al. (2023) Fei Yu, Hongbo Zhang, and Benyou Wang. 2023. Nature language reasoning, A survey. _CoRR_, abs/2303.14725.
* Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In _NeurIPS_.
* Zheng et al. (2023)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-juqge with mt-bench and chatbot arena. _CoRR_, abs/2306.05685.

 

Details of Data Collection

To generate candidate fallacious steps, we prompt GPT-4 with the following content. Figure 3 shows an example of the fallacy definition. Note that the steps generated by GPT-4 are only used as candidates. We then have human experts proofread and revise them to ensure the quality of the data. We invited 10 well-trained graduate students from universities as human expert annotators. They have passed the graduate school entrance exam and have well logical reasoning skills. They have undergone rigorous training in areas including Mathematical Logic, Computer Science, Programming, and Statistics.

```
Frompt for Generating Candidate Fallacies Steps You are a faulty reasoner. I will describe a logical fallacy to you, and then you generate a reasoning step that belongs to this logical fallacy. Here is the description: {fallacy definition} Now, generate a reasoning step that contains this type of logical fallacy. The generated step should be related to this element: [entity] Use square brackets [I] to include propositions. The format of the output is "Since [XXX] and [XXXI], therefore, [XXX]
```

In the end of the appendix, Table 11 shows the detailed categorizations and descriptions of the 232 fallacies in our dataset.

## Appendix B Details of Experiments

### Models

Table 7 shows the version and source URL of the LLMs used in our experiments. For all series of LLMs, we select their versions of instruction fine-tuned or chat fine-tuned, since these models are closest to realistic applications. We follow the licences of these models to use them.

### Prompt Selection

To select suitable prompts and to explore the impact of prompts on model performance, we conduct an ablation study on the prompts. Table 8 demonstrates the prompts we used. Prompt 1 is the simplest prompt. The model is expected to answer "Yes" for the correct reasoning steps and "No" for the incorrect reasoning steps. Prompt 2, alternatively, replaces the response "Yes/No" with "True/False." Prompt 3 adopts a Chain-of-Thought-style prompt (Kojima et al., 2022) that allows the model to generate some relevant rationales before giving the prediction. Prompt 4 describes the task in more detail. Moreover, in contrast to Prompt 1, Prompt 4 requires the model to answer "Yes" for incorrect reasoning steps (which contain logical fallacies) and "No" to correct reasoning steps (which do not contain logical fallacies).

Table 9 demonstrates the results. We can find that Prompt 1 works well for all models. For Prompt 2, some of the models show significant performance degradation after replacing the response words. Prompt 3 introduces chains of thought, but does not achieve significant performance gains on all models. For Prompt 4, although it describes the task in more detail, it does not seem to help improve the LLMs' performance. Among all the models, GPT-4 exhibits the strongest robustness to prompts, achieving similar performance with different prompts. Comprehensively, we finally chose Prompt 1 as our prompt.

### Few-shot Setting

We also conduct experiments under few-shot setting. We include four demonstrations in the prompt, covering the correct and fallacious steps related to formal and informal fallacies.

```
Input for identifying falseboxes Stage Under Rew-shot Settings Is the following reasoning step correct? You can only answer "Yes" or "No." Since [If it's raining then the streets are wet] and [It's raining now], therefore, [The streets are wet]. Yes. Since [I found a shell on the beach] and [this shell was beautifully shaped and colored], therefore, [all shells are beautifully shaped and colored]. No. Since [I am at home or I am in the city] and [I am at home], therefore, [I am not in the city]. No. Since [heavy snowfall often leads to traffic jams] and [traffic jams cause delays], therefore, [heavy snowfall can lead to delays]. Yes. [reasoning step]
```

Table 10 demonstrates the performance of identifying fallacious steps under few-shot setting. We can find that most of the LLMs achieve comparable performance under few-shot setting compared to that under zero-shot setting. Moreover, our findings in Sec 4.3 stand under the few-shot setting as well



The **description** of the fallacy of type "Affirming the Consequent" is "An error in formal logic where if the consequent is said to be true, the antecedent is said to be true, as a result."

The **abstract logical form** is

_If P then Q._

_Q._

_Therefore, P._

The following are **some examples and their explanations** of the fallacy of type "Affirming the Consequent."

Example #1:

_If taxes are lowered, I will have more money to spend._

_I have more money to spend._

_Therefore, taxes must have been lowered._

Explanation: I could have had more money to spend simply because I gave up crack-cocaine, prostitute solicitation, and baby-seal-clubbing expeditions.

Example #2:

_If it's brown, flush it down._

_I flushed it down._

_Therefore, it was brown._

Explanation: No! I did not have to follow the, "if it's yellow, let it mellow" rule -- in fact, if I did follow that rule I would probably still be single. The stated rule is simply, "if it's brown" (the antecedent), then (implied), "flush it down" (the consequent). From this, we cannot imply that we can ONLY flush it down if it is brown.

Figure 3: The definition of the fallacy of “Affirming the Consequent”, one of 232 types of fallacies in our dataset.

\begin{table}
\begin{tabular}{|l l l|} \hline \hline
**Model** & **Version** & **URL** \\ \hline Flan-T5-Large Chung et al. (2022) & google/flan-t5-large & [https://huggingface.co/google/flan-t5-large](https://huggingface.co/google/flan-t5-large) \\ Flan-T5-xl Chung et al. (2022) & google/flan-t5-xl & [https://huggingface.co/google/flan-t5-xl](https://huggingface.co/google/flan-t5-xl) \\ Flan-T5-xxl Chung et al. (2022) & google/flan-t5-axl & [https://huggingface.co/google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl) \\ \hline Llama2-7B Touvron et al. (2023) & meta-llama/Llama-2-7b-chat-hf & [https://huggingface.com/meta-llama/Llama-2-7b-chat-hf](https://huggingface.com/meta-llama/Llama-2-7b-chat-hf) \\ Llama2-13B Touvron et al. (2023) & meta-llama/Llama-2-13b-chat-hf & [https://huggingface.com/meta-llama/Llama-2-13b-chat-hf](https://huggingface.com/meta-llama/Llama-2-13b-chat-hf) \\ Llama2-70B Touvron et al. (2023) & meta-llama/Llama-2-70b-chat-hf & [https://huggingface.co/meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) \\ \hline Baichuan2-7B Yang et al. (2023a) & baichuan-inc/Baichuan2-7B-Chat & [https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat](https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat) \\ Baichuan2-13B Yang et al. (2023a) & baichuan-inc/Baichuan2-13B-Chat & [https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat](https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat) \\ \hline ChatGLM-6B Du et al. (2022) & THUDM/chtaglm-6b & [https://huggingface.co/THUDM/chtaglm-6b](https://huggingface.co/THUDM/chtaglm-6b) \\ ChatGLM2-6B Du et al. (2022) & THUDM/chtaglm2-6b & [https://huggingface.co/THUDM/chtaglm2-6b](https://huggingface.co/THUDM/chtaglm2-6b) \\ \hline InternLM-7B Team (2023) & internlm/internalm-chat-7b-v1\_1 & [https://huggingface.co/intermlm/internalm-chat-7b-v1](https://huggingface.co/intermlm/internalm-chat-7b-v1)\_1 \\ InternLM-20B Team (2023) & internlm/internlm-chat-20b & [https://huggingface.co/internlm/internalm-chat-20b](https://huggingface.co/internlm/internalm-chat-20b) \\ \hline Falcon-7B Almazrouei et al. (2023) & tiuize/falcon-7b-instruct & [https://huggingface.co/tiuiuea/falcon-7b](https://huggingface.co/tiuiuea/falcon-7b) \\ \hline WizardML-13B Xu et al. (2023) & WizardLM/WizardLM-13B-V1.2 & [https://huggingface.co/WizardLM/WizardLM-13B-V1.2](https://huggingface.co/WizardLM/WizardLM-13B-V1.2) \\ \hline Vicuna-7B Zheng et al. (2023) & Imsys/vicuna-7b-v1.5 & [https://huggingface.co/Imsystolicuna-7b-v1.5](https://huggingface.co/Imsystolicuna-7b-v1.5) \\ Vicuna-13B Zheng et al. (2023) & Imsys/vicuna-13b-v1.5 & [https://huggingface.co/Imsystolicuna-13b-v1.5](https://huggingface.co/Imsystolicuna-13b-v1.5) \\ \hline Owen-14B Bai et al. (2023) & Quen/Quwen-14B-Chat & [https://huggingface.co/Queen/Quwen-14B-Chat](https://huggingface.co/Queen/Quwen-14B-Chat) \\ \hline GPT-3.5 Peng et al. (2023) & gpt-3.5-turbo & [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5) \\ GPT-4 OpenAI (2023) & gpt-4 & [https://platform.openai.com/docs/models/gpt-4](https://platform.openai.com/docs/models/gpt-4) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Detailed information about the models we experiment with.

 



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c|}{**Formal**} & \multicolumn{6}{c|}{**Informal**} \\
**Model** & **prop.** & **quant.** & **syl.** & **prob.** & **Avg.** & **amb.** & **incon.** & **irrel.** & **insuf.** & **inappr.** & **Avg.** & **Avg.** \\ \hline Random & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 & 50.0 \\ \hline Flan-T5-Large & 58.3 & 64.2 & 71.9 & 62.5 & **64.2** & 62.3 & 85.0 & 73.8 & 73.3 & 69.7 & 72.8 & 68.5 \\ Flan-T5-xl & 47.5 & 74.2 & 81.9 & 68.8 & 68.1 & 65.3 & 85.0 & 70.0 & 70.3 & 70.6 & 72.3 & 70.2 \\ Flan-T5-xcl & 44.2 & 70.8 & 79.4 & 56.2 & 62.7 & 64.0 & 76.7 & 73.5 & 74.6 & 71.4 & 72.0 & 67.3 \\ Llama2-7B & 65.0 & 66.7 & 56.2 & 63.8 & 62.9 & 80.3 & 75.0 & 81.5 & 87.0 & 81.8 & 81.1 & 72.0 \\ Llamaa2-13B & 65.8 & 65.8 & 60.0 & 62.5 & 63.5 & 69.0 & 73.3 & 78.6 & 80.3 & 80.6 & 76.4 & 70.0 \\ Llama2-70B & 64.2 & 77.5 & 78.8 & 72.5 & 73.2 & 83.3 & 81.7 & 86.8 & 90.9 & 88.1 & 86.2 & 79.7 \\ Baichuan2-7B & 55.0 & 70.0 & 62.5 & 62.5 & 62.5 & 73.3 & 76.7 & 72.8 & 77.4 & 73.9 & 74.8 & 68.7 \\ Baichuan2-13B & 60.8 & 72.5 & 68.8 & 62.5 & 66.1 & 76.3 & 75.0 & 78.7 & 82.3 & 79.1 & 78.3 & 72.2 \\ ChatGLM-6B & 71.7 & 72.5 & 68.1 & 61.3 & 68.4 & 70.7 & 81.7 & 73.7 & 78.5 & 76.9 & 76.3 & 72.3 \\ ChatGLM-2eB & 64.2 & 65.8 & 78.8 & 57.5 & 66.6 & 74.0 & 83.3 & 79.7 & 81.6 & 76.2 & 79.0 & 72.8 \\ InterLM-7B & 60.0 & 68.3 & 76.9 & 47.5 & 63.2 & 69.3 & 80.0 & 74.1 & 75.6 & 71.8 & 74.2 & 68.7 \\ InternLM-20B & 58.3 & 69.2 & 78.8 & 60.0 & 66.6 & 69.3 & 80.0 & 74.9 & 77.6 & 75.8 & 75.5 & 71.1 \\ Falcon-7B & 56.7 & 58.3 & 55.0 & 53.8 & 55.9 & 70.0 & 71.7 & 74.0 & 73.0 & 72.5 & 72.2 & 64.1 \\ WizardLM-13B & 76.7 & 72.5 & 77.5 & 71.2 & 74.5 & 82.3 & 80.0 & 85.8 & 90.8 & 84.4 & 84.7 & 79.6 \\ Vicuna-7B & 59.2 & 72.5 & 73.1 & 60.0 & 66.2 & 76.7 & 81.7 & 84.2 & 87.2 & 83.5 & 82.6 & 74.4 \\ Vicuna-13B & 67.5 & 75.0 & 83.1 & 62.5 & 72.0 & 82.7 & 85.0 & 85.8 & 90.8 & 85.5 & 85.9 & 79.0 \\ Qwen-14B & 68.3 & 75.0 & 83.1 & 67.5 & 73.5 & 81.7 & 88.3 & 87.8 & 91.2 & 85.2 & 86.8 & 80.2 \\ GPT-3.5 & 72.5 & 81.7 & 78.1 & 78.8 & 77.8 & 86.0 & 86.7 & 87.9 & 89.4 & 85.6 & 87.1 & 82.4 \\ GPT-4 & 91.7 & 84.2 & 87.5 & 88.8 & 88.0 & 84.7 & 85.0 & 89.2 & 93.0 & 87.8 & 87.9 & 88.0 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Accuracy results (%) of identifying fallacious steps under few-shot setting.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt}}
**No.** & **Content** \\
**Prompt 1** & Is the following reasoning step correct? !n You can only answer “Yes” or “No"..n {reasoning step} \\
**Prompt 2** & Is the following reasoning step correct? !n You can only answer “True” or “False”.n {reasoning step} \\
**Prompt 3** & Is the following reasoning step correct? !n Let’s think step by step and then answer “Yes” or “No“.n {reasoning step} \\
**Prompt 4** & You are a trained model capable of detecting reasoning errors and logical fallacies. !n As a detector, your task is to analyze the given reasoning steps and determine whether they involve any logical fallacies. In If a logical fallacy is present, your response should be ”Yes”. un If no logical fallacies are detected, your response should be “No”.n You can only answer “Yes” or “No”. n {reasoning step} \\ \hline \hline \end{tabular}
\end{table}
Table 8: The different prompts used to prompt large language models to identify fallacious steps.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c|}{**Prempt 1**} & \multicolumn{4}{c|}{**Prompt 2**} & \multicolumn{4}{c|}{**Prompt 3**} & \multicolumn{4}{c}{**Prompt 4**} \\
**Model** & **Formal** & **Informal** & **Avg.** & **Formal** & **Informal** & **Avg.** & **Formal** & **Informal** & **Avg.** & **Normal** \\ \hline Flan-T5-Large & 61.0 & 73.6 & 67.3 & 59.5 & 71.7 & 65.6 & 60.9 & 73.9 & 67.4 & 40.1 & 41.3 & 40.7 \\ Flan-T5-xl & 67.9 & 74.1 & 71.0 & 68.2 & 68.5 & 68.4 & 67.3 & 74.7 & 71.0 & 41.0 & 47.6 & 44.3 \\ Flan-T5-xl & 63.9 & 73.0 & 68.4 & 65.1 & 74.2 & 69.6 & 63.4 & 73.6 & 68.5 & 43.1 & 59.1 & 51.1 \\ Llama2-7B & 61.3 & 71.8 & 66.5 & 50.7 & 53.3 & 52.0 & 49.8 & 52.2 & 51.0 & 50.0 & 49.9 & 50.0 \\ Llama2-13B & 59.9 & 63.2 & 61.5 & 57.5 & 67.3 & 62.4 & 49.8 & 51.0 & 50.4 & 50.2 & 51.1 & 50.7 \\ Llama2-70B & 73.3 & 88.2 & 80.7 & 72.3 & 87.8 & 80.1 & 63.8 & 67.7 & 65.7 & 54.7 & 59.1 & 56.9 \\ Baichuan2-7B & 52.7 & 52.3 & 52.5 & 60.6 & 75.0 & 67.8 & 54.4 & 71.1 & 62.7 & 45.3 & 46.5 & 45.9 \\ Baichuan2-13B & 68.0 & 80.9 & 74.4 & 61.7 & 82.2 & 72.0 & 65.9 & 75.9 & 70.9 & 50.9 & 49.9 & 50.4 \\ ChatGLM-6B & 55.8 & 54.4 & 55.1 & 42.9 & 48.1 & 45.5 & 53.3 & 61.3 & 57.3 & 48.2 & 46.3 & 47.3 \\ ChatGLM-6B & 61.7 & 75.2 & 68.4 & 59.1 & 76.5 & 67.8 & 61.8 & 71.3 & 66.5

* [108] M. C. Bender, A. K. Sridharan, and R. J. Smith, "A new approach to the development of a new method for solving the problem of finding an optimal solution in a multi-objective optimization problem," _Journal of Computational Physics_**263**(4):579-602, 2000.
* [109] D. E. Kingma and P. Ba, "Adam: A stochastic gradient descent algorithm with applications to machine learning," _arXiv preprint arXiv:1412.6090_**, 2014.
* [110] G. L. Hinton, "Deep residual networks: A survey on deep neural networks," _Advances in Neural Information Processing Systems_**26**(1):1055-1068, 2011.
* [111] I. Goodfellow, Y. Bengio, and J. Pouget, "Generators for large scale data recognition," _Neural computation_**4**(4):433-448, 2015.
* [112] V. Khoslavi, N. Mirza, and T. Darrell, "Learning from tiny images using deep convolutional nets," _IEEE Transactions on Pattern Analysis and Machine Intelligence_**5**(1):1038-1048, 2016.

[MISSING_PAGE_POST]

 

* [105] M. Cacciari, R. G. Caldwell, and A. De Paola, "The automated computation of the event generator for a single-event simulation," in _Proceedings of the 23rd International Workshop on Advanced Computing Systems (ACAT)_, p. 467, 2009.
* [106] J. Alwall et al., "Event generation with the CMS detector at LHC," _JINST_**8** (2011) P06005, doi:10.1088/1748-0221/8/06/P06005, arXiv:1106.2502.
* [107] T. Sjostrand, S. Mrenna, and P. Z. Skands, "PYTHIA 6.4 physics and manual", _Comput. Phys. Commun._**191** (2006) 159, doi:10.1016/j.cpc.2006.01.024, arXiv:hep-ph/0603175.
* [108] E. Barberio, F. Piccinini, and L. Pitelli, "A new method to simulate events using PYTHIA 6.4 physics and manual", CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2006-003, CERN Report CERN-LHCC-2 

* [108] M. C. Bader, A. D. Kessler, and R. J. Sorensen. The role of the agent's ability to generate a reward for an agent in a reinforcement learning environment. In _Proceedings of the 27th International Conference on Artificial Intelligence_, pages 35-46, 2019.
* [109] P. F. Goyal, L. E. Oliva, and T. Vinyals. Reinforcement learning with reinforcement learning: A review. _Machine Learning Research_, 20(1):1-10, 2019.
* [110] H. G. Hinton, I. Goodfellow, and Y. Bengio. Generative adversarial nets. In _Advances in neural information processing systems_, volume 1, MIT press, Cambridge, MA, USA, 2016.
* [111] W. Tehgaler, N. Srivastava, and Z. Chen. Deep residual learning for image recognition. In _International conference on computer vision and pattern recognition_, pages 1-10, 2019.

[MISSING_PAGE_POST]

 

* [10] M. C. Bader, A. D. Kessler, and R. J. Sorensen. The role of the agent's ability to predict a behavior in the environment. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 35-46, 2007.
* [11] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 89-90, 2007.
* [12] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [13] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine learning research (ICML)_, pages 103-113, 2007.
* [14] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [15] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [16] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [17] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [18] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [19] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [20] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [21] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [22] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [23] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [24] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [25] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [26] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [27] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [28] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [29] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [30] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [31] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [32] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [33] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [34] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [35] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [36] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [37] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [38] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [39] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [40] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [41] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [42] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [43] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [44] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.
* [45] P. F. Goyal, L. E. Oliva, and T. V. LeCun. An empirical study of the agent's ability to predict the behaviour of the agents' behavior. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 103-113, 2007.

 

* [25] M. C. Bader, A. D. Krizhevsky, and R. G. Hinton. The output of a deep neural network for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 1-40, 2006.

[MISSING_PAGE_POST]

 

* [25] M. C. Bader, A. D. Kessler, and R. J. Schreiber, _A new approach to the design of a single-layer model for the design of a single layer_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [26] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _The design of a single layer model for the design of a single layer_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 109-112, 2017.
* [27] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [28] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [29] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [30] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [31] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [32] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [33] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [34] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [35] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [36] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [37] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [38] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [39] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [40] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [41] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [42] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [43] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [44] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [45] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [46] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [47] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [48] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [49] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [50] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [51] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [52] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [53] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [54] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [55] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [56] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [57] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [58] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [59] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [60] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.
* [61] S. F. Gershman, P. H. Schultz, and T. W. L. Smith, _Designing the design of a single layer model for the design of a single layer model_, in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1036-1048, 2017.

 

* [25] M. C. Bader, A. D. Kessler, and R. J. Sorensen. The role of the social network in a social world: an overview. In _Proceedings of the 10th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 349-368. ACM, 2007.
* [26] G. F. Hinton, P. H. Friedman, and I. Goodfellow. Learning to be effective for image recognition. _arXiv preprint arXiv:1407.1199_, 2014.
* [27] L. Bottou, Y. Bengio, and T. Salakhutdinov. Generative adversarial nets. In _Advances in neural information processing systems_, volume 1, pages 103-113. MIT Press, Cambridge, MA, USA, 2014.

[MISSING_PAGE_POST]

 



\begin{tabular}{|p{142.3pt}|p{85.4pt}|} \hline Argument from Silence & Drawing a conclusion based on the silence of the opponent, when the opponent is refusing to give evidence for any reason. \\ \hline Argument from Hearsay & Presenting the testimony of a source that is not an eyevitness to the event in question. \\ \hline Anonymous Authority & When an unspecified source is used as evidence for the claim. \\ \hline Insignificant Cause & An explanation that posits one minor factor, out of several that contributed, as its sole cause. \\ \hline Just Because Fallacy & Refusing to respond to give reasons or evidence for a claim by stating yourself as the ultimate authority on the matter. \\ \hline Appeal to the Law & When following the law is assumed to be the morally correct thing to do, without justification, or when breaking the law is assumed to be the morally wrong thing to do, without justification. \\ \hline Appeal to Normality & Using social norms to determine what is good or bad. \\ \hline False Effect & Claiming that the cause is true or false based on what we know about the effect in a claim of causality that has not been properly established. \\ \hline Appeal to Consequences & Concluding that an idea or proposition is true or false because the consequences of it being true or false are desirable or undesirable. \\ \hline Retroressive Causation & Invoking the cause to eliminate the effect, or calling on the source to relieve the effect of the source. \\ \hline Confusing Currently Unexplained with Unexplainable & Making the assumption that what cannot currently be explained is, therefore, unexplainable (impossible to explain). \\ \hline Appeal to Desperation & Arguing that your conclusion, solution, or proposition is right based on the fact that something must be done, and your solution is "something." \\ \hline Regression Fallacy & Ascribing a cause where none exists in situations where natural fluctuations exist while failing to account for these natural fluctuations. \\ \cline{2-3}   Causal Reductionism & Assuming a single cause or reason when there were actually multiple causes or reasons. \\ \cline{2-3}   Questionable Cause & Concluding that one thing caused another, simply because they are regularly associated. \\ \hline Informal-Inappropriate Presumption & Hedging & Refining your claim simply to avoid counter evidence and then acting as if your revised claim is the same as the original. \\ \cline{2-3}   Circular Definition & A circular definition is defining a term by using the term in the definition. \\ \cline{2-3}   Homunculus Fallacy & An argument that accounts for a phenomenon in terms of the very phenomenon that it is supposed to explain, which results in an infinite regress. \\ \cline{2-3}   Circular Reasoning & A type of reasoning in which the proposition is supported by the premises, which is supported by the proposition. \\ \hline Tokenism & Interpreting a token gesture as an adequate substitute for the real thing. \\ \cline{2-3}   Appeal to Novelty & Claiming that something that is new or modern is superior to the status quo, based exclusively on its newness. \\ \hline Two Wrongs Make a Right & When a person attempts to justify an action against another person because the other person did take or would take the same action against him or her. \\ \hline Appeal to Nature & When used as a fallacy, the belief or suggestion that "natural" is better than "unnatural" based on its naturalness. \\ \hline Naturalistic Fallacy & When the conclusion expresses what ought to be, based only on what is, or what ought not to be, based on what is not. \\ \hline Moralistic Fallacy & When the conclusion expresses what is, based only on what one believes ought to be, or what isn't is based on what one believes ought not to be. \\ \hline Suppressed Correlative & The attempt to redefine a correlative (one of two mutually exclusive options) so that one alternative encompasses the other, i.e. making one alternative impossible. \\ \hline \end{tabular}



* [10] M. C. Bender, A. K. R. Wilson, and J. S. W. Tukey, "The statistical significance of the mean-variance test for a class of random variables," in _Proceedings of the International Conference on Machine Learning Research_, pages 235-246, 2007.
* [11] D. E. Brown, "A survey of machine learning algorithms for data mining," _IEEE Transactions on Knowledge Discovery and Data Engineering_, vol. 89, no. 1, pp. 103-113, Jan 2000.
* [12] G. F. Chan, "An introduction to machine learning techniques," _Machine Learning_, vol. 1, no. 1, pp. 1-12, 1996.
* [13] H. Chen, Y. Li, and L. Zhang, "Learning the number of machines with a large number of machines," _Journal of Machine Learning Research_, vol. 1, no. 1, pp. 1-10, 2008.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{|p{142.3pt}|p{85.4pt}|} Package Deal Fallacy & Assuming things that are often grouped together must always be grouped together, or the assumption that the ungrouping will have significantly more severe effects than anticipated. \\ \hline Subjectivist Fallacy & Claiming something is true for one person, but not for someone else when, in fact, it is true for everyone (objective) as demonstrated by empirical evidence. \\ \hline Distinction Without a Difference & The assertion that a position is different from another position based on the language when, in fact, both positions are the same – at least in practice or practical terms. \\ \hline Hypothesis Contrary to Fact & Offering a poorly supported claim about what might have happened in the past or future, if (the hypothetical part) circumstances or conditions were different. \\ \hline Shoehorning & The process of force-fitting some current affair into one’s personal, political, or religious agenda. \\ \hline Appeal to Self-evident Truth & Making the claim that something is “self-evident” when it is not self-evident in place of arguing a claim with reason. \\ \hline Subverted Support & The attempt to explain some phenomenon that does not actually occur or there is no evidence that it does. \\ \hline Double Standard & Judging two situations by different standards when, in fact, you should be using the same standard. \\ \hline Fantasy Projection & Confusing subjective experiences, usually very emotionally charged, with objective reality, then suggesting or demanding that others accept the subjective experience as objective reality. \\ \hline Argument to Moderation & Asserting that given any two positions, there exists a compromise between them that must be correct. \\ \hline Broken Window Fallacy & The illusion that destruction and money spent in recovery from destruction, is a net-benefit to society. \\ \hline Self Sealing Argument & An argument or position is self-sealing if and only if no evidence can be brought against it no matter what. \\ \hline Unfalsifiability & Confidently asserting that a theory or hypothesis is true or false even though the theory or hypothesis cannot possibly be contradicted by an observation or the outcome of any physical experiment, usually without strong evidence or good reasons. \\ \hline Conspiracy Theory & Explaining that your claim cannot be proven or verified because the truth is being hidden and/or evidence destroyed by a group of two or more people. \\ \hline Confusing an Explanation with an Excuse & Treating an explanation of a fact as if it were a justification of the fact, a valid reason for the fact, or evidence for the fact. \\ \hline Limited Depth & Failing to appeal to an underlying cause, and instead simply appealing to membership in a category. \\ \hline Alleged Certainty & Asserting a conclusion without evidence or premises, through a statement that makes the conclusion appear certain when, in fact, it is not. \\ \hline \end{tabular}
\end{table}
Table 11: Categorizations and descriptions of the fallacies in our dataset.