CLOMO: Counterfactual Logical Modification with Large Language Models (2024)

Abstract: In this study, we delve into the realm of counterfactual reasoning capabilities of large language models (LLMs). Our primary objective is to cultivate the counterfactual thought processes within LLMs and rigorously assess these processes for their validity. Specifically, we introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. To effectively evaluate a generation model’s counterfactual capabilities, we propose an innovative evaluation metric, the decomposed Self-Evaluation Score (SES) to directly evaluate the natural language output of LLMs instead of modeling the task as a multiplechoice problem. Analysis shows that the proposed automatic metric aligns well with human preference. Our experimental results show that while LLMs demonstrate a notable capacity for logical counterfactual thinking, there remains a discernible gap between their current abilities and human performance. Code and data are available at https://github. com/Eleanor-H/CLOMO.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11012–11034 August 11-16, 2024 ©2024 Association for Computational Linguistics
In this study, we delve into the realm of counterfactual reasoning capabilities of large language models (LLMs). Our primary objective is to cultivate the counterfactual thought processes within LLMs and rigorously assess these processes for their validity. Specifically, we introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. To effectively evaluate a generation model’s counterfactual capabilities, we propose an innovative evaluation metric, the decomposed Self-Evaluation Score (SES) to directly evaluate the natural language output of LLMs instead of modeling the task as a multiplechoice problem. Analysis shows that the proposed automatic metric aligns well with human preference. Our experimental results show that while LLMs demonstrate a notable capacity for logical counterfactual thinking, there remains a discernible gap between their current abilities and human performance. Code and data are available at https://github. com/Eleanor-H/CLOMO.

heading: 1 Introduction
text: Despite large language models (Arkoudas, 2023; OpenAI, 2022) perform strikingly in plenty of reasoning benchmarks (Cobbe et al., 2021; Hendrycks et al., 2021a), late studies observe an internal inconsistency in their reasoning processes (Saparov and He, 2023; Arkoudas, 2023). The inconsistency is attributed to misunderstanding and misapplication of logical relations. However, logical relations in complex language reasoning are not yet properly quantified and evaluated.
Current studies on evaluating model reasoning are limited in both form and content. On the one
∗ These authors contributed equally to this work. Work is done during R. Hong’s internship at Tencent AI Lab.
† Corresponding author.
hand, benchmarking complex reasoning is generally applying discrimination tasks such as multiplechoice questions (Huang et al., 2023b; Hendrycks et al., 2021a; Chen et al., 2023; Suzgun et al., 2023), where accuracy and pass rate serve as the main evaluation metric. However, such evaluations oversimplify the goal of uncovering essential and subtle pitfalls in complex reasoning. For example, the reasoning processes could contain misconceptions in logical relations but give correct answers due to the data distribution (Elazar et al., 2021; Saparov and He, 2023). Therefore, evaluating the generated content would provide a more realistic measurement of model reasoning. On the other hand, unlike widely studied reasoning tasks such as math reasoning (Cobbe et al., 2021; Hendrycks et al., 2021b) and standard exams (OpenAI, 2023; Huang et al., 2023b), counterfactual reasoning (Starr, 2022) as a fundamental evaluation of logical relations is less explored in the context of large language models. Previous literature studies counterfactual reasoning either in a multiple-choice manner (Tandon et al., 2019; Qin et al., 2021) or applying labored human study to evaluate counterfactual generation (Qin et al., 2019), leaving an effective evaluation of counterfactual generation unexplored.
In our study, we delve into the realm of evaluating large language models’ (LLMs) ability to generate counterfactually coherent thoughts. Figure 1 demonstrates the paradigm. Specifically, we proposed an innovative evaluation system that quantitatively measures the evolution of information in statement pairs, ensuring that they adhere to a specified logical relationship. Our approach includes designing a specialized task where models are presented with mismatched argument-premise pairs bound by a specific logical relation. The objective for these models is to adeptly modify the argument text until the specified logical relation is satisfactorily established. In conjunction with this task, we have created the first dataset of its kind, compris-
11012
R M Counterfactual logical modification Logical relations
• Provides a necessary assumption to
• Provides a sufficient assumption to
• Strengthen
• Weaken
Argument
Premise 1 Premise 2
R
M
R
State 1 State 2
Input:
Output:
Argument: Statement1: After the second world war, the charter of the newly
formed united nations established an eleven-member security council and charged it
with taking collective action in response to threats to world peace. Statement2: The
charter further provided that the five nations that were then the major powers would
permanently have sole authority to cast vetoes. Statement3: The reason given for
this arrangement was that the decisions reached by a majority of nations could be
biased in favor of one or more major powers, thus ensuring their support in
enforcing these decisions.
In the following, you will see an argument and 2 premises, where Premise 1
provides a sufficient assumption to the Argument. Please modify the Statements
in the Argument until Premise 2 provides a sufficient assumption to the
Argument instead, while Premise 1 fails to provides a sufficient assumption to
the Argument. Note that no additional statement should be added.
Argument: Statement1: After the second world war, the charter of the newly
formed united nations established an eleven-member security council and charged
it with taking collective action in response to threats to world peace. Statement2:
The charter further provided that the five nations that were then the major powers
would permanently have sole authority to cast vetoes. Statement3: The reason
given for this arrangement was that the burden of maintaining world peace would
rest on the world's major powers, and no nation should be required to assume the
burden of enforcing a decision it found repugnant.
Premise1: No nation that was not among the major powers at the end of the second
world war would become a major power.
Premise2: Decisions reached by a majority of nations in response to threats to
world peace would be biased in favor of one or more major powers.
Please write the modified argument below:

heading: 2 Related Works From Complex Reasoning to Counterfactual
text: R State 1 State 2
Necessary Assumption
(38.5%)
Argument: Statement1: Journalist: the advice of social scientists is frequently overlooked by politicians making social policy. Statement 2: Because it is not unreasonable to discount scientific assertions backed by weak evidence, politicians should not generally be criticized for ignoring social science, for social scientists, unlike physical scientists, seldom agree on the claims made even within their own specialty. Premise1: The failure of scientists to agree that a claim within their specialty is true can indicate that the evidence for the claim is not strong. Argument′: Statement 1: Journalist: the advice of social scientists is frequently overlooked by politicians making social policy. Statement 2: Because it is not unreasonable to discount scientific assertions, politicians should not generally be criticized for ignoring social science unless social scientists agree on the same claim, for social scientists, unlike physical scientists, seldom agree on the claims made even within their own specialty. Premise2: Politicians should follow the advice of experts on issues about which those experts agree among themselves.
Sufficient Assumption
(6.6%)
Argument: Statement 1: Caffeine can kill or inhibit the growth of the larvae of several species of insects. Statement 2: One recent experiment showed that tobacco hornworm larvae die when they ingest a preparation that consists, in part, of finely powdered tea leaves, which contain caffeine. Statement 3: This result is evidence for the hypothesis that the presence of non-negligible quantities of caffeine in various parts of many diverse species of plants is not accidental but evolved as a defense for those plants. Premise1: Caffeine-producing plants or their ancestors have sometimes been fed upon by creatures sensitive to caffeine. Argument′: Statement 1: Caffeine produced for plant species’ own defense can kill or inhibit the growth of the larvae of several species of insects. Statement 2: One recent experiment showed that tobacco hornworm larvae die when they ingest a preparation that consists, in part, of finely powdered tea leaves, which contain caffeine. Statement 3: This result is evidence for the hypothesis that the presence of non-negligible quantities of caffeine in various parts of tobacco plant is not accidental but evolved as a defense for it. Premise2: The tobacco plant is among the plant species that produce caffeine for their own defense.
Strengthen (18.7%) Argument: Statement1: In contemplating major purchases, businesses often consider only whether there is enough money left from monthly revenues after paying monthly expenses to cover the cost of the purchase. But many expenses do not occur monthly ; taking into account only monthly expenses can cause a business to overexpand. Statement2: So the use of a cash-flow statement is critical for all businesses. Premise1: A cash-flow statement is the only way to track both monthly expenses and expenses that are not monthly.
Argument′: Statement 1: In contemplating major purchases, businesses often consider only whether there is enough money left from monthly revenues after paying monthly expenses to cover the cost of the purchase. But there are many expenses every month ; taking into account these expenses incorrectly can cause a business to overexpand. Statement 2: So the use of a cash-flow statement is critical for all businesses. Premise2: Only a cash-flow statement can accurately document all monthly expenses.
Weaken (36.2%)
Argument: Statement1: The country of baurisia has, until now, been self-sufficient in both grain and meat. However, with growing prosperity in baurisia has come a steadily increasing per capita consumption of meat, and it takes several pounds of grain to produce one pound of meat. Statement2: Therefore, since per capita income in baurisia is almost certain to rise further but increases in domestic grain production are highly unlikely, baurisia is soon likely to become an importer of grain. Premise1: It is more economical for baurisians to import meat than grain. Argument′: Statement 1: The country of baurisia has, until now, been self-sufficient in both grain and meat. However, with growing prosperity in baurisia has come a steadily increasing per capita consumption of meat. Statement 2: Therefore, since per capita income in baurisia is almost certain to rise further but increases in domestic meat production are highly unlikely, baurisia is soon likely to become an importer of meat. Premise2: The per capita consumption of meat in baurisia is roughly the same across all income levels.
Table 1: Example questions from the CLOMO benchmark, with the proportion of each logical relation. Counterfac-
tual logical modifications regarding the change of state by a premise are highlighted.
et al., 2022; Yang and Deng, 2021). Moreover, real-scenario compositional reasoning (Yu et al., 2020; Liu et al., 2020; Dalvi et al., 2021; Huang et al., 2022) joins commonsense, consider the uncertainty of events in multi-step logical reasoning, which challenge current models (Bao et al., 2023; Xu et al., 2023; Jiao et al., 2023, 2022; Huang et al., 2023a, 2021a) to solve real-world reasoning problems with faithfulness. Additionally, the more faithful models should be able to consider counterfactuals. For example, answering questions given counterfactual conditions (Yu et al., 2023; Tandon et al., 2019; Qin et al., 2021), or narrating a counterfactual scenario (Qin et al., 2019). However, previous studies on counterfactual reasoning barely pay attention to the logical consistency or faithfulness of models. Therefore, in this work, we propose Counterfactual Logical Modification that challenges the model to satisfy a given logical relation restriction while generating counterfactuals. Evaluation of LLM Reasoning Currently, there is an increasing interest in the reasoning ability of LLMs. Evaluations include several perspectives, such as mathematical reasoning, commonsense reasoning, logical reasoning, and domain-specific reasoning (Chang et al., 2023; Zhong et al., 2023; Bang et al., 2023; Liu et al., 2023). However, most current reasoning evaluations focus primarily on the accuracy of the final answer and neglect a comprehensive assessment of the reasoning process. Such evaluation is not ideal for understanding the reasoning ability of models, as it ignores situations where models may obtain correct answers through unfaithful or spurious reasoning shortcuts (Saparov and He, 2023). Some recent research has started to evaluate the validity of the intermediate reasoning steps of LLMs (Golovneva et al., 2022; Prasad et al., 2023). However, they mainly focus on the relationship between the intermediate step and the final answer rather than measuring whether the model understands the intermediate reasoning process. This paper proposes a novel logical reasoning benchmark that requires intermediate counterfactual thinking under logical relation restrictions. This leads to a more in-depth study of the model’s intermediate reasoning process.
11014

heading: 3 CLOMO Benchmark

heading: 3.1 Task Definition
text: The desideratum is to harvest LLM counterfactual thinking and then investigate the validation of the thinking and its alignment with human counterfactual thinking. To achieve this, the LLM should generate its counterfactual thinking under proper logical scenarios. We design a task of counterfactual modification of argument text given a pertur-

heading: 3.2 Benchmark Construction
text: R
M Counterfactual modification
Logical relations • Provides a necessary assumption to • Provides a sufficient assumption to • Strengthen • Weaken
Argument Argument’
Premise 1 Premise 2
R
M
𝒄(𝒓|𝒑𝟏, 𝒂) 𝒄(𝒓|𝒑𝟐, 𝒂′)

heading: 4 SES: Self-Evaluation Scores

heading: 3.3 Data Statistics
text: Max Min Mean Medium
CoT-Train 620 231 382.7 379 CoT-Dev 576 247 378.9 376 CoT-Test 620 216 374.7 371
Few-Train 1,569 1,180 1,331.7 1,328 Few-Dev 1,525 1,196 1,327.9 1,325 Few-Test 1,569 1,165 1,323.7 1,320
Zero-Train 609 220 371.7 368 Zero-Dev 565 236 367.9 365 Zero-Test 609 205 363.7 360
Table 3: CLOMO input length statistics by number of tokens. CoT: The chain-of-thought setting. Few: The few-shot setting. Zero: The zero-shot setting.
have easy access to standard/human-tested answers. Specifically, we split the scenario of complex logical reasoning evaluation into several discriminative tasks that LLMs have already seen and have been heavily trained on through logical conceptual graphs for the model to perform high-precision reasoning evaluation. We then collect the evaluations of these simple tasks and compute the ratings of the complex reasoning tasks based on the structure of the logical concept graph.
Counterfactual Modification Concept Graph Figure 2 demonstrates the graph. To make the logic of counterfactual reasoning hold, we have a pair of primitive states p1 (Premise1) and a (Argument) from p1 and a satisfying the relation r, i.e., Pr(r|p1, a) approaches 1. The other claim p2 (Premise2) and the modified a′ (Argument′) satisfy the same relation r, i.e., Pr(r|p2, a′) approaches 1. In contrast, the relation between p2 and the a should not satisfy the relation r, i.e., Pr(r|p2, a) approaches 0.
Decomposed Self-Evaluation Score As the demonstrated complex logical reasoning ability of current large language models, we let the large language model estimate Pr(r|p1, a), Pr(r|p2, a′), and Pr(r|p2, a), respectively. Specifically, we design a few binary classification tasks for the large language models, so that the probabilities are simplified to c(r|p1, a), c(r|p2, a′), and c(r|p2, a) ∈ {0, 1}. Table 4 demonstrates the prompts. The overall logical modification score is computed by:
s = c(r|p1, a)× c(r|p2, a′)− c(r|p2, a)× c(r|p2, a′) (1)
The intuition of Eq.(1) is that, according to the concept graph in Figure 2, a desired Argument’ results in: i. The logical relation r is satisfied in both pairs (Premise2,Argument′) and (Premise1,Argument), that is Pr(r|p1, a)× Pr(r|p2, a′), where Pr(r|p1, a) denotes the probability of logical relation r holds given premise p1 and argument a, and Pr(r|p2, a′) denote the probability of logical relation r holds given the modified argument a′ and premise p2. Practically, we prompt an LLM to classify the logical relation given an argument-premise pair and collect the responses c(r|p1, a) and c(r|p2, a′). As a result, the first term in Eq.(1) is c(r|p1, a) × c(r|p2, a′). ii. The logical relation R in Figure 2 can not hold between (Premise2,Argument). In other words, the probability of r holds between (Premise2,Argument) should be distinguished from that between the modified (Premise2,Argument′) as much as possible, which is −Pr(r|p2, a) × Pr(r|p2, a′). We use an LLM to do classification, resulting in c(r|p2, a)×c(r|p2, a′) as the second term in Eq.(1).
Alignment with Human Evaluation We randomly select 50 samples from the test set and examined how well the self-evaluation score (SES) matches human evaluation. Specifically, we use GPT-4 to implement SES. We recruit experts in argumentation to evaluate the modified arguments generated by GPT-4 on the selected 50 samples, scoring 1 for good answers and 0 for bad answers. The Cohen’s Kappa coefficient between the human annotators is κ = 0.6785, indicating substantial consistency of human perspective. We then use the self-evaluation score to evaluate the same group of modified arguments again. Cohen’s kappa coefficient between human and self-evaluation score is κ = 0.4391. This indicates that the self-evaluation score is a safe reference and assistance for humans as the first study on automating the evaluation of the highly challenging counterfactual modification task. Therefore, we can apply the SES score for more efficient logical modification evaluation. Also, we believe in a further improved automated score for this task, which we leave as a future work. Additionally, the SES score can adjust to the lat-
11016
SES SESNA SESSA SESS SESW
Human Performance 0.580 0.456 0.500 0.771 0.639
GPT-3.5-Turbo (OpenAI, 2022) 0.335 0.405 0.143 0.486 0.222
GPT-4 (OpenAI, 2023) 0.475 0.544 0.643 0.714 0.250
GPT-4o1 0.680 0.696 0.667 0.800 0.625

heading: 5.2 Fine-Tuning with Counterfactual Data

heading: 5.1 Main Results

heading: 5 Experiments
text: CoT Few Zero R w/o R Full w/o R Full w/o R Full
Necessary Assumption 0.434 0.494 0.052 0.633 0.234 0.557 Sufficient Assumption 0.286 0.500 0.000 0.357 0.234 0.557 Strengthen 0.457 0.771 0.114 0.800 0.343 0.771 Weaken 0.222 0.611 0.014 0.750 0.243 0.708

heading: 5.3 Performances of Small Language Models

heading: Sufficient Assumption. S: Strengthen. W: Weaken.
text: better than Vicuna-7B-v1.5. (4) For the four different relations, it is shown that Sufficient Assumption and Weaken are significantly more challenging than the other two reasoning types. The possible reason is that compared to the other two reasoning types, Sufficient Assumption and Weaken require more reasoning steps such as reversed thinking. Two cases are further shown in Section 5.4. To sum up, the CLOMO task is challenging to current language models. Therefore, further investigation is needed on the counterfactual reasoning abilities of language models.

heading: 5.4 Case Study
text: We compare the modification by GPT-4 and humans. Figure 4 shows a case about sufficient assumption. That is, whether premise is a sufficient assumption for argument. The focus of the two premises here is to predict the impact of the charter provision. Switching from Premise 1 to Premise 2, the focus of discussion changes from the group of major powers to other nations in response to threats
to world peace. This mainly affects the elaboration of Statement 3. GPT-4 has revised Statement 3 accordingly. Humans also made changes to Statement 3. Both revisions emphasized the influence of the five major powers in Statement 3, which corresponded to Premise 2. We find that GPT-4 can handle complex logical reasoning and counterfactual reasoning to a certain extent.
Figure 5 is about weakening an argument. The argument is on human intellectual development. Statement 2 provides evidence to support Statement 1. Premise 1 is on medical conditions and treatments providing counterexamples, and Premise 2 is on inaccuracies in research data. Human modifies Statement 2 to emphasize conversation, thus satisfying the logical conflict with the inaccuracies in research data described in Premise 2, thus satisfying the weaken relation. GPT-4 modifies Statement 1 by replacing intellectual activities with physical activities such as sports or gym. Intellectual/physical activities have some counterfactual
11019
Good Case
In the following, you will see an argument and 2 premises, where Premise 1 provides a sufficient assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a sufficient assumption to the Argument instead, while Premise 1 fails to provides a sufficient assumption to the Argument. Note that no additional statement should be added. Argument: Statement1: After the second world war, the charter of the newly formed united nations established an eleven-member security council and charged it with taking collective action in response to threats to world peace. Statement2: The charter further provided that the five nations that were then the major powers would permanently have sole authority to cast vetoes. Statement3: The reason given for this arrangement was that the burden of maintaining world peace would rest on the world's major powers, and no nation should be required to assume the burden of enforcing a decision it found repugnant. Premise1: No nation that was not among the major powers at the end of the second world war would become a major power. Premise2: Decisions reached by a majority of nations in response to threats to world peace would be biased in favor of one or more major powers. Please write the modified argument below:
Argument: Statement1: … … Statement3: The reason given for this arrangement was that the decisions reached by a majority of nations could be biased in favor of one or more major powers, thus ensuring their support in enforcing these decisions.

heading: GPT
text: Human

heading: Input
text: Inferior Case
contrast, but in the context of the argument at hand, the logical relation restriction is not satisfied. We find that GPT-4 can still be flawed in counterfactual reasoning. In conclusion, complex counterfactual reasoning is challenging for large language models and needs improvements.

heading: 6 Conclusion
text: In this paper, we study large language models’ counterfactual reasoning capability under the constraint of proper logical relations. To this end, we introduce a novel task, Counterfactual Logical Modification, that requires the LLMs to conduct counterfactual modification with logical restriction, where LLMs need to appropriately modify an argument text so that a specified logical relation stands. To ensure a comprehensive evaluation, we then construct a benchmark dataset CLOMO. More-
over, we propose a Self-Evaluation Score (SES) that decomposes the evaluation into several LLMs basic discrimination tasks, which is demonstrated aligned with human evaluations. We further evaluate smaller language models in inference-only and fine-tuning manners. The fine-tuned models’ performances are significantly improved, but there is still a large gap with human performance. Our findings thus underscore the need for further development in LLMs’ counterfactual reasoning.

heading: 7 Limitations
text: The major limitation of this work is that we do not include multiple reference sentences in this version of CLOMO. However, the proposed SES score leverages LLM to conduct a human-aligned evaluation, so it reduces the impact of reference in evaluating the model performances.
11020

heading: Acknowledgements
text: This work was supported in part by the National Science and Technology Major Project under Grant No. 2020AAA0109704, the Research Grants Council of the Hong Kong SAR under Grant GRF 11217823 and Collaborative Research Fund C104223GF, the National Natural Science Foundation of China under Grant 62371411, InnoHK initiative, the Government of the HKSAR, Laboratory for AI-Powered Financial Technologies. The authors thank Dr. Xingchi Su, Guangyan Sun, and Cen Li for their great effort in carefully reviewing the data.

heading: A Ethical Considerations
text: The data and annotations are collected without personal or confidential information. Therefore, there is no ethical concern to the best of our knowledge.

heading: B Samples from CLOMO
text: Tables 8 and 10 demonstrate examples from the CLOMO dataset.

heading: C Data Source
text: Since applicable data for the proposed task is lacking, we build a benchmark dataset by carefully selecting argument texts and collecting human annotation of the modified Argument′. We choose to use ReClor (Yu et al., 2020) as the source data, considering that ReClor includes standardized multiple-choice questions on argument texts and logical relations from LSAT. We then recruit domain experts to conduct the annotation.
D Implementation Details of Three Inference-Only Settings
Tables 11 to 22 demonstrate examples of three inference-only samples with input and output. The three inference-only settings are:
• Few-shot setting: We first give four demonstrations in the prompt, and then provide an unseen question for the language model. The four demonstrations are randomly selected from the CLOMO training set, each of which is from one of the four reasoning relations.
• Zero-shot setting: We directly provide the question to be solved in the prompt.
• Chain-of-thought setting: We first provide the question, and then remind the language model to think step-by-step in the prompt.
Please see the Tables for the detailed demonstrations and prompts.

heading: E Compared Models
text: We use the following language models in the experiments.
11024
Flan-T5 (Chung et al., 2022) is a family of language models that are instruction-finetuned on the T5 models (Raffel et al., 2020). The models perform well on commonsense reasoning, mathematics, history, law, and medicine. In this paper, we use Flan-T5-Large with 780M parameters, Flan-T5XL with 3B parameters, and Flan-T5-XXL with 11B parameters.
ChatGLM2 (Du et al., 2022a) is an open-source bilingual (Chinese-English) chat model based on GLM (Du et al., 2022b). The models are refined with data with a context length of up to 32K. The model has strong performance on multiple reasoning tasks.
Baichuan2 (Baichuan, 2023) is a family of language models trained on a high-quality corpus with 2.6 trillion tokens. The models achieve good performance on multiple authoritative Chinese, English, and multi-language general and domain-specific benchmarks. In the experiments, we use the Chat models Baichuan2-7B-Chat and Baichuan2-13BChat.
InternLM (Team, 2023) takes trillions of highquality tokens for training to establish a powerful knowledge base. It has outstanding comprehensive performance. We use the Chat model InternLMChat-7B in our experiments.
Vicuna-v1.5 (Chiang et al., 2023) models are fine-tuned on the LLaMA 2 (Touvron et al., 2023b) models with supervised instruction finetuning with user-shared conversations collected from ShareGPT3.
Qwen (Bai et al., 2023) is a series of comprehensive language model. The Qwen-Chat models are further fine-tuned with human alignment techniques such as Reinforcement Learning with Human Feedback (RLHF). The chat models have advanced tool use and planning capabilities.
WizardLM (Xu et al., 2024) is fine-tuned based on LLaMA (Touvron et al., 2023a) with a mixture of generated instruction data. The model shows its benefits in various skills such as philosophy, technology, and physics.
LLaMA (Touvron et al., 2023a) is a collection of foundation language models trained on trillions of tokens with publicly available datasets.
LLaMA 2 (Touvron et al., 2023b) is a family of pre-trained and fine-tuned language models that can be adapted for a variety of natural language generation tasks.
3https://sharegpt.com/
F Implementation Details of Fine-Tuning
We conduct full-parameter fine-tuning to LLaMA and LLaMA 2. The fine-tuning data is the CLOMO training set. Each model is fine-tuned by 10 epochs with a batch size of 4. The learning rate is 2e− 5 and is adapted by the cosine scheduler with a warmup proportion of 0.03. The best checkpoint is selected by the minimum perplexity in the validation split. In the inference phase, the models use beam search and a temperature of 0.7.
11025

heading: Input
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a necessary assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a necessary assumption to the Argument instead, while Premise 1 fails to provides a necessary assumption to the Argument. Note that no additional statement should be added. Argument: Statement1: In 1963, a young macaque monkey was observed venturing into a hot spring to retrieve food which had fallen in. Soon, other macaques began to enter the spring, and over a few years this behavior was adopted by the entire troop. Prior to 1963, no macaques had ever been observed in the hot spring ; by 1990, the troop was regularly spending time there during the winters. Statement2: Thus, these macaques are able to adopt and pass on new patterns of social behavior, and are not complete captives of their genetic heritage. Premise1: New patterns of behavior that emerge in macaque populations over the course of a few years or decades are not necessarily genetically predetermined. Premise2: The social behaviors of macaques are completely independent of their genetic heritage. Please write the modified argument below:

heading: Output
text: Argument: Statement1: In 1963, a young macaque monkey was observed venturing into a hot spring to retrieve food which had fallen in. Soon, other macaques began to enter the spring, and over a few years this behavior was adopted by the entire troop. Prior to 1963, no macaques had ever been observed in the hot spring ; by 1990, the troop was regularly spending time there during the winters. Statement2: Thus, these macaques’ new patterns of social behavior must be determined by others.

heading: Input
text: In the following, you will see an argument and 2 premises, where Premise 1 weakens to the Argument. Please modify the Statements in the Argument until Premise 2 weakens to the Argument instead, while Premise 1 fails to weakens to the Argument. Note that no additional statement should be added. Argument: Statement1: Measurements of the motion of the planet uranus seem to show uranus being tugged by a force pulling it away from the sun and the inner planets. Statement2: Neptune and pluto, the two known planets whose orbits are farther from the sun than is the orbit of uranus, do not have enough mass to exert the force that the measurements indicate. Statement3: Therefore, in addition to the known planets, there must be at least one planet in our solar system that we have yet to discover. Premise1: There is a belt of comets beyond the orbit of pluto with powerful gravitational pull. Premise2: Neither neptune nor pluto is as massive as uranus. Please write the modified argument below:

heading: Output
text: Argument: Statement1: Measurements of the motion of the planet uranus seem to show uranus being tugged by a force pulling it away from the sun and the inner planets. Statement2: Neptune and pluto, the two known planets whose orbits are farther from the sun than is the orbit of uranus. Statement3: Therefore, one of the two planets must tug uranus.

heading: Input
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a necessary assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a necessary assumption to the Argument instead, while Premise 1 fails to provides a necessary assumption to the Argument. Note that no additional statement should be added. Argument: Statement1: Critic : historians purport to discover the patterns inherent in the course of events. But historians actually impose, rather than find, such patterns by choosing what to include in and exclude from their historical narratives. Statement2: Thus, properly understood, histories reveal more about the presuppositions underlying different historians’ attempts to understand what happened than about what actually happened. Premise1: Which pattern a historian imposes upon events is affected by that historian’s presuppositions. Premise2: Historians have many presuppositions in common with one another. Please write the modified argument below:

heading: Output
text: Argument: Statement1: Critic : historians purport to discover the patterns inherent in the course of events. But historians actually impose, rather than find, such patterns by presupposing to choose what to include in and exclude from their historical narratives. Statement2: Thus, properly understood, histories reveal more about the presuppositions underlying different historians’ attempts to understand what happened than about what actually happened, and these patterns are similar.

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a necessary assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a necessary assumption to the Argument instead, while Premise 1 fails to provides a necessary assumption to the Argument. Note that no additional statement should be added.
Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, to be a moral agent one must have free will, because one can not be a moral agent without desiring to conform to a principle. Premise1: Desiring to conform to a principle requires free will. Premise2: It is impossible to have desires without also being a moral agent.
Please write the modified argument below: Modified Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, one can not have desires without desiring to conform to a principle, because one can not be a moral agent without desiring to conform to a principle.
Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only supervisor in the shipping department other than larson and franks. Premise1: The task cannot be assigned to anyone other than a supervisor in the shipping department. Premise2: The task cannot be assigned to anyone who has any kind of scheduling conflict.
Please write the modified argument below: Modified Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only one who do not have scheduling conflict in the shipping department other than larson and franks.
Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often does not enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will not boost sales and, therefore, will not increase profits either. Premise1: The $3. 00 coupons, if distributed, would not induce new customers, who might not have purchased the product otherwise, to try wart-gone. Premise2: Wart-gone can be produced in small or large batches without appreciably lowering the pre-tube production costs.
Please write the modified argument below: Modified Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often can enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will boost produce and, however, will not change pre-tube production costs much.
Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : 65 percent of its graduates went on to internships or jobs in print or broadcast journalism. Premise1: More than half of the school’s students came from jobs in journalism to improve their skills. Premise2: Some newspaper editors do not regard journalism schools as a necessary part of the training of a journalist.
Please write the modified argument below: Modified Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : the journalism regards journalism schools as a necessary part of the training of a journalist.
Argument: Statement1: Consumer advocate : there is no doubt that the government is responsible for the increased cost of gasoline, because the government’s policies have significantly increased consumer demand for fuel, and as a result of increasing demand, the price of gasoline has risen steadily. Premise1: The government can bear responsibility for that which it indirectly causes. Premise2: Consumer demand for gasoline cannot increase without causing gasoline prices to increase.
Please write the modified argument below:

heading: Output:
text: Statement1: Consumer advocate : there is no doubt that the government partly leads to the increased cost of gasoline, because the government’s policies have significantly increased consumer demand for fuel, and as a result of increasing demand, the price of gasoline has risen steadily undoubtedly.
Table 11: Few-shot setting inference-only sample with input and output. Logical relation: Necessary Assumption.11027

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a sufficient assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a sufficient assumption to the Argument instead, while Premise 1 fails to provides a sufficient assumption to the Argument. Note that no additional statement should be added.
Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, to be a moral agent one must have free will, because one can not be a moral agent without desiring to conform to a principle. Premise1: Desiring to conform to a principle requires free will. Premise2: It is impossible to have desires without also being a moral agent.
Please write the modified argument below: Modified Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, one can not have desires without desiring to conform to a principle, because one can not be a moral agent without desiring to conform to a principle.
Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only supervisor in the shipping department other than larson and franks. Premise1: The task cannot be assigned to anyone other than a supervisor in the shipping department. Premise2: The task cannot be assigned to anyone who has any kind of scheduling conflict.
Please write the modified argument below: Modified Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only one who do not have scheduling conflict in the shipping department other than larson and franks.
Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often does not enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will not boost sales and, therefore, will not increase profits either. Premise1: The $3. 00 coupons, if distributed, would not induce new customers, who might not have purchased the product otherwise, to try wart-gone. Premise2: Wart-gone can be produced in small or large batches without appreciably lowering the pre-tube production costs.
Please write the modified argument below: Modified Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often can enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will boost produce and, however, will not change pre-tube production costs much.
Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : 65 percent of its graduates went on to internships or jobs in print or broadcast journalism. Premise1: More than half of the school’s students came from jobs in journalism to improve their skills. Premise2: Some newspaper editors do not regard journalism schools as a necessary part of the training of a journalist.
Please write the modified argument below: Modified Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : the journalism regards journalism schools as a necessary part of the training of a journalist.
Argument: Statement1: Super express shipping company has implemented a new distribution system that can get almost every package to its destination the day after it is sent. The company worries that this more efficient system will result in lower sales of its premium next-day delivery service, because its two day service will usually arrive the following day anyway. Statement2: The company plans to encourage sales of its next-day service by intentionally delaying delivery of its two-day packages so that they will not be delivered the following day, even if the package arrives at its destination city in time for next-day delivery. Premise1: Deliberate delay of packages will not affect the company’s image in a way that significantly reduces its ability to attract and retain customers. Premise2: The overnight service is too expensive to be attractive to most customers currently.
Please write the modified argument below:

heading: Output:
text: Statement1: Super express shipping company has paused to implemente a new distribution system that can get almost every package to its destination the day after it is sent. The company worries that this more efficient system will result in lower sales of its premium next-day delivery service, because its two day service will usually arrive the following day anyway. Statement2: The company plans to encourage sales of its next-day service by stopping accelerating delivery of its two-day packages so that they will not be delivered the following day, leaving the package arriving at its destination city overnight for next-day delivery.
Table 12: Few-shot setting inference-only sample with input and output. Logical relation: Sufficient Assumption.11028

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 strengthens the Argument. Please modify the Statements in the Argument until Premise 2 strengthens the Argument instead, while Premise 1 fails to strengthens the Argument. Note that no additional statement should be added.
Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, to be a moral agent one must have free will, because one can not be a moral agent without desiring to conform to a principle. Premise1: Desiring to conform to a principle requires free will. Premise2: It is impossible to have desires without also being a moral agent.
Please write the modified argument below: Modified Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, one can not have desires without desiring to conform to a principle, because one can not be a moral agent without desiring to conform to a principle.
Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only supervisor in the shipping department other than larson and franks. Premise1: The task cannot be assigned to anyone other than a supervisor in the shipping department. Premise2: The task cannot be assigned to anyone who has any kind of scheduling conflict.
Please write the modified argument below: Modified Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only one who do not have scheduling conflict in the shipping department other than larson and franks.
Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often does not enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will not boost sales and, therefore, will not increase profits either. Premise1: The $3. 00 coupons, if distributed, would not induce new customers, who might not have purchased the product otherwise, to try wart-gone. Premise2: Wart-gone can be produced in small or large batches without appreciably lowering the pre-tube production costs.
Please write the modified argument below: Modified Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often can enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will boost produce and, however, will not change pre-tube production costs much.
Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : 65 percent of its graduates went on to internships or jobs in print or broadcast journalism. Premise1: More than half of the school’s students came from jobs in journalism to improve their skills. Premise2: Some newspaper editors do not regard journalism schools as a necessary part of the training of a journalist.
Please write the modified argument below: Modified Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : the journalism regards journalism schools as a necessary part of the training of a journalist.
Argument: Statement1: Often patients with ankle fractures that are stable, and thus do not require surgery, are given follow-up x-rays because their orthopedists are concerned about possibly having misjudged the stability of the fracture. Statement2: When a number of follow-up x-rays were reviewed, however, all the fractures that had initially been judged stable were found to have healed correctly. Statement3: Therefore, it is a waste of money to order follow-up x-rays of ankle fracture initially judged stable. Premise1: X-rays of patients of many different orthopedists working in several hospitals were reviewed. Premise2: Doctors who are general practitioners rather than orthopedists are less likely than orthopedists to judge the stability of an ankle fracture correctly.
Please write the modified argument below:

heading: Output:
text: Statement1: Often patients with ankle fractures that are stable, and thus do not require surgery, therefore, many general doctors do not order an x-ray to patients with stable ankle fractures in order to avoid wasting medical resources. Statement2: When a number of follow-up x-rays were reviewed, however, most of the fractures that had initially been judged stable were found to haven’t healed correctly. Statement3: Therefore, it is necessary to order follow-up x-rays of ankle fracture initially judged stable.
Table 13: Few-shot setting inference-only sample with input and output. Logical relation: Strengthen.11029

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 weakens the Argument. Please modify the Statements in the Argument until Premise 2 weakens the Argument instead, while Premise 1 fails to weakens the Argument. Note that no additional statement should be added.
Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, to be a moral agent one must have free will, because one can not be a moral agent without desiring to conform to a principle. Premise1: Desiring to conform to a principle requires free will. Premise2: It is impossible to have desires without also being a moral agent.
Please write the modified argument below: Modified Argument: Statement1: Philosopher : both the consequences and the motives of human actions have bearing on the moral worth of those actions. Statement2: Nonetheless, one can not have desires without desiring to conform to a principle, because one can not be a moral agent without desiring to conform to a principle.
Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only supervisor in the shipping department other than larson and franks. Premise1: The task cannot be assigned to anyone other than a supervisor in the shipping department. Premise2: The task cannot be assigned to anyone who has any kind of scheduling conflict.
Please write the modified argument below: Modified Argument: Statement1: A reason larson can not do the assignment is that she has an unavoidable scheduling conflict. Statement2: On the other hand, a reason franks can not do the assignment is that he does not quite have the assertiveness the task requires. Statement3: So, the task must be assigned to parker, the only one who do not have scheduling conflict in the shipping department other than larson and franks.
Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often does not enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will not boost sales and, therefore, will not increase profits either. Premise1: The $3. 00 coupons, if distributed, would not induce new customers, who might not have purchased the product otherwise, to try wart-gone. Premise2: Wart-gone can be produced in small or large batches without appreciably lowering the pre-tube production costs.
Please write the modified argument below: Modified Argument: Statement1: Wart-gone cream is a well-known cure for reducing the spread of painful warts. A dime size amount rubbed on the infected area daily is the prescribed dosage, and applying the cream more often can enhance the cream’s healing effects. Statement3: Hence, distributing coupons giving $ 3.00 off the price of wart-gone will boost produce and, however, will not change pre-tube production costs much.
Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : 65 percent of its graduates went on to internships or jobs in print or broadcast journalism. Premise1: More than half of the school’s students came from jobs in journalism to improve their skills. Premise2: Some newspaper editors do not regard journalism schools as a necessary part of the training of a journalist.
Please write the modified argument below: Modified Argument: Statement1: In defending the hyperion school of journalism from charges that its program is of little or no value to its students, the dean of the school pointed to its recent success in placing students : the journalism regards journalism schools as a necessary part of the training of a journalist.
Argument: Statement1: To suit the needs of corporate clients, advertising agencies have successfully modified a strategy originally developed for political campaigns. Statement2: This strategy aims to provide clients with free publicity and air time by designing an advertising campaign that is controversial, thus drawing prime-time media coverage and evoking public comment by officials. Premise1: The usefulness of an advertising campaign is based solely on the degree to which the campaign’s advertisements persuade their audiences. Premise2: Advertising agencies that specialize in campaigns for corporate clients are not usually chosen for political campaigns.
Please write the modified argument below:

heading: Output:
text: Statement1: To suit the needs of corporate clients, advertising agencies have successfully modified a strategy originally developed for political campaigns. Statement2: This strategy aims to provide clients with free publicity and air time by designing an advertising campaign that is controversial, thus drawing prime-time media coverage and evoking public comment by officials. often such advertising agencies are chosen as partners for political campaigns.
Table 14: Few-shot setting inference-only sample with input and output. Logical relation: Weaken.11030

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a necessary assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a necessary assumption to the Argument instead, while Premise 1 fails to provides a necessary assumption to the Argument. Note that no additional statement should be added.
Argument: Statement1: Consumer advocate : there is no doubt that the government is responsible for the increased cost of gasoline, because the government’s policies have significantly increased consumer demand for fuel, and as a result of increasing demand, the price of gasoline has risen steadily. Premise1: The government can bear responsibility for that which it indirectly causes. Premise2: Consumer demand for gasoline cannot increase without causing gasoline prices to increase.
Please write the modified argument below:

heading: Output:
text: Statement1: Consumer advocate : there is no doubt that the government partly leads to the increased cost of gasoline, because the government’s policies have significantly increased consumer demand for fuel, and as a result of increasing demand, the price of gasoline has risen steadily undoubtedly.
Table 15: Zero-shot setting inference-only sample with input and output. Logical relation: Necessary Assumption.

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a sufficient assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a sufficient assumption to the Argument instead, while Premise 1 fails to provides a sufficient assumption to the Argument. Note that no additional statement should be added.
Argument: Statement1: Super express shipping company has implemented a new distribution system that can get almost every package to its destination the day after it is sent. The company worries that this more efficient system will result in lower sales of its premium next-day delivery service, because its two day service will usually arrive the following day anyway. Statement2: The company plans to encourage sales of its next-day service by intentionally delaying delivery of its two-day packages so that they will not be delivered the following day, even if the package arrives at its destination city in time for next-day delivery. Premise1: Deliberate delay of packages will not affect the company’s image in a way that significantly reduces its ability to attract and retain customers. Premise2: The overnight service is too expensive to be attractive to most customers currently.
Please write the modified argument below:

heading: Output:
text: Statement1: Super express shipping company has paused to implemente a new distribution system that can get almost every package to its destination the day after it is sent. The company worries that this more efficient system will result in lower sales of its premium next-day delivery service, because its two day service will usually arrive the following day anyway. Statement3: The company plans to encourage sales of its next-day service by stopping accelerating delivery of its two-day packages so that they will not be delivered the following day, leaving the package arriving at its destination city overnight for next-day delivery.
Table 16: Zero-shot setting inference-only sample with input and output. Logical relation: Sufficient Assumption.
11031

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 strengthens the Argument. Please modify the Statements in the Argument until Premise 2 strengthens the Argument instead, while Premise 1 fails to strengthens the Argument. Note that no additional statement should be added.
Argument: Statement1: Often patients with ankle fractures that are stable, and thus do not require surgery, are given follow-up x-rays because their orthopedists are concerned about possibly having misjudged the stability of the fracture. Statement2: When a number of follow-up x-rays were reviewed, however, all the fractures that had initially been judged stable were found to have healed correctly. Statement3: Therefore, it is a waste of money to order follow-up x-rays of ankle fracture initially judged stable. Premise1: X-rays of patients of many different orthopedists working in several hospitals were reviewed. Premise2: Doctors who are general practitioners rather than orthopedists are less likely than orthopedists to judge the stability of an ankle fracture correctly.
Please write the modified argument below:

heading: Output:
text: Statement1: Often patients with ankle fractures that are stable, and thus do not require surgery, therefore, many general doctors do not order an x-ray to patients with stable ankle fractures in order to avoid wasting medical resources. Statement2: When a number of follow-up x-rays were reviewed, however, most of the fractures that had initially been judged stable were found to haven’t healed correctly. Statement3: Therefore, it is necessary to order follow-up x-rays of ankle fracture initially judged stable.

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 weakens the Argument. Please modify the Statements in the Argument until Premise 2 weakens the Argument instead, while Premise 1 fails to weakens the Argument. Note that no additional statement should be added.
Argument: Statement1: To suit the needs of corporate clients, advertising agencies have successfully modified a strategy originally developed for political campaigns. Statement2: This strategy aims to provide clients with free publicity and air time by designing an advertising campaign that is controversial, thus drawing prime-time media coverage and evoking public comment by officials. Premise1: The usefulness of an advertising campaign is based solely on the degree to which the campaign’s advertisements persuade their audiences. Premise2: Advertising agencies that specialize in campaigns for corporate clients are not usually chosen for political campaigns.
Please write the modified argument below:

heading: Output:
text: Statement1: To suit the needs of corporate clients, advertising agencies have successfully modified a strategy originally developed for political campaigns. Statement2: This strategy aims to provide clients with free publicity and air time by designing an advertising campaign that is controversial, thus drawing prime-time media coverage and evoking public comment by officials. often such advertising agencies are chosen as partners for political campaigns.
Table 18: Zero-shot setting inference-only sample with input and output. Logical relation: Weaken.
11032

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a necessary assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a necessary assumption to the Argument instead, while Premise 1 fails to provides a necessary assumption to the Argument. Note that no additional statement should be added.
Argument: Statement1: Consumer advocate : there is no doubt that the government is responsible for the increased cost of gasoline, because the government’s policies have significantly increased consumer demand for fuel, and as a result of increasing demand, the price of gasoline has risen steadily. Premise1: The government can bear responsibility for that which it indirectly causes. Premise2: Consumer demand for gasoline cannot increase without causing gasoline prices to increase.
Please write the modified argument below. Let’s think step by step:

heading: Output:
text: Statement1: Consumer advocate : there is no doubt that the government partly leads to the increased cost of gasoline, because the government’s policies have significantly increased consumer demand for fuel, and as a result of increasing demand, the price of gasoline has risen steadily undoubtedly.
Table 19: Chain-of-thought setting inference-only sample with input and output. Logical relation: Necessary Assumption.

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 provides a sufficient assumption to the Argument. Please modify the Statements in the Argument until Premise 2 provides a sufficient assumption to the Argument instead, while Premise 1 fails to provides a sufficient assumption to the Argument. Note that no additional statement should be added.
Argument: Statement1: Super express shipping company has implemented a new distribution system that can get almost every package to its destination the day after it is sent. The company worries that this more efficient system will result in lower sales of its premium next-day delivery service, because its two day service will usually arrive the following day anyway. Statement2: The company plans to encourage sales of its next-day service by intentionally delaying delivery of its two-day packages so that they will not be delivered the following day, even if the package arrives at its destination city in time for next-day delivery. Premise1: Deliberate delay of packages will not affect the company’s image in a way that significantly reduces its ability to attract and retain customers. Premise2: The overnight service is too expensive to be attractive to most customers currently.
Please write the modified argument below. Let’s think step by step:

heading: Output:
text: Statement1: Super express shipping company has paused to implemente a new distribution system that can get almost every package to its destination the day after it is sent. The company worries that this more efficient system will result in lower sales of its premium next-day delivery service, because its two day service will usually arrive the following day anyway. Statement3: The company plans to encourage sales of its next-day service by stopping accelerating delivery of its two-day packages so that they will not be delivered the following day, leaving the package arriving at its destination city overnight for next-day delivery.
Table 20: Chain-of-thought setting inference-only sample with input and output. Logical relation: Sufficient Assumption.
11033

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 strengthens the Argument. Please modify the Statements in the Argument until Premise 2 strengthens the Argument instead, while Premise 1 fails to strengthens the Argument. Note that no additional statement should be added.
Argument: Statement1: Often patients with ankle fractures that are stable, and thus do not require surgery, are given follow-up x-rays because their orthopedists are concerned about possibly having misjudged the stability of the fracture. Statement2: When a number of follow-up x-rays were reviewed, however, all the fractures that had initially been judged stable were found to have healed correctly. Statement3: Therefore, it is a waste of money to order follow-up x-rays of ankle fracture initially judged stable. Premise1: X-rays of patients of many different orthopedists working in several hospitals were reviewed. Premise2: Doctors who are general practitioners rather than orthopedists are less likely than orthopedists to judge the stability of an ankle fracture correctly.
Please write the modified argument below. Let’s think step by step:

heading: Output:
text: Statement1: Often patients with ankle fractures that are stable, and thus do not require surgery, therefore, many general doctors do not order an x-ray to patients with stable ankle fractures in order to avoid wasting medical resources. Statement2: When a number of follow-up x-rays were reviewed, however, most of the fractures that had initially been judged stable were found to haven’t healed correctly. Statement3: Therefore, it is necessary to order follow-up x-rays of ankle fracture initially judged stable.

heading: Input:
text: In the following, you will see an argument and 2 premises, where Premise 1 weakens the Argument. Please modify the Statements in the Argument until Premise 2 weakens the Argument instead, while Premise 1 fails to weakens the Argument. Note that no additional statement should be added.
Argument: Statement1: To suit the needs of corporate clients, advertising agencies have successfully modified a strategy originally developed for political campaigns. Statement2: This strategy aims to provide clients with free publicity and air time by designing an advertising campaign that is controversial, thus drawing prime-time media coverage and evoking public comment by officials. Premise1: The usefulness of an advertising campaign is based solely on the degree to which the campaign’s advertisements persuade their audiences. Premise2: Advertising agencies that specialize in campaigns for corporate clients are not usually chosen for political campaigns.
Please write the modified argument below. Let’s think step by step:

heading: Output:
text: Statement1: To suit the needs of corporate clients, advertising agencies have successfully modified a strategy originally developed for political campaigns. Statement2: This strategy aims to provide clients with free publicity and air time by designing an advertising campaign that is controversial, thus drawing prime-time media coverage and evoking public comment by officials .often such advertising agencies are chosen as partners for political campaigns.
Table 22: Chain-of-thought setting inference-only sample with input and output. Logical relation: Weaken.
11034

heading: Future Work
text: Also, we believe in a further improved automated score for this task, which we leave as a future work. Additionally, the SES score can adjust to the lat-
11016
SES SESNA SESSA SESS SESW
Human Performance 0. 580 0. 456 0. 500 0. 771 0. 639
GPT-3. 5-Turbo (OpenAI, 2022) 0. 335 0. 405 0. 143 0. 486 0. 222
GPT-4 (OpenAI, 2023) 0. 475 0. 544 0. 643 0. 714 0. 250
GPT-4o1 0. 680 0. 696 0. 667 0. 800 0. 625