LogiGAN: Learning Logical Reasoning via Adversarial Pre-training
Authors: Xinyu Pi
Wanjun Zhong
Yan Gao
Nan Duan
Jian-Guang Lou
Venue: NeurIPS 2022 Accept (2022)
Keywords: NLP
Reasoning
Logic Pre-training

Abstract: We present LogiGAN, an unsupervised adversarial pre-training framework for improving logical reasoning abilities of language models. Upon automatic identification of logical reasoning phenomena in massive text corpus via detection heuristics, we train language models to predict the masked-out logical statements. Inspired by the facilitation effect of reflective thinking in human learning, we analogically simulate the learning-thinking process with an adversarial Generator-Verifier architecture to assist logic learning. LogiGAN implements a novel sequential GAN approach that (a) circumvents the non-differentiable challenge of the sequential GAN by leveraging the Generator as a sentence-level generative likelihood scorer with a learning objective of reaching scoring consensus with the Verifier; (b) is computationally feasible for large-scale pre-training with arbitrary target length. Both base and large size language models pre-trained with LogiGAN demonstrate obvious performance improvement on 12 datasets requiring general reasoning abilities, revealing the fundamental role of logic in broad reasoning, as well as the effectiveness of LogiGAN. Ablation studies on LogiGAN components reveal the relative orthogonality between linguistic and logic abilities and suggest that reflective thinking's facilitation effect might also generalize to machine learning.

LogiGAN: Learning Logical Reasoning via Adversarial Pre-training
Xinyu ;i Wanjun Zhong"*, Yan Gao², Nan Duan Jian-Guang Lou
University of Illinois Urbana-Champaign. Urbana, USA 2Sun Yat-Sen University Miicrosoft Research Asia
xinyupi2@illinois edu, zhong=j25@mail2 syau. edw.uuedu..eo edu. cn {yan.gas jlou, nanduan) cwicroooot..ooommmmmmm com
Abstract
We present LogiGAN, an unsupervised adversarial pre-training framework for im- proving logical reasoning abilities of language models. Upon automatic identifica- tion of logical reasoning phenomena in massive text corpus via detection heuristics, we train language models to predict the masked-out logical statements. Inspired by the facilitation effect of reflective thinking in human learning, we analogically simulate the learning- cearning-thinking process with adversarial Generator- Verifier chitecture to assist logic learning. LogiGAN implements novel sequential GAN approach that (a) circumvents the con-differentiable challenge of the sequential GAN by leveraging the Generator as a sentence-leve generative likelihood scorer with learning objective of reaching scoring consensus with the Verifier; (b) is computationally feasible for large-scale pre-training with longer target length. Both base and large size language models pre-trained with LogiGAN demonstrate obvious performance improvement on 12 datasets requiring general reasoning abilities, revealing the fundamental role of logic in broad reasoning. as well the effectiveness of LogiGAN. Ablation studies on LogiGAN components reveal the relative orthogonality between linguistic and logic abilities and suggest that reflective thinking's facilitation effect might also generalize to machine learning ?
Introduction
"Learning without thinking is labor lost; thinking without learning is perilous."
Confucius
Pre-trained Language Models (PLMs) (Devlin et al. 2018 Brown et al. 2020 Raffel et al. 2020 are approaching human-lewe level performance in numerous tasks requiring basic linguistic abilities (Rajpurkar al., 2016 Wang et al., 2018), setting off huge wave of interest in Natural Language Processing (NLP). Despite the emerging fervor, researchers soon realized that PLMSs are relatively incompetent in their reasoning abilities, which seems be an insurmountable bottleneck for PLMs with even better linguistic abilities (Kassner & Schütze 2019: Helwe et al. (2021). Following this, researchers delve into reasoning from multitudinous aspects, striving to improve PLMs' reasoning abilities.
From our perspective, reasoning (in natural language) is essentially an inferential process where an unstated statement is drawn based on several presented statements, and Logic is the systemic of principles that provides reasoning with correctness and consistency assurance (Hurley (982).
Regardless of the variability of contents, logical reasoning generally incorporates two invariant forms: drawing conclusions based on some premises (aka. deduction induction, (Reichertz) 2013 or
indicates equal contribution. Work done during internship Microsoft Research Asia.
The code is released in
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

hypothesizing premises to explain some conclusions (aka. abduction (Douven 2021)). Most existing tasks requiring general reasoning ability, such as natural language inference (Nie et al., 2019) and complex machine reading comprehension (T.i et al. 2017), can be readily interpreted by this criterion. Other tasks requiring specialized reasoning skills can be considered either as (i) providing sufficient premises but requiring specific ways of premise extraction to draw conclusions, such as multi-hop (Yang et al., 2018b) or hybrid (Chen et al. 2020) reasoning; or (ii) requires external knowledge, such as commonsense (Sap et al. 2020) or numerical (Dua et al., 2019) knowledge, as premises to draw conclusions, hence could also be interpreted by the two forms of logical reasoning. Following this analysis on the relation between logic and reasoning, Logic ability will be an essential foundation for a broad scope of reasoning, and should be prioritized in improving PLMs' reasoning abilities
Conventional pre- pre-training training via randomized Masked Language Modeling (MLM) and auxiliary tasks are generally developed upon Firth (1957)'s distributional hypothesis of semantics - "a word is characterized by the company it keeps." Under this paradigm, models efficiently learn to capture grammatical structures and contextualized semantics. However, since logical consistency is beyond correctness on linguistic level, it is less obvious how MLM could help with logical reasoning abilities. Do models harvest logic ability for free from MLM? Or is that something that needs further learning beyond language acquisition? Motivated by these questions, we propose unsupervised pre-training method aiming at enhancing the logical reasoning ability of PLMs: we automatically identify occurrences of logical reasoning phenomena in large corpus via detection heuristics, and then require PLMs to predict the masked-out logical statements made in the original context (Section 3). For example, in the case "Bob recently made up his mind to lose weight. Therefore, [MASK] ", the prediction goal is the masked original statement "he decides to go on a diet".
However, statements different from the original statement could also be logically consistent with respect to a given context. For example, "he decides to exercise from today on" is also a reasonable inference in the case above. Since Generators trained merely from recovering original statements are not encouraged to explore the possibilities of other reasonable statements, their overall learning effectiveness of logic could potentially be degraded. Therefore, it makes sense to provide additional feedback based on the degree of logical consistency between statements predicted beforehand and the original context we realize this much resembles humans' reflective thinking process. Inspired by research from cogmitive psychology (Moon 2013 Boud et al.; [013] Di Stefano et al. 2016) advocating for the vital role of reflective thinking in improving the experiential efficiency of human learning, we hypothesize that machines might also benefit from reflective thinking in their learning processes. Following this hypothesis, we analogically simulate humans' bearning-thinkee process with Generator- Verifier architecture, and propose LogiGAN, a novel adversarial training approach tailored for sequential GAN training further facilitate the learning of logical reasoning.
In LogiGAN's design, the Generator learns not only to recover the original masked statements, but also to score candidate statements (based on their generative likelihood) in manner that could reach scoring consensus with the Verifier, who learns to make judgments on the logical consistency between premises and conclusions. The more logically consistent the Verifier thinks of statement w.r.t. the input context, the higher generative likelihood score is expected to be assigned by the Generator. To encourage the exploration of broader possibilities of reasonable statements other than the original one, we also apply a diversified sampling strategy for candidate statement generation. Both Generator and Verifier scoring processes are continuous throughout the adversarial training, thereby circumventing the non-differentiable barrier in sequential GAN posed by the discrete beam-search. Moreover, LogiGAN does not involve token-wise Monte Carlo Search for policy gradient estimation, and scoring processes of Generator and Verifier are decoupled, so that parallel score computation is possible. This makes large-scale pre-training with longer target length computationally feasible.
To test the effectiveness of LogiGAN, we extensively experiment on 12 datasets requiring general reasoning ability. The apparent performance improvements of both base and large size PLMS across all tasks reveal models' harvest of logic ability, shoring up the fundamental role of logic in general reasoning. We also carry out ablation studies to understand the functionality of LogiGAN components, the results of which shed light on the relative orthogonality between linguistic and logic ability and suggest that the facilitation effect of reflective thinking is also generalizable to machine learning.
We expand this analysis in-depth in App. A, and refer intrigued readers there.
2

---

2 Logic Pre-training
In this work, primarily focus on improving PLMs' ability of informal logic (Groarke 2021) We include the three most classical types of logical reasoning: deductive, inductive, and abductive reasoning conducted in the form of natural language (Reichertz 2004, Kennedy & Thornberg 2018 Reichertz, 2007, Douven 2021) in our consideration. Note that our coverage is broader than the informal logic strictly defined in the philosophy community (Munson 1976) that primarily focuses on analyzing the soundness and cogency of the application of the aforementioned reasoning in real-life arguments. The other half of logic investigation the normative study of formal logic (typically conducted in a symbolic form), such as truth-function logic (Buvac & Mason 1993), modal logic (Priest 2008). and fuzzy logic (Dote 1995). is beyond the scope of this paper.
Logic Indicators as Detection Heuristics. To set up an unsupervised pre-training aiming at improv- ing models' logic ability, the very first step will be to identify logical reasoning phenomena from vast-scale unstructured text corpus. While invocations of logic are not explicitly stated in most cases, writers' usage of logic indicators usually marks their logical reasoning processes with high precision (uureyy 1982). thereby serving as an ideal heuristic device for our detection purpose. We consider two standard types of logic indicators: (i) conclusion indicator such as "Therefore", "We may infer that", which denotes drawing conclusion deductively or inductively from given premises; And (ii) premise indicator such as "Due to",, "The reason that", which denotes abductively hypothesizing premises that explain or provide evidence to some stated conclusions.
Corpus Construction. For text corpus, we detect every occurrence of pre-defined logic indicators (listed in App. C), and mask out (i.e., replace with [MASK]) the entire statement subsequent to the indicator (each training example will have exactly one masked-out statement). Then models' task will be to perform language modeling and predict the masked statement. We emphasize that statements are declarative sentences or declarative clauses, owning complete subject and predicate structures, and are capable of being factually true or false. To supply sufficient context information for these predictions, we keep x complete sentences previous the [MASK], as well as y sentences after the [MASK], where x' and y can be sampled from a geometric distribution with pre-defined hyper-parameters. Fig. iillstraates the input and output format, and we discuss details in Sec.
Masked Logical Statement Prediction. In the simplest setting, the Generator learns to infill the masked statement via a single-task pre-training, which fulfills the training process of a typical masked language modeling task. The only difference is that models no longer predict randomly masked tokens or spans but instead logic-targeted masked complete statements. Models are trained to perform Max Likelihood Estimation (MLE) for masked statements under typical teacher forcing loss. Practically, generative pre-trained language models such as T5 (Raffel et al. (2000) could take up the position of Generator G. Given a single input context output statement pair (c,s), the teacher forcing loss can be mathematically expressed as
T log (wx(s) (s);e)
Ligle,s)
1
Σ
(1)
T
3 The Adversarial Training Framework
Since Generators trained merely from recovering masked original statements miss out opportunities of exploring other reasonable statements, LogiGAN implements an adversarial mechanism for providing Generators with extra signals based on logical consistency between aseudo-statements (sampled from Eq. 3) and context to encourage explorations. The adversarial framework has two major components: (i) a Verifier V that learns to judge logical consistency between statements and context; (ii) a Generator G that learns both to recover masked original statements, and scores secudo-statement (based on their generative likelihood) in a manner that could reach scoring consensus with the Verifier - The more logically consistent the Verifier thinks of statement w.r.t. the input context, the more likely the Generator is expected to generate the statement under the input context (i.e., assign higher generative likelihood score). The overall objective of LogiGAN can be expressed as the minimax objective:
jo min max Ext~ [log Vo s st]] + E.- Pa===un((1Bo)c.tt
[log(1 Volc, ))} (2)
(we(.) denotes the th token of input string.
3

---

Original Text
All men are mortal and Socrates is ian. Thendfore, Suacaies is mustal.
Veoer (Logical Consistency Score)
Ground-trath Statement
Verifier
Socrates mortal
Pscudo Statements
Context with Logic Indicator men momal, and Socntes man. Therefore. [MASK].
a mortal can never be god
Feedback
Socrates ameng the mortal things
External
Source
Socrales will oveniually die sense, Socrates
Generator
Ketrieved
Sucwr (Generative Likelihood Score)
Gererated
Self-Sampiing
Figure I: LogiGAN Overview. Generator targets to predict the masked-out logical statement and scores candidate statements, while Verifier justifies the logical correctness of statements. The blue path indicates the process where the Generator helps Verifier learning, while the yellow path denotes the process of giving Verifier feedback for Generator training.
where Go and Va denote the Generator and the Verifier with model parameters and Φ, respectively, s* +s/s" represents ground-truth statements from original text sampled seeudo-tatemment We discuss sampling details of secudo-statement later in this section in Eq.
Classical GAN settings (Goodfellow et al. 2014 Zhu et al.) 2017 fall short in sequential generation because the gradient propagation from the Verifier to the Generator is blocked by a non-differentiable beam-search during text generation. Previous approaches such as (Yu al. 2017) address this chal- lenge by token-wise policy gradient estimation via Monte Carlo Search. However, since the sampling run-time grows exponentially with the length of the target sequence, their original implementations are not applicable to million-scale pre-training with relatively longer target length as in our scenario.
Different from them, LogiGAN omits the token- wise Monte Carlo Search for policy gradient timation, and realizes a similar goal via measuring the similarity scoring distributions between Verifier and Generator. The main procedures of LogiGAN several candidate pseudo-statement are sampled on a sentence level; (b) the Verifier assigns the logical consistency scores Vscore based how logical consistent these candidates original context; (c) the Generator assigns the sentence-leve generative likelihood score Gscore to each candidate to reflect how likely it will produce the pseudo-statement under the given context. (d) The similarity between Generator and Verifier score distributions is computed as new signal encourage the Generator reach scoring consensus with the Verifier i.e., the more logically consistent the Verifier thinks of the statement, the higher likelihood score the Generator is expected assign. Since both scoring processes are continuous, the oon-iifferentialee barrier is successfully bypassed. Meanwhile, this design does not involve sequential token-level sampling and decouples the Generator and Verifier scoring processes, thereby enabling parallel score computations. This makes large-scale pre-training with relatively longer target sequence length computationally feasible.
be summarized in four steps: (a)
the
The overall framework overview is illustrated in Fig. I and the detailed training procedure is summarized in Algorithm []] To diversify the candidate pseudo-statements, we sample pseudo- statements from two sources: (i) self-sampling via diversified beam-search from the Generator; (ii) retrieving similar statements from the corpus, and the sampling process can be summarized as:
Preg Go, c+) = (8a ~ Gol- U 83 ~ R(s+)).
where (. c) denotes self-sampled statement Sa given context from Generator Go. and R(s*) denotes retriever" that retrieves textually similar statements 83 with ground-truth statement s from the corpus. Note that this process is conducted separately for the corpus of Verifier and Generator.
'Note: real practice, there a gap between sampled psendo-statements and logically inconsistent statements. We keep current symbolic denotations for simplicity only and discuss this issue in App. B. Any retriever is feasible and adopt BM25 the retrieving method here.

---

Algorithm 1: Adversarial Training Process
Dependencies (()) A Pre-trained Generative Language Model Generator Go
(2) A Pre-trained Discriminative L.anguage Model as Verifier Vo
(3) Generator Source Training Coprus Cg with size M
(4) Verifier Source Training Corpus Cy with size N
(5) Pre-defined Warmup epochs E, max iterations of GAN training Q
(6) Pre-defined training sample size m, n for V. 9 per iteration
Random partition Cg into Cg., Cg. with size Ma. M3;
Go Warmup Ga on Cgo for E epochs with Lif:
3 for in 1:Q do
G + 9:-1;
Ov Cg. + Sample m examples from Cy, and examples from Og., w/o replacement;
Ovn Cg. + Sample pscudo-statement for Cy.. Cg; with G. and BM25, as in Eq.
Vi Train V Cv. for epoch with Lvers in Eq. 4 (Verifier Training)
8
for T in batch Co. ) do
Vecore: Gscore & Va, Gi do scoring on T, in Eq. and 6
Egen X1 (st from 2) X2 DKI(V (score) in Eq.
11
9, +- Update 9, for 1 step with Lgeni (Generator Training)
12
end
13
end
3.1 Training of Verifier
The Verifier serves as a critic to judge whether a statement is logically consistent w.r.t. the context. Therefore, the training task of Verifier be viewed binary classification problem. Pre-trained language models that could perform discriminative classification tasks such as BERT (Devlin et al. 2018 ALBERT (Lan et 2019), and RoBERTa (Liu et al. 2019), will be suitable for the role of Verifier. With
for both ground-truth and logically consistent pseudo-statements, and 0 the binary classification loss for single pair of context/statemeet/labl (c, s,s) y) of Verifier can be mathematically expressed as (omitting average):
for other pseudo-statemen
Leer (c, v, y log Voly [G $)) - (1 y) log(1 - V [c)])),
(4)
3.2 Training of Generator
The Generator targets both recover the original masked statements, and score pseudo-statements based on their generative likelihood in a manner that could reach sentence-level scoring consensus with the Verifier. This corresponds to the two sources learning signals received by the Generator: (i) the original generative objective with teacher forcing loss defined in [q as signal: and (ii) the distribution similarity between sentence-level generative likelihood score assigned by Generator and logic consistency score assigned by Verifier. To achieve the goal of (ii), we first sample pseudo- statements {s] ...., s, from (. Go, c, *²). Then the Verifier assigns logical consistency score Vicore based on how logically consistent the pseudo-statements are w.r.t. the context, expressed as:
Vicare (c: ST s = (o(si Vo(s,, c); Vo(s,,
(5)
After this, the Generator assigns sentence-level sentence-levelgeeeerttie statement reflect how likely the pscudo-statement will be produced under the given context:
generative likelihood score Gsoone for each pseudo-
Sxcore (c: ST s = [[o c); (o(s c); ...; to(s, [c]]
(6)
where (o(s) is the accumulated log ligeelhoddh of the statement conditioned on the context c.
Afterward, each statement with high Verifier score Vo(s, c) is also expected to receive high generative score (s c) to facilitate the Generator's capturing the Verifier's judgment criterion based on logic consistency. KL-divergence (Kullback & Leibler 1951) DxL. is therefore appropriate measure for the similarity between the score distribution Vecore and Gucore. For the purpose of smoothing the gradient to stabilize the GAN training process, we gather both the ground-truth
5

---

Fror and pou INSSC exgh dul sution
pan Mood
2127M
Logle Indicstors
30000
2/5
130
prambe
condia indistory
Length Sterbution ofectuur
50604
40008
1004
2000
20004
P5
Figure 2: Corpus statistics. Histograms on the left side display length of masked statements (bottom) and prev-and-post statement context (top). The right-side pic chart displays indicators' distribution.
(learned with saccher-forcing loss) and pseudo statements (learned with KL loss) inside the same batch w.r.t. single input context c. In our case, there is exactly one ground-truth statement and pseudo-statementt for each input context c. For a batch (c; s+ stt,sj :8 ..... 8 the overall objective of the Generator is defined as (in App. F we show how Eq. 7 commits to the optimization of Eq. 2
Lgen(ess , .T...., s A Lep(e,s*) d2 D DKL(Usorolf ....,, s, Iscoro ...., ....,,)) s,). (7)
.....
4 Experiment Setup
4.1 Datasets
To test the effectiveness of LogiGAN, extensively experiment 12 datasets requiring reasoning via natural language. Specifically, ReClor (Yu et al. 2020). LogiQA (Liu et al. 2021a), Adversarial NLI ANLI, (Nie et al. 2019), focuses especially on abuductive reasoning, HotpotQA (Yang et al. 2018a on multi-hop reasoning, QuoRef (Dasigi al. 2019) on reasoing with co-reference resolution, MuTual (Cui al., 2020), DREAM (Sun et al.
logical reasoning, Tell hy (Lal al. 2021)
2019)). SAMSum (Gliwa et al. 2019) on reasoning in conversational scenarios, and Narrative Ko ciský et al., 2018). RACE (Lai et al. 2017), XSum (Narayan al. 2018) on general verbal reasoning. These datasets make most, if not all, necessary premises for drawing logically consistent conclusions available their provided context, and require few external premises like commonsense or numerical knowledge. Hence, they fit nicely for testing our hypothesis that LogiGAN brings PL.Ms logic ability beyond their intrinsic linguistic ability, which could benefit general reasoning processes.
4.2 Pre-training Corpus
We apply the corpus construction methodology ($ 2 on the widely used BookCorpus Kobayashi 2018). which consists of e-books and movies with topics crawled from general domains. Although some corpus featuring debates and arguments (Walker et al. 2012 Abbott et al., 2016 Swanson et al. 2015) appears to be more suitable for our emphasis on logic, we not elect them due to their high domain specificity in fields such as politics, law, and economics. We discard overly short statements and instances where indicators do not indicate logical reasoning (e.g., "since 2010" indicating time point rather than premises, "so happy" indicating degree of the subsequent adjective rather than conclusions). This results in 3.14 million (1.43 and 1.71 million from conclusion and premise indicators, respectively) instances. Corpus statistics visualized in Fig. 2

---

4.3 Models
Baseline Choice. Since our primary goal of the experiment is to test the effectiveness LogiGAN and test our hypothesis that logic ability can be further enhanced beyond PLMs' intrinsic linguistic ability, we only compare models pre-trained with LogiGAN against their vanilla versions. After LogiGAN pre-training, we discard the auxiliary Verifier (discussed in Sec. 6 and employ the Generator only to solve all downstream tasks in a purely end-to-end manner. For our main experiments, we initialize Generators from both base and large size pre-trained T5 (Raffel et al. (220), and Verifier from pre-trained ALBERT-large (Lan et al.) 2019). We leave discussions of the rest implementation details and syper-parameter settings of pre-training and downstream fine-tuning in Appendix D.
Elastic Search vs. Self Sampling. As stated earlier in section 3.2 candidate pseudo-statement have two possible sources they could either be sampled via beam search from the Generator's 's self- distribution, or could be retrieved from some external resources. We earry out two variant versions of LogiGAN whose Generator is trained purely from self-sampled sentences as pseudo-statements (LogiGAN base (SS)), and from extra pscudo-statements retrieved from corpus by Elastic Search Gormley & Tong (0155) (LogiGAN (S+tes) For the large model, we use LogiGAN large (es+ss) as default. Our database consists of 3.14 million sentences discovered by the corpus construction process, and we keep the top-5 similar retrieved sentences along with self- samples from Generator.
5 Experiments
5.1 Experimental Results
Table 1: Main results of LogiGAN on 12 downstream tasks (development sets).
Multiple Choice & Classification Datasets
Models/ / Dataset
ReClor
LogiQA Abc
RACE
DREAM
ANLI
MuTual
Avg.
Metrics
Acc
Acc
Acc
Vanilla desc
35.20
27.19
63.89
59.36
44.10
67.38
49.52
LogiGAN Sasc (ss)
40.20
34.72
67.13
63.38
49.50
69.41
54.06
LogiGAN base (ss+es)
40.00
37.02
67.27
63.73
19.70
69.98
51.62
Vanilla T5 targe 1 LogiGAA farge
50.40
38.56
78.99
78.98
58.00
76.41
63.56
54.80
40.55
80.67
81.42
63.50
77.88
66.47
Generation Datasets
Models Dataset
QuoRef
HotpotQA EM/F;
NarrativeQA Rougez.
TellMeWhy Rouges
SAMSum
XSum Rougez
Avg.
Metries
EM/F;
Rouges
Vanilla T5
70.76/ 74.58
61.11/ 74.86
48.11
30.03
39.32
29.14
36.65
LogiGAN base (8S)
75.02/78.
62.687 78.
49.44
39.92
30.26
37.70
LogiGAN (ss+es)
74.94/78.40
62.80/ 76.
49.46
31.15
40.21
30.27
37.77
Vanilla T5
80.06/ 83.25
G6.11 79.80
51.09
31.42
41.40
31.58
38.87
LogiGAN Gorge
81.92/ 85.25
67.04/ 80.36
51.79
32.72
43.13
33.49
40.28
As presented in Table both base and large size PL.Ms further pre-trained with LogiGAN surpass their vanilla baselines across both discriminative and generative task formats, through wide scope of downstream tasks requiring general reasoning abilities. We can make the following observations: Among all observed improvements, those on tasks with particular emphasis on logic (ReClor, LogiQA, and ANLI) are most noticeable. These positive results manifest the effectiveness of LogiGAN in injecting logic ability into PLMs, while testifying to our primary hypothesis that logic ability is fundamental to general reasoning well. This conclusion answers the two questions in the intro section D suggesting that randomized MLM pre- training might fall short in endowing language models with logic ability, and a logic-targeted pre-training approach like LogiGAN may further assist logic learning beyond language acquisition. Furthermore, extra retrieved pseudo-statements (ss+es) bring some additional performance improvement compared with the pure self-sampling (ss) LogiGAN variant, revealing the important role of pseudo-statements' diversity in adversarial training.
5.2 Ablation Study and Analysis
Observing the apparent performance enhancement, we now aim at pinpointing the truly functional components of LogiGAN through ablation studies and deriving the origins of observed improvements.
"Is logic ability obtained for free from MLM? Could it be further learned beyond language acquisition?
7

---

For fair comparison purposes, we hold all pre- trainini and downstream settings (including hyper- parameters, implementation designs, and evaluations) unchanged from full LogiGAN. All variations are initialized from T5base, and we report performance variance on 7 datasets.
Table 2:
Ablation Results on 7 datasets. The last column shows average performance variance, along with relative percentage improvement against vanilla T5base as the baseline.
Models/ Dataset
ReClor
LogiQA Acc
RACE
DREAM
ANLI
QuoRef EM/F,
NarrativeQA Rouges
Metrics
Asc
Acc
Ace
Average
Vanilla T5 luse
35.20
27.19
63.80
59.36
44.10
70.76/74.58
48.11
49.80 (0.0%)
LogiGAN base (8s+es)
40.00
37.02
67.27
63.73
49.70
74.94/78.40
49.46
(4.59
Random Sentence
36.00
30.56
11.26
58.15
45.40
70.96/74.30
48.38
50. 50.10 (0:1001:0000 (069))
11. MLE Logic Pre Irain
38.80
35.02
64.55
61.71
46.00
73.61 /6.96
49.30
(4.71(4
III. Iterative Multi-task
37.20
34.25
64.01
62.06
46.20
71.67/75.14
49.15
52.08 (4.69)
I. Random Masked Sentence Prediction Pre-training. To explain the observed improvements, our
first hypothesis is: Models harvest extra linguistic ability from masked statement prediction compared with masked token (or span) prediction. Quite intuitively, filling entire sentences with complete subject-predicatt structures might put additional demands on models to capture more abundant syntactic information beyond the coverage of masked token (or span) prediction. Since LogiGAN involves recovering masked sentences, it is then necessary to determine to what degree, if any, that the observed performance gain is attributable to models' plausible linguistic ability improvement. We therefore carry out a variant pre- pre-training where the prediction objects are randomly masked sentence.
Results (shown in Table 2 displays that masked sentence prediction training barely brings im- provement against the vanilla baseline. This suggests it is unlikely that masked sentence prediction empowers PL.M trained from masked token prediction significantly better linguistic ability, nor likely that the extra pre-trainne pre-training corpus per se significantly raises the performance. Therefore, we reject the first hypothesis and conclude that observed improvements should derive from somewhere else.
II. MLE-only Logic Pre-training. Our second hypothesis is that logic-guided, masked statement prediction enhances models' intrinsic ability of logical reasoning, thereby lifting the downstream performance. Having addressed the potential impact of learning randomized complete sentence generation, we next aim to check how learning logic-targeted statement generation affects models' behavior. We ablate the entire adversarial training process, and train models to perform maximum likelihood estimation (MLF) with teacher-forcing loss only on masked-out logical statements.
Results 2 of MLE-only logic pre-training reveals quite a notable improvement across almost all datasets against both vanilla baseline and I., suggesting that learning to generate logical statements indeed injects extra abilities into the model. Since results of eliminate the possibility that models harvest stronger linguistic abilities from complete sentence prediction, it is safe to partially ascribe the better downstream performance This reveals the relative orthogonality between logic ability and models' inherent linguistic ability, suggesting that logic ability could be enhanced through further Dogic-targeted pre-training.
models' enhanced ability in modeling logical reasoning.
I. Iterative Multi-task Pre-training. Since II. only partially explains the observed improvements, here is our last hypothesis: the adversarial training procedure of LogiGAN explains the unexplained rest part beyond the coverage of II. Here a multi-task pre-training with both generation and verification tasks will be the most natural intermediate setting between the single-model generation-only setting of II. and LogiGAN's dual-model adversarial setting. However, since the verification task relies on Generator's self-sampil sampled statements, we adopt an iterative self-critic pre-training manner following Nijkamp et al. (2021). J. Unlike typical multi- tasking training that simultaneously carries different tasks and then sums the losses, our generation and verification tasks happen alternately
Surprisingly, the iterative multi-task pre-training barely brings any positive effects to models' down- stream performance compared with II. One possible explanation for this might be that the drastically different mechanisms between the verification and generations task intervene with each other, making the single-model & multi-task setting non-beneficial. Now that we have confirmed that an extra verification task fails to explain the rest improvement, we can accept our final hypothesis and conclude that it is indeed the adversarial mechanism between the Generator and Verifier that truly facilitate learning of logical reasoning, thereby further improving the downstream performance beyond II.
Veriffcattinn is formulated as a generation task model outputs natural language token "good" and "bad".
8

---

6 Discussion
Psycholinguistic Interpretation of Logic-oriented MLM From the first glance, the idea of Logic-oriented MLM seems to be naive and simply. However, we argue that to fully appreciate the value and potential of logic-oriented MLM, going beyond the superficial appearance of masked text and touching down to their underlying psycholinguistic essence is necessary.
It is neither the linguistic patterns of the masked-out text, nor the masking technique to corrupt them that makes logic-oriented MLM (and its potential follow-ups) unique. What truly matters is the distinctive cognitive processes proceeding in the minds of writers when they put down different pieces of text - language is a window into human minds (Pinker 2007).
Consider the following examples when human is filling in the [MASK]'s:
(I) "19 + 69 = [MASK]." (Numerical cognition).
(2) "Windows is founded by [MASK]." (Declarative memory retrieval).
(3) "Socrates is a mortal, so he will eventually [MASK]" (Logical reasoning).
(4) "If I feed my dog more than 2 treats per day, it will get [MASK]." (Causal inference).
(5) "A crow immediately stands out in swans because it's [MASK]." (Common sense reasoning).
(6) "Mike deeply bows to his teacher to show his [MASK]." (Social perception).
Though answers to these [MASK]'s ss are similar in string length, they nevertheless involve different information pathways and substantially distinctive cognitive processes in writers' minds.
Logic-oriented MLM shines in that it consistently captures exactly one type of such cognitive process, and trains LMs to model humans' logic reasoning mechanism, which well beyond modeling language per se. On the contrary. Randomized MLM does not capture consistent underlying cognitive processes, which could significantly lower LMs' efficiency of learning advanced intelligence mechanisms beyond language itself. The empirical effectiveness of Logic-oriented MLM provides positive evidence for the practicability of this paradigm, suggesting that LMs might be able to learn various advanced human cognitive processes other than logical reasoning via similar approach. Combined with LogiGAN's natural analogy of humans' lcarning-thinkee mechanism during logic development, LogiGAN made an encouraging attempt unify cognitive modeling and language model pre-training.
Adversarial Training Might Assist Downstream Generation Tasks. Although in our experi- ments, we discard the Verifier and solve downstream tasks with the Generator only, some previous works (Shen et al. 202T Cobbe et al.) (02T) reveal that the Verifier can be used for ranking multiple generation results, thereby effectively enhancing overall downstream accuracy. However, in their paradigm, the information propagates unidirectionally from the Generator to the Verifier, and the Generator cannot directly benefit from the Verifier's 's discriminative feedback. In contrast, our Logi- GAN adversarial training paradigm surmounts the non-differentiable obstacle and could potentially enlighten a new paradigm of both pre- veeitrain and downstream fine-tuning.
Improving Logical Pre-training. Our paper demonstrates that PLMS' logic ability can be further enhanced beyond their inherent linguistic ability, and adversarial training may bring extra benefits beyond the learning of logic-targeted masked statement prediction. However, our heuristic- approach to identifying logical phenomena in text corpus and the single mask prediction setting can be further improved. Logic recognition methods with higher recall and better unsupervised task designs (e.g., logical indicator prediction, or logic-guided sentence shuffling) are worthwhile to explore in the further work. Besides, since we are adopting a general domain pre-training corpus (i.e., BookCorpus) with bare emphasis on logic, understanding the impacts of extending pre-training the domain-specinn specific corpus (c.g., law corpus) or others emphasizing logical reasoning is also substantial.
7
Related Works
Generative Adversarial Training in NLP. Unlike conventional GAN (Goodfellow al., 2014
Mirza & Osindero, 2014 Zhu et al. 2017) that generates continuous output such as images, sequential

---

GAN generates discrete sequences via con-differential searches. This makes feedback from the discriminator not propagatable to the generator. To tackle this challenge. SeqGAN (Yu al. 2017) borrows an idea from reinforcement learning, treating each output token as a single action, and estimates token-wise policy gradient via Monte Carlo search. RankGAN (Lin et al. 2017) adopts a similar approach but breaks the cassiiccassiiiictionn assumption of discriminator task design, and ranker provides feedback to the generator. Their generator attempts to generate verisimilar sentences to deceive the ranker into ranking synthetic sentences higher over multiple human-w -wiiten ones. In our scenario, however, the gold ranking is hard to determine because measuring which statements are more logically consistent w.r.t. context than others is non-trivial, and multi-gold cases are possible. While successfully enabling communication between generator and discriminator, the original designs of SegGAN, RankGAN, as well as other works such as (Guo al. 2017: Fedus et al. 2018 Caccia
et al. 2018 Rekabdar et al. 2019). generally formulate text generation as sequential action decision problem, thereby involving heavy sampling for policy gradient estimation, and are sensitive the
length of the target sequence. Since large-scale pre-training (with arbitrary target length) puts a high demand on scalability and computational efficiency, the above approaches are not readily applicable in our scenario. Furthermore, previous work leverages adversarial training to improve qualities of generated examples, whereas our focus is on enhancing models' intrinsic logic ability.
recent work, AR2 (Zhang et al. 2021), leverages adversarial training to improve dense document retrieval. With a retriever-ranker architecture, the learning objective of retriever is to maximize the agreeableness between its own score assignment and that of the ranker for input documents. This is conceptually similar to LogiGAN,
Generator also aims at reaching consensus with Verifier. However, AR2 does not fall into the sequential GAN paradigm, since it does not involve any sequential text generation, and there is no non-differentiable barrier between the retriever and ranker.
Pre-training for Reasoning Ability Improvement. Previous works have extensively investigated the possibility of injecting specific type of reasoning via pre-training, such as numerical (Geva et al.
2020; Yoran et al. 2021; Pi et al.] (022), commonsense (Zhong et al. [019 Tamborrino et al. 2020
Staliunaite et al. (022)) I). formal logic (Wang et al. 2021 Pi et al. 2022), , multi-hop (Deng et al. 2021
Zhong et | 2022). and tabular (Liu et al. b)2 reasoning. Different from them, LogiGAN focuses on logic reasoning, which plays a fundamental role in general reasoning via natural language.
8 Conclusion
In this work, we hypothesize that (i) logic ability plays a key role in a wide scope of tasks requiring general reasoning; and (ii) PLMs' logic ability can be further improved beyond their original linguistic ability. We correspondingly propose LogiGAN, an unsupervised adversarial pre-training framework for logical reasoning enhancement. LogiGAN cireumvents the non-differentiable challenge of sequential GAN via a novel Generator- scoring consensus mechanism, and enables large- scale pre-training with longer target length. Extensive experiments and ablation studies reveal the effectiveness and functional components of LogiGAN, providing evidence to our major hypothesis.
References
Rob Abbott, Brian Ecker, Pranav Anand, and Marilyn A. Walker. Internet argument corpus 2.0: An sql schema for dialogic social media and the corpora to go with it. In LREC, 2016.
David Boud, Rosemary Keogh, and David Walker. Reflection: Turning experience into learning. Routledge, 2013.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165. 2020. WTRL.httts:/// /ttp::///axxvvvve wtw.:oog////aoaooooeoe http:::///r.....og///////.. 14165
Sasa Buvac and Ian A Mason. Propositional logic of context. In AAAI, pp. 412-419, 1993.
10

---

Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. CoRR, abs/1811.02544 2018. URL http://arxiv. org/ abs/ 1811 02549
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1026-1036, Online, November 2020. Association for Computational Linguistics. doi: 10.1 86533v11/2020 1665/////2022.nnnngg-mmmlppppp
URL
https://acllnttholgg. org. 22200.020..00 findings-emnlp 91
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. Mutual: dataset for multi-turn dia- logue reasoning. CoRR, abs/2004.04494, 2020. URL httts::/aarae hrrg/abs/2004... 04494
Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In EMNLP, 2019.
Xiang Deng, Yu Su, Alyssa Lees, You Wu, Cong Yu, and Huan Sun. ReasonBERT: Pre-trained to reason with distant supervision. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 6112-6127, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclantholoogy htmp2222.emmmmpmmmia.. 494
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http::/arxivv http://////bb///r/ra// 1100//801100000555
Giada Di Stefano, Francesca Gino, Gary P Pisano, and Bradley R Staats. Making experience count: The role of reflection in individual learning. Harvard Business School NOM Unit Working Paper, (14-093):14-093, 2016.
Y. Dote. Introduction to fuzzy logic. In Proceedings of IECON '95 21st Annual Conference on IEEE Industrial Electronics, volume 1, pp. 50-56 vol. 1995. doi:
Igor Douven. Abduction. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2021 edition, 2021.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019.
William Fedus, Ian Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the, 2018. URL http::tttt/rrrttti org/ abs/ 1801.
Firth. A synopsis of linguistic theory 1930-1955. In Studies in Linguistic Analysis. Philological Society, Oxford, 1957. reprinted in Palmer, F. (ed. 1968) Selected Papers of J. R. Firth, Longman, Harlow.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numericaa reasoning skills into language models. CoRR, abs/2004.04433 2020. URL httpt:://ssssiiv../. org/ hts///b//2000 04487
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 1 1/ [tt. org/ D19-5409]
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL. https
// //arxiv.. org/ absaaaa///4/ talss/146..20 2661
11

---

Clinton Gormley and Zachary J. Tong. Elasticsearch: The definitive guide. 2015.
Leo Groarke, Informal Logic. In Edward Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2021 edition, 2021.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation 2017. URL http
via adversarial training with leaked information. CoRR,
7/ //arxiv... org/ abs/1709.... 08624
Chadi Helwe, Chloé Clavel, and Fabian M Suchanek. Reasoning with transformer-basee mod- els: Deep learning, but shallow reasoning. In 3rd Conference on Automated Knowledge Base Construction, 2021.
Patrick Hurley. A Concise Introduction to Logic. Belmont, CA, USA: Wadsworth, 1982.
Nora Kassner and Hinrich Schütze. Negated LAMA: birds cannot fly. CoRR, 0tbs/1911.0034333 2019. URL http://arxiv.oree wttp://ww..g//////111...1 03343
Brianna L Kennedy and Robert Thomberg. Deduction, induction, and abduction. The SAGE handbook of qualitative data collection, pp- 49-64, 2018.
Sosuke Kobayashi. Homemade bookcorpus. httpp://githhb..c////CLLLLLLLL 2018.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-866 1951.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv: 1ee..440443... 2017.
Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasuhramanian. Tellme- why: A dataset for answering why-questions in narratives. CoRR, abs/2106.0611113 2021. 2021. URL
sctps://arxxv.orr/abb...o www.o/rggg.....o..e////r// 06132
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909. 2019. http::tta/a/x...orr//////////
11942
Kevin Lin, Dianqi Li, Xiaodong He, Ming-Ting Sun, and Zhengyou Zhang. Adversarial ranking for language generation. In NIPS, 2017.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCAI'0, 2021a. ISBN 9780999241165.
Qian Liu, Bei Chen, Jiaqi Guo. Zeqi Lin, and Jian- Guang Lou. TAPEX: table pre-training via learning a neural SQL executor. CoRR, abs/2107.07653, 2021b. URL https:://arrtiii http://aavvvv... www.aiiww...........oo....ooo hatg//bs/2111 2107 07653
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019. http/ttp::/// /tttp://arxiv
11692
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abs/1411.1784, arg/abs//a arg/abs//411... 1411, 1784
2014. URL UtL::tt:://arr
Jennifer Moon. Reflection in learning and professional development: Theory and practice. Routledge, 2013.
Ronald Munson. The Way of Words an Informal Logic. Boston, MA, USA: Houghton Mifflin School, 1976.
12

---

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. CoRR, 1et111900115999 2019. URL
httt:://axivt org/ abs/ 1910. 14599
Erik Nijkamp, Bo Pang. Ying Nian Wu, and Caiming Xiong. SCRIPT: Self-critic PreTraiiiig of transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5196-5202, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18553/v1/2021 naacl-main. mail 409. URL https: httpp///aaaaaahhoooooooo org/ 2021 naacl main. 409
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, and Weizhu Chen. Reasoning like program executors. CoRR, abs/2201.11473, 2022. URL
11473
S. Pinker. The Stuff of Thoughu: Lunguage Window Into Human Nature. Viking, 2007. ISBN 9780670063277. URL httthtttp///oooooo google. http//bookssiddjyllTrTTS
G. Priest. An Introduction to Non-Classical Logic: From If to Is. Cambridge Introductions to Philosophy. Cambridge University Press, 2008. ISBN 9781 139469678. URL https://books google. com//ookss7iddrrXWWooboo.o
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http://jmlr axggpapprss/v212200 074 html
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250, 2016. htRp:/ttt///rxii.or/////
1606.05250
Jo Reichertz. 4.3 abduction, deduction and induction in qualitative research. Companion to, pp. 159, 2004.
Jo Reichertz. Abduction: The logic of discovery of grounded theory. Sage London, 2007.
Jo Reichertz. Induction, deduction. The SAGE handbook of qualitative data analysis, pp. 123-135. 2013.
Banafsheh Rekabdar, Christos Mousas, and Bidyut Gupta. Generative adversarial network with policy gradient for summarization. 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pp. 204-207, 2019.
Tomáš Ko cisky. Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeAA reading comprehension challenge. Transactions of the Association for Computational Linguistics, TBD:TBD, 2018. https/sts:///B)
Maarten Sap. Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. Commonsense reasoning for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: Tutorial Abstracts, pp. 27-33, Online, July 2020. Association for Computational Linguistics. doi: 10. act53////22200//ttttuorii https://aclanthhllgg .orgooyyyoooroooooooooooo 2020 acl tutorials.7
URL.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu, Generate & rank: A multi-task framework for math word problems. CoRR, abs/2100.03034, 2021. URL
https://arxiv.orgg/ 2aabs//10 /2009. 03034
leva Staliunaite, Philip John Gorinski, and Ignacio lacobacci. Improving commonsense causal reasoning by adversarial training and data augmentation. CoRR, 101.0996,, 2021. URL
https://arxiv. wwtp:/o/////////// att///bs/////.////////////aaaa/t 04966
13

---

Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge dataset and models for dialogue-basee based reading comprehension. Transactions of the Association for Computational Linguistics, 2019. URL T /ettps://rxiv.... http:///.wvvv....aaabbassa wwwaas///9...... 00164v1...vvvv
Reid Swanson, Brian Ecker, and Marilyn Walker. Argument mining: Extracting arguments from online dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group on
Discourse and Dialogue, pp. 217-226, Prague, Czech Republic, September 2015. Association for Computational Linguistics. doi: 10.18653/11/11554400
org/
W13-
Alexandre Tamborrino, Nicola Pellicano, Baptiste Pannier, Pascal Voitot, and Louise Naudin. Pre-training is (almost) all you need: An application to commonsense reasoning. CoRR, abs/2004. 2020. URL tanttps://axiiv . org/ abs 14074
Marilyn Walker, Jean Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. A corpus for research on deliberation and debate. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pp. 812-817, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL http: /ttt:://www... http://wwwwwwwwwcc..o.o/oeooe..t conf org. org/proceedings/ 1rec2012/pdf/ http:ppddddo0__ppppapra .df
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR,
abs/ has//804.074611, 2018. URL
http:/iww..rrrbb//////ea///e/// 1804. 07461
Siyuan Wang. Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. Logic-driven context extension and data augmentation for logical reasoning of text. arXiv preprint arXiv:2105.03653 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. CoRR, Abs/1809.0960M 2018a. UtL/attp://arxi..oom org/ abe//80ss.//.//./////// 09600
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D.. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018b.
Ori Yoran, Alon Talmor, and Jonathan Berant. Turning tables: Generating examples from semi- structured tables for endowing language models with reasoning skills. CoRR, abs/2107.07261. 2021. / org/ abs////////0 07261
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty- AAAI Conference on Artificial Intelligence, AAAI'17, pp. 2852-2858. AAAI Press, 2017.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. Adversarial retriever-ranker for dense text retrieval. CoRR, abs/2110.03611,, 2021. URL https://arxiv org/ 03611
Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. Improving question answering by commonsense-basel pre-training. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 16-28. Springer, 2019.
Wanjun Zhong, Junjie Huang, Qian Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. Reasoning over hybrid chain for table-and-text open domain qa. arXiv preprint arXiv:2201. 008800 2022.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-imag translation using cycle-consistent adversarial networks. CoRR, abs//703. 10593, 2017. ( org/ abs/
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We will release the code and data upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "zripqZxaxL2", "reviewer": "Reviewer_1vTP", "summary": "The paper presents a LogiGAN, a generative adversarial approach inducing logic reasoning language models. The core claims made in this paper are: (a) logic helps with reasoning, (b) adversarial pre-training improves performance on NLP tasks requiring reasoning, (c) Generator-Verifier mechanism enables large scale pre-training with arbitrary target length. The claims are supported by experiments over 12 datasets and an ablation study.", "questions": "- 330-331: What is the difference between 'qualities of generated examples' and 'intrinsic logic ability'?\n- What is 'x' in equation 1? Is it supposed to be 'c'?", "limitations": "The authors have not mentioned either. The work does not seem to pose any threat to society beyond those inherent to a large language model.", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has an original approach to GAN for language models that demonstrates improvements of results. The quality of the paper is in its evaluation on 12 downstream tasks, although it would be nice to see a comparison with SOTA on the given task for reference.\n\nStrengths:\n- The paper is well written and presents consistent improvements over all experiments.\n- Novel way of circumventing the non-differentiability issue of SeqGAN\n\nWeaknesses:\n- The paper claims that their method allows for large scale pre-training with arbitrary target length. This is not clear to me since (a) the verifier is based on a ALBERT-large and therefore, as far as I understand, is still limited to 512 tokens, (b) I would expect a more detailed explanation as well as empirical results of how much more efficient the LogiGAN is compared to sequential methods to substantiate this claim as a contribution.\n- The paper claims that linguistic and logic ability is orthogonal. However, the evidence for this comes from Ablation I, where the model is trained to predict random masked sentences. I do not understand how this demonstrates that linguistic ability does not play a role in improving the model performance. The random selection of masked sentences surely removes logical and linguistic merit equally. Furthermore, in Ablation II, it is showed that the model does considerably improve with MLE pre-training.\n\nFormatting: \n- Figure 2 is hard to read in print\n- Table 1 has a typo in the title (devleopment setsjudgement).\n- paragraph names should end with a period\n- 75- GAN posed (GAN put*)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "v25wZ5dJ88_", "reviewer": "Reviewer_aP9K", "summary": "This paper proposes an unsupervised adversarial pre-training framework called LogiGAN to improve the ability of logical reasoning in pre-trained models. The authors adopt heuristic rules to detect logic indicators and replace the vanilla masked language modeling with masked logical statement prediction. To explore more logically reasonable statements, the authors devise a GAN-like training process which iteratively trains the generator and verifier. Experimental results on 12 datasets show the superior performance of LogiGAN.", "questions": "I have included my questions in the weaknesses part.", "limitations": "The authors have not adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The motivation to improve the logical reasoning abilities of pre-trained models is intuitive. The proposed method directly supports the motivation.\n2) Experimental results show that LogiGAN can outperform T5 on 12 datasets.\n3) This paper is overall well-written and easy to follow.\n\nWeaknesses:\n1) The novelty of the proposed method is somewhat limited. In my view, the main contribution of this paper includes the pre-training task of masked logical statement prediction and the adversarial training framework. However, masked logical statement prediction seems like a directly improved version of masked language modeling based on logic indicators. And the adversarial training framework which avoids Monte Carlo search and adopts the sentence-level training loss has also been deeply studied in the community of text GANs, such as [1] and [2]. Thus, I feel that the contribution of this paper is mainly on the applicational side.\n2) The sampling of pseudo-statements plays an important role in the training of verifier. I wonder why self-sampling and retrieving similar statements based on BM25 can provide high-quality negative samples to benefit the verifier. From my perspective, the negative samples should be logically inconsistent, which are not intuitively reached by these two methods. The authors say that they use a NLI model to assign labels in the appendix. But this only affects the label assignment without improving the quality of negative samples.\n3) The authors mask the task-specific texts (which are related to logical reasoning) in the pre-training task, which may degrade the generalization ability of the pre-trained models to general NLP tasks. Other similar works which enhance the pre-trained models with specific abilities mostly conduct experiments on general benchmarks such as GLUE in addition to task-specific datasets, aiming to show that their method can maintain the generalization ability of pre-trained models. I suggest that the authors should also do this to make readers understand the ability of LogiGAN comprehensively.\n\n\n[1] Maximum-Likelihood Augmented Discrete Generative Adversarial Networks.\n\n[2] ARAML: A Stable Adversarial Training Framework for Text Generation. EMNLP 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pyinWqZ72hC", "reviewer": "Reviewer_P5x5", "summary": "This paper proposed LogiGAN. It circumvents the non-differentiable challenge of sequential GAN via a novel Generator-Verifier scoring consensus mechanism, and enables large scale pre-training with arbitrary target length. Extensive experiments and ablation studies reveal the effectiveness and functional components of LogiGAN. Experiments on 12 datasets requiring general reasoning abilities, revealing the fundamental role of logic in broad reasoning, as well as the effectiveness of LogiGAN.", "questions": "1. Are the vanilla T5 base model's results from your implementation or they are reported by some other people?", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper is well organized and easy to follow.\n\n2. The motivation of this paper is quite attractive. It firstly hypothesizes that (i) logic ability plays a key role in a wide scope of tasks requiring general reasoning; and (ii) PLMs’ logic ability can be further improved beyond their original linguistic ability. Upon automatic identification of logical reasoning phenomena in massive text corpus via detection heuristics, they train language models to predict the masked-out logical statements.\n\n3. The experiments are solid. One concern here is the numbers in table 1 are kind of close. Is that possible to make a significant test for the improvement? Also it will be better to have a column for avg performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HWCprmqgHNd", "reviewer": "Reviewer_n4Wt", "summary": "This paper aims at improving the logical reasoning ability for language models via a pre-training architecture LogiGAN. The adversarial framework adopts a Generator-Verifier style. The Generator learns to recover the masked statements instead of random tokens/spans in traditional pre-training settings. It also learns to score candidate statements to reach a scoring consensus with the Verifier. The verifier, on the other hand, learns to judge the logical consistency between the context and the predicted statement. Additionally, in this way, LogiGAN overcomes the challenge of the non-differentiability of sequential GAN. LogiGAN is verified on 12 datasets designed to test reasoning abilities and surpasses the baseline model.\n\nThe key contribution of this paper, as I understand, is to leverage sententious logical relations in pre-training to enhance the reasoning abilities for language models.\n", "questions": "1. How to ensure that the masked-out statements “are declarative and have a complete subject and predicate structures and are capable of being factually true of false”?\n2. How to ensure that the Detection Heuristics indeed reflect ‘conclusion’ or ‘premise’? For example, `so` in “so happy” could be just an adverb indicating degrees. \n3. What are the statistics of parameters for LogiGAN? This could be important for a fair comparison. \n", "limitations": "The authors provide an insightful `Discussion` section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. This paper is overall well-written with clear motivations and good structures.\n2. The research topic is interesting, i.e., improving general reasoning ability for language models via taking Logic as an entry point. \n3. The proposed LogiGAN is technically sound and surpasses the baseline models in a broad range of reasoning datasets with different kinds of task settings.\n4. The ablation studies are nice.\n\n**Weaknesses**:\n1. The major weakness of this paper is, in my eyes, the superficial modeling of logical relations. Deductive, Inductive, and Abductive reasoning types are only manifestations of logical reasoning, they do not touch down to the essence. Based on this classification, this paper only models the consequential relations in a sentence-level style, which ignores a broad range of other logical reasoning expressions such as negation, disjunction, and conjunction. \n2. The so-called automatic identification of logical reasoning phenomena is in fact requires many manual labors. The authors have to pre-define words/expressions that indicate conclusion or premises, which greatly limits the universality of the proposed approach.\n3. The authors only report the experimental results on the development set, which made the results less convincing. Also, the performance improvement over generation datasets is less noticeable than in classification datasets. It's hard to know that the model is really behaving as advertised rather than just adding capacity on top of the baselines.\n5. In the related work section, especially `Pre-training for Reasoning Ability Improvement`, the authors describe several papers of other relation types that LogiGAN is not compared with. This makes it difficult to place this work in comparison with prior literature and understand the novelty of the proposed framework. Also, the key contribution of this paper could be blurred with improvement over GAN-like architecture rather than pre-training tasks or something else. \n6. I do not see any open-source statements or code submissions in the supplementary materials, so I take issues with the reproducibility of this paper.\n\n**Minor Issues**:\n1. Line 19: “Learning without thought is labor lost” -> “learning without thinking is labor lost”\n2. Caption of table 1: “development setsjudgement” -> “development set judgment”\n\nOverall, I'm borderline with this paper. I think the paper tests three aspects: 1) GAN-style training in language models. 2) predict the masked-out logical statements as the pre-training task. 3) detection heuristics for logical reasoning phenomena. It certainly demonstrates great experimental results compared with vanilla T5 baselines. But none of them **alone** is novel enough. The most important part in my eyes, leveraging logical reasoning phenomena for pre-training, is after all a sententious logical relation extraction, whose effectiveness is already demonstrated in [1].\n\n[1] Huang, Yinya, et al. \"DAGN: Discourse-aware graph network for logical reasoning.\" NAACL 2021.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept