QA-NatVer: Question Answering for Natural Logic-based Fact Verification
Authors: Rami Aly
Marek Strong
Andreas Vlachos
Venue: EMNLP 2023 Main (2023)
Keywords: fact-checking
fact extraction and verification
claim verification
natural logic
natural language inference
faithfulness
TL;DR: We propose to use question answering to predict natural logic operators in natural logic proofs for fact verification to benefit domains without substantial resources.

Abstract: Fact verification systems assess a claim's veracity based on evidence. An important consideration in designing them is faithfulness, i.e. generating explanations that accurately reflect the reasoning of the model. Recent works have focused on natural logic, which operates directly on natural language by capturing the semantic relation of spans between an aligned claim with its evidence via set-theoretic operators. However, these approaches rely on substantial resources for training, which are only available for high-resource languages. To this end, we propose to use question answering to predict natural logic operators, taking advantage of the generalization capabilities of instruction-tuned language models. Thus, we obviate the need for annotated training data while still relying on a deterministic inference system. In a few-shot setting on FEVER, our approach outperforms the best baseline by 4.3 accuracy points, including a state-of-the-art pre-trained seq2seq natural logic system, as well as a state-of-the-art prompt-based classifier. Our system demonstrates its robustness and portability, achieving competitive performance on a counterfactual dataset and surpassing all approaches without further annotation on a Danish verification dataset. A human evaluation indicates that our approach produces more plausible proofs with fewer erroneous natural logic operators than previous natural logic-based systems.

DA-NatVer: Question Answering for Natural Logic-based Fact Verification
Rami Aly, Marek Strong, Andreas Vlachos
Department of Computer Science and Technology
University of Cambridge (rmya2,ms2518.. a,a cam.ac.uk
Abstract
have been made improve their explainability, for instance via highlighting salient parts of the evi- dence (Popat et al., 2018), or generating summaries (Kotonya and Toni, 2020), there is no guarantee that the explanations are faithful, i.c. that they ac- curately reflect the reasoning of the model (Jacovi and Goldberg, 2020; Atanasova et al., 2023).
Fact verification systems assess claim's verac- ity based on evidence. An important considera- tion in designing them faithfulness. i.c. gen- erating explanations that accurately reflect the reasoning of the model. Recent works have fo- cused natural logic, which operates directly on natural language by capturing the seman- tic relation spans between an aligned claim with its evidence via set- set-theoretic operators. However, these approaches rely on substantial resources for training, which are only available for high-resource languages. To this end, propose to use question answering to predict natural logic operators, taking advantage of the generalization capabilities of instruction -uned language models. Thus, we obviate the need for annotated training data while still relying a deterministi sticiiitiiiii inference system. In few-shot setting on FEVER, our approach outperforms the best baseline by 4.3 accuracy points, includ- ing a state-of-the-art pre-trained seq2seq nat- ural logic system, as well state-of-the of-the-art prompt-based classifier. Our system demon- strates its robustness and portability, achiev- ing competitive performance on counterfac- tual dataset and surpassing all approaches with- out further annotation on a Danish verification dataset. A human evaluation indicates that approach produces more plausible proofs with fewer erroneous natural logic operators than previous natural logic-based systems.
Contrarily, proof systems like NaturalLI (An- geli and Manning. 2014), perform natural logic inference as proofs, and are faithful by design. Their transparent reasoning empowers actors make informed decisions about whether to trust the model and which parts of the prediction to dispute (Jacovi and Goldberg, 2021). Recently Krishna et al. (2022) constructed a natural logic theorem prover for claim verification, using an autoregres- sive formulation with constrained decoding. How- ever, important limitation of this approach is its dependence on substantial resources for train- ing, relying on large datasets for claim verifica- tion, and structured knowledge bases like the Para- Phrase DataBase (Ganitkevitch et al., 2013), Word- Net (Miller, 1994), and Wikidata (Vrandečić and Krötzsch, 2014). However such manually curated resources are typically accessible for high-resource languages, thus limiting its applicability.
To this end, we propose QA-NatVer: Question Answering for Natural Logic-based Fact Verification, a natural logic inference system that composes a proof by casting natural logic into a question answering framework. As illustrated in Figure 1, a proof is a sequence of steps, with each step describing the semantic relation between a claim span and an evidence span via set-theoretic natural logic operator (NatOp), and this sequence of NatOps determines the veracity of the claim following a deterministic finite state automaton (DFA). QA-Nat predicts NatOps using operator-specific boolean questions (cf. Table for examples for all operators). For instance, the relation between the claim span
1 Introduction
Automated fact verification is concerned with the task of identifying whether a factual statement is true, with the goal of improving digital literacy (Vlachos and Riedel, 2014) 40. A typical fact verifi- cation system consists of an evidence retrieval and claim verification component (i.e. the judgement of whether claim is true). The latter is typically implemented as a neural entailment system (Guo et al., 2022, inter alia), which is not transparent in regards to its underlying reasoning. While efforts

---

Anne Rice born in New Jersey Evidence: Born in New Orleans, Rice spent much of early [...]
Claim:
start
S
Claim is Supported
R
Claim is Refuted
Is Anne Rice peraphrase Rice? Yes.
N
Not enough information
Rice was born in New Jersey
Any
N
5.5
born paraphrase of Bor?? Yes.
start
Rice Born in New Jersay
S
J
Does New Jersey entail New Orleans? No.
Does New Jersey exclude New Orleans? Yes. _1
Rice Born in New Orleans
Deterministic Finite State Automaton
Figure 1: At each inference step, a claim span mutated into an evidence span via a natural logic operator (NatOp). The current veracity state and mutation operator determine the transition the next state, via fine state automaton (DFA). Starting at the span Anne Rice mutated the equivalence operation (=), resulting in S according to the DFA. The inference ends in R indicating the claim's refutation. We use question-answeriig to predict the NatOps, taking advantage the generalization capabilities of instruction-tun tuned language models.
was born and the evidence Born is ascribed the equivalence NatOp (=), which we predict with questions such Is "was born" paraphrase of "Bor""?. This formulation enables to make of instruction-finetune language models, which are powerful learners, even with limited supervision (Sanh et al., 2022, inter alia). Since the input format to our ceessioo-anssering formulation constrains the context to the aligned claim-evidence spans, we generate claim-evidence alignments between overlapping spans of varying length, and individually predict the NatOp for each pair of aligned spans. To select the best proof over all possible proofs, we combine the answer scores to the questions associated with each proof.
nario on Danish fact verification dataset (Nørre- gaard and Derczynski, 2021, DanFEVER) without further annotation for training, our system outper- forms all baselines by 1.8 accuracy points, high- lighting the potential of the question-answering formulation for low-resource languages. An abla- tion study indicates the benefits of QA-NatVer's ireeston-answeerigg formulation, outperforming ChatGPT (OpenAl, 2022) (over 430x the size of BARTO) prompted with in-context examples to pre- dict NatOps by 11.9 accuracy points. Finally, we show in a human evaluation that QA-NatVer im- proves
previous natural logic inference sys- tems in terms of explainability, producing more plausible proofs with fewer erroneous NatOps.
In few shot setting with 32 training instances FEVER. QA-Nat outperforms all baselines by 4.3 accuracy points, including ProofVER, LOREN (Chen et al., 2022a), and state-of-the-art few-shot classification system, T-Few(Liu et al., 2022). By scaling the instruction-tuning model from BARTO to Flan-T5, we achieve a score of 70.3 + 2.1, clos- ing the gap to fully supervised models, trained over 140,000 instances, to 8.2 points. On adversarial claim verification dataset. Symmetric FEVER (Schuster et al., 2019), QA-NaIVER scores higher than few-shot and fully supervised base- lines, including ProofVER, demonstrating the ro- bustness of our approach. In a low-resource sce-
Related Work
Natural logic (Van Benthem, 1986; Sanchez, 1991) operates directly on natural language, making it an appealing alternative to explicit meaning represen- tations such as lambda calculus, since the transla- tion of claims and evidence into such representa- tions is error-prone and difficult decode for non- experts. NatLog (MacCartney and Manning, 2007, 2009) proposes the use of natural logic for textual inference, which has then subsequently been ex- tended by Angeli and Manning (2014) into the Nat-
'Code and models are available at https: http://gittthbccc com/

---

Janet Leigh incapable writing Evidence: [ JJanet Leigh ] She also wrote four books between 1984 and 2002. including two novels.
Claim:
Janet Leigh
was incapable
of writing
Does incepable writing entail four books?
Janet Leigh
was incapable
writing
Janet Leigh
She also wrote four
books
Janet Leigh
She also wrote four
books
Is incapable negation of She also wrote four books?
Janet Leigh incepable
incapable writing
REFUTES
Janet Leigh She also wrote four
(She also wrote four books
Answer:
Janet Leigh
was incapable of wriling
Janet Leigh
She also wrote four books
Janer Leigh was incapable of writing
Janet She wrote four
NetOp: Negation (-)
Multi-granular Chunking & Alignment (Sec. 3.1)
NatOp Assignment via QA (Sec. 3.2)
Proof Selection
(Sec. 3.3)
Figure 2: QA-NatVer's proof construction. We first chunk the claim and the evidence, and align them at multiple granularity levels. We then assign natural logic operator each aligned claim-evidence span using question- answering. Finally, we select the proof by combining the answer scores to the questions associated with the proof.
uralLI proof system. With the surge of pre-trained language models, multiple works have attempted to integrate natural logic into seuro-symbolic reason- ing systems (Feng et al., 2020, 2022). In particular, ProoFVer, a natural logic inference system specifi- cally designed for fact verification, achieves com- petitive performance yet remains faithful and more explainable than its entirely neural approaches (Kr- ishna et al., 2022). Stacey al. (2022) propose an alternative framework of logical reasoning, evalu- ating the veracity of individual claim spans (or of atomic facts in Stacey et al. (2023)) and determin- ing the overall truthfulness by following a simple list of logical rules. Chen et al. (2022a) use a simi- lar list of logical rules but aggregate the outcomes with a neural network component. However, all pre- vious approaches have in common that they require substantial training data perform well, limiting their use resource-rich languages and domains.
binary entailment, i.e. either supported or refuted. Zeng and Zubiaga (2023) explore active learning in combination with PET (Schick and Schütze, 2021), I), a popular prompt-base -baee few-shot learning method, and Pan et al. (2021) and Wright et al. (2022) gen- erate weakly supervised training data for zero-shot claim verification. However, none of the aforemen- tioned methods produces (faithful) explanations.
3 Method
Given a claim e and a set of evidence sentences E, the task of claim verification is to predict a ve- racity label ÿ and to generate a justification for the selected label. QA-NatVer is a system that returns natural logic inference proof which consists of sequence of relations P mi..... my, each of them specifying a relation between claim and evi- dence span and NatOp operator o.? The sequence of operators
01,.... oj then the input a deterministic finite state automaton that speci- fies the veracity label ğ = (OA (c.f. Figure I). The proof P itself serves as the justification for the predicted label ý.
Casting natural language problem to question- answering setting has previously been explored in a variety of tasks such as relation classification (Levy et al., 2017: Cohen et al., 2022), and semantic role labeling (He et al., 2015; Klein et al., 2022). For fact verification, in particular, previous works have considered formulating it as question generation task, decomposing claim into relevant units of information to inquire about, followed by question answering find relevant answers large knowl- edge base (Fan et al., 2020; Chen et al., 2022b). Yet, these works do not consider aggregating nor con- structing proofs from the answers to the questions.
Qer constructs its proofs following a three-step pipeline, illustrated in Figure 2: multi- granular chunking of the claim and its evidence sen- tences and the alignment between claim-evidence spans (Sec. 3.1), assignment of NatOps to each aligned pair using question answering (Sec. 3.2), and a proof selection mechanism over all possible proofs by combining the answer probabilities to the questions associated with the proof (Sec. 3.3).
Finally, work on few-shot claim verification limited. Lee et al. (2021) explore using a perplex- ity score, however, their approach is constrained to
II the natural logic literature these operators define mu- tations that would convert the claim so that it follows from the evidence (A (Angeli and Manning, 2014), however we do not make use of the edited claim in this paper.

---

3.1 Multi-granular Chunking & Alignment
late the prediction for each NatOp o question- answering task (cf. Table 1), each of which is in- stantiated by one or more boolean question prompts To- Only exception is the independence operator (#) which is applied when none of the other oper- ators are predicted. To predict whether a NatOp holds between claim span e; aligned with ev- idence e, we compute the log probabilities aver- aged over all question prompts To:
We chunk the claim initially into I non-overlapping consecutive spans c Cl...., Ct using the chun- ker of Akbik et al. (2019), and merge spans that do not contain any content words with their sub- sequent spans. To align each claim span e with the information of the highest semantic relevance in the evidence F,, we use the fine-tuned contextu- alized word alignment system of Dou and Neubig (2021) to first align individual words between the claim and each evidence sentence Ej. These word- then mapped back the span oj to form an evidence span eij from the sentence Fy. Since multiple spans in the evidence sentences could align with C₁ we measure the cosine similar- ity between & and each aligned evidence span Eij- using latent span embeddings via Sentence Trans- formers (Reimers and Gurevych, 2019).
QA Ca e,, = [T]
Σ LET.
log po(a Po ca, e,,t)
level alignments
(1)
with E No). [T] being the number of question prompts, and QA being our seq2seq instruction-tuned language model (see App. A for details). We apply an argmax function to select the most probable answer dg = argmax, argmax,QA(a Ca e,, T.). An answer prediction d, = Yes indi- cates that the NatOp o holds for the aligned claim- evidence spans. This formulation enables us to effectively use of the generalization abilities of instruction-tune -unnon language models.
It is of essence that the granularity of a claim's span matches the evidence span to capture their semantic relationship correctly. Therefore, we additionally consider merging the claim chunks C1,..., into more coarse-grained chunks. Con- cretely, we concatenate m consecutive claim chunks into single new span Gas m- with m being up to length of 4. The merging process results in total of at most q 4 6 chunks. Addi- tionally, we consider the claim itself as the most goarse-grained unit and align to evidence in F. Consider the example in Figure 2. A system that only considers a single chunking might consider was incapable and of writing as separate phrases. However, the evidence spans aligned to these indi- vidual phrases (She also wrote four and books, re- spectively) do not provide enough context individ- ually to infer their semantic relations with respect to the claim spans. However, when merged into a single chunk, their semantic relation becomes obvi- ous, as the span was incapable of writing is negated by she also wrote four books. Hence, a more flexi- ble variable-length chunking enables finding more semantically coherent alignments.
As illustrated in Figure 2, given the aligned claim-evidence spans was incapable of writing and She also wrote four books, we ask questions for each of the five NatOps, with the spans embedded in them. In the figure, the negation NatOp (-) is selected due to its corresponding boolean question being positively answered. Since predictions are made independently for each NatOp, it can occur that the model predicts multiple NatOps for sin- gle pair of aligned claim-evidence spans. In these instances, we select the NatOp with the highest probability (as computed in Eq. 1). On the con- trary, if none of the five NatOps is predicted, we assign the independence operator (#).
NatOp
Task
Question Example
Equivalence (=)
Paraphrase
in New Jersey a para- phrase of New Orleans?
identification
Given the premise New
Fwrd Entailment
Entailment
Orleans does the hypothe- sis New Jersey hold?
Rov. Entailment
Does in New Jersey entail New Orleans?
3.2 NatOp Assignment via QA
Entailment
(2)
the phrase in New Jer- sey a negation in New Orleans?
Each claim-evidence pair has to be assigned a NatOp, specifying their semantic relationship. We assign one out of six NatOps o E (,, I, #) to each claim-evidence span. We formu-
Negation (-)
Negation
classification
Alterntiin
Altemation
Does in New Jersey ex-
(I)
classification
clude in New Orleans?
Table 1: Natural logic operators and the corresponding natural language task and boolean question examples.
Siiillarly to Angeli and Manning (2014) and Krishna et al. (2022), we do not consider the rarely appearing cover operator, but instead replace it with the independence NatOp.

---

3.3 Proof Selection
3.4 Training
Since we expand the initiall non-overlapping claim chunks with gulti-granular merging into 9 over- lapping ones, we can construct total of C(t) En C(i) proofs, with C(i) being the number of proofs for chunks, C(0) = 1, and m being the maximum merge length.' 4 To select the most appropriate proof compute score for each one, defined the sum of NatOp probability score Sp (Eq. 2) and a NatOp verdict score 8y (Eq. 3) introduced later this section. We select the proof with the highest score.
We fine-tune the word-level alignment system as well as our question-answering system for QA- NatVer. The alignment system is trained on multi- ple training objectives as defined in Dou and Neu- big (2021) for parallel corpora, namely masked lan- guage modelling, translation language modelling, and their self-training objective. We consider the claim e and each gold evidence e E Fg as a sen- tence pair. We create further pairs using gold proofs Pg by considering all possible substitu- tions of claim chunks with their respective evi- dence chunk. Our question-answering system is fine-tuned following Liu et al. (2022), by optimiz- ing the maximum likelihood estimation (cf. Eq. 1), complemented by an unlikelihood loss which dis- courages the model from predicting incorrect target sequences. The NatOps the gold proofs Pa are used as positive QA samples, while we sample neg- ative training instances (i.e. NatOp questions with the answer being "No") from the gold proofs by randomly selecting wrong NatOp for an aligned claim-evidence span.
Since the probability for each NatOp is com- puted independently, define the score Sp as the average of the predicted NatOp probabilities:
1 Σ QA(aYes Ca, C, To),
Sp = n 1=1
(2)
with being the length of the proof P, and To being the questions for the NatOp assigned the i-th aligned span in the proof. Since no probability is explicitly assigned to the independence operator (#) as we have no question to directly capture it (cf. Section 3.2), we use the lowest scoring option to be added to the average in these cases (i.e. 0.5 since the predictions are boolean).
Evaluation
4.1 Few-Shot Claim Verification
Data We manually annotated 68 training in- stances of FEVER (Thorne al., 2018) with nat- ural logic proofs. Since FEVER does not contain gold evidence for instances labeled with NEI, we use retrieved evidence instead. The samples were drawn maintain a balanced label distribution, with 22 Supported, 25 Refuted, and 21 NEI in- stances. This results in proofs with a total of 183 Equivalence (=), 55 Forward Entailment (_), 16 Reverse Entailment (2), 29 Negation (-), 31 Alter- nation (l),, and 10 independence (#) NatOps. We train all systems 32 samples unless stated other- wise, randomly sampling from the aforementioned annotated instances. We evaluate our system on the development split of FEVER (Thorne et al., 201 8), consisting of 19, 998 claims, using retrieved evidence. We use the document retriever of Aly and Vlachos (2022), and the sentence reranker of Stammbach (2021) to select the top A 5 evi- dence sentences E. To assess the robustness of QA-NatVer, also evaluate the systems on Sym- metric FEVER (Schuster al., 2019), a binary clas- sification dataset (Supports, Refutes) consisting of 712 instances, which is built to expose models that learn artefacts and erroneous biases from FEVER.
The NatOp verdict score 8, considers the aligned claim with its evidence in its entirety as the most coarse-grained chunk, for which our question- answering system computes a score over textual representation of the veracity labels y:
8y = QU(DFAA(O) | c, F, T,),
(3)
with T, being the veracity question templates, and O being the NatOp sequence in the proof P. The score is defined to be the probability assigned to the veracity label associated with the state in the DFA in which the proof P would terminate in, i.e. (F)(O). In our example, the proof where was inca- pable and of writing are considered separate spans receive low score due to the two independence NatOps and its termination in the Not enough info (NEI) state, while the one where they are merged in a single span is selected due to high answer con- fidence in the predicted NatOps.
*While the number of potential proofs can grow large, com- putations for each claim- ciii-evvdenne span are only made once, resulting in linear complexity with respect to I, c.f. Section 4.5.

---

Baselines As state-of-the-art faithful inference system, ProoFVer (Krishna et al., 2022) is the main baseline we compare against, which is based on GENRE (Cao et al., 2021), , an end-to-end en- tity linking model, fine-tuned on BART (Lewis et al., 2020). We evaluate a version of ProoFVer that is trained on the same data as Qe-NatVer to compare both system's data efficiency. We re- fer the version of ProoFVer trained 140, 000 FEVER instances using additional knowl- edge sources as outlined in Sec. 1) as ProoFVer- full. Moreover, we evaluate in our few-shot setting LOREN (Chen al., 2022a), which decomposes claims into phrases, and predicts their veracity us- ing neural network regularized on the latently encoded phrases' veracity by simple logical rules, Similarly ProofVer, LOREN was trained on the entirety of FEVER.
art instruction-tume language model, which has explicitly seen natural language inference amongst many more tasks. Results are averaged over five runs with standard deviation indicated unless oth- erwise noted.
Accuracy
Macro-Avg. F1
DeBERTa DEBERTA-NLI
39.1 + 2.5
36.7 + 2.3
59.7 + 1.9
59.4
T-Few
5.2
58.8 + 3.1
LOREN
3.8
27.9 1.7
ProoFVer
36.2 + 1.3
32.5 113
QA-NatVer
64.0 + 0.9
63.1 1.5
+Flan-T5
70.3 + 2.1
70.0 = 1.4
Table 2: Results FEVER with 32 claim annotations for training.
Main Results Results are shown in Figure 2. QA-NatVer achieves an accuracy of 64.0 and a macro-averaged F1 of 63.1, outperforming T-Few by 1.6 and 1.3 absolute points respectively. More importantly, our system substantially outperforms ProoFVer which performs only marginally better than random when trained on the same amount of data. We improve accuracy by 27.8 8 points, high- lighting the limitation of the Prool'eer formulation in a few-shot scenario, and this observation also holds for LOREN. Despite QA-NatVer's BARTO not having seen any natural language inference task in its instruction-tunin DEBERTav3-NLL model. When training our QA- NatVer model with the Flan-T5 model instead, our system accuracy improves 70.3 + 2.1 with an Fi of 70.0 + 1.4 while being trained only on 32 claim annotations. For comparison, ProoF achieves an accuracy of 78.5, highlighting the data efficiency of QA-Nat
We further consider few-shot baselines that do not guarantee faithfulness or provide the same level of interpretability. These include state-of titte-of-ther the- art few-shot learning method, T-Few (Liu et al.. 2022), which uses two additional loss terms to improve few-shot fine-tuning. While Liu et al. (2022) is based on TO (Sanh et al., 2022), we in- stead use BARTO (Lin et al., 2022), BART model -estruttoonfinetuueed instruction- finetuned on the multi-task data mix- ture described in Sanh et al. (2022), to keep the baselines comparable. They observe comparable performance between BARTO and TO, which we confirmed in preliminary experiments. Finally, also evaluate finetuned DeBERTa, arGe model (He et al., 2021), and a DeBERTa model addition- ally fine-tuned on SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018), both being common and very competitive claim verification baselines (Stammbach, 2021; DeHaven and Scott, 2023).
it outperforms the
We are sampling K train- ing samples and do not consider a validation set for peerparrametertuuinng following the real-world few-shot learning setting of Alex et al. (2021). We use Awesomealign (Dou and Neubig, 2021) as the word-level contextualized alignment system. To stay comparable with our ProoFVer and T-Few baselines, we also use BARTO tuned language model. Notably, natural language inference not task BARTO has seen during in- struction fine-tuning. To take advantage of a more powerful QA system, we also evaluate QA-NatVer using Flan-T5 (Chung et al., 2022), a state-of-tt
Experimental Setup
Accuracy
Macro-Avg. Fj
DeBERTa DEBERTA-NLI
52.3 + 4.0
47.5 1 8,0
83.1 + 1.6
81.4 + 0.9
T-Few
73.7 + 3.8
73.0 13.0:44 4.7
ProoFVer
53,4 + 2.7
52.6 + 2.7
QA-NatVer
80.9 + 1.2
80.9
+ Flan-T55
85.8 + 0.9
85.8 + 1.0
instruction-
Table 3: Results on Symmetric-FEVIRR
Robustness As shown in Table 3. QA-NatVer performs competitively against all baselines when run without adjustment on Symmetric FEVER. QA- NatVer outperforms ProoFVer by 28.7 accuracy

---

points and T-Few by 7.2 points. Our system is also competitive with models trained on the entietty of FEVER, including ProoFVer-full, performing 1.2 accuracy points worse than ProoFVer-full (82.1). Training DeBERTa on NLI datasets improves ro- bustness substantially, performing even better than ProoFVer-full. Using Flan-T5 our instruction- tuned model instead, QA-Nat surpasses all other models, improving scores by about 4.9 accuracy points reach an accuracy of 85.8 0.9.
Performance over dliim length
20000
Verdict accuracy
Number samples
17500
15000
12500
5000
2500
40
Varying sample sizes. Table compares our ap- proach against the baselines when trained with varying amounts of data a few-shot setting, rang- ing between 16, 32, and 64 samples. QA-NatVer consistently outperforms our baselines across sam- ple sizes. Notably, while DeBERTav3 sees im- provements with 64 samples, ProoFVer's improve- ments marginal, indicating a larger need for data. The variability decreases across all mod- els with increasing sample size, with QA-NatVer having the lowest standard deviation, indicating its training robustness. QA-NatVer with Flan-T5 achieves sore of 71.6 + 1.7 when trained with 64 samples. The question-answering formulation might also be beneficial to obtaining large-scale proof annotations (ef. Section 2 cheaply, reducing the workload for the annotators. Investigating this option left to future work.
Hirimum Longth tokenss)
Figure 3: QA-NatVer on claims of varying length.
Ablation We perform three different ablation studies reported in Table 5. First, examine the performance of QA-Nat Ver without gaattlrga chunking. We observe a 7.7 average score drop accuracy, demonstrating that considering evidence spans at different levels of granularity improves per- formance. Second, we ablate our proof selection method by omitting the veracity score observ- ing accuracy drop by 3.7 points.
Finally, compare approach for NatOp assignments that prompted to predict NatOPs for claim- evidence spans as multi-class problem without multi multi-granular chunking. We use ChatGPT (Ope- nAI, 2022) and Llama-2 (Touvron et al., 2023) as state-of-the-ar few-shot language models and prompt them with in-context examples from our annotated proofs. The other parts of QA-NatVer kept identical. We observe that the non-QA ap- proach leads predicting more independence op- erators (#), resulting in an 11.9 points drop in accu- racy with ChatGPT, and a 16.6 drop with Llama2- 13B. For details see Appendix D.
question question-answerte answering model
Num. Samples
16
32
64
DeBERTav3
36.5
+ 2,4
39.1 = 255
47.4 5.6
DEL
55.2 3.0
59.7
+ 1.9
63.4 +10
T.Few
53.4
59.4
61.6
ProoFVer
33.7 40.5
36.2 + 1.3
37.7 +: 1.3
QA-NatVer
60.2+
64.0 + 0.9
65.5 + 0.4
Table 4: Results on FEVER with 16, 32, and 64 samples.
Claim length. Real-world claims typically longer than the example claim shown in Figure 1. MultiFC (Augenstein et al., 2019), a large-scale dataset of real-world claims, has an average claim length of 16.7 tokens compared to FEVER with 9.4. We subsequently measure the performance of Ve-NatVer as function of claim's 's minimum length, as shown in Figure 3. QA-NatVer shows only very small performance decline for claims of up a minimum of 18 tokens, indicating its robust- ness towards longer claims, correctly predicting the veracity of claims such as "The abandonment of populated areas due to rising sea levels is caused by global warming".
Setup Qa-Nat w/o mualti-granular chunking
Acc
Fi
64.0 + 0.9
63.1 + 1.5
56.3 0.6
55.3 + 0.7
w/o verdiet score & w/o QA (w/ ChatGPT) w/o QA (w/ Llama2-13B)
60.3 1.4
57.1 +
52.1
51.9
47.4
42.4
Table Ablation study of QA-NatVer on FEVER.
4.2 Application to Lower-Resouree Language Data To assess QA-NatVer's performance for lan- guages with fewer resources than English, we eval- uate it without further training (apart from the 32 FEVER annotated English claims) on DanFEVER

---

Claim: Highway to Heaven is something other than drama. Evidence: Highway to Heaven is American television drama series which NBC from 1984 to 1989.
Verdict: Refutation.
ProoF Ver Proof:
Claim Span
Highway Highway
Heaven
something other than drama is an American television drama series
scries
Evidence Span
to Heaven
drama
NatOp DFA State
2
R
R
R
QA-NatVer Proof:
Claim Span
Highway Highway
Heaven is something other than drama series from drama series
Evidence Span
Heaven
NatOp DFA State
S
R
Figure 4: FEVER example where ProoFVer and QA-NatVer reach the correct verdict (refutation). QA-NatVer produces more plausible proofs with fewer erroneous NatOp assignments than PeroFVer.
(Nørregaard and Derczynski, 2021), a three-way claim verification task for Danish, consisting of a total of 6, 407 instances. To eliminate additional variability from a multilingual retrieval system, we use the gold evidence for evaluation, except for NEL-labeled instances for which we retrieve evi- dence via BM25.
Results on DanFEVER are shown in Ta- ble Our system achieves accuracy and F1 of 61.0 and 56.5, outperforming all other baselines by 1.8 and 2.7 points, respectively. The ProoFVer base- line trained on the same data as our model achieves score of 17.9. Notably, in this setting our ap- proach even outperforms ProoFVer-full, where the claims and evidence being translated from Danish into English. Running ProoFVer-full in this set- ting is computationally expensive due to the trans- lation required and still has worse accuracy than QA-NatVer. The variability in this language trans- fer setup is higher than for FEVER, particularly for T-Few, but remains low for Ve-NatVer.
Results
Accuracy
Mucro-Avg. Fj
ALM-RoBERTL XLM-RO ROBERT--NLLI
41.9 3,0
31.7 3.3
54.3 1.3
52.2 0.7
T-Few
59.2 4.8
53.8 + 12.7
ProoFVer
47.9 =3.2
21.6 066:22 2.9
furoFVer-full w/ MT
56.9
52.1
QA-NatVer
61.0 3.5
56.5 4.9
Table Results on DanFEVER, using 32 FEVER
4.3 Correctness of Natural Logic Operators
claims using multilingual language model.
Assessing the quality generated proofs exclu- sively by the verdict they result in, ignores that an incorrect proof might lead to the correct verdict. For instance in Figure 4, ProoFVer fails to assign equivalence (=) even to identical spans, such Highway and Highway, yet it still produces the cor- rect veracity label. To intrinsically evaluate the quality of proofs, human subjects (not the authors of this paper) annotated a total 114 NatOp assign- ments from 20 claims and their associated proof from both ProoFVer and Ve-NatVer for their cor- rectness. Each NatOp assignment was annotated by 3 annotators, resulting in 342 data points. The claims selected via stratified sampling, ensuring that each class is equally represented. We further ensure that both models predict the same verdict. All three subjects assigned the same correctness label to a NatOp in 84.8% of cases, thus indicat- ing high inter-annotor agreement. QA-NatVer's
We use our baselines with multilingual backbones, namely T- Few, and ProoFVer with an mTOO backbone (Muen- nighoff et al., 2022), as well as finetuned XLMM RoBERTa (Conneau et al., 2020) model. We ad- ditionally consider ProoFVer-full by translating the claim and evidence from Danish into English, using the translation system by Tiedemann and Thottingal (2020). QA-NatVer also uses mTO (Muennighoff et al., 2022), multilingual T5- based model, instruction-tunn tuned on multilingual tasks. Similarly to BARTO, mTO has not seen nat- ural language inference in its fine-tuning. We use the chunking system of Pauli al. (2021) and keep all other components unchanged. The language of the natural logic questions and answers remains English for all experiments.
Baselines & Experimental Setup:

---

NatOp assignments are correct in 87.8% of the cases, while ProoFVer is only correct in 63.4%, in- dicating that the quality of NatOp assignments by QA-NatVer is superior to those by ProoFVer.
(Fleiss, 1971). Regarding the plausibility as- sessments, the subjects rate QA-NatVer proofs an average score of 1.61, while PeooFVer is rated 4.16 out of 5 points.
Considering the very high label accuracy of ProoFVer (outperforming QA-NatVer by almost 10 accuracy points), these results are surprising. We hypothesise that ProoFVer might have learned "shortcuts" to arrive at the correct verdict in its proof due to the noisy signals in the weakly super- vised proof dataset it has been trained on, due to the dataset-specific heuristics that have been applied construct dataset of sufficient size to train it on. To validate this, we inspect mutations where the claim and evidence spans are identical. These are trivial cases where the model expected to pre- dict the equivalence operator. However, ProoFVer produces a wrong NatOp for about 16.3% of cases, mostly the independence operator (13%), but our system always predicts equivalence (see App. C).
4.5 Efficiency
QA-NatVer remains computationally efficient since the typical bottleneck of transformer models, the input and output length, remain short every stage of the pipeline. Concretely, the alignment mod- ule encodes each evidence sentence with the claim independently. The QA model uses as input a ques- tion with a single embedded claim span and its evidence with the output being either Yes/No or short phrase. The average input length to the QA model on FEVER is 20 tokens while its output is in most cases single token. This is substan- tially cheaper computationally than cross-encoding the claim and all evidence sentences and autore- gressively generating the proof at once, as done by ProoFVer, with 195.2 input and 31.1 output tokens on average. The entire runtime of the QA module can be described as (1 + + with being the number of spans, Rspen being the input length for the aligned claim-evidence spans (for the NatOp probability score) and nall being the length of the claim and its evidence sentences (for the NatOp verdict score). We measure the wall-clock time (in minutes) with the BART-large backbone, using the same hardware configuration as deseribed in Ap- pendix B. DeBERTa, T-Few, PrroFVer, LOREN, and QA-Nat train in 5.3, 22.3, 21,4 4, 27.5, and 36.4 minutes, and inference on the FEVER devel- opment set of 19998 instances runs in 20.6, 7.3, 185.2, 116.5, and 89.1 minutes, respectively.
4.4 Plausibility
To assess the plausibility of the natural logic proofs predicted by QA-NatVer, we run forward predic- tion experiment (Doshi- Velez and Kim, 2017). Hu- man subjects are asked to predict the veracity label solely from the justification (i.e. proof) generated by the model and to specify on a five-level Likert scale, ranging from very plausible to not plausi- ble, how plausible the justification appears to them. Since we are evaluating proofs as an explanatory mechanism to humans, we ensured that no subject was familiar with the deterministic nature of natu- ral logic inference. To enable non-experts to make use of the proof, we replaced the NatOps with En- glish phrases, similar to (Krishna et al., 2022) (see Appendix C).
Conclusion
The evaluation consists of 120 annotations from 6 subjects. The same 20 claims used in the hu- man evaluation of correctness are paired with a ProoFVer or QA-NatVer proof explanation and are annotated by three subjects. No subject annotates the same claim for both models, as otherwise a subject might be influenced by the explanation it has seen before for the same claim. Using the Ver proofs, subjects correctly predict the model's decision in 90% of cases, compared to ProolVer'' 76.9%. All three subjects selected the same verdict in 70% and 91.7% of the cases, for ProoFVer and Qa-NatVer, respectively, with an inter-annotator agreement of 0.60 and 0.87 Fleiss
This paper presented Ve-,at a natural logic inference system for few-shot fact verification that frames natural logic operator prediction as question-answering We show that our approach outperforms all baselines while remaining faithful. Human evaluation shows that QA-Nat produces more plausible proofs with fewer erroneous natu- ral logic operators than the state-of-the-ae natural logic inference system, while being trained on less than thousandth of the data, highlighting QA- NatVer's generalization ability. Future work looks at extending the capability of natural logic infer- ence systems to more complex types of reasoning, including arithmetic computations.

---

Limitations
Cheng, Sana Kidwai, Nedjma Ousidhoum, Andre Schurat, and leva Raminta Staliunaite. We finally thank the anonymous reviewers for their time and effort giving us feedback on our paper.
While natural logic provides strong explainability by operating directly on natural language, it is less expressive than alternative meaning representations that require semantic parses such as lambda caleu- lus (Zettlemoyer and Collins, 2005). For instance, temporal expressions and numerical reasoning are beyond the expressive power of natural logic (Kr- ishna et al., 2022) but are frequently required when semi-structured information is available (Aly et al., 2021). Moreover, cases of ambiguity like cherry- picking, are difficult process with natural logic. Addressing the limits of natural logic inference systems is out-of-scope for this paper. Similar to ProoFVer, the proof we constructed intended to be executed in the DFA from left to right, however, natural logic-based inference is not constrained to such execution. Furthermore, all benchmarks ex- plored in the paper use Wikipedia the knowledge base which is homogeneous compared heteroge- neous sources professional fact-checkers use (e.g., news articles, scientific documents, images, videos)
References
Alan Akbik, Tanja Bergmann, Dunean Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy masy-to-use framework for bate-of-the the- art NLP, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pages 54-59. Minneapolis, Minnesota. Association for Computational Linguisties.
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Car- lier. Michael Noetel, and Andreas Stuhlmüller. 2021. RAFT: real-world few-shot text classification benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 2).
Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull,
James Thome,, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocaraseu, and Arpit Mittal, 2021. FEVEROUS: Fact extraction and VERification over unstructured and structured information. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Ethics Statement
Our system improves the explainability of claim verification models and empowers actors to make more informed decisions about whether to trust models and their judgements, yet actors must re- main critical when evaluating the output of auto- mated claim verification systems and not confuse explainability with correctness. We emphasize that we do not make any judgements about the truth of statement in the real world, but only consider Wikipedia as the source for evidence to be used. Wikipedia is a great collaborative resource, yet it has mistakes and noise of its own similar to any encyclopedia or knowledge source. Thus we dis- courage users using our verification system to make absolute statements about the claims being verified, i.e. avoid using it to develop truth-tellers,
Rami Aly, Xingjian Shi, Kaixiang Lin, Aston Zhang. and Andrew Gordon Wilson. 2023. Automated few-shot classification with instruction- finetuned lan- guage models. arXiv preprint arXiv: 12576.
Rami Aly and Andreas Vlachos. 2022. Natural logic- guided autoregressive multi-hop document retrieval for fact verification. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6123-6135, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Gabor Angeli and Christopher D. Manning. 2014. Nat- uralLI: Natural logic inference for common sense reasoning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 534-545, Doha, Qatar. Association for Computational Linguistics.
Acknowledgements
This work was supported by the Engineering and Physical Sciences Research Council Doctoral Training Partnership (EPSRC). Andreas Vlachos is supported by the ERC grant AVeriTeC (GA 865958) and the EU H2020 grant MONITIO (GA 965576). Further, the authors would like to thank the subjects who volunteered to be part of the hu- man evaluation, namely Mubashara Akhtar, Julius
Pepa Atanasova, Maria Camburu, Christina oma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023. Faithfulness tests for natural language explanations. arXiv preprint arXiv:2305./8029,
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Chris- tian Hansen, and Jakob Grue Simonsen. 2019. Mul-

---

tiFC A real-world multi-domain dataset for evidence- based fact checking of claims. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP), pages 4685-4697, Hong Kong, China. Association for Computational Linguistics.
(FEVER), pages 58-65, Dubrovnik, Croatia, Associ- ation for Computational Linguistics.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational I Linguisties.
Finale Doshi- Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning, arXiv preprint arXiv: 170200868..
Greg Brockman, Peter Welinder, Mira Murati, and Ope- nÂI. 2020. Openai: Openai api. htpps://openai. com/ble http:/llloppppiaaaaaeaa/
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 2112 1122- Online. Association for Computational Linguisties.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representa- tions.
William A Falcon. 2019, Pytorch lightning, CitHub, 3.
Jiangjie Chen, Qiaoben Bao, Changzhi Sun, Xinbo Zhang, Jiaze Chen, Hao Zhou, Yanghua Xiao, and reason-
Angela Fan, Aleksandra Piktus, Fabio Petroni, Guil- laume Wenzek, Marzich Sacidi, Andreas Vlachos, Antoine Bordes, and Sebastian Riedel. 2020. Gen- erating fact checking briefs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Lei Li. 2022a. Loren: Logic-regularee ing for interpretable fact verification. Proceedings of the AAAI Conference on Artificial Intelligence, 10482-:0442210499
Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022b. Generating literal and implied sub- questions to fact-check complex claims. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3495-3516, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yufei Feng, Xiaoyu Yang, Xiaodan Zhu, and Michael Greenspan. 2022. Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inference. Transactions of the Association for Com- putational Linguistics, 10:240-256.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. language models.
Yufei Feng. 7i ou Zheng, Quan Liu, Michael Greenspan, and Xiaodan Zhu. 2020. Exploring end-to -od differ- entiable natural logic modeling. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1172-1185, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
2022. Scaling instruction-finette arXiv preprint arXiv:2210.1 11416.
Amir DN Cohen, Shachar Rosenman, and Yoav Gold- berg. 2022. Supervised relation classification as two- way span-prediction. In 4th Conference on Auto- mated Knowledge Base Construction.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asxo- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17333
Juri Ganitkevitch, Benjamin Van Durme, and Chris Kallison-Burchh 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758-764, Atlanta, Georgia. As- sociation for Computational Linguistics.
Mitchell DeHaven and Stephen Scott. 2023. BEVERS:
A general, simple, and performant framework for automatic fact verification. Proceedings of the Sixth Fact Extraction and VERification Workshop

---

Zhijiang Guo, Michael Schlichtkrull, and Andreas Vla- chos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178-206.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequeneceto-ssqqeeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Länguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Us- ing natural language to annotate natural language. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 643-653, Lisbon, Portugal. Association for Compu- tational Linguistics.
Bill Yuchen Lin, Kangmin Tan, Chris Scott Miller, Bei- wen Tian, and Xiang Ren. 2022. Unsupervised cross- task generalization via retrieval augmentation. In Advances in Neural Information Processing Systems.
Pengcheng Hc, Jianfeng Gao, and Weizhu Chen. 2021. 3ebertav3; Improving deberta using electra-style pre- training with miaetent-disentanlleddd embedding shar- ing. arXiv preprint arXiv:2111.0954
Haokun Liu, Derek Tam, Mugeeth Mohammed, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficiee fine-tuning is bet- ter and cheaper than in contett learning. In Advances in Neural Information Processing Systems.
Alon Jacovi and Yoav Goldberg. 2020. Towards faith- fully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198 4205, On- line. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Bill MacCartney and Christopher D. Manning. 2007. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 193-200, Prague. Associa- tion for Computational Linguistics.
Alon Jacovi and Yoav Goldberg. 2021. Aligning faithful interpretations with their social attribution. Transac- rions of the Association for Computational Linguis- tics, 9:294-310.
Ayal Klein, Eran Hirsch, Ron Eliav, Valentina Pyatkin, Avi Caciularu, and Ido Dagan. 2022. QASem pars- ing: Text-to-text modeling of QA-based semantics. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceed- ings of the Eight International Conference on Com- putational Semantics, pages 140-156, Tilburg. The Netherlands, Association for Computational Linguis- tics.
George A. Miller. 1994. WordNet: A lexical database for English. In Human Language Technology: Pro- ceedings of Workshop held Plainsboro, New Jersey, March 8-11, 1994.
Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7740-7754, Online. Association for Computa- tional Linguisties.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman. Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint 2eXiv:2211.07220
Amrith Krishna, Sebastian Riedel, and Andreas Vlachos. 2022. ProoFVer: Natural logic theorem proving for fact verification. Transactions of the Association for Computational Linguistics, 10:1013-1030.
Jeppe Nørregaard and Leon Derczynski. 2021. Dan- FEVER: claim verification dataset for Danish. In Proceedings of the 23rd Nordic Conference on Com- putational Linguistics (NoDaLiDa), pages 422-428, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
Nayeon Lee, Yejin Bang, Andrea Madotto, and Pascale Fung. 2021. Towards few-shot fact-checking via perplexity. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. pages 1971-1981, Online. Association for Computational I Linguistics.
OpenAl. 2022. Openai:
Introducing chatgpt. https:
//openai hatm//log/chhhatgge
Omer Levy, Minjoon Sco, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the s Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver, Canada, Association for Computational Linguistics.
Liangming Pan, Wenhu Chen, Wenhan Xiong, Min- Yen Kan, and William Yang Wang. 2021. Zero-shot fact verification by claim generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

---

(Volume Short Papers), pages 476-483, Online. Association for Computational Linguistics.
pages 253-269, Online. Association for Computa- tional Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary Veiito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style. terf-forfrmmece deep learning li- brary. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay, 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing SEMNLP-IJCNLI pages 3419 3425, Hong Kong, China. Association for Com- putational Linguistics.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Oana-Maria Camburu, and Marek Rei. 2023. Log- ical reasoning for natural language inference using generated facts as atoms.
Amalie Brogaard Pauli, Maria Barrett, Ophélie Lacroix. and Rasmus Hvingelby, 2021. DaNLP: An open- source toolkit for Danish natural language process- ing. In Proceedings of the 23rd Nordic Conference on Compulational Linguistics (NoDuLiDa), pages 460-466, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. 2022. Logical reasoning with span-level predictions for interpretable and robust NLI models. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 3809-3823, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018. DeClarE: Debunking fake news and false claims using evidence-aware deep learning. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pages 22-32. Brussels, Belgium. Associ- ation for Computational Linguistics.
Dominík Stammbach. 2021. Evidence selection as token-level prediction task. In Proceedings of the Fourth Workshop on Fact Extraction and VERifica- rion (FEVER), pages 14-20, Dominican Republic. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguisties.
James Thorne, Andreas Vlachos, Christos Christoddoulopoulos,, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages
Victor Sanchez. 1991. Studies on natural logic and categorial grammar. Ph.D. thesis, University of Am- sterdam.
New Orleans, Louisiana. Association for Computational Linguistics.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS- MT - building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. pages 479-480, Lisboa. Portugal, European Associa- tion for Machine Translation.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Hugo Touvron, Louis Martin, Kevin Stone, Peter AL- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09220 09988..
Johan Van Benthem. 1986. Natural Logic. pages 109- 119. Springer Netherlands, Dordrecht.
Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACI. 2014 Workshop on Lan- guage Technologies and Computational Social Sci- ence, pages 18 22, Baltimore, MD, USA. Associa- tion for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the Earopean Chapter of the Asso- ciation for Computational Linguistics; Main Volume,

---

Denny Vrandečić and Markus Krötzsch, 2014, Wiki- data: A free collaborative knowledgebase. Commun. ACM, 57(10):78-85.
QA(y c, E,To, =
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Humun Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
1 M ΣΣ log poly c, Hemm), (4)
IE7
with M being the length of the textual representa- tion of y.
The input is structured so that the claim fol- lowed by the evidence sentences, each separated by end-of-sentence tokens: с 
 60, 
... 
 Cj- Each evidence sentence in E, is preceded by the corresponding document title in square brackets, e.g. "[James McBrayer] Jack McBrayer (born May 27... [Tom Bergeron] Tom Bergeron (born May 6, 1955)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 3844, Online, Association for Computational Linguistics.
Training We follow Liu et al. (2022) when fine- tuning our model on multiple prompts, randomly selecting prompt t E To. We fine-tune all weights of the model (both for BARTO and Flan-T5). We experimented with sampling negative training in- stances for QA-Nat that follow the positive negative sample distribution as present during infer- ence (i.e. 1:4 positive to negative question-answer pairs), yet the training process frequently diverged due to the high label imbalance. The question- answering module is trained jointly on veracity question templates and natural logic operator tem- plates.
Dustin Wright, David Wadden, Kyle Lo. Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022. Generating scientific claims for zero- shot scientific fact checking. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2448-2460, Dublin, Ireland. Association for Compu- tational Linguistics.
Xia Zeng and Arkaitz Zubiaga. 2023. Active PETs: Ac- tive data annotation prioritisation for few-shot claim verification with pattern exploiting training. Find- ings of the Association for Computational Linguistics: EACL 2023, pages 190-204, Dubrovnik. Croatia. As- sociation for Computational Linguistics.
B Implementation Details
All models are implemented using Py Torch (Paszke et al., 2019), making use of PyTorch Lightning (Fal- con, 2019). All experiments using BARTO as the instruction-finetuned QA model were run on a ma- chine with a single Quadro RTX 8000 with 49GB memory and 64GB RAM memory. To fine-tune the Flan-T5 model (Flan-T5-x] with its 3 billion parameters, use single Ampere A100 GPU with 80GB memory. We have also run some early tests with TO (Sanh al., 2022) but noticed that the performance of BARTO is compurable, confirming observations made in Lin et al. (2022). All models are implemented using Huggingface (Wolf et al.. 2020).
Luke S. Zettlemoyer and Michael Collins. 2005. Learn- ing to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, UAI'05, page 658-666, Arlington, Virginia, USA. AUAI Press.
A QA-Nat Ver
Question-Answering System BARTO is a pre- trained seq2seq model which has been instruction- finetuned to autoregressively generate the predic- tion for a given input prompt. In the case of boolean questions where only a single token is predicted, the generation can be simplified to Equation 1. Oth- erwise, such as for equation 3, where the answer coice y E (Supports, Reftues, Not enough info). is longer than a single token, the probability as- signed QA-NatVer can be described via an autore- gressive formulation as:
QA-Nat The language model we used with AwesomeAlign is the multilingual BERT model (Devlin et al., 2019). To fine- tune our word alignment system, we use

---

Determine an entailment relation between two expressions.
Use one of the following entailment relations: Equivalence, Negation, Alternation, Forward-Entallmeen, Reverse- Entailment, Independence.
Equivalence means that two expressions have the same meaning. This includes synonyms and paraphrases. Examples of Equivalence:
[N examples of Equivalence)
Negation means that one expression negates the other and that both cannot be true at the same time. Examples of Negation:
10
[N examples of Negation)
13 Alternation means that expressions are mutually exclusive.
Examples of Alternation:
(N examples of Alternation)
17 Forward- Entailmeet means that the first expression entails the second expression. Examples of Forward-EEtalimenn Entailment:
(N examples of Forward-Entailment
20
Reverse-Entailmen means that second expression entails the expression.
Examples of Enverse-nttallmett.
[N examples of
24
Independence means there informative relation between expressions. Examples of Independence:
[N examples of Independence)
Listing 1: The instructive part the prompt used in ChatGPT and Llama2 ablation experiments. For each NatOp. we provided short explanation and examples (colored in blue) formatted (evidence, triples.
the AwesomeAlign repository the sen- tence transformer we use for alignment is wtwrrraaforrrrraaammaaarara
provided us access to their ProofVER model. For our ablation experiment with ChatGPT, We use OpenAI's API (Brockman et al., 2020) to query ept-3.5-turbo-00
addition to the similarity score from the sentence transformer to select the most appropriate evidence span from all options Cij. we take into account the evidence retriever's ranking of the evidence sentences, down-weipgs down-weighting the similarity score by factor that scales negatively with the rank, i.c. (j 0.8).
and ask ChatGPT to assign NatOPs to batches of claim-evidence pairs (at most 25 spans per query). For ablation experiments with Llama2, we ran 13B parameter models locally. We used the GPTQ (Frantar al., 2022) version of these models with 4-bit quantization to lower the computational requirements and speed up the infer- ence.
Baselines We evaluate the T-Few baseline using the provided repository which also provided the basis for the QA-Nat code implementation. Liu et al. (2022) also propose a parameter-efficient fine- tuning method (named (IA)") ) in addition to the loss functions, however, we observe much more stable training loss and better results in preliminary exper- iments when tuning the entire model, particularly for BARTO. This observation of degrading perfor- mance on BARTO with (IA)3 is also observed in Aly et al. (2023). Therefore, for all experiments, and all models (baselines + QA-NatVer) we trained all model weights. Krishna et al. (2022) kindly
Hyperparameters Since no development data was used to tune hyperparameters, we set them to the default values described in Liu et al. (2022). We only reduced the learning rate for QA -ettVr as we noticed that the training loss was unstable. Specifically, we set the hyperpa- rameters learning_rate: le-5, batch_size: 4, training_steps
8. @ead_acccm_facdee 2000, use the AdamW optimizer (Loshchilov and Hutter, 2019), and linear decay with warmup scheduler. The same hyperparameters are used with Fla
Pht http://gggthbbboooneeee/a//sssss hhttps://github.. wtwpgtthub.ccmmrrtteeeetttteee

---

C
Human Evaluation
All subjects in the human evaluation are graduate/- postgraduate students in either computer science or linguistics. 3 subjects are male, 3 female. None of the subjects had prior knowledge of natural logic in- ference. Table 7 shows the textual description used for the NatOps in the human evaluation and Table 8 shows the predicted NatOps by both ProoFVer and our system for aligned claim-evidence spans that are identical.
NatOP
Textual Description
Equivalent Spans
Claim span follows from the evidence span Claim span is negated by the evidence
Evidence span contradicts the claim span Incomplete Evidence
Unrelated claim span evidence span
Table 7: NatOPs and their corresponding textual de- scription as shown to human anmotators, following (Krishna al., 2022).
ProoFVer 9832
Ours
# Eq. Spans
17774
Eq. Natop
8233 (83.7%)
17774 (100%)
Fwd. Entail Natop
84 (1%)
0 (0%)
Rev. Entail Natop
193 (2%)
0 (0%)
Neg Natop
0 (0%)
0 (0%)
Natop
(0.5%)
(0%)
Indep Natop
1274 (13%)
0 (0%)
Table Predicted NatOps for claim and evidence spans that identical. Despite the triviality, ProoFVer does assign the wrong operator in about 16.3% of cases. In contrast, our system does consistently predict equality in every instance.
D Prompting
Listing 1 shows template for the instructive part of our prompts with ChatGPT and Llama2. For ChatGPT, we used OpenAl's API (Brock- man et al., 2020) to query
and asked ChatGPT to assign NatOPs to claim- evidence pairs. We used batches of at most 25 spans per query to lower the running costs. For Llama2 experiments, we the models locally and asked the models to assign NatOP per query.


Reviews:
[{"review_id": "C6TnzoPL9l", "reviewer": "Reviewer_5F8N", "summary": "", "questions": "QA. How does this method perform on complex claims?\n\nQB. Have you run any experiments where BART0 and Flan-T5 are directly fine-tuned for fact-checking tasks? I expect them to make a strong baseline, especially for Flan-T5 models.\n\nQC. What's the baseline performance of LLMs (like InstructGPT or ChatGPT) in this task? \n\nQD. Can you design a chain-of-thought prompting for LLMs that resembles the NatOps and Proofs used in the proposed method? You can also add several annotated in-context examples for this purpose. I'm curious whether such decomposition and reasoning method could be useful for LLMs, because the reasoning process seems to be the major contribution of this work.\n\nQE. How is the efficiency of this method?\n\nQF. How does the size of backbone model (i.e. scaling law) influence the effectiveness of this method?\n\nQG. How is the expressiveness of natural logic, and what is the limitation of it? Can LLMs reason properly with natural logic?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and comments on our paper! We address the reviewer's questions and the modifications we make to our paper based on the reviewer's suggestions below.\n\n**Lack of discussions on LLMs (Question C, Question D, Question F)**\n\nWe agree with the reviewer that LLMs are worth taking into account and that their integration into a natural language inference system is of interest. In fact, our paper already contains some experiments using LLMs and how their abilities can be integrated into our system:\n\n1. In our ablation (lines 469- 479) we investigated how our question-answering formulation for the prediction of NatOps compares to using ChatGPT for the same task, the latter being prompted in a multi-class setting with 5 in-context examples for each NatOp and a short description. Our system outperforms ChatGPT as the latter overpredicts independence operators (#), indicating that our design carries merit (the details of our experiments with ChatGPT will be added in the camera ready if accepted). We will conduct further experiments exploring various prompting strategies to predict NatOps as suggested by the reviewer (chain-of-thought prompting, self-correcting prompting, etc.) and also how LLMs can compose predicted NatOps into proofs. These experiments have been difficult to conduct given the time constraint of the rebuttal but we will report additional findings in the paper.\n\n2. We show that our approach benefits from more powerful backbones, as we see substantial performance improvements when scaling up from BART0 with 400M to FlanT5-xl with 3B parameters (one might already consider the latter an LLM), highlighting that our method is able to take advantage of recent advancements of LLMs. We have run additional experiments on smaller and intermediate-sized models (BART0-base: 60.1 +- 1.9, and FlanT5-large: 67.1 +-1.4), which further validate the observed trend. Moreover, we managed to improve performance on FlanT5-xl from 69.1 in the last version of the paper to 70.3, with only 32 training samples (64 training samples: 71.6 accuracy points). Our approach with 32/64 samples with the Flant5-xl backbone comes close to state-of-the-art fully supervised models on FEVER, trained on over 140,000 instances. These results highlight how the increased capabilities of LLMs benefit our method. However, due to computational constraints, we have not yet been able to run experiments on the largest Flan-T5 model with 11B parameters. We plan to compute results on this for the camera-ready version if accepted.\n\n**The proposed method seems to only work on simple claims in FEVER. (Question A)**\n\nWe agree with the reviewer that some classes of complex claims, for instance, ones that require temporal and numerical reasoning, or cases of ambiguity like cherry-picking, are difficult to process by our method. Yet, these types of claims are difficult to handle by most contemporary verification systems. Further, in most instances, a naturally occurring claim does not span multiple sentences but is instead expressed in a single one, see e.g. MultiFC [1], a large-scale dataset of real-world claims. However, we observe a difference in the claim’s length: MultiFC has an average claim length of 16.7 tokens, versus 9.4 for FEVER. We subsequently measure the performance of QA-NatVer on claims in FEVER that are at least 14 tokens long, which constitute 1286 out of about 20,000 instances. We see only a small performance decline (BART0: 0.62 +- 1.2, FlanT5-xl: 0.67 +- 1.1), indicating that more complicated structures (e.g. “The abandonment of populated areas due to rising sea levels is caused by global warming”) are not problematic per se. We will ensure to add a more detailed analysis of this point into the paper in the additional space if accepted.\n\n**I somehow hold concerns about natural logic on real-world scenarios (Question G)**\n\nWe do not attempt to extend the capabilities of natural logic inference systems in our paper.  Instead, we address the severe limitation of faithful inference systems of requiring substantial resources for training. We discuss the limitations of natural logic’s expressiveness in our limitations section. It would be interesting to explore in future work possible ways of mapping more complex semantics (e.g. algorithmic operations) to the set-theoretic semantics of natural logic, but this is beyond the scope of this paper.\n\n**Missing reference: LOREN [...] they adopted a very similar idea to this work [...] constructing proofs with them, and aggregate them via logic regularization.  I'd suggest the authors to include more discussions with this work and its ideas.**\n\nThank you very much for the additional reference, we will ensure to relate to it in our work! \n\nWe conducted experiments for LOREN in a few-shot setting using the experimental setup described in our paper. LOREN requires substantial amounts of training data (it is trained on all of FEVER, 145K instances), thus suffering from similar limitations as other previous work. Without further adjustments (besides upgrading the backbone of the MRC model from BART-base to BART-large to keep the comparison fair), LOREN achieves an accuracy of 33.4 on FEVER. We subsequently attempted to improve the performance by adjusting the hyperparameters to match the ones used in our work (a much higher number of epochs and a smaller learning rate), resulting in a score of 35.7. These results are comparable with ProoFVer in a few-shot setting and confirm the difficulty of building interpretable reasoning systems in low-resource scenarios.\n\nMoreover, the logical rules LOREN defines are similar to the three simple logical rules used in Stacey et al. (2022) [2]. These rules do not constitute a logical proof, as their outcomes are merely aggregated by a neural network component, while in the case of Natural Logic these are aggregated by a finite state automaton. Consider for instance a case of double negation such as for the claim “Paul wants to be dry” and the evidence “Nobody wants to be wet”. While natural logic can easily handle such cases, LOREN’s logical rules are not able to handle these explicitly. Instead, LOREN’s MRC system that generates the “local premises” that the logical rules operate on must be powerful enough to cover for the logical rules’ restrictions (i.e. not generating a local premise that can be refuted since a single refuted local premise results in the refutation of the entire claim,  e.g. not doing a substitution that results in a local premise such as “Paul wants to be wet“). However, shifting the reasoning implicitly to the MRC model results in reduced transparency compared to natural logic, since the double negation is never covered explicitly by the logical rules.\n\nThe LOREN comparison will be added to our paper, following further experiments and an analysis of its error sources.\n\n**Have you run any experiments where BART0 and Flan-T5 are directly fine-tuned for fact-checking tasks? (Question B)**\n\nYes. We initially explored using BART0 as a traditional classifier (similar to DeBERTa, using a linear layer on top), yet observed substantially lower performance than when using the model in the generative setting (i.e. T-Few) (41.6 versus 59.4). For clarification: With T-Few we fine-tune the backbone (BART0) directly for fact-checking, complemented by additional loss functions defined by T-Few. As seen in our paper, the results of T-Few still trail behind QA-NatVer, similar for Flan-T5.\n\n\n**The pipeline of this method seems to be a little lengthy, [...] Adding efficiency analysis would be helpful to clarify this. (Question E)**\n\nWe will add a section on our method’s complexity to the paper in the additional space if accepted. Overall, our method remains efficient since the input length to the models at every stage of the pipeline is short, which is typically the computational bottleneck due to the quadratic runtime of transformers with respect to the input length. Concretely, the alignment module encodes each evidence sentence independently with the claim. Similarly, the QA model for the NatOp prediction takes as input a question with a single embedded claim and evidence span. The average input length is 19.95 tokens, substantially shorter than cross-encoding the entire claim and all evidence sentences, with 195.2 tokens on average. The entire runtime of the QA module can be described as  $\\mathcal{O}(l * n_{\\text{span}}^2 +n_{\\text{all}}^2)$, with $n_{\\text{span}}$ being the input length for the aligned claim-evidence spans (for the NatOp probability score) and $n_{\\text{all}}$ being the length of the entire claim and its evidence sentences (for the NatOp verdict score).\n\nWe measure the following wall-clock time (in minutes) training on 32 instances, and running inference on the FEVER development set of 19998 instances, using the same hardware configuration as described in the paper. All models use the BART-large backbone (except DeBERTa):\n\n| Model | Train      | Inference |\n| -----------  | ----------- | ----------- |\n|DeBERTa       |5.3      | 20.6       |\n|T-Few      | 22.3   | 7.31        |\n|ProoFVer     | 21.35   | 185.22        |\n|Loren     | 27.49   | 116.46        |\n|QA-NatVer    | 36.39   | 89.11    |\n\nAs shown above, QA-NatVer performs comparably to the baseline methods, taking the longest for training (but still manageable with about half an hour of training time), and less than both LOREN and ProoFVer during inference.\n\n[1] https://aclanthology.org/D19-1475.pdf\n\n[2] https://aclanthology.org/2022.emnlp-main.251.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include:\n1. This paper proposes a reasonable method of using QA to predict natural logic operators, contributing to the explainability of neural fact verification systems.\n2. This paper describes in detail about the design and implementation of the QA-NatVer system, including the process of proof construction, which is very helpful for understanding and reproducing the system.\n3. The system performs well on multiple datasets, and enjoys an additional edge over previous method on the correctness and plausibility of proofs.", "reasons_to_reject": "1. Lack of discussions on LLMs. I would really love to see how this idea could be integrated into current large language models (LLMs), like InstructGPT and ChatGPT, which have strong few-shot in-context learning and chain-of-thought reasoning abilities. Conveniently, the major ideas and techniques in this paper can be easily adapted to the LLM CoT prompting setting (few-shot, reasoning chains, etc.). In a time when the foundations of most systems are shifted toward LLMs, I think it's important to add them into discussions. I would really love to see if smaller models could beat LLMs using some good designs.\n2. The proposed method seems to only work on simple claims in FEVER. However, it is unknown if naturally-occurring claims with multiple sentences and complex structures can be correctly processed by this method, especially for the chunking & alignment module. \n3. The pipeline of this method seems to be a little lengthy, which raises concerns for its practical value. Adding efficiency analysis would be helpful to clarify this. \n4. I somehow hold concerns about natural logic on real-world scenarios, for example, claims with first-order logic like \"not all US senators are hispanic\".", "questions_to_authors": "QA. How does this method perform on complex claims?\n\nQB. Have you run any experiments where BART0 and Flan-T5 are directly fine-tuned for fact-checking tasks? I expect them to make a strong baseline, especially for Flan-T5 models.\n\nQC. What's the baseline performance of LLMs (like InstructGPT or ChatGPT) in this task? \n\nQD. Can you design a chain-of-thought prompting for LLMs that resembles the NatOps and Proofs used in the proposed method? You can also add several annotated in-context examples for this purpose. I'm curious whether such decomposition and reasoning method could be useful for LLMs, because the reasoning process seems to be the major contribution of this work.\n\nQE. How is the efficiency of this method?\n\nQF. How does the size of backbone model (i.e. scaling law) influence the effectiveness of this method?\n\nQG. How is the expressiveness of natural logic, and what is the limitation of it? Can LLMs reason properly with natural logic?", "paper_topic_and_main_contributions": "This paper primarily investigates how to use QA to predict natural logic operators to improve fact verification systems. The main contributions include:\n1. Proposing a novel method of using QA to predict natural logic operators, which avoids the need for extensive annotated training data while still relying on deterministic reasoning systems.\n2. Introducing the QA-NatVer system, a natural logic reasoning system that constructs proofs by transforming natural logic into a QA framework.\n3. Outperforming all baselines in a low-resource setting on FEVER, including a state-of-the-art pre-trained seq2seq natural logic system and a state-of-the-art prompt-based classifier. The system demonstrates its robustness and portability on adversarial datasets and Danish verification datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {"Missing_References": "1. The claim in Line 151-153 that \"these works do not consider aggregating nor constructing proofs from the answers to the questions\" is not very accurate. In [LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification](https://ojs.aaai.org/index.php/AAAI/article/view/21291) (Chen et al., in AAAI 2022), they adopted a very similar idea to this work: decompose claims (chunking) and acquire veracity information from evidence with QA, construct proofs with them, and aggregate them with logic regularization. I'd suggest the authors to include more discussions with this work and its ideas."}}, {"review_id": "86JkCom1k8", "reviewer": "Reviewer_Esj4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and comments on our paper! Indeed, our approach inherits the limitations of Natural Logic (see Krishna et al. 2022 [1] for a discussion), and we will mention this in the camera ready if accepted.\n\n[1] https://aclanthology.org/2022.tacl-1.59/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and straightforward to follow.\n\nThe focus on designing language model-based systems that produce explanations that can be trusted is appropriate given the current emphasis on pure LLM-based systems.\n\nThe emphasis on low-resource languages is good to see\n\nThe paper thoroughly tests their method, showing performance across 1) multiple datasets, 2) different languages, and 3) has a human evaluation", "reasons_to_reject": "The main limitation in my opinion would be that the range of situations where this method is applicable would be limited to situations that can be handled by natural logic. I do not consider this a significant issue, however.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a system for constructing a proof of a claim from a set of evidence statements using Natural Logic. However, rather than use traditional natural logic operators (where the applicability of a particular operator is determined by a model trained on a large amount of data), the authors cast the operators into questions. This allows their approach to use strong pretrained language models to assess whether a particular operator applies when constructing a proof. Consequently, their approach requires less labeled data and exhibits stronger performance", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lzLix38QhK", "reviewer": "Reviewer_5v7q", "summary": "", "questions": "1. The proof selection part isn't very clear, is this similar to the CYK algorithm?\nThe authors should give state transition equations or figure to clarify it.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the time and comments on our paper! We address the reviewer's questions and the modifications we make to our paper based on the reviewer's suggestions below.\n\n**There are too few proof annotations on the test set (only 32), which weakens the solidity of the work.**\n\nWe believe the reviewer meant the training set (we always evaluate on the full test set of 19998 instances). In Table 4, we also show performance when training our method and the baselines on 64 instances. We observe a clear and consistent performance improvement, from 16, 32, to 64 samples. 64 samples is often considered an upper limit for few-shot learning (see for instance Alex et al. 2021 [1])\n\n**Multiple baselines use different pre-trained models, and this paper also uses external tools, the experimental comparison is not very fair.**\n\nTo clarify: we generally used the same pre-trained model, but deviated from it when it was *in favour* of the baselines:\n\n* T-Few, as the most powerful autoregressive approach, uses the same pre-trained model as QA-NatVer (BART0).\n\n* DeBERTav3+NLI, as the strongest three-way classifier, performs much better than BART0 when used directly as a classifier (41.6 versus 59.7 accuracy).\n\n* ProofVer uses GENRE (BART fine-tuned on entity-linking [2]) as it performed better than BART0 (only 34.3 accuracy). \n\nAssuming the reviewer is referring to the chunking and alignment system as the external tool, we would like to emphasize that these are commonly used models for these tasks and are typically incorporated into more complex pipelines like ours. Other baselines make use of tools as well and even additional data, such as ProoFVer with its entity-linking component (GENRE) or DeBERTav3 with the MNLI training data.\n\n**Question 1: The proof selection part isn't very clear, is this similar to the CYK algorithm?**\n\nWe will clarify the description of our proof selection in our paper. The selection is not similar to CYK, instead, the dynamic program is only used to construct all possible proofs efficiently. Each proof is then scored individually via a NatOp probability score (Eq. 2) and a NatOp verdict score (Eq. 3). Note that all proofs are constructed from 4 x l − 6 aligned claim-evidence spans as defined in section 3.1 (with l being the number of chunks the claim is initially decomposed into). The same span is used across multiple proofs but its NatOp probability score is only computed once, for efficiency reasons. For a more detailed discussion on QA-NatVer’s efficiency see our response to reviewer 1.\n\n**Missing Reference: LOREN**\n\nWe thank the reviewer for the additional reference and will incorporate it into our related work, see our response to reviewer 1 (in the section \"Missing reference: LOREN\") for comparison against it.\n\n[1] https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/ca46c1b9512a7a8315fa3c5a946e8265-Paper-round2.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a novel approach to fact verification that combines natural logic with question answering, which is a big contribution to the field.\n\n2. The proposed system outperforms existing baselines and state-of-the-art systems in terms of accuracy in a few-shot setting on FEVER.", "reasons_to_reject": "1. There are too few proof annotations on the test set (only 32), which weakens the solidity of the work.\n\n2. Multiple baselines use different pre-trained models, and this paper also uses external tools, the experimental comparison is not very fair.\n", "questions_to_authors": "1. The proof selection part isn't very clear, is this similar to the CYK algorithm?\nThe authors should give state transition equations or figure to clarify it.", "paper_topic_and_main_contributions": "This paper proposes QA-NatVer, a natural logic inference system for fact verification that frames natural logic operators in a QA setting.\nThe main contribution of the paper is a dynamic programming approach to proof selection that uses a multi-granular merging mechanism to construct proofs from non-overlapping claim chunks.\nThe paper also introduces a scoring function that combines NatOp probability scores and NatOp verdict scores to decode the optimal assignment.\nOverall, it provides a novel approach for fact verification that can improve the accuracy and explanation of the process.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Logic-Regularized Reasoning for Interpretable Fact Verification. AAAI 2022."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers overall agreed on the novelty of the approach and its good performance on multiple datasets. However, they also raised concerns about the limited applicability of the method to complex claims, the lack of discussions on integrating the approach into large language models, the efficiency of the pipeline, the scarcity of proof annotations on the test set, and the experimental fairness in the comparison with baselines.  The authors have provided a fair amount of discussion to the questions raised by the reviewers.
Decision: Accept-Main