Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need
Authors: Jinxuan Wu
WenHan Chao
Xian Zhou
Zhunchen Luo
Venue: EMNLP 2023 Main (2023)
Keywords: Scientific Claim Verification
Heterogeneous Graph
reasoning
TL;DR: Verify scientific claims with qualitative causal structure.

Abstract: A scientific claim typically begins with the formulation of a research question or hypothesis, which is a tentative statement or proposition about a phenomenon or relationship between variables. Within the realm of scientific claim verification, considerable research efforts have been dedicated to attention architectures and leveraging the text comprehension capabilities of Pre-trained Language Models (PLMs), yielding promising performances. However, these models overlook the causal structure information inherent in scientific claims, thereby failing to establish a comprehensive chain of causal inference. This paper delves into the exploration to highlight the crucial role of qualitative causal structure in characterizing and verifying scientific claims based on evidence. We organize the qualitative causal structure into a heterogeneous graph and propose a novel attention-based graph neural network model to facilitate causal reasoning across relevant causally-potent factors. Our experiments demonstrate that by solely utilizing the qualitative causal structure, the proposed model achieves comparable performance to PLM-based models. Furthermore, by incorporating semantic features, our model outperforms state-of-the-art approaches comprehensively.

Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need
Jinxuan Wu
Wenhan Chao
Xian Zhou
Zhunchen Luo
School of Computer Science and Engineering, Beihang University,
Information Research Center of Military Science, PLA Academy of Military Science,
{ (wujinxuan, chaowenhan @buaa edu. cn zhouxian@alumni. . sjtu. edu. cn,
Abstract
are tasked with identifying the essential elements, variables, or factors involved and furnishing com- prehensive comprehension of the claim's essence and extent.
A scientific claim typically begins with the for- mulation of a research question or hypothesis, which is tentative statement or proposition about a phenomenon or relationship between variables. Within the realm of scientific claim verification. considerable research efforts have been dedicated to attention architectures and leveraging the text comprehension capabilities of Pre-trained Language Models (PLMs), yield- ing promising performances. However, these models overlook the causal structure informa- tion inherent in scientific claims, thereby failing establish a comprehensive chain of causal in- ference. This paper delves into the exploration highlight the crucial role of qualitative causal structure in characterizing and verifying scien- tific claims based on evidence. We organize the qualitative causal structure into a hetero- gencous graph and propose novel attention- based graph neural network model to facili- tate causal reasoning across relevant causally- potent factors. Our experiments demonstrate that by solely utilizing the qualitative causal structure, the proposed model achieves parable performance to PLM- --se- models. Furthermore, by incorporating semantic fea- tures, our model outperforms state-of-the-art approaches comprehensively.
In the field of scientific claim verification, the majority of research efforts have been de- voted to dedicating attention architectures (Zhang et al., 2021) and leveraging the text comprehen- sion capabilities of Pre-trained Language Models (PLMs) (Sarrouti et al., 2021; Wadden et al., 2022) to improve performance. These studies follow the end-to-end paradigm and employ transformers the foundational framework, enabling the model to foeus on relevant parts of the claim and capture the essential elements and relationships within the text. However, despite their proficiency to capture key elements and their interactions, these models neglected the causal structure information inherent in the claims that should be characterized during the process of scientific claim verification, let alone establish comprehensive chain of causal inference between the claim and the evidence.
Following the schema proposed in SciClaim (Magnusson and Friedman, 2021), we provide example of qualitative causal structure in scien- tific claim in Figure 1. The claim describes the re- lationship between 'temperature" and 1COVID-19 transmission', which we identify as the key factors (highlighted in green). The core meaning conveyed in the claim is the causal connection between these factors, expressed by the term 'change' indicating an association (highlighted in red in Figure 1). The arrows from association to factors indicate that 'temperature' serves as the cause, while 'COVID- 19 transmission' is effected by this cause. The yellow spans represent modifiers of the association, denoting its likelihood, extent, or direction, con- straining the applicability or scope of the claim. As shown in Figure 1, there could also exist qualita- tive causal structure in the evidence.
1 Introduction
A scientific claim refers to statement propo- sition that puts forward particular hypothesis, theory, or discovery within the realm of science. The process of verifying scientific claim typi- cally entails thorough examination and assessment to ascertain its validity, relying on evidences that can support, refute, or remain neutral regarding the claim. In order to accomplish this, researchers must diligently serutinize and characterize the specific features, qualities, attributes of the claim. They
Corresponning authors.
!Our code is available at: https::/ / www.. com/ VulnDetector /CC.
Unlike factoid claim verification, a cognitive sys-

---

modiff
sffect
CLAIM:
Temperatures
significantly
changes
COVID-19 transmission
(sub)tropical cities Brazil.
factor
modifier
asvociation
factor
modiifeer
EVIDENCE:
The
weather
can
affect
the spread of coronavirus.
factor
modifier association
factor
modify
effect
Figure 1: An example of qualitative causal structures present in claim-evidencs pair.
tem developed for verifying scientific claims is an- ticipated to make sense of specialized descriptive language and employ causal reasoning concerning interactions between quantities or entities (Wad- den et al., 2020). Of utmost importance is the understanding of the qualitative causal structure, we hypothesize that it can characterize sci- entific claim and facilitate the traversal of causal reasoning among relevant causally-poters factors for verification. For instance, the widdnce--bsse verification of the claim in Figure can be achieved by exploring the relevant key factors and the asso- ciations that aggregate features from factors. More- over, the modifiers play crucial role in determin- ing the direction of traversal, e.g., replacing 'signif- icantly' with 'hardly' or 'never' could completely change the meaning of the claim.
of sub-graphs also provides insights into which sub- claims are supported by evidence and which are not. Through extensive experiments, we demon- strate that our proposed model, utilizing only the qualitative causal structure information, achieves comparable performance to Pre-trained Language Model (basM) models. Additionally, by in- corporating the semantic features of the original sentences, our model achieves state-of-the-art per- formance.
Related Work
2.1 Scientific Claim Verification
The fact-checking task aims to design model to automatically verify given claims. The FEVER task (Thome et al., 2018) is a typical fact-checking dataset where claims human-generated, and the evidence is extracted from Wikipedia. Recently, verifying scientific claims has drawn significant attention from the NLP community. There are serveral scientific claim verification datasets: Sci- Fact (Wadden et al., 2020), PubHcalth (Kotonya and Toni, 2020), COVID-Fact (Saakyan et al.. 2021), Health Ver (Sarrouti et al., 2021). Especially, Health constructs claims from snippets returned by a search engine for questions about COVID- According (Sarrouti et al., 2021), the claims from Health Ver are more complex, which is a more real-world situation setting.
With the hypothesis, the objective of this pa- per is to examine the role of qualitative causal structure to characterizing and verifying scien- tific claims. To achieve this goal, we employ the schema introduced in SciClaim (Magnusson and Friedman, 2021) to construct simplified qualita- tive causal structures for claim-evidence pairs in the datasets HEALTH V (Sarrouti et al., 2021) and SCINLI (Sadat and Caragea, 2022). These qualitative causal structures are then organized as heterogeneous graphs to enable reasoning. Fur- ther, we propose a novel attention-based graph neu- ral network model that performs causal reasoning between claim and evidences. Considering that compound claims need to be divided into several atomic subclaims, we introduce a graph decom- position algorithm and compute representations for sub-graphs, which are then aggregated using a weighted-sum operation to obtain final representa- tions for the entire graph. The weight distribution
Most current research formulates the claim veri- fication task as performing the Natural Language Inference (NLI) task (Chen et al., 2017; Ghacini et al., 2018; Parikh et al., 2016; Radford al., 2018; Peters al., 2018; Luken et al., 2018). Simi- larly, in scientific claim verification, most existing work concatenates the claim and the evidence to- gether to establish the semantic relation between

---

them. ARSJoint (Zhang et al., 2021) uses a hier- archical attention network (Pappas and Popescu- Belis, 2017) to perform word attention and sen- tence attention. MultiVerS (Wadden et al., 2022) concatenates the claim and each sentence of evi- dence together. By leveraging the NLI ability of the pre-trained language model and the pre-training procedure, MultiVerS achieved state-of-the-an per- formance the SciFact dataset. Similarly, the best performing model (Sarrouti al., 2021) on the Health Ver dataset also leverages the NLI ability the pre-trained model T5.
1OOL
can
pass
pass
angt
no proof
can
arg!
arg
argh
cats
to
to
the animal
arg0
arg
ung
virus
other cats
human
covid-19
Cats can pass the virus to other they discovered no proof that the animals can pass COVID-19 humans.
2.2 Causal structure in elaims
Several recent studies have integrated structural reasoning process into their models. GEAR (Zhou et al., 2019) proposes that interaction among sen- tences in the evidence should be considered and constructs fully-connected evidence graph by con- catenating the claim with each sentence in the evi- dence as a node and connecting all node pairs with edges. Similarly, KGAT (Liu et al., 2020) builds a graph in the same way but provides more fine- grained evidence selection using neural matching kernels. In cognative system, the schema proposed by SciClaim (Magnusson and Friedman, 2021) l) pro- vides meaningful approach. They develop multi- ple entity types such as Factors, Association, and Magnitudes, well construct different relations between these entities. The schema is adopted simplify complex claims (Wührl and Klinger, 2022) to solve the mismatch between trained mod- els on ScifFact and real-world claims.
Figure 2: An illustrative graph depicting the qualitative causal structure within the scientific claim. The edges in blue are modifying relations.
(Hu et al., 2020; Linmei et al., 2019; Hu et al.. 2020) since graph is a powerful tool to represent entities and their complex interactions.
Qualitative Causal Schema
SciClaim (Magnusson and Friedman, 2021) pro- poses a fine-gras sine-grained graph schema for scientific claims. We simplified the schema by removing cer- tain entity and relation types that are not essential for the task as follows.
Entities are labeled text spans. We aggregate six entity types into three: factor, association, and modifier, furthermore, to conduct aggregation, we design novel node type root. Factor remains unchanged as variables tested or asserted within a claim, typically nouns or noun-phrases, such as 'cats' and 'covid-19' in Figure 2. Association are the same the original schema, which is explicit phrases associating one or more factors that de- scribe the structural effect of the entity, such as 'pass' in Figure 2. Modifier is new entity type, which aggregates the original entity types magni- lude, evidence. epistemics, and qualifiers since they perform similarly scientific claim verification modifiers of the association. Root is designed to aggregate tree reprexentations into a single vector representing the whole scientific sentence.
2.3 Graph Representation and Reasoning
We also refer to the work of heterogeneous graphs when designing the model. Different from ho- mogenous graphs, heterogeneous graphs (Sun and Han, 2013) contain multiple node relation types, posing unique challenges for graph-based learning methods like Graph Convolutional Network (GCN) and Graph Attention Network (GAT) that can only handle homogenous graphs. To address this issue, meta-path (Sun et al., 201 1) is a typical method that leverages rich information in heterogeneous graphs, but it requires human knowledge to define meta- paths. HERec (Shi et al., 2018) is typical meta- path-based work that extracts several homogenous graphs using meta-path and performs aggregation on the representations learned on the homogenous graph respectively. Additionally, there has been research on applying heterogeneous graphs to NLP
Relations are directed edges between labeled en- tities, and preserve the original argO and argl relation types, which relate associations their cause or effect, respectively. We also retain the modifier edge with new name modifying to dis- tinguish the entity modifier. Modifying relates asso- ciations to their modifying components (modifiers

---

greater
more likely
Patienis
more likely
benefit from
rod
argf
argl
argi
ang
Patients greater baseline LAP volume
benellit from statin therapy
baseline IAP voltuee
greater
starin therapy
(a) The overlap annotation in SciClaim (Magnusson and Fried- man, 2021).
(b) Our proposed non-overlapped nested tree design notation.
Figure 3: entities red, yellow and green are associations, modifiers and factors repectively. We illustrate a typical antit--vvrrlappe annotation modification with the text: Patients with greater baseline LAP volume more likely to benefit from statin therapy.
in our schema), and comp_ to which represents comparison in scientific claim verification. How- ever, remove other high-order edges such q+ and q- since these relationships are contained in the semantics of associations.
4 Proposed Approach
In this section, we introduce the proposed model for scientific claim Verification by utilizing the Qualitative Causal Structure, also referred to as VerQCS. We explain the big picture of VerQCS here. VerQCS utilizes causal graphs within claims and evidence. The graph structures extracted by our dedicated extraction model. With both the graphs and the textual content, VerQCS performs reasoning and ultimately predicts the relationship. Modules like factor cross-attention, causal graph decomposition, and causal graph reasoning are in- troduced to capture the structural characteristics of claims and evidence, which are subsequently employed to predict the relationships.
In addition modifying the schema, propose the introduction of nested structure to replace the annotation setting that allows for entity overlap- ping. The original schema permits the overlapping two entity spans, posing a challenge for mod- els to accurately identify and differentiate between distinct entities. We present typical example of entity overlapping in Figure 3a, where the entire span 'Patients with greater baseline LAP volume' is annotated as a factor, despite the overlapping as- sociation 'greater'. Furthermore, the schema does not enable the subsequent model to concentrate on specific scientific entities within the sentence, such as "baseline LAP volume' and 'statin therapy', as these concepts are not extracted as entities. There- fore, we propose the nested design to decompose the overlapped entity into smaller, nov-overlg sub-trees present in Figure 3b, the span 'Pa- tients with greater bascline LAP volume' in Fig- ure 3a is decomposed into a sub-tree rooted by "greater" in Figure 3b. The tree structure enhances the model's 's comprehension of hierarchical relation- ships between entities and their modifiers, thereby facilitating more precise identification of relevant entities and relations for claim verification.
Prior introducing the model, two properties of the graph structure are introduced:
First, the associations always serve as the mot nodes of the trees. In Figure 2, both sub-claims have 'pass' as their root, which is an association; Second, if we designate the tree directly con- nected to the rool the root tree, we observe that root tree corresponds to the semantic span of sub-claims or sub-evidence. In Figure 3b, the sub- tree rooted by 'greater' corresponds to the noun- property phrase 'patients with greater baseline LAP volume" which does not provide any assertive infor- mation that can be argued or serve as evidence. We have observed that only the root tree corresponds to the semantic span, which is the tree rooted by 'benefit from' in the figure.
The properties we observe reveal the intrinsic connections between structure representations and semantic meaning. Sub-claim or sub-evidence can be extracted via identifying the root tree. Conse- quently, we employ structural reasoning on their respective root trees facilitate reasoning between sub-claims and sub-evidence.
Following the schema, we annotate the structure of each sentence in HEALTHVER Aataset(Sarrouu et al., 2021), which is complex scientific claim verification dataset. We proceed train joint entity and relation extraction model for extracting the sentence structures. Additionally, we utilize the extraction model to conduct experiments on another dataset SciNLI (Sadat and Caragea, 2022).
The tree representation learning module, which we further detail in section 4.5, encodes a tree struc- ture using an embedding vector and employs a re-

---

INPUT
Semanbe Regresentation
Graph
Embr.
Model
Apuregation
Embo.
Emby
Lunity
C
1
Ta
GGNN
Azero
Cross atfention
Descrppostoo Algorithm
factor graph
modifying
Emby
Prediction
Label
Label
Ensity
Cr
m
graph
Encoder
Graph
T
Homogenizatin
Pmbo
Extraction Model
Gp
Cross-attention
Causal Graph Decompasition
Cansal Graph Reasoning
Exideace: Eoi,, the reconery rate und mortallly rubk currelation ....
Scrantic Represeniation
Figure 4: The framework of VerQCS with an input example of green represent associations, modifiers factors respectively.
pair. The nodes in red, yellow and
cursive design to handle the nested tree structure. For instance, Figure 3b illustrates sample con- taining two trees. The input of the tree represen- tation module is the root tree rooted by 'benefit from'. Upon parsing the tree structure, the module identifies a dependent sub-tree rooted by 'greater'. Subsequently, the module recursively calls itself, generating an embedding of the sub-tree that sub- sequently utilized to generate the embedding of the root tree.
perform entity pooling:
cap(W (W. his + bb)
04,
Ljnh
ing
exp(W hij + b)
(1)
Cj ait hik
Where e, is the entity representation of entity: the matrix W maps the hidden representation token into a scaler.
4.3 Factor Cross-Attention
4.1 Joint Entity and Relation Extraction
We perform factor cross-attention between Ge and Gg, initializing each node with the corresponding entity embedding. The motivation of the compo- nent is that hypothesize the interaction between factors crucial for subsequent reasoning. This teraction should not only occur within the scientific claim or evidence individually, but also between the pair. Intuitively, if the evidence reasons the claim, it is likely that the factors share commonalities, such as co-re co-reference or domain domain-specific relationships. We concatenate all the factor entity representations: F ....
Using the annotated structure representations of sentences in HEALTHVER, we train joint model to extract the structure representations. VerQCS utilizes the predicted structures as input to perform structural reasoning and predicts the final relation- ships within the claim-evidence pairs.
Iustrated in Figure 4, the extraction model takes the claim and evidence texts as input. By represent- ing entities nodes and relationships edges, the extraction model predicts the structural graph of the claim (Gc) and evidence (Gg).
..., ................................ fei..... ... fe.. where na and nee are the numbers of factors in claims and evidence respectively, therefore F p(nctn,)xd, where d, is the dimension of the entity represen- tation. To perform factor cross-attention on factors from both sides:
factor interactions, we employ
4.2 Entity Encoder
The entity encoder generates embeddings for the extracted entities. The module consists of a to- ken encoder and entity pooling. Considering the scientific property, we use SciBERT (Beltagy al., 2019) as the token encoder generate to- ken embeddings. Let entity; hig.... denote the embeddings of the tokens in entity, we employ an attention-bar based entity representation to
Fnew (FV) softmar (FQ)- (FK)T (2)
ddg
Q.K.V the matrixes which map da dimdennnn vectors into dy.d; and dy dimensional vector space respectively. All factor in Go and Gg are then updated with the new embeddings, leading to the new graphs Go and Gg-

---

rection or credibility of the association. These two graphs offer distinct perspectives and contribute a comprehensive understanding. ings, we design the module to decompose the causal graph into trees and identify root trees. In Figure 4, since there are two trees in Gc, Tc1 and Tc2 are the decomposition results. The pseudocode is presented in Algorithm 1. The output E and R
4.4 Causal Graph Decomposition
Since the graph forms the structure of nested trees with root trees containing specific semantic mean-
Graph neural networks are employed to generate embeddings for these graphs. The embeddings the individual graphs are learned independently and subsequently concatenated to form the final embedding of the tree.
Emby Wtree (tree [reemodi fyigg) (3)
Algorithm 1: Decomposition Algorithm
Input: The graph of the scientific sentence:
Cross-attention Aggregation The module com- bines the embeddings of individual trees to create the graph embedding. For example, in Figure 4, the embedding of Gc. denoted as EmbGg. is cal- culated based on the embeddings of Tei and Tc2. Since sub-claims and sub-evidence are represented by root trees, the aggregation process focuses on the embeddings of these root trees.
Entities: (c1.02.... C::):
Relations: R = (Tiyles - ej
Function TreeGrow (e, (e,E,
foreach entity in {eile m e do
E = E U entily;
R = RRRrkee
if type(entity) association then flagstd/ False;
The rationale behind the aggregation is that if a sub-evidence tree strongly reasons a sub-claim tree, the weight of the sub-evidence should be assigned higher value when calculating the embedding of root for evidence, and the same applies to the claim. To describe the relationships between trees in the scientific evidence and claim, utilize matrix S E R Xn,, where Tee and ne denote the number of root trees in the claim and evidence, respectively.
return E, R.R.flags;
else
L TreGGrowentittyEERR.foo
return E,R.flags;
associations set: A = {a,,a......... am, flags = list(m);
Initialize flags with True; foreach in A do
The aggregation process conducted by leveraging the matrix S as follows:
F = Φ; R= d;
Output: flR.flags.
W sim EMbTO, Wsim EmbTE
8ij
Vagim
represent the entities and relations of each tree, re- spectively, with as the root. The flags variable indicates root trees.
Welaim soʃ soffmaa max S) dim=0
the Welaim,
roolelamm Wevi
EmbTO (4)
4.5 Causal Graph Reasoning
i=1
sofftmat max S)
Tree Representation Learning The objective of the proposed component is to represent the tree structure using embeddings, thereby preserving the structural information. However, since the tree a heterogeneous graph with multiple relations and entities, we propose graph homogenization as a solution to split the heterogeneous graph into two graphs: the factor graph and the modifying graph. The factor graph is obtained by preserving the fac- tors and associations with their connecting edges, while the modifying graph consists of the associa- tions and modifiers with modifying relations. This division is based on the notion that the factors are noun properties, while the modifiers modify the di-
dim=1
The
rool
Wevi;
Embres
Let Sij denote the scalar value in matrix S. Wim be the mapping function that projects the tree em- bedding into dsim -iimenninai space, and Welaim and Wevi represent the computed weights for claim and evidence root trees, respectively. Additionally, Viham serves as the scaling factor, max generates dim=0
an n.-dimensional vector by selecting the maxi- mum value along each row, and max indicates dim=1
selection along each column.

---

Label Prediction In our final prediction, we ac- knowledge the value of semantics as an advanta- geous complement to the structural representation. Consequently, we concatenate the claim and evi- dence tokens using [SEP] as a separator, consider- ing the oulput of the [CLS] token as the semantic feature. Subsequently, we combine the structural and semantic representations of the pair, employing cross-entropy the loss function.
model is presented in Appendix A.1. In our experiments conducted on the HEALTHVER dataset, we compare the per- formance of VerQCS with T3 (Raffel et al., 2020), which the current state-of-the-art model, as well as similar PLMs such as BERT (Devlín et al., 2019) and SciBERT (Beltagy et al., 2019), as presented in Table 1. Since semantic models do not require structure input, there is no performance difference between the HEALTHVER ERtuhh and HEALTHVERpred settings.
5 Experiment
In this study, we perform two distinct experiments: structure extraction and scientific claim verifica- tion, respectively. We present the experiment of scientific claim verification in the section, more details about structure extraction are presented in Appendix A.I.
Models denoted by StrOnly represent semantic- feature-free reasoning models, and the sub- script indicates the adopted GNN method. In HEALTH the performance of StrOnlyGCN is superior to BERT, when introduc- ing the error in the parsing model, StrOnly models perform slightly worse than PLMs. Therefore, we hypothesize that by incorporating the extracted reasoning element information alone, the model can achieve comparable performance models fine-tuned on PLMs. Furthermore, observed that when the StrOnly models are augmented with semantic features, VerQCS outperforms these PLMs, even in the presence of errors introduced by the parsing model.
5.1 Experimental Setup
Dataset The extraction model is trained on the training set of HEALTHVER since we annotate the structure the dataset. We train VerQCS on the training set with the ground truth of the struc- tures, and VerQCS is required to predict the cases in test with the structure predicted by the ex- traction model. To evaluate how the error in the extraction model may propagate to VerQCS, we also leverage the ground truth structures in test for comparison. We name the experiment settings as V and HEALTH VERL respec- tively. We also experiment SciNLI (Sadat and Caragea, 2022) without any an- notation. The structures in SciNLI are derived from the parsing model trained on HEALTHVER.
We observe that in StrOnly models, GCN is a better choice than GAT, which appears to contra- dict findings in previous studies. We hypothesize that the input the GNN consists of decomposed factor or modifying graph on the extracted trees. These graphs consist one association and several nodes with the same label, lacking complex struc- tures and numerous nodes. Consequently, GAT, which has more parameters than GCN, may tend to overfit the input. Additionally, the attention layer in GAT may provide limited performance gains due the presence of nodes belonging the same class in the graph.
another dataset
Implementation Details In scientific claim ver- ification, We optimize VerQCS with AdamW (Loshchilov and Hutter, 2017) for 50 epochs with learning rate of 4e-5 and weight decay of le- 2. C.Coniideiing the scientific nature of the task, we employ SciBERT (Beltagy al., 2019) as the token embedding layer. Within the factor cross-ates attoenatten module, we set the multi-head parameter to 3 to account for the three different roles. The entity dimension remains consistent at 768 for both factor cross-attention and tree representation learning. In the training process, the batch size to 1 with the accumulation step set 32.
Additionally, we evaluate performance by com- paring with a large language model: GPT-4. To perform reasoning with GPT-4, we utilize the API provided by OpenAl. The prompts are de- signed with three distinct settings. "Instruct GPT-4" prompts contain solely the task description. "3-shot GPT-4" refers to demonstrations of GPT-4 using three sample cases, with one case for each label. "6- shot GPT-4" involves six cases, each featuring the task description, and two cases for each label. The results are represented in Table 2. We conclude that in the "Instruct GPT-4" setting, where only
5.2 Overall Performance
We present the result of scientific claim verification experiment here. The performance of the extraction

---

HEALTHV VER
HEALTHVER pred
Model
F1
rec
F1
BERT SciBERT
75.19
73.71
74.17
75.19
73.71
74.17
80.51
78.14
78.71
80.51
78.14
78.71
81.10
80.23
80.52
81.10
80.23
80.52
StrOnlyccw
76.58
75.17
75.24
73.12
73.00
73.05
Surnlyear
74.89
73.66
74.10
73.01
72.34
72.66
VerQCS
82.79
81.36
82.14
81.10
81.65
81.33
w/o Factor Cross-attention
79.08
77.20
77.68
76.37
77.10
76.66
w/o Causal Graph Decomposition
81,10
80.23
80.52
79.21
79.93
79.45
w/o Graph Homogenization
76.67
75.22
75.64
74.90
75.32
75.11
Table 1: Experimental results of the comparison methods. StrOnly models exclusively utilize the qualitative causal structure information, And the models started with "w/o" indicate the approaches considered in the ablation study.
root
QD
poss
pass
argo
arg
proof
had occurred
further transmission events
"90
the onimal
orgl
persons
argl
animals
to
angit
the virus
other
human
covid
other animals
Cats can pass the virus to other cats, they discovered no proof that the animals can pass COVID-19 to humans.
No further transmission events from animals to other animals or persons had occurred.
The weight distribution of the claim.
(b) The weight distribution of the evidence.
Figure 5: An example where the evidence refutes the claim.
Prompt construction
F1
Model
C
R
E
N
F1
Instruct GPT-4
68.08
BERT
77.46
71.74
75.09
76.47
75.19
3-shot GPT-4
68.50
SciBERT
80.30
74.18
75.90
79.76
77.53
6-shot GPT-4
71.05
RoBERTa
81.18
74.22
77.99
78.86
78.06
XLNet
81.53
75.95
77.63
77.63
78.18
Table 2: Performance of GPT-4 with different settings.
StrOnlyacN
71.15
68.33
69.14
71.00
70.66
StrOnlyGAT
72.30
67.77
70.17
62.42
68.16
the task instruction was provided, GPT-4's perfor- mance less impressive. By providing GPT-4 with example cases and leveraging its in-context learning ability, performance improved as more examples were given. We observed that GPT-4's performance was comparable to fine-tuned BERT models. This aligns with findings from similar research (Liu et al., 2023).
VerQCS
83.46
75.20
77.44
80.08
79.27
Table Experimental results SciNLI.
ing model being trained annotated data from HEALTHVER, which leads difficulties in gener- alization to SCINLI. Nevertheless, VerQCS contin- ues to outperform the PLMs. More details about the analysis are presented Appendix A.I.
conducted experiments SCINLI, and the corresponding performance is presented in Table 3. The StrOnly models exhibit lower performance compared to PLMs. We hypoth- esize that this discrepancy arises due to the pars-
Furthermore,
5.3 Ablation Study
To demonstrate the effectiveness the designed components in our model, we conducted ablation

---

studies where we removed factor cross-attention. causal graph decomposition, and graph homoge- nization. Based on the results presented in Table I, the ablation study provides evidence of the effec- tiveness of these components. Without factor cross- attention, the performance drops, indicating the interaction among. anoooooccooos is the key to conduct verification. We remove the graph decomposition with model reasoning directly the entire graph. The result demonstrates the effectiveness of the component. In our reasoning process on the hetero- gencous graph, we partition the graph into a factor graph and a modifying graph. By conducting an ablation experiment that removes graph homog- enization, we provide evidence of the method's effectiveness.
qualitative causal structures in claim verification. Morcover, we hypothesize that the factor entities within causal graphs can be seamlessly linked external knowledge graphs, thus offering opportu- nities for further exploration.
Limitations
We suppose the VerQCS contains two major lim- itations. The first limitation the choice of the strcture extraction model, we leverage an entity re- lation extraction model SpERT, while the annotated structure adheres to the tree structure that closely resembles the dependency parsing task, we have not yet explored whether utilizing successful meth- ods from dependency parsing could yield improved performance. The second limitation pertains our method operating pipeline paradigm, where an extraction model predicts structures and VerQCS conducts further reasoning using the predictions as input. The experiment results reveal an issue of error propagation. we hypothesize that employing joint model could be more effective approach, and this avenue will be explored in our future work.
5.4 Case Study
In the case study, we aim present the weight distribution assigned the root trees by VerQCS with a typical case Figure 5. As depicted in Fig- ure 5a, the claim consists of two sub-claims, each of which is mapped to a root tree in the graph repre- sentation while the structure of the evidence forms a single tree in Figure 5b. The evidence refutes the claim since the first sub-claim conveys the con- flicting information, although the second sub-claim is consistent with the evidence. VerQCS has the capability to reveal which sub-claim is reasoned by evidence by assigning the weight distribution over trees, the tree corresponds to the first sub-claim is assigned with weight 1.0 in the case. Therefore, the weight distribution provides insights into how the reasoning process is conducted in VerQCS, which can be leveraged to further help researchers.
Ethics Statement
Scientific claim verification and fact-checking tasks are widely studied tasks in NLP, and we do not iden- tify any notable ethical concerns associated with these tasks. Furthermore, the datasets utilized in our study are publicly available. While the Sci- Claim schema of the graph is derived from pub- lished work, we obtained permission and approval from the original author to modify the schema for better alignment with our task.
We also illustrate a more complex case where both claim and evidence contain multiple root trees in Appendix A.2.
A Acknowledgements
This work supported by National Natural Sci- ence Foundation of China (No. 61976221). We would like to express our gratitude the authors of SciClaim for their invaluable assistance. We also extend our appreciation to the reviewers for their patience and valuable feedback.
6 Conclusions
This paper introduces the utilization of qualitative causal structure in scientific claim verification. To this end, we propose the novel model VerQCS, which performs structural reasoning on the causal graph. The framework has been demonstrated to be effective even without semantic features enhancing. Additionally, VerQCS demonstrates its capability of selecting sub-claims and sub-evidence by as- signing weights to the sub-graphs, which provides insights of how VerQCS conducts reasoning. In conclusion, we have substantiated the efficacy of
References
Iz Beltagy, Kyle Lo. and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNIPP) pages 3615-

---

3620, Hong Kong, China. Association for Computa- tional Linguistics.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced Istm for natural language inference. arXiv preprint 160.00669..0038.
Jackson Luken, Nanjiang Jiang, and Cathe-Catherine de Marneffe. 2018. QED: A fact verification sys- tem for the FEVER shared task. In Proceedings of the First Workshop on Fact Extruction and VERifi- cation (FEVER), pages 156-160, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
lan Magnusson and Scott Friedman. 2021. Extracting fine- ane-grained knowledge graphs of scientific claims: Dataset and transformer-based results. In Proceed- ings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 4651-4658, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Markus Eberts and Adrian Ulges. 2019. Span-based joint entity and relation extraction with transformer pre-training, arXiv preprint
Nikolaos Pappas and Andrei Popescu-Belis. 2017. Mul- tilingual hierarchical attention networks for docu- ment classification. In Proceedings of the Eighth International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 1015- 1025, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Reza Ghaeini, Sadid A. Hasan, Vivek Datla, Joey Liu, Kathy Lee, Ashequl Qadir, Yuan Ling, Aaditya Prakash, Xiaoli Femm, and Oladimeji Farri. 2018. DR- Dependent reading bidirectional LSTM for natural language inference. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Pa- pers), pages 1460-1469, New Orleans, Louisiana. Association for Computational Linguistics.
BiLSTM:
Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference Empirical Methods in Nat- ural Language Processing, pages 2249-2255, Austin, Texas, Association for Computational Linguistics.
Linmei Hu, Chen Li, Chuan Shi, Cheng Yang, and Chao Shao. 2020. Graph neural news recommendation with long-term and short-term interest modeling. In- formation Processing & Management, 57(2):102142..
Matthew E. Peters, Mark Neumann, Mohit Tyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7740-7754, Online. Association for Computa- tional Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training. OpenAl.
Hu Linmei, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. 2019. Heterogeneous graph attention net- works for semi-supervised short text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- pages 4821-
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(():5485-5511
guage Processing 4830, Hong Kong, China. Association for Computa- tional Linguistics.
Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. Covid-fact: Fact extraction and veri- fication of real-world claims on covid-19 19 pandemic. arXiv preprint 2eXiv:2106.0777
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng. Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the logical reasoning ability of chatgpt and gpt-4.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342-7351, On- line. Association for Computational Linguistics.
Mobashir Sadat and Cornelia Caragea. 2022. SciNLI: A corpus for natural language inference on scientific text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Papers). pages 7399-7409, Dublin, Ireland, Association for Computational Linguistics.

---

Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fusbman. 2021. Evidence- wridence-based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499-3512, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
claim
cvidence
Statisties
Train
Dev
Test
Train
Dev
Test
entity/token relation/entity
3.13
3.15
3.32
3.99
4.67
3.91
1.25
1.22
1.45
1.21
1.19
0.88
(a) The statistics of HEALTHVER
Chuan Shi, Binbin Hu, Wayne Xin Zhao, and S Yu Philip. 2018. Heterogeneous information network embedding for recommendation. IEEE Transactions on Knowledge and Data Engineering, (2):3357 370.
Statistics
sentencel Dev
sentence2
Train
Test
Train
Dev
Test
entity/token
5.05
4.98
4.88
5.02
4.86
4.49
relation/entity
2.25
2.16
2.09
2,30
2.24
2.22
(b) The statistics of SCINLI
Yizhou Sun and Jiawei Han. 2013. Mining hetero- geneous information networks: structural analy- sis approach. Acm Sigkdd Explorations Newsletter, 14(2):20-28.
Table 4: The statistics of the datasets. The entity/token is the average number of entities divided by the average number of tokens, indicating the density of entities, relation/entity indicates the density of relations.
Yizhou Sun, Jiawei Han, Xifeng Yan, Philip Yu, and Tianyi Wu. 2011. Pathsim: Meta path-based
top-k similarity search in heterogeneous information networks. Proceedings of the VLDB Endowment, 4(1 4(11):992-1033
A Appendix
A.1 Additional Expriment Result
James Thore,, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. 2018. Fever: large-scale dataset for fact extraction and verification. arXiv preprint 18030558..
More details about the experiments are presented in the section.
Joint Entity Relation Extraction The extraction model is trained on the training set of HEALTHVER since we annotate the structure on the dataset. For the structure extraction task, we leverage SpERT(Eberts and Ulges, 2019) to perform the structure extraction, the batch size is set to 32 with 50 epochs. The performance of the model in struc- ture extraction is presented in Table 5. The model can achieve an Fl-score of over 80 in entity ex- traction. However, it struggles to perform well in relation construction, as this task requires strong logical reasoning ability.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. As- sociation for Computational Linguistics.
David Wadden, Kyle Lo. Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. Mul- ti veVe::: Improving scientific claim verification with weak supervision and full-document context. In Find- ings of the Association for Computational Linguistics: NAACI 2022, pages 61-76, Seattle, United States. Association for Computational Linguistics.
Amelie Wührl and Roman Klinger. 2022. Entity-based claim representation improves fact-checking of medi- cal content in tweets. In Proceedings of 9th Workshop on Argument Mining, page 187.
Label Prec
Rec
Fi-score
association
91.05
90.00
90.52
factor
87.45
86.20
86.82
modifiers
82.02
85,98
83.95
Zhiwei Zhang, Jiyi Li, Fumiyo Fukumoto, and Yanming Ye. 2021. Abstract, rationale, stance: A joint model for scientific claim verification. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 3580-3586, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
argO 80.77
80.77
80.77
argl 80.19
73.07
76.46
comp to 53.33
42.11
47.06
modifying
69.10
75.93
72.36
Table 5: the performance of the joint extraction model.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and rea- soning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Compu- lational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.
Dataset Statisties Additional dataset statistics are provided in Table 4, where both the entity/token ratio and the relation/entity ratio in SCINLI signif- icantly surpass those HEALTHVER. This dis- crepancy suggests that the extraction model inade-

---

0.01
0,99
carly
wise
0.1
0.09
determine
masintai
disinfect
peotection against
kill
how
nondrinkers
lowrisk drinkers
that status
the virus
COVID-I
for
alcohol
affect
covid- 19
Consumption alcohol
in the inhaled air
progression
(a) The weight distribution of the claim.
(b) The weight distribution of the evidence.
claims: Consumption of alcohol will not kill the virus in the inhaled air; will not disinfect your mouth and throat; and will not give you any kind of protection against COVID-19.
evidence: While remains much too early to determine precisely how alcohol use might affect risk for or progression of COVID19. any efforts to connect the dots from past alcohol research would suggest it wise for nondrinkers and lowrisk drinkers to maintain that status during this public health crisis.
Figure 6: complex case where evidence Supports the claim.
quately extracts fine-grained reasoning elements in SCINLI.
A.2 Complex Case
We illustrate more complex case where both claim and evidence contain multiple trees in Figure 6, the weight distributions on trees of claim and evidence disclose the fine-grained relationships on tree level, which reveals the functional sub-claim and sub- evidence in the reasoning process.


Reviews:
[{"review_id": "hHux11RR3m", "reviewer": "Reviewer_7skt", "summary": "", "questions": "1. In Algorithm 1, where did you use flags? I see only setting flags to False but they are not used in anywhere.\n2. What were the performances of proposed models for building the ground truth tree structures?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review of our paper and for raising important questions. We deeply appreciate your acknowledgment of the novelty and significance of our approach. Additionally, we are grateful for your patience and thoughtful feedback on typos, grammar, style, and presentation-related issues.  We provide answers to your questions below. \n\n**Q1**:Baselines could be improved by fine-tuning or prompt tuning on HEALTHVER tasks for more fair comparison as newly proposed models were trained on these tasks specifically.   \n**A1**: With regard to the training of baselines, we would like to kindly clarify that they were indeed fine-tuned using the HealthVER training set, and the performance presented in the paper aligns comparably with that in the HealthVER paper. For further elaboration and comprehensive training specifics, we invite you to refer to the dataset section in Section 5.1 of our paper.\n\n**Q2**: In Algorithm 1, where did you use flags? I see only setting flags to False but they are not used in anywhere.    \n**A2**: The notation \"flags\" in Algorithm 1 is utilized in cross-attention aggregation and serves to denote whether a tree, rooted by an association, qualifies as a root tree. Indeed, the \"flag\" notation is briefly mentioned in section 4.4 and we acknowledge that it lacks a comprehensive explanation within the algorithm. We've found a better way to rectify this and ensure that the role of the \"flag\" is clearly articulated. Your feedback has been invaluable in this regard, and we appreciate your attention to this matter.\n\n**Q3**: What were the performances of the proposed models for building the ground truth tree structures?    \n**A3**: In consideration of the space constraints, we have included the performance results related to building the tree structures in Table 3, which can be found in Appendix A.1.\n\nOnce again, we extend our gratitude for your meticulous assessment of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and the models are thoroughly analyzed by architectural variants and ablation tests.\n- Proposed schema can capture more fine-grained \n- The new model VerQCS obtained the highest performance for HEALTHVER dataset tasks.\n", "reasons_to_reject": "- Baselines could be improved by fine-tuning or prompt tuning on HEALTHVER tasks for more fair comparison as newly proposed models were trained on these tasks specifically.", "questions_to_authors": "1. In Algorithm 1, where did you use flags? I see only setting flags to False but they are not used in anywhere.\n2. What were the performances of proposed models for building the ground truth tree structures?\n", "paper_topic_and_main_contributions": "The authors propose a novel pipeline for scientific claim verification leveraging qualitative causal structures “Verification by utilizing the Qualitative Causal Structure (VerQCS)”. Specifically, it extracts entities (which gets encoded by SciBERT for factor entity representations) and generates graphs for claims and evidence. Then, it calculates factor cross-attention for generating new claim and evidence graphs and decomposes them to tree structures. Then, those tree structures get homogenize to factor graphs and modifying graphs which get encoded as final tree embeddings. These embeddings get aggregated using cross-attention. Finally, the model predicts labels using both this structural representation and semantic features extracted by ‘claim [SEP] evidence’.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- P.2 L142-143 “COVID-19 ,according to (Sarrouti et al.,2021)” ->  “COVID-19. According to (Sarrouti et al.,2021)”\n- L138 ‘2020),PubHealth’ -> ‘2020), PubHealth’\n- L139 ‘2020),COVID-Fact’ -> L139 ‘2020),COVID-Fact’\n- L140 ‘,HealthVer’ -> ‘ ,HealthVer’\n- L143 ‘,according to’ -> ‘, according to’\n- L183 ‘claims(‘ -> ‘, ‘claims (‘\n- L216 ‘Factor remain’ -> ‘Factor remains’\n- L235 ‘modifier,.' -> 'modifier.'\n- L237 'comp_to,' -> 'comp_to'\n"}}, {"review_id": "WBwMMw1aI8", "reviewer": "Reviewer_8wEx", "summary": "", "questions": "The method uses SciBERT embedding layer - How important is this component in performance?\n\nAre there any other graph-based baselines?\n\nIt would be interesting to see the effectiveness of the proposed model in low-label scenarios or domain generalization scenarios, compared to existing models -  Are such experiments possible?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our gratitude for your thorough review of our paper and for bringing forth significant questions. Your recognition of the uniqueness and importance of our approach is greatly valued. We provide answers to your questions below. \n\n**Q1**: The method uses SciBERT embedding layer - How important is this component in performance?    \n**A1**: Thanks for raising this question. We are also interested in assessing the significance of SciBERT as an embedding layer. Hence, we have conducted the experiments to substitute SciBERT with BERT on $HealthVER_{truth}$ dataset. Here is the result:    \n|          Model          |          F1(SciBERT)          | F1(BERT) |\n| ----- | ----- | ---- |\n| $StrOnly_{GCN}$ | 75.24 | 73.19 |\n| $StrOnly_{GAT}$ | 74.10 | 72.22 |\n| $VerQCS$ | 82.14 | 80.23 |    \nFrom the results, we have the following observations:\n- The substitution leads to an average 2-point decrease in F1-score, indicating the crucial role of SciBERT in enhancing our model's performance.    \n- VerQCS with SciBERT as the embedding layer outperforms PLMs using SciBERT. Even when replacing SciBERT with BERT as the embedding layer, VerQCS still outperforms PLMs using BERT, which demonstrates the effectiveness of VerQCS.\n\n**Q2**: Are there any other graph-based baselines?      \n**A2**: We wish to kindly emphasize that VerQCS is the first work of entity-level graph-based approach in scientific claim verification. While we are aware of similar graph-based methods like GEAR[1], which uses GCNs with sentences as nodes, it's important to highlight that GEAR primarily focuses on paragraph-level evidence and doesn't delve into the causal structure within claims and evidence.\n\n**Q3**: It would be interesting to see the effectiveness of the proposed model in low-label scenarios or domain generalization scenarios, compared to existing models - Are such experiments possible?    \n**A3**: We have devised a domain transfer experiment in which we assess the performance of models fine-tuned in HealthVER when applied directly to SciNLI in a zero-shot scenario. Here is the result:\n   | Model           | F1(fine-tuned) | F1(zero-shot) | Performance Drop | percent |\n   | --------------- | -------------- | ------------- | ---------------- | ------- |\n   | BERT            | 79.20          | 50.98         | 28.22            | 35.63   |\n   | SciBERT         | 80.13          | 53.19         | 26.94            | 33.62   |\n   | $StrOnly_{GCN}$ | 73.10          | 51.66         | 21.44            | 29.33   |\n   | $StrOnly_{GAT}$ | 70.89          | 50.42         | 20.47            | 28.88   |\n   | VerQCS          | 81.33          | 55.76         | 25.57            | 31.44   |    \nThe classification task differs between SciNLI and HealthVER. We combined entailment and reasoning into a support label, while the contrasting label is considered refuting, with neutral labels remaining unchanged. To ensure a fair comparison, we fine-tuned models in SciNLI using the same labels as HealthVER, resulting in an improved F1 score compared to the results in Table 2 of our paper. Our observations can be summarized as follows:\n- Adapting models fine-tuned on HealthVER for zero-shot evaluation in SciNLI resulted in an average F1-score drop of over 20 points. This experiment highlighted an interesting phenomenon: while semantic models showed greater performance degradation, models grounded in causal structure reasoning demonstrated greater robustness.   \n- We find this difference reasonable since dataset distributions vary based on language style and construction. However, the difference in causal graphs is likely to be smaller than that in language. \n\nOnce again, we extend our gratitude for your meticulous assessment of our paper.\n\nReference:   \n[1] Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019.GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 892–901, Florence, Italy. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed model is well motivated and has good performance. Reasoning over causal structures in claim verification is an interesting research direction and the paper would encourage further research in this area.\n\nThe model is verified on two datasets, HealthVER and SciNLI.\n\nThe semantic-free models perform on par or better than PLM-based ones, and the model performs well with and without ground truth structure information.", "reasons_to_reject": "My main concern is the limited number of evaluated datasets, and the clarity of writing.", "questions_to_authors": "The method uses SciBERT embedding layer - How important is this component in performance?\n\nAre there any other graph-based baselines?\n\nIt would be interesting to see the effectiveness of the proposed model in low-label scenarios or domain generalization scenarios, compared to existing models -  Are such experiments possible?", "paper_topic_and_main_contributions": "This paper proposes a scientific claim verification model based on a GNN model operating on causal structure. The proposed model constructs a tree representing causal structure for both claim and evidence, and compares the two structures to predict the label. The proposed model outperforms the current state of the art (T5) on HealthVER claim dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lg6NzshuWr", "reviewer": "Reviewer_54T1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and raising your concerns about the work. Below, we'd like to provide our insights.\n\n**Q1**: The authors introduced a very complex pipeline and fail to explain clearly the big picture. I found the paper quite difficult to follow.      \n**A1**: We appreciate your feedback and would like to provide a more concise response.\n- This paper focuses on emphasizing the pivotal role of qualitative causal structures in characterizing and verifying scientific claims based on evidence. We organize these structures into a heterogeneous graph and propose an attention-based graph neural network called VerQCS to facilitate causal reasoning across relevant causally-potent factors. Our pipeline comprises three main steps: 1) causal structure extraction (Section 4.1), 2) entity/factor representation learning (Sections 4.2 and 4.3), and 3) causal graph decomposition and reasoning (Sections 4.4 and 4.5).    \n- We regret any potential confusion by not including an introduction to the general framework in Section 4, as there is so much we would like to present but space does not permit. We deeply regret any inconvenience this may have caused. In response, we are committed to a careful paper revision, which will include an introduction to the overall framework and a more comprehensive explanation of each model component's motivation. \n\n**Q2**: The examples justifying each component are not convincing enough.   \n**A2**: We appreciate your feedback. To clarify the motivation behind the examples, we provide the following explanations:   \n- Figure 1 introduces the use of causal structures for verification.\n- Figure 2 illustrates the schema of the heterogeneous graph.\n- Figure 3 highlights the differences between the original and modified schemas through a typical example.\n- Figure 5 presents a graph representation of a case study, showcasing the enhanced capabilities of VerQCS, which we believe can contribute to the understanding of performance improvements.   \nWe hope these clarifications provide a more compelling rationale for the examples used in our paper.\n\n**A3**: The paper claims to compare with state-of-the-art models while the systems compared (BERT, SciBERT and T5) are quite old. Instruct LLMs are not compared.   \n**Q3**: Thanks for your question. Your insights have made a meaningful contribution to our future endeavors. We were also curious about the performance of Large Language Models (LLMs) in the reasoning task, so we conducted experiments using the GPT-4 API. The results are as follows:   \n| Approach                     | F1 score   |\n| ---------------------------------- | ----- |\n| Instruct GPT-4                   | 68.08 |\n| 3-shot GPT-4                 | 68.50 |\n| 6-shot GPT-4                 | 71.05 |\n| BERT                                   | 74.17 |\n| $StrOnly_{GCN}$                  | 75.24 |\n| $StrOnly_{GAT}$                  | 74.10 |\n| $VerQCS$                         | 82.14 |\n\nWe evaluated GPT-4 in different scenarios:   \n- In the \"Instruct GPT-4\" setting, where only the task instruction was provided, GPT-4's performance was less impressive.\n- By providing GPT-4 with example cases and leveraging its in-context learning ability, performance improved as more examples were given. We observed that GPT-4's performance was comparable to fine-tuned BERT models. This aligns with findings from a related paper[1].\n\nIn summary, our results suggest that while Large Language Models (LLMs) demonstrate robust semantic understanding of general text, their capabilities in reasoning for scientific claim verification tasks are an area worthy of further exploration. Your attention to this matter is greatly appreciated.\n\nOnce again, we extend our gratitude for your meticulous assessment of our paper.\n\nReference:   \n[1] Hanmeng Liu and Ruoxi Ning and Zhiyang Teng and Jian Liu and Qiji Zhou and Yue Zhang.2023.Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4.arXiv", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Experiments seem correctly conducted\n- The model offers a means to decompose complex scientific claims into atomic subclaims, allowing for more detailed scrutiny and interpretation of the claims.", "reasons_to_reject": "- The paper claims to compare with state-of-the-art models while the systems compared (BERT, SciBERT and T5) are quite old. Instruct LLMs are not compared.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a graph-based method for verifying scientific claims. The authors introduce a complex pipeline made of 5 modules.  They present an attention-based graph neural network model that facilitates the reasoning of causally crucial factors in scientific claims.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a scientific claim verification model based on a GNN model operating on causal structure. The proposed model constructs a tree representing causal structure for both claim and evidence, and compares the two structures to predict the label. The proposed model outperforms the current state of the art (T5) on HealthVER claim dataset.

The shared perspective of the reviewers include:
* the topic is very interesting
* the proposed model is well-motivated and the performance is extremely promising
* the work presented here is not only interesting within the scope of the paper, but it opens the door to research such as more fine-grained structures or in general the topic of reasoning over casual structures in claim verification

The issues of concern for all reviewers seem to be aspects of the paper or the analysis which can be addressed in a camera-ready, such as implementation using LLMs. In response, the authors seem to have thoroughly addressed each point clearly. 

Based on the comments by the reviewers, I would hope the authors dedicate time to improve the overall clarity in the writing in a final version of the paper.
Decision: Accept-Main