Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation
Authors: Xin Yuan
Jie Guo
Weidong Qiu
Zheng Huang
Shujun Li
Venue: EMNLP 2023 Main (2023)
Keywords: out-of-context
stance analysis
misinformation
Internet evidence
TL;DR: A method for detecting out-of-context mis- and disinformation by analyzing the stance of internet evidence

Abstract: Mis- and disinformation online have become a major societal problem as major sources of online harms of different kinds. One common form of mis- and disinformation is out-of-context (OOC) information, where different pieces of information are falsely associated, e.g., a real image combined with a false textual caption or a misleading textual description. Although some past studies have attempted to defend against OOC mis- and disinformation through external evidence, they tend to disregard the role of different pieces of evidence with different stances. Motivated by the intuition that the stance of evidence represents a bias towards different detection results, we propose a stance extraction network (SEN) that can extract the stances of different pieces of multi-modal evidence in a unified framework. Moreover, we introduce a support-refutation score calculated based on the co-occurrence relations of named entities into the textual SEN. Extensive experiments on a public large-scale dataset demonstrated that our proposed method outperformed the state-of-the-art baselines, with the best model achieving a performance gain of 3.2% in accuracy.

Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis- and Disinformation
Xin Yuan and Jie Guo' and Weidong Qiu and Zheng Huang School of Cyber Science and Engineering, Shanghai Jiao Tong University, China [yuanxin, guojie, qiuwd, huang- wuang-zheng)esjtue edu. cn
Shujun Li
Institute of Cyber Security for Society (iCSS) & School of Computing, University of Kent, UK S. S.J.Li@kent.ae ac.
Abstract
platforms such images and short videos has led to more and more mis- and disinformation in differ- ent modalities beyond text (Vosoughi et al., 2018; Wang et al., 2021). In web pages such as news articles, texts and images often co-exist but they may not be correctly associated, therefore leading to the so-called out-of-context (OOC) mis- and disinformation where readers may be misled to be- lieve some false narratives caused by such false or inaccurate text-image associations (Thomas and Kovashka, 2020). OOC mis- and disinformation have become a common phenomenon on many on- line platforms, and have received attention from many researchers recently (Luo et al., 2021; Aneja et al., 2023; Abdelnabi et al., 2022; Zhang et al., 2023).
Mis- and disinformation online have become major societal problem major sources of online harms of different kinds. One com- mon form of mis- and disinformation out- of-context (OOC) information, where differ- ent pieces of information are falsely associated, e.g., real image combined with false tex- tual caption or a misleading textual description. Although some past studies have attempted defend against OOC mis- and disinforma- tion through external evidence, they tend to disregard the role of different pieces evi- dence with different stances. Motivated by the intuition that the stance of evidence rep- resents bias towards different detection re- sults, propose stance extraction network (SEN) that can extract the stances of differ- ent pieces of multi-modal evidence in a uni- fied framework. Moreover, we introduce support score calculated based on the co-occurrence relations of named entities into the textual SEN. Extensive experiments on a public large-scale dataset demonstrated that proposed method outperformed the state-of- the-art baselines, with the best model achieving a performance gain of 3.2% in accuracy.
A major area of research on OOC mis- /disinformation is about development of automated detection methods and construction of datasets for evaluating and comparing performances of differ- ent methods. Operations such as random match- ing (Jaiswal et al., 2017). manipulating named enti- ties (Sabir et al., 2018) and strategic matching (Luo et al., 2021) have been used to automatically gener- ate OOC mis-/disinformation samples. Rather than relying static datasets and detection models' in- trinsic ability detect OOC mis-/disinformation (often via cross-modal inconsistencies), Müller- Budack et al. (2020) and Abdelnabi al. (2022) also leverage available evidence on the Internet to help detect OOC mis-/disiiformationn. However, their methods do not adequately perform effective modeling of stance relations between evidence and the claim being checked. This can miss impor- tant information useful for detecting OOC mis- /disinformation, since given piece of evidence supporting or refuting the claim can help detect mis- miggalign between texts and images. An ex- ample can found in Figure
1 Introduction
The proliferation of mis- and disinformation' and its erosion on democracy. justice, and public trust have increased the need for detection and interven- tion (Lazer et al., 2018: 8; Islam et al., 2021). While such information consists of primarily text, the in- creasing popularity of nen-tel data on online
"Corresponding co-authors
'In the literature the terms "misinformation" "dis- information" often have inconsistent definitions. In our work. we adopt the more established definitions by the United Nations (https: /ttps:://www. undp. org/ eurasia/dis/ misinforms Lion): misinformation refers to information that is false but not created with the intention of causing harm and disinformation to information that is false and deliber- ately created to cause harm. Our work can be applied to hoth mis- and disinformation, so we will mostly use the term "miss/disiffoomation..
In this paper, to fill the aforementioned research gap on the lack of consideration of stance analysis

---

Visual Stance Extracton Network
Vixual Evidence
Ensoder
8968
OCGB
Nath-dlnnc
Exiianca Cratton
Textual Stance Extraction Network
Captice Represerttioee
Image
Encadeer
OOGO
Classifer
Caption
Matt-Stance
0000
Dridensa Regreeetttoe
ENOInA Datters
Support Refutation Score
Textual
Supaon
Foo
Ouileng
Ri Me-icrten Towe
Figure I: The architecture of our proposed method. Independent stance extraction networks are used for semantic stance comparisons between image and visual evidence, as well as caption and textual evidence, respectively. The textual extraction network incorporates support-refutation score calculated based the co-occurrence relationship of named entities.
for OOC miss/disinforrmatinn detection, we pro- pose unified framework that aims to comprehen- sively incorporate the stances of multiple pieces of evidence towards claims during the detection process. More specifically, for image-based claim and evidence, text-based claim (image captions in our work) and evidence, we utilize different but in- dependent stance extraction networks (SENs) with similar structure, which allow for cluster-specific presentations of evidence semantics and can extract and fuse multiple stances. In the textual SEN, we further emphasize the stance relationship through a support-refutation score calculated based on the co-occurrence relationship of named entities to re- place the binary named entity indicator (NEI) com- monly used in the literature (Tan et al., 2020; Ab- delnabi et al., 2022). Our analyses based on pub- lic large-scale dataset showed that the score follows a significantly different distribution for pristine in- formation and OOC miss/disinforration.. Further experiments demonstrated that our SENs are more efficient at mining useful information in evidence that can be used to verify claims. The major contri- butions of this paper are summarized below.
We conducted extensive experiments prove that our proposed method can significantly outperform statc-of-the-ard (SOTA) methods while being highly explainable.
The rest of the paper is organized as follows. In the next section, we overview some related work. Section explains our methodology, and experi- mental setup and results covered in Section 4. Section concludes the paper, with the sec- tions after discuss limitations and ethical consid- crations.
2 Related Work
Fake News Detection, Most fake news detection algorithms work solcly on plain text. Some studies employ fact-checking based on knowledge bases or knowledge graph to identify fake news (Thorne et al., 2018: Zhou et al., 2019; Zhong et al., 2020). In recent years, more approaches have attempted use multimodal feature to address the challenge of multimodal deepfakes, which requires bridging the semantic gap between multiple modalities (Imran et al., 2020). Xue et al. (2021) mapped text features and visual semantic features to the same semantic space to obtain cross-modal feature representations, and considers the consistency between them. Sun et al. (2023) designed mual-inconsistency network to simultaneously detect cross-modal inconsistency and content-knowlede inconsistency.
We propose unified framework for compre- hensively modeling the stance of external evi- dence detect OOC mis-/disinformation. To the best of our knowledge, we are the first reporting the significance of stance relation for detection of OOC mis-/disinformation.
Image Repurposing and Out-of-Context Mis-

---

information. Early studies on image repurposing check the integrity of package through reference datasets containing closely related metadata, such as MAIM (Jaiswal et al., 2017) and MEIR (Sabir et al., 2018). AIRD (Jaiswal et al., 2019) adopted an adversarial upproach to simultancously train a bad actor who forges metadata for image repurpos- ing and watchdog for consistency verification of images and accompanying metadata. However, it is unrealistic to assume existing an available and reli- able reference datasets. Therefore, Müller- Baddekkkkccckk et al. (2020) turned to the Internet for visual evi- dence to verify the images in news.
other words, the task can be described as function
as follow: (I ,T, Vc) - L
The architecture of our proposed method is shown in Figure 1. Comparisons between the dif- ferent subsets of input, i.e., between the image claim and visual evidence, and between the cap- tion claim and textual evidence, are performed by independent SENs. Our proposed method allows cluster-specifie presentations of the semantics of evidence, preventing evidence with different se- mantic stances from being ignored. According to Tan et al. (2020), contextual information about named entities is crucial in fake news detection and they leveraged this feature using the binary NEI, which is simplistic. So our proposed method calculates support-refutation score based the co-occurrence relationships of named entities emphasize the stance of each piece of evidence in the textual SEN.
To eliminate the linguistic bias caused by ma- nipulating named entities in the text to obtain in- consistencies, given caption, Luo et al. (2021) re- trieved irrelevant but convincing image through CLIP (Radford et al., 2021) and other models. Ab- delnabi al. (2022) proposed the concept of multi- modal sycle-consi check that starting from image/caption, they searched for textual/visual evi- dence on the Internet to verify the caption/image. However, most of them only focus on evidence with similar semantics, ignoring evidence with differ- ent stance that may be essential for out-of-context misinformation detection.
3.2 Support- Support-Refutation Refutation Score
The support-refutation score (SRS) is based on a simple observation: if there exist same named enti- ties between the textual evidence and the caption claim, then they are usually related to the same or similar context. Conversely, if the named enti- ties that occur frequently in the textual evidence do not appear in the caption claim, the two con- texts are likely to be significantly different. That is, the co-occurrence relationship of named entities can reflect the stance of textual evidence towards the caption claim. This is more evident in OOC disinformation scenarios, where the disinformation creators often tamper with named entities therefore leading to such contextual inconsistencies (Sabir et al., 2018; Müller-Budack al., 2020).
Stance Detection. Stance detection is a classi- fication task where the classification results are in Favor, Against, Neither, or other similar forms (AL- Dayel and Magdy, 2021; Hardalov et al., 2021). Zubiaga et al. (2018) demonstrated that stance is helpful for rumor detection. Some works have attempted to introduce stance detection into fake news detection, but mainly focus on the plain text situation (Thorne et al., 2018, 2019). multimodal situations, Yao et al. (2022) generated a stance representation of evidence during the declaration verification phase to predict a truthfulness label. However, there is no work that can comprehen- sively introduce stance relation into the detection of oft-of-context mis-ddisinformation..
To calculate SRSs, We use the NER module the Spacy library (Honnibal and Montani, 2017) to perform named entity recognition on C and To, and obtain the named entity set E. for C and Ee; for the i-th textual evidence TV, respectively. We obtain statistics on all Ec,, and rank the named entity set E, by their frequencies following de- scending order. Then, we calculate the SRS for the i-th evidence according the following equations:
3 Methodology
3.1 Problem Statement
Given an image-caption pair X consists of an image claim _e and caption claim C and two of external evidence, namely tex- tual evidence set T [TY, -.. I and visual evidence set Ve = [V[.... ... vg]. our task to accurately predict a binary OOC label L that indi- cates if X's two claims are falsely associated. In
= X=
which
Eise- Ee; n.:
(1)
E = Eec, -Ee
(2)
EjeR g(o,)
SRS, = Fiico
(3)

---

where oj refers to the order of named entity j in Ee, and 5 is threshold. Only named entities with an occurrence frequency in Ee no less than 7 will be considered, which can reduce the impact of ir- relevant evidence since the quality of evidence re- trieved from the Internet is not always guaranteed. Additionally, we use scaling factor C that scales down as # Eise decreases to emphasize conflict- ing (mis-aligned) relation, otherwise scales up to emphasize an aligned relation. It should be noted that, due the imperfect accuracy of the Spacy NER module, we utilize fuzzy matching based on the edit distance (Levenshtein Distance more pre- cisely) to filter out inconsistencies such as "Barack Obama" and "Obama's" s" when comparing named entities.
put of textual evidence Hī, we concatenate it with the SRS calculated as explained in Section 3.2 obtain its final semantic representation:
1î; = Concat( (1f, SRS)
(4)
For the same of convenience and simplicity, in the following we hide the subscripts i and t.
The semantic comparison of multiple pieces of evidence versus a single claim is a complex issue. In order to avoid the evidence with different seman- tics towards the claim being ignored in the attention mechanism due to its low weight, our method per- forms hierarchical clustering process using the cosine distance the claim and evidence based on the representations extracted above, and obtain the following three clusters:
Meanwhile, conflicting named entity with a higher frequency should express a stronger refuta- tion stance. So we use g(x) = a+e V= a variant of the family of Logistic functions control the impact of conflicting named entities on the SRS. Ç. a and b are all undetermined parameters.
1) Supporting cluster (SuC), consisting of evi- dence in the same cluster as the claim at clustering threshold Tg. typically expressing support stance;
2) Representative cluster (RcC), the maximum evidence cluster at a clustering threshold Tys which best represents the semantics of all evi- dence;
Although the SRS is concise and intuitive ap- proach suitable for many scenarios, it may not be sufficient for addressing a more complex OOC sce- nario where the same named entities exist but with inconsistent contexts. To address this limitation, we propose an SEN which requires full considera- tion of semantics.
3) Complementary cluster (CoC), not actually a cluster, consisting of the residual evidence, often containing noise, which we retain to pre- vent the loss of potentially useful information.
3.3 Stance Extraction Network
It should be noted that all the above three clusters may be empty, and SuC and ReC may also overlap. When there are two clusters containing the max- imum amount of evidence, we regard the cluster whose semantic closer to the claim as ReC.
A stance extraction network (SEN) aims to com- prehensively consider the semantic stance of all evidence while preventing the neglect of important evidence by enabling the cluster-specific presen- tations of different semantics. Our method uses independent SENs with a similar structure for vi- sual evidence and textual evidence.
Referring to the way how the memory net- work (Sukhbaatar et al., 2015; Abdelnabi et al.. 2022) processes multiple sentences, our method uses a fully connected layer to process the claim representation H and obtain He as query in the attention mechanism, and the other two indepen- dent fully connected layers to process the evidence representation 1î and obtain Hद and Hद as key and value, as described in the equations below:
We map claims and evidence into semantic rep- resentations in visual and textual semantic spaces. We use the same encoders the consistency- checking network (CCN) (Abdelnabi al., 2022). Specifically, for image claim Ie and visual evi- dence Ve, we use ResNet152 (He et al., 2016) pretrained on ImageNet and ResNet50 pretrained on Places365 (Zhou et al., 2017) as two indepen- dent encoders. For caption claim C and textual evidence T We use Sentence-Ber (Reimers and Gurevych, 2019). We denote the representation of / as Hi and the representation of V Ve/Te as He It should be noted that for the encoder out-
Hc = oW HC +
(5)
Hi H6/0 o (WE, - + 12/c))
(6)
where represents ReLU.
As shown in Figure 2, we complete the atten- tion mechanism shown in Eqs. (7)-(8) for cach cluster. We first compute an attention distribution

---

between the claim and evidence using softmax, and then treat the weighted sum as a cluster-specific se- mantic presentations the evidence in this cluster. Then, we added H to the -luster-specific semantic presentations to obtain the semantic stance repre- sentation of cach cluster as follows:
them into classifier consisting of two fully- connected layers, following the approach of Ab- delnabi et al. (2022). We use the cross entropy loss to train the model.
Experiments
0x2 Sootmax H Hg
(7)
In this section, we explain the datasets we used, the experimental setup, and the experimental results regarding the performance of our proposed method. We also conduct qualitative analysis of the results to provide an intuitive illustration of our method.
H. BN Hi + 1
(8)
where E {SuC, ReC, CoC) represents different clusters, and BN represents batch normalization. We fuse the semantic stance representations of the above three clusters to obtain a stance fusion representation of the overall evidence. We tried several commonly used fusion strategies includ- ing element-wist multiplication, max-pooling avg- pooling, but the simple concatenate showed the best performance, which is described follows:
4.1 Datasets
NewsCLIPpings (Luo et al., 2021) is a large-scale and currently the most challenging OOC mis- /disinformation dataset, which contains both pris- tine and falsified (i.e., O0C) image-caption pairs. According to certain strategies, matches real im- ages and real captions to simulate four OOC sce- narios: (1) Both Scenarios a and b, i.e., Seman- tics/CLIP Text-Image and Semanties/CLIP Text- Text, are about attempts of portraying the subject of images as other named entities and use CLIP em- beddings for mismatching. The main difference is that Scenario tries to form OOC pairs by finding non-matching image-caption pairs with the highest similarity directly, while Scenario by first find- ing non-matching caption-captin pairs with the highest similarity and then using the corresponding image of the second caption. (2) Scenario c, i.c.. Person/SBERT-WY Text-Text, portrays the same person in false context, such the wrong events or places. It uses the lowest SBERT-WK text-text similarity ensure the person appearing in differ- ent contexts. (3) Scenario d, i.e., Scene/ResNet Place, describes the event in the image as another event of the same event type.
H = CConcat(Hsace Hsuc, HREC; Hco.),
(9)
fusion ((WII + I19).
(10)
We added He in Equation (10) again to emphasize the stance relation. The dimension of W and that of b depend on the dimension of H. Generally speaking, if H Idd, then W E Rdx3d and E Rdd. We set d to 1024 and 768 for the visual and textual SENs, respectively.
SuC
ReC
CoC
Evidence Representation
OOO
OOO
OOO
Atcention
Hlue
Flse
Rico
Coo
We evaluated our proposed method on its Merged/Balanced subset, which mixes equal sam- ples from the above four scenarios to achieve a more realistic scenario and consists of 71,072 sam- ples for training. 7,024 for validation and 7,264 for testing.
Concatt Houty Haues Hose)
Claim
FC ReLU
Representation
O
For visual and textual evidence used to validate image-caption pairs, we used the results retrieved by Abdelnabi al. (2022). Given textual cap- tion, retrieves up to 10 pieces of visual evidence. Given an image, retrieves textual evidence both entity and sentence types.
Stance Fusion Representation Heato
Figure Stance extraction fusion in the SEN.
Finally, we concatenate all obtained stance fu- sion representations from different SENs and feed

---

4.2 Experimental Setup
encoded images and captions. We choose the model using ViT-B/32 like CLIP and CCN.
We computed SRSs for both entities and sen- tence evidence, and set the threshold min ; # E.). We used binarized Ç and set g(x) e V=z as explained in Appendix A. We used two SENs for the two different representations of im- age claim and visual evidence, and one SEN for sentence evidence. Due to the fact that almost all entities are general concepts or named entities com- posed of one or two words, they lack the complete context express stance through semantics, re- sulting the small difference (about 0.1%) between using an SEN and a memory nerwork. So we de- cided to use the memory network, since has fewer parameters. Since the retrieved evidence is usually in the same field as the claim, for example, both singers' performances or politicians' speeches, we set the threshold of SuC and ReC to be the same. Specifically, we set the text clustering thresholds r and Tf 0.500 and the image clustering thresholds T.. and r. 0.166.
MNSL (Zhang et al., 2023) parses text into reaprictt graph and process it together with visual input.
CCN (Abdelnabi et al., 2022) uses Internet evidence retrieved across modalities to fact- check image-caption pairs. For comparison, choose the model using Sentence-BERT for text encoding like ours.
Evidence
All
Pristine
Falsified
CLIP
X
60.2%
70.1%
50.4%
SSDL
X
65.5%
68. 68.7%
62.2%
DT
X
65.7%
73.7%
57.6%
MNSL
X
68.2%
65.4%
70.5%
CCN
83.9%
82.8%
84.9%
Ours
87.1 %
85.5%
88.6%
Table I: Performance on Merged/Balanced subset of our method in comparison with baselines using Accuracy metric.
To measure the performance gain from introduc- ing stance analysis, we used all the constraints and gains in the baseline CCN (Abdelnabi et al., 2022), such as evidence filtering constraint that will re- duce the accuracy rate but are closer the real situation,, and the fine-tuning CLIP model, image label and domain gain.
Table 1 shows the performance of all compared models on all pairs, pristine pairs and falsified pairs. Compared with all baselines, our method achieves the best performance, improving the accuracy by 3,2 3.2% compared to the second best method. The pre- diction of both pristine pairs and falsified pairs has been significantly improved, with accuracy gain of 3.7% on falsified pairs, slightly higher than the 2.7% on pristine pairs, which confirms that consid- ering the stance of evidence will benefit detection of OOC miss/diisiffrrmtiinn.
We conducted experiments on a computer equipped with one NVIDIA Geforce RTX 3090 GPU. The model was trained with batch size of 64 for 60 epochs. The optimizer used is Adam (Kingma and Ba, 2014). The learning rate is a cyclical learning rate with the maximum value of 6c-5 and minimum value 9c-6.
Table also demonstrates the importance of external evidence for detection of OOC image- caption pairs. Models using external evidence achieve significantly higher accuracy than those not. Furthermore, due to the highly realistic and plausible nature of falsified samples, models not re- lying on evidence have over- pre-ppredeted the Pristine or Falsified label, which means it is difficult for them to mine more useful features relying solely on pairs.
4.3 Results
We compared our method against the following SOTA baseline methods. In Table 1 we indicate whether these SOTA methods use external evidence or not.
CLIP (Radford et al., 2021) generates seman- tically similar representations for images and text describing the same concept or event;
We further compared the accuracy of our method with CCN on the data from the four scenarios in the Merged/Balanced subset to analyze their abil- ity deal with various types of OOC. As shown in Figure 3, compared with CCN, our method ex- hibits significant improvements for Scenarios a and by 3.8% and 4. 4.7%, respectively. For Scenario b, due to the use of indirect mismatching method
SSDL (Mu et al., 2023) assesses intra- modal and inter-modal self-consistencies by wwo-phased learning strategies using self- supervised supi-super method;
DT (Papadopoulos et al., 2023) uses transformer- based detector to process CLIP

---

which is easier to detect, both CCN and our method have achieved an accuracy rale of more than 90% and the performance improvement at this time be- gins to be affected by the quality of the both pairs and evidence, but our model still achieved a 1.3% improvement. As discussed in Luo et al. (2021), Scenario is the most challenging one, and our method achieves 3.0% performance improvement, indicating that our method has better understand- ing ability and can comprehensively mine informa- tion beyond named entities.
$0,005
$2,00%
78,0096
74,00%
70,009
25%
50%
75%
100%
Triining Set Proportion OCN Ours
Figure 4: Performance analysis in limited data environ- ment.
95.009%
w/o SENs: Use Memory Network to replace all Stance Extraction Networks.
90.00%
w/o Cluster: Remove evidence belonging to the specific cluster.
80.00%
The ablation results in Table 2 confirm that both SRS and SEN are indispensable for the best per- formance. Compared with w/o SRS, the widely used binary NEI (Tan et al., 2020; Abdelnabi al., 2022), although improving the detection accuracy of pristine pairs, weakens the ability to detect falsi- fied pairs, while SRS is able to improve both. Al- though the SEN with both modalities may slightly impair the detection of pristine pairs (up to 1.4%), substantially improves the detection of falsified pairs (up 5.7%). This consistent with our hy- pothesis that both SRS and SEN are intended highlight the refuting relation while preserving the supporting relation implied in the baseline.
75.00%
70.00%
Soctario
Ouns
Figure 3: Performance comparison between CCN and our method in different out-of-context scenarios.
4.4 Performance Analysis in Limited Data Environment
To evaluate the performance stability of our method in a limited data environment, we randomly sam- pled the training set at varying proportions. The results, depicted in Figure 4, indicate that with only 25% of the entire training set required, our method has approached the accuracy performance of the baseline CCN the entire training set, reach- ing 83.6%. Furthermore, our proposed method can have an even higher performance gain with a smaller number of training samples. Detailed data can be found in Appendix B. This finding high- lights our method's superior capacity to effectively explore and utilize useful features for the OOC mis-/disinformationn detection task.
In order to verify the effectiveness of semantic clustering, we removed different clusters for impor- tance analysis. As shown in Table 2, when differ- ent clusters are removed, performance decreases varying degrees, with CoC having the least impact, as there is more noise rather than useful informa- tion. Notably, given the possibility evidence intersection between SuC and ReC, we removed both of them at the same time, resulting in signif- icant performance drop.
4.6 Explainability Analysis
Figure shows the distribution of the mean SRS for both pristine and falsified pairs, showing significant distribution differences. Our model has learned the distribution differences to better predict labels, im- proving the accuracy by 2.5% and 1.4% compared to w/o SRS and binary NEI respectively, as shown in Table 2. It should be noted that Scenario c, i.e., the Person/SBERT- WK Text-Text scenario, some distributions appear bigger than 0 due to the high
4.5 Ablation Analysis
We investigated the impact of different components of our proposed method on the performance by defining the following variants:
w/o SRS: Remove the SRS for textual evidence. binary NEI: Use binary NEI to replace SRS.
w/o Vi/Te-SE: Use Memory Network to replace the Visual/Textual Stance Extraction Networks.

---

Image-Caption Pair
Textual Evidence
Visual Evidence
Prediction
Unitd Sake Cour Appiaa for the Clenic Colnmila Zirual (UA)
Prwniber saman laprone Coun modked
vee reminrion arFancie allier
Pristine /
Paantt Rosentik
2011
Patncia let was nominated to the powerful Appaeda
Pnoers mamoss vake
Powerts Catull Dhemormt
Pralins
y Hour Prcee
Feodi Natar Moaar (ol), Va
AAdenciun The Cuardion 14)
atar more henyy riln (0.0)
Out-of-Context V
InAoerderr alire (8.20)
Gemmelll Nounebber 2010
Climate change incressing the risk of flodiigg g UK Outsl-Conteel
GaT siae Scrtbard The Guardbie (1.7))
Michaal Gooe reeipspll in in Dused boadang primery achioll
The hoo peaadr poire restte Neck Origo and seersaay 2013 3.30
Out-of-Context
Michuel Cove al achool Edmonion northeast London
Mctie Ryean on acadiing echoae fooce senonney coor
Contest
Ulerpe Seeerrt Yade
cecion daymp 2010 pluaas, Summer cymporunty Maiiyaaeyy
Pristine
X
Lendon Ol 2012 Pheie, Summer Dympie 2012 Phoisgaeey
Aaron Reynolds
Thares File Brigade boat Dud-el-Conneel
Figure 5: Some prediction examples. Yellow box represents SuC, blue box represents ReC, purple box represents CoC, and (orange) represents SRS for textual evidence. The ground truth labels are listed Caage-- Pair We report the prediction of our method in Prediction and use //X to indicate whether the prediction is correct. Only part of the evidence is shown to highlight key points.
Pristine Pairs
Falsified Pairs
Sumorio
Scemario
Sectario
0.2
Scmurio
220
-1.5
-1.0
.0.
0.0
0.5
1.0
1.5
2.0
-20
-1.5
-1.0
.0.5
0.0
0.5
1.0
1.5
20
Figure SRS distribution heatmap on pristine data and falsified data.
probability containing mentions the same person. To statistically verify the observation results, two equal-sized collections of both pristine and falsified image-caption pairs were sampled, with 500 sam- ples per type. Then conducted two-sample one-tail z-test the 1,000 samples. Let /4p be the mean SRS of pristine samples and μ be that of fal- sified samples. Ho represents the null hypothesis and the alternative hypothesis Hy:
< 0.001, the null hypothesis Hg can be rejected even under significance level (ar) as low as 0.001, indicating the observation results are statistically supported.
Figure 5 shows some prediction examples of our model. For pristine examples, such as the first one, in addition to having typically positive SRS due to the same named entities, SuC and ReC simultane- ously express support semantics and often overlap. For falsified examples such the second one, there usually no evidence in SuC. In addition, although (12) the textual evidence and the visual evidence have
Ho Mp M1 57
(11)
H Up 1 >
When y gradually increases to 0.7, 2 3.627, an event type similar to the image-caption pair, it

---

All
Pristine
Falsified
concise equation model this based the practi- cal background. However, unified standard or a more accurate learning model that considers seman- tics would be ideal for this task. Additionally, we measure named entities based on fuzzy matching, which may not account for more complex situations such name abbreviations or relational references, etc. Although these situations are uncommon in our task scenarios, that is, short captions, it would be beneficial to consider them for more comprehen- sive and accurate analysis.
Ours
87.1% %
85.5%
88.6 %
w/o SRS
84.6%
82.1%
87.1%
binary NEI 85.7%
86.9%
84.4%
w/o Vi-SEN 85.5%
85.9%
85.1% %
w/o Te-SEN 86.1%
85.0%
87.3%
w/o SENs
84.9%
86.9%
82.9%
w/o SuC 83.4%
83.5%
83.3%
w/o ReC
85.1%
83.9%
86.3%
w/o CoC
86.5%
85.6%
87.4%
Multi-modal fact-checking of OOC mis- /disinformation, especially news, is a complex task due to: (I) the naturally loose and abstract correspondence between the image and the caption: (2) the multi-faceted nature of verification, including verification within modalities and between modalities; and (3) the quality of evidence is not always guaranteed. There is no single pipeline that can handle all these challenges in unified manner so far, especially the first one, which remains rescarch challenge. In our future work, plan to devise more comprehensive model to address some of these challenges and generalize our method to other OOC tasks, such the more complicated one with one image and two captions addressed in Aneja et al. (2023).
w/o SuC+ReC 74.0%
75.4%
72.6%
Table 2: Performance comparison results of different variants.
is predicted correctly. The third one shows a sit- uation that is difficult to judge, due to the same event context, i.e., the same person in same event type, partial textual evidence presents supporting semantics, but low SRS and a lack of supporting visual evidence prevent the model from making wrong judgments. The last one shows wrong prediction example, the textual evidence is "generi- cally matched" and semantic conflicts are difficult to appear. In the visual evidence, the model may pay more attention to the characteristics of people holding torch, boats, seas, etc., but does not note the difference in the person due to the size.
7 Ethics Statements
In terms of ethical considerations, our method aims to address the prominent social problem of OOC miis/diiinforrmtionn by identifying characteristics that distinguish from truthful information. We acknowledge that current detection algorithms, in- cluding our method, have failed to achieve a detec- tion accuracy that can be completely independent of human intervention, not to mention that there are far more OOC scenarios than the four categories described in (Luo et al.. 2021) and considered in our work. However, manual reviews are not fea- sible for the vast amount of misnfdiiinnoooaaiinn online, and our method can serve as a fast and reasonably accurate first line of defense. For in- formation with significant influence, such as that released by important institutions, celebrities, and politicians, method is suitable auxiliary tool to aid human judgment. However, also rec- ognize that our method not perfect and may have limitations such as potential biases, which should be taken into account when interpreting its results. We strive to transparent about our methodology
5 Conclusion
We propose unified framework capable of incor- porating stance relation of evidence for automatic out-of-context miis//isinforrmation detection. The support-refutation score calculates a stance related score for each textual evidence based on the co- occurrence relationship of named entities, while stance extraction networks able to uniformly consider the stance of all evidences. Our proposed method outperforms the existing basclines and pro- vides a new benchmark for evidence-assiste ve- racity assessment of image-caption pair task. Fur- thermore, our method offers good interpretability, allowing for clear understanding of the role of each evidence and how it contributes to the overall decision-making making process.
6 Limitations
We are not aware of a model for stance relation analysis based on named entities, so we propose a

---

and encourage ongoing ethical discussions in the field of miss-disinformationn detection.
Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmmeeed. and Premkumar Natarajan. 2017. Multimedia semantic integrity assessment using joint embedding of images and text. In Proceedings of the 25th ACM Interna- tional Conference on Multimedia, pages 1465-1471. ACM.
Acknowledgments
This work was partly supported by the National Natural Science Foundation of China under the Reference Number 61972249.
Ayush Jaiswal, Yue Wu, Wael AbdAlmaeeed, lacopo Masi, and Premkumar Natarajan. 2019. Aird: Ad- versarial learning framework for image repurposing detection. In Proceedings of the 2019 IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, pages 1130-113333 IEEE.
References
Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. 2022. Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 14940-14949. IEEE
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. [cs.LG].
David M. J. Lazer. Matthew A. Baum, Yochai Ben- kler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menezer, Miriam J. Metzger, Brendan Nyhan, Gor- don Pennycook, David Rothschild, Michael Schud- son, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zit- train. 2018. The science of fake news. Science, (19963300:1044100999
Abeer AaDayel and Walid Magdy, 2021. Stance detec- tion on social media: State of the art and trends. Infor- mation Processing & Management, 58(4): 10259711 102597:22.
Shivangi Aneja, Chris Bregler, and Matthias Nießner. 2023. COSMOS: Catching out-of misinfor- mation with sulf-supe learning. Proceedings of the AAAI Conference on Artificial Intelligence. 37(12); 10044-149.2.
Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021. NewsCI.Ppings: Automatic generation out-of- context multimodal media. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6801-6817. ACL.
Momehil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2021. A survey on stance detection for and disinformation identification. arXiv:2103.0022 [cs.CL].
Michael Mu, Sreyasee Das Bhattacharjee, and Junsong Yuan. 2023. Self-super distilled learning for multi-modal misinformation identification. In Pro- ceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, pages 28 2819- 2828. IEEE.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778. IEEE.
Eric BBulck,, Jonas Theiner, Sebastian Diering. Maximilian Idahl, and Ralph Ewerth. 2020. Multi- modal analytics for real-world news using measures of cross-modal entity consistency. In Proceedings of the 2020 International Conference Multimedia Retrieval, pages 16-25. ACM.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embed- dings, convolutional neural networks and incremental parsing.
Mordansss Papadopoulos, Christos Koutlis, Symeon Papadopoulos, and Panagiotis Petrantonakis. 2023. Synthetic misinformers: Generating and com- bating multimodal misinformation. Proceedings of the 2nd ACM International Workshop on Multime- dia Al against Disinformation, pages 36-44, ACM.
Muhammad Imran, Ferda Ofli, Doina Caragea, and An- tonio Torralba. 2020. Using AI and social media multimodal content for disaster response and man- agement: Opportunitics, challenges, and future di- rections. Information Processing & Management, 57(5): 102261:1-1022611:
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 9748-8763 PMIRR
Md Saiful Islam, Abu-Hena Mostofa Kamal, Alam- gir Kabir, Dorothy L. Southern, Sazzad Hossain Khan, S. M. Murshid Hasan, Tonmoy Sarkar, Shayla Sharmin, Shiuli Das, Tuhin Roy. Md Golam Dostogir Harun, Abrar Ahmad Chughtai, Nusrat Homaira, and Holly Seale. 2021. COVID-19 vaccine rumors and conspiracy theories: The need for cognitive inocula- tion against misinformation to improve vaceine adher- ence. PLoS ONE, 16(5):0025 1605: e-25 1605:17. 7..
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on

---

Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing, pages 3982-3992. ACI.,
social media. Proceedings of the International AAAI Conference on Web and Social Media, 15:776-786.
Junxiao Xue, Yabo Wang, Yichen Tian. Yafei Li, Lei Shi, and Lin Wei. 2021. Detecting fake news by exploring the consistency of multimodal data. Infor- mation Processing & Management, 58(5):10261011 102610:13.
Ekraam Sabir, Wael AbdAlmageed, Yue Wu, and Prem Natarajan. 2018. Deep multimodal image- repurposing detection. In Proceedings of the 26th ACM International Conference on Multimedia, pages 1337-1345. ACM.
Barry Menglong Yao, Aditya Shab, Lichao Sun, Jin- Hee Cho, and Lifu Huang. 2022. End-to-end multi- modal fact-checking and explanation generation: A challenging dataset and models. arXiv:2205. 12487 [cs.CL]]
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. Advances in Neural Information Processing Systems, 28:2440-2448.
Mengzhu Sun, Xi Zhang. Jianqiang Ma, Sihong Xie, Yazheng Liu, and S. Yu Philip. 2023. Inconsistent matters: A knowledge-guide
Yizhou Zhang, Loc Trinh, Defu Cao, Zijun Cui, and Yan Liu. 2023. Detecting out-of-context multimodal misinformation with interpretable neural-syman -sybolc model. arXiv:2304.07633 [cs.CL].
dual-consisteney net- work for multi modal rumor detection. IEEE Trans- actions on Knowledge and Data Engineering.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 6170-6 6180. ACL
Reuben Tan, Bryan A. Plummer, and Kate Saenko. 2020. Detecting cross-modal inconsistency to defend aainst neural fake news. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing, pages 208 1-2106. ACI..
Christopher Thomas and Adriana Kovashka. 2020. Pre- serving semantic neighborhoods for robust cross- modal retrieval. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, volume 12363 of Lecture Notes in Computer Science, pages 317-335. Springer.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 40(6):1452-144 14444
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and rea- soning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 892-901. ACI.
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. 2018. FEVER: a lcage-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pages 809-819. ACI..
Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018. Detection and reso- lution of rumours social media: A survey. ACM Computing Surveys, 51(2):32:1-322
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopouuls,, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the 2nd Workshop on Fact Extraction and VERificution (FEVER), pages 1-6. ACI..
A SRS Parameter Optimization
Since we do not have any gold standards ground truth available, conducted small-scale exper- iment to select an excellent curve for modeling support-reftttatoon relations in SRS and illustrate the effectiveness of each item.
Tian Tian, Yudong Liu, Xiaoyu Yang. Yuefei Lyu, Xi Zhang, and Binxing Fang. 2020. QSAN: quantum- probuubiiybb based signed attention network for explainable false information detection. In Pro- ceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1445-1454, ACM.
It should be noted that for each hyperparame- ter, we mainly conducted experiments based on grid search within small and reasonable range. Considering that each parameter can theoretically take many values in large range, this experiment is obviously not exhaustive. But our aim is make SRS achieve representative performance and demonstrate the effectiveness of each item in SRS, rather than the optimal performance with minimal differences. Larger ranges and more detailed values
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread true and false news online. Science, 359(6380); 14--1101 146-1151.
Yuping Wang, Fatemeh Tahmasbi, Jeremy Black- burn, Barry Bradlyn, Emiliano De Cristofaro, David Magerman, Savvas Zannettou, and Gianluca Stringh- ini. 2021. Understanding the use of fauxtography on

---

will exponentially increase time and computational resources, which not in line with our original intention. To this end, we report the validation accuracy when selecting Ç and g(r).
A.2 Selection of g(x)
In our final model, we set g(x) - ev since it has the simplest form and relatively smooth descent. However, we later obtained the performance of dif- ferent g(r) curves by tuning a E {0, 2,2,4, 8) and E (1, as reported in Figure 8. Interestingly, although not optimal, the g(x) we adopted also exhibits competitive performance. Meanwhile, all g(x) curves outperform the results obtained using
A.1 Selection of Scale Factor
C controls the emphasis on corresponding or con- flicting relationships in the Support-Refutation Score (SRS) calculation. We consider two ap- proaches setting the value of Ç. The first ap- proach, shown in Equation (13), which we call the Proportion Setting, intuitive and commonly used. However, when the number named entities in the evidence and caption overlap, i.e., # E4sce is too large, the conflicting relationship may be weakened. Therefore, we consider a second approach, Bina- rization Setting, where Ç is binarized, as shown in Equation (14).
binary NEI.
87.00%
86.50%
86.00%
85.50%
85.00%
- a(# Eiteo + 1)
(13)
0.25
0.5
{
23
# Exec
b=1
b=2
ç s=
(14)
3 # Eixco
Figure 8: Selection of g()).
The specific expression of the function g(x) should not significantly affect the comparison re- sults between the two settings Ç. For our experi- ments, we set g(x)= evaa.
A.3 Validity Verification
We tune both a and / within (0.25, 0.5, 1, 2, 4J. The result is shown in Figure (7).
We can discuss the validity of the terms in the SRS by analyzing the impact of removing or modifying them on the performance. When we removed the negative term and only keep the positive term SRS, the performance drops by 0.22%. Similarly, when we fixed the negative term to 1, the perfor- mance drops by 0.24%. These results indicate that the negative term representing refutation relation is important and contribute the overall performance of our method.
87.00%
86.50%
86.00%
85,50%
Furthermore, we found that the performance dropped by 0.42% when g(x) was fixed so that it was independent of the input (we set g(x) 0.5). This confirms observation that named entities with higher frequency of conflicts should express stronger conflicting stance. When the scaling fac- tor ( does not scale as # Eigcc gets smaller (we set - 2), the performance dropped by 0.26%. This indicates that dynamically emphasizing support or refutation relations based on # E is effective.
85.00%
0.25
0.5
2
Proportion
Binarization
Figure Selection of Ç.
It can be observed that on average, the Binariza- tion Setting of Ç performs slightly better than the Proportion Setting, although the difference is not significant. This may be due to the short text in task, resulting a small number of named entities. Based on these results, we choose the Binarization Setting and set 3 since the accuracy almost the highest while having smaller loss in this case.
B Details about Figures and 4
Details about Figures 3 and 4 are presented in Ta- bles and 4, respectively.

---

Scenario
same features as CCN and our method. Specif- ically, Equations (15)-(17) demonstrate how we combined approach (a), while Equations (18)-(19)
a
d
CCN
81.5%
90.9%
78.5%
84.6%
depict how we integrated approach (b).
Ours
85.3%
92.2%
81.5%
89.3%
H. (oftmax(H"HH..))) H
(15)
Table Performance comparison results between CCN and our method in different OOC scenarios.
H₁ -ofttmax- H + H
(16)
H W. "CCnncat HI .H, )
(17)
Training Set Proportion
H. = Concat W. W.H.. H W. H. H³)
(18)
25%
50%
75%
100% %
= H. +
(19)
CCN
77.9%
80.8%
82.8%
83.9%
Ours
83.6%
85.3%
86.0%
87.1%
Method
CCN
(a)
(b)
Ours
Table 4: Performance analysis results in the limited data environment.
Accuracy 83.9%
83.9%
83.8%
87.1%
Table 6: Performance comparison results of different ways introducing stance relation.
Multi-Stance Fusion
In addition to the concatenation method, we also tested other methods, including Max-Pooling, Avg- Pooling, Element-wise multiplication, and com- bination of the three. followed by a fully connected layer for dimensionality reduction. However, as shown in Table 5, none of these methods were able achieve the performance the concatena- tion method for stance fusion.
The comparison with baseline CCN and our method is shown in Table can be seen that the accuracy has not significantly improved with the assistance of the two methods, which shows that the introduction of stance in the out-of-context scenario is different from the general fake news scenario.
All
Pristine
Falsified
Ours
87.1 %
85.5%
88.6 %
Max-Pooling
86.2%
86.3%
86.0%
Avg-Pooling 85.6%
84.5%
86.6%
Multiplication 83.8%
83,4%
84.1 %
All with fc
86.0%
84.5%
87.4%
Table 5: Performance comparison results of different variants.
D Ways to Introducing Stance Relation
We notice that some studies try to introduce stance relation into fake news detection. For instance, (a) QSAN (Tian et al., 2020) emphasizes the "opposite" relationship by defining -Softmxx, and (b) Yao et al. (2022) propsoed to extract stance representa- tions of evidence towards claim in end-to-end multimodal fact-checking task by two arithmetic operations. For comparison with the way we in- troduce the stance relation, we integrated the two approaches into the baseline CCN while using the


Reviews:
[{"review_id": "VjFWJZZI6F", "reviewer": "Reviewer_2aq7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. Below please find our responses to issues and concerns raised by the reviewer.\n\n**Reason To Reject 1: The proposed method is fairly complex and depends on multiple networks and an external NER component. However, the authors show that all of the components are important for the results.**\n\n**Our response:** Although our proposed method is indeed quite complex, we would like to highlight that it uses the same number of networks as some of the SOTA baselines such as the CCN method. As the reviewer pointed out, we include all the components to help improve the performance of the designed classifier beyond other SOTA methods, therefore the high complexity is just a consequence of including such components.\n\n**Reason To Reject 2: I'm not fully convinced by the explainablity analysis presented by the authors (also the explainbility of the model itself). More precisely, even though the two distributions differ in terms of support-refute score this is a feature in the model and it unclear how it correlates with the predicted label (pristine/out-of-context).**\n\n**Our response:** Thanks for pointing out the lack of clarity of the explainability subsection. The effect of the SRS feature can be seen from the ablation analysis shown in Table 2: when we remove the SRS feature (w/o SRS) or further simplify it into a binary feature (binary NEI), the accuracy will drop by 2.5% and 1.4%, respectively. We will refine the subsection by adding this analysis in.\n\n**Typos Grammar Style And Presentation Improvements**\n\n**Our response:** We thank the reviewer for the two issues and will fix them and do further proofreading to correct any other typos, grammatical errors and presentation issues.\n\n**Reproducibility: 3: Could reproduce the results with some difficulty. The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available.**\n\n**Our response:** We appreciate the reviewer's concerns on the difficulty about reproducibility. We have released all the code in the supplementary materials, and README.md contains the steps and commands required to replicate our experimental results. The datasets we used are all public ones so readers can easily access them and run our code against the datasets. If the paper is accepted, we will release the code and relevant documents on GitHub as a public resource to facilitate reproduction of our experimental results. We will also release the random samples used in our experiments to allow reproducing the results reported in this paper. We also welcome any further advice from the reviewer on what we can do to further improve reproducbility of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Multimodal mis-/disinformation detection is an important and timely topic.\n2. The proposed approach models simultaneously the textual and visual evidences using multi-stance fusion.\n3. The proposed framework outperforms significantly state-of-the-art approaches both in low-resource and full-resource setting.", "reasons_to_reject": "1. The proposed method is fairly complex and depends on multiple networks and an external NER component. However, the authors show that all of the components are important for the results\n2. I'm not fully convinced by the explainablity analysis presented by the authors (also the explanbility of the model itself). More precisely, even though the two distributions differ in terms of support-refute score this is a feature in the model and it unclear how it correlates with the predicted label (pristine/out-of-context). ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a framework for automatic out-of-context mis-/disinformation detection using the stance (support/refutes) of multi-modal evidences (texts and images). The authors' proposed architecture leverages stance extraction networks for semantic stance comparisons between image and visual evidence. Finally, they incorporate support-refutation score calculated based on the co-occurrence relationship of named entities. The proposed approach outperforms strong state-of-the-art baselines on the NewsCLIPpings datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* L428 STOA -> SOTA\n* The metric is unclear in: L026 \"a performance gain of 3.2%.\" and in the caption of Table 1."}, "extra_text": {}}, {"review_id": "X4lO8oFp3v", "reviewer": "Reviewer_V426", "summary": "", "questions": "A. Why did your data sample your training data? Did you re-train and re-evaluate other models in your sample environment? Would this shift in the dataset change the model's performance, either in data efficacy or overall performance?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. Below please find our responses to issues and concerns raised by the reviewer.\n\n**Reasons To Reject 1: The paper writing is too dense and somewhat hard to follow.**\n\n**Our response:** We thank the reviewer's comment, and we will conduct a more careful proofreading to see if the writing can be further improved.\n\n**Reasons To Reject 2: In the paragraph starting at line 470, the paper has compared their method to the CLIP approach, which performs considerably lower than theirs. (In Figures 3&4, the model's name shows CCN, but in the text, it refers to the CLIP model). Comparing must be done your model with the existing best model on OOC scenarios.**\n\n**Our response:** Thanks for spotting this inconsistency. We made a mistake in the main text. The model we compared our method with is indeed the CCN approach, the best existing model. We will fix this error in the final paper.\n\n**Reasons To Reject 3: The scenarios (a, b, c, d) are unclear. It makes the analysis lines 476 and 482 hard to follow.**\n\n**Our response:** Thanks for pointing out that the descriptions about the four scenarios are not sufficiently clear. Note that our proposed method outperforms the best baseline CCN in all the four scenarios, although we decided to focuses more on the most challenging Scenario c. In the following, we clarify more about the four scenarios and why we focused on Scenario c.\n\nThe four different OOC scenarios were originally defined by authors of the NewsCLIPpings paper. They defined the scenarios in a relatively complicated way, so that they had to use more than two pages for the descriptions. For our paper, due to the limited space, we decided to focus on the most challenging Scenario c in lines 481-484 and explain other three scenarios only briefly. If the paper is accepted, we will add the following more detailed descriptions of all the four scenarios to the final paper (which we understand is possible because accepted papers will be allowed to have one more page for the main body).\n\n* Both Scenarios a and b, i.e., Semantics/CLIP Text-Image and Semantics/CLIP Text-Text, are about attempts of portraying the subject of images as other named entities and use CLIP embeddings for mismatching. The main difference is that Scenario a tries to form OOC pairs by finding non-matching image-caption pairs with the highest similarity directly, while Scenario b by first finding non-matching caption-caption pairs with the highest similarity and then using the corresponding image of the second caption.\n\n* Scenario c, i.e., Person/SBERT-WK Text-Text, portrays the same person in a false context, such as the wrong events or places. It uses the lowest SBERT-WK text-text similarity to ensure the person appearing in different contexts.\n\n* Scenario d, i.e., Scene/ResNet Place, describes the event in the image as another event of the same event type.\n\nWith these explanations, we believe that the analysis between lines 476 and 482 will be much easier to understand. Since Scenario b uses the similarity of caption-caption instead of the similarity of image-caption for mismatching, it is relatively easy to detect in a multi-modal framework. Due to the special mismatching method of Scenario c, only models with a strong semantic understanding ability can discover the subtle differences in the context, so we consider it more difficult to detect.\n\n**Reasons To Reject 4: The conclusion of the two-sample one-tail z-test is not clearly explained.**\n\n**Our response:** We have shown in lines 538-540 that there are significant differences in the distribution of pristine and OOC data on SRS, and explained in lines 544-546 that the purpose of conducting the two-sample one-tail z-test is to statistically verify this observable distribution difference. After reading the description again, we now feel some parts could benefit from more explanation, e.g., how the value of gamma was handled and how the statistical significance changes w.r.t. the value of gamma. We will add such additional discussions to the final paper.\n\n**Reasons To Reject 5: The data sampling is indefinite. The chosen samples can affect the evaluation outcome.**\n\n**Our response:** We understand that the reviewer refers to the results based on a sampled subset of the whole training dataset, reported in Section 4.4. We used three sampling rates (25%, 50% and 75%) and our purpose is to study how well our method works when the training set available is smaller. This is useful because if a model can work with a smaller amount of training data, then the efforts needed to gather and label training data will be lower. As shown in Figure 4, compared with the CCN method, our proposed method can have an even higher performance gain with a smaller number of training samples, indicating that our model can learn more effectively and more efficiently (i.e., faster) than the CCN model. In the final paper, we can include a new table to show the precise performance gains under the four different sampling rates (25%, 50%, 75% and 100%). For the reproducibility purposes, we saved the random sampling data we used, although we did not include the data in the supplementary materials. In order to facilitate reprodicibility of our results, we will release all such data together with our source code once our paper is accepted. Finally, we would like to clarify that in Section 4.3, all our results are based on the whole training dataset, which give the best results for all methods, but our proposed method remains the best.\n\n**Question For The Authors A: Why did your data sample your training data? Did you re-train and re-evaluate other models in your sample environment? Would this shift in the dataset change the model's performance, either in data efficacy or overall performance?**\n\n**Our response:** Regarding why we sampled the training dataset, we have explained above. We re-trained and re-evaluated all the models in every sampling environment. The performance of the model in a limited data environment can be seen in Figure 4. While the reduced amount of training data does reduce the accuracy of all models, but our model demonstrated an even higher performance gain relatively to the best existing model (CCN).\n\n**Typos Grammar Style And Presentation Improvements: Figure 1 Textual Evidence is misspelled. Line 136, metadata is misspelled.**\n\n**Our response:** Thanks for spotting such typos. We will fix them.\n\n**Soundness: 3: Good: This study provides sufficient support for its major claims/arguments, some minor points may need extra support or details.**\n\n**Our response:** We hope the above responses have clarified those minor points.\n\n**Reproducibility: 4: Could mostly reproduce the results, but there may be some variation because of sample variance or minor variations in their interpretation of the protocol or method.**\n\n**Our response:** Hope the above response on the data sampling has clarified this reproducibility point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes an innovative approach to one of the most prominent problems in the digital world and is highly relevant to society.\n2. Their method not only detects mis/disinformation effectively but also provides interpretability, which is helpful for experts or the audience. \n3. This method surpasses the existing best approach in a pressing issue. \n4. They provide extensive analysis of their model and comparison to show their model performance.", "reasons_to_reject": "1. The paper writing is too dense and somewhat hard to follow. \n2. In the paragraph starting at line 470, the paper has compared their method to the CLIP approach, which performs considerably lower than theirs. (In Figures 3 & 4, the model's name shows CNN, but in the text, it refers to the CLIP model). Comparing must be done your model with the existing best model on OOC scenarios. \n3. The scenarios (a, b, c, d) are unclear. It makes the analysis lines 476 and 482 hard to follow.\n4. The conclusion of the two-sample one-tail z-test is not clearly explained. \n5. The data sampling is indefinite. The chosen samples can affect the evaluation outcome.", "questions_to_authors": "A. Why did your data sample your training data? Did you re-train and re-evaluate other models in your sample environment? Would this shift in the dataset change the model's performance, either in data efficacy or overall performance?", "paper_topic_and_main_contributions": "This study thoroughly investigates the widespread problem of mis- and disinformation on the internet, focusing particularly on out-of-context (OOC) information. This occurs when unrelated pieces of information are falsely linked, resulting in a misleading combination of genuine images and deceitful textual descriptions.\nIn response to this issue, the researchers introduce an innovative solution called the Stance Extraction Network (SEN). The SEN is a comprehensive framework, purpose-built to distill the stances from a wide range of multi-modal evidence. This approach signifies an evolution in the study of OOC misinformation by appreciating the multi-faceted nature of the evidence involved and recognizing their inherent biases and unique perspectives.\nFurthermore, the research presents a novel integration of a support-refutation score into the textual SEN, which is a metric derived from the co-occurrence relations of named entities. This refinement amplifies the depth of analysis possible for textual misinformation, enhancing the overall robustness of the method and providing interpretability. \nThe researchers subject their proposed solution to rigorous empirical testing using an expansive public dataset. The resulting data underscores the superiority of the new method compared to existing state-of-the-art techniques, with the highest-performing model recording a performance increment of 3.2%.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1 Textual Evidence is misspelled. Line 136, metadata is misspelled."}, "extra_text": {}}, {"review_id": "iV2giifNjC", "reviewer": "Reviewer_RUdv", "summary": "", "questions": "You have provided a good set of comparisons with other baselines, however you have not included COSMOS by Aneja et. al. (2021) (https://github.com/shivangi-aneja/COSMOS) which you cite as well. Given how close the two papers are (with the exception of COSMOS not taking into account external evidence), was there a reason a comparison was not made?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and insightful feedback. Below please find our responses to issues and concerns raised by the reviewer.\n\n**Questions For The Authors: You have provided a good set of comparisons with other baselines, however you have not included COSMOS by Aneja et. al. (2021) (https://github.com/shivangi-aneja/COSMOS) which you cite as well. Given how close the two papers are (with the exception of COSMOS not taking into account external evidence), was there a reason a comparison was not made?**\n\n**Our response:** COSMOS is indeed related to OOC image-caption mismatch detection. However, the input of COSMOS is conceptually different from our OOC image-caption pair detection: the input of COSMOS a triplet (image, caption1, caption2) and it tries to detect if the two captions associated with the same image are semantically incoherent, which indicates that either caption1 or caption2 is the consequence of an OOC image-caption pair. However, our method takes one image and one caption as the input and predict if this given image-caption pair is pristine or OOC. These two detection tasks are not directly comparable, although our method can be generalized to the task COSMOS is designed for. Note that other researchers working on OOC also mentioned such difference in their papers, e.g., the NewsCLIPPings paper and the CCN paper. We do have a plan to generalize our method and compare how it can be used to detect incoherence in a triplet (image, caption1, caption2) like what COSMOS does, but we leave this as our future work and will add it to our final paper.\n\n**Typos Grammar Style And Presentation Improvements: What is the difference between misinformation and disinformation, in the context of your paper? Does your model processes misinformation and disinformation differently? if not, it may be helpful to just use one term to describe them, as is commonly done in the literature when the two are not differentiated by the method.**\n\n**Our response:** Our method can work with both mis- and disinformation as long as the image and the caption does not match. While OOC image-caption pairs are more likely disinformation in the first instance (i.e., the creator has a clear intention to cheat, although in rare cases the creator may accidentally mismatched an image with a wrong caption), some people may spread an OOC image-caption pair without knowing that it is false -- in this case the false OOC information will be more like misinformation (i.e., the spreader has no intention to cheat). We will clarify this point in our final paper and consider using the more neutral term \"OOC false information\" to replace the potentially confusing term \"OOC mis-/disinformation\".\n\n**Typos Grammar Style And Presentation Improvements: SOTA (state of the art) is spelled incorrectly as STOA (lines 428 and 429)**\n\n**A3:** Thanks for pointing out this typo. We will carefully proofread the paper again to identify and fix as many such mistakes as possible for the final edition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper addresses an important social problem: finding repurposed images that may spread online misinformation.\n\n- Sound methodology and study design; thorough analyses of results and model capabilities.\n\n- Submission includes code for reproduciblity\n\n- Reasonable and accurate description of limitations and ethical considerations around the proposed method.", "reasons_to_reject": "I have included some suggestions for improving the manuscript, but none are major enough to be a cause for rejection IMO.", "questions_to_authors": "You have provided a good set of comparisons with other baselines, however you have not included COSMOS by Aneja et. al. (2021) (https://github.com/shivangi-aneja/COSMOS) which you cite as well. Given how close the two papers are (with the exception of COSMOS not taking into account external evidence), was there a reason a comparison was not made?", "paper_topic_and_main_contributions": "The study takes an important social issue, \"detecting out of context misuse of images to spread online misinformation,\" and proposes a multimodal processing framework of multimodal evidence and content. The paper accounts for text and visual evidence and their relationship with the image-caption input pairs in detecting out of context misinformation in the input by proposing a stance extraction network. The authors provide comparison with several baselines on the NewsCLIPpings dataset and present ablation studies and analyses that support the effectiveness of their framework in addressing out of context multimodal misinformation. This paper also includes code for reproducing its findings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "What is the difference between misinformation and disinformation, in the context of your paper? Does your model processes misinformation and disinformation differently? if not, it may be helpful to just use one term to describe them, as is commonly done in the literature when the two are not differentiated by the method.\n\nSOTA (state of the art) is spelled incorrectly as STOA (lines 428 and 429)"}}]

Author rebuttal:

Meta-review: This research focuses on out-of-context (OOC) mis-/disinformation on the Web and proposes a method to predict it using evidence and stances consisting of text and images. 

Pros:
Mis-/disinformation identification is an urgent issue, and the effort is commendable. 
The approach is based on a good understanding of the nature of the problem. 

Cons:
Key questions from reviewers have been addressed in the rebuttal, but these need to be properly reflected in the camera-ready version.
Decision: Accept-Main