Exposing Text-Image Inconsistency Using Diffusion Models
Authors: Mingzhen Huang
Shan Jia
Zhou Zhou
Yan Ju
Jialing Cai
Siwei Lyu
Venue: ICLR 2024 poster (2024)
Keywords: inconsistency detection
multi-modal learning
diffusion models
Primary area: societal considerations including fairness, safety, privacy

Abstract: In the battle against widespread online misinformation, a growing problem is text-image inconsistency, where images are misleadingly paired with texts with different intent or meaning. Existing classification-based methods for text-image inconsistency can identify contextual inconsistencies but fail to provide explainable justifications for their decisions that humans can understand. Although more nuanced, human evaluation is impractical at scale and susceptible to errors. To address these limitations, this study introduces D-TIIL (Diffusion-based Text-Image Inconsistency Localization), which employs text-to-image diffusion models to localize semantic inconsistencies in text and image pairs. These models, trained on large-scale datasets act as ``omniscient" agents that filter out irrelevant information and incorporate background knowledge to identify inconsistencies. In addition, D-TIIL uses text embeddings and modified image regions to visualize these inconsistencies. To evaluate D-TIIL's efficacy, we introduce a new TIIL dataset containing 14K consistent and inconsistent text-image pairs. Unlike existing datasets, TIIL enables assessment at the level of individual words and image regions and is carefully designed to represent various inconsistencies. D-TIIL offers a scalable and evidence-based approach to identifying and localizing text-image inconsistency, providing a robust framework for future research combating misinformation.

Published as a conference paper at ICLR 2024
EXPOSING TEXT-IMAGE INCONSISTENCY USING DIF- FUSION MODELS
Mingzhen Huang, Shan Jia, Zhou Zhou, Yan Ju, Jialing Cai, Siwei Lyu University at Buffalo, State University of New York
ABSTRACT
In the battle against widespread online misinformation, growing problem is text-image inconsistency, where images are misleadingly paired with texts with different intent or meaning. Existing classification-bass based methods for text-image inconsistency can identify contextual inconsistencies but fail to provide explain- able justifications for their decisions that humans can understand. Although more nuanced, human evaluation is impractical at scale and susceptible to errors. To ad- dress these limitations, this study introduces D-TIIL. (Diffusion-basee) Text-Image Inconsistency Localization), which employs text-to-image diffusion models localize semantic inconsistencies text and image pairs. These models, trained large-scale datasets act "omniscient" agents that filter out irrelevant infor- mation and incorporate background knowledge to identify inconsistencies. In addition, D-TIIL uses text embeddings and modified image regions to visualize these inconsistencies. To evaluate D-TIIL's efficacy, we introduce a new THIL dataset containing 14K consistent and inconsistent text-image pairs. Unlike exist- ing datasets, TIIL enables assessment at the level of individual words and image regions and carefully designed represent various inconsistencies. D-TIIL offers a scalable and evidence-basee approach to identifying and localizing text- image inconsistency, providing robust framework for future research combating misinformation. Please refer Project Page for source code and dataset.
INTRODUCTION
The widespread online misinformation (Ali 2020) has become the bane of the Internet and social media. One simple means create misinformation is to juxtapose images with texts that do not accurately reflect the image's original meaning or intention. In this work, we term this type of misinformation text-image inconsistency (Lee Choi [2099] TTn al.) 22200) Zeng et al. (023). Exposing text-image inconsistency has become important task in combating misinformation. Text- image inconsistency can be solved with binary classification, as in the recent works MAIM (Jaiswal et al.
2017h. COSMOS (Aneja et al. 2021). NewsCLIPpings (Luo et al. 2021). and CCN (Abdelnabi
ct al. 2022). which classifies an input text-image pair as contextual consistent or inconsistent. Although showing good classification performance benchmark datasets, the classification-basee methods output only the predicted categories, with little or no evidence to support the decision. On the other hand, humans often spot text-image inconsistency by locating image regions corresponding to objects or scenes inconsistent with the textual description, using knowledge of the world. In addition, humans often prefer more visual evidence of semantic inconsistency, text-image pair is explained to another human. However, when we need to analyze many text- text-image pairs, relying on human inspection is costly, time-consumin consuminn,,, and prone to mistakes and errors (Molina et al. 202T). Our work aims to make this process automatic so it can scale up.
when mii-contextuulizze
Specifically, we aim to address two challenges in localizing text-image inconsistency intrinsic to the complex nature of semantic contents across the two modalities. First, there is unrelated information in text and images irrelevant to their semantic consistency. This is usually information only represented in one modality but not in the other. Unrelated information in one modality will not have counterpart in the other, but is not the cause of semantic mismatch and cannot be accounted for inconsistency. Furthermore, many cases of inconsistency are hard to identify due to limited background knowledge of humans or algorithms. For instance, to someone who is unaware that dolphins are mammals, a text stating "a school of fish swimming in the ocean" might seem consistent with image showing

---

Published as a conference paper at ICLR 2024
dolphins swimming. Such missing information can be overcome by using more knowledgeable human or incorporating background knowledge into the algorithm.
Both challenges are addressed in our work by leveraging the text-to-image diffusion models. Text-to-image diffusion models trained on large- scale datasets, such as DALL-E2 (Ramesh et al.) 2022), Stable Diffusion (Rombach al. 2022). Glide (Nichol et al. 2021). and GLIGEN (Li al. 2023). can generate realistic images with consistent semantic content in the text prompts. We can regard these large-scale text-to-image diffusion models as an "omniscient" agent with extensive background knowledge about any sub- ject matter. Taking advantage of this knowledge representation, we describe a new method that locates the semantically inconsistent image regions and words, which is termed as Diffussion-brn based Text-Image Inconsistency Localization (D-TIIL). D-TIIL proposes two different alignment steps that iteratively align the image and update text (in the form of vectorized embeddings) with diffusion models to (i) filter the irrelevant semantic information in the text-image pairs and (ii) incorporate background knowledge that is not obvious in their shared semantic scope. The first alignment step employs diffusion models to generate aligned text embeddings from the input image. This is to filter out implicit semantics and establish textual consistency. The second alignment operation focuses on denoising the input text to be more relevant to the input image. To be more specific, we modify the input image based on the original input text to achieve semantic consistency, and then produce aligned text embeddings from this edited image. These two alignment steps yield modified yet knowledge-shared text embeddings from both the input image and text, making it easier to identify semantic inconsistencies. This approach sets our method apart from previous ones, depicted in Fig.
Frevitss Mathods
Beary desäfer
Tecpnditters
D Offannon
Tanggle colliidd with mraclrr troder Wadheedyy.
Uncres/ery tors:
Corsolary scre 218
Exposing text- cext-image inconsistency based on previous methods and our method. Instead of employing a binory classification model, D-TIII-offers interpretable evidence by localizing word- and pixel-level inconsisten- cies and quantifying them through consistency score.
Figure I:
Existing datasets (Luo et al., 2021 Aneja et al. 2021) do not provide evidence of inconsistency at the level of image regions and words that can be used to evaluate D-TIIL. To this end, we create a new Text-Image Inconsistency Localization (TIIL) dataset, which contains 14K text-image pairs. Existing datasets construct inconsistent text-image pairs by randomly swapping texts (Luo et al. 2021) or external search for the similar text of the swapped texts (Aneja et al.) (02T) to match with the original image. These methods can create inconsistent text-image pairs that either conflict with human intuitions or are totally irrelevant (see more analysis in Sec Differently, TIIL constructs inconsistent pairs by changing words in the text, and/or editing regions in the image (e.g.. changing objects, attributes, or scene-texts(] The edited words and regions are manually selected. The image editing made with the text-to-image diffusion models. Furthermore, all inconsistent text-image pairs undergo meticulous manual curating to reduce ambiguities in interpretation.
The main contributions of our work can be summarized as follows:
We develop new method, D-TIIL,, that leverages text-to-image diffusion models expose text-image inconsistency with the location of inconsistent image regions and words;
Text-to-image diffusson models are used as a latent and joint representation of the semantic contents of the text and image, where we can align text and image to discount irrelevant information, and we use the broad coverage of knowledge in the diffusion models to incorporate more extensive background;
We introduce a new dataset, TIII, built on real-world image-text pairs from the Visual News dataset, for evaluating text-image inconsistency localization with pixel-level and word-level inconsistency annotations.
2 BACKGROUNDS
2.1 RELATED WORKS
Text-image inconsistency detection has been the focus of several recent works (AbdeInabi al.,
2022; Luo et al.) 2021 Qi et al.] 2021 Aneja et al.] 2021 Abdelnabi et al.] 2022). There are
many methods (Zlatkova et al., 2019: Abdelnabi et al. 2022
Popat et al. 2018 using the reverse
"THII has mixed both original and edited images; one cannot simply rely on forensic methods that identifying image editing to expose inconsistent pairs.
2

---

Published as a conference paper at ICLR 2024
image search function provided by the Search Engine (e.g., Google Image Search) to gather textual evidence (articles or captions) from the Internet or external fact-checking sources on the Internet (e.g., P and Factchec4(). The resulting text is then compared with the original text in an embedding space such as BERT (Kenton & Toutanova 2019) to determine their consistency. Although straightforward and intuitive, these methods rely solely the results from the reverse image search and are limited by irrelevant or contradicting texts found online. Other inconsistency detection methods explore joint semantic representations of texts and images. For example, Khattar al. (2019) designs multimodal variational autoencoder for learning the relationship between textual and visual information for fake news detection. McCraee et al. (02I)) detect semantic inconsistencies in video-caption posts by comparing visual features obtained from multiple I<de-unnderstandin networks and textual features derived from the BERT (Kenton & Toutanova, 2019) language model. Aneja et al. (202T) employs self-supervised training strategy to learn correlation from an image and two captions from different sources. More recently, neural Dision-langure
models originally
designed for other asiio-language tasks (e.g.. VQA, image-text retrieval) have also been applied to
text-image inconsistency detection. For instance, CLIP (Radford et al. (2021b) is used in Luo et al. (2021) and the VLn (Zhang et al. 2021) model introduced in Huang et al. (2022).
Several multi-modal datasets exist for image inconsistency detection. MAIM (Jaiswal et al. 201 7 ). MEIR (Sabir al. 2018)) FacebookPost (McCrae et al., (022) and COSMOS (Aneja et al. 2021
are formed by swapping the original caption of an image with randomly selected ones to create inconsistent image-text pairs. The NewsCLIPpings dataset (Luo et al., (02T) utilizes CLIP (Radford
it al. 202 as a retrieve model to swap similar captions in the Visual News dataset (Liu et al. 2020). The main problem with these datasets that the semantic relations among the labeled consistent or inconsistent pairs are not precise (Huang et al., 2022), making them less reliable to be used as training data for text-image inconsistency detection methods. These methods and datasets do not report words or image regions that cause the inconsistency.
2.2 TEXT-TO-IMAGE DIFFUSION MODELS
The diffusion model (Ho et al.) (0220) has recently attained state- -the-art performance in the field of text-to-image generation |Ramesh et al. [022]] Saharia et al., 2022 Rombach et al. 2022). As category of likelihood-based models (Nihhol & Dhariwal (02T). diffusion models perturb the data by progressively introducing Gaussian noise to the input data and train to restore the original data by reversing this noise application process. The key idea involves initialization with x ~ (0, I). which represents an iteratively noised image derived from the input image Xo- At each timestep t E [0. T]. the sample x, is computed as x; = V0xxXo + vT - Oftt, where Or E (0. 1] defines the level of noise, and 6, ~ I) represents the sampled noise. Ultimately, the distribution of x approaches a Gaussian distribution. Diffusson models then iteratively reverse this process and denoise X1 to generate images given text conditioning by minimizing simple denoising objective:
E Ey,xo. || Eo (x,, t, c.))
where eg is an UNet (Ronneberger al. 2015) noise estimator that predicts e, from x,. Diffusion models have been widely used in various downstream applications, including image editing (Couairon et al. 2023; Kawar et al. 2023) where text-conditional diffusion model can be generalized for learning conditional distributions. When provided with different text conditionings, the model generates different noise estimates. Notably, the variation in noise across spatial locations reflects the semantic distinctions between the corresponding text conditions in the image space. This inspiration motivates us to utilize diffusion models for representing and exposing semantic inconsistencies in image inconsistency.
3 METHOD
This section describes the proposed D-TIIL model in detail. The input D-TIIL is pair of image I and text T, from which D-TIIL outputs the image region (as a binary mask M) and words in the text that exhibit semantic inconsistency. In addition, a consistency score r E [0, 100), with being maximum inconsistent and 100 being completely consistent, is also obtained based on the localization results. The overall process of D-TIIL is illustrated in Fig. 2 with four distinct steps. In those four steps, we iteratively align image-text semantic to produce final output as shown in Fig.
PPllitifact: https:/wwww.faccccccc..ogg Factcheck: hutps//wwww.pllifffcc.coomm
3

---

Published as a conference paper at ICLR 2024
"The 1992 ad featuring the supermudel drinking orange juice in frunt of two pubescent boys proved that sex appeal sells products. She was the start mony Pepsicelebrit collaborations iochuling Super Bowl XLVII hulfiime performer Beyonce"
Eo
Step 1
Step 2
Step
Step
M
Image- regulat-e Text Alignment
reg.lated.l.........
Mask
Image Editing
Eo Imagee- regulated Text Denoising
I,E
Mask
Generation
Generation
Eo
Eain
Step 1
Step &
"orange juice" 23.1
e/T
Naise
D ffusion
Model
Reconst/uction
Image
Neise
Diffusion Medel
Eate/Edt
Eg/E
Fddt
Diffusion Model
Noise Estimation
Edin
Figure 2: The overall pipeline of D-IILL See texts for details.
Step 1: Align Text Embedding to Input Image. We first obtain the CLIP (Radford et al. 2021 text embedding Eg of the input text T. Using a pre-trained Stable Diffusion model ç (Rombach et al., 2022). we find another text embedding Eatn that better aligns with the semantic content of the input image as shown in Fig. Step 1. Model G takes as input the text embedding Eo and noised image XT to generate an image, (xT; Ey). It simulates diffusion process that begins with the input noise and generates an image that exhibits similar semantic content to the text embedding Eg. The image-regulated text alignment D-TIIL is to solve the following optimization problem:
Eoin argmin || (xx:; E) s.l., ||E Folle 5
(2)
where the learnable E is initialized from Eg, and y > 0 is a small constant determined as a hyper- parameter. || ll and 1|-1 || are the vector & norm and matrix Frobinuous norm, respectively. The constraint is used to control the deviation from the original embedding. This optimization problem be solved using the gradient computation of G iteratively. The obtained Fain is semantically closer to the image when the original text T has inconsistencies and distracting semantic information.
Step 2: Text-guided Image Editing. Next, generate edited image, Teat which is in alignment with the original text embedding, Eg. This is intended to materialize the original text, T. within a visual context, thereby minimizing the presence of extraneous and implicit data. subsequently acts as the benchmark for estimating inconsistency. D-TIIL transposes the semantics of Eu into the image space. with both Eo and the aligned text embedding Eain guiding the editing process. Specifically, we introduce noised version of image 7 as the input and leverage the UNet architecture from the Stable Diffusion to derive two noise estimations, each corresponding to Fep as target and Eain as reference. By examining the difference between these two noise estimates in the spatial domain, we can identify regions in image 1 that are most prone to modifications due to the shift in conditioning text from Eatn to En. This disparity is then transformed into binary mask, denoted as M,, by normalizing values within the [0, 1] range and subsequently employing a thresholding operation. After obtaining M'. use the Diffusion Inpainting model (Lugmayr et al. (002)) to yield the edited image Tedt, guided by the target text embedding Eg and mask M'. The process effectively transmutes the textual embedding Eg into visual equivalent, purging any distracting or implicit details, as shown in Fig. Step 2.
Step Align Text Embedding to Edited Image. While the binary mask M' captures the regions of inconsistency between the input image / and text embedding Es, it may still include regions that are directly related to semantic consistency, such as the objects scenes in I and Ledt that does not correspond to verbal description in T. This is form of unrelated information that we use another round of operation involving the diffusion model G to reduce. Specifically, we formulate another optimization problem Edut argming || Teds G (xedt:, E) s.t., E Follr S where xedt is noised image of Tedt- Compared with input text embedding Fg, the aligned text embedding Fant includes extra implicit information from the images and excludes additional implicit information that only appears in the text as shown in Fig. 3 Step where we refer this operation as image-regulated text denoising.
Step 4: Inconsistency Localization and Detection. From the two text embeddings that are more closely aligned, Eai and Edut. we generate the difference in visual domain denoted as M. This is done by repeating the mask generation process outlined in Step 2. The M represents the pixel-level inconsistent region within the image. To detect the corresponding inconsistent words, compare the
4

---

Published as a conference paper at ICLR 2024
1/Ech
Tedh/Ead
Traee ctaney
Sunantc
Input
Taltt
Stop
Stap
Output
Eicoe Hacc fort tirt Comala dlree ald
Egst Techt
Eght
The main process of D-TIIL is illustrated conceptually with Venn diagrams, where the semantic contents of text and image are represented as two circles. The four steps gradually align the semantic contents to facilitate exposure of inconsistency: given initial image-text pair (I, Bo), the proposed method first produces a text embedding Eatn that aligned with I, and then an edited image Tedt to filter the inconsistency. In Step 3, the model optimizes Eo from the Teds to obtain a Edns which is aligned with Teas. Finally, Step 4, the model produces the inconsistency mask from well-aligned pair (I, Faln: Fddn))
Figure 3:
edited image Ledt with the inconsistency mask M. Specifically, we leverage the CLIP image encoder to obtain the image embedding from the image Leat. Next, we derive tokenized words from the rows in Eg that exhibit the greatest cosine similarity with the image embedding, such as the example of "an orange juice" depicted in Fig. 2 of image-text inconsistency, we extract the CLIP image embedding from the masked input image using M. We then compute the cosine similarity score between this CLIP image embedding of the masked image and the input text embedding Eg as the consistency score. The resulting score is rescaled to the range of [0, 100], serving as the final consistent score r of our D-TIIL model.
To further generate a regression score that quantifies the degree
4 TIIL DATASET
We also construct TIIL more carefully curated dataset for text-image inconsistency analysis. Our approach to dataset creation is different from those of existing datasets that use randomly or algorithmically identified pairs mismatched image-text pairs (Jaiswal et al., 2017, Sabir et al.) 2018: Aneja et al.] 2021). We leverage state-of-the-art text-to-image diffusion models to design text-guided inconsistencies within images and human annotation improve the relevance of the inconsistent pairs.
Data Generation. the Visual News dataset (Liu et al. 22020). which offers a rich variety of news topics and sources, providing us with diverse real-world news data. The first step in our process involves creating an edited image, Je. This is achieved by modifying a specific region in 7 using an altered text Tm by human annotators, where the text prompt corresponding to the object is replaced with different generation term. The region to be manipulated is manually selected. Through this procedure, we can generate two consistent image-text pairs, namely (I.T) and (e. ). as well as two inconsistency pairs formed by {I, Im. } and (Le, T}. The DALL-E2 model (Ramesh al. 2022), known for its capacity to generate images within specific regions based on text prompts, is leveraged to create this dataset. Table demonstrates that the DALL-F22 model outperforms real-world image-text pairs in terms of CLIP similarity scores, indicating its superior ability to capture multi-modal connections. Fig. illustrates the entire generation pipeline of our TIIL dataset. The process starts with a real image-text pair. Human annotators then identify the corresponding visual region and textual term. Subsequently, these annotators provide a different text prompt to replace the chosen term, thereby creating an inconsistency with the selected object region. The mask of the selected object region and the swapped text with the new prompt are then fed into the DALL-E2 model for image generation. In the final step, human annotators carefully assess the quality of the generated images, evaluate the image-text inconsistency, and refine the region mask to provide the ground truth for the pixel-level inconsistency mask. The TIIL dataset consists of approximately 14K image-text pairs, encompassing a total of 7,138 inconsistencies and 7,101 consistent pairs. All inconsistent instances in the dataset have been manually annotated.
Our methodology starts with real-world image-text pair, {I,T), obtained from
Manual Annotations. We also go through a manual meticulous data annotation process by a team of six professional annotators. The annotation process is carried out following defined procedure. First, the annotators select object-term pairs that align with each other, and then, they input the target text prompt that corresponds to the selected object-term pairs. The final step of the process involves
5

---

Published as a conference paper at ICLR 2024
Quality Creck
DALLE22
Instance Reg
Segmentation
Grighel Deperatez
tour bus on the Nive
"tour bus" to "school bus"
Jersey Tarrpiic coliided witha tracter trailer
collided with tractor trailer Wedhenday"
Wedresday
Text prompt
Irput
Human Annatation
Swapped
Senorated Images
Data Cluzning
Figure 4: Pipeline depicting the generation and annotaation process of the proposed THL dataset.
COSMOS
FacebookPost
MAIM
RO
EEMASST
PhoSe PaIns
Minocity Leader Nancy on Kobe Bryan's last game for the LA "Budapest in August 2014""
sunbather casching moments before bites her"
"Mike Tysun and his wife Lakibai
Lakers'
Figure S: Inconsistent examples from other datasets. Due to the constraints of using random swapping or auto-retrieval methods to produce inconsistent pairs, the resulting pairs could either be semantically consistent (as seen in the left two examples) entirely unrelated (as illustrated by the right two examples).
Amazon has snapped up the one deal one
day online retailer Wool. Above Amazon warehouse Nevada
amazon.com
TIIL: inconsistent image generated
Original image
NewsCL.Ppings
with edited "Nike"
Figure 6: inconsistent example in our THIL and News NewsCLiPpings Note that the THIL example agrees with common viewer to be inconsistent while the one from NewsCLiPpings is consistent rather than inconsistent.
Avg. CLIP Score
Total
Inconsistent
Anno
Generation
Pixel&word-lee
Real-world
30.71
Dataset
oPairs
#Pairs
#Pairs
Method
Annotation
DALL-E2
31.03
MAIM
240K
120K
Random swap
Random Swap
12.39
MEIR
58K
29K
Random swap
NewsCLIPpings
988K
494K
Auto retrieval
lurs-Inconsisten
22.68
Facebook Post
4K
2K
Random swap
COSMOS
200K
850
1.7K
Manual
Table 1: Comparison of the CLIP
TILL (Ours)
14K
7K
14K
Minual-Diffusion
scores for different pairs.
Table 2: Comparison with existing related datasets.
data cleaning to ensure the accuracy and coherence of the dataset. To maintain the highest quality, the annotations are cross-validated among the team members. This step allows for the detection and rectification any potential errors or inconsistencies. Further details about our data annotation process are in the Supplementary Material.
Comparison with Existing Datasets. As shown in Table 2 our TIIL dataset boasts several unique characteristics that set it apart from existing datasets. To the best of our knowledge, it is the first of its kind to feature both pixel-level and word-level inconsistencies, offering fine-grained and reliable inconsistency. We provide comparison of the CLIP scores between the traditional random swap- based creation method and our diffusion-basse approach in T The results demonstrate the superiority of method in achieving higher levels of semantic similarity compared to randomly swapped image-text inconsistency pairs. This higher semantic similarity makes the inconsistencies more subtle and harder detect, thereby enhancing the dataset's complexity and realism. In comparison to datasets that build inconsistency pairs through random swap (e.g., MAIM (Jaiswal al. 2017). MEIR (Sabir al. 2018). Facebook (McCrae et al., 2022) and COSMOS (Aneja et al. 2021) or automatic retrieval (such as NewsCLIPping (Luo et al. (02T)). TIIL offers more inconsistent pairs, as demonstrated in Fig. 5 and Fig 6 Moreover, our dataset is sourced from wide array of news topics and sources, ensuring diverse and rich collection of examples. For a more comprehensive understanding, we present examples of image-text pairs and annotated labels from the TilL dataset in Fig. These examples underline the range and complexity of the data within our novel dataset.
reliable consistent

---

Published as a conference paper at ICLR 2024
"View photo gallery Washington Post food critic Tom Sietseme picked the Dupont Circle location of sun FShaae Shackamong the city's best cheap
kite/drome flies International Consumer in Jamuory 2014 in Las Vegas"
YAHOO!
Google
"A ichool bu hew streetcar is placed on the rails along H St NE Figure 7: Examples in TILL dataset. Colored texts correspond to inconsistent regions of the same color in the image. The figure best viewed in color.
"GoogiefYaboo plans to spin off core web basiness"
M
M
Score: 13.2
Score: 26.3
Score: 31.6
Score: 11.8
Score:
"Britain's 's Queen Diana leaves the anmuat Braemar Highlond Gathering
"The vase sitting the object dating back to the previous
"Roast steak Pomegr- anate and Dute Molasses"
"A school bus placed on the ralls along H St NE"
"Brad Pin Years Slave"
century"
Scotland Sept 6 2014"
Figure 8: D-TIIL on TILL examplex. The detected inconsistent words are highlighted in red.
EXPERIMENTS
This section presents a comprehensive analysis of our approach, including qualitative and quantitative results, comparisons with other methods, and ablation studies to evaluate different variations.
5.1 SETTINGS
Implementation Details. We use the implementations of Stable Diffusion (Rombach et al., 2022) and CLIP (Radford et al. 2021a) ViT-B/32 model available on https: http::://hggggggaaacccc co In the diffusion model, we use the denoising diffusion implicit model (DDIM) (Song al. 2020 to sample noises, and classifier-
guidance (Ho & Salimans 2022) is set to the recommended value of 7.5. We train both text embedding Ealn and Ednt for 500 iterations with learning rate of 4e The hyperparameter is set to in our experiment. For noise estimation, we use the same random seed for two conditioned text embeddings, remove outlier values in noise predictions, and average spatial
7

---

Published as a conference paper at ICLR 2024
DeICLIP
GAE
Ours 47.12
CLIP
CLIP
CCN
Ours
mlaU (%)
14.24
27.35
AUC (9)
84.18
78.58
82.80
87.00
Accuracy (%)
76.77
71.87
74.10
79.46
Table Comparison of text-inme -maage inconsistency localization.
Table 4: Comparison of detection.
differences set of 10 input noises. After obtaining the predicted inconsistent masks, we use threshold to binarize them where the threshold is the average values among the mask. We only retain the top 3 mask regions with the largest areas. To localize inconsistent words, we follow the previous work (Radford et al. 2021a) to use a prompt template "A photo of [words]" 3 for CLIP (Rafford et al. 2021a) text embedding generation.
Evaluation Metrics. We report the mean class-wise intersection over union (mloU) EEeringham et al., 2015) to evaluate the quality of the predicted inconsistency mask. mloU is metric that aligns with the per-pixel classification formulation, making it a commonly used standard metric in semantic
segmentation tasks (Fan al. 2021 Klingner al. 2021
Gao et al. 2022 Xu al. 2022)
5.2 COMPARISON WITH EXISTING METHODS
We first present qualitative results demonstrating the pixel-level and word-level detection of incon- sistency achieved by the proposed D-TIIL model in Fig. 8 It can be observed that our D-TIIL achieves accurate results benefiting from the multi-step semantic alignment. Given the absence of prior work especially in addressing image inconsistency localization, two relevant baseline approaches for comparison: (1) a straightforward solution that uses an object detector (Zhou et al. 2022) to detect all objects segmentation mask in the image and then compares the CLIP CLIP (Radford et al. 2021a) embedding similarity between the text and each object region. The object region with the highest dissimilarity is identified, and its corresponding mask is considered as the inconsistent mask. We denote this method as DetCLIP; (2) an off-the-shelf method, GAE (Chefer) al. 2021), which provides explainability for bi-modal and encoder-decoder transformers by pre- senting co-attention maps. Specifically, it analyzes the classification relevancy of specific layers for CLIP (Radford al.] (201la) to provide a pixel-level attention heatmap. We further generate the inconsistent mask with the attention heatmap by applying a threshold. The comparison results on the TIII. dataset are shown in Table and Fig. 9 The D-THIL method demonstrates significant superiority over the baseline methods in both mloU scores and qualitative evaluation.
consider the following
5.3 ABLATION STUDIES
Text-image Inconsistency Detection. While we have emphasized that binary classification may not be the best method for revealing inconsistencies in text image pairs, we have adapted the D- TIIL model into binary framework. This allows inconsistency detectors, such as the CLIP model (Radford et al. 2021a). its fine-tuned version on NewsClippings (referred as CLIP*), and recent detector that shows the best performance CCN (Abdelnabi et al. 2022). We report the Area Under ROC (AUC) and Accuracy. As shown in Table the D-TIIL model outperforms other models terms of both AUC and Accuracy scores. This improvement can be attributed to the use of the detected inconsistency mask for image embedding extraction, which enables the exclusion of distracting and implicit regions from the image, thereby enhancing the classification performance. Moreover, compared with CCN, which requires inverse searches for inconsistency detection, our D-TIIL does not require information from external sources, especially showing superior performance pairs with manipulated images that cannot be found on the Internet. Specifically, the experimental results demonstrate that CCN achieves an accuracy of 80.12% on the subset composed entirely of original images/text sourced from the Internet, However, its accuracy drops to 68.0% for subsets containing manipulated image/text. In comparison, our subsets, respectively, indicating minimal
compare D-TIIL with current text-image
method obtains accuracy of 82.79% and 79.15% impact on the online accessibility of data.
image-rrgulande Text Denoising. We first highlight the benefits of performing additional image- regulated text denoising in Step 3 instead of using the M' in Step as the inconsistency map. Table and Fig. g reveal that M' exhibits a coarse pixel-level inconsistency map, whereas the detected inconsistency mask includes additional background areas due to the absence of denoising in the input text embeddings.
8

---

Published as a conference paper at ICLR 2024
"Shrimp Tocos on a nice plate"
TTocss" by GAE
"Shrimpp Tucos"
"The History Museum
"huxvy
by D-TIIL
displays a luxury
by GAB
by D-TIIL
Figure 9: Comparison with GAE for detecting inconsistent mask.
(a) Original image
M
Tede
(b) Original image
M
Tedt
(bj: "A stuffed animal and fowers Wills Memorial Park in La Plata MD where Jibre died last May" Figure 10: Failure cases D-THIL TIIL dataset. (a) is consistent image-texx pair and (b) inconsistent image-text pair. Detected inconsistent words are highlighted in red.
Text (al: "Rev Nace Lanier poses for Ronald Reagan National Airport"
office
white swing in
Text Embedding Alignments. We further compare our method with two variations to show the influence of the text embedding alignment process inconsistency localization. This experiment is conducted in a randomly sampled subset TIIL with 1000 image-text pairs. We consider two initialization variations, (1) random initialization of Eg from random noise: and (2) initialize E with Eo and only supervised by the reconstruction L2 loss without the text embedding constrain loss in Eq. (2). Table demonnstatts that the performance decreases when the aligned text differs from the initial text embedding Eg too much.
5.4 FAILURE CASES
Fig. 10 shows two examples that D-TIIL. does not generate results consistent with human viewers. Such cases are attributed to the limitations of the underlying text-to-image diffusion models in understanding the entailment semantic meanings of words and creating precise local edits of images. In Fig. 10 (a), the word "office" is likely to be taken too literally by the model, even though human viewer may extrapolate to this unusual setting. The case of Fig. TO (b) is clear inconsistent pair, the word "stuffed animal" not reflected in the original image, but D-TIIL finds misaligned inconsistent regions.
CONCLUSION
In this work, we describe D-TIIL to expose text-image inconsistency by employing diffusion models as an omniscient, impartial evaluator learn the semantic connections between textual and visual information. Instead of using a binary classification model, D-TIII offers interpretable evidence by determining the inconsistency score of an image-text pair and pinpointing potential areas where the text and image semantics disagree. We also provide new dataset, TIIL, built on real-worke image-text pairs, for evaluating our D-TIIL method. Experimental evaluations D-TIIL on this dataset demonstrate improved and more explainable results than the previous methods. There a few directions we would like to enhance the current work in the future. Given the limited prior knowledge of the diffusion model used, our model may effectively handle the inconsistencies with respect to specific external knowledge. One potential solution is to replace our general foundation diffusion model with Nomain-specializen diffusion models learn semantic connections specific those domains (e.g., a text-to-image diffusion model trained on a Fashion dataset (Sun et al. 2023 Karras et al. 2023) that generates fashion images would identify inconsistencies in Iashion-relates text-image pairs such as mismatched brands or styles). Furthermore, it is important to continue to enlarge our dataset with more recent text-prompt image generation models.
M'
M
Random Init w/o Emb Constrain
Ours
mioU (%)
38.43
47.12
mloU (%)
41.27
43.47
47.50
Table 5; Benefits of text denoising.
Table 6: Comparison of text embedding alignments.

---

Published as a conference paper at ICLR 2024
Ethics Statement. This work is relevant to the fight against misinformation, which is vexing problem that reduces the integrity of online information. While our method can effectively expose misinformation created with text-image inconsistency, there is risk that the misinformation creator may use our method to select more deceptive text and image pairs, for instance, only use those that pass algorithm. The mitigation such abuse is to continue improving the algorithm and only provide its access to trustworthy parties. We will only release our code as open-source with the condition that it must not distribute harmful, offensive, dehumanizing content otherwise harmful representations of people or their environments, cultures, religions, etc. produced with the model weights.
Acknowledgement. This work was supported in part by the US Defense Advanced Research Projects Agency (DARPA) Semantic Forensic (SemaFor) program, under Contract No. HR001 120C0123, and National Science Foundation (NSF) Project SaTC-21531 12. The views and conclusions contained herein are those of the authors and should be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, NSF, or the U.S. Government.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (4940-149493 2022.
Sana Ali. Combatting against covid-19 & misinformation: systematic review. Human Arenas, pp. 1-16, 2020.
Shivangi Aneja, Chris Bregler, and Matthias Nicßnnr. Cosmos: Catching out-of-context misinforma- tion with seffssuuervisso2222lllafffffer learning. arXiv preprint arXiv:2 2021.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 397-406, 2021.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- semantic image editing with mask guidance. In The Eleventh International Conference Learning Representations, 2023.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 11:98-1366 2015.
Mingyuan Fan, Shenqi Lai, Junshi Huang, Xiaoming Wei, Zhenhua Chai, Junfeng Luo, and Xiaolin Wei. Rethinking bisenet for real-time semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9716-9725, 2021.
Shanghua Gao, Zhong- Yu Li, Ming-Hsuan Yang. Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Mingzhen Huang. Shan Jia, Ming-Ching Chang, and Siwei Lyu. Text-image de-contextualization detection using vision-language models. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8967-8971, IEEE, 2022.
Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan. Multimedia semantic integrity assessment using joint embedding of images and text. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1465-1471,201
Johanna Karras, Aleksander Holynski. Ting-Chun Wang, and Ira Kemelmacher--Shizzrmmn Dream- pose: Fashion imagc-to-video synthesis via stable diffusion. In ICCV, 2023.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, pp. 2019.
Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. Mvae: Multimodal variational autoencoder for fake news detection. In The world wide web conference, pp. 2915- 2921, 2019.
Marvin Klingner, Andreas Bar, Marcel Mross, and Tim Fingscheidt. Improving online performance prediction for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-11, 2021.
11

---

Published as a conference paper at ICLR 2024
Kiljae Lee and Jungsil Choi. Image-text inconsistency effect product evaluation in online retailing. Journal of Retailing and Consumer Services, 49:279-288 227799888220 2019.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2251 2023,
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. arXiv preprint arXiv:2010.037/4 2020.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu. Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 11461-11471, 2022.
Grace Luo, Trevor Darrell, and Anna Rohrbach. Newselippings: Automatic generation of out-of- context multimodal media. arXiv preprini arXiv:2104.008895 2021.
Scott McCrae, Kehan Wang, and Avideh Zakhor. Multi-modal semantic inconsistency detection in social media news posts. arXiv preprint arXiv:2105. 2021.
Scott McCrae, Kehan Wang, and Avideh Zakhor. Multi-modal semantic inconsistency detection in social media news posts. In MultiMedia Modeling: 28th International Conference, MMM 2022, Phu Quoc, Vietnam, June 6-10, 2022, Proceedings, Part I1, pp. 331-343. Springer, 2022.
Maria D Molina, S Shyam Sundar, Thai Le, and Dongwon Lee. "fake news" is not simply false information: concept explication and taxonomy of online content. American behavioral scientist, 6502):180-212, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR 2021.
Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. Declare: Debunking fake news and false claims using evidence-aware deep learning, arXiv preprint arXiv: 180..00000 2018
Peng Qi, Juan Cao, Xirong Li, Huan Liu, Qiang Sheng, Xiaoyue Mi, Qin He, Yongbiao Lv, Chenyang Guo, and Yingchao Yu. Improving fake news detection by using an entity-enhanced framework to fuse diverse multimodal clues. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1212 1220, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 202
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 2xx.x:2203.000 00020, 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 200i::2204406125,, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 241. Springer, 2015.
12

---

Published as a conference paper at ICLR 2024
Ekraam Sabir, Wael AbdAlmageed, Yue Wu, and Prem Natarajan. Deep multimodal image- repurposing detection. In Proceedings of the 26th ACM international conference on Multimedia, pp. 1337-1345, 2018.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic ext-tt- diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wrXi:22010.02502, 2020.
Zhengwentai Sun, Yanghong Zhou, Honghong He, and PY Mok. Sgdiff: A style guided diffusion model for fashion synthesis. 2023.
Reuben Tan, Bryan A Plummer, and Kate Saenko. Detecting cross-modal inconsistency defend against neural fake news. arXiv preprint XxXi:2009.006995 2020.
Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multi-class token transformer for weakly supervised semantic segmentation. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 4310-4319, 2022.
Zhi Zeng. Mingmin Wu, Guodong Li, Xiang Li, Zhongqiang Huang, and Ying Sha. Correcting the bias: Mitigating multimodal inconsistency contrastive learning for multimodal fake news detection. In 2023 IEEE International Conference Multimedia and Expo (ICME), pp. 2861-2866. IEEE, 2023.
Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang. Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations vision-language models. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühh. and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pp. 350-368. Springer, 2022.
Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. Fact-checking meets fauxtography: Verifying claims about images. arXiv preprint arXiv:1908.1 11722, 2019.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
In the Appendix, we provide more details about TILL dataset and conduct additional ablation experiments.
A TIIL DATASET
We provide more annotation details and statistics about the TILL dataset in this section. More examples in the dataset are shown in Fig. I1
Annotation Details. The annotation process was carried out by a team of six annotators with professional background, including 1 postdoc from the research team, 3 graduate volunteers, and 2 undergraduate volunteers. These annotators possess a comprehensive understanding of the data annotation task, which comprised three steps: (I) selecting matched object-term pairs and inputting the target text prompt, (2) manipulating the image or text with exact instruction and (3) conducting data cleaning as the final step. The generation of the TIIL dataset starts with real image- text pair. The first annotation process involves identifying the corresponding visual regions and textual terms through human involvement. Initially, we automatically extract separate text terms with spaCy $ human annotators then select and annotate the visual region that corresponds to the matched text term. The mask is annotated with the CVAT annotation platform ! Additionally, the annotators provide a target text prompt with the instruction that i) it should be inconsistent with the original text but match the context that may mislead the readers; ii) it should not share semantic overlap with the original term, for instance, replacing "Chiwetel Ejiofor" with "Brad Pitt" rather than "a male actor". By utilizing the selected object region and annotated prompt as inputs to the DALL-E2 model, obtain three manipulated images for each image-text pair. The second phase of the annotation process focuses on data cleaning to ensure the accuracy and coherence of the dataset. Each human annotator carefully follows three steps for quality checking purposes: (1) assessing the image quality of the generated images, (2) evaluating image-text inconsistencies, and (3) refining the region masks to establish ground truth for pixel-level inconsistency masks as the generated objects may have different shapes. To maintain the highest annotation quality, rigorous Prossvvalidation procedure was implemented within the team. This involved multiple assessments of each image-text pair by different annotators.
Dataset Statistics. The images in TIIL dataset have resolutions ranging from 256x396 to 3744x3744 pixels, and on average, the consistent mask covers 44.73% % of the entire image area. There are total of 7,101 consistent pairs in the dataset. Among them. 2.101 are composed of original images and text sourced from the Visual News dataset (Liu et al. 2020 (referred to as {1, T) in Section 4 of the main paper). The remaining pairs consist of images generated by DALL-E2 (Ramesh et al. 2022) along with their corresponding text (referred to {Ie, ). The dataset also includes 7,138 inconsistent pairs, out of which 2,101 pairs consist of original images with manipulated text (referred to as Tm,.m. }) while the remaining pairs consist of images generated by DALL-E2 (Ramesh et al.) 2022) paired with the original text (referred to as! T)). Regarding manipulation regions, the dataset contains 3,015 annotated inconsistent regions of large size (greater than 200 200), 1,548 annotated inconsistent regions of medium size (ranging between 100 x 100 and 200 x 200), and 474 annotated inconsistent regions of small size (smaller than 100 x 100).
B ADDITIONAL ABLATION STUDIES
To demonstrate the impact of mask binarization threshold and text embedding alignment initialization methods on the performance, we offer additional ablation studies in this section.
Text Embedding Alignment Initialization. Instead of initializing the text embedding with random noise, we choose to initialize it with the embedding Es of the input text. In Ta the main paper, we have provided a comparison of different embedding initialization methods in terms of mloU. Here, we further present a comparison between random initialization and our Eased initialization using CLIP scores to explain our motivation. Table Table demonstrates that the semantic similarity between inconsistent text embeddings (i.e., the input Eg and the text embedding corresponding to the input
' https///ttthh.omm/esppsssssy/yyy https//www.cvat.ai/
14

---

Published as a conference paper at ICLR 2024
"An he helicopter Alllgiian Air offering flights BWI"
"An caka Asiian Stoppy Joes
"One of the rescued eutevardog from South Corolina sholter that was in the path of Hurricane Matthew"
"Harambe 17-year-old Nordlorillaa at the Cincinnati Zoo
"Of the 216 condos 45 percent of them have one diving roondeedroom two
"A rabbid/ifongomeey lawnlovers are arms"
"A samescx marriiae supporter waves a UK Plagha rainbow flog fyont of the US Supreme Court"
Mulherin's grilled fried salmonflamb
steak with salad of thell beans"
Examples in THIL dataset. Colored texts correspond to inconsistent regions of the same color in the image. The figure best viewed in color.
Figure 11:
inconsistent image) significantly higher than the similarity of random noise embeddings (i.e., the input Eg and the randomly initialized embedding). This highlights the advantage of initializing with Ep. as it not only serves as reference for optimizing E, but can also expedite the convergence during the alignment process.
Random Init
Ours
CLIP Score
0.19
80.98
Table 7: Comparison of different text embedding initialization methods.
Comparison Between
and Text-manipulated Subsets. As TIIL dataset is and text-
able be categorized by two different kinds of manipulations: mmage-manipulated manipulated set, we further provide the performance of different methods on those two subsets for both localization (in Table R and detection (in Table 9). The text-changed samples achieved slightly worse performance than image-manipulated data, since the text changes (such replacing a word or phrase) are less impactful than image changes (where various region sizes are manipulated), therefore more difficult to be detected.
Image-changed
Text-changed
DeICLIP
11.93
15.21
GAE
28.57
26.85
Ours
50.28
45.57
Table 8: Comparison the localization performance different samples.
15

---

Published as a conference paper at ICLR 2024
Threshold:
Threshold:
Threshold:
Average threshold
M
Jede
M
Score: 24.3
Score: 20.1
Score: 70.8
Score: 11.9
"A dog stood proudly, surrounded by the crisp autumn air"
Threshold: 0.7
Threshold; 0.3
Threshold:
Averoge fhreshold (~~ 0.2)
M
Jodl
Score: 13.5
Score: 58.7
Score: 66.1
Score: 12.3
"A cat shown recovering will be adopted by a veterinary technician"
T
Figure 12: Comparison of D-TIIL TIIL examples with different thresholds.
Mask Threshold. We compare four fixed threshold strategies with our average-based method, which uses the average values among the mask as the threshold for M' and M. The results in Table 10
16

---

Published as a conference paper at ICLR 2024
AUC (image-changed) ACC (image-changed)
AUC (text-changed)
ACC (text-changed)
CLIP Ours
86.50
79.11
78.58
71.87
88.80
81.25
82.29
74.70
Table Comparison of the detection performance in different samples.
0.1
0.3
0.4
Average (Ours)
mloU (%)
39.66
42.81
43.23
42.66
47.50
Table 10: Comparison of different mask thresholds on subset of THI. with 1,000 image-text pairs.
and Fig. 12 show that using relatively smaller threshold results in a lower mloU score and larger predicted area, wherein the generated image Ledt includes more "implicit" backgrounds. On the contrary, when using larger threshold, there is increase in the mloU, but it can also lead to smaller predicted inconsistency area. This particularly evident when setting the threshold to 0.4, as depicted in Fig. 12 Our method surpasses the performance of fixed threshold strategies by utilizing an adaptive threshold for mask binarization.
painting for Rockville Connecticut Bethessa"
"Icicles hang from the caves northern Wisconsin"
(a) Completely inconsistent image descriptions
"Karim Benzema from Atletico Madrid celebrates goal in Mudrid derby"
"Roses playing basketball in October 1987"
"Andy Murvay kicking a footholl Jeryy Jonowier during their semiifinal mutch"
lovely dog ball the leaves"
(b) Objects are aligned but attributesfpredietes are not.
"Woman holding while walking throggh the city" "Portrait woman forest"
(c) Backgrounds or scenarios are inconsistent.
Figure 13: Additional examples from D-TIIL on real-world news image-text pairs. The detected inconsistent text is highlighted as red.
17

---

Published as a conference paper at ICLR 2024
Additional examples. We include additional examples to cover different scenarios of inconsistencies in Fig. 13 Fig. 13 (a) shows the case that text and images are completely misaligned where the most area of the image supposed to be masked; Fig. 13 (b) shows the case that the objects shared in the image and textual semantic space are well aligned but their attributes (e.g., actions, adjective) are complex semantics inconsistent cases where the semantics inconsistency occurs in the background or the scene.
inconsistent, the masks are supposed to cover the whole objects. Fig. [3 (c) contains
Analysis of the learned representation. D-TIIL has two alignment steps to iteratively align the image/text embeddings and filter out relevant semantic information with diffusion models. The well-aligned representations make it easier to identify and localize the semantic inconsistencies. Specifically, the learned representations are two parts, including Eatn with aligned semantic space with the input image I, and En with aligned semantic space with the input text embedding Fg. To show the effectiveness of our two alignment steps in learning the representation, we have provided the comparison of averaged cosine similarity scores on subset of our dataset. We observed that the similarity between Eain and has increased by 9.2% compared to the similarity between Eg and after the first step alignment. The similarity between Edut and Teds is increased by 2.4% compared to the similarity between Eg and Le by the second step alignment. Note that 2.4% is not subtle since only comes from the exclusion of distracting semanties from text embeddings.
18


Reviews:
[{"review_id": "5QZYQUEu7b", "reviewer": "Reviewer_qTYR", "summary": "This paper studies how to detect image-text inconsistency with diffusion models. More specifically, the author designed a pipeline that iteratively use diffusion models to edit the text and images in the image-text pairs to gradually optimize a mask that can point out where the inconsistency come from. This task is interesting and meaningful for misinformation detection, as it provides interpretable prediction results. To evaluate the proposed method, the authors collected a dataset containing image-text pairs and their inconsistency masks. Experiments shows that the proposed method outperforms baselines and gives explanable prediction on the inconsistency.", "questions": "How does the annotation and the model handles predicates?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The task studied in this paper is meaningful.\n\n2. The dataset that they collected is contributive to the community.\n\n3. The method is novel.", "weaknesses": "1. The writing is not very good. I read the methodology part several hours to understand their pipeline.\n\n2. The idea is well justified for the inconsistency of object alignment. But what if the predicate is not aligned, i.e. the person is correct but the action is not?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGAsoFzNFk", "reviewer": "Reviewer_LEKC", "summary": "This paper presents D-TIIL for identifying and localizing inconsistencies between text and images. \nA new dataset, TIIL, containing 14K consistent and inconsistent text-image pairs, is introduced for evaluating the method. The D-TIIL outperforms existing approaches in terms of Accuracy scores and demonstrates more explainable results. In a nutshell, the paper offers a scalable and evidence-based approach to identify and localize the text-image inconsistency. However, it also acknowledges the potential misuse of the method for creating deceptive text-image pairs and suggests improving the algorithm and restricting access.", "questions": "1. Regarding Step 3 in Section 3 METHOD, the proposed E_{dnt} and descriptions like “include extra implicit information from the images and excludes additional implicit information that only appears in the text” raise doubts about the effectiveness of the process of the “text denoising”. Such “text denoising” seems to be too idealistic. In Section 5.4, for example, there is the failure case of the word \"office\". This leads to the bold suspicion that the D-TIIL method is only valid for simple objects, but not for backgrounds or objects that contain more complex semantics.\n\n2. Also, the high dependency on the diffusion model affects the generalizability of the method. If text and image are not well aligned on the latent space, the validity of the method will be more affected. Semantic entanglement can also exist.\n\n3. Regarding Step 4 in Section 3 METHOD, the descriptions like “We then compute the cosine similarity score between this image embedding and the input text embedding” are confusing to the readers.\n\n4. In the Data Generation part of Section 4 TILL DATASET, T_{m} is unknown where it comes from, is it manually designed?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Originality: The paper introduces a novel method, D-TIIL, that exposes text-image inconsistency with the location of inconsistent image regions and words. Also, the new TIIL dataset is the first dataset with pixel-level and word-level inconsistency features that provide fine-grained and reliable inconsistency.\n\n2. Quality: The D-TIIL and TIIL dataset generation are thoroughly described. The paper also provides a comprehensive comparison of the proposed method with existing approaches.\n\n3. Clarity: The paper is well-structured and clearly written. The method is explained in detail and the experiment results are presented in an understandable manner.\n\n4. Significance: The D-TIIL method improves the accuracy of inconsistency detection and provides more explainable results. The introduction of the diffusion model makes it possible to align text and images in a latent and joint representation space to discount irrelevant information and incorporate broader knowledge.", "weaknesses": "1. The paper acknowledges that the D-TIIL may struggle with inconsistencies with respect to specific external knowledge, and this could reduce the effectiveness of the method in real-world application.\n\n2. The D-TIIL method relies heavily on the text-to-image diffusion models and benefits a lot from the semantic space that is already well aligned. This dependence could limit the generalizability of the proposed method.\n\n3. There are some confusing details in the method description section.\n\n4. In the comparison of methods, the reasons why D-TIIL is superior are not discussed and analyzed in detail, and the potential solutions for the failure cases are not provided.\n\n5. More specific discussions and measures could be included to prevent potential abuse rather than simply restricting access.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRny6S4ecC", "reviewer": "Reviewer_4tYr", "summary": "The authors propose D-TIIL (Diffusion-based Text-Image Inconsistency Localization), a system for automatically identifying and explaining text-image inconsistencies. D-TILL uses text-image diffusion models to locate semantic inconsistencies in text-image pairs. Diffusion models trained on large datasets filter out irrelevant information and incorporate background knowledge to identify inconsistencies. In addition, D-TIIL uses text embeddings and modified image regions to visualize these inconsistencies.\nTo evaluate the effectiveness of D-TIIL, the authors also introduce a new dataset (TIIL) with 14K consistent and inconsistent text-image pairs.", "questions": "•\tHow does the approach perform on completely wrong image descriptions?\no\tIs the whole image masked?\n•\tIs the model sensitive to the image part generated by DALE and not to the parts which do not correspondent to the text?\no\tIs there an experiment that can proof that?\no\tMaybe regenerate the image for the dataset also with the right semantic class?\n•\tIs there another dataset where the method could be compared also to other baselines?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "•\tThe paper is well written and well structured\n•\tThe problem and the related work are well introduced\n•\tThe framework is explained in detail\n•\tThe idea to build consistency scores between stable diffusion and the original image is interesting.", "weaknesses": "•\tThe general theoretical idea behind the approach lacks clearity\n•\tThe real-world application is not very clear, e.g. wrong labels have a different type of mislabeling than just objects that are swapped\n•\tSensitivity to threshold highly influences M and the consistency score\n\nWith D-TIIL, the authors have presented an interesting method for using diffusion models to evaluate the consistency of image-text pairs.\nHowever, the utility of the method is not fully evaluated in detail. Deeper insights into why this approach works are lacking. In addition, it would be nice to see how the approach works on other datasets where the labeling is just mixed up or misleading.\nIn addition, I would recommend for ICLR to investigate the method in more detail in terms of learned representations.\n\nThe paper is well written and has some interesting ideas, e.g. the usage of diffusion models for detecting image-text inconsistency. The method and the dataset, both are valuable. However, to be accepted in ICLR I would expect more and deeper investigations about the method and the dataset. What is learned, what are short comings?\nThere are some doubts, such that the model could be sensitive to the DALE generated part instead being sensitive to the text-image inconsistency. Experiments are missing that evaluate the underlying behavior. Moreover, a second evaluation on another dataset with more established baselines would be preferable to proof some of the assumptions, advantages and shortcomings of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wVFaJBiRpf", "reviewer": "Reviewer_D7Cn", "summary": "This paper develops a new method, D-TIIL, to expose text-image inconsistency with the location of inconsistent image regions and words, which is quite commonly happening in T2I generation diffusion models. To achieve this, they introduce a new dataset, TIIL, for evaluating text-image inconsistency localization with pixel-level and word-level inconsistency annotations.", "questions": "As mentioned in the above weakness, I would appreciate seeing the proposed method applied more extensively in evaluation. The inclusion of evaluation metrics beyond CLIP scores could enhance the robustness and confidence of this paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The dataset's contribution is commendable. Existing datasets lack the capacity to furnish evidence regarding inconsistencies occurring at both the image region and word levels, which is essential for evaluating D-TIIL (Diffusion-Based Text-to-Image Inconsistency Localization).\n\n2. The problem addressed in this research is of significant importance. Previous methods have primarily focused on determining the presence of inconsistencies, whereas this paper introduces a novel approach to pinpointing the specific locations where these inconsistencies occur.", "weaknesses": "1. It would be valuable to explore whether this method could be extended to evaluate other text-to-image (T2I) augmentation techniques (i.e., [1-3]). Given the abundance of research on generating images based on textual prompts, applying this method for evaluation purposes could have a broader impact and contribute significantly to the field.\n\n2. Are there alternative evaluation metrics to assess the correspondence between text and images? Based on my experience with CLIP scores, it may not consistently capture performance accurately in various scenarios.\n\n[1] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models. SIGGRAPH 2023\n\n[2] Improving Sample Quality of Diffusion Models Using Self-Attention Guidance. ICCV 2023\n\n[3] Expressive Text-to-Image Generation with Rich Text. ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)