See or Guess: Counterfactually Regularized Image Captioning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 08:32:53

Abstract: Image captioning, which generates natural language descriptions of the visual information in an image, is a crucial task in vision-language research. Previous models have typically addressed this task by aligning the generative capabilities of machines with human intelligence through statistical fitting of existing datasets. While these models demonstrate proficiency in describing the content of normal images, they may struggle to accurately describe those where certain parts of the image are obscured or edited. Conversely, humans effortlessly excel at it in this case. The weaknesses these models exhibit, including hallucinations and limited interpretability, often result in performance declines when applied to scenarios involving shifted association patterns. In this paper, we present a generic image captioning framework that leverages causal inference to make existing models more capable of interventional tasks, and counterfactually explainable. Specifically, our approach consists of two variants that utilize either total effect or natural direct effect. We incorporate these concepts into the training process, enabling the models to handle counterfactual scenarios and thereby become more generalizable. Extensive experiments on various datasets have demonstrated that our method can effectively reduce hallucinations and increase the model's faithfulness to the images, with a high portability for both small-scale and large-scale image-to-text models.

Main paper: 
 introduction: 
 As a fundamental task in vision-language understanding research, image captioning requires models to mimic the human ability to compress huge amounts of visual information into descriptive language [3, 19, 36] . A large amount of image-to-text methods [31] have been developed, among which recent large multimodal models [12, 18, 43] perform surprisingly well in describing an image in details. Despite their good performance in real scenarios, their capabilities still differ from those of humans in interventional scenarios. For example, in Figure 1 , the BLIP model [13] can generate a sentence that accurately describes the factual image at left. However, when the bicycle is masked or changed to a tree as shown in the counterfactual images, it generates incorrect descriptions such as "A man sitting on a bench in front of a river. " Such errors reveal Factual Image Counterfactual Image (masked)A man sitting on a bench in front of a river.A picture of a bridge over a body of water. that the model might not have precisely understood the image. Instead, it may make guesses based on the common association patterns in the datasets. For example, the frequent co-occurrence of a river and a man in the dataset may lead the model to form shortcut connections and wrongly generate "man" for most images with a river.The above analysis suggests that while current models may exhibit impressive performance, it does not necessarily imply their ability to accurately comprehend the contents of an image and generate appropriate descriptions, a capability inherently possessed by humans. Such weaknesses may result in hallucinations and hinder the interpretability of models since people cannot exactly tell which parts of the image correspond to the generated words in the text. Furthermore, when these learned models are applied to other scenarios with shifted association patterns, their performance may suffer a substantial deterioration.To overcome the above shortcomings, we design a novel framework that integrates causal inference into any image captioning model to mitigate the shortcut correlations. Specifically, we leverage counterfactual concepts to enhance the correspondence between visual and textual characteristics. We hope that when certain regions of an image are removed, the generated text should not include the descriptions of those regions. While this idea is intuitive, it is challenging to implement for the following reasons. First, existing counterfactual models primarily focus on classification tasks [1, 8, 39] . However, we handle a generation task that necessitates the consideration of sequential impacts between words. Moreover, in our multi-modal scenario, the influence of the image on the word can be attributed to two paths: (1) a direct influence from the image to the word, and (2) an indirect influence, where the image first impacts preceding words, which subsequently influence the current word. It presents a nontrivial challenge to distinguish between these two paths and enhance the first one to minimize hallucination while preserving linguistic fidelity.To address the above challenges, in this paper, we first formalize the image-to-caption generation task as a sequential causal graph, where each word in the generated text is determined by both its previous words and the image. Following this causal graph, we leverage the causal concepts of total effect (TE) and natural direct effect (NDE) to discriminate different reasons for the word generation process. Then we can intervene in the cause, and enhance the correspondences between the image and words while controlling the other influential factors. Finally, we propose a counterfactually regularized image captioning framework. The main contributions of our paper are as follows:• We propose a generic framework to counterfactually regularize image captioning models and thus make them more human-like, explainable, and robust.• We propose two causal methods based on total effect and natural direct effect to enhance the correspondence between the visual and textual characteristics.• We extensively experimented on various models and datasets to demonstrate the high generality and interpretability of our methods, which can effectively reduce object hallucinations and enhance the faithfulness of the model to the images. 
related work 2.1 image captioning: 
 Image captioning, a crucial aspect of image-to-text generation [31] , has evolved from convolution neural network (CNN)-based encoders and recurrent neural network (RNN)-based decoders [34, 37] to Transformer architectures [5, 11] , and further into the era of vision language pretraining (VLP) models [13, 14, 42] . Recent advancements in Large Vision Language Models [12, 18, 43] have sparked renewed interest in the field. In addition, some methodologies explore the integration of multimodal representation models like CLIP [27] to furnish visual support for language models [19] .Our proposed approach has a model-agnostic nature and flexibility. Considering the notable performance of VLP models, we opt to validate the effectiveness across various architectures (decoder-only and encoder-decoder) and model scales by employing ClipCap [19] , BLIP [13] , and BLIP2 [12] as backbone models.Alleviating hallucination of image captioning models does not solely hinge on improved image perception capability but also on factors like over-reliance on language priors or biases during sequence generation [25, 28] , potentially leading to guesswork that is not faithful to the image. Researchers [28] thus propose utilizing the CHAIR metric to quantify hallucination occurrence. Some efforts have been made to reduce model reliance on common or biased co-occurrences by adjusting object label co-occurrence statistics [4] .Other methods maintain semantic consistency to reduce object hallucinations by learning consensus representations through aligning scene and language graphs [40] , or by aligning textual tokens and visual objects using masked language modeling [6] . However, these methods may blur semantic and visual alignments and overly rely on dataset co-occurrence patterns, harming interpretability and performance in real scenarios. Our approach considers causality, aiming to establish the correct vision-to-language guiding relationship during the generation process.Causal inference seeks to unravel the causal relationships and underlying mechanisms driving observed outcomes [2, 7, 23] . Moreover, counterfactual causal inference offers a framework to enhance [1, 32] and explain [8, 9] models in counterfactual scenarios. However, the majority of these counterfactual-related works are tailored for classification tasks, such as image classification [1, 8, 39] , representations learning [32, 41] , or visual question answering [10, 15, 20] , rather than for generation tasks. Classification tasks exhibit a deterministic correspondence between input and output, whereas, in the generation process, the counterfactual image and preceding generated tokens collectively influence the subsequent token generation, creating an effect propagation. Some researchers [35] have performed Maximum Likelihood Estimation (MLE) on interventional distribution to reduce spurious correlations learned by models due to observed confounding factors. However, the applicability of their framework is limited to the strong ignorability assumption and lacks causal analysis in multi-modal scenarios. Capturing this causal correspondence [22] is challenging, especially in multimodal scenarios, and has thus received little attention in prior literature. In this paper, we endeavor to leverage counterfactual causal inference to tackle this challenge and gain deeper insights into a model's generation behavior. 
preliminaries: a causal look at image captioning: 
 This section presents the fundamental concepts and notations of causal inference [7, 22] and how we apply it in image captioning. In the following, capital letters, i.e. , cause 𝑋 , Mediator 𝑀 and Effect 𝑌 , represent random variables. The values or subscripts of these random variables indicate their observed values. As for image captioning, a model is used to process an input image 𝐼 and produce a corresponding textual description, i.e. , a sequence 𝑆 = (𝑠 1 , 𝑠 2 , . . . , 𝑠 𝐿 ) , where 𝑠 𝑖 is a token in the sequence and 𝐿 is the sequence length. The sequence of the preceding tokens of 𝑠 𝑗 is denoted as 𝑆 < 𝑗 = (𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑗 -1 ). We will later present how to treat these variables from a causal perspective.A causal graph describes the causal relations between different random variables in a graph manner [24] . In a causal graph G = {V, E}, a node 𝒗 ∈ V represents a variable and a directed edge 𝒆 ∈ E represents a causal relationship between variables. The direct effect means that there is an edge between two variables, e.g. , in Figure 2 two variables are not directly linked, but are connected via some mediator variables, e.g. , 𝑋 has a indirect effect on 𝑌 if 𝑋 → 𝑀 → 𝑌 .Considering the process of auto-regressive generation, at each step, the current token 𝑠 𝑗 is determined by all the preceding tokens 𝑆 < 𝑗 , and the visual information of the input image 𝐼 as well. As shown in Figure 2 (a), at step 𝑗, 𝑆 < 𝑗 is influenced by 𝐼 , and 𝑠 𝑗 is jointly determined by 𝐼 and 𝑆 < 𝑗 . We use 𝑌 𝐼,𝑀 𝐼 = 𝑌 (𝑋 = 𝐼, 𝑀 = 𝑀 𝐼 ) to denote the probability of token 𝑠 𝑗 when the cause 𝑋 is set to 𝐼 and the mediator 𝑀 is set to 𝑀 𝐼 = 𝑆 < 𝑗 .In causal inference, counterfactual causal effects compare hypothetical outcomes under factual and counterfactual treatments [2, 23] .As shown in Figure 2 Mathematically, the total effect can be expressed asEQUATIONTE 𝐼,𝐼 * measures the effect of all factors (i.e. , direct and indirect effects) resulting from changing image 𝐼 to 𝐼 * . Further, intervening both 𝑋 and 𝑀 allows the total effect to be decomposed into two components, namely the natural direct effect (NDE) and the total indirect effect (TIE). Unlike TIE that focuses on the effect brought by changes in the mediator 𝑀, NDE is the effect of 𝑋 on 𝑌 that results solely from changes in 𝑋 , without any influence from 𝑀, which can be denoted asNDE 𝐼,𝐼 * = 𝑌 𝐼,𝑀 𝐼 * -𝑌 𝐼 * ,𝑀 𝐼 * .(EQUATIONThe first term 𝑌 𝐼,𝑀 𝐼 * corresponds to Formula 2 describes the variation of 𝑌 when 𝑋 is changed from 𝐼 to its counterfactual 𝐼 * while 𝑀 is held constant at 𝑀 (𝑋 = 𝐼 * ). This paper explores how to use TE or NDE to reduce object hallucination in image captioning and improve interoperability. 
counterfactual regularization: 
 In this section, we introduce how to construct counterfactual data, propose our framework, and design two counterfactual regularization losses that can be universally applied to existing image captioning models.Collecting or generating counterfactual images for the factual ones is difficult. However, by adding a mask, it is easy to achieve minimal changes to the original image when constructing the counterfactual one, which can be regarded as an approximation of the idealized counterfactual image. Specifically, we construct counterfactual images by leveraging datasets that have labeled bounding boxes for corresponding phrases in the image captions. As shown in Figure 3 (a), we first select the entity to intervene ( S), e.g. , "black poodle". Then we identify its corresponding region to intervene (𝑟 ) in the image based on the labeled bounding boxes in the datasets. A black mask is used to replace the region 𝑟 to create a counterfactual image 𝐼 * . If the entity to intervene corresponds to more than one bounding box (e.g. , when S is "a group of people"), all related regions will be masked. Next, we employ the initial image captioning model to generate a counterfactual caption 𝑆 * . The counterfactual captions are used to model 𝑆 * < 𝑗 and 𝑠 * 𝑗 in the causal graphs (Figure 2 ), i.e. , modeling the generated words for the counterfactual image 𝐼 * . Please note that the goal of 𝑆 * is to facilitate the estimation of causal effects, instead of being used as a ground-truth caption for counterfactual images. Thus, it is much easier to obtain compared with ground-truth labels for counterfactual images. Accordingly, we have (𝐼, 𝑆, S, 𝐼 * , 𝑆 * ) prepared for dataset D.We propose a framework by incorporating negative log-likelihood (NLL) loss L NLL with TE or NDE regularization loss, i.e. , L TE or L NDE , which will be described later. Formally, the vanilla negative log-likelihood (NLL) loss is as follows:EQUATIONwhere 𝑓 𝜃 (•) refers to the model that takes the image and preceding text sequence 𝑆 <𝑖 as input and outputs a probability distribution on the vocabulary to generate the next token 𝑠 𝑖 , with parameters 𝜃 . We add the counterfactual regularization loss to allow the model to learn together with the NLL loss. A hyperparameter 𝛼 determines the weight of losses, ensuring balanced optimization. The final loss is denoted as:EQUATIONThe whole optimization includes two stages: (1) training the model with vanilla NLL loss (Formula 3); ( 2) training the model with either L 1 or L 2 (Formula 4) using the constructed counterfactual images and their corresponding generated counterfactual captions. When the region corresponding to "black poodle" is masked in the image (Figure 3 ), we hope the model will significantly lower the generation probabilities of the words "black poodle" to reduce hallucination. To achieve this goal from a causal perspective, we maximize the total effect of changing 𝐼 to 𝐼 * on the generation of "black poodle" (i.e. , S), which is given in Formula 1. Maximizing this total effect can be fulfilled by minimizing the following total effect (TE) loss:EQUATIONEQUATIONwhere the first part corresponds to 𝑌 𝐼,𝑀 𝐼 in Formula 1 and calculates the likelihood of generating S (e.g. , "black poodle") given the factual image 𝐼 and previously generated words, and the second part estimates 𝑌 𝐼 * ,𝑀 𝐼 * in Formula 1 with the likelihood of generating S at any position given the counterfactual image 𝐼 * and preceding tokens 𝑆 * <𝑖 that are generated from 𝐼 * . Here, s𝑗 denotes the 𝑗-th token of the entity to intervene S, of which the length is 𝐿 S . 𝑝 represents the index of the first position where the entity to intervene S appeared in the ground truth or factual caption (e.g. , if "black" and "poodle" are the 9th and 10th words, then 𝑝 = 9). 𝐿 𝑆 * is the length of counterfactual caption 𝑆 * . The specific occurrence position of the entity to intervene S is explicit in the ground truth, while it may not necessarily appear in the counterfactual caption. Therefore, we need to estimate the probability of their occurrence using the average value.In the example shown in Figure 3 , the word "black poodle" is the entity to intervene ( S). We estimate the first term by the probability of generating each token in "black poodle" at the position it appeared in the factual caption, i.e. using the preceding tokens "A man in a yellow shirt training a". The second term is calculated by the average probabilities of generating each token in "black poodle" at any position in the counterfactual caption "A man in a yellow shirt is stretching in a field. ", where the preceding tokens are those before each step.To improve the visual perception ability of the model, another option is to maximize the natural direct effect (NDE) rather than the total effect (TE). The natural direct effect is to measure the direct effect resulting from changes in the image. As shown in Formula 2, the first part is to calculate the likelihood of generating each token in the entity to intervene S at any position, from the image 𝐼 and the preceding tokens 𝑆 * <𝑖 that have been generated from 𝐼 * . Whereas, the second part is the likelihood of generating S at any position from the counterfactual image 𝐼 * and preceding tokens 𝑆 * <𝑖 that are generated from 𝐼 * , which is the same as the second part of TE. Formally, we calculate NDE loss as:EQUATIONThe first part here looks simpler than that in TE loss because both the first part and the second part in NDE loss average the probabilities over any position in 𝑆 * whose length is 𝐿 𝑆 * . Figure 3 (b) presents an example. The first term is estimated by the probability of generating each token in "black poodle" at any position in the counterfactual caption "A man in a yellow shirt is stretching in a field.", but with the factual image 𝐼 as input. The second term is again the average probabilities of generating each token in "black poodle" at any position in the counterfactual caption with the counterfactual image 𝐼 * as input. By maximizing the NDE effect, the direct influence of the image is enhanced, thereby the model is more inclined to see the image and generate the correct next token, rather than to guess it.  
experiments: 
 In this section, we conduct extensive experiments to evaluate the capability of our model for alleviating object hallucination, reducing biases in training data, and interpreting the correspondence between captions and image regions.To evaluate the effectiveness of our model, we construct counterfactual images and captions as mentioned in Section 4.1. We choose Flickr30k Entities [26] and MSCOCO [17] as our datasets, which have high-quality image annotations for constructing masked counterfactual images. Flickr30k Entities (Flickr) is built upon the existing Flickr30k dataset [38] that contains 31,783 images. The dataset provides 244k coreference chains and 276k manually annotated bounding boxes within the images. We use the original split of this dataset. Entities that occur more than once within a caption are removed to avoid confusion. After pre-processing, the final dataset consists of 29k/1k/1k samples for training/validation/test, respectively. MSCOCO (COCO) consists of more than 328k images with annotated objects, phrases, and relationships. We adopt the Karpathy split of the MSCOCO dataset and coreference relationships in the annotations are utilized to establish correspondences between the image regions and phrases (entities) to intervene in the captions. Both datasets are composed of diverse phrase categories, where the Flickr dataset covers over 1,000 categories, while the COCO dataset is more concentrated on 80 categories.Our proposed counterfactual regularization losses are model-agnostic and can be applied to various models. We conduct experiments on three backbones: Clip-Cap [19] , BLIP [13] and BLIP2 [12] , which respectively serve as representative models for decoder-only, encoder-decoder, and multimodal large language model architectures. In addition to the above three image captioning baselines, we compare our methods with another two methods that aim to alleviate the object hallucination in image captioning: (1) ObjL [4] utilizes object labels as training augmentation to diminish models' object bias on hallucination;(2) ObjMLM [6] conducts a whole object mask to mitigate object hallucination in masked language modeling. More implementation details are in supplementary pages.Compared to baselines, our methods are expected to significantly reduce object hallucination on counterfactual test sets while maintaining the generation ability on factual test sets (it is not trivial due to different distributions between training and test sets). We employ both automatic and human evaluation in our experiments for convincing conclusions. Automatic Evaluation: We evaluate hallucination by using:• CHAIR 𝑠 [28] : It measures whether models generate a masked phrase, i.e. , phrases whose corresponding regions have been masked in the counterfactual image:EQUATIONwhere a lower CHAIR 𝑠 score indicates a reduced presence of hallucination or increased faithfulness.• Ranking-based Metrics: We generate a set of five candidate captions with the highest probability of being generated for a given counterfactual image, among which the ones without the masked phrase are regarded as positive while those with the masked phrase are as negative. Precision @5 and nDCG @5 are employed1 to assess the object hallucination in fine-grained. We adopt BLEU [21] , ROUGE-L [16] , and CIDEr [33] to measure the quality of generated captions on factual image test sets. We do not evaluate the generation quality of counterfactual images using automatic evaluation due to the lack of ground-truth captions. To compensate for this, the quality of captions generated for counterfactual images is analyzed by using human evaluation. Human Evaluation: To verify whether the automatic measurements are consistent with human experiences, we further conduct a user study. First, we randomly sample 50 factual images from the Flickr and 50 from the COCO dataset. We then create counterfactual images for the 100 factual images and collect top-generated captions from all methods for both factual and counterfactual images. We conduct human evaluations on the 100 factual images and 100 counterfactual images. For each image, we shuffle the generated captions and make the methods anonymous when presented with an image to ensure a fair comparison. Three human assessors majored in English with the age range from 23 to 25, are hired to rate the captions on a 3-level Likert scale from 0 to 2 in two aspects:• Faithfulness measures the degree to which a caption accurately represents the content of the image;• Overall means the overall quality of a caption. Finally, we calculate the Fleiss' Kappa among their assessments which results in 0.43, meaning a moderate level of agreement. We use their average values as the results.We first compare our proposed models with all baselines on counterfactual test sets in terms of both hallucination and overall generation quality. The results are shown in Table 1 , where all methods with the same backbone are grouped for clarity.Automatic evaluation. In terms of the automatic metrics of measuring object hallucination in Table 1 (CH. 𝑠 , P @5 , nDCG @5 ), our proposed counterfactually regularized methods consistently exhibit superior performance over all baselines on both datasets, demonstrating their effectiveness in mitigating object hallucination. Our NDE regularization performs better than the TE ones. This indicates that maximizing the direct effect of image content on the generated tokens helps build a more precise alignment between visual regions and their corresponding entity phrases. Baselines ObjL and ObjMLM do not always alleviate hallucination effectively, e.g. , they exhibit more hallucinations on the ClipCap backbone on the two datasets. In comparison, our methods that regularize the causal effect consistently reduce hallucination in terms of different backbones, datasets, and measures. This demonstrates the effectiveness of adopting a causal perspective when handling hallucinations. Further experiments confirm this will not be affected by different decoding strategies (please see Section A in supplementary pages).Human evaluation. Human evaluation results in Table 1 show that our methods perform the best regarding both reducing hallucination (Faith.) and overall generation quality (Overall). Moreover, NDE wins more times than TE, and these two methods that we propose are consistently better than baselines ObjL and ObjMLM. Overall, the human evaluation results are consistent with the automatic evaluation results in terms of hallucination and additionally reveal the good generation quality of our methods.We compare all methods on factual test images to investigate 1) whether our regularization methods compromise any generation capability and 2) whether our method can reduce hallucination on factual images. As shown in Table 2 , our proposed methods achieve comparable or superior performance to all the baselines on both datasets in terms of automatic metrics for evaluating generation quality (BLEU, ROUGE-L, CIDEr). Human evaluation results also show that our methods can consistently outperform all baselines in reducing hallucination (Faithfulness) and increasing overall generation quality (Overall). This indicates that our methods can significantly reduce hallucinations on counterfactual images without scarification in generation performance on factual images.It would be interesting to investigate whether the proposed methods perform better when the test data has a biased distribution of some entities from training data. Therefore, we construct a biased dataset from Flickr30k Entities. First, we do statistics of all the captions and find that 9,893 captions contain male-related words, such as "man/men" and "boy/boys", and 5,963 captions contain femalerelated words. We then reconstruct a training set consisting of 8,942 male, 1,962 female, and 14,838 other captions, where the ratio of male to female is about 5:1. We reverse the ratio to reconstruct the test set, which consists of 481 males, 2,034 females, and 500 other captions. The validation set is constructed with a similar size and recipe as the test set. Finally, we train a BLIP model and evaluate its performance on the biased test set. When examining entities related to gender only, the error rates are presented in Table 3 . The results indicate that BLIP+NDE model outperforms BLIP in terms of lower error rates for both factual and counterfactual images. The results imply that models with our proposed methods are more robust in handling biased datasets.We present some examples of the generated captions on counterfactual images in Figure 4 (a), and more in supplementary pages. Overall, our methods perform better in understanding counterfactual images, avoiding generating captions containing masked information. Instead, they describe what are indeed presented in the images, such as "a wooden cross" and "a couple of black screens". Conversely, the baseline model without counterfactual regularization often guesses incorrectly. We also present some examples of generated captions for counterfactual images in the biased dataset in Figure 4(b) . The baseline model often incorrectly guesses a "man" or "boy" behind the mask, whereas our models describe other objects that are present in the image, such as "a child's bicycle" and "a horse".We further utilize a Latent Diffusion Model [29] to inpaint the masked region with a counterfactual object. As shown in Figure 4 (c), an intriguing observation is that the baseline model occasionally hallucinates "a person" in the inpainted image, despite the absence of any human presence in the image. This may be caused by the shortcut connections it learned from the training data, where our methods can robustly avoid this and correctly describe "a feather" or "a tree house" that can be seen in the inpainted images.In this experiment, we compare different image captioning models in terms of interpretability. The basic idea is that an interpretable model should generate a noun phrase by using its corresponding region. For example, when generating the phrase "little girl", the region that contains a little girl should contribute the most to the model generation compared with other regions. The contribution of a region is measured by using an efficient and effective explanation method CXPlain [30] . More specially, for each noun phrase, we first identify its corresponding region (positive sample) and then randomly generate four incorrect regions with the same size (negative samples). Next, we rank the five regions by using CXPlain [30] , which assigns a higher contribution score to a region if removing this region results in a larger change in the model's loss. A model is considered interpretable if the positive sample has the highest contribution. Figure 5 shows an example, where the correct region (the region with a little girl) contributes the most in generating the phrase "little girl" for the Blip+NDE model, demonstrating the interpretability of the model. In comparison, for other models (Blip and Blip+TE), the correct region does not have the highest contribution to generating "little girl". To show the average results over our test set, we use accuracy as the measure. Accuracy refers to the percentage of cases in which the positive sample has the highest contribution. Figure 6 shows that the TE method consistently performs better than the backbone model without regularization. The NDE method outperforms the TE method across backbone models on Flickr by a significant margin, while they perform comparably on MSCOCO. This suggests that our proposed counterfactual regularization is effective in enhancing interpretability and our proposed NDE method is the most effective. 
conclusion: 
 This paper proposes using counterfactual causal effects to model the relationship between vision and language. We employ two counterfactual regularization methods based on the concepts of total effect (TE) and natural direct effect (NDE) to improve image captioning models. Experimental results consistently show the superiority of our methods over baselines in terms of alleviating hallucination across different backbones and datasets. The NDE method performs the best in generating faithful captions for counterfactual images and accurately interpreting the most relevant image regions corresponding to a phrase in a caption. In the future, we plan to integrate the counterfactual regularization methods into more complicated multimodal generation scenarios with both image and text as input, such as visual question answering and multimodal dialogue.  
 
 Appendix: https://github.com/microsoft/rankerEval 


Decision: This work presents a framework using causal inference to enhance image captioning models for interventional tasks and counterfactual explanations, reducing hallucinations and increasing faithfulness to images across various datasets. Existing models often struggle with obscured or edited images, but the proposed approach addresses these challenges.

Three reviewers vote for acceptance, while Reviewer GZrh still has concerns. All reviewers acknowledge that the image captioning framework integrating causal inference can address limitations in existing models. Reviewer GZrh maintains concerns about the scalability of the proposed method, the evaluation metric, and its generalization ability on unseen and novel test sets. I agree with these concerns but, considering the contribution of using causal inference and the good empirical evaluation, I recommend accepting this submission. I suggest the authors provide a detailed discussion on these potential limitations or future directions. Please ensure the rebuttal (especially the newly included analysis) is well incorporated into the revised version.

Reviews: review: paper_summary: paper_summary: This paper proposes an image captioning framework based on counterfactual scenarios. The authors introduce causal graph into the model training, making the model more robust and interpretable, thereby mitigating the hallucination problem to some extent.
strength_weakness: strength_weakness: 1. Authors propose a generic counterfactural regularize image captioning framework, which mark generated descriptions more human-like, explainable and robust.

2. Authors introduce causal graph into the training of image captioning, enhancing the better alignment between images and text.

3. The writing and examples in the paper make it easy to read.
limitations: limitations: 1. In the process of generating counterfactual captions (Section 4.1), the initial model was not clearly specified. And I want to know how to ensure that hallucinations do not appear in counterfactual captions.

2. compared to other methods, the performance of proposed method is not significant in Flickr30k Entities and MSCOCO. But this is not the most important thing. In Table 3, the authors' method modeled the counterfactual data, but it only achieved about a 4% improvement in performance on the counterfactual images.

3. What are the advantages and disadvantages of TE and NED, respectively?
paper_summary: paper_summary: This paper proposes a novel image captioning framework that integrates causal inference, addressing limitations in existing models. By incorporating causal concepts into training, the models can handle obscured or edited image content more easily. Through two variants, the approach improves generalization across datasets, reduces hallucinations, and enhances faithfulness to images. This innovation signifies a significant advancement in the interpretability and robustness of image captioning systems.
strength_weakness: strength_weakness: 1) By providing counterfactually explainable descriptions, the framework advances the interpretability of image captioning tasks, fostering trust and understanding in AI-generated outputs.
2) By incorporating causal concepts, the proposed framework enhances the models' ability to handle scenarios involving obscured or edited image content, improving their robustness in real-world applications.
3) Sufficient in validation, the two variants presented in the framework enable better generalization across different datasets, making the approach applicable to a wide range of image captioning tasks and scenarios.
limitations: limitations: 1) This paper seems quite similar to ‘Show, Deconfound and Tell: Image Captioning with Causal Inference’, but there is no reference to this paper. Can the authors state the difference between this paper and the proposed method? 
2) I am curious that in table 1, it seems like NDE is working better than TE, but in factual images, the TE and NDE results seem not so as generalized as the counterfactual images. Is there any reason or explanations on this part?
3) For evaluation in biased datasets, are there any results on unbiased datasets, and see how the existing and current proposed methods differ in results? 
4) In Figure 2, so the $Y_I$/effect is an unobserved counterfactual? The figure is not so clear to me.
paper_summary: paper_summary: The paper addresses the challenge of accurately describing images, especially in scenarios where parts of the image are obscured or edited. Current image captioning models struggle with this issue due to problems like hallucinations and limited interpretability. The paper proposes leveraging causal inference in existing image-to-caption models. By incorporating concepts like total effect and natural direct effect into the training process, the models become more generalizable and capable of handling counterfactual scenarios. The main contributions include proposing a generic framework for counterfactually regularizing image captioning models, introducing two causal methods to enhance correspondence between visual and textual characteristics, and demonstrating effectiveness in reducing object hallucinations and improving model faithfulness to images.
strength_weakness: strength_weakness: The paper's strength lies in its innovative approach of incorporating causal inference into image captioning models to enhance interpretability and performance in handling counterfactual scenarios. The use of counterfactual data construction, where minimal changes are made to original images through masking, allows for improved model generalizability and robustness in generating captions for altered or obscured images. The paper's emphasis on addressing issues like object hallucinations and improving model faithfulness to images demonstrates its practical relevance and potential applications in various image captioning tasks. The paper is well-presented and organized, making it easy to read. Additionally, Figure 4 effectively illustrates how the models work.
limitations: limitations: The paper's limitation lies in the challenge of distinguishing between direct and indirect influences of the image on word generation in a multi-modal scenario. This complicates the enhancement of the direct influence to minimize hallucinations while maintaining linguistic fidelity. Additionally, I would suggest that the authors elaborate more on precision and nDCG in the "Experiments" section.
paper_summary: paper_summary: This paper proposes a universal image captioning framework that leverages counterfactual causal inference to enhance the correspondence between visual and textual features, thereby improving the explainability and robustness of existing models when processing counterfactual scenarios. See strengths and limitations.
strength_weakness: strength_weakness: 1. The paper introduces two causal methods based on total and natural direct effects, and integrates them into the training process to enhance the capability of models in processing counterfactual scenes, making them more explainable and robust. 
2. Experimental results demonstrate that this method can effectively reduce illusions and increase the faithfulness of models to images, possessing high transferability and suitability for different sizes of image-to-text models.
3. The writting and the figures are clear and easy to read.
limitations: limitations: 1. Some important references are missing:
[1] Le T, Lal V, Howard P. Coco-counterfactuals: Automatically constructed counterfactual examples for image-text pairs[J]. Advances in Neural Information Processing Systems, 2024, 36.
[2] Yang X, Zhang H, Cai J. Deconfounded image captioning: A causal retrospect[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 45(11): 12996-13010.
[3] Fei Z. Efficient modeling of future context for image captioning[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 5026-5035.
2. It has not been tested on LLM models, which may affect the scalability of the method. Validation on larger models might be necessary.
3. The automatic assessment indicators used in the document, such as CHAIRs and P@5, may not truly reflect the performance of the model on counterfactual image, due to the dataset bias. 
4. It does not consider the impact of distribution differences between test and training data. Such differences may affect the generalization ability of the model and should be taken into account during evaluation. Testing the method on more and different datasets could provide more comprehensive results, e.g. training on COCO and testing on Flickr30k.
5. It uses basic masking techniques to generate counterfactual images, which can only simulate a limited number of counterfactual scenarios. For example, replace some region with counterfactual elements.