reviews: 
Comments: This work considers the problem of bias mitigation on vision-language models (VLMs). The authors introduced Counterfactual vision-language debiasing (CVLD), a technique that can be summarized in two main contributions: 1- a data generation pipeline based on off-the-shelf generative models to create counterfactual augmentations from real data; 2- a bias mitigation strategy based on fine-tuning a VLM using the generate counterfactuals. The authors evaluate the proposed approach empirically on 3 tasks: image classification, image-text retrieval, and image captioning. Experiments are carried out using a model from the BLIP family and considering both fairness and task performance metrics. Overall, results show that the proposed yields the best trade-off between improving fairness (i.e. mitigating biases) and attaining good performance in the task.
Strengths: - The paper tackles a critical problem and very relevant open research question: how to mitigate bias on foundation models;

- The manuscript is overall well-written and most sections are easy to follow; 

- The counterfactual augmentation approach is grounded in formal definitions from the counterfactual fairness literature;

- The experimental evaluation is extensive in the number of considered tasks.
Weaknesses: - One of the central claims of the work is that the proposed approach is a unified way to mitigate bias in VLMs across multiple tasks, as per the following evidence: 
  - In the title (Debias your VLM with Counterfactuals: A **Unified** Approach, bold text by myself). 
  - Also throughout text: e.g. In Section 2 "[...] we focus on a task-agnostic fairness framework for VLMs, unifying the study of bias across different tasks and domains." ). 

   Claiming that the proposed approach is unified and task-agnostic seemed reasonable until Section 4. However, after reading through the details of how fine-tuning with synthetic data should be carried out for the three considered tasks, it seems to me that CVLD practical instantiation takes a very different format from task to task, rendering it a specific and not-unified framework for debiasing.



- One of the key parts of the introduced approach is the counterfactual data generation. However, the authors did not mention at any part of the manuscript details of the evaluation of the data generation pipeline. Moreover, it is not clear how the quality of generated counterfactuals could affect the performance of CVLD. Moreover, other fine-grained aspects such as *how much synthetic data is needed* and how the number of synthetic samples used at training time affects performance were not addressed in the manuscript, making it difficult to judge to what extent this framework would generalize to other scenarios where it might be difficult to generate high quality counterfactuals.  

- Some parts of the text do not seem to reflect the actual insights that can be extracted from the results. For example, in the introduction, the authors mentioned that CVLD "demonstrates striking effectiveness for the most studied problems in the bias literature" while it is not clear from the results that the CVLD demonstrates **striking** effectiveness, neither there are references to support the statement that the considered problems in this work are the most studied ones in the bias literature.

- The experimental results are a bit confusing and hard to parse. The employed metrics in the evaluation are not standard in the literature and it is not clear whether an improvement is observed when a metric increases or decreases. Moreover, it is not clear why some methods were grouped in different parts of the tables (e.g. in Table 1 it is not clear why both CVLDs are in different sections from BLIP-PT and the ResNet-50, aren't they directly comparable?). On a similar note, it is not clear how the different bolded numbers in the tables represent and how they should be compared against each other.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good