Automated Detection of Visual Attribute Reliance with a Self-Reflective Agent
Authors: Christy Li
Josep Lopez Camuñas
Jake Thomas Touchet
Jacob Andreas
Agata Lapedriza
Antonio Torralba
Tamar Rott Shaham
Venue: NeurIPS 2025 poster ()
Keywords: Model robustness; Interpretability; Interpretability Agent; Self-reflection; Interpretability benchmarks; Spurious correlation
Primary area: evaluation
TL;DR: Automated agent with a self-reflection mechanism that detects unintended visual attribute dependencies in vision models

Abstract: When a vision model performs image recognition, which visual attributes drive its predictions? Detecting unintended reliance on specific visual features is critical for ensuring model robustness, preventing overfitting, and avoiding spurious correlations. We introduce an automated framework for detecting such dependencies in trained vision models. At the core of our method is a self-reflective agent that systematically generates and tests hypotheses about visual attributes that a model may rely on. This process is iterative: the agent refines its hypotheses based on experimental outcomes and uses a self-evaluation protocol to assess whether its findings accurately explain model behavior. When inconsistencies arise, the agent self-reflects over its findings and triggers a new cycle of experimentation. We evaluate our approach on a novel benchmark of 130 models designed to exhibit diverse visual attribute dependencies across 18 categories. Our results show that the agent's performance consistently improves with self-reflection, with a significant performance increase over non-reflective baselines. We further demonstrate that the agent identifies real-world visual attribute dependencies in state-of-the-art models, including CLIP's vision encoder and the YOLOv8 object detector.

Automated Detection of Visual Attribute Reliance with a Self-Reflective Agent
Christy Li Josep Lopez Camuñas²
Jake Thomas Touchet Jacob Andreas',
Agata Lapedriza" Antonio Torralba',
Tamar Rott Shaham
MIT CSAIL Universitat Oberta de Catalunya
Louisiana Tech
*Nottheastern University
Abstract
When a vision model performs image recognition, which visual attributes drive its predictions? Detecting unintended reliance on specific visual features is critical for ensuring model robustness, preventing overfitting, and avoiding spurious corre- lations. We introduce an automated framework for detecting such dependencies in trained vision models. At the core of our method is self-reflective agent that systematically generates and tests hypotheses about visual attributes that a model may rely This process is iterative: the agent refines its hypotheses based on experimental outcomes and uses a self-evaluation protocol to assess whether its findings accurately explain model behavior. When inconsistencies arise, the agent self-reflects over its findings and triggers a new cycle of experimentation. We evaluate our approach diverse visual attribute dependencies across 18 categories. Our results show that the agent's performance consistently improves with self-reflection, with significant performance increase over non-reflective baselines. We further demonstrate that the agent identifies real-world visual attribute dependencies in state-of-the- models, including CLIP's vision encoder and the YOLOv8 object detector.
novel benchmark of 130 models designed exhibit
1 Introduction
Computer vision models trained on large-scale datasets have achieved remarkable performance across broad range of recognition tasks, often surpassing human accuracy standard benchmarks [He et al., 2016, Dosovitskiy et al., 2020, Tan and Le, 2019, Russakovsky et al., 2015]. However, strong benchmark results obscure underlying vulnerabilities. In particular, models may achieve high accuracy using prediction strategies that are non-robust or non-generalizable. These include relying on object-level characteristies such as pose or color [Geirhos al., 2018]. contextual cues like background scenery or co-occurring objects [Xiao et al., 2021, Alcorn et al., 2019], and demographic traits of human subjects [Wilson al., 2019, Wang al., 2019, Rosenfeld et al., 2018]. Such visual dependencies may result in overfitting, reduced generalization, and performance disparities in real-world usage [Hendrycks et al., 2021, Recht et al., 2019, Taori al., 2020, Wiles al., 2022].
Existing methods take various approaches to discover visual attributes that drive model predictions. These include saliency-base [Simonyan et al., 2013, Selvaraju et al., 2017, Kindermans et al., 2017]. feature visualizations that map activations to human-interpretable patterns [Olah al., 2017], and concept-based attribution methods that evaluate sensitivity to predefined semantic concepts [Kim et al., 2018, Ghorbani et al., 2019, Mu and Andreas, 2020]. While powerful for visualizing local behaviors. these approaches
methods that highlight input regions associated with prediction
Address correspondence to: ckl@ mit.cdu, Samarott@mit.cd
i Work done while at MIT CSAIL Website: https: httpp:::/chhhrtttt// github. wtt....i/saaaaaeasiiiii
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

What are the visual attr butes that the classher relies on to detect vases?
What are attributes CLIP on recognize the concept teacher?
What are attributes YOLO relles on to detect pedestrians?
Lotta conduct, aone caperimcntss
Latts conduct sene experizents.
Let'a conduct sone experiments...
The model mell liss on the presence of Floowrre inselce the vade over expty decorative vases,
CLIF rolles on authority sianala in conventional claseroen settinge rather than actual pedapogical engagement.
YOLO relics side-view walking in uxben eroading contexta.
Prodi high scoring images
Predicted high scoring images
Predicted high scoring images
0.49
00:41
0:42
0.20
0.45
00.38
Predicted low- images
Predicted low-scoring cooiing images
Predicted low low-scoring images
Figure I: Attribute Reliance Detection. We use a self-reflective agent to produce natural-language descriptions of the visual attributes that model relies on to recognize or detect a given concept. For each target concept (e.g., vase, teacher, or pedestrians). the agent first conducts hypothesis testing to reach a candidate description and then validates the description's 's predictiveness of actual model behavior through self-evaluation protocol. The top row shows the agent's generated explana- tions. The bottom rows show images predicted to elicit high (green) or low (red) model responses, along with their actual model scores. Results are shown for different target concepts across an object recognition model with a controlled attribute reliance (left), CLIP (middle), and YOLOv8 (right).
often rely on manual inspection and assume access to a fixed set of predefined concepts, limiting their ability to scale to modern models with complex behaviors.
In this paper, we introduce a fully automated framework designed to detect visual attribute reliance in pretrained vision models. Given a pretrained model and target visual concept (e.g., an image classifier selective for the object vase), our method identifies specific image features that systemati- cally influence the model's predictions, even when these features fall outside the model's intended behavior (e.g., the classifier relies on flowers to detect the vase, Fig. 1). At the core of our approach self-reflective agent (implemented with a backbone multimodal LLM) that treats the task as a scientific discovery process. Rather than relying on a predefined set of candidate attributes, the agent autonomously formulates hypotheses about image features that the model might rely on, designs targeted tests, and updates its beliefs based on observed model behavior. In contrast to previous interpretability agents [Schwettmann et al., 2023, Rott Shaham et al., 2024], the self-reflective agent does not stop after generating an initial finding but rather actively evaluates how well it aligns the model's 's behavior on unseen test cases. Importantly, this evaluation self-contained and does not require any ground-truth knowledge about the model's attribute dependencies. Instead, the agent generates two sets of test images: one where the candidate attribute is present (which is expected to elicit high prediction scores from the model) and one where the attribute is absent (which is expected to elicit low prediction scores). When discrepancies arise between the expected and actual model behaviors, the agent reflects its assumptions, identifies gaps inconsistencies in its current understanding, and initiates a new poothesii--estinng loop. We show that the agent's ability to reason about attribute reliance significantly improves with self-reflection rounds (see Sec. 5).
To quantitatively evaluate our method, we introduce a novel benchmark of 130 object recognition models, each constructed with well-defined intended behavior and an explicitly injected attribute dependency. The benchmark spans 18 types of visual reliances, inspired by vulnerabilities known to exist in vision models [Dreyer et al., 2023, Geirhos al., 2018, 2020, Buolamwini and Gebru, 2018, Xiao et al., 2021, Wang et al., 2019, Singh et al., 2020]. These include object-level attributes (e.g., color, material), contextual dependencies (e.g.. background, co-occurring object state), and demographic associations. Together with an automated evaluation protocol, this benchmark provides controlled environment for evaluating visual attribute reliance detection methods.
Our method is model-agnostic and can be applied to any vision model that assigns scores to input images. To demonstrate its versatility, we evaluate it on both our controlled benchmark and state- of-the-art pretrained models. Across a range of visual reliance types, our experiments show that our self-reflective agent consistently outperforms non-reflective basclines. Moreover, it successfully

---

uncovers previously unreported attribute dependencies in pretrained models (Fig. 1). For example, it identifies that the CLIP-ViT vision encoder [Radford et al., 2021] recognizes teachers based on classroom backgrounds and that YOLO088 [Jocher et al., 2023], trained for object detection for autonomous driving, relies on the presence of crosswalks to detect pedestrians. These findings highlight the efficacy of our method as scalable tool for detecting hidden dependencies in pretrained models deployed in real-world scenarios.
2 Related Work
Revealing Visual Attribute Reliance in Vision Models. Prior work has explored methods to uncover the visual cues that drive model predictions. One common strategy is to manipulate input features to isolate model sensitivities, such shape spectral biases in elassifiers [Gavrikov and Keuper, 2024], or attribute preferences in face recognition systems [Liang et al., 2023]. Other works rely on interpretability tools to identify potential dependencies for example, extracting keywords from captions of misclassified images [Kim et al., 2024], or using feature visualizations to reveal facial attribute reliance [Teotia al., 2022]. However, most of these methods target specific types of biases and rely on predefined concept sets or human inspection. In contrast, our framework introduces a unified, flexible approach that can detect broad range of attribute reliances without prior assumptions about the relevant features.
Interpretability and Automated Analysis. Initial work on interpretability automation produced textual descriptions of internal model features, using keywords [Bau et al., 2017]. programs [Mu and Andreas, 2020], natural language summaries [Hernandez et al., 2021, Bills al., 2023, Gandelsman et al., 2023]. While informative, these descriptions are typically correlational, lacking behavioral validation [Huang al., 2023, Schwettmann al., 2023, Hausladen al., 2024]. More recent work introduces agents that actively probe models. For instance, the Automated Interpretability Agent (AIA) [Schwettmann al., 2023] used a language model to analyze black-box systems via a single pass over the input space. The Multimodal Automated Interpretability Agent (MAIA) [Rott Shaham et al., 2024] extended this approach by incorporating iterative experimentation and multimodal tools, enabling more detailed analysis of model internals. Our work builds on this direction by focusing specifically on discovering visual attribute reliances and introducing self-reflection mechanism that allows the agent to revise faulty hypotheses based on experimental evidence, leading to more accurate and robust conclusions.
Benchmarks for Visual Attribute Reliance Detection. Standardized benchmarks for evaluating visual attribute reliance remain limited. Prior evaluations often use models trained on datasets with known biases, such as WaterBirds [Wah et al., 201 1] or CelebA [Liu et al., 2015], using label co-occurrence as a proxy for ground truth [Sagawa al., 2020]. For generative settings, Open Bias [D'Incà et al., 2024] proposes biases via LLMs, generates images from biased prompts, and assesses reliance using VQA models. However, OpenBias is not applicable to predictive models and cannot conduct controlled interventions. We introduce suite of 130 vision models with explicitly injected attribute reliances across 18 categories, providing fine-grained control over attribute type and strength. This allows rigorous, scalable evaluation of reliance detection methods in predictive setting.
Agent-Base Based Reasoning and Self- Reff-Refeective Systems. A growing body of work explores how agents can improve reasoning through reflection, feedback, and interaction. Methods like SELF- Refine [Madaan et al., 2023], Reflexion [Shinn et al., 2023], and ReAct [Yao et al., 2023] introduce multi-step loops in which agents revise their outputs via self-critique. Similarly, Du et al. [Du et al., 2023] show that multi-agent debate improves factual consistency and reasoning in language models. Our agent uses a task-specific self-evaluation protocol, enabling it to assess whether its conclusions align with actual model behavior. This integration of behavioral validation with self-refiection allows our agent to autonomously revise hypotheses and close the interpretability loop.
3 Self-Reflective Automated Interpretability Agent
Our framework is designed to automatically discover visual attributes that a pretrained model relies on to perform its task. Our approach consists of two main stages. (i) Hypothesis-Tesiing stage, in which an autonomous agent provided with subject model (e.g., an image classifier) and a target concept to explore (c.g.. vase). The agent is tasked with discovering visual attributes in
3

---

What are input factures my madel relies on to cietect suit?
Hypothesis-Tssingg
Self-Raflaction
Hypothesis-Teshi
Locking
imagen with aighest
Letts svaluate the "corporate settings" conclusion lecte look this
Poct: 13-degroc vicks of cuits
detection scores:
images with and attribute and evaleate it they match model xesponses.
00335
Self Sell-uaauttion
0:75
Predicted high-scoring magess
Teat mele
va.
Test:
relies black auite
0.39
0.32
0.34
Twat: puzporete wwil LiiLing
The model
relina
male
veating
Ietra evaluate chis conclusion
0.76
Predicted low-scoring
The nodal relles on corporate setrings.
Bone of the inacsa with corporate settings got 1oM confidenze
The conclusion natchea the model nchaviari The model rolies male presenting Ziqures to the suit.
While corporate setting still gez high scozes, Lot's zun another wagweiment.
Figure 2: Attribute reliance detection through hypothesis testing and self-reflection. To discover features that drive model prediction, SAIA starts by formulating and testing range of hypothe- ses. After reaching a conclusion (e.g.. the model favors suit in corporate settings), performs self-evaluation by testing the model responses on images with and without this feature. When inconsistencies between the conclusion and model behavior are observed (e.g., some non-corporate images yield high scores, while some corporate images yield lower scores), SAIA updates its prior beliefs according these discrepancies and test alternative hypothesized explanations.
the input image that the subject model relies on to perform recognition tasks. The agent proposes candidate attributes that may influence the model's behavior, designs targeted experiments to test its hypotheses, and iteratively refines them based on observed results. This cycle continues until the agent converges to stable explanation of the model's reliance. (ii) Self-Reflection Stage, in which the agent uses self-evaluation tool to score explanation. This done by quantifying how well the agent's 's explanation matches the behavior of the model in new input images. If the explanation fails to generalize or reveals inconsistencies, the agent reflects on its prior explanation in light the evaluation evidence and launches new hypothesis-testing stage. Both stages are demonstrated in Fig. 2 and together form our Self-reflective Automated Interpretability Agent (SAIA).
3.1 Hypothesis-Tese Testing Stage
In this stage, SAIA iteratively refines hypotheses about the attribute sensitivities of the subject model. Inspired by MAIA [Rott Shaham et al., 2024], we design SAIA to operate in a scientific loop: it begins by proposing candidate attributes that the subject model might rely on, designs multiple experiments involving generating and editing images to test these hypotheses (e.g. edit an image with a suit change its color), observes the resulting model's behavior (e.g. measure subject model scores across these experiments) and updates its beliefs accordingly. This cycle continues until SATA converges on an initial conclusion about the model's sensitivity to image features.
Agent actions SAIA interacts with the subject model through set of predefined actions imple- mented Python functions. These actions include: (i) querying the subject model with given input image to observe its prediction score; (ii) retrieving the set of images that achieve the highest output responses from a fixed dataset, identify inputs that strongly trigger the target concept; (iii) generating new images using a text-to-image model; (iv) editing existing images to manipulate specific attributes; (v) summarizing visual information across one or more images into text, to infer shared features; and (vi) displaying function which enables SAIA to log images, text, or other results notebook available throughout the experiment. SAIA designs experiments by composing multiple actions together through Python scripts. It then observes the experiment results- combination of

---

What input festures my model relies teddy
Self-Ewalustion Protocol
higheroorin magee
Messured model bohesior
Experiment Conclusion
Modal
The model relles on the teady being presented in educational aettings
Prompt gernereror
Taxt-to-iine Maxt-to-image moo-image
0.47
Model Soore
Predicted lowascoingg
O Self-Reflection
Hypothesis- Testing Experimant
images with predicted high expected, let's continue with the experiments. Bighest scores were for direct child interaction, rather than educstional. settings general. dducational setups without hunan presence scored only noderately. Tsol ated beace in various rettings and enviromenttal conditions didn't signifieantly impact sccres.
Some
than
Figure 3: Self-reflection stage. SAIA initiates hypothesis-testing experiment. After testing multiple candidate hypotheses (see Fig. 2), SAIA draws an inital conclusion (e.g., teddy bear are detected based on appearing in educational settings). SAIA then uses self-evaluation protocol that generates synthetic images via a text-to-image model and computes the subject model's scores on these images. The self-evaluation protocol compares the predicted and actual model scores, triggering another round of hypothesis testing if results deviate from expectations. In this example, SAIA observes that the highest scores correlate with direct child interaction rather than generic educational settings, leading to refined future hypotheses.
text and images-and and decides whether to continue with more experiments to output conclu- sion. We implement SAIA with a Staude-Sonnet. backbone. Please refer to Appendix A for implementation details, full prompts, and API.
3.2 Self-Reflective Stage
Once the hypothesis testing stage is complete and SAIA reports its conclusion for the initial hypothesis- testing stage, initiate self-reflection stage. In this stage, the conclusion from the hypothesis testing stage is scored using a self-evaluation protocol. This protocol is completely unsupervised and does not require any ground-truth labels or external information. Instead, it measures how well the detected reliance matches the actual behavior of the subject model. SATA's detected reliance sufficiently matches the model's behavior, it terminates the experiments and returns the current conclusion. Otherwise, if inconsistencies between SAIA's conclusion and the model behavior are found, the information collected from the self-evaluation stage is returned to SAIA, which reflects over its previous conclusion and initiates another sypothesis-testing round. This process is demonstrated in Fig. 3.
Self-Evaluation protocol Self-evaluation serves two key purposes: (i) to assess whether the current explanation matches the model's behavior, and (ii) guide further experimentation if it does not. The process begins by querying a separate language model instance (we use Claude-Sonnet-3, generate two diverse of image prompts, that according to SATA's description predicted to elicit high and low detection scores, respectively. The first set, termed the "predicted high-scoring images", contains instances of images with the target concept (c.g., teddy bear) along with the detected reliance attribute (e.g., educational settings) that SAIA detected the model to be selective for. The second set, the "predicted low-scoring images", contains instances of the target concept, but with the absence of the detected attribute. This prompt generation strategy emphasizes attribute-contre controlled diversity, where each prompt in the high low-scoring group keeps the core attribute constant while allowing other visual factors to vary. These prompts are then used as inputs to a text-to- text-to-image model (we use Flux. 1-dev Labs [2024]) that generates the corresponding images, which are then fed to the subject model, and the output scores are recorded. If SAIA's explanation is accurate, the model should exhibit systematically higher scores on the "predicted high-scoring" images and lower scores on the "predicted low-scoring" set.
5

---

rehaavor-matching protocols of this kind have been shown to be effective in other evaluation settings, particularly for the task of producing textual labels of neurons' behavior in pretrained models [Rott Shaham et al., 2024, Kopf et al., 2024, Huang et al., 2023]. In domains where ground- truth explanations are unavailable, they provide a way to validate hypotheses through behavioral consistency. We repurpose this evaluation method as a basis for self-reflection: SAIA uses it to validate its own conclusions, determine whether further experimentation is necessary, and reflect on its own findings based measured model behavior.
Agent self-reflection After observing the model's 's responses to the "predicted high-scoring and "predicted low-scoring" images, SAIA reflects on whether the results align with its expectations. If the mean scores between the two groups are not sufficiently separated based on an empirically set threshold (i.e. the conclusion is not sufficiently discriminative), SAIA may decide that its current conclusion is incomplete or inaccurate. then analyzes which visual attributes within the generated images might explain these discrepancies. In doing so, SAIA updates its hypothesis narrowing the original explanation (e.g., refining "educational settings" to "child interaction") or by generating alternative hypotheses altogether. This reflective process closes the experimental loop and allows SAIA to reinitiate the hypothesis-testing stage with better-informed guidance.
either by
In practice, we cap the total number of agent rounds (hypothesis-festing followed by self-reftection) to 10. If no hypothesis meets the self-evaluation threshold by that point, SAIA returns the hypothesis that achieved the best alignment between predicted and actual model behavior, typically the most recent one. Empirically, SAIA converges well before reaching this cap - in most runs, stops after just 2-4 rounds. This behavior is visualized in Figure 5a, which shows that the predictiveness score (see definition in Sec. 5.1) generally improves monotonically across rounds, providing a natural convergence signal. Please refer to Appendix A.2 for the full self-reflection instructions.
4 A Benchmark of Models with Controlled Attribute Reliance
To evaluate the capabilities of SAIA, we constructed a benchmark of 130 unigue object recognition models that exhibit 18 diverse types of visual attribute reliance. All simulated behaviors are inspired by known vulnerabilities of vision models [Dreyer et al., 2023, Geirhos et al., 2018, 2020, Buolamwini and Gebru, 2018, Xiao et al., 2021, Wang et al., 2019, Singh et al., 2020], and mimic spurious correlations between the target object and image attributes such as object color, background context, co-occurring object state, or demographic cues. To assess the generalizability of our method, the benchmark also includes a subset of models with counterfactual attribute reliance that are intentionally rare or unnatural in real-world pretrained models (c.g.. a suit detector responds more strongly when women wear the suit). Each benchmark model includes input parameter that controls the strength of the injected reliance, allowing for precise control over model behavior. Importantly, because these models explicitly engineered with known intended behavior, they serve as a controlled testbed for evaluating and comparing feature reliance detection methods.
4.1 Simulating attribute dependencies
Figure 4 illustrates simulated attribute reliance scenario. Given target object class (e.g. bird) and an intended injected attribute reliance i (e.g. setting; beach), we simulate a model C.,, that detects l under the condition i. Each benchmark model is composed of two components; an object detector Or and an attribute condition detector A,, which modulates the output of O2 based on the presence or absence of the specified attribute. To compute the final output of the model Ct,; on an input image img, we first pass the image through the object detector O, If the target object class is not detected, the model returns low random bascline score. If the object is detected, the image is then evaluated by the attribute condition detector A,. If the attribute condition is satisfied, the original subject model score O is returned, simulating full model response. Otherwise, the subject model score is discounted by multiplication factor of simulating attenuated confidence due to the missing attribute. The scalar a E [0. 1] controls the magnitude of the injected reliance: higher values of a simulate stronger reliance on the attribute. Please refer to Appendix B.2 for empirical evaluation of reliance magnitude as a function of a. In all the benchmark models, we use Grounding DINO [Liu et al., 2023] as the object detector O2 and SigLIP [Zhai et al., 2023] as the attribute condition detector As, which in practice is guided by a textual description of the injected attribute condition For demographic attribute dependencies, FairFace [Karkkainen and Joo, 2021] is used for A, instead

---

Input Images
Conditioning visual attr attribute A
Object attribute: Color
Yes
Is the beach setting condition satisfied?
Cuting 0,(ing)
Object attribute: Material
O₂ is the bird present?
Context attribute: Image background
Context attribute: Object state or configuration
(ing
(1-e) o,(ing)
Demographic attribute: Gender of people present
Diacount
Demographic attribute: Age of people present
rand(e.0.
factor
Figure 4: Simulating visual feature reliance. We simulate feature reliance by modulating object recognition scores based on the presence of specific visual attributes (c.g., bird detector that relies on the presence of beach background). Given an input image and object category t, Or produces object recognition score for object presence. If the object is not detected, low random score is assigned as the final model score of the image. If the object is detected, we simulate an attribute dependency (e.g., presence of "beach background" for bird detection) through the procedure described in Sec. 4.1. If the condition is satisfied, the final model score equals O₂ (img) recognition score. Otherwise, the score discounted by a factor a to represent the model's 's weaker response in the case that the attribute condition is not met.
of SigLIP (see implementation details below). Notably, this model composition approach is highly flexible-on one could engineer any object-condition pairing to construct an object detection model with desired attribute reliance.
4.2 Attribute Condition Categories
We categorize the attribute conditions used to inject reliance into four groups: object attributes, context attributes, demographic attributes, and counterfactual demographic attributes. These categories reflect different types of visual dependencies observed (or intentionally constructed) in our benchmark models, and guide the choice of attribute detector used in each case. Please see the full list of constructed models in Appendix B.
Object attributes These attribute dependencies relate visual properties of the object itself. We include reliance on object color and material, using SigLIP as A for zero-shot classification of object-specific attributes (e.g. SigLIP guided with the prompt red bus to inject color reliance to a bus detector), A color-reliant system returns the full score from O, only if the object has specific color, otherwise the response is discounted; similarly, material-reliant model will have full response only if the object is of the intended material (c.g., vases made of ceramic).
Context attributes These dependencies reflect properties of the object's surrounding context. We simulate reliance on the specific setting of the object (e.g. keyboard only only if it is in urban environment). Here well, we
being typed)
and object background (e.g. SigLIP-guided text for detecting the intended attribute.
Demographic attributes These dependencies are based on the age gender of people interacting with the target object. We use FairFace as AA to detect demographic attributes and construct systems relying on these (e.g., apron detector that relies on the apron to be worn by women, and glasses detector that relies on the glasses to be worn by older individuals).
Counterfactual demographic attributes To test whether SAIA can discover atypical or out-of- distribution dependencies, we include models with counterfactual demographie reliance (e.g., an apron detector that activates only when worn by men, or a glasses detector that prefers younger wearers), which rarely co-occurs real-world data. These systems allow us to test whether SAIA can detect unexpected or counterintuitive reliance patterns that do not follow natural co-occurrence statistics. This distinction allows us to assess both SAIA's ability to uncover realistic demographic biases and its robustness to rare previously unknown dependencies.
7

---

1.0
Reliance
MAIA
Factuals
Predictiveness Score
0.5 Reliance
Milan
Counterfactuals (0.9)
0.9
Ground-Truth
Muit-Agent
Factuals (0.5)
... Counterfactuals (0.5)
0.8
0.7
0.6
2
4
6
8
10
2
10
Self-Reflection Rounds
Self Reff-Refeecton Rounds
(a) Predictiveness Scores
(b) Factuals vs. Counterfactuals Reliances
Figure 5: Predictiveness Score over Self-Reflection Rounds. (a) We plot the average predictiveness score over all models in the benchmark for a harsh reliance magnitude of a = 0.9, as well as a softer reliance magnitude of 0.5. We see a steady increase in the predictiveness scores of SAIA's conclusions over rounds for both discount factors, approaching their respective theoretical upper bounds as given by the ground truth baseline. As expected, the scores for softer reliance models (a 0.5) are consistently lower than those of the stronger reliance models (a 0.9), illustrating that subtler attribute reliances are more challenging to detect. SAIA outperforms all nonreflective baselines (MAIA, Milan, and Multi-Agent) by significant margin for both reliance magnituee values. (b) We compare the predictiveness scores of SAIA's reliance descriptions over factual models with dejjoo-demoggaphicc
more intuitive demographic attribute reliances against counterfactual reliances associations that are not commonly observed. Although SAIA's descriptions of the counterfactual models achieve lower predictiveness scores, the performance still reliably improves over increased rounds of self-reflection for both a settings.
Experiments
We evaluate the performance of SAIA on both our synthetic benchmark models and on pretrained vision models widely used practical settings. Examples of attribute reliance discovered by SAIA, as well as evaluation results, are shown in Figures 1, 2, 2, and
5.1 Evaluation protocol
We quantitatively evaluate the accuracy the detected attribute reliances generated by SAIA and compare its performance to four different baselines.
Predictiveness score Following Kopf et al. [2024], Schwettmann et al. [2023], quantify how well candidate reliance description matches model behavior. Similar to the self-evaluation score, given candidate explanation, high model scores and 10 that are expected to elicit low scores. We then pass these images through the model and record its actual responses. Each image is assigned binary prediction label (high or low predicted response), and we threshold the model's scores to obtain a binary outcome (high or low measured response). The predictiveness score is computed as the proportion of images where the predicted label matches the model's 's actual binary output. This reflects how well the explanation predicts individual model responses.
start by generating 10 synthetic images that expected to elicit
LLM us Judge We use a language model as a judge [Zheng et al., 2023] in a two-alternative forced choice (2AFC) setting. Given ground-truth explanation of benchmark model and two candidate explanations, one produced by SAIA and one from a baseline, the LLM judge is asked to choose which candidate better matches the ground-truth description. For each description pair, repeat the test 10 times, and report the average preference rate for SAIA's descriptions.
Baselines We compare SAIA against the following alternatives: (i) Milan-style reliance detection: Following Milan [Hernandez et al., 2021], this approach avoids iterative experimentation and detects the reliance based precomputed set of image exemplars that maximize the model's scores.

---

What are the visual attr butes that CUP relies on to recognize the conzept scientist?
What are attributes that CLIP on recognize the concept wine glass?
What are attributes YOLO relies on to detect bikers?
Lotta conduct, aone exporimcnis..
Locts conduct seme experiments.
Latta conduct sone caperiments..
CLIPP telles on traditional laboratory sett ings over compalatioonal sclent Cie wark.
CLIP proters singuier wine glacses against simple paakgrounds.
YOLO relles an 43-dogrse and rear Fose ot bikere
Prodicted high scoring imapes
Predicted high scoring images
Predicted sighis
scoring images
-40-79
0.07
0.34
0.16
0.16
0.49
0.39
0.18
Predicted low www-icoring images
Predicted low scoring impes
Prodicted low now-scoorng images
Figure 7: Detected feature reliance in CLIP and YOLO. SAIA identifies visual attribute dependen- cies in state-of-the-art pretrained models that have not been previously documented. For each concept (e.g., scientist, wine glass, biker). SAIA infers an attribute reliance through natural-language explanation and tests it by comparing predicted high and low-scoring images (scores are normalized for clarity). The examples reveal that CLIP-ViL VIT relies traditional laboratory settings to recognize scientists, while YOLOv8 favors de-degr and rear views for detecting bikers.
Ground-truhh descr
LLM as judgo: cancidiate is mor simlari meaning to the ground truth clese issiniptio?
(ii) MAlA-style agent: Based Rott Sha- ham et al. [2024], this method performs hy- pothesis testing but does not engage in self- reflection to revise its explanation. For a
resgonetog festunee of analog clock
A: the classifier zelles on tradit anslog clocke appearance
B: The classićioz rolics en Rosasi numerlla on the olock face
fair comparison, we equip MAIA with the
same set of tools and backbone model that method uses. (iii) Multi-agent ensemble: We run 10 independent MAIA-style, non- self-reflective agents and select the expla- nation with the highest predictiveness score. This tests whether repeated sampling alone can match the performance gains from self- reflection. (iv) Ground-truth descriptions: For benchmark models, we include ideal nat- ural language descriptions of the injected re- liance as an upper bound on performance.
Agent's destription
Competitive description
SAIA vs. Milan
SAIA vs. MAIA
SAIA Multingent 0.53 + 0.03
0.5
0.51 + 0.02
0.56 + 0.03
0.9
0.59 + 0.02
0.6 0.02
0.62 + 0.03
Figure 6: 2AFC evaluation. We use language model (GPT-4) a judge. Given a ground-truth description, the LLM compares two candidate ex- planations: one generated by SAIA and one from competitive baseline (MAIA, Milan, or Multiagent). The LLM selects the candidate it finds most semanti- cally similar to the ground truth. We report average preference rate for SAIA in the table.
5.2 Evaluating Benchmark Models
Self-reflection enhances reliance detection
As showed in Fig. 5a, predictiveness scores steadily improve over the course of self-reflection rounds, suggesting that SAIA's explanations become increasingly aligned with the model's actual behavior. Notably, performance exceeds the MILAN one-shot baseline, a MAIA-style agent, and the non-reflective multiagent system, indicating that self-reflection offers a distinct advantage.
Robust performance across different degrees of model reliance Figure 5a shows consistent performance gains for both strong (a 0.9) and weak (a 0.5) reliance settings, demonstrating that SAIA is effective across a range of dependency strengths. While stronger dependencies lead to higher absolute scores, the relative improvement from self-reflection remains significant even in more ambiguous scenarios. The same trend is seen in the 2AFC test (Fig. 6), where the final conclusion from SAIA is more frequently preferred over the baselines for the stronger reliance.
SAIA discovers counterfactual feature reliances In addition to recovering realistic dependencies, SAIA successfully identifies counterfuctual attribute reliances (Fig. 5b). This indicates that SAIA is capable of discovering surprising or non-intuitive patterns of reliance, rather than simply mirroring familiar dataset biases.

---

5.3 Revealing attribute reliance in pretrained vision models
We deploy SAIA to detect attribute reliances in two pre-trained vision models: the CLIP-ViT image encoder [Radford et al., 2021] trained to align image and text representations, and the YOLOv8 model [Jocher et al., 2023] trained for object detection in autonomous driving settings. With CLIP, we perform object recognition by measuring the cosine similarity of the image with target prompt (e.g., "A picture of a scientist") "). For YOLOv8 we measure the detection score of the target object class. Figures and 7 show that SAIA can generate natural-languan language descriptions of attribute reliance in various contexts. The generated descriptions are shown to be predictive of model behavior, as model scores increase when the reliance is satisfied and decrease when it is absent. Surprisingly, SAIA reveals dependencies that were never observed before, such as the reliance of clip on traditional laboratory settings when detecting scientist, and YOLOv8 dependency on bikers' poses. We note that SAIA's goal is to surface such dependencies rather than to assess their desirability or harm, allowing practitioners to make informed judgments based on specific downstream use cases.
5.4 Revealing compositional visual reliance
Vision models can rely on combinations of multiple attributes to detect certain concepts. To evaluate SAIA's performance in such settings, we created ten additional synthetic benchmark models that each exhibit two attribute reliances. Five models depend on the simultaneous presence of both attributes (e.g., bench detector that relies on benches that are wooden AND in beach setfings), while the other five exhibit high confidence if either attribute is present (c.g., wooden benches OR benches in beach settings). Model details are provided in Table 6 of the Appendix.
For models requiring both attributes simultancously (AND logic), SAIA successfully identified both dependencies in 80% of cases, whereas MAIA (the non-self-reflectivve baseline) failed to detect both attributes in any case. This performance gap stems from SAIA's self-evaluation protocol, which rigorously tests multi-attribun hypotheses by constructing positive exemplars containing all relevant attributes and negative exemplars containing none or only subset of candidate attributes. For models relying on at least one of two attributes (OR logic), both methods struggle more significantly. SAIA recovered both dependencies in only 20% of synthetic models, while MAIA again failed to detect both dependencies. This difficulty likely arises from terminating the process once attribute is found. Future extensions could address this limitation with a hicrarchical or trre-structured experimental design that explicitly enforces multi-reliance exploration. See Appendix C.2 for further results and analysis of these experiments.
Overall, these findings demonstrate that self-reflection enhances SAIA's robustness and compositional reasoning, leading to more reliable explanations even when multiple visual dependencies are involved. Beyond multi-attribute detection, the self-reflective stage also improves performance across broader experimental settings. One limitation of agentic interpretability methods is that relying on external tools introduces errors stemming from tool inconsistencies or biases. A key distinction in our framework is the inclusion of a self-reflcction loop, which help to mitigate such artifacts. Please refer to Appendix C.3 for experimental details and analysis.
6 Conclusion
We introduced SAIA, a self-reflective agent for discovering attribute reliance in pretrained vision models. Treating interpretability scientific discovery process, SAIA autonomously generates hypotheses, tests them through synthetic image generation and editing, and refines its conclusions via a self-evaluation loop. Our experiments on controlled benchmarks and real-world models as CLIP and YOLOV8 show that SAIA outperforms non-reflective baselines and uncovers both expected and previously unreported attribute dependencies. While the method provides a scalable framework for behavioral model auditing, its performance depends on the quality of generated images, which can limit robustness for rare or cott-of-ditributte concepts. Moreover, the agent's 's backbone model inherits known priors and biases that may constrain the hypothesis space and introduce confirmation bias-likely explaining the reduced performance on counterfactual demographic models, though self-reflection still improves predictive reliability. Despite these challenges, self-reflective agents mark promising step toward transparent and automated model auditing, and their effectiveness will continue to grow with advances in image generation and agent reasoning.
10

---

Acknowledgments
We are grateful for the support of ARL grant #W91 1NF-114--F22000 MIT-IBM Watson Lab grant #W1771646, Hyundai Motor Company, ONR MURI grant #033697-00007, and PID2022-1388 139222-187 100, funded by MCIN/AEI/I0, 13039/501 10001 1033 and FEDER, EU. This research (partially) supported by the DARPA AIQ program through the DARPA CMO contract number HR001
References
M. A. Alcorn, Q. Li, Z. Gong, C. Wang. L. Mai, W.-S. Ku, and A. Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4845-4854, 2019.
D. Bau, B. Zhou, A. Khosla, Oliva, and Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6541-6549, 2017.
S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. URL htps://openniipulic blob. core. windows. nethheunon-explaieer//pppperr 2023.
html.(Date accessed: 14.05. 2023), 2.
J. Buolamwini and T. Gebru. Gender shades: Intersectional accuracy disparities commercial gender classification. Proceedings of Machine Learning Research, 81:77-91, 2018.
M. D' O'Ina,, E. Peruzzo, M. Mancini, D. Xu, Goel, X. Xu, Z. Wang. H. Shi, and Sebe. Openbias: Open-set bias detection in text-to-image generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1222--1235, 2024.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
M. Dreyer, Achtibat, T. Wiegand, W. Samek, and S. Lapuschkin. Revealing hidden context bias in segmentation and object detection through concept-specific explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3829-3839, 2023.
Y. Du, A. Zeliger, S. Gehrmann, D. Kiela, Tenenbaum, and J. Andreas. Improving factuality and reasoning in language models through multiagent debate. In NeurlPS, 2023.
Y. Gandelsman, A. Efros, and Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprint arXiv:2310.059) 2023.
Gavrikov and J. Keuper. Can biases in imagenet models explain generalization? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22184-22194, 2024.
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet-trained cnns biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint 1811 12231, 2018.
R. Geirhos, J.-H. Jacobsen, C. Michaelis, and et al. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Z. Geng, B. Yang. T. Hang, Li, S. Gu, T. Zhang, J. Bao, Z. Zhang, H. Hu, D. Chen, and B. Guo. Instructdiffusion: A generalist modeling interface for vision tasks. CoRR, 055/233900389, 2023. doi: 40..50//aXXiv.2339.oo .03895 URL https: /dii. org/10 48550/ arviv. 2309. 03895.
A. Ghorbani, J. Wexler, and et al. Towards automatic concept-based explanations. In NewrIPS, 2019.
C. Hausladen, M. Knott, C. Camerer, and Perona. Social perception of faces in a vision- language model. arXiv preprint arXiv:2408..444 2024.

---

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. CVPR, 2021.
E. Hernandez, S. Schwettmann, D. Bau, T. Bagashvili, A. Torralba, and J. Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2021.
J. Huang, A. Geiger, K. D'Oosterlinck,, Z. Wu, and C. Potts. Rigorously assessing natural language explanations of neurons. arXiv preprint arXiv:2309. 10312, 2023.
G. Jocher, A. Chaurasia, and J. Qiu. Ultralyties yolov8, 2023. URL https://github. com/ ultrralttics/ultrrayyii..
K. Karkkainen and J. Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548-1558 2021.
B. Kim, M. Wattenberg, Gilmer, C. Cai, J. Wexler, and F. Viégas. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In Proceedings of the International Conference on Machine Learning (ICML), pages 2668-2677, 2018.
Y. Kim, S. Mo, M. Kim, K. Lee, J. Lee, and J. Shin. Discovering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1102--10099 10922 2024.
P.-J. Kindermans, S. Hooker, and et al. The (un)reliability of saliency methods. arXiv preprint arXiv:1711.00867. 2017.
L. Kopf, P. L. Bommer, Hedström, S. Lapuschkin, M. Höhne, and K. Bykov. Cosy: Evaluating textual explanations of neurons. Advances in Neural Information Processing Systems, 37:34656- 34685, 2024.
B. F. Labs. Flux. http::///gchhbbccmmbbbbeeeetttt
hwt.huu...cmbbbcc.kk..... www.fforetttteee labs/flux, 2024.
H. Liang, P. Perona, and G. Balakrishnan. Benchmarking algorithmic bias in face recognition: An experimental approach using synthetic faces and human evaluation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4977-4987 2023.
S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-trate rre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang. S. Welleck, B. P. Majumder, S. Gupta, A. Yazdanbakhsh, and Clark. Self-refine: Iterative refinement with self-feedback, 2023.
Mu and J. Andreas. Compositional explanations of neurons. NeurlPS, 2020.
C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017.
OpenAI. Dall-e 2023. attps:://openai.co//aaall//
OpenAl. Gpt-4o technical report, 2024. https://openai.com http://penai.cco//////////////// hetp/index/gpti
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 5748-8763. PmLR, 2021.
12

---

B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In Proceedings of the International Conference on Machine Learning (ICML), pages 5389-5400, 2019.
A. Rosenfeld, R. Zemel, and J. K. Tsotsos. The elephant in the room. arXiv preprini 1808.08000033000 2018.
T. Rott Shaham, S. Schwettmann, F. Wang, A. Rajaram, E. Hernandez, J. Andreas, and A. Torralba. A multimodal automated interpretability agent. In Forty-first International Conference on Machine Learning, 2024.
O. Russakovsky, J. Deng, H. Su, Krause, Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211 1-252, 2015.
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.0873. 2020.
Schwettmann, T. Shaham, Materzynska, N. Chowdhury, S. Li, J. Andreas, D. Bau, and A. Tor- ralba. Find: A function description benchmark for evaluating interpretability methods. Advances in Neural Information Processing Systems, 36:75688-7577 75715, 2023.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via caudient-bbsed localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 618-626, 2017.
N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36:8634-8652 2023.
K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6033 2013.
A. Singh, Y. Goyal, D. Batra, and D. Parikh. Don't just assume; look and answer: Overcoming priors for visual question answering. In CVPR, 2020.
M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances Neural Information Processing Systems (NeurIPS), 2020.
D. Teotia, A. Lapedriza, and S. Ostadabbas. Interpreting face inference models using hierarchical network dissection. International Journal of Computer Vision, 13065); 2022.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongic. The caltech-uesd birds-200-201 I dataset. 2011.
T. Wang, J. Zhao, M. Yatskar, K.-W. Chang, and V. Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5310-5319, 2019.
O. Wiles, A. Black, B. Avila Pires, and Riquelme. fine-grained analysis of distribution shift on imagenet-21k. In International Conference on Learning Representations (ICLR), 2022.
B. Wilson, J. Hoffman, J. Morgenstern and N. De Freitas. Predictive inequity in object detection. arXiv preprint arXiv: 1902. 1097, 2019.
K. Y. Xiao, L. Engstrom, A. Ilyas, and A. Madry. Noise or signal: The role of image backgrounds in object recognition. International Conference on Learning Representations, 2021. URL https: http://pppneeeeeeeeeee/e wttp://wwww..eettttt///i//e/eeee xY7wLQ.
13

---

S. Yao, J. Zhao, D. Yu, B. Yu, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models, arXiv preprint arXiv:2210.03629, 2023.
X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 11975 1975-11986, 11986, October 2023.
L. Zheng, W.-L. Chiang, Y. Sheng, Zhuang. Z. Wu, Zhuang, Z. Lin, 2. Li, D. Li, E. Xing. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, systematically support all claims made in the abstract introduction in the following sections the main paper and appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, we perform detailed analysis of the limitations of the work in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we discuss the implementation details of the self-reflective agent in Section 3 and SM, the benchmark models in Section 4 and SM, and the experiments and evaluations in Section 5 with additional experimental results in SM.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and model benchmark will be made available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
somoiiiiiiiicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ttpp.cccppbllll/// http:ccccpppccccccccceeccccccccc htppuuliccguuieeesseeeeppp guides/ /CoeeSubmdbibiii iss ssisssoolooooooooo oonooiiyyyylllooooiomoooomiiiilp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimentation setup and results are discussed in detail in Section 5 and S.M.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Confidence intervals are shown for both the 2AFC test and the ground truth predictiveness scores.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The GPU and API resources required to reproduce the experiments are de- scribed in SM.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https ///e://///eeee /tt.reeurrp..peee cet...e//ppe cc/public/ hiteee/Ehhhhhhiiee csQuidelllllllc ines?
Answer: [Yes]
Justification: Yes, the authors have reviewed the NeurlPS Code of Ethics and the research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Both potential positive and negative societal impacts of the work discussed in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work presented in the paper does pose significant risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators original owners of assets properly cited throughout the work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code and model benchmark documentation will be released upon accep- tance.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The usage of LLMs as the backbone the self-reflective agent is described in Section 3, and the usage of LLM-as-a-judge for an external evaluation is described in Section
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //teuripes..///// wtt.ccccccooooeocco C for what should or should be described.
21

---

Appendix
A
SATA Implementation
A.1 Hypothesis loop instructions
You will get an access to a classifier in the final layer of a deep neural network that is sensitive to the presence of <concept> within photo-realistic images. Your overall task is reveal biases in the classifier's behavior. That is, your task is to find specific features for which the classifier tends have relatively higher confidence scores, but only covers partial aspect of the <concept> class, or specific features for which the classifier tends to have relatively lower confidence scores although <concept> is still present.
Try to identify and explain the cause of biases in the network. These biases could reflect real-world biases or they could be spurious. Some bias categories include:
1. Contextual Biases: Settings/environmentt where confidence score differs 2. Demographic Variations across race, gender, age, etc. 3. Technical Biases: Lighting, angle, scale, or quality dependencies 4. Spurious Biases: Correlations between classes and spurious features To do that, you are provided with a library of Python functions to run experiments on each unit (inside the "System" class), given the functions provided in the "Tools" class. Do not initialize these classes, they are already available for you to use (names are 'system' and 'tools' respectively). Make sure test each hypothesis from the list and to variety of tools from the library to maximize your experimentation power. At each experiment step, write Python code that will conduct your experiment the model, using the following format: [CODE]: write Python code using your knowledge of Python and the System and Tools # class for running experiments. You must display values if you want # to see them. They will not be displayed automatically.
Biases:
Include only a single instance of experiment implementation at each step. Your code will be executed and the results will be displayed. IMPORTANT: Wait see the resulting images and confidence scores before providing an analysis.
Each time you get the output of the classifier, try summarize what inputs that achieve high confidence scores have in common (where that description is not influenced by previous hypotheses). Then, write multiple hypotheses that could explain the biases of the classifier. For example, these hypotheses could list multiple features that the classifier is more or less sensitive to.
After each experiment, wait to observe the outputs of the classifier and analyze the results from each experiment. [ANALYSIS]: Let's break this down step by step. The experiments imply that...
Based on these conclusions, write a list of initial hypotheses about the classifier biases in the format: [НYPOTHESIS LIST): Hypothesisi: <hypothesisi> ... Hypothesis, <hypothesis,>> Now, write additional experiments to test them. Test the effects of both local and global differences in images using the different tools in the library. If you unsure about the results the previous experiment, you can also rerun it, or rerun a modified version with additional tools.
Use the following format: [CODE]: ##fconduct additional experiments using the provided python library test *ALL* the hypotheses. Test different and specific aspects of each hypothesis using all of the tools in the library. Write code to run the experiment in the same format provided above. Include only single instance of experiment implementation. Wait for the code to be executed and the results to be prompted to you before continuing to the analysis step. [ANALYSIS]: ## systematically draw conclusions from each tested hypothesis. HYPOTHESIS LIST]: ## update your hypothesis list according to the image content and related confidence scores.
Continue running experiments until you prove or disprove all of your hypotheses. Only when you are confident in your hypothesis after proving multiple experiments and no additional experiments need to be executed, output your final description of the classifier bias in the following format. Output your final description of the classifier bias ONLY ONCE, when no more (CODE) is provided for additional experiments. DO NOT write any code in the round you
22

---

want to output your final bias description and bias label: [BIAS DESCRIPTION: <description of bias>
In addition, output one-sentence summary of the bias description in the following format:
[BIAS DESCRIPTIONI: <succinct one-sentence summary of bias description>
Examples
# Example [BIAS LABEL]: The classifier should classify dogs, but it has bias to prefer white dogs. ======_
Example 2 [BIAS LABEL]: The classifier should classify surfboards, but has bias prefer surfboards in beach setting.
Remember: Document all images and their confidence scores by using the display function Wait for experiments execute and results to be shown to you before drawing any conclusions or revising any hypotheses Verify findings with multiple approaches DO NOT use any functions other than the ones provided
You should maintain a systematic approach while being creative in hypothesis generation and testing. Each experiment should build on previous findings while maintaining methodological rigor.
23

---

A.2 Self-reflection instructions
You will get access to a classifier in the final layer of deep neural network that is sensitive to the presence of within photo-realistic images. Your overall task is to reveal biases in the classifier's behavior. That is, your task is to find specific features for which the classifier tends to have relatively higher confidence scores, but only covers partial aspect of the class, or specific features for which the classifier tends to have relatively lower confidence scores although 0 is still present.
Try to identify and explain the cause of biases in the network. These biases could reflect real-world biases or they could he spurious. Some bias categories include:
1. Contextual Biases: Settings/environeens where confidence score differs 2. Demographic
Biases:
Variations across race, gender, age, etc. 3. Technical Biases: Lighting, angle, scale, or quality dependencies 4. Spurious Biases: Correlations between classes and spurious features A fellow scientist has proposed the following hypothesis about this classifier's bias:
HYPOTHESIS: "BIAS"
We have conducted experiments on three sets of relevant images to help you assess this hypothe- sis:
1. [DATASET EXEMPLARS] 10 real-world images achieving highest classifier confidence score These represent the classifier's "natural" high-confidence score patterns Each includes confidence score and classification (HIGHMMDDERTTE/LLLOOO
2. [POSITIVE EXAMPLES]: 10 synthetic images designed to test the bias hypothesis Expected to trigger high confidence score based on the hypothesis Includes generation prompt, confidence score, and classification (HIGHMODERATE/OOO)))
3. INEGATIVE EXAMPLES]: 10 synthetic images hypothesis controls Expected to trigger low confidence score based on the hypothesis Includes generation prompt, confidence score, and classification (HIGHMODERATT/LLO)) ANALYSIS GUIDELINES:
1. High Confidence Score Pattern Analysis: Group all high-activating images together, regard- less of which set they' re from - Identify common visual elements across these high-confidence cases Extract shared prompt elements that consistently produce high confidence scores Note cases where similar prompts produced different confidence score levels
2. Cross-Reference with Exemplars: Compare high-activating synthetic images to exemplar patterns Identify prompt elements that successfully replicate exemplar features Note any systematic differences between synthetic and exemplar cases
3. Confidence Assessment: Evaluate consistency of coonpt-to-confiden score relationships Identify which prompt patterns reliably produce high confidence scores Note any context- dependent variations in prompt effectiveness
Based on these experimental results and analysis, generate a comprehensive list of possible biases explore with further experimentation. Present your hypotheses in the following format: Hypothesis,, <hypothesis,,>
[HYPOTHESIS LIST]: Hypothesis <hypothesis Strong hypotheses should explain both intended and unintended high-confidence score patterns while identifying reliable prompt elements that trigger the classifier.
24

---

A.3 Image generation for self-evaluation
You will get access to a classifier in the final layer deep neural network that detects the presence of 0 within photo-realistic images for autonomous driving. Your task is validate hypothesis about the classifier's 's biases. A researcher has concluded:
HYPOTHESIS: <BIAS>
Generate two sets of image prompts: 1. 10 prompts for images containing 0 that should trigger HIGH classifier confidence score based on the hypothesis 10 prompts for images containing O that should trigger LOW classifier confidence score based on the hypothesis If the hypothesis suggests multiple bias scenarios, distribute the prompts to cover all cases. Make sure each image only tests ONE bias scenario at time so different possible sources of bias do not confound.
Return your results in this exact format:
<POSITIVE PROMPTS> 1. [First high-confidence score prompt] [Second high-confidence score prompt] 10. [Tenth high-confidence score prompt] </POSITIVE: PROMPTS>
<NEGATIVE PROMPTS> [First low-confidence score prompt] [Second low-confidence
score prompt] 10. [Tenth low-confidence score prompt] </NEGATIVE PROMPTS> Remember to start the positive examples with the opening tag <POSITIVE PROMPTS> and end the positive examples with closing tag </POSITIVE PROMPTS>. Remember start the negative examples with opening tag <NEGATIVE PROMPTS> and end the negative examples with closing tag </NEGATIVE PROMPTS>.
25

---

A.4 SAIA API and implementation details
SAIA's API based on that of MAIA Rott Shaham et [2024] with few important modifications: (i) SAIA is able to define experiments in free-form code blocks, where initially MAIA had to define all its code within a function (execute command) that was then executed by the codebase. This allows SAIA to both write multiple blocks of code per experiment and to access variables defined previous experiments. (ii) SAIA now able to log and display any text/images generated during its experiments in the format of its choosing with single, flexible function (display), whereas MAIÁ had to rely on individual tools display their own results in predetermined formats. (iii) SAIA can use more recent VLLMs, particularly Claude-3.5-sonnce where the original codebase used gpt-t-Vision-previee
A.5
Agent tools
To support experimentation and interpretability amalysis, we provide Tools class with utilities for:
Text-to-image generation from prompts via pretrained diffusion models.
Prompt-based image editing to create controlled counterfactuals.
Image summarization that identifies common semantic or non-semantic features across set of images.
Region- based image descriptions that generate textual descriptions of highlighted activa- tion regions.
Exemplar retrieval for a given classifier unit, returning representative images and their scores.
A.6 Supported backbone models.
SAIA ships with a small self-contained toolkit that lets us run end-to-end vision experiments (image generation, editing, and logging) through single, uniform interface.
Text-to-Image Generation:
Flux Image Generator Labs [2024]
- -DALL-E 3 OpenAl [2023] (OpenAI API) (used for the evaluation)
Image Editing:
InstructDiffusion (Stable Diffusion variant with instruction tuning) Geng et al. [2023]
Image Description and Summarization:
GPT-40 OpenAl [2024], used via API to describe image regions and summarize visual commonalities across images.
A.7 Interface and Logging.
All generated or edited images are stored in Base64 format for transmission and display. The framework logs each experiment (prompt, image, activation, description) and supports export as an interactive HTML report for reproducibility.
Overall, the toolkit enables SAIA to generate or edit images for hypothesis testing, score them different models, analyse the outcomes, and package the entire into report, making each experiment swift, scalable, and fully reproducible. For comprehensive overview of hardware requirements, see Table 1.
A.8 Resources
All our experiments were conducted on single NVIDIA RTX 3090 (24 GB) GPU. SAIA's backbone Slaude-3.5-sonned was used through Anthropic API, and the prompt generator for self-reflection (GPT4o) and the evaluator modern in the 2AFC experiment (GPT4) were accessed through Open- AI API. An experiment with 10 rounds of hypothesis testing followed by self-refiection costs approximately $3 and takes about 10-20 minutes per round. Note that most experiments are concluded before 10 rounds, this is upper bound.
26

---

Model (Inference)
Peak |RAM (GB)
#Paarams (M)
SAM ViT-H
~7.0
632
Grounding DINO Swin-T
00.5
174
SigLIP So400m (P14/384)
~21
878
CLIP ViT-L/14 (OpenAl)
22,0
428
FairFace ResNet-34
00.0
21.8
YOLOv8-m (Ultralytics)
~0.07
25.9
Stable Diffusion 3.5 Medium (FP16)
99-10 10
2500
FLUX.dev (12B,4-5) offload)
100-11
12000
InstructPix2Pix (SD-1.5 base)
~6-7
890
Instruction Diffusion
~10
1000
RetinaFace MobileNetV3
~0.02
1.7
Average Experiment
199.5-2
Table Peak GPU memory and parameter scale of all models used at inference time.
27

---

A.9 API prompt
System:
Python containing interact with.
specific classifier
Attributes
classifier
The unit number of the classifier.
layer string
The name of the layer chare the classifier is lacated.
model_nane string
The of the vision madel.
model : nn Module
PyTorch model.
Methods
call, call_classit classifier timage_list List (torch. Tensor]) >> Taple (List [int], List [str)] Raturns the classifier confidence for image in the input inage_ list as well as the original inage (encoded into Base64 string)
def iiinit.. (self, claseifier_num: int, layer: atr, model_name str): n.w
device:
Initielizes classifier object by specifying its nunber and layer location model the classifier belongs to.
Paramoters
classifier, int
The unit number of the classifier.
layer
The name of the layer vhere the classifier is located. model_nane
The name of the vision model that the classifier is part of.
device atr
The conputational device ('epu' teuda!),
claesifier_ claseifier_aun
self layer - layer torch.. (f* (derice)" curch.cudd..c itd..ii..veillallee alsa "cpu")
(drrcccc
def call_ claseiieeee (self, (self, image_list: List [torch. Tensor]) Tuple [List (fleat),
List [str]]: nen
The function returns the classifier's 's naximum cenfidence score (in float the inages the
format)
original
inage (ancoded into Base64 string).
Parameters
image list List (torch. Tensor] The input image
Returna
Tuple (Mppe(Liitiin) (int), [ist[str]]
For each inage in inage_list returns the confidence score of the classifier on image, original inage encoded Base64 string.
28

---

Examples
>>> I teat the confidence
of the classifier for the prompt "a
standing on the grass"
prompt [*a dog standing
grass"]
>>> inage - tools. - text2inage score_list, image_
cysten.call_claeefffi (inage))
>>> for score, image in zip(score_list, image_list) tools. (image, t"Canfidence Score: (score)*)
>>> test the confidence score of the classifier for the prompt "a dog graas" >>> prompte ["a dog standing on the graes"]*5
standing
robustneas noise
images image_list list I"Confidence Score: (statiatics. >>> score_list,
text2image(proaptt cal.ea.aallllaaiaaaaaaaaaaaa (images)
www.dddlllyyy1mg.ll11 score_list))")
>>>
confidence landscape with tree and river'
of classifier
prompt
image >>> prompta ["a landscape with tree and river"]*3
different seasons:
>>> original.imagea tools. text2image (pronpta)
edits ["make
"make spring" 'nake it winter"]
>>> all-images, all-pronpta tools.edit_ adt..t..aggggarrgggalll.....
edita)
acore_list, image_list syatem.call.claseiiee (al1,
>>> for score, inage, prompt in zip cip(soreeliist inage_list, all.prompts):
tools. (inage, t"Pronpt: (prompth\nCoffidene,
{score)"
claas Tools:
n*n
A Python class containing tools to interact with the units implemented in the system class,
In order to run experimenta it.
Attributes
text2inage snaesgge__ode___emmme seee__odee___eee deneeaneee The name of the text-to inage madel. taxt2image_ : any
The loaded text-to-inage model. int
images_per.pront
Number of
generate per prompt.
path2eave : str
saving output images. threshold any
Confidence score threshold classifier analysis.
device torch device
The device (CPU/GPU) used for computations. experinent_log:
A log of all the experiments, including the code and the output from the analysis..
exemplars : Dict
dictionary containing the exemplar inages for each unit. exemplars_scorer Dict
dictionary containing the confidance for each exemplar inage. threrholls Dict
dictionary containing
results_liat List
list of the results from the classifier analysie.
29

---

dataset_exemplare (systen: (systen:: Systen)
[Tuple strl]
This experiment provides good coverage ef the behavior observed very large dataset of images and therefore represents the typical images. This function characterizes
babavior of the classifier prototypical behavior of the classifier by computing its confidence score
all images in the Imagellet dataset and returning the 15 highest confidence images that produced than in
scores
string format.
edit adit_images (self, base_insges: List (str), editing.prompts: List [str]) >> Tuple List [strt List List[str]] [str]]
function enables localized testing specific hypothesea variatione on the content of a single image affect classifier confidence scores.
Geta list of input images in Basedd encoded string fornat and list of on instructions given in the pronpt using a text-basee inage aditing madel.
correspending editing inatructions, then adita each provided image
the
function returns a list of images in Base64 encoded string format and list relevant prompts. This function is useful for testing the causality of the clasaifier controlled way, example testing how the classifier confidence
score use negative terminelogy
by changing sepect inage. IMPORTANT: ...", try
terminology "replace
with
or "change the color of ... to ...", taxt2image (prompt_ atr) List[atr]
Geta list of text prompta as an input and generates image for each proapt using
image model. list of images in Baseé4 encoded string fornat.
function returns
munnarize_inagee (salf, imaga_list: List [str])
str:
This function is useful to sunnarize the autual visual appears in set insget. It gets List of images at input and describes (eyathetiic, describe_ interibee_inne(((e harggbeeeiinneeeeeyyaeeiiiieeee
that
inw.beetiiiinng_ttttttetee List(ete),
>> str
Provides inpartial descriptions of images. Do not use this function dataset exemplars.
list imagas and generatea description of the semantic content of each of then.
The functiou is blind to the current hypotheses list and therefere provides unbiased description of the visual display (self, earge: Union Insge, Image)):
your vay dieplaying experiment this on rosults/variables that you wish to viey in ordar to view then.
This
call
def init (self, path2save: str, device: str, DatasetExeuplare: DatasetExemplare None, pageetlprr.rropptll, aoxtiingge_mode__noo
Initializes the object.
Parametere
path2save str
Path saving output images.
device str
computational DatasetExemplara object
('epu' 'euda
an object from the claes DatasetExenplare
30

---

Number of images to generate prompt. text2inage_ nane
The of the text inzagee- model.
daf dataset cra aystem: Systen) List (Tatllutpetfloat, str]]
nethod images
Inagellet dataset produce highest confidence scores for specific classifier.
returns
confidence scores that used generate these confidence
correspending exemplar images
This experiment is parformed on real images and will provide a good approxination the classifier behavior.
Parameters
systen Syatem
The systen representing the specific classifier and layer vithin the neural network.
systen
'layer' and 'classifier_ attributee, dataset_exenple fuaction
return exemplar confidence scores images specific classifier.
Raturns
For each exemplar inage, stores tuple containing tuo elements: element classifier
specified
The second element is the exemplar inages (aa Baae64 encoded strings) correaponding the confidence
Example
exemplar_date
dxtpaaet_axemplars (aystem)
>>> for acore, image in exemplar sxemplar_dtta:
dools.dieple (image, I d'Cafiidence Score: (score)*)
n..
def edit_inages (self,
base_images List[atr) adsting.pronpta: [atr]] [apla [atr]], 11:
Generates provided base inages, then edits each base image with corresponding editing prompt. Accepts either text prompte or Base64 eacoded strings as
the inages.
The function returns list containing lists inagea (original and edited
interleaved) in Base64 encoded string format, and a list of the relevant string and editing prompt, interleaved).
prompts (original
base inages List[str]
A list of inages Base6d encoded strings. These edited by the prompts in aditing.pronpts. oditing-prompts [ist[atr]
A list of instructions for how to edit the base inages derived from inase_
same length basgeinages'.
31

---

Tuple (List [List List[str]]
all_images: A list where elements alternate between:
list Base6d strings for the original image from a source. Base64 strings for the aditad image from that
Example: [lorigi_ingl origi_ing2), [editi,ingl, editl_ing2), imgg),
all.prompta: [edit2_ingl].
alements
The original
string (text prompt Base64) used.
The editing prompt used.
Exanple: [sourcel,
aource2, ....
The order in alll,nages corresponde to the order in 'sll.prompta',
Raises
ValueBrror
If the langtha of "base_inages" and 'aditing-prompta" are not aqual.
Examples
>>> # test the confidence score of the classifier for the prompt "a landscape with tree and river"
# the
with different
>>> prompta ["a landscape with a tree and a river"]*3
originsl inagge
text2inage (pronpts) (proopa)
>>> edits - [*maze it autumn"," "make it spring", "nake it vinter")
hat...ddd..aagcssrggggll....... >>> acore_liat, image_list ayaten. cal.n..cll.classffaar (all.,inga
all_images, all.prompts
edits)
>>> for score, image, prompt in zip(score, inage_list, all.prompts): tools.display (image, t"Prompt: prooptt\ncooriddene Score: {scora)"
>>> # test the confidence score of the classifier on the highest scoring dataset exemplar
>>> # under different conditions
>>> exemplar_date tools dataset (syeten)
tigbett.acoring.exepp
exemplar_data [0] [1]
>>> edits - ['make it night", 'make it daytime", "nake it snoving")
all_images, all.proupts edols.edit,inages (baghess_scriing len(adits), edits)
acore_liat, image_list
aetta..call.laasttfiar (allg
>>> for score, inage, prompt in 2iporeee__liet inage_lies all.pronpts);
>>>
tools. (inage, f"Prompt: [proupt]\aConfiden Score: (score)"
daf text2image (self, prompt_liat: List [str]] >> (Liat
Takes list of text prompts and generates inages_per_prout images for
each using
model. The function inages_ inages_por_pte per_pronpt images each proupt.
list list of
Parametere
prompt_lis List [str]
à list
pronpta image generation.
List [Lir] A list of list of images_por.pprops images Base64 encoded string fornat for input prompts.
32

---

Examples
555 | Generate images from list of prompts
>>> prompt.list [sAua dog standing the grassali,
>>>
aAlla dog sitting on couchait,
>>5
alMa dog running through fieldsli)
images tools. text2image (prentt..ap >55 score_list, image_ list
calten.call.claseiiee (inages)
559 for score, image in zip(scoro_list, image_liet).
tools. display (image, f ""Confidence Score: (scora)*)
def display (self, earge: Union (str, Image Image]):
Diepleys series of inages and/or text the chat, similar Jupyter notebook.
Paraueters
arze Union Image Inage] The content to be displayed in the chat. Can be multiple stringe or Image objects.
Nates
Displays directly
Exemple
>>>
Display single image
dog standing grass"] >>> insges toole. terieeigee22ii
prompt
score_list. image_list cysten..all_claeeiitii (inages)
>>> for score, image in zip(score__ist imago_list): dools.
3>>
f"Confidence Score: (score)")
>>> Diaplay a single image fron a 1ist
prompte ["a deg standing
grass"]a5
>>> images tertingeet2iineepprron
score_list, image_ list system. cppten..alllccaaacffee (inages)
>>> daola.diaplay (inage_ [0], f" ""Confidenca Score: (statiatics. nean score_list))")
>>> # Diaplay 1ist of inages
>>> prompt_list [aAlla dog standing on the grassšIi,
alNa dog sitting on couchali, alNa dog running through a fieldshi)
>>>
tools. text22imeeerooopt..sss >>> acore_list, image_ list score, image zip (icoree..iit, image_list):
(inages)
>>> tools. tools.dispia display (inage, ["Confidence Score: (score)*)
nen
def sunnarize intieee (self, image_ List [str]) >> str:
Gata 1ist of imagas and deacribea shat ia common to all of them.
Parameters
image_list list
4 1ist of imagas Basa64 encaded string
33

---

Returns
A string with descriptions zhat is common to all the images.
Example
>>> # Sumnarize classifier's dataset exemplars
exemplars [axemplar for - examplar
dat.sett (syaten
}] # Get exemplars
>>> sunnarization dioplay (aumaarization)
sumnarize images (exemplars)
def imacrsbe (self, inage_list: List [str],
[str]) >> str:
Ganerates textual descriptions for list images, focusing specifically on highlighted regiona. The final descriptions are concatenated returned single string, each description associated with the corresponding image title.
Paramoters
image_list [ist(
A 1iat of imagea in Base64 encoded string format.
image_ -tttl List[etr)
A list of titles for each image in the image_list.
Raturns
A concatenated string of descriptions for each inage, where each description
is associated with the images title and focuses on the highlighted regions
in the image.
Example
prompt_
dog standing couch",
grass",
>>>
"a dog sitting
dog running through field"]
>>> inages toole. text2ingge (pr
score_list, image list systen.call. cla.eelleeiiirrr (images)
(smage_ll (inage_list, pronpt_list) >>> tools. displyy (descriptions)
descriptions
34

---

B Benchmark models
B.1 Systems specification
We provide below the full list of objects and categories used for our benchmark.
Gender
Age
Female
Male
Young
Old
Apron ("kitchen")
Tie
Laptop
Glasses
2
Umbrella
Beer
Cell phone
Book
3
Scarf
Skateboard
Skateboard
Hat
Cat
Suit
Bicycle
Tie
Book
Laptop
Teddy bear
Wine glass
6
Handbag
motorcycle
7
Wine glass
surfboard
Hair drier
Frisbee
Teddy bear
Guitar
Dress
Cap
Table 2: Feature categories and corresponding objects associated with gender and age stereotypes.
Color
Material
Red
Green
Blue
Black
White
Wooden
Ceramic
Bus
Bus
Bus
Bus
Bus
Table
Vase
Umbrella
Umbrella
Umbrella
Umbrella
Umbrella Tie
Chair
Bowl
Tic
Tic
Tic
Tie
Bench
Cup
Kite
Kite
Kite
Kite
Kite
Frisbee
Frisbee
Frisbee
Frisbee
Frisbee
Table 3: Feature categories and corresponding objects associated with color and material properties.
Setting
State
Kitchen
Living Room
Office
Wilderness
City
Beach
Misc.
Table
Table
Table
Bird
Bird
Bird
Airplane (Flying)
Chair
Chair
Chair
Car
Car
Car
Bicycle (Ridden)
Cat
Cat
Cat
Dog
Dog
Dog
Clock (Analog)
Dog
Dog
Dog
Horse
Horse
Horse
Keyboard (Typing)
5
Vase
Vase
Vase
Bench
Bench
Bench
Kite (Flying)
6
Wine glass
Wine glass
Wine glass
Umbrella (Open)
Vase (With flowers)
Table 4: Feature categories and corresponding objects associated with different settings and states.
35

---

Gender
Age
Female
Male
Young
Old
1
Tie
Apron
Glasses
Laptop
2
Beer
Umbrella
Book
Cell phone
Skateboard
Scarf
Hat
Skateboard
Suit
Cat
Tie
Bicycle
5
Laptop
Book
Wine glass
Teddy bear
6
motorcycle
Handbag
surfboard
Wine glass
Frishee
Hair drier
9
Guitar
Teddy bear
10
Cap
Dress
Table 5: Feature categories and corresponding objects with flipped gender and age associations.
B.2 Dependency strength versus discount factor
We investigate how discount factor E [0, 1] of our synthetic model attenuates the score of the synthetic model henever a predefined attribute condition is not satisfied. Figure 8 shows the mean classification accuracy for six attribute groups: AGE, COLOR, GENDER, SETTING, SIZE, and STATE.
No discount (a=1.0). Baseline accuracies for all groups remain high (2 0.73).
Small discount (0 < < 0.3). Accuracy drops slightly (under five percentage points), indicating that mild penalties leave decisions largely intact.
Medium discount (0.3 < a S 0.5). Accuracy decreases almost linearlya&fGGNNEER is most robust, while SIZE and SETTING degrade faster.
HighâA discount (a: 0.5). A sharp collapse occurs; COL.OR and SIZE fall below 0.20 at a 0.7,, and all groups eventually saturate between 0.05 and 0.25.
1.0
Color
Material
State
0.8
Setting
Age
Gender
Acmirayy
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Discount factor
Figure 8: Mean accuracy versus discount factor for six attribute groups.
36

---

C
Results
C.1
Additional Analysis
Figures 9a and 9b report the average self-evaluation scores across ten rounds of self-reflection. under discount factors of a=0.5 and a=0.9, respectively. We observe consistent upward trend in performance across all six attribute categoriesåAfGEnDE This trend holds across both mild and severe reliance scenarios, suggesting that the iterative refinement process is effective improving the quality of SAIA's hypotheses and explanations. While early rounds exhibit fluctuations (especially at a=0.5), later rounds show stabilization and convergence toward higher evaluation scores. The improvement is more pronounced under the lighter discounting condition (o=0..), where SAIA starts from lower scores but achieves comparable gain. It also noticable that for disambiguate attribute dependency (a=0.5) more self-reflection rounds are necessary, whereas with (a=0.9) saturation is achieved earlier. This demonstrates that self-reflection enables SAIA to recover explanatory accuracy even in challenging scenarios.
AGE, COLOR, STATE, and SETTING.
0.85
0.90
Self-Evaluation Score
Self-Evaluation Score
0.80
0.85
0.75
0.80
gender
gender
age
0.75
age
color
0.65
state
0.70
state
setting
setting
0.60
6
10
2
4
10
Self Reflection Round
Self Reflection Round
(a) 0.5
(b) 0.9
Figure 9: Average self-evaluation scores across ten rounds of self-reflection under different discount factors broken down by reliance attribute category.
C.2 Multiple feature reliances
For the models that rely on the presence of both attributes simultancously (Table 7), found that SAIA is able to accurately uncover both reliances for four out of the five systems, while MAIA was not able detect both reliances correctly in any of the systems. For the models that rely on the presence of at least one of the two attributes (Table 8, SAIA was only able to recover both reliances for single model, while MAIA outperforms MAIA on multi-attribute reliance systems overall, both are less effective that detecting notice that for these models, during
again not able detect both reliances for any. While SAIA
multi-attribute reliances in this second setting. Qualitatively, the hypothesis testing phase, it is more challenging for the agent to isolate the reliance attributes, sometimes identifying other co-occuring attributes instead. For example, for the bench detector that relies on benches that are wooden OR in beach settings, SAIA concludes that the classifier is biased toward aAJattatiiional park benches with backrests natural settings,äAl presumably because traditional park benches are often wooden.
C.3 Text-to-image tool robustness
Text-to-image (T2I) models can carry societal and representational biases. This is general limitation of agentic interpretability methods that rely T2I models for generating stimuli. To quantify SAIA's robustness to T2I errors, we conducted two experiments:
Random failure: In 50% of T2I calls, we replaced SAIA's prompt with empty string, resulting the model generating unrelated content that ignored the intended experimental manipulation. We found that in 80% of these corrupted trials, SAIA successfully noticed the issue, either during the hypothesis-tesiee testing stage or self-reflection, and responded by revising approach or ignoring the incorrect images.
37

---

Target Concept
Attribute 1
Attribute
bus
Color: Red
Setting: City
tie
Gender: Male
Setting: Office
bench
Material: Wooden
Setting: Beach
State: With flowers
Setting: Home
apron
Gender: Female
Setting: Kitchen
Table 6: Target concept and attribute reliances pairs for the systems with multiple attribute reliances. Each system that relies on the presence of both attributes simultancously will detect the Target Concept with high confidence if both Attribute AND Attribute are present in the image. Each system that relies on the presence of at least one attribute will detect the Target Concept with high confidence if either Attribute OR Attribute are present in the image.
No Reliances Detected
One Reliance Detected
Both Reliances Detected
MAIA
20%
80%
0%
SAIA
0%
20%
80%
Table 7: MAIA and SAIA success rates on the multi-attribute reliant systems that rely on the presence of both attributes simultaneously.
We attack the prompts generated by SAIA in experiments on a tie detector system that relies on the presence of a man. We attack the T21 model by systematically replacing instances of the phrase åÄlJa personåAI with åÄlJa manâÄí x% of the time for x [0, 25, 50, 75, 100). and aÄlJa womanåAf the remaining (100 - x)% of the time to simulate a controlled gender bias in the T21. We found that SAIA was robust to attack and able to detect the correct reliance in all simulated bias ratios. Interestingly, SAIA required more iterations when the T2I model was biased toward generating images of men (i.e. when the bias of the T2I matched the model's feature reliance). We noticed this phenomenon the base T2I model as well, which almost always generates a man when prompted with åÄlJ person wearing a tie.âÄI This is another motivating factor for including the study on counterfactual demographic attribute reliant systems (e.g. tie detector that relies on the presence of woman AåAA such systems are out of distribution for not only the multimodal LLM backbone but tools like the T2I as well.
Injected bias:
Interestingly, when SAIA notices such unaligned behavior, it usually uses a different experimental design to get the intended behavior (e.g. if the T2I tool doesn't follow the prompt correctly, it uses the editing tool the edit one of the dataset exemplars to achieve the desired stimuli) or ignores the incorrect images and focuses its analysis on the successful generations.
C.4 Diversity of Hypotheses
We measured the diversity of SAIA's hypotheses by computing the average pairwise similarity between hypotheses generated across all rounds of SAIA for each synthetic benchmark model. Specifically:
For each system, we parsed all hypotheses produced across rounds.
We computed the average cosine similarity between the CLIP text embedding of all pairs of hypotheses for that system.
We then averaged these values across all systems to obtain overall similarity score.
To contextualize this result, we constructed a baseline using the ground-truth descriptions of all 130 synthetic benchmark models (which vary in both object class and attribute dependence). We computed the average pairwise cosine similarity between these ground-truth descriptions and between SAIA's hypotheses and found that the similarity score for SATA's hypotheses was 0.073 compared to 0.094 for the baseline. The lower similarity score for SAIA's hypotheses indicates greater diversity than the baseline, suggesting that SAIA explores wide range of explanations.
38

---

No Reliances Defected
One Reliance Detected
Both Reliances Detected
MAIA
60%
40%
0%
SAIA
60%
20%
20%
Table 8: MAIA and SAIA success rates on the multi-attribute reliant systems that rely on the presence of at least attribute.
C.5 Validation of CLIP and YOLOv8 Reliances on Real Images
We used SAIA's discovered explanations (e.g., reliance classroom settings to detect the concept of teacher or pose to detect the concept of biker) to select real images that either contained or lacked the relevant attributes. We then measured the model's responses and found that the same attribute dependencies were reflected in all the real-world image scores, providing strong support for the alignment between synthetic and real domains. The
CLIP Concepts
YOLO Concepts
teacher
scientist
wine glass
biker
pedestrian
Generated images
0.22 0.17
0.22 / 0.16
0.27 0.21
0.35 0.32
0.54 / 0.33
Real images
0.25 0.23
0.21 0.17
0.21 0.20
0.18 / 0.03
0.49 / 0.07
Table 9: Raw average positive/negative exemplar score pairs for target CLIP and YOLO concepts for which SAIA discovered attribute reliances, evaluated on both generated and real images.
Teacher
Sclentst
Hight stomn
0.29
10:24
0.23
0:22
Lower Scoring
Pedestrian
Higher Stoting
0.70
0.52
0:41
0.83
0.78
Lower Scoring
D90
Figure 10: Real images that elicit high (green) and low (red) scores from the (a) CLIP and (b) YOLO classifiers across different object categories. Each triplet shows the top and bottom scoring examples per class for CLIP and YOLOv8.
39


Reviews:
[{"review_id": "5kMIpdptEy", "reviewer": "Reviewer_wjA6", "summary": "The authors introduce an agentic framework for identifying visual attribute dependencies in trained vision models. The core of the contribution is a \"self-reflective agent,\" implemented with a multimodal LLM, that systematically generates and tests hypotheses about the unintended visual attributes that a model may rely on. They also introduce a new benchmark of 130 models with synthetically injected, known attribute dependencies, which they use to evaluate their framework.", "questions": "See weaknesses.\n\nMinor: What the computational cost (e.g., API calls, time) for analysing a single concept in a model?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nFraming model interpretability as an iterative, scientific discovery process driven by a self-reflective agent is novel and potentially impactful for practitioners. \n\nThe introduced benchmark is highly useful for the field and its construction is well described. \n\nExperiments are well designed and the framework is compared to several reasonable benchmarks.\n\n\nWeaknesses\n\nThe framework is dependent on a TTI model to generate images for hypothesis testing and self-evaluation. This introduces a major potential confounder that the paper does not adequately address. TTI models are themselves trained on large web-scale datasets and often use CLIP-like embeddings; thus, they encode their own strong societal and representational biases. For example, when the agent tests a hypothesis like \"the model relies on corporate settings for the concept *suit*,\" it generates images with and without this attribute. However, when it prompts the TTI model for \"a person in a suit *not* in a corporate setting,\" the TTI model might, due to its own biases, change other attributes—it might generate a woman instead of a man, or change the style of the suit. The agent might then observe a drop in the subject model's score and incorrectly conclude that the \"corporate setting\" was the key attribute, when in fact the reliance was on the person's gender...\n\n\nThe paper frames the discovered dependencies as \"unintended\" or \"vulnerabilities\". However, some of the examples seem to be strong, statistically valid features that are part of a concept's representation. For example, the finding that CLIP's vision encoder recognizes \"teachers\" based on \"classroom backgrounds\" is not necessarily a spurious shortcut. In many contexts, the background is a defining feature. How else would a model distinguish a teacher from  another person in a professional setting? The framework currently lacks a mechanism to differentiate between a harmful shortcut (e.g., reliance on gender for a job-related concept) and a benign, contextually relevant feature. This makes the practical significance of some findings unclear.", "was_revised": true, "final_justification": "Based on the detailed author response and the extra experiments that were supplied and I have raised my score. Below I explain how the rebuttal mitigates my earlier concerns and note the points that are still only partly resolved.\n\nDependence on the T2I Backbone My principal worry was that bias or drift in the text-to-image (T2I) generator could mislead the agent and yield false positives. The new ablation (random-empty prompts) and the controlled gender-swap attack are helpful. In particular, the report that the agent still recovers the correct reliance in all injected-bias conditions—even if it needs more iterations—suggests a degree of robustness I did not anticipate. That said, the experiments cover just one attribute class (gender) and one corruption type.\n\nClarification of the Self-Evaluation Protocol The cap-wearing example and the explanation that the system generates two diverse prompt pools, rather than a single minimal pair, remove most of the ambiguity in the original manuscript. The three documented failure-handling behaviors further clarify what happens when generation and text become misaligned.\n\n“Harmful vs. Benign” Reliances The authors now acknowledge that the agent surfaces dependencies without judging their desirability and that task-level judgment is left to practitioners. This reframing should be spelled out early in the paper, especially for readers who might otherwise assume the tool is a bias detector by design.\n\nOutstanding Issues\n\n- The new robustness study is currently limited in scope; expanding to other attributes (e.g., background, material) would strengthen the claim.\n- A concise comparison with MAIA’s bias analysis would help further clarify the contribution of the paper.", "rebuttal": "We thank the reviewer for their meaningful comments. We were glad to see that, like all other reviewers, reviewer wjA6 found our benchmark and synthetic dataset to be highly useful, and the experiments to be well designed. Below, we address their comments. To give a short summary of the main points: \n\n(1) We clarify the usage of T2I models in the self-evaluation phase and run additional quantitative evaluations to test the agent’s sensitivity to T2I generation failures and biases and find it to be mostly robust to these failure modes. This aligns with the behavior of the agent we observed during experimentation (see description below). We will expand this evaluation and include it in the appendix of the paper.\n\n(2) We clarify the definition of the task and discuss whether all found reliance are harmful—we believe that this depends on the downstream tasks. \nPlease see full replies below. \n_____\n\n**T2I models have their own biases**\n\nWe agree that T2I models can carry societal and representational biases. This is indeed a general limitation of agentic interpretability methods that rely on T2I models for generating stimuli (and was noted and evaluated in the MAIA paper [1], see their section 4.4).  However, a key distinction in our framework is the inclusion of a self-reflection loop, which we find makes the system more robust to such artifacts.\n\nFor example, in the provided supplemental experiment folder “male_cap_0.5”, we provide the full trace of an experiment on a *cap* detection model with an injected bias towards men wearing caps. We observe that the T2I model exhibits gender bias: despite gender-neutral prompts (e.g. “a person wearing a black baseball cap”), all generated images in the first hypothesis-testing round (“round0.html”) depict men wearing caps. As a result, the agent is initially unable to draw any conclusions about potential gender dependencies through hypothesis-testing alone. During self-reflection in round3.html, the agent is evaluating the hypothesis: *“The classifier demonstrates a strong contextual bias toward caps being actively worn on human heads…”* It does so by passing the conclusion to a separate LLM that generates two diverse sets of prompts predicted to elicit high (e.g. *“A young man wearing a red baseball cap while smiling at the camera”*, *“Woman jogger wearing a white sports cap during her morning run”*, *“Teen boy wearing his baseball cap backwards at a skate park”*) and low (e.g. *“New baseball cap displayed on store shelf”*, *“Collection of caps hanging on wall hooks”*, *“Baseball cap floating in swimming pool”*) detection scores, respectively. As a result of the diverse evaluation prompts, the agent is able to recognize a discrepancy in its previous conclusion and formulates a new hypothesis: *“The classifier exhibits a gender and age bias, favoring adult males wearing baseball caps in athletic or casual contexts.”* This insight leads the agent to revise its prompts and explicitly query for images of women wearing caps, ultimately uncovering the correct dependency, despite the T2I model's initial bias.\n\nIn several other cases, the agent was able to detect inconsistencies between its generated prompts and the resulting images. Interestingly, when the agent notices such unaligned behavior, it usually uses a different experimental design to get the intended behavior (e.g. if the T2I tool doesn’t follow the prompt correctly, it uses the editing tool the edit one of the dataset exemplars to achieve the desired stimuli) or ignores the incorrect images and focuses its analysis on the successful generations. \n\nTo quantify the agent’s robustness to T2I errors, we conducted two experiments:\n\n- *Random empty prompt:* in 50% of T2I calls, we replaced the agent’s prompt with an empty string, resulting in the model generating unrelated content that ignored the intended experimental manipulation. We found that in 80% of these corrupted trials, the agent successfully noticed the issue, either during the hypothesis-testing stage or self-reflection, and responded by revising its approach or ignoring the incorrect images. While this is a simple baseline, it gives an approximation of the agent’s sensitivity to breakdowns in the generation pipeline.\n\n- *Injected bias:* We attack the prompts generated by the agent in experiments on a tie detector system that relies on the presence of a man. We attack the T2I model by systematically replacing instances of the phrase “a person” with “a man” x% of the time for  x = [0, 25, 50, 75, 100], and “a woman” the remaining (100-x)% of the time to simulate a controlled gender bias in the T2I. We found that the agent was robust to attack and able to detect the correct reliance in all simulated bias ratios. Interestingly, the agent required more iterations when the T2I model was biased toward generating images of men (i.e. when the bias of the T2I matched the model’s feature reliance). We noticed this phenomenon in the base T2I model as well, which almost always generates a man when prompted with “a person wearing a tie.” This is another motivating factor for including the study on counterfactual demographic attribute reliant systems (e.g. a tie detector that relies on the presence of a woman)—such systems are out of distribution for not only the multimodal LLM backbone but tools like the T2I as well. This is a first quantitative evaluation that we will expand for the final version of the paper (include more attacks and more systems), and we will report the overall sensitivity of the agent to controlled biases in T2I.\n\n\n[1] Shaham et al. \"A multimodal automated interpretability agent.\" ICML 2024.\n\n\n**Image generation in the self-evaluation might be inaccurate**\n\nTo clarify, the self-evaluation protocol does *not* operate by prompting the T2I model with a minimal pair of prompts (e.g. *“a cap on a human head”* vs. *“a cap not on a human head”*). Rather, as described in the previous section, the agent passes its current conclusion (e.g., *“The classifier demonstrates a strong contextual bias toward caps being actively worn on human heads”*) to a separate LLM, which then generates two diverse sets of prompts predicted to elicit high and low detection scores, respectively (e.g. predicted high detection score prompts: *“A young man wearing a red baseball cap while smiling at the camera”*, *“Woman jogger wearing a white sports cap during her morning run”*, *“Teen boy wearing his baseball cap backwards at a skate park”*, and low detection score prompts: *“New baseball cap displayed on store shelf”*, *“Collection of caps hanging on wall hooks”*, *“Baseball cap floating in swimming pool”*). We thank the reviewer for highlighting this unclear description of the self-evaluation protocol and will incorporate a more comprehensive explanation in the revision.\n\n\nWe found this strategy to be more robust compared to naive prompting with direct negations (which we indeed found prone to semantic and visual drift). It emphasizes attribute-controlled diversity, where each prompt in the high or low group keeps the core attribute constant (e.g., presence or absence of humans) while allowing other visual factors to vary. \n\n\nWhen mismatches between textual prompts and generated images is the self evaluation stage do occur, we found three main behaviors:\n\n- The agent is able to detect the misalignment and disregard the artifacted sample (as explained in the previous reply)\n\n- The agent was not able to detect the misalignment, however, the T2I artifact yields an image that doesn’t match the predicted behavior (because of the artifacts), this is typically reflected in a lower predictiveness score. The agent uses this feedback as a signal to reinitiate hypothesis testing (as shown in Fig. 2)\n\n- The agent was not able to detect the misalignment, but the T2I artifact doesn’t affect the related attribute and therefore doesn’t harm the predicted behavior and doesn’t affect the experiments. \n\nWe agree that misalignment between textual prompts and generated images in T2I is an important concern. We will add this discussion and the empirical evaluation to the appendix of the paper.\n\n**Found reliance is not necessarily spurious**\n\nWe agree. Our framework does not attempt to judge whether a discovered reliance is inherently harmful or contextually appropriate, nor is this distinction communicated to the agent. Rather, the goal is to surface potential dependencies that might not have been previously anticipated, enabling practitioners to make informed judgments based on their specific downstream use case. In some applications, reliance on context (e.g., background for \"teacher\") may be a valid, even desirable, part of the concept's representation. In others (e.g., medical imaging or fairness-sensitive applications), such dependencies could lead to unwanted generalization or bias. Notably, the agent sometimes concludes that no strong attribute reliance is present. We will make sure to clarify this in the definition of the problem setting in the intro and the rest of the paper. \n\n**Computational cost**\n\nAn experiment with 10 hypothesis testing + self-reflection rounds costs approximately $3, and each hypothesis testing + self reflection round takes about 10-20 minutes (Note that most experiments are concluded before 10 rounds, so this is an upper bound)."}, {"review_id": "yDAEdUF5jc", "reviewer": "Reviewer_Cd4M", "summary": "This paper proposes an automated agent for visual attribute reliance. The idea is to use an LLM agent that can propose hypothesis, tests them and iteratively refines them based on observations. One key contribution of the method is the self-reflection stage in which the agent uses a self-evaluation tool to score it's explanation. The authors also construct a synthetic dataset to measure the effectiveness of their proposed approach.", "questions": "1. Is there any error analysis that is done to show how sensitive or robust the method is when text to image models fail to generate the presence or absence of some attributes. How much error can we tolerate.\n2. T2I models can suffer when generating some rare classes, like rare breeds of dog etc. How can the self-reflective agent work in these settings.\n3. Can the LLMs come up with diverse set of candidate hypothesis? Is there a way to quantify this?", "limitations": "I feel the contribution over MAIA is limited. The over-reliance on text to image generators is a bit concerning when they themselves can have biases or failure modes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The use of self-reflection agent is quite interesting. It is a natural extension to MAIA style approaches.\n2. The proposed synthetic dataset can be useful. The method for constructing the dataset makes sense.\n\nWeakness:\n1. Main limitation is that the contribution is limited. I feel the use of agents for visual explanations have been proposed in works like MAIA. This work just adds a self reflecting agent on top of it.\n2. The self-reflecting agent assumes text to image model is good. But text to image model can have biases in itself. They might not be able to generate the presence or negation of some attributes. T2I models have especially been shown to be poor at negations. Given this, it is hard to conclusively tell how good the self-reflection stage is.\n3. There are a lot of components cascaded in this system. So, if one stage breaks, the entire system can fail. This concerns me.\n4. The diversity of generations from T2I models can also be low. Is that a concern?\n5. The synthetic dataset is constructed with simple object classes and attributes. What about the long tail category objects?", "was_revised": true, "final_justification": "The reviewer clarified my concern on over-reliance of T2I models. As mentioned by the reviewer, the self-reflective agent can help mitigate such biases due to an external LLM judge. Since my misunderstanding is clarified from the rebuttal, I increase the score to borderline accept.", "rebuttal": "We thank the reviewer for their meaningful comments! We were glad to see that, like all other reviewers, the reviewer Cd4M found our benchmark and synthetic dataset to be useful, and the self-reflection to be a meaningful contribution. Below, we address their comments. To give a short summary of the main points: \n\n- We clarify the contribution over MAIA\n- We clarify the usage of T2I models in the self-evaluation phase. Instead of prompting a T2I with an attribute and its negation, we use a separate LLM to generate diverse prompts that we find to be much more robust to inaccurate generations from the T2I tool.\n- We provide an initial evaluation of the ability of the agent to recover from failures of tools\n- We conducted additional experiments on ResNet-152 for classifying rare classes in ImageNet \n- We provide a quantitative evaluation of the diversity of the hypotheses \n___\n**Limited contribution compared to MAIA**\n\nOur work builds upon MAIA by introducing a *self-reflective loop*, which we believe is a significant step forward, both in terms of methodological design and empirical performance. \n\nWhile MAIA concludes the experiments after a single hypothesis-testing pass, our agent critically evaluates the accuracy of its own conclusions and initiates further experimentation when inconsistencies arise. This enables the agent to revise or refine hypotheses, correct misattributions, and uncover more robust explanations. For example, as shown in Fig. 2, the agent initially concludes that a model relies on *“corporate settings”* to detect suits, but revises this to *“male-presenting figures wearing suits”* after it measures behavior in the self-evaluation phase didn’t follow the predicted one. The reviewer can find many more results like this in the provided html files in the supplementary material.\nQuantitatively, this reflective capability leads to substantial gains: across our benchmark (Fig. 5a), self-reflection improves predictiveness scores significantly beyond MAIA, and even outperforms ensembles of MAIA-like agents. This suggests that the improvement yields systematic advantages in model auditing.\n\n**T2I tools might be biased**\n\nWe agree that T2I models can carry societal and representational biases. This is indeed a general limitation of agentic interpretability methods that rely on T2I models for generating stimuli (and was noted and evaluated in the MAIA paper, see their section 4.4).  However, a key distinction in our framework is the inclusion of a self-reflection loop, which we find makes the system more robust to such artifacts.\n\nFor example, in the provided supplemental experiment folder “male_cap_0.5”, we provide the full trace of an experiment on a *cap* detection model with an injected bias towards men wearing caps. We observe that the T2I model exhibits gender bias: despite gender-neutral prompts (e.g. “a person wearing a black baseball cap”), all generated images in the first hypothesis-testing round (“round0.html”) depict men wearing caps. As a result, the agent is initially unable to draw any conclusions about potential gender dependencies through hypothesis-testing alone. During self-reflection in round3.html, the agent is evaluating the hypothesis: *“The classifier demonstrates a strong contextual bias toward caps being actively worn on human heads…”* It does so by passing the conclusion to a separate LLM that generates two diverse sets of prompts predicted to elicit high (e.g. *“A young man wearing a red baseball cap while smiling at the camera”*, *“Woman jogger wearing a white sports cap during her morning run”*, *“Teen boy wearing his baseball cap backwards at a skate park”*) and low (e.g. *“New baseball cap displayed on store shelf”*, *“Collection of caps hanging on wall hooks”*, *“Baseball cap floating in swimming pool”*) detection scores, respectively. As a result of the diverse evaluation prompts, the agent is able to recognize a discrepancy in its previous conclusion and formulates a new hypothesis: *“The classifier exhibits a gender and age bias, favoring adult males wearing baseball caps in athletic or casual contexts.”* This insight leads the agent to revise its prompts and explicitly query for images of women wearing caps, ultimately uncovering the correct dependency, despite the T2I model's initial bias.\n\nTo quantify the agent’s robustness to T2I biases, we conducted an experiment to deliberately inject a bias to the T2I model and show that the agent was able to recover the correct reliance despite it across different levels of injected biases. We invite the reviewer to refer to the reply to reviewer wjA6 above **\"T2I models have their own biases\"** for the experimental setting and results.\n\n**The T2I model might not be able to generate the presence or negation of some attributes**\n\nTo clarify, the self-evaluation protocol does *not* operate by prompting the T2I model with a minimal pair of prompts (e.g. *“a cap on a human head”* vs. *“a cap not on a human head”*). Rather, as described in the previous section, the agent passes its current conclusion to a separate LLM, which then generates two diverse sets of prompts predicted to elicit high and low detection scores, respectively. We thank the reviewer for highlighting this unclear description of the self-evaluation protocol and will incorporate a more comprehensive explanation in the revision.\n\nWe found this strategy to be more robust compared to naive prompting with direct negations (which we indeed found prone to semantic and visual drift). We refer the reviewer to the reply to reviewer wjA6 above **\"Image generation in the self-evaluation might be inaccurate\"** for more discussion about the effect of T2I failure on self-reflection.\n\n**Diversity of T2I**\n\nPlease note that we never generate multiple samples from the same prompt. Instead, in the self-reflection stage we ask a language model to generate diverse sets of prompts corresponding to the presence or absence of the detected attribute (as described in Section 3.2). Each prompt is sampled once, so the resulting image set reflects conceptual diversity across different instantiations of the attribute, rather than relying on stochastic sampling from a fixed prompt. We invite the reviewer to explore the submitted html files to judge the diversity of generated images.\n\n**Rare Classes**\n\nWhen investigating attribute dependencies in ResNet-152 for classifying rare ImageNet classes, we found that the agent was able to effectively reason about certain classes, e.g. loupe, and tools like the T2I generated reliable outputs. However, for other classes, such as the pickelhaube, the T2I was only able to reliably generate pickelhaubes in isolation. When prompted to generate a pickelhaube worn by a soldier, the T2I would generate a regular army helmet, leading to low scores from the model and the agent to conclude that the model has a bias against pickelhaubes worn by humans. Thus, in rare classes, it is true that it may be difficult for the agent to disentangle model attribute dependencies from nuanced failures of the T2I—this is a limitation of the framework that we will discuss further in the final version of the paper. In cases where the subject model operates on uncommon concepts, the users might adjust the tools for the agent.\n\n**Components cascade can cause the framework to break:**\n\nWe agree that cascading multiple components might introduce the risk of compounding errors, which is a known challenge in agentic frameworks. However, our self-reflective agent is better at addressing this concern. The self-evaluation protocol systematically compares predicted and actual model behavior, allowing the agent to detect inconsistencies that may result from failures in earlier stages. When such discrepancies are found (whether because of a misinterpretation of the agent or a failure of one of the tools), the agent re-enters the hypothesis-testing loop and attempts to revise or refine its explanation. This feedback loop improves the system’s robustness by preventing premature or faulty conclusions from being returned to the user. \n\nTo quantify the agent’s robustness to failure of one of its tool, we conducted the following experiment: We intentionally elicit failures in the T2I tool such that in 50% of T2I calls, we replaced the agent’s prompt with an empty string, resulting in the model generating unrelated content that ignored the intended experimental manipulation. We found that in 80% of these corrupted trials, the agent successfully noticed the issue, either during the hypothesis-testing stage or self-reflection, and responded by revising its approach or ignoring the incorrect images. We plan to expand this evaluation by introducing more targeted “tools attacks” and we will include the results in the appendix of the paper.\n\n**Are the hypotheses diverse? Can we quantify this?**\n\nWe conducted a quantitative analysis to measure hypothesis diversity: we computed the average pairwise cosine similarity between hypotheses generated across all rounds of the self-reflective agent for each synthetic benchmark model seperatly. Specifically:\n\n- For each system, we parsed all hypotheses produced across rounds.\n- We computed the average cosine similarity between all pairs of hypotheses for that system.\n- We then averaged these values across all systems to obtain an overall similarity score.\n\nWe also constructed a baseline using the ground-truth descriptions of all 130 synthetic benchmark models (which vary in both object class and attribute dependence). We computed the average pairwise cosine similarity between these ground-truth descriptions.\n\n*Results:*\n- Agent hypotheses similarity score: 0.073\n- Ground-truth similarity score: 0.094\n\nThe lower similarity score for the agent’s hypotheses indicates greater diversity than the baseline, suggesting that the agent explores a wide range of explanations. We invite the reviewer to explore the agent's hypotheses in the experiments provided in the SM."}, {"review_id": "atbcXJDqBn", "reviewer": "Reviewer_H1cw", "summary": "This paper proposes an automatic framework for detecting which visual attributes an object recognition model uses when identifying a visual concept, especially focusing on detecting unintended attributes. The framework uses a \"self-reflective agent\" that works in two stages.\n\nFirst, in the \"Hypothesis-Testing\" stage, the agent (based on Claude-Sonnet-3.5) is provided with a subject recognition model and a target visual concept and must discover on which visual attributes on the input image the model relies on in order to recognize the target visual concept. To that end, the agent proposes test hypotheses about the attributes, generates synthetic images with and without these suspected attributes, scores the images with the subject recognition model, and refines its proposed hypotheses based on the model's responses until it reaches a \"stable explanation\". \n\nNext, in the \"Self-Reflection\" stage, another language model evaluates the explanation from the first stage. It checks if the explanation matches the recognition model’s confidence scores by again generating and scoring (with the model) more synthetic images. If inconsistencies are found, it launches a new \"Hypothesis-Testing\" stage. \n\nThe paper introduces a new benchmark of synthetic models with controlled attribute dependencies to evaluate the framework. Experiments show that the framework improves with more self-reflection stages. It is also tested on real models like CLIP and YOLOv8, successfully identifying their visual attribute dependencies.", "questions": "Please answer my concerns/questions/concerns listed on the weaknesses section above. Specifically:\n- How do you respond on my comment about the assumption that the recognition model depends on a single, \"decisive\" visual attribute?\n- How well the detected visual attributes from the synthetic images match the model's behavior on real-world images\n- What is the difference between the testing phase in the \"Hypothesis-Testing\" stage and the \"Self-Reflection\" stage\n- Questions about workings of the proposed framework", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**    \n- The paper introduces a novel agentic framework for automated interpretation of vision models (i.e., automatically detecting the visual attributes on which they rely on), addressing an important and emerging research direction in explainable AI.     \n- The proposed framework appears to be well-designed and makes (overall) sense. The results demonstrate the advantage of integrating into the framework \"self-reflection\" stages.    \n- Also, it is supported by a novel benchmark, which can be proved useful for future works in this area.     \n- The paper is generally well-written and accessible, successfully communicating its key ideas (however, as discussed below, some descriptions lack technical clarity)     \n\n**Weaknesses:**    \n- The work appears to assume that the recognition model depends on a single, \"decisive\" visual attribute (meaning the model succeeds only if the attribute is present and fails otherwise). However, in practice, recognition models might rely on multiple attributes, and their decisions may depend on a combination of features rather than a single absolute factor. For instance, a model might recognize a concept when a sufficient subset of relevant attributes is present, rather than requiring all or none.     \n- Based on the above, and given that the framework’s reliance on synthetic images, it remains unclear how well the generated explanations align with the model’s behavior on real-world images (if I understood correctly, both the proposed framework and the proposed benchmark rely on synthetic images for providing explanations and evaluating them respectively). Did the authors test whether the generated explanations (e.g., for the real models CLIP and Yolov8) match the model outcomes on real-world images instead of synthetic ones?     \n- It is not clear what is the difference between the testing phase in the \"Hypothesis-Testing\" stage and the \"Self-Reflection\" stage. Both generate synthetic images with and without the visual attribute, then score these images with the recognition model, and observe if the hypothesis matches the model's score outcomes. So, why a different \"Self-Reflection\" stage is needed is unclear.     \n- While on high-level it is easy to understand how the approach works, the main paper's description of how exactly the two main stages work is not entirely clear. For instance, it is not clear how exactly the agent determines whether a hypothesis matches the observed results in the \"Hypothesis-Testing\" and \"Self-Reflection\" stage. What is the exact criterion for determining if the hypothesis and the results match or requires further refinement?  Does the framework relies exclusive on synthetic images for generating explanations about the visual attributes?\n- The proposed framework appears to be somewhat complex.", "was_revised": true, "final_justification": "My initial review of this work was positive, highlighting its novelty, well-designed framework, and the introduction of a novel benchmark. However, I raised several concerns: 1) the assumption that a subject recognition model relies on a single decisive visual attribute, 2) the unclear behavior of the model on real-world images (i.e., whether the generated explanations align with subject model outcomes on the these real-world images), 3) confusion about the difference between the \"Hypothesis-Testing\" and \"Self-Reflection\" stages, and 4) the framework’s apparent complexity.\n\nThe rebuttal addressed most of these concerns effectively. The authors described additional experiments that demonstrate that their framework’s explanations match subject model behavior on real-world images and clarified the distinction between the two stages. However, further experiments during the discussion period revealed a limitation: the framework struggles with subject models that exhibit high confidence when relying on multiple attributes.\n\nDespite this limitation, I still find the work interesting and valuable. The authors’ commitment to additional evaluations will help clarify the operational scope of their system and may inspire future research to address this issue. Based on the authors’ responses, it also appears that they have addressed the other reviewers’ concerns to a reasonable extent.\n\nThus, I maintain my initial positive assessment and recommend acceptance.", "rebuttal": "We thank the reviewer for their meaningful comments! We were glad to see that, like all other reviewers, the reviewer H1cw found our benchmark and synthetic dataset to be useful and the self-reflection to be a meaningful contribution. Below, we address their comments. To give a short summary of the main points: \n- We constructed additional systems with dual-attribute dependency and showed that the agent can successfully identify both attributes. We will add these and more complex systems with multiple attribute reliances to the final version of the paper.\n- We evaluated the findings from the agent on real images and found the behavior to be as expected. We will add this evaluation to the appendix of the paper.\n- We clarify the different between the different stages and the stopping criteria\n____\n\n**Assumption for a single, \"decisive\" visual attribute**\n\nFollowing this comment we created 5 more synthetic benchmark models that have 2 attribute reliance (e.g. a bench detector that relies on *wooden benches* in *beach* settings) and found the self-reflective agent to be able to accurately uncover both reliances for 4 out of the 5 systems, while MAIA was not able to detect both reliances correctly in any of the systems. We will create more systems with multiple attribute reliances (more than 2) and report the tradeoff between performance and reliance-complexity in the appendix of the paper.\n\nWe would like to note that even in the benchmark systems that have only one attribute reliance, the agent commonly initially concludes reliance on multiple attributes and narrows down its conclusion over further rounds of self-reflection and hypothesis-testing. For example, in the provided supplemental experiment folder “black_umbrella_0.9”, we provide the full trace of an experiment on a *umbrella* detection model with an injected bias towards black umbrellas. After the first hypothesis-testing round (“round0.html”), the agent concludes the following: *“The classifier has a strong bias towards black umbrellas being used for rain protection in moody urban settings.”* In the subsequent self-reflection and hypothesis-testing rounds, this conclusion is found to be overspecified, and the agent ultimately discovers the more general ground truth reliance: *“The classifier identifies umbrellas primarily through their complete geometric form and prototypical appearance (especially black color)”* \n\n**How well the generated explanations align with the model’s behavior on real-world images?**\n\nGreat point! Following the reviewer’s suggestion, we conducted additional evaluations using real images for all attributes found for CLIP and YOLOv8. In these evaluations, we used the agent’s discovered explanations (e.g., reliance on classroom settings or biker pose) to select real images that either contained or lacked the relevant attributes. We then measured the model’s responses and found that the same attribute dependencies were reflected in all the real-world image scores, providing strong support for the alignment between synthetic and real domains. We will include these new results in the final version of the paper and are currently working on a larger-scale validation using real-world image datasets to systematically assess the generalizability of explanations.\n\n**The difference between the testing phase in the \"Hypothesis-Testing\" stage and the \"Self-Reflection\" stage.**\n\nThank you for raising this important question. While both stages involve generating and evaluating synthetic images, their roles, timing, and structure within the agent’s reasoning process are fundamentally different.\n\n- Hypothesis-Testing Stage: This stage is part of an open-ended, exploratory process. The agent iteratively proposes hypotheses and actively designs multiple experiments to test them, this may involve generating contrasting image pairs, editing dataset exemplars, or manipulating visual attributes in various ways. The agent then analyzes the pattern of scores across these experiments to decide whether the hypothesis is supported. Crucially, the agent may test several hypotheses during this stage.\n\n- Self-Reflection Stage: This stage occurs after the agent has reached a conclusion. The agent now uses a standardized self-evaluation protocol: it calls another LLM that takes the conclusion regarding the attribute reliance and generates 10 image prompts predicted to elicit high scores (with the attribute) and 10 image prompts that should elicit low scores (without it). These prompts are then fed to a T2I model, and the generated images are fed to the subject model to measure whether the model’s responses match this predicted contrast. This protocol acts as an external consistency check, validating whether the conclusion generalizes. If it fails, the agent reflects on the mismatch and re-enters the hypothesis-testing loop with updated insights.\nThis structure is essential to enable the agent to self-correct: if the self-reflection stage reveals inconsistencies between the conclusion and model behavior, the agent revises or rejects the conclusion entirely. We will clarify this distinction more clearly in the final version of the paper.\n\n**How does the agent determine whether a hypothesis matches the observed results**\n\nWe incorporate two concrete mechanisms to guide and constrain the agent's decision process:\n\n- Empirical Threshold for Self-Evaluation: In the self-reflection stage, the agent compares model scores between two groups of synthetic images: those predicted to yield high scores and those predicted to yield low scores under the current conclusion. If the mean scores between the two groups are not sufficiently separated based on an empirically set threshold (i.e. the conclusion is not sufficiently discriminative), the agent concludes that the hypothesis lacks support and initiates a new hypothesis-testing round.\n\n- Maximum Round Limit: We cap the total number of agent rounds (hypothesis-testing + self-reflection) to 10. If no hypothesis passes the self-evaluation threshold by that point, the agent returns the hypothesis that achieved the best self-evaluation score (typically the most recent one). In practice, the agent converges before reaching this cap. This behavior is visualized in Figure 5, which shows that the self-evaluation score tends to improve monotonically across rounds.\n\nWe will make sure to clearly explain this in the method section. \n\n\n**Does the framework rely exclusively on synthetic images?**\n\nThe method relies on both looking at activation of images from a fixed dataset of real images (“dataset exemplars”) to get an initialization for the search, and then on generated synthetic images that test different aspects of the hypotheses. In order to make sure that the conclusions are valid, we conducted additional evaluations using real images for all attributes found for CLIP and YOLOv8 and found that the same attribute dependencies were reflected in all the real-world images."}, {"review_id": "xqlCvjyQy9", "reviewer": "Reviewer_aRwo", "summary": "Understanding the features utilized by a visual recognition model in its prediction is necessary for detecting improper functionality. While many methods utilized predefined concepts lists and/or manual inspection, the authors propose an automated method leveraging a multimodal LLM. This method incorporates a self-reflective mechanism that separates it from other automated methods and results in improved performance in attribute reliance detection. This is achieved by setting up the LLM to iteratively generate and test hypotheses about a recognition model's reliance on a particular attribute. Additionally, the author curate a benchmark of 130 models that are diversely prepared with reliance on 18 diverse types of visual attributes based on known vulnerability of visual models (e.g., spurious correlations). The models are given predefined reliance thus giving users the ability to assess attribute detection/reliance methods. Finally, the authors demonstrate their automated pipeline's ability to uncover attribute dependencies in pretrained models such as CLIP and YOLO.", "questions": "- With the lack of concrete threshold / criterion for when a stable explanation is achieved, will the agent ever determine that the model doesn't rely on unintended attributes, or will it keep trying to find one? Basically, what if the model works as intended?\n- The counterfactual setting is unclear to me, could you clarify that intended functionality this experiment is measuring?\n- Is having a reliance variable always an issue? For example, when assessing whether an image includes a teacher or scientist, one cannot determine from looking at the person alone, we have to use attributes such as \"a class room setting\", \"a laboratory environment\", etc.", "limitations": "- Limited by the conceptual understanding and capability by the chosen pretrained multi-modal model\n\t- The method involves the LLM proposing candidate attributes, generating & editing the images (which would then include accurate understanding of the attributes). If the LLM is capable of accurately classifying, then why use the other model? Are there cases where the LLM fails to classify or detect well, but successfully improves the model it is diagnosing?\n- The approach depends on a several larger and more complex models to explain smaller models.\n\t- Evaluation and and effectiveness is entirely reliant on the strength of the chosen LLMs, summarizers, and image generators.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- (S) Paper organization, problem communication, and method overview are very clear and well written.\n- (S) The method successfully detects reliance attributes in the recognition models and outperforms other reliance detection methods.\n- (W) The method seems to be reliant on several hyper parameters such as:\n\t- The choice of multi-modal LLM\n\t- The choice of text-to-image model\n\t- The choice of editing method to said text-to-image model\n- (S) The addition of a benchmark of 130 models with known attribute reliance is a valuable contribute to reliance detection method, but also explainable & interpretable methods in general\n- (W) The attribute reliance of these models is simplistic as it uses an external attribute classifier (either SigLIP or FairFace) to detect the attribute and simply lowers the classification score if the attribute isn't detected. Additionally, all of the 130 models are based on Grounding DINO.\n- (W) For the \"Counterfactual demographic attributes\" described in section 4.2, it is unclear to me what the main difference to \"demographic attributes\" is.\n- (W) The evaluation criteria is entirely dependent on the LLM, so it is unclear when hypothesis testing will end and when self-reflection will end.", "was_revised": true, "final_justification": "Given the many clarifications and expansions to the synthetic models benchmark, and the experiment showing the robustness to failure at the T2I component of the baseline, I'm raising my score to 4.\n\nHowever, I still think there are limitations to the simplicity of the injected attribute reliance in the synthetic models (based on a single model: Grounding DINO). Additionally, the reliance on the T2I and image editing models may become problematic. Even though you showed that the agent can adapt by \"revising its approach or ignoring the incorrect images\", it still seems that in these cases the agent may fail to explain the problem with the model (limitation).", "rebuttal": "We thank the reviewer for their meaningful comments! We were glad to see that, like all other reviewers, the reviewer aRwo found our benchmark and synthetic dataset to be useful, and the method's performance to be convincing. Below, we address their comments. To give a short summary of the main points: \n- We clarify the reason for the way we constructed the synthetic benchmarks and its advantages, and provide an initial evaluation for the agent’s ability to recover from an error in the pipeline.\n- We add more complex systems and show the agent is mostly able to work with these. For the final version of the paper we will add an increasingly growing complexity experiment to find the agent’s breaking point.\n- Clarify the definition of the counterfactual systems\n- Clarify the evaluation criteria\n- Add a discussion on whether having a reliance is always an issue—to summarize, we believe it depends on the downstream task.\n- Clarify the reasoning for using large models like LLMs to audit smaller, more specialized models\n____\n\n**Framework is reliant on several hyperparameters and models**\n\nIndeed, our framework is modular by design, and its behavior may depend on the specific models used as the agent backbone and for the tools. We see this modularity as a strength: as new, more capable tools become available, they can be seamlessly integrated into the framework. To support this, we have:\nDesigned the benchmark of 130 synthetic models as a consistent testbed that can be reused to evaluate alternative model configurations\n\nFollowing the reviewer comment, we conducted the following experiment as an initial evaluation of the framework’s robustness to failures in specific components: in 50% of T2I calls, we replaced the agent’s prompt with an empty string, resulting in the model generating unrelated content that ignored the intended experimental manipulation. We found that in 80% of these corrupted trials, the agent successfully noticed the issue, either during the hypothesis-testing stage or self-reflection, and responded by revising its approach or ignoring the incorrect images. While this is a simple baseline, it gives an approximation of the agent’s sensitivity to breakdowns in the generation pipeline. We are currently expanding this evaluation to simulate degraded versions of each component (e.g., T2I, image editing, image summarization) and will report the agent’s error recovery rate across modules in the appendix of the final version of the paper.\n\n**The reliance of these models is simplistic, as they use an external attribute classifier**\n\nThe benchmark models are indeed based on a simplified mechanism. This design was chosen intentionally to allow explicit control over what and how severe ground-truth dependency is, enabling precise and scalable evaluation of whether an interpretability method can recover it. Importantly, this setup allows us to vary the strength of the injected reliance, simulating models with mild to strong biases. In the SM (sec 2.2 and Fig. 1), we show that this mechanism enables us to simulate a wide range of bias levels, from weak reliance to strong or even exclusive reliance. This setup allows us to quantitatively evaluate how well the agent recovers dependencies of varying strength.\n\nA benefit of our mechanism is it enables the construction of more synthetic systems that can be engineered to exhibit a wide range of behaviors. To further extend realism and complexity in the benchmark, we have started to build multi-attribute models with compositional reliance (e.g., concepts that depend on both setting and gender). We observed that the self-reflective agent successfully recovers both dependencies in 80% of systems while MAIA was not able to recover these at all. We will add systems of increasing complexity to test when the agent breaks.\n\nIn addition to this controlled benchmark, we validate our method on pre-trained models (e.g., CLIP, YOLOv8), demonstrating that the agent can recover attribute dependencies that emerge organically. These results suggest that our synthetic benchmark complements real-model analyses, providing both a rigorous evaluation ground and a path for future expansion.\n\n**What are the “Counterfactual demographic attributes”?**\n\nDemographic attribute systems simulate reliance on naturally co-occurring demographic signals (e.g., the presence of a male face often coinciding with ties in training data). These are designed to reflect the types of biases commonly found in real-world models and datasets. Counterfactual demographic systems, by contrast, are constructed to simulate atypical or out-of-distribution dependencies e.g. a model that detects a tie only in the presence of a person who is female, which rarely co-occurs in real-world data. These systems allow us to test whether the agent can detect unexpected or counterintuitive reliance patterns that do not follow natural co-occurrence statistics. This distinction allows us to assess both the agent’s ability to uncover realistic demographic biases and its robustness to rare or previously unknown dependencies. We will make this difference more explicit in the final version of the paper.\n\n**The evaluation criteria are entirely dependent on the LLM**\n\nWhile it is true that the agent is built from a multimodal LLM to reason over experimental outcomes, its decision process is guided by a structured protocol that provides consistency. \nIn more detail: In the hypothesis-testing phase, the agent runs a number of experimental rounds, where each round involves proposing a hypothesis, designing targeted stimuli (via prompting or editing), and evaluating the model’s behavior. The self-reflection stage follows a fixed diagnostic protocol: the winning hypothesis from the previous hypothesis-testing stage is passed to a separate LLM that generates two diverse sets of prompts predicted to elicit high and low detection scores. Then we measure the model scores on these images. If the resulting score distributions do not align with the agent's expectations (i.e., little to no score separation between the two sets of images), the agent concludes that the hypothesis lacks support and re-enters the hypothesis-testing phase with a revised prompt.\n\nWe incorporate two concrete mechanisms to guide and constrain the agent's decision process:\n\n- Empirical Threshold for Self-Evaluation: In the self-reflection stage, the agent compares model scores between two groups of synthetic images: those predicted to yield high scores and those predicted to yield low scores under the current conclusion. If the mean scores between the two groups are not sufficiently separated based on an empirically set threshold (i.e. the conclusion is not sufficiently discriminative), the agent concludes that the hypothesis lacks support and initiates a new hypothesis-testing round.\n\n- Maximum Round Limit: We cap the total number of agent rounds (hypothesis-testing + self-reflection) to 10. If no hypothesis passes the self-evaluation threshold by that point, the agent returns the hypothesis that achieved the best self-evaluation score (typically the most recent one). In practice, the agent usually converges before reaching this cap. This behavior is visualized in Figure 5, which shows that the self-evaluation score tends to improve monotonically across rounds.\n\nWe will make sure to clearly explain this in the method section. \n\n**Will the agent ever determine that the model doesn't rely on unintended attributes**\n\nGreat point! Yes, in some cases the agent concludes that there are no dependencies, and we will include such examples in the supplements of the paper.\n\n\n**Is having a reliance variable always an issue?**\n\nOur framework does not attempt to judge whether a discovered reliance is inherently harmful or contextually appropriate, nor is this distinction communicated to the agent. Rather, the goal is to surface potential dependencies that might not have been previously anticipated, enabling practitioners to make informed judgments based on their specific downstream use case. In some applications, reliance on context (e.g., background for \"teacher\") may be a valid, even desirable, part of the concept's representation. In others (e.g., medical imaging or fairness-sensitive applications), such dependencies could lead to unwanted generalization or bias.\n\n**Why not use the backbone LLM for the recognition task itself?**\n\nThe large models used in our framework are too large and expensive to be used in some practical deployment scenarios, such as on-device inference or embedded systems. Many real-world models are much smaller and more efficient. Our approach allows to use the strengths of foundation models, like strong semantic understanding and language grounding, to systematically identify dependencies in other models that might be less capable, more compact, and even task-specific."}]

Author rebuttal:

Meta-review: The submission initially received three ratings leaning toward rejection and one recommending acceptance. Reviewers found the proposed self-reflection agent to be novel and interesting, the benchmark useful, the experiments well-designed, and the paper well written. Initial concerns included the reliance on image embeddings or T2I models that may carry inherent biases, the model’s inability to distinguish between harmful and benign attribute dependencies, missing comparison to MAIA, unclear differences between the framework stages, and other technical issues.

The authors rebuttal effectively addressed most of these concerns, and all reviewers shifted to recommending acceptance in their final ratings.

The AC concurs with the reviewers consensus and recommends acceptance. The AC asks the authors to incorporate the rebuttal and reviewers feedback into the final version.
Decision: Accept (poster)