Counterfactual Explanations on Robust Perceptual Geodesics
Authors: Eslam Zaher
Dr Maciej Trzaskowski
Quan Nguyen
Fred Roosta
Venue: ICLR 2026 Poster (2026)
Keywords: Interpretability
Visual Counterfactual Explanations
Explainability
Primary area: interpretability and explainable AI
TL;DR: We propose Perceptual Counterfactual Geodesics (PCG), a method that generates counterfactuals by tracing geodesics in a perceptually aligned latent space, outperforming prior methods and avoiding failures from misaligned geometry

Abstract: Latent-space optimization methods for counterfactual explanations—framed as minimal semantic perturbations that change model predictions—inherit the ambiguity of Wachter et al.’s objective: the choice of distance metric dictates whether perturbations are meaningful or adversarial. Existing approaches adopt flat or misaligned geometries, leading to off-manifold artifacts, semantic drift, or adversarial collapse. We introduce Perceptual Counterfactual Geodesics (PCG), a method that constructs counterfactuals by tracing geodesics under a perceptually Riemannian metric induced from robust vision features. This geometry aligns with human perception and penalizes brittle directions, enabling smooth, on-manifold, semantically valid transitions. Experiments on three vision datasets show that PCG outperforms baselines and reveals failure modes hidden under standard metrics.

Published as a conference paper at ICLR 2026
COUNTERFACTUAL EXPLANATIONS ON ROBUST PERCEPTUAL GEODESICS
Eslam Zaher!.2 Maciij Trzaskowski Quan Nguyen 13355 Fred Roosta!
1 ARC Training Centre for Information Resilience (CIRES)
2 School of Mathematics and Physics, University of Queensland
3 Institute for Molecular Bioscience, University of Queensland
4 Profenso
5 QIMR Berghofer Medical Research Institute
ABSTRACT
Latent-space optimization methods for counterfactual explanations framed as minimal semantic perturbations that change model guity of Wachter et al.'s objective: the choice of distance metric dictates whether perturbations meaningful or adversarial. Existing approaches adopt flat misaligned geometries, leading to off-manifold artifacts, semantic drift, or ad- versarial collapse. We introduce Perceptual Counterfactual Geodesics (PCG), a method that constructs counterfactuals by tracing geodesics under a perceptually Riemannian metric induced from robust vision features. This geometry aligns with human perception and penalizes brittle directions, enabling smooth, on- manifold, semantically valid transitions. Experiments on three vision datasets show that PCG outperforms baselines and reveals failure modes hidden under standard metrics.
the ambi-
INTRODUCTION
As deep learning models grow in scale and impact. interpretability becomes paramount as it offers a crucial lens into their internal reasoning. Traditional saliency-based methods, which highlight influential input features (Simonyan et al., 2014; Sundararajan et al., 2017; Smilkov et al., 2017, 7; Kapishnikov et al., 2021; Ribeiro et al., 2016; Selvaraju et al., 2016; Lundberg & Lee, 2017), have been widely adopted for vision models but produce static, often noisy attributions that lack guidance on how predictions could be altered. Counterfactual explanation (CE) methods have emerged complementary paradigm grounded in the fundamental human capacity to contemplate "what if?" scenarios (Wachter et al., 2017; Ustun et al., 2019; Joshi et al., 2019; Artelt & Hammer, 2019). Rather than merely highlighting salient regions, CEs specify which semantic features should be modified-and how-to to produce a different prediction. Wachter et al. (Wachter et al., 2017) formalized this notion as a solution to an optimization problem:
min
(((()))))
Similarity Distance
Classification
where x* is the original input, s the desired class, f the classifier, é loss function (c.g., cross- entropy), a distance metric, and À hyperparameter balancing classification and similarity.
Considerable debate has emerged around whether a CE is fundamentally distinct from an adversarial example (AE), as both arise from the same optimization problem (Wachter et al., 2017; Browne & Swift, 2020; Pawelczyk et al., 2022; Freiesleben, 2022). The choice of distance metric plays central role: while it may support meaningful CEs, it can also encourage AEs if it favors imperceptibly small, distributed perturbations. Wachter et al. Wachter al. (2017) acknowledger this ambiguity, noting that "AEs are counterfactuals by another name," proposing distinction two grounds: (i) misalignment of the distance metric with meaningful feature changes- since metrics typically used for AEs favor such dispersed modifications, thereby diminishing their explanatory value, and (ii) adversarial perturbations are non-semantic signals that displace inputs out of the possible off-manifold regions that do not correspond to valid examples under the data distribution.
Rather than directly solving eq. (1), some approaches leverage generative models to produce visual CEs by exploiting low-dimensional semantic representations (Augustin al., 2022; Mertes et al.,

---

Published as a conference paper at ICLR 2026
2022; Looveren et al., 2021: Singla et al., 2020; Lang al., 2021; Khorram & Fuxin, 2022). For instance, Singla et al. (2020) trained a conditional GAN to produce exaggerated CEs, while Lang et al. (2021) used conditional STYLEGAN2-busse approach generate sparse visual CEs along disentangled classifier-relevant style-space directions. Khorram & Fuxin (2022) used cycle-consistent losses train transformations between factual and counterfactual distributions in generative latent spaces. Though visually compelling, these methods rely on exhaustive techniques that depart from the direct optimization formulation and ignore the geometry of the data manifold.
Other research adopt eq. (1) in the latent space of generative models (Joshi et al., 2019; Duong al., 2023; Dombrowski al., 2024; Pegios al., 2024), but either assume flat Euclidean geometry (Joshi et al., 2019; Dombrowski et al., 2024), failing to capture the manifold's ' intrinsic curvature, or use geometrically informed yet adversarially vulnerable distance metrics (Pegios et al., 2024). For example, REVISE (Joshi et al., 2019) solves the objective in eq. (1) in a VAE latent space under Euclidean assumptions, using explicit / distance terms. Dombrowski al. (2024) discard explicit similarity terms and employ Stochastic Gradient Descent (SGD) assuming flat geometry misaligned with the underlying data manifold.
Off-manifold.oo AE
Beeaser
Gonerate
Cluss
On-manifald
Figure 1: Schematic of PCG. An input is mapped through an encoder-generator pair. A linear latent path to perceptually plausible target-class sample (Class B, brown region) is refined in Phase I into the blue geodesic by minimizing robust perceptual energy. In Phase 2, the endpoint and intermediate points are jointly optimized under classification loss and robust energy, resulting the red counterfactual geodesic. The green trajectory (REVISE, VSGD) ignores manifold geometry, strays off manifold and produces off off-manifold AEs. The yellow trajectory (RSGD/-C) conforms to fragile geometry, getting stuck in on-manifold adversarial regions (Class B, outside brown region).
This misalignment often causes perturbations to stray off-manifold, leading to implausible or off- manifold AEs. Pegios et al. (2024) proposed equipping the latent space with Riemannian metric induced by the generator and optimizing with Riemannian SGD (RSGD) to account for the geometry of the data manifold. However, their induced metric is typically derived by pulling back either the pixel-space l or a standard classifier's feature space metric. Both are problematic in the vision domain: the & norm is poor proxy for human perception (Sinha & Russell, 201 1; Jordan al., 2019; Rybkin, 2022), while a standard cesturee-bseed metric is semantically brittle as it inherits the adversarial vulnerabilities of non-robust vision models (Sjögren al., 2022; Ghazanfari et al., 2024b).
Such methods acutely fail in the high-dimensional vision domain, where the counterfactual optimiza- tion process can't 't distinguish between CEs and ABs. Browne & Swift (2020) proposed the notion of semantic divide- distinction between perturbations that affect human untraantnnndtttaaalllll semantic features or low-level, uninterpretable features. Perturbations with rich semantic content fall on the explanatory side; pixel-level or low-level ones fall on the adversarial side. Browne & Swift (2020) argue that neither distance metries nor appeals to "possible worlds" fully resolve this distinction; instead, semantic relevance only determines whether result is valid CE or AE
We agree with Browne Swift (2020) that the second criterion proposed by Wachter et al.- displacement to off-manifold regions-fails to adequately differentiate AEs from CEs. Several studies have shown that on- manifold AEs exist (Ilyas et al., 2019; Garcia et al., 2023; Song et al., 2018), and can be generated via generative models (Stutz al., 2019; Zhao al., 2018), representing a subclass of AEs that reside within Wachter et al.'s "possible worlds". However, we challenge the assertion that distance metrics inherently incapable of making the distinction. We show that if the
2

---

Published as a conference paper at ICLR 2026
data manifold is endowed with semantically robust Riemannian metric, solving the counterfactual can cetimization-wher guided appropriately- can cross the semantic divide and produce valid CEs.
Failure Modes of Previous Approaches. We attribute the failure of previous latent-space counter- factual optimization methods in the high-dimensional image data regime to three core limitations:
Off-manifold Traversal. Optimization in latent space often disregards the geometry of the data manifold, leading to off-manifold AEs or semantically implausible counterfactuals (Pegios al., 2024).
(ii) Local Gradient Optimization. Without glohal structural guidance, single-point geometry- aware gradient methods operate locally and overlook the global manifold structure, including the existence of on-manifold adversarial regions. As result, they often converge to either semantically distant counterfactuals or on-manifold AEs.
(iii) Versatility of Generators. Even when accounting for manifold geometry, high-capacity generators can exploit non-robust or misaligned distance metrics to produce on-manifold AEs (Stutz al., 2019; Zhao al., 2018; Gilmer et al., 2018), fooling the metric rather than producing semantically meaningful perturbations that genuinely cross the semantic barrier.
Contributions. Motivated by findings in adversarial robustness that show robust models exhibit perceptually aligned gradients (Ganz et al., 2023b; Srinivas et al.; Shah et al., 2021; Kaur et al., 2019b), robust saliency maps (Etmann al., 2019; Zhang & Zhu, 2019; Tsipras et al., 2019), and meaningful CEs (Boreiko et al., 2022; Santurkar et al., 2019; Augustin et al., 2020), we introduce a semantically grounded, data-manifold basddd approach for perceptually progressive CEs. We emphasize that our focus lies not in interpreting robust classifiers themselves, but in generating explanations for standard models, positioning our work orthogonally to efforts aimed at explaining robust models (Boreiko et al., 2022; Santurkar et al., 2019; Augustin al., 2020). Our key contributions
follows:
(i) Counterfactual Generation: We introduce Perceptual Counterfactual Geodesics (PCG), which leverages robust Riemannian metric on the latent space of STYLEGAN2/3 generator (Karras et al., 2020b; 2021). This metric induced from feature spaces of robust vision models. PCG optimizes counterfactual trajectories along geodesie paths, ensuring that counterfactual evolution adheres to robust perceptual perturbations that cross the semantic barrier, avoiding off- or on-manifold adversarial regions.
(ii) Perceptual Geodesic Interpolation: We show that the robust latent geometry underlying PCG enables smooth and semantically robust interpolations between samples. Our exper- iments demonstrate that trajectories aligned with the robust Riemannian metric preserve class coherence and perceptual structure. In contrast, other metrics collapse into visually ambiguous or brittle transitions due to geometric misalignment.
2 BACKGROUND
2.1 DIFFERENTIAL GEOMETRY OF DEEP GENERATIVE MODELS
Deep generative models, such as VAžs and GANs, offer powerful framework for learning high- dimensional data distributions through low- www-dimessional latent representations (Kingma & Welling, 2022; Higgins al., 2016; Goodfellow et al., 2014; Karras al., 2018). These models define generative function 7 - X, where Z C Rd is a latent space and X c RR. is high-dimensional data space, typically d < D. The image of Z under g. denoted M = g(Z) C X, forms subset of the data space, often referred to as the data manifold. Under mild regularity conditions-suce as smoothness of y with a full-rank Jacobian mapping J, A 0g/0z 7 + IR Dxd this image is a smooth, d-dimensional immersed submanifold of X (Shao et al., 2017: Arvanitidis et al., 2017). This construction supports the manifold hypothesis, which posits that real-world high- -igh-dimensional data concentrates near such a low-dimensional manifold (Brahma et al., 2016; Fefferman et al., 2013; Tenenbaum et al., 2000).
However, while Z is typically treated as Euclidean, this assumption misaligns with the geometry induced by g. as the nonlinear generator significantly distorts its structure. As result, distances and directions in Z do not reflect the true relationships of the data manifold. This motivates equipping the latent space with geometry that faithfully reflects the structure of the image manifold. M.
3

---

Published as a conference paper at ICLR 2026
2.2 PULLBACK METRICS AND THE GEOMETRY OF GENERATORS
A smooth manifold M C X inherits a tangent space T.M at each point I E M, consisting of directions along which one can move locally. To measure lengths and angles, we define a smoothly varying inner product (z on each tangent space. This defines a Riemannian metric G(z), and the pair (M, G) forms Riemannian manifold.
Given a smooth generator 9 Z + via pullback from the ambient space X, assumed to have metric Gx (x) RDDD For any u, T.Z = Rd, we define:
X, we equip the latent space Z with a Riemannian metric
(u, (J)())u, Jg())
u Jz(2) (x()()))dg())),
where (2) is the Jacobian of g at z. If 12(2) has full column rank, this defines the pullback metric as Gz(z) = (xx=)))(((=))))](C))
While mathematically well-defined, this construction inherits the limitations of the ambient metric. When Gx(x) I. the geometry is induced from the canonical pixel-wise lo metric. In high- dimensional vision tasks, such distances misalign with human perception and are highly sensitive to small, imperceptible perturbations. This issue is not limited to Euclidean metrics; it also applies to other ambient geometries that lack robust semantic grounding. For example, Pegios et al. (2024) pulls back a feature- based metric from standard classifier, which operates in feature space but still inherits the adversarial vulnerabilities of non-robust models. As a result, the induced latent geometry reflects local structure relative brittle and semantically misaligned notion of similarity, often leading to adversarial trajectories (Browne & Swift, 2020).
2.3 LATENT SPACE COUNTERFACTUAN OPTIMIZATION
We summarize several methods that solve variations of eq. (I) in the latent space of generative models.
REVISE. Joshi et al. (2019) introduced an approach based on VAEs for tabular data, where the latent code z of an input x= updated via SGD on the objective = d(x,, g())) + ((([((())) v/). This method relies on two assumptions: that pixel-wise Euclidean distances in ambient space provide meaningful similarity, and that Euclidean SGD updates in latent space correspond to smooth semantic transitions. Both assumptions fail in high-dimensional vision domains, where distances are misaligned with perception and SGD updates stray off off-manifold.
Vanilla SGD (VSGD). To adapt vision settings, Dombrowski et al. (2024) proposed eliminating the distance term in REVISE and directly applying vanilla SGD to the classification loss:
+ z 1) V= ((F(o())), v
While sidestepping metric misalignment in X, it still assumes a flat Euclidean geometry in Z, ignoring the curvature induced by g. Since y is highly nonlinear expressive models, such updates often stray off the manifold and lead to off offfmmanioldd AEs or perceptually implausible counterfactuals.
Riemannian SGD (RSGD). Pegios et al. (2024) proposed RSGD account for the curvature the data manifold by replacing Euclidean gradients with Riemannian ones derived from a pullback metric on the latent space. Given stochastic VAE generator (=))) = f=z) + (z) @ e, with N/ T), the latent metric is defined as the expected pullback of the ambient & metrie:
G2(z) = (x) Jz(()) + (z) (2).
and optimization proceeds via: 2 6 |r|
where r =
A variant, RSGD-C, replaces the ambient metric with the pullback of a classifier-basse feature metric, using the final-layer representation of a standard classifier. This introduces auskawareesess by aligning updates with
directions.
Both methods remain limited by their underlying metrics. Pixel-wise l distances are fragile and misaligned with perception. and standard classifier-basee features inherit adversarial vulnerabilities. RSGD/-C does not enforce geodesic paths and has been applied only in low-dimensional domains where adversariality is less evident.
4

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
Prior approaches fail in the vision domain due to three tightly coupled issues: the use of perceptually misaligned metrics (e.g., lg in pixel space or fragile Classifier-basee metrics), reliance on local gradient updates that ignore global manifold structure, and the expressive power of high-capacie generators that exploit these misaligmments to produce adversarial perturbations.
Our method, PCG, addresses these limitations by casting counterfactual generation global curvature- optimization over latent trajectories on Riemannian manifold, where the generator induces a latent geometry aligned with human perception. To define this geometry, construct perceptually robust ambient metric. Unlike standard classifiers, robust models learn representations that are resistant to adversarial perturbations and aligned with human perceptual similarity. These robust intermediate activation spaces exhibit linearly separable structure and encode grounded, semantically meaningful features. As result, the Euclidean metric becomes more reliable proxy for perceptual similarity in these robust semantic spaces, unlike its failure in pixel or fragile semantic spaces. We leverage this structure to define a composite ambient metric by aggregating pullbacks of the Fuclidean metric from robust feature spaces into the input space, capturing hierarchical, perceptually coherent variations. Formally, we define the robust perceptual metric as:
K Σ wk Jnn T
1 Jny (x), Wk
Gn(2)=
N
where K is the number of selected intermediate layers of a pretrained robust vision model, h2 (z) denotes the activation of the k-thh layer with dimensionality dę < D, Jhy (x) E Rd is its Jacobian with respect to the input E TR. and Ng denotes the total size (number of elements) of the activation hy (xr), which normalizes each layer so that no single feature space dominates due its size. Pulling back GR through the generator g Z X defines the latent-space metric
Gz(2) = J=()) Gn(o(=))J,,()),
which induces a latent geometry that penalizes brittle or non-robust directions and favors perturbations that produce perceptually smooth, semantically aligned variations in the image space.
We seek smooth latent trajectory 7: [0, + Z such that ((+(t)) evolves through robust semantic regions. The perceptual length of this trajectory, where (t) = dy/dtt is the latent-space velocity, evaluated under GR. is
[(ght)) -(g())- -f. (Y(T
and minimizing this length under constant-speed parametrization is equivalent to minimizing the robust perceptual energy (Jost, 2017):
E(g())) = J. (z((())))))
Expanding Gz using the composite metric shows that the pullback energy weighted sum of squared velocities in each robust feature space:
K Σ wk k=1
Y0) GZG())))))))))
Minimizing E(g(7)) thus amounts to finding geodesic whose generator outputs move smoothly and consistently across all robust semantic layers. To do this, we discretize y into T + points (zo...., where 2g is the latent encoding of the input x², and 27 is initialized as the latent encoding of arbitrary target-class sample from the dataset. This initialization is critical: unlike which often converge
previous methods that perform iterative updates from single starting to on-manifold adversarial endpoints-we initialize between two manifold-conformal guide global transitions across semantically valid regions under the robust metric. Using forward finite differences as in Shao al. (2017), approximate the robust feature-space velocity at I,
points to
5

---

Published as a conference paper at ICLR 2026
as dhg (o)()()(t)))/d 1=,, (ha(o(2(42) h=(((22)))// This gives the discrete robust energy equivalent of eq. (2):
T K
Wk öt
ha(o(=(+1) - -ha(o(=))) [|: where z - [zo.-... 2T] and öt = 1/T.
Frobust (z) =
Optimization proceeds in two stages, In Phase 1, we fix 20 and and minimize Eroba( with respect to the intermediate points to obtain a geodesic consistent with the robust semantic geometry induced by the generator. In Phase 2, we release 27 and jointly optimize the energy and classification loss to ensure the endpoint maintains the desired prediction under f. The combined loss is
[(z) Exobusa (z) + X + (((g(Zz)),
In practice, Phase 2 is implemented as coarse-to- coarse-to-fine refinement of the Phase I geodesic. We start from the path connecting Zorig to a target-class exemplar and optimize C for a fixed budget, with a small initial À that is increased over time. This schedule gradually shifts the optimization from purely geometric regularization (when À small and Erchust dominates) towards enforcing the target-class prediction at the endpoint (as À grows). To avoid either collapsing back to the input or drifting too far into the target region, we periodically apply a vreannhhring step: at fixed intervals, we scan along the current path for points that are already classified as the target class and select the one closest to Corig (in the induced perceptual geometry) as the new endpoint. We then reparameterize the path by inserting midpoints between successive latent codes restore the original number of waypoints and resume optimization from this shortened, re-anchored trajectory. Iterating this procedure has the effect of progressively "pulling" the endpoint towards the input while keeping the target label fixed, so that the path converges to a counterfactual that remains on the robust geodesic manifold and is as close as possible to Forig under the induced metric. The overall structure of our two-stage optimization and the contrast with prior methods is illustrated in Figure 1; full algorithm, induced metric, and optimization details are provided in Appendix A.
EXPERIMENTS
We evaluate PCG against prior latent-space optimization methods. section 4.1, we first show the failure mode of interpolation methods inherent in their geometrical assumptions, and demonstrate the effect of our proposed robust Riemannian metric in generating perceptually smooth geodesics that underpins PCG. In section 4.2, we compare PCG with other approaches in terms of the perceptual plausibility of the generated counterfactuals. Finally, we quantitatively evaluate PCG under both typical and geometry- distance measures. Code for experiments is available here.
Datasets. We evaluate our method on three high-dimensional real- real-image datasets: (1) AFHQ (Choi et al., 2020), with high-resolution images of cats, dogs, and wild animals; (2) FFHQ (Karras et al., 2019), containing 70,000 diverse human face images; and (3) Vlant (Hughes & Salathé, 2015), with labeled images of healthy and diseased plant leaves across species.
Models. We train STYLEGAN2 generators from scratch on AFHQ and PlantVillage (=140 NVIDIA H100 GPU-hours per model) (Karras al., 2020a). For AFHQ, we also use a pretrained STYLEGAN3 generator (Karras et al., 2021). For FFHQ, we use pretrained STYLEGAN2 and STYLEGAN3. Post hoc, we train image-to-latee and then briefly fine-tune the encoder-gente generator pair jointly. For classifiers, we train binary models based on the VGG-19 backbone (Simonyan & Zisserman, 2014): one per AFHQ class pair and a -snuhhy classifier for PlantVillage. Because FFHQ lacks labels, we train attribute classifiers on CelebA (Liu al., 2015) and apply them to FFHQ Architectural and training details appear in Appendix A.3.
encoders (used for all counterfactual optimization in z-space)
Baselines. We compare PCG against the following latent-space based approaches:
REVISE (Joshi al., 2019). Latent-space equivalent of Wachter et al.' objective based on SGD, VSGD (Dombrowski et al., 2024). It performs distance- freee vanilla SGD in the latent space. RSGD/-C (Pegios al., 2024). In these variants, a Riemannian metric is used to guide SGD. The metrics are pull- back from either the Euclidean metric in the ambient space or in the final layer of the classifier under explanation.

---

Published as a conference paper at ICLR 2026
4.1 EFFECT OF LATENT GEOMETRY ON INTERPOLATION
In Figure 2, we illustrate how latent-space geometry shapes interpolation. The top row linearly interpolates latent space under Euclidean assumption, which ignores the nonlinear distortion induced by the generator and produces mid-path off- aff-manifold artifacts such as class ambiguity, unnatural warping, and deformed textures. The second row minimizes pixel-space MSE in X. which induces latent-space geometry by pulling back the Euclidean metric from X to Z: transitions remain brittle and semantically incoherent, with midway blends of disparate attributes that expose the fragility and misalignment of pixel-wise distances. The third row uses the pullback of feature metric from a standard ResNet-50 (He et al., 2016) (see appendix A.4); semantics improve, yet fading, illumination shifts, and class discontinuities persist. These instabilities reflect the vulnerability of non- robust models to adversarial perturbations and reliance on brittle features, with similar failure modes reported Laine (2018) using VGG-19. In contrast, the fourth row applies our robust perceptual metric derived from robust ResNet-50, producing smooth, on- mn-manifold trajectories with consistent semantics and coherent evolution. This confirms our hypothesis that robust Riemannian geometry enables smooth, semantically valid on- on-manifold interpolations while avoiding adversarial collapse.
Z lerp
X' MSE
F MSE
R MSE
Figure 2: Interpolation paths under four latent geometries based on STYLEGAN2 (top-bottom). (a) Z-linear (Euclidean): flat latent metric; off- mffifolifold artifacts. (b) Pixel MSE pullback: Euclidean metric pulled back to Z; brittle, incoherent paths. (c) Standard feature pullback: non-robust ResNet- 50; better semantics but still fading and discontinuities. (d) Robust perceptual pullback (ours): robust ResNet- 50; smooth, consistent, on- manifold trajectories. See Appendix B. for STYLEGAN3 results.
4.2 PERCEPTUAL COUNTERFACTUAL GEODESICS
Having established smooth perceptual geodesics under our proposed metric, their refinement into plausible CFs. Figure 3 showcases the two-stage nature of our approach. In Phase (rows) 1 and 3), we generate initial perceptual geodesic between the input and an arbitrary larget-class sample, such as a dog image for a cat input, or a non-blonde face for a blonde input. Although the target semantically distant, the path remains coherent, illustrating the alignment of our metric with perceptual structure. In Phase 2 (rows 2 and 4), we release the endpoint and jointly optimize with the path under the classification loss, allowing the counterfactual move closer to the input while maintaining geodesicity. The resulting counterfactual geodesics trace robust regions of the data manifold and maintain consistent semantics throughout the trajectory, retaining the semantic continuity and avoiding adversarial shortcuts abrupt transitions. This step ensures the whole path travels through perceptually robust regions on the manifold as shown Fig 1. We show that different choices of the target-class exemplar lead optimization to converge within small neighborhood of the input, producing diverse yet faithful counterfactual explanations; see Appendix B.3
demonstrate
Comparison with Baselines. We now evaluate the final counterfactuals produced by PCG against existing latent-space optimization methods. As shown in Figure 4, our method consistently produces semantically valid CEs that remain close to the input while effecting the desired class transition. In
7

---

Published as a conference paper at ICLR 2026
contrast, RSGD and RSGD-C, despite accounting for local curvature, rely fragile metrics (e.g.. pixel-space & or non-robust classifier features) that remain vulnerable to adversarial manipulation. Many of the generated counterfactuals collapse into on-manifold AEs-as seen in and 6. Like the outputs of other baselines, they fall on the adversarial side of the semantic divide. Even when RSGD variants converge (e.g., row 3), the output is visibly distant from the input in pose and structure, reflecting the lack of geodesic constraint and a tendency to traverse longer manifold paths. VSGD, which assumes flat Euclidean geometry, produces off-manifold perturbations that either perceptually implausible, or adversarial. In row 2. the generated counterfactual exhibits class ambiguity and disoriented eye alignment: in row 3, the face is unnaturally elongated with distortions under the chin; in row 6, the leaf counterfactual contains the expected symmetry, fullness, and surface continuity of leaves. These artifacts arise from ignoring the data manifold altogether. REVISE exhibits similar failure modes: the strong pixel-wise distance penalty constrains outputs to remain close in & norm, but adversarial. All REVISE outputs in the figure represent off aff-manifold AEs, driven by the optimization pressure to minimize distance rather than induce meaningful semantic change. In contrast, PCG navigates robust regions of the manifold along perceptual geodesics, producing minimal, semantically faithful changes.
1, 2, 4, 5,
unnatural cusp-like protrusion that breaks
Initial Geodesic from Phase
0.00
0.15
0.37
0.81
0.92
0.97
1.00
1.00
Counterfactual Geodesic from Phase 2
0.04
0.05
0.25
0.01
0.21
0.46
0.72
0.85
0.98
1.00
1.00
1.00
L.00
0.01
0.02
0.03
0.04
0.15
0,25
0.45
0.62
0.71
0.80
Figure 3: Perceptual Counterfactual Geodesics. Rows and 3: initial geodesics from Phase between an input and a target-class sample. Rows and 4: counterfactual geodesics after Phase 2, where the endpoint is optimized with the path. from Phase 2 stay in robust regions of the manifold and preserve semantic continuity. Results from STYLEGAN2 (see Appendix B.2 for STYLEGAN3)
Table Quantitative comparison across datasets for STYLEGAN2 (see Appendix B.4 for STYLE- GAN3 and Appendix B.5 for runtime complexity). Columns report Li (pixel &1), L2 (pixel f2), L (pullback from standard VGG-16), and LR (pullback from robust Inception V3). Lower is better.
Method
AFHI
FFHQ
PlantVillage
62
Ey
6x
Cx
4
E2
Ey
Ex
REVISE:
1,2081112
0.730018
1,0HL.00
2.70H0.55
088200088
0.32H0.13
08240008
27840.06
0.5080.13
0.3860.15
0.66006
2.5740.07
1.3140.11
1,4900115
1.6040099
0.7960.11
0.9610.10
1.50H0.12
286:0.07
03340.13
09940.17
1.180007
+00
RSGD
0.8540.88
1,3210.09
0.7010.07
1.8540.05
0.6110.55
08410.07
0611004
24110.05
0.7860.8
08260.11
0.440005
04
RSGD-C
0,9310.10
14550.17
0.6520.88
L.7540.06
0.6R20.06
0.9310.09
0.1840.04
2:1140.44
0.80.0.10
0.8600.33
0.452005
2:00.0006
(ours)
0.7940.07
1.14+0.10
0.5340.06
0.31+0.02
0.42:0.03
0.72+0.09
0.39+0.05
0.22:0.06
0.36:0.3
0.56+0.5
0.340004
0.20
Distance-based Evaluation. We assess counterfactual proximity using four distance metrics: C (pixel-wise & L2 (pixel-wise f2), L,= (distance induced by the pullback from standard ResNet- 50 features), and Lx (pullback from robust ResNet-50 features). Each induced metric is com- puted between the input and the final counterfactual in image space using the local quadratic form La(z0, 2T) / (g((r) o(20)) ((g(z)))((zr)) g(zo)), where G (Gr, GR) is the respec-
8

---

Published as a conference paper at ICLR 2026
tive ambient metric. This approximates perceptual distance in the feature space around the input. To avoid entanglement between optimization and evaluation, we compute Cx using an independent VGG-16 model that never involved in training or counterfactual optimization, and compute LR using a robustly trained Inception- V3 model (Alfarra et al., 2022h) separate from the robust ResNet-50 that defines metric. As shown in Table 1, our method achieves the lowest distances across all geometry-awate metrics and also under L1, indicating sparse, perceptually meaningful changes. The margin is largest under LR. and extends to LF. since our robust geodesics stay closer even under weaker perceptual proxies. REVISE and VSGD often stray off-manifold, producing AEs that appear close under L2 (unsurprisingly, as REVISE directly minimizes this metric) but deviate sharply in all perceptual geometries. RSGD and RSGD-C operate under their metrics, but lack geodescity and remain vulnerable to on-manifold AEs Aestuurturaations smooth under & and L,r yet semantically fragile. These cases highlight that our proposed LR serves more faithful evaluation metric, exposing failure modes that remain hidden under non-robust distances. Low scores in L1, L2. or CF do not guarantee proximity and can coincide with adversarial behavior.
Input
PCG (ours)
RSGD
RSGD-C
VSGD
REVISE
0.91
0.87
0.94
0.89
0.96
cat
dog
wild
cat
0.2
0.89
0.86
0.08
not smile
smile
0.84
0.90
0.93
0,37
bald
hairy
0.91
0.94
0.52
healthy
unhealthy
0.92
0.95
0.55
0.97
0.90
Figure 4: Qualitative comparison of counterfactuals across methods with STYLEGAN2. Columns show input images followed by counterfactuals from PCG (ours), RSGD, RSGD-C, VSGD, and REVISE. Rows indicate input and target /class. PCG produces minimal, semantically faithful changes along robust geodesics, while baselines often show off off-manifold artifacts, semantic drift, or adversarial collapse. Optimization details for baselines are presented in Appendix A.2.
Explanandum-basee Metrics. We quantify realism using the standard FID (Heusel et al., 2018) and its robust variant R-FID (Alfarra et al.. 2022a). As sparsity measure, report representation based COUT following Khorram & Fuxin (2022). For closeness to the original input, we use LPIPS (Zhang et al., 2018) and its robust counterpart R-I.PPS (Ghazanfari al., 2023). For validity,
9

---

Published as a conference paper at ICLR 2026
propose Semantic Margin (SM) metric to evaluate whether generated counterfactuals move into regions of the data space that are genuinely associated with the target class. We further use a Manifold Alignment Score (MAS) to measure how the direction of change between original images and their counterfactuals aligns with manifolds induced by different geometries. Finally, we report flip rate for completeness. All evaluation metrics are detailed in Appendix A.5. We additionally evaluate the smoothness of our counterfactual geodesics (Appendix B.7), scaling of our method with respect image resolution and path length (Appendix B.5 ). ablations on (Appendix B.6), the effect of the selected robust backbone and layer aggregation scheme (Appendix B.8), and more results for classifiers beyond the VGG-19 backbone extended to multiclass classification, and comparisons with counterfactuals from robust models (Appendix B.9).
Table 2 summarizes the explanandum- based metrics discussed above. Under standard FID, PCG attains the lowest score, but standard FID alone is known to be insensitive to adversarial artifacts (Alfarra et al., 2022a). When we move to the robust variant R-FID, the gap widens: PCG remains close to the real target distribution, whereas baselines degrade more strongly, indicating that their improvements in standard FID are at least partly driven by non-robust directions. LPIPS and its robust counterpart R-LPIPS quantify the perceptual displacement between original images and their counterfactuals, PCG achieves the smallest distances, with the clearest separation under R-LPIPS, in line with the fact that our trajectories constructed geodesics in robust perceptual geometry and therefore change content more gradually in robust feature space. The COUT scores follow the same pattern: PCG induces more concentrated. lower-magnitude changes in the classifier's internal representation than baseline methods, rather than diffusing changes across many low-level features. Finally, the mean SM shows that PCG counterfactuals move into regions of robust feature space that are genuinely populated by target-class data (positive margins), while baseline methods more frequently remain mixed or non-target neighborhoods. Finally, MAS scores reveal how the direc- tion of change interacts with different geometries: methods that do not explicitly model geometry (e.g, REVISE, VSGD) exhibit weak alignment or specialize to the geometry they implicitly optimize (e.g, RSGD), whereas PCG achieves strong alignment in the robust feature space it induces, and this behavior generalizes to standard feature geometry, consistent with its construction as a robust percep- tual geodesic. Taken together, these metrics support our central claim that standard FID/LPIPS alone can be overly optimistic in adversarially vulnerable regimes, whereas their robustified counterparts and semantic diagnostics (COUT, mean SM, and MAS) draw out the advantage of semantics-awar robustly induced geodesics over existing latent-space counterfactual methods much more clearly.
Table 2: Evaluation results across realism, closeness, faithfulness, manifold-alignmennn metrics, and flip rate for various methods (STYLEGAN2 on AFHQ).
Realism
|
Closeness
Faithfulness 1
MAS f
Flip
Method
FID
R-FID
LPIPS
R-LPIPS
COUT
Mean SM
Pixel
Standard
Robust
Rate
REVISE
18.5
50.1
0.85
0.67
0.09
-0.48
0.21
0.18
0.14
98%
VSGD
23.5
46.7
0.93
0.79
0.10
-0.14
0.17
0.21
0.19
92%
RSGD
12,9
37.8
0.61
0.68
0.13
0.03
0.82
0.45
0.21
96%
RSGD-C
12.7
28.3
0.59
0.53
0,25
0.05
0.68
0.84
0.47
94%
PCG (ours)
8.3
9.1
0.24
0.17
0.43
0.74
0.65
0.87
0.91
95%
5
CONCLUSION
We introduced Perceptual Counterfactual Geodesics (PCG), method for generating semantically faithful counterfactuals by optimizing smooth trajectories on a latent Riemannian manifold equipped with a robust perceptual metric. Our two-phase framework operationalizes established ideas from pullback geometry and robust perception into a practical algorithm. Empirically, PCG outperforms latent-space baselines and avoids their common failure modes (off- and on-manifold adversarial collapse, semantic drift). In addition, the robust geometry-aware evaluation LR exposes errors that remain hidden under standard distances, providing more reliable yardstick for counterfactual quality, Conceptually, the contribution is algorithmic: we show that when the latent space is endowed with a robust. perceptually aligned geometry and optimized globally along paths, counterfactuals become smooth, diverse, and faithful. Final notes on our scope, limitations, and future work are discussed in Appendix C
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors have read and will adhere to the ICLR Code of Ethics. Our experiments use publicly available vision datasets under their licenses; no new human-subject data were collected, and do not perform re- re-identification or demographic inference. Any released code is intended for research use and will include guidance discouraging harmful or deceptive applications. LLM usage disclosure: in line with ICLR policy, we used a large language model only for light copy-editing (grammar, typos, minor phrasing/formatting) it did not contribute to research ideation, analysis, or claims.
REPRODUCIBILITY STATEMENT
All methodological details, derivations, and hyperparameter settings required to reproduce our experiments are described in the main text (Section 3) and in Appendix A.I, where we also provide pseudocode for our two-stage optimization procedure. Architectural specifications, training protocols for generators, encoders, and classifiers, and additional results (including sensitivity initialization) are included in Appendices A and B. Anonymized source code implementing PCG and all evaluation metrics is provided in Section enable full replication of our experiments.
ACKNOWLEDGMENTS
This research was partially supported by the Australian Research Council through an Industrial Trans- formation Training Centre for Information Resilience (IC200100022), Quan Nguyen is supported by a NHMRC Investigator Grant (GNT2008928), Maciej Trzaskowski the Managing Director of Profenso; this work was conducted independently of Profenso.
REFERENCES
Motasem Alfarra, Juan C. Pérez, Anna Frühstück, Philip H. S. Torr, Peter Wonka, and Bernard Ghanem. On the robustness of quality measures for gans, 2022a.
abs/2201. 13019.
Motasem Alfarra, Juan C. Pérez, Anna Frühstück, Philip H. S. Torr, Peter Wonka, and Bernard Ghanem. On the robustness of quality measures for gans, 2022b. URtps/t:://axx 0tts/2201.13011
André Artelt and Barbara Hammer. On the computation of counterfactual explanations - A survey, November 2019.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv: Machine Learning, 2017.
Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in- and out- distribution improves explainability, 2020. URL https:
200...0000000000 094469466 61.6.
Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein. Diffusion Visual Counterfactual Explanations, October 2022.
Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. Dig-in: Diffusion guidance for investi- gating networks uncovering classifier differences neuron visualisations and visual counterfactual explanations, 2024. URL. http::t/t///c /aappv::///////////////sissasssa www.iiivorrrooo www./oo.////..o...oo.o.........t
rxiv. http//////////oo//////////////// a23/ 13111.781111.....1 17833.
Valentyn Boreiko, Maximilian Augustin, Francesco Croce, Philipp Berens, and Matthias Hein. Sparse Visual Counterfactual Explanations in Image Space, pp. 133-148. Springer International 9. URL http:
Publishing, 2022. ISBN 978303 doi:
//dx. doi. wwo.orgg///// /00g 111110779777----1--7--------8881
Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: A manifold disentanglement perspective. IEEE Transactions on Neural Nenworks and Learning Systems, 27
1197:0997-0088, 2016. doi: 10. 1..09/NNI..20155.4490000.

---

Published as a conference paper at ICLR 2026
Kieran Browne and Ben Swift. Semantics and explanation: Why counterfactual explanations produce adversarial examples in deep neural networks, December 2020.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8188-8197, 2020.
Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. A light recipe to train robust vision transformers, 2023. URL https: //ariiv...iiv.///aaa org/ httgg/bbs/2205 2209.. 07399.
Ann-Kathrin Dombrowski, Jan E. Gerken, Klaus-Robert Müller, and Pan Kessel. Diffeomorphic Counterfactuals With Generative Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3257-33
May 2024. ISSN 1939-3539, doi: 10.1 1...1099PAMM..22333339900
Tri Dung Duong, Qian Li, and Guandong Xu. CeFlow: A Robust and Efficient Counterfactual Explanation Framework for Tabular Data using Normalizing Flows, March 2023.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL httpt::////// /ttpp::///thubb. com Maoo//MdryLab//aa/ /aaaaaayaaabbbbuuuseeeessss wtw.a//rbbuuutss.......
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola Schoenlicb. On the connection between adversarial robustness and saliency map interpretability. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp- 1823-1832. PMLR, 09-15 Jun 1 9a html.
2019. URL
httpsss:/pocceeeddeeee . mlr prwssppsssseeeeeeeppee /vasss/991
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis, 2013.
URL https:tt//// //t:://axxxa /tt///axvivv..///// org/ abs/ 11100. 0425.
Timo Freiesleben. The Intriguing Relation Between Counterfactual Explanations and Adversarial Examples. Minds and Machines, 32(1):77-109, March 2022. ISSN 0924-6495, 1572-8641. doi: 10.1007/s1 1023-021-09580-
Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply adversarial robustness?, 2023a. URL https:tpps.........t. http:/axvvvv.o/// org abs 2207.11377
Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply robustness? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. mlr. prwss / w22//222/22//2/ aaaaaaaaaa.a. aam.m22........
PMLR, 23-29 Jul
2023b. URL https: /ttp://prooooc pttpp/ppceeeeg..eeeeeppp
Washington Garcia, Pin- Yu Chen, Hamilton Scott Clouse, Somesh Jha, and Kevin R.B. Butler. Less is more: Dimension reduction finds on-manifold adversarial examples in hard-label attacks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 254-270, 2023. doi: 10. 10.110//S 555.
Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, and Alexandre Araujo. R-lpips: An adversarially robust perceptual similarity metric, 2023. URL https:
// //aaxiiv.oo. org/ abs 22t//230...0 15157.
Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Farshad Khorrami, and Siddharth Garg. Lipsim: provably robust perceptual similarity metric, 2024a. URL https://arxiv, org//ba/ 2310 18274.
Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Farshad Khorrami, and Siddharth Garg. LipSim: A Provably Robust Perceptual Similarity Metric, March 2024b.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres, 2018.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https: / www....wggbbbe//.ooeoooo 2661.
12

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVP.20016.90. URL https: /ttp:////aaivv///ae org / abs/ 1512. 03385.
Magnus R. Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards, (9(6):40944383 1952. doi: 10.6028/jres. 049.044.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium, 2018. URL
https: /ttp//aaxvvvv//v wwwpgvi.v.rr.a.ooooooooo org/abs/////ooooooooo/ooo//..arr 1706. 1700008500.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: VAE: Learning Basic Visual Concepts with Constrained Variational Framework. November 2016.
David P Hughes and Marcel Salathé. An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv preprint 1 2015. URL
https: / /tw.vv.vvvv..///ava//.a.//aaaava Bergg/bss/10 /1511 1511.08060.....
Andrew Iyas,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial Examples Are Not Bugs, They Are Features, August 2019.
Matt Jordan, Naren Manoj, Surbhi Goel, and Alexandros G. Dimakis. Distortion of Adversarial Examples, February 2019.
Quantifying Perceptual
Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijithenjaronk, Been Kim, and Joydeep Ghosh. Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems, July 2019.
Jürgen Jost. Riemannian geometry and geometric analysis. Springer, 7 edition, 2017.
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. Guided Integrated Gradients: An Adaptive Path Method for Removing Noise, June 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019. URL https: aaxivii org/abs/ 812. 04 9448
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NeurIPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan, 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks, 2021. URL https://arxiv orgg abs/ 2106. 1266.1243 12423.
Simran Kaur, Jeremy Cohen, and Zachary C. Lipton. Are perceptually--alggned gradients a general property of robust classifiers?, 2019a. URL http:///wwwwwwwo.a//a//////s//// 0.4.0
Simran Kaur, Jeremy Cohen, and Zachary C. Lipton. Are peceetuuuull gradients general property of robust classifiers?, 2019b. URL
Markus Kettunen, Erik Härkööen,, and Jaakko Lehtinen. E-lpips: Robust perceptual image simi- larity via random transformation ensembles, 2019. URL https: htttts:://arxiv.orr//bbbb 6.060 03973.
Saeed Khorram and Li Fuxin. Cycle-consistent counterfactuals by latent transformations, 2022. URL ptttss://ariiie otg/abss/////oooooooooooos... 2203 15064.
13

---

10.3389/frai.22222.55555 Published as a conference paper at ICLR 2026
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2022.
Samuli Laine. FEATURE-BASED METRICS FOR EXPLORING THE LATENT SPACE OF GENERATIVE MODELS. 2018.
Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T. Freeman, Phillip Isola, Amir Globerson, Michal Irani, and Inbar Mosseri. Explaining in Style: Training GAN explain classifier in StyleSpace, September 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Arnaud Van Looveren, Janis Klaise, Giovanni Vacanti, and Oliver Cobb. Conditional Generative
Models for Counterfactual Explanations, January 2021.
Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions, 2017. URL
https:/ /tap::://axxeee /ttpvva.avvv////////////////aav owg/oogg//ss.. htwp///b////ao aaas///////// /1705 078744.
Tung Luu, Nam Le, Duc Le, and Bac Le. From visual explanations to counterfactual explanations with latent diffusion. In 2025 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 420-429, IEEE, February 2025, doi: 10.1100/wacv6104/ w0e.:000112.2.000000 2025.0005.. //dx. doi. org 10. 1109 /WWCV6104111111 2025. 00051.
URL http:
Silvan Mertes, Tobias Huber, Katharina Weitz, Alexander Heimerl, and Elisabeth André. GANterfactual Counterfatal Explanations for Medical Non-experts Using Generative Ad- versarial Learning. Frontiers in Artificial Intelligence, 5, April 2022. ISSN 2624-8212. doi:
Martin Pawelczyk, Chirag Agarwal, Shalmali Joshi, Sohini Upadhyay, and Himabindu Lakkaraju. Exploring Counterfactual Explanations Through the Lens of Adversarial Examples: A Theoretical and Empirical Analysis. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pp. 4574-4594. PMLR, May 2022.
Paraskevas Pegios, Aasa Feragen, Andreas Abildtrup Hansen, and Georgios Arvanitidis. Counterfac- tual Explanations via Riemannian Latent Space Traversal, November 2024.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In I Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 11 1335-1144, San Francisco California USA, August
2016. ACM. ISBN 978- 1-54------2222-22 doi: 10.11552939677..939777.
Oleg Rybkin. The reasonable ineffectiveness of mse pixel loss for future prediction (and what to do about it), 2022.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with single (robust) classifier, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization. 2016.
Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features?, 2021. URL https: w/w.xiivvvooo. abs / 22102 02. 12781.
Hang Shao, Abhishek Kumar, and P. Thomas Fletcher. The riemannian geometry of deep generative models, 2017. URL https: / arxia. wwwpxxvvoorgaeeo http://abb///.....a./ 08014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint ArXiv: 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
14

---

Published as a conference paper at ICLR 2026
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by Progressive Exaggeration, February 2020.
Pawan Sinha and Richard Russell. A Perceptually Based Comparison of Image Similarity Metrics. Perception, (6(11))12291122 1281, November 2011. ISSN 0301-0066, 1468-4233. doi: 10. 1068/ p7063.
Oskar Sjögren, Gustav Grund Pihlgren, Fredrik Sandin, and Mareus Liwicki. Identifying and Mitigating Flaws of Deep Perceptual Similarity Metrics, July 2022.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. SmoothGrad: Removing noise by adding noise, June 2017.
Bartlomiej Sobieski and Przemyslaw Biecek. Global Counterfactual Directions, pp. 72-90. Springer Nature Switzerland, November 2024. ISBN 9783031730368, doi: 10. 00010077978-33-31--7333--3.. URL http::/t/// //dx http::/xxxxdddooooo ort. 10000110000000000 / 0001//97777-301-73333----------- 73036-8
Bartlomiej Sobieski, Jakub Grzywaczewski, Bartlomiej Sadlej. Matthew Tivnan, and Przemyslaw
Biecek. Rethinking visual counterfactual explanations through region constraint, 2024. URL
attps://arxee org/ abs 2410. 12591.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models, 2018. URL https: / //arxiivv..////////////////avavr org/ abs/aa/5..11111
Suraj Srinivas, Sebastian Bordt, and Himabindu Lakkaraju. Which Models have Perceptually-Aligned Gradients? An Explanation via Off- Robustness.
Suraj Srinivas, Sebastian Bordt, and Hima Lakkaraju. Which models have perceptually-aligned gradients? an explanation via off- off-manifold robustness, 2024. URL https::/arxii...r
23os/205..r 19101.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generaliza- tion, 2019. URL https: /aaxiviiiiiv////a///aaa//aaaavva org/ /1bb/ 00740.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks, June 2017.
Joshua Tenenbaum, Vin de Silva, and John Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 29065500)2219-23320 2000. doi: 10. 20.1126scciece. 290.5500.2319.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy, 2019.
Berk Ustun, Alexander Spangher, and Yang Liu. Actionable Recourse Linear Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 10-19, January 2019. doi: 10. 145/3288756032285566
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR. SSRN Electronic Journal, 2017. ISSN 1556-5068. doi: 10.2139/ssm..003288)
Nina Weng, Paraskevas Pegios, Eike Petersen, Aasa Feragen, and Siavash Bigdeli. Fast diffusion- based counterfactuals for shortcut removal and generation, 2024. /tlLtttss//acxiv.oor
bbbs/23122 142233.
Eslam Zaher, Maciej Tizaskowski, Quan Nguyen, and Fred Roosta. Manifold integrated gradients: Riemannian geometry for feature attribution. In Proceedings of the 4 1t International Conference on Machine Learning. ICML'24 '24. JMLR.org, 2024.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric, 2018. URL https: http/xxiivvvooo/a abss/ 1801 03924.
15

---

Published as a conference paper at ICLR 2026
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural networks, 2019. URL https: / /ttp/aaavvvv/..////v//eee//..at. org wttpaaa............. 09797.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples, 2018, URL https://arxiv, wwtpoos///////a//e 11342.
16

---

Published as a conference paper at ICLR 2026
A FURTHUR DETAILS ON THE PCG ALGORITHM, BASELINES, MODELS, AND METRICS.
A.1 PCG OPTIMIZATION
Our objective minimizes the discrete robust perceptual energy of a latent trajectory under the pullback geometry. Because we differentiate the energy itself (squared feature increments along h& backprop through hgg and g automatically inserts the Jacobian factors that define the pullback metrie. Two implications follow. First, in Phase (energy-only), standard gradient descent already converges a manifold-conforming geodesic for the path variables, so a Riemannian correction brings no additional benefit. Second, in Phase we add a classification term that touches only the endpoint 2r; while one could Riemannian- correct that update in isolation, is unnecessary in our coupled objective: the energy term continues to regularize all latent points (including 2T), steering the entire trajectory to remain a counterfactual geodesic.
PCG proceeds in two phases. The first constructs smooth geodesic path between the input and target-class sample, optimized for 200 steps with fixed learning rate of 1e-3. The second refines the path into a faithful counterfactual over 300 steps, using the same learning rate and a dynamic À schedule: starting from 1e-4 and multiplying by 5 every 50 steps. At each such interval, we apply re-anchoring strategy: the path endpoint is reassigned to the closest point to the input along the trajectory that is classified as belonging to the target class. We then increase the resolution of the path by inserting midpoints between each pair of consecutive latent codes, restoring the original path length. Optimization resumes to refine the updated path, progressively giving closer counterfactuals. For completeness, the PCG optimization pseudocode given Algorithm 1.
A.2 BASELINES OPTIMIZATION
To ensure comparability, all baselines start from the same initialization 20 the same encoder-gent generator pair, and are optimized for the same number of steps as PCG (200+ + +300). We use Adam (Kingma & Ba, 2015) and select the step size by a small sweep E (le-4, 3e-4, 1e-3, 1e a held-out split; we report the best setting per method.
c(x*) use
VSGD, Vanilla latent descent minimizes only the classification loss (([(g)))),)) (no similarity term, no A). We run Adam with the learning-rai raae sweep above.
REVISE. We optimize ((())) + A ((((()()), y²) in latent space. For fairness and due diligence, À follows the same dynamic schedule used in PCG Phase (start le-4, x5 every 50 steps). We use the same Adam sweep for 17.
RSGD/-C. These variants require the inverse of the induced latent metric. We compute the natural- gradient direction by solving G2(2) r = v,L L with Conjugate Gradients (Hestenes & Stiefel, 1952), using Jacobian-vector products via autodiff; this avoids explicit Jacobian assembly and matrix inversion. Since the original code targets VAEs on tabular data and not public, we implement a deterministic metric compatible with our GAN setting (pixel fa pullback for RSGD; classifier-feature pullback for RSGD-C) and apply the same Adam step-size sweep for the outer update.
A.3 GENERATORS, ENCODERS, AND CLASSIFIERS
Style-based generator (image prior). We use the official StyleGAN2-ADA (and, where noted, StyleGAN3) implementations as our image prior. The generator provides a smooth latent manifold on which we optimize trajectories; we do not introduce architectural modifications beyond standard configuration cessolution/wcights)
Image latent encoder (inversion). To place real images on the generator's latent manifold, train a lightweight encoder that maps an input image to a single latent vector compatible with the generator's is input space. Its role is purely representational: enable mapping for endpoints and faithful reconstructions; exact layer choices are not critical to the method.
Discriminator (training-onlyy, When (re)training generator, bundled with the official StyleGAN repositories. It is only a training counterpart- never used by our optimization or evaluation procedures.
the standard discriminator
17

---

Published as a conference paper at ICLR 2026
Task classifiers (decision function f). For each dataset/attribute, image classifier (e.g.. VGG-19 from Torch Vision) as the decision function whose prediction we seek to change. These models are straightforward baselines chosen for familiarity and availability; they are not part of the perceptual metric.
conventional supervised
Robust backbones (perceptual geometry & evaluation). To define robust perceptual metric and for geometry-aware, we rely on adversarially trained ImageNet backbones sourced from public robustness libraries (Engstrom et al., 2019; Debenedetti et al., 2023). These networks are used only to induce perceptually aligned geometry and to score distances; they are distinct from the task classifier f.
Why these choices. The generator supplies a strong visual prior (manifold parameterization), the encoder puts real data on that manifold, the classifier defines the target decision boundary, and robust backbones define a perceptually grounded geometry. This separation lets us optimize counterfactual paths on high-quality manifold while keeping the decision function and the perceptual metric decoupled.
Requirements for each method. Tables and summarize practical requirements and optimization burden. All methods require a generator ! and (for real images) encoder e; only PCG additionally uses a robust backbone to induce the perceptual geometry. Unlike RSGD variants, PCG does not perform metric inversion (no CG solves), which keeps its runtime below RSGD/RSGD-C despite being path-based; qualitatively it is "Medium," " while RSGD and RSGD-C are "High" and "Highest," respectively. REVISE and VSGD remain the lightest due to single-point Euclidean updates without metric operations.
Table 3: Component requirements by method. "Yes/Optional" means the encoder is needed for real-image inversion but optional for synthetic latents.
Method
Generator g
Encoder
Classifier f Robust backbone
PCG (ours)
Yes
Ycs/Optional Yes/Optional
Yes
Yes
RSGD-C
Yes
Yes
RSGD
Yes
Yes/Optional
Yes
No
REVISE
Yes
Yes/Optional
Yes
No
VSGD
Yes
Yes/Optional
Yes
No
Table 4: Optimization and compute summary. "Metric inversion" refers to solving Gz(z)r= V.C
(e.g., via Conjugate Gradients).
Method
Optimization Style
Metric inversion
Relative compute
PCG (ours)
Path optimization (two-phase: energy then energy+els) Single-point Riemannian descent (feature-space pull- back)
No
Medium
RSGD-C
Yes (CG)
Highest
RSGD
Single-point Riemannian descent (pixel-space pullback)
Yes (CG)
High
REVISE
Single-point Euclidean descent (distance + cls)
NA
Low
VSGD
Single-poing Euclidean descent (cls only)
NA
Lowest
A.4 METRIC COMPOSITION, ROBUST BACKBONES, AND SMOOTHNESS
Backbone choice. We instantiate the perceptual geometry using adversarially trained ImageNet backbones that supply perceptually aligned, manifold-confome Zhang & Zhu, 2019; Tsipras et al., 2019; Ilyas et al., 2019; Stutz et al., 2019). Our default induced geometry in the main text uses L2-robust ResNet-50 trained ImageNet with = 3.0 (Engstrom et al., 2019), and in the appendix we report analogous results using an -g-rrbsst XCTT-S12 vision transformer trained under the same threat model (Debenedetti et al., 2023). These networks only to induce the metric; they are never the same model as the task classifier f. and their weights remain frozen throughout. The choice of robust backbones is motivated by concrete theoretical and
gradients (Ganz et al., 2023b;
used
18

---

Published as a conference paper at ICLR 2026
empirical results in adversarial robustness and Perceptually Aligned Gradients (PAGs). Etmann et al. and fo work show that adversarially trained models produce saliency maps that are more strongly aligned with human salient structure and suppress righ-frequency, non-salient directions (Etmann et al., 2019; Zhang & Zhu, 2019; Tsipras et al., 2019; Ilyas et al., 2019). Srinivas et al. provide theoretical account via off offffoanifold robustness, showing that when a classifier is trained to be more robust off the data manifold than on it, its input gradients are forced to lie approximately in the tangent bundle of the data manifold (Srinivas et al., 2024). This mechanism explains why robust models tend to have gradients that follow intrinsic manifold directions rather than adversarial spikes orthogonal to the data, and is consistent with the broader PAG literature where robust models exhibit gradients that align with human perceptual judgments (Ganz et al., 2023a; Kaur et al., 2019a; Ganz et al., 2023b).
Composite pullback metric (layer aggregation). Our composite perceptual metric is constructed by pulling back Euclidean metrie from a robust feature space through the generator. In differential- geometric terms this is the standard pullback construction: Riemannian metric the feature space induces a Riemannian metric on latent space via the generator map. Infinitesimal moves in the latent space z are measured according to how they change robust features: directions along which robust representations vary smoothly and semantically incur low cost, while directions that robust training suppresses- such as frguurreyeyccy or adversarial perturbations-and assigned high cost. Geodesics under 92 are therefore strongly biased to follow manifold-aligned, perceptually meaningful directions that the robust model uses internally, rather than the brittle directions preferred under Euclidean pixel distances or non-robust features. This picture mirrors the literature on robust perceptual similarity, which shows that distances in robust feature spaces correlate more strongly with human similarity judgments and are less vulnerable to adversarial manipulation than their standard counterparts (Kettunen et al., 2019; Ghazanfari et al., 2024a). Kettunen et al. demonstrate that LPIPS-type metrics are themselves vulnerable and can be rectified into a robust variant by using robust representations (Kettunen et al., 2019), while Ghazanfari et al. construct a 1-Lipschitz perceptual similarity metric with provable robustness guarantees and show that robustness in feature space improves both adversarial stability and alignment with human perceptual similarity (Ghazanfari al., 2024a). Together with the manifold-aligamenn results above, these works support the view that infinitesimal moves in robust feature space correspond to smooth, semantically coherent deformations along the data manifold.
In practice, we instantiate the feature map as a composite feature map obtained by aggregating intermediate representations at multiple depths of an La-robust ResNet-50 with s = 3.0 and, in the appendix, an L2-robust XCTT-S12 backbone with the same threat radius. For the robust ResNet-50, we concatenate activations from the stem (layer 0) and all four residual stages (layers 1-4), and normalize each stage with simple scalar weights so that no single block dominates the metric. For the robust XCTT-S12, we follow the same principle and aggregate embeddings from early-to-mid blocks and mid-to-deey blocks, as well as a multi-block configuration that spans early and deep layers; features from different depths are rescaled and concatenated into a single representation. This composite robust pullback metric is the default geometry used by PCG. For comparison, we also define a "standard" pullback metric based on the same layer-aggregation scheme but using a standard (non-robust) ResNet-50 backbone. Across all backbones, the role of robustness is to supply feature spaces and gradients that are better aligned with the data manifold and human perception (Zaher et al., 2024; Srinivas et al., 2024; Ganz et al., 2023a; Kaur al., 2019a; Ganz et al., 2023b; Kettunen et al., 2019; Ghazanfari et al., 2024a; Zhang & Zhu, 2019; Tsipras et al., 2019; Ilyas et al., 2019; Stutz al., 2019), while the pullback construction translates this structure into latent-space geometry that PCG can exploit.
19

---

Published as a conference paper at ICLR 2026
Algorithm Perceptual Counterfactual Geodesics (PCG)
Input image x², target class y/, encoder e, generator 9, classifier f Require: Robust feature maps {4x 2mm1 path length T, Phase-l steps Si, Phase-2 steps S2
Require:
Require:
Learning rate y/ loss weight schedule (Aa } re-anchoring period P
function ROBRGYZN (.,,
2: öt < 1/T, E t- 0
3: for i 0
do
4:
for k = 1 to K do
5:
Ugk: < (g (2)1(()))))))))))))))))))))))))) hz(((z)))
6:
E <- E
7: end for
8: end for
9: return E
10: end function
Initialization:
12: Ag 4- e(x*)
Choose a target-class sample Zigt with arg max (Zigt
14: 2r & -e((Tgg)
15: Initialize {=1 T- by linear interpolation between 26 and 2T
16: Phase 1: Robust geodesic with fixed endpoints
17: for
to Si do
18:
E <- ROBUSTENERGY ROBUSTENEERGY(EA..... 2T])
19: Compute V 2y E by backprop
for i 1 T
do
21:
2; -
V
22: end for
23:
end for
24: Phase 2: Endpoint-aware refinement under classification constraint
25: for = 1 to S2 do
26:
E + ROBUSTENERGY((a,..
27:
Less + ([((((2)))
28:
L F X₂ Let
29:
Compute V .1.... 2y by backprop
for
T do
31:
2; +
V L
end for
33:
+ 21 V 24
D endpoint update re-anchoring
34:
if mod
then Re-anchor to the closest point along the path classified as y' Densify path by inserting midpoints and resampling to T++1 points
35:
36:
37:
end if
38:
end for
39:
Return final path ......, 27l and counterfactual Ter = -(2T)
Smoothness. To ensure the induced metric varies smoothly, we replace non-smooth ReLU variants in our models with Softplus post hoc (after training). In practice this does not materially change behavior, as activations typically operate in smooth regions; it only guarantees that the metric field is differentiable along the paths optimize.
20

---

Published as a conference paper at ICLR 2026
A.5 MORE DETAILS ON EVALUATION METRICS
We briefly collect the definitions, motivations, and implementation details for all evaluation metrics used in the main text.
FID and R-FID (realism). To assess distribution-level realism, the Fréchet Inception Distance (FID), where lower values indicate that the generated distribution is closer to the real one in the chosen feature space.
In our setting, the real set comprises images from the target class in the training data, and the generated set comprises the corresponding target-class counterfactuals. As discussed in the main text, standard FID can be insensitive to non-robust or adversarial directions exploited by generative models. We therefore also report robust variant, R-FID []. obtained by replacing the model used for FID with its robust. The functional form of the metric is identical; only the feature space changes. Intuitively, improvements that rely on non-robust directions tend be reflected in standard FID but are penalised by R-FID, which is more tightly aligned with perceptual and semantic structure.
LPIPS and R-LPIPS (closeness). To quantify closeness between an original image Zorig and its counterfactual Zet we use the Learned Perceptual Image Patch Similarity (LPIPS). Lower values under this metric correspond smaller perceptual changes.
Analogously to FID/R-FID, we also use R-LPIPS that replaces the backbone with its robust coun- terpart. This yields a perceptual distance that is more stable under adversarial perturbations and better aligned with robust feature distance. In our context, LPIPS and R-LPIPS play complementary roles: they measure instance-level closeness between Forig and Tef, whereas FID/R-FID capture distribution-level realism.
COUT (representation-leve sparsity). Following the spirit of COUT from Khorram Fuxin (2022), we measure how focused the change in the explained classifier's internal representation is relative to the change in its belief about the target class. Let f X + RC be the classifier we aim to explain, with logits f fx)) and target class Yigt, and let h X Rd denote fixed internal representation of (penultimate layer). For a counterfactual pair (Torig: (eet), we define
A Aig -mn (wer) Sena (Seorig);
Ah =h(wer)) h(
The COUT score is then
Sigt
COUT xa)
(4)
|Ah +
where is a small constant for numerical stability. Higher COUT indicates that the method obtains a given increase in target-class confidence with smaller, more concentrated change in the explained model's 's own internal representation, which is desirable for counterfactual explanations that aim for targeted, rather than dispersed, changes. We adopt this representation-leewe analogue instead of the original pixel-curve COUT for two reasons. First, our setting is explicitly adversarially vulnerable: pixel-space perturbation curves can look favourable even when the underlying trajectory exploits non-robust directions, whereas measuring sparsity in probes how efficiently the counterfactual steers the actual decision-maid makinng features of f. Second, our framework is centred on feature and latent geometries rather than input-space masks, and COUT defined in h integrates more naturally with this viewpoint and is computationally lighter than evaluating per-pixel perturbation paths. Robustness and manifold faithfulness are handled separately by our semantic-marge margin and manifold- alignment diagnostics, which are computed an independent robust feature space; COUT is thus read as an explanandum-relattve efficiency measure that complements, rather than replaces, these robustness-aware metrics.
Semantic margin (SM: semantic locality in robust feature space). To probe whether counter- factuals move into regions of feature space genuinely associated with the target class, we use a semantic margin defined in separate robust feature space. Let yp X - R 2 be robust backbone (Inception- V3) that is not used to construct the PCG geometry. Let {(x,, y: 9:) M denote the labelled training set and y sxxz)).
21

---

Published as a conference paper at ICLR 2026
For counterfactual Zef with target class Vigt, we define the sets of k nearest neighbours in sp-pe
Nagg(Fet)) =
arg min (cg iES 81=k
Σ
1|x( - P0ll22
(5)
Nother(Fet) =
arg argmin & iES
Σ [1o(xet) - Poll2-
(6)
8l=k
We then define class-conditional average distances
degt (Fett)
Σ
( 8:ll2
Nother(Fe))
Σ iENotner (Fer)
[| - Gill2 (7)
(xar)
and the semantic margin
m(xet) = dother (Iet) - dego(Tet).
(8)
Intuitively, mxxr)) > means that Zef is, on average, closer (in robust feature space) to target-class training examples than to non-target examples; m (xxet) S suggests that the counterfactual resides in a mixed or non-target neighbourhood and is therefore suspect from manifold perspective (potentially off- on-manifold AE). We report the mean semantic margin across all counterfactuals for each method in the main text. In our experiments, we set A 16.
Manifold Alignment Score (MAS: tangent alignment under different geometries). To quantify whether counterfactual updates follow the tangent structure induced by different geometries, we introduce Manifold Alignment Score (MAS). For each geometry g we consider representation map
fg Z R", obtained by composing the generator G: - X' with an appropriate embedding:
fpix(2) = G(z)
(pixel geometry),
(10)
[sd(()) = (G((z))
(standard feature geometry),
(11)
frob(2) = (G((2)
(robust feature geometry).
(12)
= Let Forig and žef denote the latent codes of the original and counterfactual images, so that Worig G (Zorig and Tef = G(Zet). For geometry g. we define the ambient displacement
Ug = fy(Ee) - fg(2orig)
E RP*
(13)
[[a( (
Using the Jacobian Jy( (Zorig) = (z) smorii"
we construct an orthogonal projector Po (Zooi)))
onto the tangent space of the corresponding manifold at fz( for example via
Pg (Zorii) Jg (Zarig) (Jz(Zorig) J (F((rr)))) J Jg(Zorig)"
(14)
The MAS for a pair (Torig: Tat) under geometry is then
8g (yoorrig Tee) = Pa (Zorig 2
E [0.
(15)
|0g113
This score measures the fraction of the squared norm of the counterfactual displacement that lies in the tangent space induced by geometry g. High scores indicate that the update is predominantly tangent maanifold-aligneee whereas low scores indicate a large normal component (off-manifold soometty--issligneee We report averages of sy over all counterfactuals for each method and geometry.
Path-based LPIPS and R-LPIPS (geodesic smoothness). To assess the smoothness of counterfac- tual trajectories, we use path-bases perceptual metrics centred around the PCG geodesic and compare them to standard latent-space interpolations. Given discrete path {2a )r between Forig and a (PCG geodesic, linear latent path, or spherical latent path), we define the average LPIPS-step as
T-1
ALPIPS T
LPIPS( (11,)++)),
(16)
and analogously AR-LPIPS by replacing LPIPS with R-LPIPS. Smaller values indicate more gradual perceptual change along the path.
22

---

Published as a conference paper at ICLR 2026
B MORE RESULTS & ANALYSIS
B.1 INTERPOLATION RESULTS
Figures and compare straight-line interpolations under four geometries based on STYLEGAN3. From top to bottom in each panel: (i) Z-linear interpolation (flat latent space), (ii) pixel-space MSE pullback (AMsE), (iii) standard feature pullback (FMsE), and (iv) our robust perceptual pullback (RMSE). The robust metric produces smooth, on-manifold transitions with consistent semantics (identity/pose for faces; class coherence for animals), while Z-lerp and pixel MSE exhibit mid- trajectory artifacts and blends. The standard feature pullback improves semantics but still suffer from similar failure modes. These visuals mirror the trends discussed in the main text and motivate using a robust geometry for PCG.
2 lerp
MSE
A'
R MSE F MSE
Figure 5: Interpolations on FFHQ under four geometries. Rows (top to bottom): Z-lerp, XASE pullback, FMSE pullback, and robust RASE pullback. The robust shows smooth, semantically consistent evolution (e.g., gradual attribute change without identity drift), whereas the other geome- tries introduce off-manifold blends and texture/ilumiination artifacts mid-path.
Z eerp
MSB
F MSE
R MSE
Figure 6: Interpolations on AFHQ under four geometries. Same ordering as Fig. 5. The robust RMSE path preserves class coherence and yields clean transitions, while Z-lerp and XMSE produce ambiguous hybrids and brittle textures; FMSE reduces but does not eliminate these effects.
23

---

Published as a conference paper at ICLR 2026
B.2 PERCEPTUAL COUNTERFACTUAL GEODESICS ACROSS AFHQ AND FFHQ
Figures 7 (AFHQ, two examples) and (FFHQ, two examples) visualize the two-phase PCG procedure with STYLEGAN3. In each panel, the top row the initial linear path in Z (straight interpolation between the encoded input and target exemplar), which often drifts off-manifold or blends semantics mid-trajectory. The middle row is the Phase robust geodesic with fixed endpoints: transitions become smooth and class-consistent. The bottom row is the Phase 2 counterfactual geodesic, where the endpoint is jointly refined with the classification loss; the endpoint moves closer to the imput while achieving the target class/attribute, and the entire path remains on-manifold. Qualitatively, AFHQ preserves species structure and textures, while FFHQ preserves identity and pose attributes change, supporting the claims about semantic fidelity and geometry-awan paths.
Initial linear Path
Resulting Geodesic from Phase 1
Counterfactual Geodesic from Phase 2
Initial linear Path
Resulting Geodesie from Phase
Counterfactual Geodesic from Phase 2
Figure 7: PCG on AFHQ (STYLEGAN3). examples (Cat + Dog Wild ++ Dog). Rows (top to bottom): initial linear path in Z between the encoded input and target exemplar; Phase robust geodesic (energy-only) with fixed endpoints; Phase 2 counterfactual geodesic after endpoint refinement with classification loss. The geodesic rows remove mid-path blends and keep species-level semantics while reaching the target class.
24

---

Published as a conference paper at ICLR 2026
Initial linear Path
Resulting Geodesic from Phase
Counterfactual Geodesic from Phase
Initial linear Path
Resulting Geodesic from Phase
Counterfactual Geodesic from Phase 2
Figure 8: PCG on FFHQ (StyleGAN3), two examples (Glasses - No-glasses & Female > Male). Same layout as Fig. 7. Phase 1 produces smooth, on- on-manifold transitions; Phase moves the endpoint toward the input while satisfying the target classifier. Identity and pose are largely preserved as the target attribute changes, and intermediate frames remain perceptually coherent.
B.3 SENSITIVITY TO DIFFERENT TARGET CLASS SAMPLES
Figures and test how PCG depends on which target-class exemplar is used to initialize the path. For each input run PCG twice, once per exemplar. We observe that the Phase geodesic reflects the chosen exemplar (different coarse routes latent space), but after Phase 2 (endpoint refinement with classification loss) the counterfactual geodesics converge to tight neighborhood around the input while achieving the target label/attribute. This yields diverse yet faithful counterfactuals and supports the main-text claim about robustness target initialization.
25

---

Published as a conference paper at ICLR 2026
Initial Geodesic from Phase for Target Class Initialization
Resulting Counterfactual Geodesic
Initial Geodesic for Different Target Initialization
Resulting Counterfactual Geodesic
Figure 9: Sensitivity to target exemplar on AFHQ (StyleGAN3) (Cat + Wild). Rows: (1) Phase geodesic initialized with target exemplar A, (2) resulting Phase counterfactual geodesic, (3) Phase geodesic with a different exemplar B. (4) resulting Phase counterfactual geodesic. Although the Phase routes differ, the Phase 2 counterfactuals converge near the input and satisfy the target class, indicating low sensitivity to the exemplar choice and producing diverse but faithful variations.
Initial Geodesic from Phase for Target Class Initialization
Resulting Counterfactual Geodesie
Initial Geodesie for Different Target Initialization
Resulting Counterfactual Geodesic
Figure 10: Sensitivity to target exemplar on FFHQ (StyleGAN3) (Blonde - Non-blonde). Same layout as Fig. 9. Two different target exemplars lead to distinct Phase 1 paths, yet the Phase counterfactual geodesics converge to a small neighborhood around the input while achieving the target attribute; identity and pose remain largely preserved.
Quantitative sensitivity (multiple initializations, M=15 per input). To quantify low sensitivity to target initialization, PCG M=15 times per input with different target exemplars and measure
26

---

Published as a conference paper at ICLR 2026
how close and consistent the resulting counterfactuals (CFs) are. We use LPIPS and its robust variant R-LPIPS, and report three intuitive, scale- aware metrics: (i) CF dispersion ratio (CDR) - how tightly CFs cluster compared to typical variation within the target class, (ii) Initialization-output contraction ratio (IOCR) - how strongly optimization contracts the diversity of target initializations into tight CF cluster, and (iii) CF diameter the worst-case dissimilarity among the CFs. All three metrics are computed both in standard LPIPS space and in the robust feature space underlying R-LPIPS.
be the CF endpoints for one input x", with M = 15. Let T = f ( denote the corresponding target-class initializations used initialise the path endpoints. Let -) denote perceptual distance, instantiated either as LPIPS or R-LPIPS, Let degt be the average d-distance between random pairs sampled from the targer class (estimated once per dataset/attribute using 30 random pairs).
Definitions. Let C = (
(1)
of af (A1)
(I) CDR (CF dispersion ratio):
der M(M - 1)
2
(m)) 2 Edad
CDR
dcr
degt
m<n
CDR <> indicates CF' form a cluster much tighter than typical target-class variability.
(2) IOCR (Initialization-ouuluu contraction ratio): First, measure the average dispersion of the initial target exemplars:
2 M(M - 1)
dini
Σ
d Azige
We then define
TOCR der
dinit Here IOCR < means that optimization contracts diverse set of target initializations into much tighter CF cluster (strong insensitivity to the choice of target exemplar), whereas TOCR x indicates that CFs are about as diverse the initial targets.
(3) CF diameter:
Diamer max d (m) (n) Tee : F
mnn
so a small value guarantees even the most dissimilar CFss among the M runs remain close. We report all three quantities under both d = LPIPS and R-LPIPS to jointly capture sensitivity in standard and robust perceptual feature spaces.
Sensitivity summary (AFHQ and FFHQ) based on STYLEGAN2. Using the LPIPS- based metrics defined above, Table 5 reports the CF dispersion ratio (CDR), the Initialization--uuppu contraction ratio (IOCR), and the CF diameter for M=15 target initializations per input (mean + std). For each dataset/task, CDR is the intra-CF mean perceptual distance normalised by target-class baseline dugt computed from 30 random target-class pairs: by construction, CDR corresponds to the variability two random target-class samples, while CDR < 1 indicates that CFs form a much tighter cluster than generic target-class variation. IOCR compares the intra-CF dispersion to the dispersion of the initial target exemplars for that input; here 1OCR = means the CFs are about diverse the initial targets, whereas IOCR- < 1 indicates that optimization contracts diverse set of initial targets into tighter CF neighborhood. CF diameter is the maximum pairwise distance among the 15 CFs and captures the worst-case gap within the cluster. All three metries are evaluated both in standard LPIPS space and in the robust R-LPPPS feature space. In all cases, lower is better: tighter clustering (CDR), stronger contraction of initial diversity (IOCR), and smaller worst-case separation (diameter).
For all AFHQ and FFHQ tasks, we observe CDR & 1, showing that CFs produced from different target exemplars lie in cluster that is several times tighter than typical target-class variability, consistent with our claim that PCG converges on a stable, input-specific counterfactual neighborhood rather than scattering across the class. IOCR values well below 1 further indicate that optimization strongly contracts the diversity of the initial target exemplars into this neighborhood, i.c. the final counterfactual is largely insensitive which target exemplar was used. The small CF diameters
27

---

Published as a conference paper at ICLR 2026
confirm this in the worst case: the most dissimilar CFs remain close in both standard and robust perceptual feature spaces, The same trends hold under LPIPS and robust counterpart R-LPIPS (denoted LPIPS and R-LPIPS in the table), indicating that this insensitivity is preserved when measured a robustness-aware feature space.
Table 5: Sensitivity to target initialization (15 runs per input).
CDR
IOCR
CF Diam.
LPIPS
R-LPIPS
LPIPS
R-LPIPS
LPIPS
R-LPIPS
AFHQ: cat -+ dog
0.21 + 0.08
0,16 0.05
0.23 + 0.03
0.17 + 0.08
0.17 + 0.03
0.15 0.04
FFHQ not-smile smile
0.22 + 0.05
0,18 0.07
0.28 1: 0.02
0.21 +: 0.03
0.13 4: 0.05
0.12 0.06
FFIQQ hairy
0.02
0.17
B.4 QUANTITATIVE RESULTS BASED ON STYLEGAN3
As in the main text, PCG consistently achieves the lowest values under the geometry- metrics LF and CR and remains competitive under pixel metrics. These appendix results, obtained on STYLEGAN3, show that the robust geodesic formulation retains its advantage without re-tuning and confirm the stability of PCG's behaviour across model choices.
Table 6: Quantitative comparison across datasets.
AFHQ
FFHQ
Method
4
L2
LF
LR.
L1
C2
DF
CR.
REVISE
1.18#0.12
0.72+0.17
1.05+0.10
2.6820.04
0.81+0.07
0.33#0.12
0.81#0.09
2.75+0.06
VSGD
1.30=0.11
1.48=0.15
1.57+0.09
2.88+0.08
0.78+0.11
0.9520.10
1.49=0.12
2.83+0.08
RSGD
0.8420.08
1.3020.09
0.68+0.07
1.8340.05
0.6010.05
0.8320.07
0.60#0.04
2.3940.05
RSGD-C
0.9240.10
1.43#0.16
0.6340.08
1.7340.06
0.67+0.06
0.9140.09
0.4740.04
2.08
PCG (ours)
0.78+0.07
1.13#0.10
0.51+0.06
0.30+0.02
0.41#0.03
0.71
0.38+0.05
0.21
B.5 RUNTIME COMPLEXITY SCALABILITY ON AFHQ
On AFHQ, measured on single NVIDIA H100 GPU, Table 7 reports per-sample wall-clock runtimes and speedups across methods based on STYLEGAN2. VSGD the fastest (1.6 min). PCG runs 3.4 min per sample despite being path-based (here 7=10): with GPU, all path nodes and robust-feature evaluations are batched (5.7 min) because each step requires solving Gz(2)r V.C with Conjugate Gradients; the inner CG iterations and repeated Recobian-vector products through (and, for RSGD-C, the feature backbone) dominate wall-clock. Absolute times depend precision and batch sizing, but the relative ordering was consistent across runs.
single forward/backward. so the extra cost is modest. RSGD is slowest
Table 7: AFHQ per-sample wall-clock runtime (minutes). RSGD serves as representative for RSGD/RSGD-C. VSGD represents standard Euclidean-gee Buclidean-gradient methods.
Method
Time (min)
Speedup RSGD Notes
VSGD (rep. Euclidean)
3.56x
Classification loss only; lowest cost.
PCG (ours)
3.4
1.68x
Path-based with T=10 nodes; nodes batched on GPU.
RSGD (rep. RSGD/-C)
5.7
1.00x
Natural-gradien via CG: Jacobian- prod- ucts dominate.
28

---

Published as a conference paper at ICLR 2026
Trade-off between Path Length and Smoothness. PCG parameterizes counterfactual trajectory as a discretized path with T points between Forig and zef- Increasing T refines the discretization of the underlying robust geodesic: more points allow the optimizer to distribute semantic change across more intermediate states, but also increase compute and memory usage because each point carries its own latent code, generator activations, and robust features.
Scaling of time and memory with T. Table 8 reports runtime and peak CUDA memory for 512 x 512 images on a single GPU as we vary E (10, 15, 20, 25). Empirically, wall-clock time grows approximately linearly with T: going from 1'=10 to T-20 roughly doubles path length and increases runtime from 3.4 to 7.2 minutes (~ 2.1x), while T-22 yields 10.9 minutes (~ 3.2x the cost of T=10 for a 2.5x longer path). This reflects the fact that each additional node contributes its own set of forward and backward passes through the generator and robust backbone. Peak CUDA memory also increases with T, but more gently: from 23.4GB at T=10 27.3GB at 7 (an increase of ~ 17%). Here, peak memory reflects the joint footprint of all components needed for our setup (pretrained generator and encoder, the classifier to be explained, the robust backhone used to induce geometry, and the activations and latents required to run PCG), not just the path itself.
Scaling of time and memory with image resolution. At fixed path length (T=10), PCG also scales smoothly across image resolutions (Table 9). Runtime increases roughly in line with the number of pixels: moving from 256² to 512² images doubles the resolution in each dimension and approximately doubles wall-clock time (from 1.7 to 3.4 minutes), while going from 512 to 1024 yields a further ~ 2.3x increase (from 3.4 to 7.8 minutes). Peak CUDA memory grows more substantial components (latents, robust features,
moderately- from 19.8GB at 256 23.4GB at 512 and 27. 27.7GB fraction of the footprint comes from csolution-independeee and model parameters), with higher resolutions mainly contributing larger generator and classifier activations in the ambient image space. In practice, this means PCG remains feasible up to 1024² on a single 24 40GB GPU, with runtime being the primary limiting factor at very high resolutions rather than memory exhaustion.
Effect of T on perceptual smoothness. To quantify the smoothness side of this trade-off, we measure the path-based average LPIPS-step and its robust counterpart, ALPIPS and AR-LPIPS (see Appendix A.5 for definitions), for different path lengths T E {3. 5, 8, 10, 15, 20) while keeping all other hyperparameters fixed. The resulting values are visualized in Figure 11. We observe clear pattern: very short paths (7=3 and 7= exhibit noticeably larger ALPIPS and AR-LPIPS, indicating that the class change is implemented via a small number of relatively large perceptual jumps. Increasing the path length to 7'=8 and T'= 10 yields a substantial reduction in both metrics, as the optimizer can resolve the robust geodesic with a finer discretization and distribute semantic change more evenly along the trajectory. Beyond 1-10 the gains become marginal: T=15 provides a small additional improvement in ALPIPS and AR-LPIPS, and 2-20 essentially plateaus, with only minor decreases in step size despite the increased cost.
Practical operating range and comparison to RSGD. Taken together, the runtime and path- smoothness measurements provide concrete view on the trade-off. Short paths are computationally cheap but produce perceptually coarser trajectories; very long paths yield only slightly smoother geodesics at a near-linear increase time and non-trivial increase in peak memory. In our experi- ments, 7-=810 emerges as a practical operating range: it captures the majority of the smoothness gains observed when increasing T. while keeping runtime and memory within a comfortable budget on single high-memory GPU. When resources permit and extremely fine-gr mne-grained trajectories are desired, 1515-20 can be used, but beyond this range we expect the cost to dominate the marginal improvements in ALPIPS and AR-LPIPS. For reference, at 100 and 5122 resolution, PCG runs in 3.4 minutes with a peak of 23.4GB. compared to RSGD/RSGD-C. which require 5.7 minutes and 29GB under the same conditions due to repeated conjugate- sojjugate-radient solves and Jacobian- products for metric inversion. VSGD, by contrast, is fastest (1.6 minutes, no path structure), but lacks any geometric regularization.
29

---

Published as a conference paper at ICLR 2026
Path-based perceptual smoothness vs. path length
0.25
LPIPS R-LPIPS
Perceptual smoothness
0.20
0.15
0.10
0.05
0.00
3
8
10
15
20
Path length T
Figure 11: Path-based perceptual smoothness versus path length T, measured by average LPIPS and R-LPIPS step (lower is better).
Table 8: Resource usage across different path lengths (resolution 512?).
Resources
Path Length T
10
15
20
25
Time (min)
3.4
5.3
7.2
10.9
Peak CUDA Memory (GB)
23.4
24.1
25.5
27.3
Table Resource usage across different image resolutions (T=10 points).
Resources
Image Resolution
256
5122
1024
Time (min) Peak CUDA Memory (GB)
1.7
3.4
7.8
19.8
23.4
27.7
B.6 ABLATIONS ON THE CLASSIFIER WEIGHT FOR PCG
We study how PCG's behavior depends on the classifier-loss weight À under a fixed optimization budget of 300 steps. Recall that PCG initializes each path with endpoint sampled from the target class, so the classifier already predicts the target at the start of optimization. quantify whether optimization preserves this, report target-class retention metric:
Target retention
Σ
[[f(= (1)
( where 21(0) is the final endpoint after optimization and Vigi is the prescribed target class. High values mean that the endpoint remains in the target region; low values indicate that the path has lapsed back towards the source class.
30

---

Published as a conference paper at ICLR 2026
We compute FID, R-FID, LPIPS, and R-LPIPS shared subset of examples: first filter, for each configuration, to those counterfactuals whose endpoint is in the target class, and then take the intersection across all configurations. All reported distributional and perceptual metrics evaluated on this common subset.
Table 10 reports results for of static À values (kept constant throughout optimization) and for dynamic schedules of the form X Xn 5/4/501,
Table 10: Effect of the classifier-loss weight PCG under a 300-step budget. Static settings keep
fixed; dynamic schedules start from and multiply by 5 every 50 steps. Lower is better for FID, R-FID, LPIPS, and R-LPIPS; higher is better for Target retention, (STYLEGAN2 on AFHQ)
À type
Setting
FID
R-FID
LPIPS +
R-LPIPS +
Target retention f
0.0001
11.8
14.9
0.05
0.03
0.37
0.005
10.3
11.2
0.07
0.04
0.54
0.010
9.1
9.5
0.25
0.21
0.68
Static
0.100
8.5
9.3
0.31
0.27
0.78
1.000
7.8
8.6
0.42
0.38
0.81
4.000
7.4
8.2
0.51
0.41
0.87
Au = 0.0001
8.4
9.5
0,21
0.13
0.95
Dynamic
An 0.001
7.9
9.8
0.47
0.32
0.96
Ao = 0.01
7.5
9.1
0.51
0.47
0.97
For very small static (e.g. 0.0001), the path-energy term dominates the objective, so the optimiser prefers trajectories that remain extremely close to the original point. This yields low LPIPS R-LPIPS at a higher FID, but the target-class retention is poor: the endpoint is often pulled back towards the source-class region within the 300-step budget, the initial target-class endpoint is not maintained.
As À increases, target retention rises monotonically: moderate values (around 0.01-0.1 and up to 1.0) strike better balance between path energy and classifier loss, achieving substantially higher retention while keeping FID and R-FID near their best values and only moderately increasing LPIPS. Pushing larger static values (e.g. A 4.0) further increases retention but the cost of noticeably higher LPIPS R-LPIPS, reflecting more aggressive, less conservative moves away from the original image.
Dynamic schedules achieve even higher target retention under the same budget. Starting from small and growing it multiplicatively allows early iterations prioritize finding a low-energy path structure, while later iterations increasingly emphasize staying in the target-class region. This yields retention above 0.9 with FID R-FID comparable the best static settings. However, as the initial weight À increases (e.g. starting from 0.001 or 0.01), the classifier term dominates earlier in optimization: target retention approaches 0.96-0.97, but LPIPS and R-LPIPS increase, indicating relatively larger semantic displacement from the input. Overall, these trends highlight the intended trade-off: smaller values favor geodesics that cling too tightly to the original image and often lose the target class at the endpoint, whereas larger and scheduled À values improve target-class retention at the expense closeness, with moderate dynamic schedules offering the best compromise between geodesic regularity and counterfactual validity.
B.7 SMOOTHNESS OF THE GENERATED COUNTERFACTUAL GEODESICS.
Table 11 compares different paths connecting the same endpoints, Torig and zet. in terms of path- based perceptual smoothness. For each trajectory, we compute the average I.IPS- step AT.PIPS and its robust counterpart AR-LPIPS between consecutive points along the path. Both latent-space baselines-linear and spherical interpolation-exiieee indicating that the perceptual change between successive frames is uneven and occasionally abrupt, particularly when measured robust feature space. In contrast, the PCG geodesic achieves markedly smaller values of both ALPIPS and AR-LPIPS. This is consistent with the construction of PCG as a discrete approximation to a geodesic in the induced robust perceptual geometry: by explicitly minimizing path energy in that metric, PCG produces trajectories that change content gradually
relatively large ALPIPS and AR-LPIPS.
31

---

Published as a conference paper at ICLR 2026
and approximately constant "perceptual speed" in robust feature space, whereas naive latent interpolations do not respect this geometry and therefore yield less regular, less semantically smooth transitions.
Table 11: Path-based perceptual smoothness for different interpolation schemes between Torig and Fef- We report the average LPIPS and R-LPIPS step along each path, ALPIPS and AR-LPIPS (lower is better) (STYLEGAN2 on AFHQ).
Path type
ALPIPS 1
AR-LPIPS $
Linear latent interpolation
0.51
0.83
Spherical latent interpolation
0.48
0.64
PCG geodesic (ours)
0.07
0.05
B.8 EFFECTS THE CHOICE OF ROBUST BACKRONE AND AGGREGATED LAYERS.
Table 12 investigates how the induced geometry depends on the choice of robust backbone (CNN vs. Vision Transformers) and on which layers are aggregated. all cases, the ResNet-50 and XCTT-S12 backbones are adversarially trained under an l threat model with e = 3 from (Engstrom et al., 2019; Debenedetti et al., 2023). For each backbone, we consider three aggregation configurations. Early-to- mid captures low-level and intermediate structure from early-to-mid layers/blocks (ResNet-50: stem and conv2_x; XCiT-S12: 12: blocks and 5). Mid-to deep focuses on higher-leve semantics from later layers/blocks (ResNet-50: conv3 x-conv5__ XCIT-SIII -12 blocks 7, 9, and 11). Finally, multi-block (our default configuration) aggregates features across early-to-depp layers.
Two patterns clear. First, within each backbone, there is consistent progression from early-to- mid, to mid-to-deep, to multi-block aggregation: R-FID decreases, while both MAS and mean SM increase. For the robust ResNet-50, moving from early-to-mid to mid-to-deep layers yields gain in robust realism and semantic structure, and aggregating across all blocks further improves all three metrics. The XCTT-S12 backbone shows the same behaviour: early-only aggregation underperforms, mid-to-deep layers bring a clear improvement, and multi-block aggregation gives the strongest overall performance, albeit with slightly lower absolute values than ResNet-50. Second, across architectures, the relative trends and performance gaps remain stable: regardless of whether the robust features come from a CNN or vision transformer, PCG benefits from including higher-level, semantically richer layers, and combining early and deep features yields the best trade-off between distributional realism (R-FID), manifold alignment (MAS), and semantic locality (mean SM). Note that, even with few early layers across different backbones, PCG outperforms all baselines (refer to Table 2). This further shows that the proposed induced geometry is not tied to a specific robust backbone, while also highlighting that robust, high-level features particularly important for capturing manifold structure.
Table 12: Effect of robust backbone and layer aggregation on PCG (STYLEGAN2 on AFHQ).
Backbone
Layer set
R-FID
MAS f
Mean SM 1
ResNet-50
Farly-to-mid
12.2
0.74
0.31
Mid-to-deep
9.3
0.86
0.38
Multi-block
7.1
0.91
0.42
XCTT-S12
Early-to-mid
13.8
0.71
0.26
Mid-to-deep
11.2
0.81
0.35
Multi-block
8.3
0.89
0.39
B.9 DIFFERENT CLASSIFIERS & EXTENSION TO MULTI-CLASS CLASSIFICATION
PCG not restricted to binary classifiers and extends in a straightforward way to multi-class settings by targeting the logit (or probability) of any chosen class. In the main text, instantiate the explained classifier as a VGG-19 19 backbone ine-tumed for binary tasks to keep the exposition and visualisations focused.
32

---

Published as a conference paper at ICLR 2026
In Figure 12, illustrate this behaviour AFHQ for a three-war way classifier (cat dog / wild) using two different backbones as the explained model: ResNet-18 and a DenseNet-121. Each row starts from input image (left) and shows PCG counterfactuals towards the two target classes classes for each backbone (e.g. cat + dog and cat + wild in the first row, dog -> cat and dog - wild in the PCG produces class-consistent
second, and wild cat and wild -> dog in the third). Across transformations that primarily modify species-defining cucs-such as car and muzzle shape, fur texture, and overall facial -trieture-while preserving pose, lighting, and background. PCG is not tied to specific classifier architecture: given a different multi-class backbone, the induced counterfactuals remain geometrically smooth and visually plausible, while reliably steering the prediction towards the desired target class.
Input
PCG (ResNet-18)
PCG (DenseNet-12))
cat dog/wild
dog > cat/wild
wild -
cat/dog
Figure 12: PCG counterfactuals for multi-class AFHQ (cat / dog wild). From left to right: input images and PCG counterfactuals for ResNet-18 and DenseNet-122 classifiers, showing class- consistent cat+>dog+-wild transformations.
Robust classifier with standard latent geometry vs. PCG. line with the literature on adversarial robustness, robust models are known to exhibit gradients and saliency maps that are more aligned with the data manifold and human perceptual structure than their standard counterparts. To probe how far this helps in the counterfactual setting, we consider latent-space baseline without our robust perceptual metric, but with a robust classifier as the model to be explained. Concretely, we run VSGD in latent space using AFHQ classifier fine-tuned from the same robust ResNet-50 that we use to induce our geometry, and compare to PCG applied to a standard (non-robust) AFHQ ResNet-50 with our robust pullback metric.
Figure 13 shows qualitative examples on AFHQ. In all rows, PCG produces counterfactuals that are both class-consistent and tightly on-manifold: the species changes (ears, muzzle, fur pattern) while pose, lighting, and background remain stable, and there are no obvious local artifacts. VSGD with robust classifier (right column) clearly improves the cat->dog example yields plausible dog face rather than highly distorted image-reflecting the more manifold-alignee gradients of the robust model. However, the manifold conformity still noticeably weaker than PCG. In the first row, the VSGD+robust counterfactual exhibits a larger semantic drift relative the input than PCG: it achieves the target class, but with more drastic change in identity and fine-scale structure, consistent with the absence of geodesicity constraint, In the second and third rows, VSGD+robust produces dog/cat-like animals, but with local off-manifold artifacts in the fur and facial regions (e.g., irregular texture and shape in the mane/ears), whereas the PCG results remain visually smoother and more coherent. This aligns with our claim that a robust classifier guards against some adversarial behavior, but without an induced geometry that encodes its structure latent space, trajectories can still slip into directions that are only weakly aligned with the data manifold.
VSGD with non-robust classifier-e.g.
33

---

Published as a conference paper at ICLR 2026
Input
PCG
VSGD Robust Clf
cat - dog
wild dog
wild cat
Figure 13: PCG vs. VSGD with robust classifier AFHQ (cat / dog wild). Each row shows an input image (left), the PCG counterfactual for standard ResNet-50 classifier (middle), and the counterfactual from VSGD applied robustly fine-tuned ResNet-50 classifier (right) for cat->dog and wild -dog/cat targets. VSGD with a robust classifier produces more meaningful counterfactuals than its non-robust counterpart but still exhibits larger semantic drift and local artifacts compared to PCG. For clarity, standard VSGD results with non-robust classifier are omitted here, as they predominantly yield AEs rather than plausible counterfactuals.
To make this comparison systematic, measure distributional realism (FID, R-FID), closeness (LPIPS, R-LPIPS), and manifold alignment under the robust geometry (MAS,ot)::
Table 13: Comparison of PCG, VSGD, and VSGD with a robust classifier on AFHQ counterfactuals. Lower better for FID, R-FID, LPIPS, and R-LPIPS; higher better for MAS (STYLEGAN2 on AFHQ).
Method
FID
R-FID |
LPIPS 1
R-LPIPS |
MAS,ob 1
VSGD
25.4
39.7
0.89
0.71
0.17
VSGD + robust clf
13.2
16.6
0.41
0.32
0.74
PCG (ours, standard clf)
8.5
9.4
0.23
0.19
0.89
Table 13 mirrors the qualitative picture. Switching from a standard to a robust classifier inside VSGD improves all metries: FID and LPIPS decrease slightly, while R-FID, R-LPIPS, and MAS improve substantially, confirming that robust features reduce adversarial exploitation and encourage more semantic changes. Nevertheless, PCG remains clearly ahead, especially on the robust and gewmetry-aware metrics: its R-FID is close to the real target distribution, its R-LPIPS indicates smaller robust perceptual displacement from the original images, and its MAS is significantly higher, meaning that its counterfactual directions lie much more in the robust tangent space. In other words, VSGD+robust clf sits in an intermediate regime- -eeteereterr than non-robust latent optimization, but still less manifold-alige alignned than PCG. By contrast, PCG's behavior is driven primarily by the induced robust geometry: enforcing geodesicity forces the entire trajectory to conform tightly to the robust feature manifold, independently of whether the explanatory classifier is itself robust.
34

---

Published as a conference paper at ICLR 2026
C FINAL NOTES ON SCOPE, LIMITATIONS & FUTURE WORK
Our work is deliberately centered on specific but fundamental line of methods: latent-space instantiations of the Wachter et al. counterfactual objective, where one optimizes a counterfactual loss in the latent space of generator and decodes back to image space. This family has been extremely influential, but, as we discuss in the introduction, it has accumulated chronic geometric failure modes in realistic vision settings (off-manifold traversals, on manifold adversarial examples, and semantic drift along latent trajectories). These issues have pushed much of the field either to restrict latent- Wachter formulations to low-dimensional or tabular data, or to move to different paradigms altogether. PCG is best read as a geometric re-framing of this core -eeet-r line: we make the choice of geometry explicit, induce it from robust perceptual features, and require trajectories to follow counterfactual geodesics rather than arbitrary latent updates. Our empirical focus and claims are therefore scoped to this regime: Wachter-style latent counterfactuals for non-robust classifiers in high-dimensional vision, under a fixed pretrained generator.
Within that scope, PCG targets the failure modes above and, under geometry- and robustness aware diagnostics, improves both on realism, semantic proximity, and on manifold faithfulness relative to existing latent-space Wachter-based baselines. We do not claim to subsume all contemporary visual counterfactual methods. Other directions, discussed in our introduction, including pixel-space diffusion approaches (e.g. Weng et al., 2024; Sobieski et al., 2024) and latent diffusion methods (Sobieski & Biecek, 2024; Augustin al., 2024; Luu al., 2025) instantiate counterfactual generation through diffusion sampling, often with classifier guidance and regional constraints, in either pixel space or the latent space of a diffusion model. These are complementary directions: they work with different generative families and through optimization mechanisms different from the Wachter-style latent optimization.
Methodologically, PCG inherits several structural dependencies that can be viewed as limitations. First, like all latent-space approaches, PCG relies on a pretrained generator (and encoder, when present) whose latent space has reasonable manifold fidelity. Our robust pullback metric and two- stage path refinement mitigate artifacts from imperfect generators, but they cannot fully repair a severely mis- nis-specified collapsed latent space; in such cases, all latent methods, including PCG, are constrained by the quality of the underlying generative model. Second, the induced geometry is built from an adversarially trained robust backbone on ImageNett. This is motivated by theoretical and empirical work on adversarial robustness, which shows that robust models align gradients and feature variations more closely with the data manifold and human perceptual structure (Srinivas et al., 2024; Ganz et al., 2023a; Kaur et al., 2019a; Kettunen et al., 2019; Ghazanfari et al., 2024a). However, robustness is intrinsically local and ceerriiutoonddepeddee are robust only within an L2 ball of radius around images drawn from (or close to) its training distribution. Our induced geometry is most trustworthy when (i) the generator and robust backbone operate approximately the same domain, (ii) trajectories stay within regions where the robust model's predictions and gradients remain stable, and (iii) geodesic paths are reasonably well sampled with enough discretization of the underlying robust geodesic to allow the optimizer to distribute smooth semantic change across more intermediate states. Under strong distribution shifts, under- sampled paths, or strong displacements along trajectories that leave these neighhorhoods, the semantic interpretation of the metric can degrade even though the construction remains mathematically valid.
the backbones used in our experiments
These observations naturally point to future work. On the robustness side, it would be interesting to study how different robustness norms, radii, and training regimes (e.g., Los vs. L2, certified vs. empirical robustness) reshape the induced geometry and the behavior of geodesics, and whether one can adapt the metric or the path optimization schedule to local estimates of robustness. On the generative side, our current choice of SeiieGAN-faamily generators is partly pragmatic: they provide a single, well-defined latent space in which it is tractable to induce and analyze a metric via pullback geometry. Latent diffusion models, by contrast, involve multiple interacting spaces (the VAE latent, dependenn diffusion states, and conditioning latents), and stochastic score dynamics over time. Transferring our geometric framework to that setting would require deciding where to place geometry, how to define geodesics consistently along a stochastic trajectory, and how pullback metric should interact with the score field. We expect the core phenomenon we study - that a strong generative prior alone does not preclude mis-specified - to persist in diffusion architectures, but a careful treatment is non-trivial and we view this as a distinct line of future work rather than trivial extension.
adversarial behavior when geometry is
35

---

Published as a conference paper at ICLR 2026
Finally, our experiments are confined to images. Extending PCG beyond vision raises both modeling and geometric questions. Multimodal extensions could couple text and image spaces via joint latent geometries (e.g., CLIP-style or diffusion backbones) and cross-modal robust metrics; video counter- factuals would need to incorporate temporal coherence and spatiotemporal perceptual geometry; and applying similar ideas to graphs language would require suitable generators and domain-specific robust features. In low-resource regimes, training full-scale robust backbones may be impractical, suggesting the need for lightweight robust surrogates, few-shot adaptation of perceptual metrics, or self supervueee proxies. We see PCG as a first step towards bringing explicit, robustly induced geometry into latent counterfactual explanations in vision, and anticipate that both the limitations and the structural ideas outlined here will be useful in guiding subsequent work across other generative families and data modalities.
36


Reviews:
[{"review_id": "VNY97XaXge", "reviewer": "Reviewer_Bs9x", "summary": "This paper presents a method, Perceptual Counterfactual Geodesics (PCG), for generating counterfactual explanations by optimizing trajectories in a generator's latent space. The core contribution is the introduction of a robust Riemannian metric to define the latent geometry. This is induced from the feature spaces of robust vision models. This perceptually-aligned geometry enables the generation of smooth, semantically valid explanations that successfully avoid both off-manifold artifacts and on-manifold adversarial regions, which are common failure modes for existing latent-space methods.", "questions": "(restating some of the weaknesses)\n1. Can you provide theoretical analysis or intuition for why (and when) robust features guarantee semantically meaningful geodesics? Are there failure cases where robustness alone is insufficient?\n2. The runtime analysis in Appendix B.5 mentions using T=10 nodes for the path. How does the computational cost (both time and memory) scale as T increases? Is there a practical limit. What is the trade-off between a smoother geodesic and computational cost?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong motivation: Paper clearly articulates three speicific failure modes of prior latent space methods: 1. off-manifold traversal leading to artifacts, 2. local gradient optimization that ignores global structure, 3. generator exploitation of non-robust metrics.\n- Perceptual Counterfactual Geodesics (PCG) is logical and well-structured. It employs a two-phase optimization process. This involves first finding an energy-minimizing geodesic, and tehn jointly refining the path and its endpoint with a classification loss.\n- The experimental validation is comprehensive. Uses three high dimensional datasets (AFHQ, FFHQ, Plant Village), and two different generator architectures (STYLEGAN2, STYLEGAN3).\n- The paper provides compelling visual evidence that PCG produces smooth, semantically coherent explainations while baselines collapse into off-manifold artifacts or on-manifold AEs. PCG also acheives the best scores in all geometry aware metrics.\n- The authors introduce a robust, geometry-aware evaluation metric, $ L_R $. This proves to be more faithful to perceptual similarity and successfully exposes teh adversarial failure modes of baselines that remain hidden under standard $ L_1 $, $ L_2 $, or even non-robust feature metrics, $ L_F $.", "weaknesses": "- The paper's own conclusion frames it's contribution as operationalizing established ideas from pullback geometry and robust perception. The technical novelty is incremental, rather than a fundamental new theory.\n- The paper's main quantitative results is incomplete, with runtime comparisions relegated to appendix. There is limited discussion of scalability to longer paths or higher resolution images.\n- The justification for why robust features necessarily yield semantically meaningful geodesics rests on empirical demonstrations and appeals to prior adversarial robustness literature. It could be strengthened with a self contained theoretical proof for why this alignment must exist.\n- The paper notes that the baseline RSGD methods were mostly proposed for tabular data settings. The authors had to implement their own adaptations for the vision domain, which could mean the comparison isn't fully representative of those methods intended use, and hence somewhat unfair.\n- The sensitivity analysis in Appendix B.3 is limited. It relies on only three initialization runs per input and uses a single perceptual metric (LPIPS) for comparison. A broader statistical analysis would make the claims of low sensitivity more robust.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5lj8bI0lAO", "reviewer": "Reviewer_RJrE", "summary": "This paper addresses an important problem of the potential adversarial nature of counterfactual explanations (CEs) in computer vision. To avoid off- and on-manifold adversarial examples (AEs) as solutions to the empirical CE optimization problem, the authors propose to perform the optimization within the latent space of a generative model with the geometry induced by the aggregated representations of a robustly trained classification model. This optimization takes into account the entire path traced from the true sample to the CE and aims at finding a geodesic between these points wrt. the proposed geometry. Within the experimental section, the authors aim at showing improved smoothness in comparison to baselines, superior results in terms of several geometric measures and the ability to avoid off- and on-manifolds AEs.", "questions": "Overall, I am a big fan of the proposed approach and consider the problem to be extremely relevant. My greatest concerns lie in the scope and scale of the current evaluation scheme, which does not provide sufficient evidence for some of the claims and does not shed light on important aspects of the proposed method, e.g, the $\\lambda$ hyperparameter. During the rebuttal, please refer to the weaknesses above, which either contain the questions directly or implicitly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. The considered problem is often overlooked in the CE context, especially in the computer vision domain, where human perception may be easily fooled. It is also of extreme importance to the explainability community -- since CEs stand at the top of the Pearl's causality ladder, it is crucial to generate explanations that are truly valid. The authors provide an elegant solution to this problem, with clearly highlighted motivation and proper mathematical formalism.\n\nS2. The proposed PCG algorithm is actually a novel, two-phase approach with clear motivation behind its construction, especially for phase I. The clarity of the description is great, the baselines are properly described, and the overall flow of the paper is enjoyable for the reader.\n\nS3. Prior methods in the undertaken research direction were mainly considered in the tabular setting. The authors did a very good job in transferring and adapting these methods to the vision domain.", "weaknesses": "W1. Robust models are never \"infinitely\" robust (lines 225-227), meaning that their robustness is preserved up to some nieghborhood of each point. What are the ways of measuring this robustness? How does the level of this robustness influence the resulting induced geometry? What limitations spark from that and how can they be overcome?\n\nW2. Is there some theoretical justification for the definition of the robust perceptual metric $G_R$ (its equation is not numbered, but can be seen at line 234)? Or is it purely a heuristic choice? I fully understand the motivation behind it, but this particular choice of the definiton appears in a slightly surprising way without any reference, especially a theoretical one about the robustness itself.\n\nW3. Influence of $\\lambda$ seems crucial as it may significantly impact the robust energy and eventually make the trajectory misaligned with the induced robust semantic geometry. The paper currently lacks any ablation study related to $\\lambda$'s influence.\n\nW4. I am fully aware that evaluating the claimed improved smoothness and manifold consistency is very challenging. However, I must argue that simply providing a few example trajecteries is not enough to properly back the claims. One way of addressing this is through a user study, in which participants are asked about the perceived smoothness, i.e., the ground-truth for the semantic similarity perception. The other, probably less demanding and more suitable to the rebuttal, is finding some proxy measure to quantitatively assess the properties.\n\nW5. While I agree that counterfactuals from other methods seem to be off- or on-manifold AEs, how can this be measured at scale? What is the guarantee that these samples are not just cherry-picked? A quantitative comparison here is important to assess whether PCG is not suffering from the same problems.\n\nW6. The paper currently does not contain any explanandum-related metrics and is missing the typical ones from prior work. While I do understand that baseline works did not use them, the extension to vision domain deems it necessary (at least for me). The paper should additionally contain a comparison in terms of FID ([1], closeness at the distribution level), COUT ([2], sparsity relative to the explained model) and flip/sucess rate (effectiveness of the method).\n\nW7. Important ablations are missing: what is the influence of the choice of the robust model? how does its training data influence the resulting geometry (the proposed method relies on ImageNet-based model everywhere)? How does the choice of the number and weights of the layers affect it?\n\nW8. The paper is currently ignoring two important research directions. First one is related to current state-of-the-art solutions based on pixel-space diffusion-based models, e.g. [3] and [5]. These should be at least mentioned as recent methods that do not focus on the problem considered in the paper, which will only strengthen its relevance. The other direction considers latent-diffusion-based methods, which also perform some kind of optimization in the latent space of a VAE, but with an addition of a diffusion model [4,6,7]. What is the influence of the VAE+diffusion combination in the considered problem of off- and on-manifold AEs? These issues should be at least mentioned in the paper to ensure maximum impact.\n\nMinor:\n- line 095: the mentioned methods are mostly in the vision domain, except the first one\n- line 238: neural networks (for prediction tasks) typically compress data, so it should probably be $d_k << D$\n- line 921: is the change to Softplus actually relevant? If yes, some experiment should be showing that in the paper. If not, why not just remove this procedure and stick with ReLU?\n- the main text should be slightly more precise about phase II, the re-anchoring procedure and the overall details. It is fully understandable only after referring to the pseudocode from the appendix.\n\n[1] Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NeurIPS, 2017\n\n[2] Khorram et al., Cycle-Consistent Counterfactuals by Latent Transformations, CVPR, 2022\n\n[3] Weng et al., Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation, ECCV, 2024\n\n[4] Sobieski and Biecek, Global Counterfactual Directions, ECCV, 2024\n\n[5] Sobieski et al., Rethinking Visual Counterfactual Explanations Through Region Constraint, ICLR, 2025\n\n[6] Augustin et al., DiG-IN: Diffusion Guidance for Investigating Networks - Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations, CVPR, 2024\n\n[7] Luu et al., From Visual Explanations to Counterfactual Explanations with Latent Diffusion, WACV, 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ozfwDVrdNW", "reviewer": "Reviewer_T7oH", "summary": "The paper studies the problem of generating counterfactual explanations for image classifiers.\n\nThe paper adopts a latent-space approach: the counterfactual explanation is found by traversing the latent space of a generative model.\n\nThe introduction clearly describes the problems faced by existing latent-space approaches and methods that search for counterfactual explanations more generally. For example, we don't want to provide on-manifold adversarial examples as counterfactuals.\n\nThe paper then proposes a novel method  to generate counterfactuals (PCG). The idea is to equip the latent space with a robust perceptual metric via pullback from a robust vision classifier (this is not the same classifier for which they generate the explanations). \n\nThe paper states that when counterfactual explanations are generated in the latent space using the robust perceptual metric, they correspond to smooth, on-manifold, semantically valid transitions between images of different classes.\n\n**Experiments:** The novel method PCG is compared against other latent-space counterfactual methods. The paper contains the following evaluations:\n\n- Visual inspection of interpolation paths in the latent space (Figure 2, Figure 3)\n- Quantitative comparison between different CF generation methods according to standard metrics (Table 1)\n- Qualtitative dedication of counterfactuals generated by different methods (Figure 4).", "questions": "**Question 1:** Why is the proposed approach not applied to arbitrary ImageNet classifiers? (or other standard classifiers from PyTorch repositories)\n\n**Question 2:** If we were to generate counterfactual explanations using a latent-space approach without your robust perceptual metric, but for a robust computer vision classifier (like the robust model that you derive your metric from), would this result in similar explanations?\n\n**Question 3:** Is the direction from the original image to the generated counterfactual approximately aligned with the tangent space of the data manifold at the original image, or is this not the case?\n\n**Reason for Final Score:**  The strengths of this paper is a very well-written introduction and a very good motivation for the proposed method, including the mathematical details. The main weakness of this paper is the limited scope of the empirical results, which fall somewhat below the bar for a conference like ICLR. To me, this makes the current version of the paper a marginal paper. I decided to assign the paper a score of 4 for now.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is well-motivated and extensively described in the paper\n- The contribution of the paper is situated within the existing literature\n- The empirical results in the paper are encouraging, both for the geodesics and the generated counterfactual explanations\n- The paper is clearly written", "weaknesses": "**Limited empirical results:** While the motivation and description of the proposed method are extensive, the breadth of the empirical results is surprisingly limited. By this I mean that:\n\n- The qualitative examples in the paper are almost all either for humans or for cats and dogs. So there is a limited breadth in terms of the classes for which the counterfactuals are explored.\n- The classifiers for which the counterfactuals are generated are VGG-19 backbones trained on a binary classification task. This seems like a rather specific choice; it should be relatively easy to extend this to a multi-class setup with other architectures? \n- Some details, like class probabilities assigned by the classifiers, are not reported\n\nFor example, I compared the empirical results in this paper to the results in \"Diffusion Visual Counterfactual Explanations\" NeurIPS 2022 (https://proceedings.neurips.cc/paper_files/paper/2022/hash/025f7165a452e7d0b57f1397fed3b0fd-Abstract-Conference.html). This paper does not use a latent-space approach, but, similarly to the submitted work, it proposes a method to generate robust counterfactual explanations for arbitrary classifiers. This paper has similar limitations to the submitted paper, insofar as it must rely on a visual inspection of the generated counterfactuals and some proxy metrics. However, the empirical results in this paper are much more detailed.\n\n**Unclear if the proposed approach beats the state of the art:** While the proposed method is very well-motivated, the limited empirical evaluation does not make clear to me in what precise sense the obtained counterfactuals can be considered state of the art among latent space methods, or method for generating counterfactual explanations for non-robust classifiers more generally (if we were to compare with diffusion-based approaches as well, as in the above reference)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)