FeatInv: Spatially resolved mapping from feature space to input space using conditional diffusion models
Authors: Nils Neukirch
Johanna Vielhaben
Nils Strodthoff
Venue: Accepted by TMLR ()

Abstract: Internal representations are crucial for understanding deep neural networks, such as their properties and reasoning patterns, but remain difficult to interpret. While mapping from feature space to input space aids in interpreting the former, existing approaches often rely on crude approximations. We propose using a conditional diffusion model - a pretrained high-fidelity diffusion model conditioned on spatially resolved feature maps - to learn such a mapping in a probabilistic manner. We demonstrate the feasibility of this approach across various pretrained image classifiers from CNNs to ViTs, showing excellent reconstruction capabilities. Through qualitative comparisons and robustness analysis, we validate our method and showcase possible applications, such as the visualization of concept steering in input space or investigations of the composite nature of the feature space. This approach has broad potential for improving feature space understanding in computer vision models.

Published in Transactions on Machine Learning Research (12/2025)
FeatInv: Spatially resolved mapping from feature space to input space using conditional diffusion models
Nils Neukirch
mils.neukirchhaol... de
Division AlfHealth
Carl von Ossietzky Universität Oldenburg
Johanna Vielhaben
Eaplainable Artificial Intelligence Group
Fraunhofer Cenarric--Hett-hiie
Nils Strodthoff
nils. mils.strudthof@@oo..dd de
Division Al4Healtth
Carl Ossietzky Universitää Oldenburg
Reviewed on OpenReview:
net/ forum? UtE1YnPNg2
Abstract
Internal representations are crucial for understanding deep neural networks, such as their properties and reasoning patterns, but remain difficult interpret. While mapping from feature space input space aids in interpreting the former, existing approaches often rely crude approximations. We propose using conditional diffusion model pretrained high-fidelity diffusion model conditioned on spatially resolved feature maps to learn such mapping in a probabilistic manner. We demonstrate the feasibility of this approach across various pretrained image classifiers from CNNs to ViTs, showing excellent reconstruction capabilities. Through qualitative comparisons and robustness analysis, we validate our method and showcase possible applications, such the visualization of concept stecring in input space investigations of the composite nature of the feature space. This approach has broad potential for improving feature space understanding in computer vision models.
FeatIny
Original
ResNet50
ConvNeXt
SwinV2
Figure FeatIny learns a probabilistic mapping from feature space input space and thereby provides a visualization of how sample is perceived by the respective model. The goal is identify input samples within the set of natural images whose feature representations align most closely with the original feature representation of given model. In this figure, visualize reconstructed samples obtained by conditioning on the feature maps of the penultimate layer from ResNet50, ConvNeXt and SwinV2 models.

---

Published in Transactions on Machine Learning Research (12/2025)
1 Introduction
The feature space is vital for understanding ncural network decision processes it offers insights into the internal representations formed by these models as they process inpul data. While it serves as the foundation for many modern explainability approaches (Rai et al. extends beyond interpretability. The feature space provides rich resource for investigating fundamental
2024 Bereska & Gavves 2024). its importance
properties of deep neural networks, including their robustness against perturbations, invariance characteristics, and symmetry properties (Bordes et al., 2022). By analyzing the geometry and topology of these learned representations, researchers can gain insights into model generalization capabilities, failure modes, and the emergence of higher-order patterns in the data. This perspective enables advancements in theoretical
understanding of neural networks while informing practical improvements in architecture design and training methodologics.
An important challenge in examining the feature space is establishing a connection back to the input domain, especially for classification models that map to labels rather than the same domain as the input. One aspect of this challenge involves identifying which part of the input a particular region or unit in feature space is sensitive to. GradCAM (Selvaraju al. 2017] pioneered this by linearly upsampling a region of interest in feature space to the input size. However, linear upsampling imposes rather strong implicit assumption. As alternative, one might consider the entire receptive field of feature map location, yet in deep architectures these fields tend to be broad and less informative.
The more intricate second aspect of this challenge is to derive a mapping from the entirety of the feature space representation back to the input domain beyond mere localization. Recent works proposed to leverage conditional generative models to learn such a mapping by conditioning them on feature maps (Bordes et al.,
2022 Dosovitskiy & Brox 2016 Rombach et al., 2020). However, these approaches either build on pooled feature maps (discarding finegrained spatial details of the feature map), only provide deterministic mappings (overlooking the inherent uncertainty of this ill-posed problem), or do not utilize state-of-the-art generative models. Related approaches such as diffusion autoencoders (Preechakul et al. 2022) show that diffusion models can indeed be fitted with meaningful and decodable latent representations that enable near-exact reconstructions and the manipulation of semantic attributes. However, their latent codes are global and not spatial, whereas our focus is on conditioning spatially resolved feature maps to preserve the fine-grained structure. We argue that a lack of faithfulness, i.e., the inability to reconstruct spatially resolved feature maps, is detrimental for downstream applications, c.g., in explainable AI or for robustness analysos, as it fails to reflect the true relation between feat ure space and input space in the model. In addition, pooled representations are not meaningful for the purpose of studying feature representations of intermediate layers. To the best of our knowledge, there is no probabilistic model that provides high-fidelity input samples when conditioned on a spatially resolved feature map thereby integrating both aspects of the challenge described above. We aim to close this gap with this submission.
More specifically, in this work we put forward the following contributions:
1. We demonstrate the leasibility of learning high-fidelity mappings from spatially resolved feature space to input space using conditional diffiusion model of the ControlNet-flavol as exemplified in Fig. | We investigate this for different computer vision modcls, ranging from CNNs to ViTs.
2. We provide quantitative evidence that generated samples align with the feature maps of the original samples and that the samples represent high-fidelity natural images, see Tab. T and carry out a qualitative model comparison, see Fig.
3. We provide robustness analysis, see Tab. 2 as well as further analysis demonstrating the genoralizability of our approach across different ConvNeXL stages (see Fig. its behavior on out-of-disstrbuttion data (see Fig. as well as its robustness to corrupted images (see Sec. 4.2
4. We provide a specific usc-cases for the application of the proposed methodology to visualize concopt- steering in input space, see Fig. 0 as well as to provide insights into the composite nature of the feature space, see Fig.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2
Methods
Classification
Poeling
Previous work
Hy
W
Stable Diffusson
Jadcc
Cr
Cf
!
ControlNet
df mudi
Model (CNN ViT)
Featiny (ours)
convolution
Conditional Enceder
SILU
convolation
Text prompt: "a high-quality. detailed and professional image"
SiL.U
Cf
convolution
x
upompling
Input
Visualization model represention ey input space
Figure 2: Schematic overview of the FeatInv approach. Left: Given a spatially resolved feature map Of of some pretrained model, we aim infer an input x' within the set of natural images, whose feature representation aligns as closely possible with cy, i.e. to learn a probabilistic mapping from feature space input space. Previous work consider spatially pooled feature maps, whereas this work conditions spatially resolved feature maps. Middle: We leverage pretrained diffusion model, which gets conditioned means of ControlNet architecture, which parametrizes an additive modification on top of the frozen diffusion model. Right top: The ControlNet. adds trainable copies of blocks in the stable diffusion model, which are conditioned on the conditional input and added to the output of the original module, which is kept frozen. Right botlom: The feature map ey is processed through bilinear upsampling and a shallow convolutional encoder to serve as conditional input for the ControlNet.
of by
Approach In this work, propose method called FeatInu to approximate inverse mapping from model's feature space imput space. Our method conditions a pretrained stable diffusion model on spatially resolved feature map extracted from pretrained CNN/ViT model of our choice. As described in detail in the next paragraph, the feature maps are provided as conditional information along with an unspecific text prompt ("a high-quality, detailed, and professional image") to a conditional diffusion model of the ControlNet (Zhang al. 2023a) flavor. Importantly, rather than achieving precise reconst ruction of the original sample reconstruction
in input space, our goal is infer high-fidelity, synthetic images whose fcature represcntations align with those of the original image when passed through a pretrained CNN/VIT model.
Architecture and training procedure We ControlNet Zhang et al. 2023a architecture, building pretrained diffusion models, in our case MiniSD (Pinkney 2023) model operating at an input resolution of 256 x 256. The ControlNet is a popular approach to condition a pretrained diffusion model on dense inputs such as segmentation maps or depth maps. It leverages a pretrained (text-conditional) diffusion model, whose weights are kept frozen. The trainable part of the ControlNct model mimics the internal structure of the pretrained diffusion model, with additional layers introduced incorporate conditioning inputs. These conditional inputs processed by dedicated encoder and inserted into the corresponding computational
3

---

Published in Transactions on Machine Learning Research (12/2025)
blocks, where their outputs are added to those of the original diffusion model. Convolutional layers initialised to zero ensure that optimisation of the ControlNet model starts from the pretrained diffusion model.
Conditional input encoder An important design choice is the conditional input encoder, which maps the feature map (with shape H x W x Cj, where , correspond to the height, width and channels of the feature map, respectively) to the diffusion model's internal representation space (with shape Ha x Wa x Ca)- As a definite example for 224 x 224 input resolution, for the output of ResNet50's final convolutional block with H W =, Cy 2048 we learn mapping to the diffusion model's internal representation space with Ha = W = 32 and Ca = 320. To this end, we first use bilinear upsampling to reach the target resolution. Then, we allow for shallow CNN to learn suitable mapping from the model's representation space to the diffusion model'ss representation space.
Pooled vs. unpooled To demonstrate superiority over prior work (Bordes et al. 2022 we also consider the case of pooled feature representations obtained from avcrago-pooling spatial tokens feature maps. In order to process them using the same pipeline as for conditioning on spatially resolved feature maps, we copy the Ca-dimensional input vector along Ha and Wa times to reach an input tensor with shape Ha X Wa X Ca as before.
Training The ControlNet is trained using the same noise prediction objective as the original diffusion model (Ho et al., 2020). Control signals are injected at multiple layers throughout the network, rather than being restricted to the middle layers, allowing them to influence the denoising process at various stages. Training was conducted on the ImageNet traiming set with a batch size of and a learning rate of 1e-5 using an AdamW optimizer with the stable diffusion model locked. The ControlNet was trained on ImageNet. for three epochs over approximately 45 to 60 hours (depending on backbone) of compute time two NVIDIA L40 GPUs. During the course of this project, about five times more models were trained until the described setup was reached.
Full pipcline We work with the original input resolution of the respective pretrained models, which varies between 224 x 224 and 384 x 384 for the considered models, see the Supplementary Material A.1 for a detailed breakdown. Even though the approach allows conditioning on any feature map, we restrict ourselves to the last spatially resolved feature map, i.c., directly before the pooling layer, and learn mappings to MiniSD's internal feature space. The MiniSD model always returns an image with resolution 256 x 256, which we upsample/ hwwpp///////www/eleee to the model's expect input resolution via. bilinear upsampling/downsmmppii The full generation pipeline is visualized in Fig. 2
3 Related Work
Conditional diffusion models Achioving spatially controllable image generation while leveraging a pre- trained diffusion model has been a very active area of research recently, ly, see (Zhang el al. 2023b for a recent. review. Applications include the conditional generation of images from depth maps, normal maps or canny maps. Popular approaches in this direction include ControlNet (Zhang et al. 2023ab GLIGEN (Li et al., 2023). The mapping from feature maps conditional input is structurally similar to the mentioned cases of spatially controllable generation. However, there is key distinction. In the previously mentioned cases, the conditional information typically matches the resolution of the input image. This often necessitates downsampling to reach the diffusion model's internal representation space. In contrast, commonly used classification models (including CNNs and vision transformers) leverage feature maps with a reduced spatial resolution. Consoquently, the spatial resolution of the conditional information is typically lower dimensional than the diffusion model's internal representation space. This difference necessitales an upsampling operation before conditioning on feature maps.
Feature visualization The idca to reveal structures in feature space to understand what neural network has learned is an old one. Approaches range from identifying input structures or samples that maximize the activation of certain feature neurons (Erhan et al. 2009 Nguyen et al. 2016) to approximate inversion of the mapping from input to features space (Zeiler 2014). Our approach clcarly stands in the tradition of the latter approach. Previous work has attempted to learn a deterministic mapping that "inverts" AlexNet feature maps (Dosovitskiy & Brox 2016). This approach was recently extended to invert vision transformer representations
4

---

Published in Transactions on Machine Learning Research (12/2025)
RRtthjens et al. 2024). In contrast, FeatInv learns a probabilistic mapping using state-of-the-art diffusion models and investigates state-of-the-art model architectures. Other approaches tackle the problem using invertible neural networks to connect VAE latent representations to input space (Rombach et al.) 2020) and/or disentangle these reprosentations using concept supervision (Esser et al., 2020 In contrast, Featinv does not rely on a particular encoder encoder/decoder /eeooder structure but can use any pretrained neural network as encoder. The closest prior work to our approach is (Bordes et al., 2022), which also uses a diffusion model to learn a mapping from feature space to input space. However, uses pooled representations as input, i.e. neglects the spatial resolution of the feature map. We argue that pooled representations are too coarse for many applications as they disregard the finegrained spatial structure of the feature space. Diffusion autoencoders (Preechakul et al. 2022) also explore how diffusion-basse representations can be made both meaningful and decodable, but their latent. codes are global vectors. In contrast, we condition directly on spatial fealure maps to preserve fine- aue-grained structure.
Representation surgery Finally, related feature inversion approaches have also been explored beyond computer vision, for example in natural language processing (Morris al. 2023). Here, the ability to invert. latent representations is seen an essential component for representation surgery approaches (Avitan et al. FeatInu enables similar approaches for computer vision models.
2025
4 Results
We investigate three models ResNet50 (He et al. 2016) (original torchvision weights), ComvNeXt (Li et al. 2022b) and SwinV2 (Liu et al. 2022a all of which have been pretrained pretrained/fnettnned /fnetuned on ImageNetlk. ConvNeXt and SwinV2 represent modern convolution- based and vision-transformer-beee
architectures,
identified as strong backbones in (Goldblum et al. 2024h. We include ResNet50 due to its widespread adoption. For each model, we train a conditional diffusion model conditioned on the representations of the last hidden layer before the final pooling layer to reconstruct the original input samples. Below, report quantitative and qualitative aspects of our findings.
4.1 Quantitative and qualitative comparison
Experimental setup For each ImageNet class, we reconstructed 10 validation set samples with FeatInv, resulting in 10,000 reconstructed samples. We adjust the diffusion model's control strength and guidance scale to optimize the match of classification outputs between original and reconstructed samples on the validation set, resulting in different control strengths and guidance scales for each model. Since our goal is to represent the feature space as accurately as possible, we also observed the cosine similarity wetween the feature maps of the original and reconstructed samples and observed a strong correlation between classification match and cosine similarity, which further supports our choice of paramcters. In our experience, it also possible to achieve good results with the same control strength and guidance scale for all three models, see the Supplementary Material A.2 for details. We reconstruct with sample step size of 50. For each model this took roughly 12 hours on single NVIDIA L40 GPU. We only generate one sample per feature map but it is also possible to generate multiple to observe the variability across reconstructions (given the same conditional input), see Supplementary Material A.3 for insights. The generated samples are assessed according to two complementary quality criteria, reconstruction quality and sample quality:
Reconstruction quality The encoded generated image should end up close to the feature representation of the original samples, which can be understood as a reconstruction objective that implemented implicitly by conditioning the diffusion model on a chosen feature map. (1a) The most obvious metric is cosine similarity between both feature maps. However, not all parts of the fcature space will be equally important for the downstream classifier. (1b) Most reliable measure is the classifier output itself. Focusing on lop-predictions one also compare top-k predictions to the top prediction for the original sample. More general alignment measures between generated input and original feature representation are not helpful in this context, as we require a precise reconstruction of the original feature space for the downstream classifier above the layer under
1 timm model weights: convnext baanneet__bssef..eeeeeee in22k ft. in1k, swinv2 base window11 192tn384_22kfttk 22k
5

---

Published in Transactions on Machine Learning Research (12/2025)
consideration. A feature inversion method is deemed faithful if the reconstructed samples ex- hibit strong correspondence with the original samples, as assessed by the two metrics introduced above.
2. Sample quality We aim to generate samples within the set of high-fidelity natural images. In our case, this objective is implemented through the use of pretrained diffusion model. Apart from qualitative assessments in the following sections, we rely on FID-scores as established measures to assess sample quality.
Table Reconstruction quality and image quality of the individual models: For the three considered backbones, we indicate three performance metrics to assess the quality of our reconstructions: Cosine similarity in feature space (cosino-sim), calculated by averaging the cosine similarity of all superpixels while maintaining the dimensions of the original inpul (288 288) and upscaling the reconstruction (from 256 x 256), top5(1) matches using the top-1 prediction of the original sample as ground truth (top5(1) match), and FID scores (FID) to assess the quality of the generated samples. We consider generative models conditioned on unpooled feature maps (rows 1-3) and models conditioned on pooled feature maps (rows 4-6). The results indicate that the proposed approach produces faithful reconst ructions with high fidelity that are both perceptually realistic and, when re-encoded, have similar values in the feature space as the original input.
Model
cosine-sim
top5(1) match
FID
unpooled
ResNet50
0.46
91% (70%)
11.49
ConvNeXt
0.61
94% (77%)
8.20
SwinV2
0.53
95% (80%)
12.69
pooled
ResNet50
0.12
48% (23%)
31.64
ConvNeXt
0.19
44% (20%)
31.67
SwinV2
0.16
47% (22%)
49.04
Reconstruction quality Comparing identical models conditioned either on pooled or unpooled feature maps, not surprisingly unpooled models show significantly higher reconstruction quality, Samples generated by models conditioned on unpooled feature maps show very good alignment with the feature maps of the original samples (cosine similarities above 0.53 and top5 matching predictions of 94% or higher for the two modern vision backbones). Samples conditioned on pooled feature maps show some alignment but fail to accurately reconstruct the respective feature map and are therefore unreliable for investigations of structural properties of models. These findings support the hypothesis that the approach yield feature space reconstructions that closely match the original feature represenlations. We also directly compared our pooled results the approach proposed by Bordes al. (2022). see Supplementary Material A.5 for details.
Sample quality The corresponding class-dependen. diffusion model achieves FID score around 29, which is typically considered as good quality. The models conditioned on pooled representations still show acceptable FID scores between 31 and 49. Interestingly, models conditioned on unpooled representations show a significant increase in image quality with FID scores between 8 and 12. These results support the statement that the created samples were sampled from the space of high-fideityy natural images.
Backbone comparison Within cach category (pooled vs. unpooled), there gap between the two most recent model architectures ConvNeXt and SwinV2, notwithstanding the architectural differences (CNN vs. Vision transformer) between the two, in comparison to the older Resnet50 models. The former achieve cosine similarities of .61 or higher and top5 matches of 94% or higher in the unpooled category. This suggests that there is a qualitative difference between the representations of Res-eet5l--ppreeentaaioo and representations of more modern image backbones.
Qualitative comparison In Fig. we present qualitative comparison based on randomly selocted samples. The visual impressions of ConvNeXt and SwinV2 reconstructions are similar each other while also being close to the input sample despite the fact that they were trained on high-level semantic feature maps, i.e.,
6

---

Published in Transactions on Machine Learning Research (12/2025)
without reconstruction objective in input space. The ResNet50 reconstructions seem in many cases interpretation of the sample's semantic content (see e.g. 2. toucan or 5. file), albeit with the correct spatial composition, while matching specific color composition and textures much less accurately than ConvNeXt and SwinV2. We primarily attribute the differences between ResNet and Conv.Nett/Swii S to the nature of the Teature spaces themselves, ves, stressing qualitative difference between modern architectures such as ConvNeXi and SwinV2 and older model architectures such as ResNet50, which are much more pronounced than the differences between different model architectures such as ViTs and CNNs. The samples obtained from conditioning on pooled feature representations often seem to capture overall semantic content of the image correctly (file, space shuttle, traffic light), but fail to reflect the details of the composition of the image. This can further be observed in the Supplementary Material A.4
Original
SwinV2 ConvNeX: RedNes00
E67
0L8
8.9
2.99
ComeNeXt
pooled
Figure 3: Qualitative comparison of reconstructed samples for the ResNet50, ConvNeXt, SwinV2 and ConvNeXt pooled models. The cosine similarity of the original feature map and that of the reconstruction is noted at the bottom edge of the images. The qualitative comparison confirms the insights from the quantitative analysis Tab. The two modorn vision backbones, ConvNeXt and SwinV2, show reconstructions that resemble the original very closely, not only in lerms of semantic content and spatial aligument but also in terms of color schemes and finegrained details. Semantic content and composition also mostly matches in case of the ResNet50, but not even the semantic content seems to be captured when using pooled representations example).
(ConvNeXt pooled
Conditioning on intermediate feature layers The experiments so far focused on the last feature layer before the classifier in order to be able to visualize the features that ultimately lead to the classification decision (e.g., for FeatTve- Viz in Section 4.3) However, FealTwe can also applied to intermediate feature layers. For demonstration, we trained three additional Featine models conditioned on the respective last feature of the first, second, and third stage within the ConvNeXt model. As expected, conditioning on earlier feature provides more specific spatial context and therefore leads to more accurate and detailed reconstructions, see the qualitative examples in Figure 14 in the supplemental material. As Table T we can also assess the reconstruction quality and the image quality quantitatively. The main difference compared to Table D is the ability to consider not only the cosine similarity compared to the feature layer but also compared to other feature layers, which were here taken identical to the three intermediate layers used for conditioning above. The results of this experiment are compiled in Figure g Most notably, the faithfulness extends beyond the layer that was used for conditioning, i.e., the models generally achieve good riconstruction across all considered feature layers. In line with expectations, the similarity increases when conditioning on earlier feature layers as they provide more finegrained information on the composition of the image. Furthermore, it is interesting to note that the similarity increases consistently for all models with the later stages the feature maps extracted from (except the last one). This suggests that the last feature maps from the second and
7

---

Published in Transactions on Machine Learning Research (12/2025)
third stages are less sensitive to reconstruction--rlltee pixel deviations and instead represent more robust, semantically stable image features.
edtacde from
secand
Fet.
3.5
2.44
0.8
5.79
377
3.0
A31
2.5
4.77
Biid Tunsd Paalarts Elage
<ecrnd
Int
1re seredd mind bet Pused Tudaroo tur Suags
tntt
excond
Iver
Thared Peaters Stage
a cosine-similarity
(b) FID score
(c) top5(1) matching
Figure Comparison of the cosine similarity, FID score and (1 for the models trained on the last feature maps of the first, second, and third stage of ConvNeXt calculated by averaging the cosine similarity of all superpixels. The models trained on earlier features and reconstructing them perform better in terms of cosine-sim, FID score and matchings than the one which was trained on the very last feature map
4.2 Robustness evaluation
Cross-model evaluation To assess the robustness of the presented results, we carry out cross-model comparisons where we measure model performance based on samples generated by conditioning on the feature map extract from a different model. The results for this experiment are compiled in Tab. - It turns out that all three sets of samples (conditioned features generated by the three different backbones) transfer quite remarkably to other models.
Table 2: Cross-model evaluation: Percentage of matching of the actual predictions (Lop5/ (opp/lpl and the predictions based on the reconstructions for different models. The Featlnv models based on the ResNet50, ConvNeXt and S2inV2 fcatures were used for the reconstruction and evaluated by the same three models.
evaluated by ConvNeXt
conditioned on
Nes5e
SwinV2
ResNet50
91% / 70%
90%
68%
91%
71%
ConvNeXt
92% / 73%
94%
77%
96%
81%
SwinV2
94% / 77%
94%
78%
95% / 80%
Evaluation with OOD samples In Fig. we qualitatively test the robustness of our findings by conditioning on samples that are slightly out of the model scope of the original models finetuned on ImageNet. To this end, we use the BAID (Yi et al., 2023) datasct, which differs in style from the samples ImageNet. We reconstructed 10,000 images from this dataset. The ControlNet trained on CouvNeXt features still shows good reconstruction quality of the semantic content but the style of the reconstruction and the original image differ more strongly than in the in-distribution case in Fig. 3 The average cosine similarity is 0.48 and the FID value is 12.98. Nevertheless, the results speak for the robustness of the proposed approach.
Evaluation with adversarial images We tested FeatInv with adversarial images, which are often mis- classified by models trained on ImageNet. For this, we used the ImageNet-A (Hendrycks al. (2019)) dataset, which contains 7,500 samples. In the case of misclassification, the model can be misled by image characteristics such small objects confusing patterns. We reconstructed all images in this dataset using
8

---

Published in Transactions on Machine Learning Research (12/2025)
Original
Reconstructed
0.53
0.51
0.76
0.51
0.59
2.46
D:69
0.51
Figure 5: Reconstruction of OOD samples. Comparison of randomly selected original and generated samples from the BATD (Yi et al., 2023) dataset, which differs in style from the samples in ImageNet. The basis for the reconstruction was the ControlNet trained on ConvNeXt features, which received the ConyNeXt features of the samples shown as input. The cosine similarity between the original feature map and that of the reconstruction is noted at the bottor edge of the images
our FeatInv model trained on ImageNet ConNeXt feature maps. The average cosine similarity is 0.56 and the FID score is 12.44. FeatInv is therefore able to reconstruct these unknown difficult images to a good degree. The t1)5(1) classification match is 0.71(0.38), which means that more than a third of the images are classified as the original image, suggesting that parts of the image relevant for classification are reconstructed in a meaningful way. Nevertheless, these matching values are smaller than those of the reconstructed simple images, indicating that the adversarial images are more complex and difficult to reconstruct.
Evaluation with corrupted images We also tested FeatIny with corrupted images. To this end, we used the ImageNet-C dataset (Hendrycks & Dietterich (2019)), which contains various categories of image corruption of varying degrees of intensity. Here, we focused on the weakest (level 1) and the strongest (level 5) corruption levels. In the supplementary material, we present the FID scores for the respective categories and the respective cosine similarity values as well as both the top 5(1) accuracy matches, which correspond to the cosine similarity results, and qualitative results (see Supplementary Material C As expected, all corruptions have a negative effect on both reconstruction quality and sample quality. Noise and contrast in particular have high negative impact the scores and cannot be reproduced well by FeatInv. However, it is worth noting that level 5 corruption represent represent a significant distortion of the original images and therefore represents a challenging robustness scenario.
4.3 Application: Featinv- Viz Visualizing concept steering in input space
Concept steering in input space In generative NLP, steering is sometimes used to verify concept interpretations by reducing or magnifying concepts the model activations and observing how this changes the generated output text, as famously demonstrated at the example of the Golden Gate concept (Bricken et al. 2024) in Claude 3 Sonnet. This approach is not directly applicable to vision classifiers. However, with our method of inverting model representations from feature to input space, we can observe the effect of concept steering within hidden model activations in the imput representation space instead of the output. This enables a novel method for concept visualization, with benefits over existing approaches (see below).
Concept definition Concepts are typically defined as structures feature space such as individual neurons, single directions or multi-dimensional subspaccs. Many concept-based XAI methods define a way to decomposc feature vector into concepts from dictionary / cconcept bank (Fel al. 2023). In this work, we use concepts from multi-diimensional concept discovery (MCD) (Vielhaben et al.) 2023), which defines concepts linear subspaces in feature space. Nevertheless, our approach is applicable to any concept discovery method.
Concept visualization through attenuated feature maps A common challenge for unsupervised concept. discovery methods is inferring the meaning of discovered concepts. To address this, we steer a concept in feature space and observe the effect in input space. Specifically, we attenuate coefficients for the concept under consideration to 25%, see the Supplementary Material D for details. Then, we use FeatIny to map the original and the modified feature map to input space using identical random seeds for the diffusion process.
9

---

Published in Transactions on Machine Learning Research (12/2025)
By comparing the resulting images, gain insights into how the concept is expressed in input space. We call this method FeatJnv- and present it in Algorithm
Algorithm 1: Feathnu- Viz: Visualization of concept steering in input space
Input: Model m, concept decomposition Φ E; 0i, concept with c
Output: Visualization of concept e in input space
Notation: RaxHxW where 20) refers to color channels with j E G, d' + Eo 0, + 0.25 de;
// Attenuated feature map
for i to n do
$; RandomSeed()
<- FeatInv( (Ø, seed = 8i)
// Original reconstruction
r + Featlnv (o' seed = 8:)
// AAttenuatted reconstruction
A, V EEE(RR, (2 24 11)
Euclidean distance
return median A
// Madian along sample axis
Exemplary results Fig. shows exemplary concopt stcoring visualizations for four samples from the Indigo Bunting class. Here, we decomposed ConvNeXt's feature space into three linear concept subspaces. FentInu- Viz provides visualization of these concepts input space. The method provides a very finegrained visualization of which specific regions in input space change upon steering each concept in feature space. More examples be found in the Supplementary Material [.I
Benefits We emphasize that Feather- Via extends commonly used concept activation maps in two ways: First, it provides a finegrained visualization rather than a coarse upscaling Bau et al. (2017); Vielhaben et al. 2023 of lower-resolution feature map. Second, it goes beyond merely verifying alignment with a predefined concepts Bau et al. (2017) by providing counterfactual information from targeted feature-map manipulations.
Original
Concept
Concept
Concept
Figure 6: FeatInv- Viz visualization of three concepts identified within ConvNeXt's feature space of the Indigo Bunting class, which can be associated with sky/backkrons background, bird head/breast and branches Seanches/leaves. leaves. For the visualization normalize the respective outputs of Algorithm D and threshold it below 0.33 as binary mask to indicate unaffected regions of the image.
10

---

Published in Transactions on Machine Learning Research (12/2025)
4.4 Application: Investigating the composite nature of the feature space
In NLP, well-known examples of feature-space arithmetic e-g- king man + woman queen Mikolov et al. 2013 have shaped our understanding of embedding geometries. Featinv offers insights into the composite nature of the feature space in vision models by conditioning feature maps from two samples. In particular, we investigate the effect of convex linear superpositions of two feature maps. To this end we linearly interpolate between the feature representations of two input samples and visualize reconstructions for different weighted combinations, as shown in Fig. [ We also indicate the cosine similarity between the reconstruction and the weighted feature map, which is highest for the original feature maps and typically reaches its lowest value for the equally weighted interpolated feature map. This can be seen as an indication that the weighted average of two feature maps is in general not well-defined operation. Nevertheless, foreground objects from one image and background from a second, seem to be reasonably combined through linear superposition (see egg. bird, landscape).
Original
100%
75%
Original
665
3.49
644
0.99
19
471
Figure 7: Reconstructions from weighted combinations of two ConvNeXt feature maps. The cosine similarity between the weighted feature map and that of the reconstruction is noted at the bottom edge of the images.
In Fig. g we show spatially composed combinations of two feature maps. The results indicate that feature maps exhibit very local influence, which aligns well with the simple upscaling of the fcature map resolution to the input resolution.
4.5 Limitations and future work
Our work is subject to different limitations, which provide directions for future investigations: First, the present work focuses exclusively the domain of natural images. It would be very instructive to extend the approach to other domains, such as medical imaging. Second, the proposed approach building the ControlNct method, builds on pretrained diffusion model, which might not be readily available in any application contexts. Third, every model and layer choice requires training a dedicated FeatIny model, which represents a computation hurdle. First experiments and the results in Tab. iidicate that finetuning could be beneficial to alleviate this issue. Finally, both application scenarios rely on modifications of the feature space. In order to obtain reliable results, it would be instrumental to introduce measures to detect input samples, i.e., feature maps that are outside the scope of the model.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Original
fell
left
oetter
bottom
full
Original
0.2
6:43
645
3,37
6,32
0.93
Figure 8: Reconstructions of spatially composed mixtures of two ConvNeXt feature maps. The cosine similarity between the manipulated map and that of the reconstruction is noted at the bottom edge of the images. The yellow outlines show the part of the feature map that was manipulated
5 Summary and Discussion
In this work, address the problem of obtaining insights into the structure of given model's feature map by means of a learned probabilistic mapping from feature space to input space, implemented as a conditional diffusion model. We demonstrate the feasibility of training such a model in a ControlNet-stylle achieving very accurate and robust reconstruction results across different model architectures. We present two possible applications both which relate to gaining inside into manipulated feature maps. However, we believe that the proposed approach could be widely applicable to further applications. We envision potentially positive societal impact through improved model understanding, along the lines of the concept steering use case. The source code underlying investigations is available at https://gittee com wwwm///TCeeaaltttO///eeatt.tooa
Acknowledgements
This work supported by the Bundesministerium für Forschung, Technologie und Raumfahrt (BMFTR) under Project INGVER, grant number 01KX2419.
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
Matan Avitan, Ryan Cotterell, Yoav Goldberg, and Shauli Ravfogel. A practical method for generating string counterfactuals. arxi preprint 1402.11355, 2025. URL
11355
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6541-6548 2017.
Leonard Bereska and Stratis Gavves. Mechanistic interpretability for Al safety a review. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openrevier: net cott/oorm?id- ePUVetPKu6 Survey Certification, Expert Certification.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self- supervised representation knows about. Transactions on Machine Learning Research, 2022. TSSN 2835-8856.
URL https::/openrrevie: http:/rwwwwwweeeeeeeeeeeeeeeeeee www.:ommmudddt///iiimm
Trenton Bricken, Ajay Reddy, Tim Conerly, Vikrant Varma, Lawrence Chan, Catherine Burns, and Neel Nanda. Scaling monosemanticity: Learning features that resist polysemanticity in large language models. Trans- former Cirewils, 2024. URL hittps://trans ormer-oorrrreoo circuits. wptpaaat....bb///////////// http://////saaalnmmoooeeeeeeeee/ index. Accessed: April 29, 2025.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29, 2016.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. 2009.
Patrick Esser, Robin Rombach, and Bjorn Ommer. A disentangling invertible interpretation network for explaining latent representations. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0223-9232. 2020.
Thomas Fel, Victor Boutin, Louis Béthune, Remi Cadene, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance cstimation. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Nearal Information Processing Systems, volume 36, pp. 54805-54 54818. Curran Associates, Inc., 2023.
Micah Goldblum, Hosscin Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. Battle of the backbones: large-scale comparison of pretrained models across computer vision tasks. Advances in Neural Information Processing Systems, 36, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advonces in neural information processing systems, 33:6840-6855 2020.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with itcrative attention. In International conference on machine learning, pp. 4651-4664. PMLR, 2021.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jac Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 12611122221, 2023.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEFE/ IEFE/CVP /CVF conference on computer vision and pattern recognition, pp. 12009 12019, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976 1777--11866 11986, 2022b.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word repre- sentations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL
John Xavier Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. Text embeddings reveal (almost) as much as text. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL https://openrevievv. net http://www..tttoommoododeeee/o/
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016.
Justin Pinkney, minisd. http:s//wwugnnngsdeeco//////t/tr 2023, Hugging Face Model Repository.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer--ased language models. arXiv preprint arXiu:2407.02646 2024.
Jan Rathjens, Shirin Reyhanian, David Kappel, and Laurenz Wiskott. Inverting transformer based vision models. arXiv preprint 2412.06534, 2024. URL https://arxivy wtt.://wwrrr/abb//2..e.ee.
Robin Rombach, Patrick Esser, and Björn Ommer. Making Sense of CNNs: Interpreting Deep Repre- sentations and Their Invarionces with INNs, pp. 647-664. Springer International Publishing, 2020.
ISBN 9783030585204.
doi: (011007/978-3-000-555220 38. URL hot./xxxxooo...o
978-3-030-58820-41
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE international conference on computer vision, pp. 618 626, 2017.
Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensionall concept discovery (MCD): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023.
ISSN 2835-8856. URL httts:://peerreee httt:/oorum2/ddd4xx2/P2///
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. In NewrIPS 2021 Workshop on ImageNel: Past, Present, and Fulure, 2021. URL https:
/openreviev w/wppereevww..ee//oooroo http:rrwwwww..etttooouiiiei.eeee http/fforummidddNN6JJJ......
Ran Yi, Haoyuan Tian, Zhihao Gu, Yu-Kun Lai, and Paul L. Rosin. Towards artistic image aesthetics assessment: A large-scale dataset and a new method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22388 22388-22397, June 2023.
MD Zeiler. Visualizing and understanding convolutional networks. In European conference on computer mision, volume 1311, 2014.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 2023a.
Tianyi Zhang, Zheng Wang, Jing Huang, Mohinddin Muhammad Tasnim, and Wei Shi. A survey of diffusion based image generation models: Issues and their solutions. arXiv preprint Xrx::2330. 13142, 2023b.
15

---

Published in Transactions on Machine Learning Research (12/2025)
A Ablation studies
A.1 Additional models
For the ablation studies, we consider ResNet550F (Wightman et al. 2021 as second ResNet-model to study the impact of the training procedure, in addition to the models studied in the main text.
Choice of conditional encoder Next to the bilinear upsampling followed by shallow CNN, explore different design choice for the conditional encoder: To this end, we explore the use of cross-attention as used in the Perceiver architecture (Jaegle et al. 2021), which uses learnable representation of predefined size and connects via cross-allention to the representation that serves as input for the mapping. Unlike the previous approach, this approach is not subject to any locality assumptions and therefore the most flexible approach that is in particular suitable for model architectures without built-in locality assumptions such as vision transformers. We carry out a comparison between the two conditional encoder models for the of the SwinV2 VIT, where expect the impact to be most pronounced the VIT operates visual tokens, which might not align well with the upscaling operation in the convolutional encoder. On the contrary, results indicates that the convolutional encoder yields better classification results and therefore for the following experiments we restrict ourselves to the convolutional encoder.
Class-condiitonal baseline We tested the models on an unconditional baseline to see MiniSD is able to generate good representations of the classes and if they can be correctly classified. To define the classes for the input prompt as precisely as possible,
the WordNet hierarchy and create the prompt follows: 'a high-quality, detailed, and professional image of CLASS, which is kind of SUPERCLASS as for example aa high-quality, detailed, and professional image of tench, which is kind of cyprinid'. The unconditional baseline created consists of 10 samples per class, ie. 10.000 samples in total.
Table Additional model insights: For the different backbones, we indicate the corresponding canonical input sizes as well as the top5(1) accuracy on an unconditional dataset, created by MiniSD. Since only the base model is used for the prediction of these unconditional samples, the results do not differ between unpooled and pooled as well as for the different SwinV2 archilectures. We show three performance metrics: Cosine similarity in feature space (cosine-sim), calculated by taking the mean of the cosine similarity of all superpixels, top5(1) matches using the top1 prediction of the original sample as ground truth (top5(1) match) and FID-scores (FID) to assess the quality of the generated samples. We consider generative models conditioned on unpooled leature maps (rows 1-5) and models conditioned on pooled feature maps (rows 6-9). While reconstructions were generated using model-specific values for control strength and guidance scale, also observed that good results can be achieved with uniform values. A control strength of 1.5(1) and guidance scale of 8(1.75) for (un)pooled conditional input seemed to work well for this scenario.
Model
input
MiniSD t1p5(1)
cosine-sim
top5 match
FID
ResNet50
89% (68%)
0.46
91% (70%)
11.49
umpooled
224 x 224
ResNet50B
88% (65%)
0.50
90% (70%)
11.51
ConvNeXt
288 x 288
92% (71%)
0.61
94% (77%)
8.20
SwinV2
0.53
95% (80%)
12.69
384 x 384
93% (72%)
SSinV2((erceiver)))
0.36
76% (53%)
22.65
ResNet50
89% (68%)
0.12
48% (23%)
31.64
224 x 224
pooled
ResNet50B
88% (65%)
0.14
44% (21%)
37.44
ConvNeXt
288 x 288
92% (71%)
0.19
44% (20%)
31.67
SwinV2
384 x 384
93% (72%)
0.16
47% (22%)
49.04
2timm model weights: resnet50.a1 in1k
16

---

Published in Transactions on Machine Learning Research (12/2025)
A.2 Control Strength Guidance Scale
In Fig. 9 we present several original images alongside their respective reconstructions, generated using the model trained with the ConvNeXt backbone. These samples were produced with varying levels of control strength and guidance scale. Since both significantly influences the output, we evaluated the model's performance across different scale settings, again using the ConvNeXt-basse model. To quantify the quality of the generated images, we calculated the Top-1 classification match as well as the cosine similarity, illustrated in Fig. I0 and [I] Both correlate strongly and show the same goog performing ranges. This suggests that the unpooled feature map requires minimal text guidance, whereas for the pooled variant, guidance is of substantial importance, natural looking images cannot be generated without its influence.
Original
Unponked
Pooled
(a) Reconstructions for class Magpie
Original
Unpuoled Calaaic mide
Pooled
A38
A39
(b) Reconstructions for class Kuvasz
Figure 9: Unpooled and pooled reconstructions with different control strength and guidance scales from ConvNeXt feature maps for class magpie and kuvasz. The cosine similarity between the original feature map and that of the reconstruction is noted at the bottom edge of the images. A low control strength already results in a sufficient reconstruction. While increasing the guidance scale in unpooled samples results in higher color saturation and more distorted object shapes, the same increase improves object fidelity in pooled feature maps. This trend is also roughly reflected in the corresponding cosine similarity.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Pooled Top-1 Accuracy
Unpooled Top-1 Accuracy
022
0.2
83
60
075
0.5
52
Contro Strongh (ed)
10
Control Strengt tsst
10
1.23
15
25
1.25
25
25
175
1.75
10
22
225
225
23
61
10
182
16.7
5122
512
Culdance Scale (91)
Guidence Scale 191)
(a) ResNet50
Top-1 Accuracy
Unpooled Top-1 Classification Accuracy
20
10 33 14
03
175
073
130
Control Srenge (es)
10
Control Strence (CS)
125
1:23
11
13
15:35
15
10
10
10
30 10 20
70
14
75
74 32
1000
173
2.
12
20
123
5
11
23
20
20
29
32 30
235
12
23
10
68
Guidance Scale (05)
Guidance Scale
(b) ConvNeXt
Pooled Top-1 Classification Accuracy
Unpooled Top-1 Classification Accuracy
25
023
8-
679
Control Shergth (ed)
13
Control S Steegth (ed)
32
122
18
24
24
33
22 22
123
71
113
12
1.7
23
35
30
20
123
20
22
71
21
29
32
21
P0
25
16
19
10
18
11
18
123
25
24
300
105
11
100
105
116
126
Guidance Scale (qs)
Guidance Scale
(c) SwinV2
Figure 10: Impact of control strength and guidance scale generation quality (top-1 accuracy). Top-1 classification accuracy for ResNet50, ConvNeXt and SwinV2 backbone with and without pooling. One image per class was reconstructed for each combination. Higher values indicate better performance. The unpooled model does not seem to rely on guidance as much as the pooled model.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Similarity
Unpooled Similarity
025
26
Contr Strengh tell
10
Conera Steeng nel
123
15
00
172
25
-2
23 223
223
530
23
-.05
1950
(a) ResNet50
Pooled Cosine Similarity (x100)
Unpooled Cosine Similarity (x100)
866
2- 1-11
33
15:14
12
075
40
51
32
60
51
01
01
92
60
59
57 57
15
Contro Steogh ros)
21
0.11
Exetren Smereghh fest
13
D12
122
20
20
21
21
31
31
125
10
222
2,25
33
255
13
53 132
32
2333
19.5
12.0
(gs)
(b) ConvNeXt
Pooled Cosine Similarity (x100)
Unpooled Cosine Similarity (x100)
0.55
-55
03
05
579
375
0.54
54
Commo Strergh (es)
10
14
15
17
51 52
Contel Smenghh
CoATe WET
L22
0.22
13
A32
20
15 15
28
2.29
24
23
10
10
60
100 155
11.
120
(c) SwinV2
Figure 11: Impact of control strength and guidance scale on generation quality (cosine-sim). Cosine similarity (x100) for ResNet50, ConvNeXt and SwinV2 backbone with and without pooling. One image class was reconstructed for each combination. Higher values indicate better performance. The unpooled model does not seem to rely on guidance as much as the pooled model.
A.3 Variability across reconstructions
As the model we use for reconstruction is probabilistic model, we show the variance of the generations qualitatively when different seeds used. This can be seen in Figure 12 The observed variability supports
19

---

Published in Transactions on Machine Learning Research (12/2025)
the necessity of probabilistie rather than deterministic mapping for the ill-posed problem of mapping from feature space to input space.
Irpootal Carehids
Servi
Origeal
itadeers
SeeV
Tepooaa Combboo
pablil
RonNunt
Senii Gamalod: RuNNrr Spae
Origial
potla
Figure 12: Reconstruction variations feature maps from ResNet50, ConvNeXt, and SwinV2, each unpooled and pooled for the classes volcano, strawberry, and flamingo. It noticeable that the unpooled reconstructions mostly resemble each other; only ResNet50 displays noticeable differences. However, the reconstructions pooled feature maps different. Here, the images appear very diverse, even though they are semantically similar. For example, the volcano class in ResNet50 pooled shows lype of underwater landscape consistently, while the pooled images in ConvNext all show red bushes. Consistency between the reconstructions the pooled feature maps can also be seen in the other classes, although these are never as strong as those the unpooled feature maps.
20

---

Published in Transactions on Machine Learning Research (12/2025)
A.4 Generation unpooled vs. pooled feature maps
We analyzed the generated samples for each model using both unpooled and pooled feature maps. As shown in Tables | and 3 the samples generated from pooled feature maps consistently exhibit lower visual quality compared to those generated from unpooled feature maps. This indicates that preserving spatial detail during feature extraction is crucial for high-quality image generation. Fig. 13 illustrates this difference with examples from the class zebra, clearly highlighting the superior fidelity of samples generated from unpooled features.
Original
pooled unpooled
Figure 13: Comparison of unpooled and pooled generated samples showing examples of the class Zebra using. All samples were generated using the model trained with the ConvNeXt backbone. The unpooled feature maps result in sharper and more realistic generations, while the pooled versions show a noticeable loss detail and structure, mostly showing general context of the original image. The cosine similarity between the original feature map and that of the reconstruction is noted the bottom edge of the images
A.5 Comparison to RCDM
Approach In this section, we provide a direct comparison to the RCDM approach Bordes et al. (2022 as our closest competitor. The RCDM approach conditions on pooled fcature representations and can therefore be most meaningfully compared to Feal/ne F We stress at this point, that this experiment facilitates a comparison of different approaches to train diffusion models conditioned on pooled fcature maps but does not touch the central novelty of the submission, i.c., conditioning on unpooled feature maps. Whereas RCDM trains a conditional diffusson model from scratch, FeatInw's 's ControlNet approach leverages a pretrained diffusion model.
pooled feature maps, which receives the same input.
ResNet50 We present the results for RCDM model Bordes et al. 2022 conditioned pooled ResNet50 feature maps, which produces samples at resolution of 128x 128 pixels. Here, we leverage model weights provided by the original authors. To evaluate the quality of the generated images, we computed the same metrics as for the Feat/ne models presented in the main text, as shown in Table 4 The results are slightly non-trivial to compare between RCDM and FeatInva as the output resolutions differ (128x 128 vs. 256x256) which leaves the option of either evaluating RCDM outputs at their native resolution or scaling it up 224x224. Irrespective of the precise evaluation procodure, RCDM shows slightly higher cosine similaritics (0.16-0.29 vs. 0.12) and higher FID scores (13.0-28.3 vs. 31.6) than pooled FealIve models. However, these values remain lower than those achieved by approach using unpooled feature maps.
ConvNeXt We also trained an RCDM model oursclves based on ConvNeXt as ResNet50 showed the worst results across all investigated models in the main (ext. We therefore used the official RCDM code repository to train a RCDM model conditioned on pooled ConvNeXt feature maps, as before on a resolution 128 128, After 95 hours of training on four L40 GPUs with a batch size of eight, the model had reached 520,000 steps. It performs worse than our FeatIne equivalents which shows that it either did not fully convergence, indicating that further training would be needed or is not capable of recreating matching samples based
21

---

Published in Transactions on Machine Learning Research (12/2025)
the ConvNeXt features. Our FeatInv models, which were trained using smaller batch size of two for less time, saw less images after 1,920,000 steps and performed better. This demonstrates the benefit of using a pre-trained diffusion model rather than training one from scratch.
Conclusion While RCDM is technically capable scaling to higher resolutions than 128x128, doing so requires long training times. In our experiments, we trained RCDM at half the resolution of our Feathra models and for longer duration, yet its performance still lagged behind FeotIne. In contrast, Featlwe could be adapted to even higher imput resolutions as it leverages pretrained diffusion model. Results are difficult to comparable directly due to different imput resolutions. RCDM seems slightly superior in the case of ResNet50, but clearly inferior in the case of ConvNeXt. In any case, the RCDM approach is not able to deliver models conditioned on unpooled feature maps, which yield clearly superior reconstruction quality and better FID scores. This clearly positions FeatInva as the more versatile approach.
We computed cosine similarity between feature maps of generated and original images. Since there a significant gap between the dimensions of the original and generated samples, we also examined the results when both were scaled to the same low imput size (resulting in 4x4 feature maps), which increased the similarity. Classification accuracy and FID score likewise improved when both inputs were resized to the lower resolution. The ConvNeXt RCDM does not reach the results of our FeatInv equivalett after seeing more than the same amount of images in training.
Table 4: Comparison between FeatInv and RCDM:
Model
input
cosine-sim
top5(1) match
FID
RCDM
ResNet50
128 128
0.29
69% (42%)
13.02
ResNet50
224 224
0.16
65% (37%)
28.26
ConvNeXt
128 x 128
0.12
26% (10%)
37.20
ConvNeXt
224 x 224
0.14
26% (10%)
57.46
Featluv
ResNet50
224 x 224
0.46
91% (70%)
11.49
ConvNeXt
288 288
0.61
94% (77%)
8.20
ResNet50 pooled
224 x 224
0.12
48% (23%)
31.64
ConvNeXt pooled
288 x 288
0.19
44% (20%)
31.67
B Conditioning on intermediate feature layers
To condition on intermediate feature layers, we adjusted the input and hidden dimensions of our input encoder to the different dimensions in the model. Unlike the model on the last feature map, each of these models was trained for only one epoch. We observed that training on the last feature maps from early ConvNeXt stages in particular led good reconstruction results much faster than training on features from the last layer. For these models, we also reconstrueted 10 validation images per class (see Figure 14] and calculated cosine similarity, FID and matching accuracies (see Figure For the reconstruction we provide the last. feature map from the stage the model was trained
In Figure 14 it can be seen that the models that reconstruct earlier feature maps work more accurately and display details better (e.g., watermark text). Colors and edges within the image are also displayed better.
C Reconstruction of corrupted images
Both the FID score (see Figure 15) and cosine similarity (sce Figure 16 suggest that FeatInv can only represent these manipulation up to certain level. We show the top5(1) accuracy matches for the corrupted images and their reconstructions (see Figure 17 18 In addition we present some examples of what the reconstruction of the different categories
We used Featlnv to reconstruct corrupted images as described in Section 4.1
with Featlny looks like (see Figures
19
20
21
22 and 23
22

---

Published in Transactions on Machine Learning Research (12/2025)
Original
first
CHANGE
8:50
Feature Block sccond
third
last
0.32
663
CL92
5.0
0.35
055
Figure 14: Comparison of reconstructions of models trained on the last feature maps from different ConvNeXt stages with dimensions (128, 72, 72), (256, 36, 36), and (512, 18, 18). The cosine similarity of the original feature map and that of the reconstruction is noted at the bottom edge of the images.
Level
Level
30
10
Bouistan, relis
shet, nolse
impulse roloe
deloccus blur
glass blu
meton blur
zoin, blr
frast
krow
brighteess
fog
elastic Transform
contrast
loeg compression
pisalate
specite ralse
gaussian. blur
spatter
saturate
Noise
Blur
Weather
Digital
Extra
Figure 15: Comparison of the FID scores of the ImageNet-CC categories. While a slight corruption of the images, as in level 1, only leads to a slight increase in the FID score, high corruption has a significant impact on the score. The higher the corruption, the more difficult is for FeatInv to reconstruct the corrupted images. It should be noted that the model was not trained any of these corruptions. Noise and contrast in particular have a high negative impact on the scores and cannot be reproduced well by FeatInv.
23

---

Published in Transactions on Machine Learning Research (12/2025)
0.6
Level 1 Level
0.5-
Cosine Similrity
0.3
0.2
0.1
gausstan ralse shot, nolse Impuise.
0.0
deforas Buur
metio blr
zoom, blu
frost
snow
brightnees
log
elnatic transtorm
contrast
pirelate
compression
speckle nolse
gaussian blu
spottor
saturate
glani
lpeg,
Noise
Blur
Weather
Digital
Figure 16: Comparison of the cosine similarity of the feature maps of the corrupted input images and their reconstructed images. Both minor (level 1) and major corruption (level 5) have a strong influence on the cosine similarity of the final feature maps and cause them to drop. It should be noted again that the model was not trained on any of these corruptions. Unlike the FID scores of the images in these different categorics, there does not seem to be any category in which the change differs significantly from that of the other categories.
Level 1
Level
Match Ratio
0.6
0.2
gauestan noise
0.0
shot, noise Inguise mider
defocis blur
glast blur
metion bur
zom, Blur
frost
know
brigitness
fog
elastic transform
contrast
pirelate
speclle noise
gruselan blur
Spatter
saturate
Ipeg_
Noise
Blur
Weather
Digital
Extra
Figure 17: Comparison of the top5 accuracy matches of the ImageNet-C categories. While slight corruption such as in level 1 achieves almost the same value in top5 accuracy matching as the unaltered images, stronger corruption such as in level 5 has much greater negative impact on the matching. Compared to the cosine similarity in Figure [6] the matching works unexpectedly well, which is most likely due to the fact that the model assigns the classes of the two images, one of which was reconstructed by featinv and may not contain the complete corruption, to the same class. The result is different when there large difference the reconstruction.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Level
0.6
Match Ratio
0.5
0.4
0.2
0.1
gausstan robe stot, nolse Impuise reise
defoccs blur
glass Blyr
mation Blr
zoomn blu
frrst
brightnees
bgg
elatic transferm
contrast
pirelata
speakle notse
paussian blur
Spattor
sa Saturate
compressoo
Ipeg
Noise
Blur
Weather
Digital
Extra
Figure 18: Comparison of the topl accuracy matches of the ImageNet-C categories. As expected, the results of topl accuracy matching correspond to those of lop5 accuracy matching. As with the unaltered images, the values for topl matching are lower than those for top5 matching.
level
gaussian noise
leval 5
Seconntrumed
level
shot noise
leve! 5
Gredral
Recontmicte
impulse noise
Dergioal
Figure 19: Reconstructed corrupted images: Noise (gaussian noise, shot noise, and impulse noise). In each category, the original corrupted images be seen the top and the reconstructed images below. On the left is level 1, iee, weak corruption, and on the right is level 5, i.e., strong corruption. It can be seen that FeatIny is not very good at reconstructing detailed noise, and the images lose detail. In particular, the colors of the noise are lost.
25

---

Published in Transactions on Machine Learning Research (12/2025)
level
defocus blur
level 5
Seccostinnted Oaginal
level
glass blur
level 5
Becontimntee Grestnl
level
motion blur
leval 5
level
zoom blur
Deccomanie Gropoal
Figure 20: Reconstructed corrupted images: Blur (defocus blur, glass blur, motion blur, and zoom blur). 0 In cach category, the original corrupted images are shown above and the reconstructed images below. On the left is level 1, i.e., weak corruption, and on the right is level 5, i.e., strong corruption. It can seen that FeatIny is quite capable of reconstructing the images. Both levels and show good results.
26

---

Published in Transactions on Machine Learning Research (12/2025)
frost
level
level 5
Reconshuuee Oagaa
level
snow
level 5
Decoontummd Cropni
level
fog
level
Reco
brightness
Origiaa
Figure 21: Reconstructed corrupted images: Weather (frost, snow, fog, and brightness.) In each category, the original corrupted images are shown above and the reconstructed images below. On the left is level 1, i.c., weak corruption, and on the right is level 5, i.c., severe corruption. It can be seen that FeatIny is quite capable of reconstructing the images. However, there are images where the reconstruction does not work as well, such as the second image, the flying bird. In all categories except fog, the reconstruction differs greatly from the original corrupted image.
27

---

Published in Transactions on Machine Learning Research (12/2025)
contrast
originnl
Ressnerrutte
elastic transform
level
Ohiginal
Deccontt ucheed
level
pixelate
level
Chighol
level
jpeg compression
level
Reconntrumed Orggaa
Figure 22: Reconstructed corrupted images: Digital (contrast, elastic transform, pixelate, and jpeg compression.) In each category, the original corrupted images are shown above and the reconstructed images below. On the left level 1, ie., weak corruption, and on the right level 5, Le., severe corruption. It can be seen that FeatInv quite capable of reconstructing the images in the last three categories well. However, this does not work for contrast. Especially at level 5, the reconstructed images differ significantly from the original corrupted image. The main object is more emphasized than in the image with low contrast.
28

---

Published in Transactions on Machine Learning Research (12/2025)
lvel
speckle noise
level 5
Reccnehuuee Orgia
level
spatter
level 5
Deccontuumed Gripts
level
gaussian blur
level
saturate
Origiaa
Figure 23: Reconstructed corrupted images: Extra (speckle noise, spatter, gaussian blur and saturate.) In each category, the original corrupted images are shown above and the reconstructed images below. On the left is level 1, i.c. weak corruption, and on the right is level 5, i.c. strong corruption. Excopt for the first category, speckle noise, FeatIny is able lo reconstruct the images well. In the case of spatter, the dots are reconstructed, but some are located differently or are fewer number. In the case speckle noise, similar to the reconstructions Figure 19 the noise is well represented in the reconstructed images.
29

---

Published in Transactions on Machine Learning Research (12/2025)
D Details on concept steering
For concept discovery, we rely on multi-dimensional concept discovery (MCD) Vielhaben et al. 2023). For every feature vector MCD provides concept decomposition o Σ i= @i. where oi is associated with the concept i (of in total nee concepts), which represents linear subspace of the feature space, and concept + 1 corresponds to the orthogonal complement of the span of all concept subspaces. The latter is necessary to achieve a complete decomposition not explicitly captured through concept subspaces. For a given feature vector, one can now quantify the contribution o, from concept and visualize its magnitude | across the entire feature map to obtain spatially resolved concept activation map. One option to align such a coarse concept activation map with the input image to use bilinear upsampling. This process often leads MCD for demonstration, this alignment step is
rather diffuse concept activation maps. Even though common challenge for most concept-based attribution maps.
We used the learned mapping from feature space to imput space to infer high-resolution concept visualizations. To this end, the component &i associated with concept was multiplied by 0.25 to attenuate them. In our experiments, decreasing the feature values worked better than increasing them. We speculate that increasing feature vector components can eventually result in feature vectors that exceed the magnitude of feature vectors seen during training of the FeatInv model.
To obtain higher-resolution representations, both the original feature map and the concept-maniitlatee feature map. Using the same random seed for each pair ensured that the original and manipulated reconstructions were directly comparable, with differences attributable solcly to the feature manipulation. For cach pair, we computed the pixcl-wise difference, and produce a representative difference map for each concept, we look the median across the five resulting difference maps. This yielded high-resolution (256x256) activation map that highlights the specific regions of the image affected by the manipulation.
used Featlnv to reconstruct five samples for each concept using
D.1 Featinv- Viz visualization of other classes
To provide a closer look at FeatIne- Viz, we present visualizations of three concepts for the classes Great White Shark, Fly, African Elephant, and Water Ouzel. For the visualization we normalize the respective outputs of Algorithm 1 and threshold it below 0.33 binary mask to indicate unaffected regions of the image.
Original
Concept
Concapt
Concept
Figure 24: FeatIne- Viz visualization of Great White Shark concepts identified within ConvNeXt's feature space of the Great White Shark class, which can be associated with shark/ /water, water and shark. Concept 1 and 2 both seem to show water while concept also highlights parts of the shark.
30

---

Published in Transactions on Machine Learning Research (12/2025)
Original
Concept
Concept
Concept:
Figure 25: FeatInv- Viz visualization of Fly concepts identified within ConvNeXt's feature space of the Fly class, which can be associated with fly's body, outline/part of fly/leas /leavee and leaves/green. Interestingly, Concept 3 focuses only on the supposedly green background in (see images 3 and 4). The rest of the background does not appear to be part of the concept.
Original
Concept
Concepe
Contept
Figure 26: FeatInv- Viz visualization of African Elephant concepts identified within ConvNeXt's feature space of the African Elephant class, which can be associated with tusks, outline/part of elephant and grass/ske sky. Unlike with the Great. White Shark or Fly class, only the outlines of the elephant concept can be soen here.
31

---

Published in Transactions on Machine Learning Research (12/2025)
Original
Concept
Concept2
Concent
Figure 27: FeatInv- Viz visualization of Water Ouzel concepts identified within ConvNeXt's feature
space of the Water Ouzel class, which can be associated with water/sky boundary, outline/ Maaiinee/rrrtt.ee of bird and water/sky. As in the concepts of the African Elephant, only an outline of the bird can be seen in these reprosentations.
32


Reviews:
[{"review_id": "iOtcDB3Q1A", "reviewer": "Reviewer_d8wL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes FeatInv, a conditional diffusion model for reconstructing input image from feature representation of pre-trained vision encoders. The goal is to obtain high-fidelity image reconstructions whose encoded features align with those of the original input. The main idea is to finetune a pretrained diffusion model in a ControlNet fashion with conditions from pre-pooling features. \n\n**Summary of contributions**\n- Implements FeatInv, a ControlNet variant that can reconstruct natural images from their pre-pooling feature maps of vision encoders. \n- Produces visually coherent and realistic reconstructions, suggesting that diffusion-based inversion is feasible for feature-space visualization.\n- Presents qualitative applications of FeatInv for visualization of concept direction and concepts composition in the feature space. \n\n**Strengths**\n- Provide both quantitative and qualitative results to show that pre-pooling features preserve sufficient local structure for high-quality reconstruction. \n- Consistent reconstruction and classification alignment across three diverse backbones.\n\n**Weaknesses**\n- The main methodological difference from RCDM (Bordes et al. 2022) is limited. The primary changes include the use of pre-pooling feature instead of post-pooling feature, and the use of ControlNet for conditioning. \n- To the best of my knowledge, the value of FeatInv is unclear beyond visualization. \n\t- FeatInv does not discover, analyze, or interpret internal concepts. It assumes known concept vectors (e.g., from MCD) and visualizes their attenuation in input space.\n\t- Feature composition results do not seem realistic. \n- The authors did not discuss the choice of pre-trained diffusion model. Does the diffusion model need to be trained on the same distribution as the vision encoder? More importantly, to what extent the reconstruction is biased by the pre-trained diffusion model?", "claims_and_evidence_yn": "Yes", "claims_explanation": "The qualitative and quantitative results convincingly demonstrate that conditioning on spatial (pre-pooling) features leads to higher reconstruction fidelity. The concept-steering and feature-composition results are mostly qualitative demonstrations.", "audience_yn": "No", "audience_explanation": "While FeatInv confirms that diffusion-based inversion of feature maps is feasible and produces visually realistic images, its scientific contribution is limited. The best use case might be visualization tool for feature space but it does not provide new insights into model representations.", "requested_changes": "- Beyond the use of pre-pooling features and ControlNet conditioning, what fundamental problem or capability does FeatInv address that RCDM cannot?\n- Could you discuss how the pre-trained diffusion model prior might bias the reconstruction? For instance, what happens if the diffusion model is trained on a dataset different from the encoder’s (e.g., FFHQ vs. ImageNet)?\n- Can FeatInv support quantitative interpretability, debugging, or representation analysis? Could you discuss what additional insights it can provide beyond rendering visually plausible images?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "12TwK4KEO0", "reviewer": "Reviewer_4GxC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces FeatInv, a novel method for mapping the internal feature representation of a deep neural network to a high-fidelity image. The process is inversion in the sense that the generated image should produce a similar feature representation to the original image. The core contribution is the use of a pretrained conditional diffusion model to learn this mapping. Unlike previous methods that often rely on deterministic or approximate approaches, FeatInv leverages the generative power of diffusion models to produce high-fidelity, spatially resolved images conditioned on specific feature maps. The authors explore the potential of this idea and validate the method's effectiveness through numerical experiments. Furthermore, the paper illustrates practical applications, such as visualizing concept steering and exploring the composite nature of feature representations, thereby offering a powerful new tool for network interpretability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are backed by numerical experiments.", "audience_yn": "Yes", "audience_explanation": "This work will be of high interest to researchers and practitioners in machine learning, particularly those focused on computer vision and model interpretability. The problem of understanding what neural networks learn is a fundamental and persistent challenge in the field. This paper presents a clear, well-motivated, and technically sound approach.", "requested_changes": "The paper is well-written and the core methodology seems sound. I appreciate the fact that the proposed idea is simple, yet well-explored.\n\nIn the \"Architecture and training procedure\" section, the final sentence is grammatically incorrect.\n\nIn the first sentence of the \"Conditional input encoder\" section, the word \"import\" should be corrected to \"important\".", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ciiopMxQVk", "reviewer": "Reviewer_qsaR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\nThis paper introduces FeatInv, inverting feature representations back to input space by conditioning a pretrained diffusion model (ControlNet-style) on spatially resolved feature maps. While feature inversion for explainability has been studied extensively in the deep learning community, this work provides an empirical extension by leveraging ControlNet to reconstruct high-resolution, visually realistic inputs. The work also demonstrates applications such as concept steering and compositional feature visualization.\n\n\n## Strengths\nI believe this work will be of interest to the community.\n1. While feature inversion has been extensively studied, this paper leverages diffusion models (ControlNet-style conditioning) to achieve higher-fidelity and probabilistic reconstructions, offering a clear improvement over prior deterministic approaches.\n2. This paper showcases interesting applications, including concept steering visualization and compositional feature analysis, highlighting the broader utility of the method.\n\n## Weaknesses\n1. **Incremental advances in feature inversion**. Prior work [1-3] already explored feature inversion with generative models. The main difference here is conditioning diffusion models on unpooled feature maps.\n2. **Justification of feature map choice**. The argument regarding **pooled** and **unpooled** representations not fully justify the design decision. The paper claims that conditioning on feature maps before pooling leads to better reconstructions. However, this naturally raises the question: why not condition on even earlier layers, which may contain richer spatial detail? The work would benefit from a clearer rationale for focusing specifically on the last pre-pooling feature map, rather than earlier alternatives. \n3. **Faithfulness of explanations**. While the reconstructions are visually compelling, the work does not establish whether they are faithful explanations of the underlying model’s reasoning [4]. I feel reconstruction resolution is not a major concern in explainability. Without guarantees or evaluations of faithfulness, it remains unclear to what extent the reconstructed images truly reflect the model’s decision process rather than artifacts introduced by the generative prior. \n4. **Experiments**. It would be valuable to examine how the method explains perturbed samples, such as adversarially perturbed inputs, corrupted images, or other distribution shifts. \n\n\n\nReferences:\n[1] Alexey Dosovitskiy, Thomas Brox, Generating Images with Perceptual Similarity Metrics based on Deep Networks\n[2] Florian Bordes, Randall Balestriero, Pascal Vincent, High Fidelity Visualization of What Your Self-Supervised Representation Knows About\n[3] Robin Rombach, Patrick Esser, Björn Ommer, Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs\n[4] Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, Pradeep Ravikumar, On the (In)fidelity and Sensitivity for Explanations", "claims_and_evidence_yn": "Yes", "claims_explanation": "NA", "audience_yn": "Yes", "audience_explanation": "I believe this work will be of interest to the community.", "requested_changes": "## Requested revisions.\n1. I would like to suggest the authors re-state the claims in the contributions. In particular, regarding the positioning of this work regarding how it advances the feature inversion research line. \n2. While the method achieves good  reconstruction fidelity, this alone does not establish its value for explainable AI. High-fidelity reconstructions demonstrate that the model can generate realistic images aligned with feature representations, but they do not by themselves clarify how those features contribute to the model’s decision-making. A stronger connection between reconstruction fidelity and explanatory faithfulness is needed to justify the interpretability claims.\n3. Regarding the argument the pooled versus unpooled. The high reconstruction quality can't justify this work. We need the good causal connection between explanations and decisions instead of the reconstructed quality.\n4. Experiments. Would it be possible to assess how adversarial samples, corrupted samples, OOD affect this framework?", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)