Uncovering Semantic Selectivity of Latent Groups in Higher Visual Cortex with Mutual Information-Guided Diffusion
Authors: Yule Wang
Joseph Yu
Chengrui Li
Weihan Li
Anqi Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Latent Discovery
Diffusion Models
Visual Cortex
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose MIG-Vis, a method that uses mutual information-guided diffusion to synthesize images, thereby uncovering neural latent groups that exhibit clear semantic selectivity for pose, inter-category variation, and intra-category content.

Abstract: Understanding how neural populations in higher visual areas encode object-centered visual information remains a key challenge in computational neuroscience. Prior work has examined representational alignment between artificial neural networks and the visual cortex, but these findings are indirect and provide limited insight into the structure of neural population coding. Decoding-based methods can recover semantic features from neural activity, yet they do not reveal how these features are organized. This leaves an open question: how feature-specific visual information is distributed across neural populations, and whether it forms structured, semantically meaningful subspaces. To address this, we introduce MIG-Vis, a method that leverages diffusion models to visualize and validate visual-semantic attributes encoded in neural latent subspaces. MIG-Vis first learns a group-wise disentangled neural latent representation using a variational autoencoder, and then applies mutual information (MI)-guided diffusion synthesis to visualize the visual-semantic features encoded in each latent group. We validate MIG-Vis on multi-session neural spiking datasets from the inferior temporal (IT) cortex of two macaques. The synthesized results show that MIG-Vis identifies neural latent groups with clear semantic selectivity to diverse visual features, including object pose, inter-category variation, and intra-category content. These findings provide direct and interpretable evidence of structured semantic representation in higher visual cortex.

Published as a conference paper at ICLR 2026
UNCOVERING SEMANTIC SELECTIVITY OF LATENT GROUPS IN HIGHER VISUAL CORTEX WITH MUTUAL INFORMATION-GUIDER DIFFUSION
Yule Wang, Joseph Yu, Chengrui Li, Weihan Li, Anqi Wu Georgia Institute of Technology Atlanta, GA 30332, USA (yulewang, jyu425, cn enlichengrui, weihanli, angivu]]gatech. edu
ABSTRACT
Understanding how neural populations in higher visual areas encode object- centered visual information remains key challenge in computational neuro- science. Prior work has examined representational alignment between artificial neural networks and the visual cortex, but such findings are indirect and provide limited insight into the structure of the neural population coding. Decoding becoddng-based methods can recover semantic features from neural activity, yet they do not reveal how these features are organized. This leaves an open question: how feature- specific visual information distributed across neural populations, and whether it forms struclured, semuntically meaningful subxpaces. To address this, we in- troduce MIG-Vis, method that leverages diffusion models to visualize and vali- date
attributes encoded in neural latent subspaces. MIG-Vis first learns group-wise disentangled neural latent representation using a variational autoencoder. It then uses mutual information (MI)-guided diffusion synthesis visualize the visual-semantic features encoded each latent group. We validate MIG-Vis on multi-session neural spiking datasets from the inferior temporal (IT) cortex of two macaques. The synthesized results show that MIG-Vis identifies neural latent groups with clear semantic selectivity diverse visual features, in- cluding object pose, inter-category variation, and intra-category content. These findings provide direct and interpretable evidence of structured semantic repre- sentation in higher visual cortex. The code repository of MIG-Vis is available at: com /B//BBRAIII /TT////AAAMMMGGTMIG----A
INTRODUCTION
Determining how populations of neurons in higher visual areas represent object-centred visual in- formation remains a central question in computational neuroscience (DiCarlo al., 2012 Yamins & DiCarlo 2016). A major line of research (Lindsey & Issa 2024) Xie et al. 2024 has explored representational alignment between deep neural networks (DNNs) and primate visual cortex, show- ing that DNNs trained for object acognition-esseciin
those with disentangled representation subspaces-elosely resemble inferior temporal (IT) cortex activity. Nevertheless, these findings are indirect, relying on artificial model architecture and specially designed representational similarity metrics. Meanwhile, previous works have focused on single-unit selectivity (Rust & DiCarlo 2010)
or decoding- based methods (Freiwald & Tsao 2010: Chang & Tsao 2017) that quantify semantic features like object category or viewpoint higher visual cortex. Earlier studies have also used pre-trained diffusion models with fMRI data (Luo et al., 2023a by Cerdas et al. 2024) to verify and
classify that various brain regions specialize in processing certain categories' information.
However, existing approaches do not extract semantically interpretable neural representations from sioloialy recordings in higher visual cortex. No study has mapped the organization and structural patterns of higher visual area neural populations to distinct visual attributes. In practice, addressing this scientific problem is challenging due to single-unit neural activities in the higher visual cortex exhibit mixed selectivity to multiple visual-semane We also verify this finding through an empirical study on the IT cortex of macaques in a passive object recognition task (Majaj et al. 2015) (the results are presented in Fig. 1.
features (Chang & Tsao 2017).

---

Published as a conference paper at ICLR 2026
(A)
(B)
Rotation_ns
Macaque
Macaque
Nouron #13
Rotation
Rotation
Calegory ID
Neuron #61
Category
Neuron #10
Catagory
Object_ ID
Object
Object
Figure 1: (A) Single-neuron decoding results show that IT neurons exhibit mixed selectivity, con- tributing to both low-level pose attributes (e.g., rotation) and high-level semantic features (e.g., cat- egory identity) different extents (indicated by line thickness). (B) Example images varying along each visaal-semantic feature.
To address this gap, we propose MIG-Vis (Mutual Information- Guidedd Diffusion for uncovering semantic selectivity of neural latent groups in higher Visual cortex), method that identifies inter- pretable neural latent subspaces and visualizes the semantic meaning encoded in each subspace. We assume that multiple latent dimensions form group that encodes specific type of semantic feature. For example, object category may be represented by one latent group, while rotation represented by another. To capture this structure, we use group-wise disentangled variational autoencoder (Li let al. 2025), which learns latent groups composed of multiple dimensions, each corresponding to distinct semantic feature type. Within each group, individual dimensions can still capture different aspects of that feature type, such as shape, texture, or lighting within a category category-related reeltted group.
Given the learned neural latent space, our next goal is understand the visual-semantic features that a certain latent group encodes. We achieve this by perturbing the latent and generating cor- responding images, then comparing them to the original image observe the semantic changes introduced by the perturbation. Traditionally. this can be done using neural-to-ing map perturbed neural latent to an image [Whiteway et al. 2021). However, such a decoder tends to produce single best reconstruction and may smooth out subtle variations in the latent space. As result, small perturbations may not lead to clearly distinguishable semantic changes. An alternative is to use diffusion with guidance. Prior work typically guides diffusion by optimizing simple sta- tistical moments of target neural dimensions, such as absolute activation values (Luo et al. 2023a) variance (Wang et al. 2024). Such approaches where neural activity is directly manipulated: zero indicates no activation, and larger positive values correspond to stronger responses. In our case, the neural representation lies in learned latent space where both positive and negative values carry meaningful but distinct semantics. Thus, maximizing magnitude or variance does not result in meaningful semantic change encoded in neural latent space.
decoder to
especially common in the fMRI literature,
To address limitations in existing methods, we perturb the latent representation by adding sub- tracting values along specific dimensions and seek to generate images that reflect these perturbed latents. Rather than relying a decoder or diffusion guided by magnitude or variance maximiza- tion, we steer the diffusion process by maximizing the mutual information (MI) (Hjelm et al. 2018) between the synthesized image and the perturbed latent. MI captures the full statistical dependence between them, encouraging the generated image to retain the semantic changes introduced by the perturbation. Compared to direct neural-to-imane of collapsing semantic variations into an averaged reconstruction and is better suited to reveal how visual meaning changes along different latent directions.
decoding, MI-guided diffusion reduces the risk
We evaluate MIG-Vi. Vis on multi-session neural spiking datasets (Majaj et al. 2015) from the IT cortex of two macaques performing passive object recognition task. Diffusion- based synthesis shows that MIG-Vi Vis learns disentangled neural latent groups with distinct visual-semantic selectivity. By generating images guided by maximizing mutual information between the output and each latent group, we identify clear semantic roles across groups, including pose, inter-category variation, and intra-category content details. We further apply this interpretation method within each group to reveal fine-grained semantic structure.
based
2 PRELIMINARIES
Problem Formulation. For each single trial, data is composed of neural population and stimulus image pairs: y). The neural population is denoted as x E RN, where is the number of recorded neural units. The corresponding image is represented as y E BIxHxW where H and W
2

---

Published as a conference paper at ICLR 2026
(A)
(B)
Latent Dim-3
Deterministic DDIM Inversion
MI Guided Ve
DDIM Sampling
Latent
yo
Latent Dim-1
Figure 2: (A) The semantic image editing procedure. This consists of deterministic forward process from & = to an intermediate timestep 1', followed by a neural-guided deterministic synthesis process back to = O. (B) Perturbation process. We show the neural manifold of the latent group 71 which encodes object pose. We traverse along its first group. Starting from the original image, slightly adjust the latent variable to obtain perturbed latents and generate the cor- responding images by maximizing mutual information with these perturbed representations. Since the perturbations are applied within the pose subspace, the resulting image changes consistently flect rotation variations.
denote the height and width of the gray-scale image. We develop a group-wise disentangled VAE based on Li et al. (2025) to infer the neural latent groups, denoted as = (......, 2G] RRD. in which D is the latent dimension number and G is the number of groups. Each latent group is set to have the same dimensionality dy. thus D G x dy. Our ultimate goal is to investigate the specific visual-semantic encoding of the g-th neural latent group.
Mlasiffer-Guider Diffusion Models. Given a set image data samples y. diffusion probabilis-
tic model (Ho et al. 2020 Wang et al. 2023 estimates its density Paata (y) by first perturbing the data points in step forward process: y (y: yo) N (yr; Vowyo, (1 o4) I), where {ar !?_ denotes a noise-schedule. In the reverse process, a neural network Eg (yr, t) parameterized by trained estimate the introduced noise at each timestep t. On the other hand, classifier- guided diffusion models (Dhariwal & Nichol 2021) enable conditional generation p(y z) given class label z. Formally, given the class label z and a guidance scale rj > 0, by Bayes' rule, the conditional distribution of sampled images would be: py z) x p(y)p(z y)". By taking the log-derivative on both sides, this gives:
Vy, logp" (ye z) = Vy, log (y: +1 Vy. log lo (z yy)
est. 6s (y).t)
by guidance
The unconditional score on the right-hand side (RHS) is scaled form of the predicted noise, i... Vy, logp(y;) -6a (y,, f)/v/ ar. We propose a novel MI-based classifier parameterized by & to estimate p (z yt). and the second term on the RHS corresponds to its derivvaave
METHODOLOGY
In the following, we first describe how MIG- Vis infers a neural latent space via group-wiin wise disen- tangled VAE. We then present our approach for visualizing and interpreting tions encoded by each latent group through mutual information-guided diffusion synthesis.
varia-
3.1 INFERRING GROUP- CRSUP-WISE DISENTANGLED NEURAL LATENT SUBSPACE
To uncover interpretable visual-semantic structure in neural populations, we use group-wise dis- entangled VAE that learns latent subspaces corresponding to different semantic factors. Traditional disentangled VAEs (Higgins et al., 2017] Chen et al.) 2018) assume that each semantic factor is represented by single independent latent dimension. However, this assumption is restrictive for high-level visual attributes, such as object category or 3D rotation, which typically require multiple latent dimensions to be represented adequately.
3

---

Published as a conference paper at ICLR 2026
Hence, relax the single-dimension assumption by using group-wise disentangled VAE (Es- maeili et al. 2019; Li ct al. 2025), which encourages statistical independence between multi- dimensional neural latent groups. As suggested by previous works (Locatello et al. 2019), well- disentangled semantic latents seemingly cannot be inferred without (implicit) supervision. We in- corporate certain image attributes (i.e., rotation angles and category identity) as supervision form the latent subspace learning. The supervision labels are concatenated into a vector u E RM, where M denotes its dimensionality. We decompose the neural latent vector into supervised latent
in-
groups 2(+) and unsupervised latent groups 2("), such that = [z(*) 2x"]] The latent groups within ( are informed by labels, while the groups in z are inferred without supervision. We propose to optimize the following lower bound of evidence Px( u):
log Pg(X, u) 2 Ro (x x) [log Pe (x z)] + E (xlx)) pe 2(0)
Neural Reconstraction
Weak Label Supervision
(2)
DKL (g x, u) p(z))
B Del
Lo (z)
II To (zn)
Prior Regularization
Partial Correlation
where the probabilistic encoder (-) and decoder pe(-) are parameterized by ab and &, respectively. Ies (z) _ Qu (z x())) q (x(*)) is the aggregated posterior over sample
of size N, g E f1,2..... G} denotes the latent group index, and hyperparameter / controls the penalty scale. We note that the group-wise factorized density in the partial correlation (PC) term is intractable, thus use the importance sampling (IS) estimator (Li et al., 2023 2025) approximate the PC during training. Importantly, the use of weak-supervision labels and the PC penalty here does not degrade the neural reconstruction quality, as verified by the results in Section [
3.2 MUTUAL INFORMATION MAXIMIZATION GUIDANCE
Given an inferred neural latent z, the next goal is to characterize the semanicc factors encoded by specific latent group zy. By perturbing the neural latent zg. observe the resulting changes in the generated images, which helps reveal the underlying semantic variations. Traditionally, this is done by learning neural-to-image decoder that directly maps the perturbed latent However, such decoder often captures only the most dominant patterns and may miss subtle but meaningful variations in the latent. Instead, we use diffusion to generate an image that maximizes its mutual information with the perturbed latent. This encourages the synthesized image to preserve all relevant information encoded in the latent, rather than averaging into a single reconstruction. As result, the generated image better reflects the semantic changes introduced by the perturbation. In this sense, maximizing mutual information provides a more informative guidance signal than directly mapping from z to y using standard decoder.
image.
Guidance. We start with an original image y and its corre- sponding neural latent group Zy. To visualize the semantic features encoded in Zy. we perturb it
Mutual Extormation-Maxiiiaaattio
as 2 2, + 71:7 u(-a,a) a). where is predefined half-range and 1 E R& is an all-ones vector matching the dimension of the g-th latent group. This perturbation moves 2g in both positive and negative directions. We then add noise to the image y and use the diffusion model to gradually denoise it, while guiding the process so that the generated image stays closely related to zy. Because by is slightly perturbed from the original latent Zy. the new image is also modified version of y. The difference between the two images shows the semantic change caused by moving in the neural latent space. Concretely, we use classifier-guided diffusion (Eq. []. since its explicit conditional gradient term makes scientific interpretation more accessible. We aim to approximate the two terms on the RHS of Eq. The unconditional score term is estimated using denoiser eø (Ye, t). We then introduce an MI-based guidance to construct the classifier, from which we derive the conditional score Vy. log p (z y.). Formally, the MI between Z, and y is given by:
MI (Zg.y) y) sp)))))) log P (Mg. y) p(zy)p (y) Ept,, log (y (22) p p(y)
Note that mutual information captures the full statistical dependence between 2, and the image y. During diffusion sampling, given the perturbed ky. maximizing mutual information guides the
4

---

Published as a conference paper at ICLR 2026
process that the synthesized image y reflects the semantic information encoded in zg. Based on Eq. I the classifier-guidee guafierr--giedd conditional score becomes:
y. logp" (y: Zgg Vy, log (y.) + vyV y. Ye).
Estimation of the Group Mutual Information. However, the mutual information term in Eq. intractable and notoriously difficult to compute in practice (Hjelm et al. 2018). According to InfoNCE (Oord et al. 2018), we approximate the density ratio portion p(y in Eq Businng a neural p(x)
network So (. To construct the InfoNCE loss, we need get positive and negative samples for 2g. For an image y, a positive sample 2g comes from ds (Zig
x) where x is the corresponding
neural signal for y, while a negative sample comes from qá (Zg x) with & unrelated to y. During
training. for each y and batch size B, we collect 2g =zz- ..... Zg (B)), where zy is the positive
(1)
sample and zip (i E B)) are negative samples. The noise- contrastive loss LN is optimized follows:
(1) Zg .y)
exp
s6
CN (0) -plla
log Σ
(5)
EZ exP
Note that -xx provides a lower bound (Oord al. 2018 on the group-wise mutual information, i.e. MI(Zg: 2 (B) - LN. S6 (Zg,y) y) specifically approximates the density ratio component
of the mutual information MZ)zg.)) for the g-th latent group, Since the RHS in Eq. provides normalized probability function, we have:
Vy, log Po (Zg yr) =- -v, (Ø) R MI (2g.
(6)
During MI-guided image synthesis, at each step f, we first estimate voise-free approximation by Yo = -y: VI a₁ co(y,, t)) / Var, which is then used as input to the s& (-) network. Fig. 2
shows the perturbation of the neural latent and the corresponding images generated under MI guid- ance. MI steers image synthesis within the first latent group with the guidance strength controlled by the scale parameter 7,
3.3 SEMANTIC IMAGE EDITING WITH DETERMINISTIC DDIM
In diffusion models, early-step noise perturbations primarily distort semantic attributes (e.g., object and category identity) of the clean image, while largely preserving its structural information (e.g.. layout, contours, and color composition) (Choi et al. 2022 Wu et al.) 2023). As our goal is to uncover the conceptual semantic features encoded within specific neural latent groups, we employ the image editing approach (Meng et al. [2221) This method stops the noise-perturbation process an intermediate timestep = (0, 7') and then initiates the backward synthesis process from that step. This approach is employed to retain the basic structure of the reference image. Otherwise, these foundational components will also be generated from pure noise, making it difficult interpret the effect of neural semantic features built upon them. Furthermore, to ensure that our interpretation is not compromised by the stochastic sampled noise from the standard reverse process, we use the deterministic DDIM sampler.
(A)
Formally, we employ a two-stage deterministic image synthesis process. First, we take original image Yo and apply determinis-
tie DDIM Inversion (Song et al. 2020 Mokady et al. 2023) using the diffusion denoiser ca(-) from
up to predefined timestep
t = t'. This inversion timestep is calibrated to corrupt the semantic attributes the original image while preserving its structural infor- mation. In the second stage, we reverse the process from & = t' using a classifier-guies guided,, deterministic DDIM sampling. This reverse procedure is guided by our proposed MI maximization objective, ultimately yielding the synthesized image. This overall process is illustrated in Fig. A). The complete synthesis proce- dure and a detailed algorithm are provided in Appendix
(B)
Chair
Ship
Animal
Table
Face
Plane
Frait
Figure 3: Macaques IT cor- tex dataset. (A) Experimen- tal Setting. (B) Example im- ages from the eight object cat- egories in the dataset.
4 EXPERIMENTS
Macaque IT Cortex Dataset. We use single-unit spiking responses from the IT cortex of two macaques (M1 and M2) during a passive
5

---

Published as a conference paper at ICLR 2026
Original Image
Posttive Traverool
Positive Troversal
Traversal
Truveral
Image
y=
V+
Y+
Group-2: Inter-Catagory
Group-2: Inter-Catagory
Group-3: Intra-Category
Group-3: Intra-Category
Group-4: Intra- Categryy
Group-4: Intra- Intra-Category
Negative Latent Traversal
Original Image
Powttive Latent Troversal
Origial Image
Travensa!
Traversol
Y-
y=0
Group-1: Pose
Group-1: Pose
Group-2: Inter-Category
Group-2: Inter-Category
Group-3: Category
Group-3: Matra-Category
Intra-Category
Group-d: Entra-Category
Figure 4: Synthesized images by MIG-VE Vis under varying guidance strengths. Results using frontal-view face, car, strawberry, and table the original images.
object recognition task (Majaj et al. 2015). Each fixed macaque passively viewed grayscale naturalistic object images while clectrophysiological activity was recorded from 58 IT channels in M1 and 110 in M2. Each image shown for 100 the center of gaze, and neural responses were measured within 70-170 ms post-stimulus window. The dataset contains 5760 images span- ning eight basic-level categories (Fig- 3). To introduce Keetityypreserving variation, images generated via ray-tracing with randomized object position, scale, and pose. Following prior work (Lindsey & Issa
2024), segment the foreground in each image reduce background effects.
Model Settings. (1) Neural VAE. We adopt two-layer MI.P for both the encoder and the decoder. We set the neural latent dimension number to 24 and partition it into with a group rank of D2 6. The first two groups are supervised: Group 1 is informed by the 3D rotation angles provided in the dataset, and Group uses the 8-way one-hot category iid labels. The remaining groups, Group 3 and Group 4, are learned in an unsupervised manner. (2) Neural Encoder. We first extract the DINO embeddings (Caron al. 2021 from the raw images. adopt the ViT-B/1 6 architecture and its image embedding size is 384. The density ratio estimator so (Zy, y) is implemented as three-layer convolutional neural network. (3) Diffusion model. We downsample images to resolution of 128 128 and trained an image diffusion model based U-Net architecture (Ronneberger et al. 2015). The diffusion step set as 150 and the set as 135. Further details on the model architecture and hyper-paramen parametess are listed in the Appendix
4 latent groups, each
DINO, we

---

Published as a conference paper at ICLR 2026
A
Negative Latent Traversal
Stmmaus
Poetive Latent Traversal
Negative Latent Traversal
Original Shimllu Image
Postivve Latent Traversal
Y=
y=0
SLT
SLT
AP-CFG
AP-CFG
W/o MI Guidance
Wie Mi Guidance
MIG-Vis (Ours)
MIG-Vis (Ours)
Comparison of images synthesized by MIG-Vis and baseline methods. (A) Results Figure 5:
from probing Latent Group (pose). (B) Results from probing Latent Group 2 (inter-category).
4.1 DIFFUSION VISUALIZATION OF SEMANTIC SELECTIVITY IN LATENT GROUPS
To perform diffusion-based visualization with MIG-Vi we select two key components: an original image Yo. which serves the starting point for the editing process described in Section 3.3 and a neural latent group Zy. which we perturb for MI-guided synthesis. We generate images using MIG-Vis under different guidance strengths, 7 E {-10, 5, 5, 10). In Fig. 4 frontal- -iew face, car, a strawberry, and a table are used as the original images. For each case, we perturb the corre- sponding neural latent with both positive and negative 7. The resulting visual-semantic selectivity is summarized as follows:
(1) Latent Group Pose. Probing Latent Group primarily modulates pose-related features, such as the rotation of faces and cars. This aligns with the rotation supervision applied to this group's dimensions. Importantly, object category remains unchanged across all synthesized images, indicating that this latent group separates pose variation from other semantic content.
(2) Latent Group 2: Inter-Category Semantic Attributes. Notably, despite being supervised
only with high-level category identity and no explicit semantic features, Latent Group 2 learns to control inter-category semantic attributes. For example, our MI-guidance transforms the face image into strawberry. We also observe direct relationship between activation strength and semantic distance; a larger magnitude of 7 results in a generated image that is more semantically distinct from the original, demonstrating that the of this group subspace encode high-level categorical information.
(3) Latent Group 3 and 4: Intra-Category Content Details. Latent Groups 3 and 4, discovered without supervision, both encode intra-category content variations. Group 3 primarily modulates the appearance faces and strawberries with little effect on cars, whereas Group significantly alters cars and tables while minimally affecting faces. This selective behavior likely reflects dis- tinct patterns of visual variation across object categories. These results suggest that the neural latent manifold in these dimensions is locally structured rather than globally aligned. Different object categories occupy different regions of the manifold and vary along different directions. Geometri- cally, this implies that the manifold is anisotropic and curved: the principal directions of variation are category-depon -deeedeed. Faces and strawberries extend strongly along the dimensions Group 3, whereas cars and tables extend more along Group 4. Thus, intra-category variation is organized into separate local tangent directions rather than a single shared global axis across all objects.

---

Published as a conference paper at ICLR 2026
4.2 NEURAL GUIDANCE BASELINE METHODS FOR COMPARISON
As prior works are designed for different types of neural data (c.g., fMRI), here we isolate their core neural guidance approaches during image synthesis for fair comparison with our method: Standard Latent Traversal (SLT): a standard approach for latent variable manipulation (Chen
et al., 2018 Esmaeili et al., 2019), which trains decoder to map neural latents to images. We
perform latent traversal with 2g = 2g + +y1.
Activation Probing via Classifier-Free Guidance (AP-CFG): the neural manipulation method
used in BrainACTIV (Cerdas et al.) (024), which synthesizes images guided to maximize or mini- mize the activation of a target cortical region. It employs classifier-free guidance diffusion (Ho & Salimans| 2022) that jointly models the denoising and guidance gradients.
Ours w/o MI Guidance: We perform latent traversal by shifting the group latent as 2g = 2yg + Instead of mutual information, we train probabilistic encoder that maps an image y to the neural latent zg i.e., learning ly), Given b.y. we generate a new image y via diffusion, guided by Vy log p(zz)(l ly), which encourages the generated image to produce a latent with highest likelihood matching zn.
Comparison Results Analyses. In Fig. [ we compare MIG-Vis with the three baselines intro- duced above, using a frontal-view face as the original image. We focus on two representative latent groups for this analysis: Group 1 and Group 2, which have been identified controlling pose and inter-category factors, respectively. We first observe that images manipulated by SLT exhibit some rotation when probing Group 1, but the changes are not as clean those produced by other meth- ods. When probing Group 2, the object category remains unchanged. This suggests a limitation of decoder-based image generation compared diffusion-based generation. AP-CFG performs rea- sonably well in capturing rotation semantics; however, its perturbations for inter-category variations are less clean compared to our Mis.
Importantly. in the ablation of our framework w/o MI guidance, where diffusion is guided by Rikelihood-based alignment Vy log p(z,, ly),, we observe that pose-related rotation semantics can be captured moderately well. Both ours with and without MI rely on linear activation probing to define the latent subspace. However, the key difference lies in the guidance objective. Likelihood- guidance only enforces that the synthesized image be mapped back to the target latent 2, by the learned probabilistic encoder. In other words, diffusion merely needs to generate an image that the encoder recognizes as having latent z. This constraint is one-sided and encoder-dependent as the encoder assigns high likelihood to Zg. the objective is satisfied. For simple, low-dimensional variations such rotation, this may be sufficient. However, for inter-category semantic features, whose latent structure is complex and nonlinear, this objective becomes too weak. Multiple visually different images may map similar latent values under the encoder, and subtle semantic structure may be averaged out. As a result, likelihood-basse guidance can produce inconsistent or unrealistic categorical transitions.
long
In contrast, our MIG-Vis employs MI guidance, which imposes stronger constraint. Rather than asking whether the encoder can recognize the image as having latent zg. MI maximization requires that the synthesized image and Zg share maximal statistical dependence. Intuitively, likelihood guidance asks: Can the encoder recognize this image as having latent zy? MI guidance asks: Does this image truly express the information contained in zg? ? The latter is a stricter requirement. It forces the diffusion process to generate images that visibly and structurally reflect the semantic content encoded in the perturbed latent group, rather than merely satisfying an encoder consistency check. Consequently, MI guidance better preserves complex inter-category structure and produces smooth, realistic transitions across object instances.
4.3 FINE-GRAINED NEURAL SELECTIVITY WITHIN LATENT GROUP
In Figs. and A),, we prohe pair-wise latent dimensions within the Group 3 intra-category sub- space by starting from both a face image and strawberry image, and within the Group 2 inter- category subspace by starting from a table image. We observe that perturbing the same dimension pairs results in different semantic changes across object categories.
In Fig. 6. for dimension pair [1,2], perturbations applied to the face primarily modulate orientation and gaze direction (e.g., upward gaze left orientation shift). In contrast, perturbing the same di- mensions for the strawberry mainly produces texture smoothing and global lighting changes. Sim- ilarly, within the [3, 5] subspace, faces exhibit structural variations such right-side gaze shifts,
8

---

Published as a conference paper at ICLR 2026
(A)
(B)
Negetive Traversal
Original Stimulus Image
Positive Traversal
Negalive Traversal
Original Stimulus Image
Positive Traversal
y=0
Y+
+++
Y--
y=
Y-
Y++
Dim[3,2]
Dim]]1.]]
left orientation shift
upward gaze
lighting
texture smoothing
structural collapse
right-side gaze
intra-cate change
darken surface
Dim[4,6]
Dim[4.8]
dowmward viewpoint
facial deformation
flatten shape
stripe variations
Figure Pair-wise visual-semantic probing in latent Group 3. (A) MIG-Vis synthesis from
frontal-view face. (B) MIG-Vi Vis synthesis from strawberry. These results show that different
dimension pairs within the same latent group encode distinct, fine-grained visual-semantic attributes.
Originll Image
Positive
Group-1 Pose
rotation+
rotation-
Face
Car
DimL1.2]
Strawberry
Table
category
shape
Group-3 3 Intra-Category
(C)
rectanglar Dim[3,5]
category
feature
variations
formation
texture smoothing
Face
dim
gaze direction
Dim[4,6]
Strawberry
category shift to plane
shape rounding
Figure 7: (A) Pair-wise visual-semantic feature investigation within latent Group 2. Results of MIG-Vis from probing table object. (B) and (C): Neural manifolds of two latent groups.
whereas strawberries show intra-category changes and surface darkening. For [4, 6]. perturbations deform facial features and alter viewpoint in faces, but induce shape flattening and stripe variations in strawberries.
With these semantic probing visualizations produced by MIG-Vis, we gain insight into the structure of the high-dimensional neural space in the higher visual cortex. Here, we illustrate plausible neural manifolds within example subspaces.
For the latent dimensions associated with Group-1 (pose), increasing 7 produces systematic ro- tational changes across all object categories (Fig. 4.. However, the direction of rotation depends on the object: faces rotate clockwise while cars rotate counterclockwise. A similar pattern is ob- served for strawberries and tables. This indicates that perturbations along these latent dimensions consistently encode rotation semantic, regardless of object identity. In other words, manipulating same latent axis always induces rotation, even though the visual manifestation of that rotation may differ across categories. This observation suggests that the underlying geometry of the latent space in these dimensions is structured rather than linear. A natural hypothesis is that the manifold along these dimensions resembles torus (Fig, BB). In this view, different object categories occupy different locations on the toroidal manifold: cars and tables lie closer on one side, while faces and strawberries lie closer the opposite side. When latent perturbation is applied (vertical arrows

---

Published as a conference paper at ICLR 2026
in Fig. B), its projection onto the toroidal manifold induces motion along the surface in opposite directions for different object regions. This results in clockwise rotation for some objects and coun- terclockwise rotation for others. Crucially, despite these differences in local behavior, the semantic meaning of movement along this latent axis remains consistent across categories: it corresponds to object rotation. We therefore interpret these dimensions forming globally consistent semantic manifold in neural latent space.
In contrast, the latent dimensions associated with Group-3 (intra-category) exhibit fundamen- tally different behavior. For example, when perturbing a face along dimension 1 versus dimension 2 with changing 7, we observe distinct semantic changes such as gaze direction and feature forma- tion. However, applying the same perturbations to different object, such as a strawberry, produces entirely different semantic variations, such as changes in texture smoothing or stripe patterns. This stands in sharp contrast to the Group-1 pose dimensions, where latent perturbations consistently cor- respond to the same semantic transformation across objects. Here, the same latent directions do not induce a shared semantic effect when applied to different object categories. This suggests that the underlying manifold in this subspace is substantially more complex and nonlinear (e.g., Fig. 7 Rather than forming a globally structured surface like the toroidal rotation manifold in Group- the geometry here appears warped and irregular. Perturbations along different latent axes project onto the manifold in different ways: some may induce motion along a locally structured trajectory (e.g., along an "8-shaped" curve), while others act in directions that are effectively orthogonal to it. As a result, the semantic interpretation of these dimensions becomes object-dependent. Unlike the globally consistent rotation semantics observed in Group-1, no shared transformation emerges across categories. Instead, semantic meaning can only be interpreted locally, relative to each ob- ject's position on the manifold. Taken together, this indicates that while some latent subspaces encode globally consistent semantics (e.g., pose), others correspond to highly warped intra- categry manifolds whose effects are inherently local rather than universal.
Thus, MIG- Vis serves as an intuitive tool for visualizing neural manifolds and generating hypothe- ses, while also pointing toward future neuroscience research on formally characterizing the geometry of neural subspaces in higher visual cortex.
4.4 NEURAL RECONSTRUCTION EVALUATION
Table 1: Neural reconstruction quality compar- isons on IT cortex dataset. We report the mean and standard deviation of explained variance (R2 in %) over five runs.
To verify that our neural VAE module maintains high-quality neural reconstruction with the par- tial correlation regularization and weak label su- pervision, we quantitatively evaluated its perfor- mance. Table I presents the R-squared (R²2 in %) values for both macaques, comparing our mod- ule to standard unsupervised VAE. These results demonstrate that the neural reconstruction quality is well-preserved, incurring only a marginal per- formance drop compared the standard VAE. We hypothesize that the weakly-supervisee labels in- duce a rotation of the neural latent subspace while preserving the information necessary for recon- struction. From the R² results of ablation studies on our VAE module's 's two introduced terms, we observe that each has only a minimal negative im- pact on neural reconstruction. We also plot the raw neural signals and their reconstruction across dimensions for four selected trials in Fig. 8 in the Appendix A.4
Subject
Method
R²(%) f
Standard VAE
78.62
(+0058)
Ours w/o Sup.
76.90
(+0.53)
M1
Ours w/o PC
77.30
(0066
Ours
76.58
(=0664)
Standard VAE
83.72
(+0.47)
Ours w/o Sup.
82.39
(+0.59)
M2
Ours w/o PC.
82.21
(=0.55)
Ours
81,86
(=0.51)
CONCLUSION
Our contributions can be summarized into the following points: (1) Leveraging advanced neural latent variable models, this is the first work to explore neural representations with semantic selec- tivity in the higher visual cortex from electrophysiologcaal data. (2) To interpret the visual-semantic features encoded within specific neural latent group, we use a DDIM-based deterministic image editing approach with a proposed mutual information maximization objective. (3) The edited stim- uli faithfully demonstrate distinct and high-level visual features, verifying the semantic selectivity within these inferred neural latent groups. Our work offers a critical step toward understanding the compositional, multi-dimensioon nature of visual coding in the primate visual cortex.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the seed grant from Microsoft GenAl.
REFERENCES
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-965000 2021.
Diego Garefa Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig, Paseal Mettes, and Iris Groen. Brainactiv: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation. bioRxiv, pp. 2024-10, 2024.
Le Chang and Doris Y Tsao. The code for facial identity in the primate brain. Cell, 1613))10013 1028, 2017.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11472-11481, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
James J DiCarlo. Davide Zoccolan, and Nicole Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-434, 2012.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and San-Willem Meent. Structured disentangled representations. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2525-2534. PMLR, 2019.
Winrich A Freiwald and Doris Y Tsao. Functional compartmenntllizatio and viewpoint generaliza- tion within the macaque face-processing system. Science, 330(6005):845-8555 8511, 2010.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Devon Hjelm, Alex Fedorov, Samuel Marohi-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint a 2018.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207.12500 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6901 2014.
Chengrui Li, Yule Wang. Weihan Li, and Anqi Wu. Forward x² divergence based variational im- portance sampling. arXiv preprint arXiv:2311.02516 2023.
Chengrui Li, Yunmiao Wang, Yule Wang, Weihan Li, Dieter Jaeger, and Angi Wu. A revisit of total correlation in disentangled variational auto-encoder with partial disentanglement. arXiv preprint MxXiv:2502.02279 2025.
11

---

Published as a conference paper at ICLR 2026
Jack W Lindsey and Elias B Issa. Factorized visual representations in the primate visual system and deep neural networks. Elife, 13:RP91685, 2024.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 41 4114-4124. PMLR, 2019.
Andrew Luo, Maggie Henderson, Leila Wehbe, and Michael Tarr. Brain diffusion for visual explo- ration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36:75740-75781, 2023a.
Andrew F Luo, Margaret M Henderson, Michael J Tarr, and Leila Wehbe. Brainscuba: Fine-grained natural language captions of visual cortex selectivity. arXiv preprint arXiv:2310.04422 2023b.
Najib J Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition perfor- mance. Journal of Neuroscience. (5(39):13402-13411 2015.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint vrXiv:2/08.01073, 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6038-6047. 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037// 2018
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-ussisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241. Springer, 2015.
Nicole C Rust and James J DiCarlo. Selectivity and tolerance ("invariance") both increase visual information propagates from cortical area v4 to it. Journal Neurociinne, 30(39): 21-222999991....1111 2010.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wrXiv:2010.02502. 2020.
Yule Wang, Zijing Wu, Chengrui Li, and Anqi Wu. Extraction and recovery of spatio-temporal structure in latent dynamies alignment with diffusion models. Advances in Neural Information Processing Systems, 36:38988-330 -9088,,900500 2023.
Yule Wang, Chengrui Li, Weihan Li, and Anqi Wu. Exploring behavior-relevant and disentangled neural dynamics with generative diffusion models. Advances in Neural Information Processing Systems, 37:34712-34736, 2024.
Matthew R Whiteway, Dan Biderman, Yoni Friedman, Mario Dipoppa, E Kelly Buchanan, Anqi Wu, John Zhou, Niccolò Bonacchi, Nathaniel J Miska, Jean-Paul Noel, et al. Partitioning variability in animal behavioral videos using semi sumi-supervised variational autoencoders. PLoS computational biology, 17(9)e 1009439, 2021.
Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
1900-1910, 2023.
Yudi Xie, Weichen Huang, Esther Alter, Jeremy Schwartz, Joshua B Tenenbaum, and James Di- Carlo. Vision enns trained to estimate spatial latents learned similar ventral-stream--ee alligeed rep- resentations. arXiv preprint arXiv:2412.0 12.22222299111 15, 2024.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356-365 2016.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A DETAILS OF THE MIG-VIS METHOD
A.1 MODEL. TRAINING DETAILS
For the partially disentangled neural VAE, we employ an architecture consisting of a two-layer MLP for both the probabilistic encoder and the probabilistic decoder networks. Each MLP uses ReLU activations. For the neural latent dimension number & to both the two macaques, we set k = 24 and partition it into four distinct latent groups, each of dimension 6 (i.e., group rank equals to 6). This group rank setting provides the expressiveness to model complex visual semantic factors (e.g.. inter-category variations) within unique latent group. For latent group learning with the 3D rotation angles of the object provided in the dataset; for latent group 2, we use the 8-way category identity labels (e.g.. "Animals" as integer label 0, "Boats" as integer label 1); latent groups 3 and 4 are learned in fully unsupervised manner. We use a learning rate of x 10-3 and set partial correlation penalty pc_we i ght to 5 x 10-5 5 on both two macaques. We use the Adam Optimizer| Kingma & Ba (2014) for optimization.
weakly supervise its
For the image diffusion model, we downsample the grayscale images to a resolution 1 x 128 x 128 and train the diffusion model on the resulting inputs. We augment the dataset using random horizon- tal flips to improve model generalization. We adopt the architecture of the image diffusion denoiser the U-Net (Ronneberger et al. 2015). We adopt the e-parameterization in diffusion model for improved training stability, given the relatively small dataset size and diffusion timestep settings. The diffusion timestep is set as 150. The embedding input dimension to the U-Net architecture is set as 64 and the U-Net has three down-sampling and up- up-sampling layers. The training batch size is as 64. We train the U-Net denoiser model on 40, 000 iterations with learning rate of 2 x 10 All experiments are conducted using PyTorch on a compute cluster equipped with NVIDIA A40 GPUs. The training phase of the image diffusion model takes approximately 20 hours. We also ap- ply exponential moving average (EMA) with decay rate of 0.98 to stabilize training and improve generalization.
A.2 DERIVATION OF THE NOISE-FREE PREDICTION ýo
We here elaborate the approximation of the clean image sample ÿo. as used in Section 3.2 It is computed from the perturbed data point y, and the prediction of the denoiser model e (y(y,,
As in the forward process of diffusion models, given clean stimulus image yo, it generates a noisy sample y. by gradually adding Gaussian noise through the following closed-form equation:
y1 Von, Yo v1 Cr 6, 6~ (0, I).
The above re-parameterized forward process represents y, a linear combination of the clean image Yo and the Gaussian noise e. By rearranging the above equation, we obtain an exact expression for the original image sample yo:
yo = 1 (y- (x
(8)
Va:
As the true noise is unknown during the sampling phase, it approximated by the learned denoiser neural network ea (y t). Inserting this estimate into the expression above provides an approxima- tion of the clean image yo:
8o Y: - (V1 as) co ty.,
Vau
this formula is used consistently across both DDPM and DDIM frameworks to obtain noise-free approximation the original image from the noisy sample at each timestep.
A.3 GROUP-WISE DISENTANGLED NEURAL LATENT SUBSPACE INVESTIGATION
To ensure that the neural VAE module within the MIG- Vis framework maintains high-quality neural reconstruction along with the partial correlation disentanglement regularization and weakly super- vised label guidance, we present the quantitative neural reconstruction results of both two macaques
13

---

Published as a conference paper at ICLR 2026
in Table 2 which records the R-squared values (R², in %) and RMSE of our module and the standard unsupervised VAE. These results indicate that the neural reconstruction quality is well well-prrserved, with only slight performance drop compared to the standard VAE.
A possible explanation is that the weakly- way that preserves most of the information necessary for reconstructing the neural activity. This observation is reasonable, given that these labels can be accurately decoded from the neural activity itself. Furthermore, we evaluate the disentanglement quality of the inferred latent subspace using the widely adopted Mutual Information Gap (MIG) metric (Chen et al.] 2018), also reported in Table 2 The results demonstrate that the neural latents learned by MIG- Vis are significantly more disentangled than those of the standard VAE.
labels rotate the neural latent subspace in a
Table 2: Performance report of our neural VAE module and standard VAE IT cortex neural activity from macaque MI and M2. We report explained variance (lt²), root mean squared error (RMSE), and mutual information gap (MIG) assess reconstruction accuracy and latent disentan- glement. Results are averaged over 5 runs; bold numbers indicate the highest MIG score on each subject.
MI
M2
Metries \ Method
Standard VAE
Ours
Standard VAE
Ours
(²)% t RMSE
78.62 (+0.58)
76.58
(+0.64)
83.72 (+0.447)
81.86 (+0.51)
48.49 (+0339)
50.47
(+0339)
40.77 (+0.28)
43.44 (=033)
MIG(%) 1
33.27 (=0)82)
44.23 (=0661
28.65 (+0.71)
49.85 (=0.55)
Table 3: Explained variance of neural signals from the supervised and unsupervised neural latents of the two macaques MI and M2.
Subject
Metries
Full Latents
Supervised Latents
Unsupervised Latents
R2 (%) f RMSE
76.13
(+0.27)
32.50
(+0.19)
20.88 (=:0.25)
MI
48.86 (+0211)
80.75 (0.))
88.95 (+.29
R² (%) f RMSE
71.82
(+0.28) 53.09 (+0.16)
28.11
(+0.22)
19.63 (++0.17)
M2
84.79 (+0)32)
89.65 (+0.31)
A.4 NEURAL SIGNAL RECONSTRUCTION
B DETAILED MUTUAL-INFORMATTON MAXIMIZATION GUIDANCE ALGORITHM
Algorithm Synthesize Stimulus Image for Semantic Latent Group Discovery
Input:
Target Neural Latent Group 2g. Guidance Scale y) Sampling Step E! Output: Synthesized Image Stimulus with Guidance yo
Initiate Original Image Stimulus Yo
for f = to t do
Y: = -V
01 Y2-1 + (1 a--1 1) E (y(
>. DDIM Inversion
ac-1
for t = t' to 1 do
é(y,, 1) = 60 (v 1) +1 Fx (bb
> Mutual Information Guided Gradient
Yt-1 V
a=-1 Yt + (V ac-1
1
E(y;, 1)
D Guided DDIM Sampling
14

---

Published as a conference paper at ICLR 2026
(A)
M1
M2
134: Original Reconstructed
Original vs. Recorsttuucedd
Original
Original
Aetlantion
This #2175: Origina vs Reconstructos
That m872 Original vs Reconstructed
2.0
Driginnl
3.0
Original
Recarstisction
Neuron Dimension
Neuren Dimencion
Figure 8: (A) Raw Neural signals and the reconstructions from the neural LVM module in MIG-Vis on the MI subject. (B) Raw Neural signals and the reconstructions from the neural LVM module in MIG-Vis the M2 subject.
(A)
Negative Traversal
Original
Nogative
Original Stimulus Image
Positive Traversal
Image
Traversal
y=
Intra-Category Group
Catra-Cattgy
G-5 Intra Catun-rrrrrrryy Group-1
G-5 Intra-Categon
6-6 Intra Category Group-2
Ga5 Intra- Category Group-2
Intra-Cetegory
Figure 9: Neural latent group number G analyses. (A) Synthesized images of MIG- from probing a frontal-view human face. (B) Synthesized images of MIG-Vis from probing a car object. (A)
Nogative Labent Traversal
Original Stimulus
Positive Latent Traversal
Negative Latent Traversal
Original Stimulus Image
Positive Latent
Traversal
y=0
Y+
7++
y
Y+
y
B-B8
B+a
Figure 10: InfoNCE negative sample B analyses in Group-1 (Pose), (A) Synthesized images of MIG-Vis from probing frontal-view human face. (B) Synthesized images of MIG-Vis from probing a car object.
15


Reviews:
[{"review_id": "XdiGpsBbmN", "reviewer": "Reviewer_X5v2", "summary": "The paper proposes MIG-Vis, a two-stage pipeline to uncover semantically selective latent groups in macaque IT cortex. First, a group-wise (weakly supervised) VAE learns a structured neural latent space. Then, the authors visualize each group’s semantics by editing real images with a deterministic DDIM sampler guided by a surrogate of mutual information (via InfoNCE), so edits stay on-manifold while moving the image toward the target group. Qualitatively, different groups control pose (intra-class), inter-category transitions, and fine intra-class details; ablations suggest stronger, more controllable edits than activation-guided baselines, while disentanglement improves with little loss in neural reconstruction.", "questions": "1. In equation (4), the conditional variable $z$ in $\\log p_\\gamma(y_t \\mid z)\\,$ is undefined. The authors need replace $z$ by $z_g$ or explicitly define $z$, and ensure consistent notation throughout.\n2. The paper states that negative samples are drawn from $q_\\phi(z_g \\mid \\hat{x})$ with $\\hat{x}$ unrelated to $y$, but how independence from the current $y$ is guaranteed. The authors need to clarify whether negative samples are drawn cross-image or provide the exact procedure.\n3. The quantity in Eq.~(2) is commonly referred to as Total Correlation. Please rename accordingly or explain the difference between your usage and the standard definition.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper combines a group-wise neural latent space (VAE) with explicit, InfoNCE-guided deterministic DDIM editing, yielding a clean ‘unconditional denoising score + semantic term’ decomposition for controllable edits—going beyond regression guidance and offering a clear, reproducible, and extensible recipe.\n2. On macaque IT data, the approach consistently maps different latent groups to pose, inter-category transitions, and fine intra-category details while preserving structure, providing a practical tool for population-level neural interpretation with promising implications for closed-loop neuroscience and interpretable generative modeling.", "weaknesses": "1. In Eqs.~(5)--(6) the authors train an InfoNCE scorer and use $-\\nabla_y L_{\\mathrm{InfoNCE}}$ to approximate $\\nabla_y \\mathrm{PMI}(z_g,y)$. By standard contrastive–learning results, however, the InfoNCE optimum estimates $\\log \\frac{p(z\\mid y)}{q(z)}$; this coincides with PMI only when the negative–sample distribution satisfies $q(z)\\approx p(z)$ and is independent of the current $y$. Therefore, the reliability of the gradient’s direction and magnitude hinges on this assumption. Please (i) specify how negatives are sampled in your implementation and (ii) systematically quantify the impact of different $q(z)$ strategies and batch sizes $B$.\n2. The implementation guides sampling with the pointwise mutual information (PMI) gradient for a specific target group $z_g$, yet the main text repeatedly refers to an \"MI gradient,'' which can be misread as maximizing the expected mutual information over $p(z\\mid y)$. The two have different statistical meanings: PMI means make the current image look more like this group,whereas MI concerns the overall dependence between $Z$ and $Y$. Using \"MI'' instead of \"PMI\" overstates the correspondence between the target and the evidence. Please state explicitly in the main text.\n3. The paper fixes the editing start at an intermediate diffusion step $t'$ and employs deterministic DDIM. However, it provides no evidence for why such a $t'$ should generally exist, how $t'$ ought to be selected, or whether the choice is robust across different noise schedules, datasets. Therefore, the observed effects may depend on carefully chosen settings or hyperparameters, rather than the method itself.\n4. The number of groups $G$ in the group-wise VAE is a key hyperparameter.  If $G$ is too large, semantic factors tend to fragment across multiple groups;  if $G$ is too small, heterogeneous semantics are merged into the same group.  The authors should provide experiments justifying their choice of $G$ together with metrics such as reconstruction and controllability of the resulting visualizations.\n5. While this is a minor issue, it was unclear if the dataset from Majaj et al., 2015 is publicly available but I must specify that MIG-Vis code has been provided on a link. The model is there but I might have missed the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rwJSaWyuWg", "reviewer": "Reviewer_NyBK", "summary": "This paper presents a method to visualize how neural populations in the primate brain encode visual information. The approach first processes neural spiking data from macaques through a group-wise disentangled variational autoencoder to identify distinct, low-dimensional latent subspaces. It then employs a diffusion model guided by an objective that maximizes the mutual information between a synthesized image and a target neural subspace, thereby generating images intended to represent the specific visual features encoded by that group of neurons. The resulting images suggest the higher visual cortex organizes information into specialized groups selective for features like object pose, inter-category transformations, and intra-category content details. However, the work's novelty lies more in its combination of existing machine learning techniques than in a fundamental conceptual breakthrough, updating the classic optimal stimulus paradigm with modern tools. Furthermore, the claim of providing direct evidence is questionable, as the visualizations are interpretations generated by a multi-stage modeling pipeline. A significant limitation that tempers the paper's broader conclusions is its reliance on a highly constrained stimulus set consisting of grayscale, segmented objects from only eight categories. This limited visual diversity means the uncovered semantic selectivity is defined by this artificial environment, and claims about the general encoding principles of the higher visual cortex may be overstated, as the findings are not shown to generalize to the complexity of natural vision.", "questions": "1. The choice of the VAE architecture, particularly the number of latent groups (four) and the dimensionality of each group (six), is important to the findings. Could you elaborate on the selection process for these hyperparameters? How sensitive are the discovered semantic selectivities to changes in this architecture, for example, if you used more or fewer groups? How would you apply this to natural RGB images?\n\n2. The \"latent groups\" are a powerful modeling abstraction. Do you have any hypotheses about how these computationally-defined groups might map onto the known anatomical or functional organization of the IT cortex? For instance, could a single latent group correspond to a specific neural sub-population or a known processing stream?\n\n3. Your results show that manipulating a single latent group can produce large, coherent changes in the image, such as transforming a face into a pear. Does this imply that the neural code for these distinct objects is adjacent or connected in this latent space, or is this transformation an artifact of the generative model's ability to interpolate between any two points?\n\n4. Have you considered applying your method to a more diverse dataset to test the robustness of these specific semantic axes? Would you expect to find similar, cleanly separated latent groups, or would you anticipate a more entangled representation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The work presents several strengths in its approach to investigating neural representations. It successfully integrates multiple machine learning models, a variational autoencoder and a diffusion model, into a cohesive pipeline for interpreting high-dimensional electrophysiology data.\n1. A key contribution is the use of a mutual information-based guidance objective for the diffusion model. The authors support the efficacy of this choice through an ablation study that compares it to a simpler activation-based guidance, showing the MI approach yields more semantically consistent results for complex transformations.\n2. The method produces clear and intuitive visualizations of the abstract information encoded in the identified neural subspaces. This translates complex neural activity patterns into understandable images, making the model's findings accessible.\n3. The study includes a comparative analysis against several baseline methods for neural guidance. This demonstrates that the proposed approach generates higher-quality and more structurally coherent images, strengthening the claims about its effectiveness.", "weaknesses": "The study has several limitations that should be considered when interpreting its conclusions, with the primary concerns relating to the novelty of the approach and the characteristics of the dataset used.\n1. The work's main contribution is the novel integration and application of existing machine learning methods to a specific neuroscience problem. The core components: variational autoencoders for disentanglement, diffusion models for generation, and mutual information for guidance. Are all established techniques. As such, the paper represents an advancement in methodology rather than the introduction of a fundamentally new theoretical concept for understanding neural computation.\n2. The conclusions are based on neural responses to a highly controlled and simplified visual environment. The dataset consists of grayscale, segmented objects from only eight categories presented on a uniform background. This lack of complexity raises significant questions about the generalizability of the findings. The \"semantic\" features discovered by the model are defined entirely by the limited variations present in the dataset and may not reflect how the visual cortex represents the much richer and more complex information found in natural scenes, which include color, texture, and cluttered backgrounds.\n3. The claim of providing \"direct\" evidence of neural representation is not strong. The final visualizations are the output of a multi-stage pipeline involving two separate deep learning models. The results are therefore an interpretation of the neural data as filtered through the specific architectures and biases of these models. Different modeling choices could potentially lead to different visual interpretations of the same underlying neural activity.\n4. The primary support for the semantic meaning of the latent groups comes from the visual inspection of the synthesized images. While the images are interpretable, the evaluation of their semantic content is largely qualitative and subjective. The study lacks quantitative metrics to formally validate that the generated images accurately capture the intended semantic transformations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hf9dr2UxJN", "reviewer": "Reviewer_bCGZ", "summary": "The paper tackles the enduring question of how information is represented in higher-level visual cortex, a challenge compounded by the mixed selectivity of neurons to multiple latent variables. The authors propose MIG-Vis, which combines a group-wise disentangled variational autoencoder (VAE) with a mutual-information-guided diffusion model to visualize and interpret the semantic content encoded by neural latent groups. Using macaque IT recordings and controlled object stimuli, the method aims to reveal how different latent groups in neural activation space correspond to features such as pose, category, or intra-category content.", "questions": "- Can the authors clarify what the supervised latent groups represent—are these indeed pose and category—and how the corresponding labels were generated? How much of the total neural variance do these supervised groups explain, and do they capture any structure beyond what would be expected from the labeled variables? For the unsupervised groups, what features do they encode, how consistent are they across neurons or datasets, and what proportion of variance do they account for?\n\n-  In terms of methodology, what is the motivation for the two-step diffusion process (inversion followed by guided synthesis)? Is this intended to constrain image generation to remain on the natural image manifold, and does the choice of starting image influence the final outcome?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important problem—understanding how visual cortex encodes semantically meaningful subspaces—using a technically interesting and timely approach. The combination of a group-wise disentangled VAE and mutual information-guided diffusion is conceptually elegant.", "weaknesses": "- While the idea is strong, the paper remains largely qualitative, and it is unclear what new insights about neural coding are gained beyond demonstrating that the model can recover known, supervised factors. Two of the latent groups are trained with supervised labels, presumably corresponding to pose and category. For these, the generated images vary sensibly along the provided latent axes, which serves as a good sanity check for the method. However, it is not at all clear how much variance these axes explain in the overall neural space. Given that a considerable fraction of IT variance is already known to relate to category and pose, these results may simply reflect that expected structure. It would be useful to quantify how much of the explained variance these supervised groups account for, and whether the learned representations capture any additional structure beyond the labeled variables.\n\n\n- The most interesting aspect of the approach lies in the unsupervised latent groups, yet these are only very briefly discussed and remain largely qualitative. It is unclear what these groups encode, how stable they are across datasets or neurons, and what fraction of neural variance they capture. A more detailed analysis of these unsupervised dimensions would greatly strengthen the paper and help assess whether the model uncovers genuinely new aspects of neural representation.\n\n\n- Methodologically, the two-step diffusion process—first inverting and then guiding image synthesis—should be motivated more clearly. Is this procedure intended to constrain the generation to remain on the natural image manifold, for instance analogous to preserving phase information in Fourier space? Does the choice of starting image determine the outcome in the second stage?\n\n\n- Finally, the introduction would benefit from clearer framing, as it currently shifts rapidly between topics without defining a specific gap in knowledge or central research question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jxwAtRMVuD", "reviewer": "Reviewer_4PLU", "summary": "This paper presents MIG-Vis, a computational method for interpreting mixed selectivity in neural population dynamics. The approach combines a semi-supervised variational autoencoder that decomposes neural latent representations with a diffusion model for stimulus generation. The method replaces variance-based optimization with mutual information maximization using InfoNCE estimation to generate visual stimuli that selectively activate specific latent factors. Experimental validation is conducted on neural recordings from macaque IT cortex during visual decision-making tasks, with reported improvements in disentanglement metrics and demonstration of generated stimuli showing selectivity to targeted neural factors.", "questions": "1. Given the suboptimal reconstruction performance of VAE, how can you demonstrate that the learned representations capture meaningful neural dynamics rather than statistical artifacts?\n2. Can you provide ablation studies isolating the contribution of MI guidance versus VAE architecture choices to determine which components actually drive the reported improvements?\n\nSuggestions: \n1. The authors should include at least one synthetic dataset with known ground truth factors to provide reliable evidence for method effectiveness and generalization capabilities.\n 2. The authors should provide explicit quantitative comparison with their prior BeNeDiff method to clearly demonstrate the specific advantages of MI guidance over variance-based approaches.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed mutual information maximization approach offers a new perspective for stimulus generation in neural interpretation tasks, demonstrating improved disentangled visual stimulus reconstruction effects.\n2. The paper is well-structured with relevant background coverage, providing sufficient implementation detail for reproducibility.", "weaknesses": "1. The paper reports poor reconstruction R² values alongside high MIG disentanglement scores, creating a concerning contradiction. If the VAE cannot adequately explain neural variance, the validity of downstream stimulus generation becomes questionable.\n2. The entire evaluation relies solely on a single real-world dataset (macaque IT cortex) with no additional validations such as synthetic data. This provides insufficient evidence for method generalizability or robustness.\n3. The lack of controlled synthetic datasets with known underlying factors prevents proper validation of whether the method recovers true latent structure versus dataset-specific artifacts.\n4. The approach bears high similarity to the authors' prior BeNeDiff method, which represents a natural and important baseline for comparison. Additional comparison would help substantiate the claimed effectiveness of MI guidance over variance-based approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)