An Analysis of Human Alignment of Latent Diffusion Models

Authors: lorenz  linhardt
marco  morik
sidney  bender
naima elosegui borras
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-02 19:21:25

Abstract: Diffusion models, trained on large amounts of data, showed remarkable performance for image synthesis. They have high error consistency with humans and low texture bias when used for classification. Furthermore, prior work demonstrated the decomposability of their bottleneck layer representations into semantic directions. In this work, we analyze how well such representations are aligned to human responses on a triplet odd-one-out task. We find that despite the aforementioned observations: I) The representational alignment with humans is comparable to that of models trained only on ImageNet-1k. II) The most aligned layers of the denoiser U-Net are intermediate layers and not the bottleneck. III) Text conditioning greatly improves alignment at high noise levels, hinting at the importance of abstract textual information, especially in the early stage of generation.

Main paper: 
 introduction: 
 Generative diffusion models have demonstrated remarkable efficacy in image synthesis and editing (e.g. (Dhariwal & Nichol, 2021; Rombach et al., 2022; Ruiz et al., 2023) ), image classification (Li et al., 2023a; Clark & Jaini, 2023; Xiang et al., 2023) , where they have been shown to make humanlike errors and shape bias (Jaini et al., 2024) , and in learning object-specific representations (Gal et al., 2023) . Finding semantically meaningful internal representations of diffusion models is thus key to better comprehending their aforementioned representations and capabilities. Success in this quest may enable better control over the generation process and yield effective representations in downstream tasks.Recent findings suggest that the U-Net architectures (Ronneberger et al., 2015) , employed as denoisers in most image diffusion models, capture the semantic information in the bottleneck layer ('h-space') (Kwon et al., 2022; Park et al., 2023; Haas et al., 2023) . However, the representations generated at medium-depth layers of the up-sampling stage appear to be the most useful for image classification (Xiang et al., 2023) but remain inferior to representations of self-supervised models (Hudson et al., 2023) . Despite these insights, the question of where and how diffusion models represent the concepts to be generated remains unsolved.In this paper, we look at representations of diffusion models from the perspective of humansimilarity alignment (Muttenthaler et al., 2023a) (henceforth 'alignment'), as measured on an imagetriplet odd-one-out task (Hebart et al., 2020) . We hope that this perspective helps us understand generative diffusion models by probing the global structure of representations. As suggested by Sucholutsky et al. (2023) , one should measure all components of a model to determine whether it is aligned with a reference system, thus we conduct our evaluation at different layers of the U-Net.Contributions We contribute to the understanding of diffusion models through an empirical analysis of their representations. For this purpose, we assess their alignment with human similarity judgments and examine the alignability of these representations. Our findings reveal that representations from different layers of the U-Net exhibit alignment comparable to classification models trained on much smaller datasets. Notably, the second up-sampling block yields the representa- tions with the highest alignment, from which semantic concepts, except for colors, are also best decodable. We find that alignment decreases with increasing levels of diffusion noise. However, we demonstrate that for high noise levels, text conditioning neutralizes the effect of noise, leading to stable alignment throughout the generative process. 
method: 
 An overview of our workflow for assessing latent diffusion models' alignment with human similarity judgments can be found in Fig. 1 . In the following section, we provide details on the individual methodological parts: Sec. 2.1 describes how representations are extracted, Sec. 2.2 contains details on how their alignment is measured, and in Sec. 2.3 additional information on the improvement of alignment is provided. In contrast to other works on semantic spaces in diffusion models (e.g. Kwon et al. (2022) ; Park et al. (2023) ; Haas et al. (2023) ), our focus is on Stable Diffusion (SD) models (Rombach et al., 2022) due to their training on large and diverse datasets, presumably leading to rich representations.To extract the representations from diffusion models, we follow the approach of Xiang et al. (2023) . Given an image x and noise level t, we feed the denoising network f θ (z t , t, c) a noisy latent z t , generated using the latent diffusion encoder, and optionally some text embedding c. We denote the noise level as the percentage of total noising steps T taken, where the exact amount of noise is determined by the scheduler1 (see Appx. B for a visualization). We then record the internal representation of the U-Net after each of its constituent blocks separately. We apply average pooling to the spatial dimensions to obtain our final (zero-shot) representations per layer r l t (see Appx. E for a comparison to alternatives).To quantify the extent of representational alignment between humans and diffusion models, we follow Muttenthaler et al. (2023a) and use the THINGS dataset, which consists of neuroimaging and behavioral data of 4.70 million unique triplet responses, crowdsourced from 12,340 human participants for m = 1854 natural object images (Hebart et al., 2020) and builds on the THINGS database (Hebart et al., 2019) . To create the THINGS dataset, humans were given a triplet odd-oneout task, consisting of discerning the most different element in a set of three images belonging to distinct object types. There is no correct choice and for any given triplet the answer may vary across participants. The odd-one-out accuracy (OOOA) is a metric used to quantify model and human alignment by assessing what fraction of the odd-one-out determined via the network's representations corresponds to the image selected by humans. The similarity matrix S ∈ R m×m of the model's representations is computed by S a,b := r T a r b /(∥r a ∥ 2 ∥r b ∥ 2 ), i.e. the cosine similarity between the representations extracted from the model f θ . For a triplet {i, j, k} ∈ T , where T is the set of all triplets and w.l.o.g. {i, j} are the indices of the most similar pair of the triplet, according to the human choice:OOOA(S, T ) = 1 |T | {i,j,k}∈T 1[(S i,j > S i,k ) ∧ (S i,j > S j,k )](1)Poor alignment does not mean that the relevant concepts are not contained in the representations. It has been shown that a linear transformation can drastically improve the OOOA (Muttenthaler et al., 2023a) . Thus, in addition to measuring the zero-shot alignment of representations extracted from diffusion models (i.e. without modifying the representations), we measure their affine alignabilty, i.e. how much their OOOA can be increased using an affine transformation. For this step, we follow Muttenthaler et al. (2023a; b) and learn a naive transform, i.e. a square weight matrix W and bias b for each set of representations:EQUATIONHere, Ŝ is the cosine similarity matrix of the transformed representations r = W r + b. Intuitively, the goal of the optimization is to maximize the relative similarity Ŝi,k of the images not chosen as the odd-one-out by the human participants. The magnitude of the transformation is kept small by the regularization term, in order not to distort the original representations too much. We use 3-fold cross-validation (CV) on the THINGS dataset and pick the best λ ∈ {10 i } 1 i=-4 . The resulting 'probed' representations can then be evaluated in the same way as the original ones. 
experiments: 
 We evaluate three latent diffusion models (Rombach et al., 2022) trained on the LAION-5B dataset (Schuhmann et al., 2022) : Stable Diffusion 1.52 (SD1.5), Stable Diffusion 2.13 (SD2.1), and Stable Diffusion Turbo4 (SDT), the latter being an adversarial distilled version of SD2, enabling generation with fewer steps (Sauer et al., 2023) . The main body of the paper focuses on SD2.1, and we refer to the appendix for results obtained from the other models. First, we analyze how well the representations of the diffusion models are aligned with human similarity judgments. Then we show how the alignment of diffusion model representations varies over noise levels and the different layers. Lastly, we show the influence of text-conditioning on the alignment.We first analyze the representations generated from x without further text conditioning. This is the most naive and perhaps faithful implementation of the image triplet tasks, as only image information is used. In Fig. 2 , it can be seen that the highest OOOA across layers is 45.31% for SD1.5, 45.47% SDT, and 43.29% for SD2.1. These values are below the average of the models evaluated by Muttenthaler et al. (2023a) and roughly comparable to self-supervised models trained on ImageNet-1k. Note that due to choice disagreement between humans, the maximum achievable accuracy is only Noise level (%) 38.6 40.0 42.0 38.3 36.4 35.8 42.8 40.8 38.7 38.6 40.3 42.7 39.2 37.2 36.0 43.2 41.2 38.6 38.3 40.4 42.7 39.8 37.9 36.3 43.2 41.2 38.4 38.2 40.2 42.3 39.9 38.3 36.6 42.9 41.0 38.1 38.0 39.9 41.8 39.7 38.6 36.8 42.3 40.7 38.0 37.8 39.6 41.1 39.4 38.5 36.8 41.4 40.2 37.9 37.6 39.1 40.4 38.8 38.1 36.5 40.4 39.6 37.8 37.5 38.5 39.3 38.1 37.4 35.9 39.1 38.7 37.7 37.3 37.9 38.2 37.2 36.7 35.5 37.7 37.7 37.5 37.2 37.5 37.1 36.3 35.8 34.8 36.5 36.5 2020), whereas the accuracy of random guessing is around 33. 3%. We conclude that the capabilities of SD models are not reflected in the human alignment of their intermediate representations.In this section, we briefly present the OOOA results obtained after applying an affine transformation, learned for each block individually, as outlined in Sec. 2.3. It can be seen in Fig. 6 that the overall pattern across layers and noise levels does not change, but alignment increases generally. While this improvement is substantial, the alignment of the transformed representations is only slightly better than that of models trained on much less data (Muttenthaler et al., 2023a) , after a similar transformation. This may indicate either that the dimensions relevant for human similarity judgments are not much better represented in SD models, or that more flexible transformations are needed to extract them.In unconditional diffusion models, the bottleneck layer of the U-Net appears to carry the most semantic information (Kwon et al., 2022) and to encode concepts as directions. This idea is further supported by recent works (Park et al., 2023; Haas et al., 2023) . We find that this does not hold for SD models.The OOOA obtained from the representations extracted at different layers and for different levels of noise are displayed in Fig. 2 . The most aligned layers are the intermediate up-sampling layers, which corresponds to the layers found to be most useful for linear classification (Xiang et al., 2023) , albeit we find little to no degradation until noise levels of at least 30%. Furthermore, one might assume that for small t, the model would not need to involve the deeper layers to remove the little noise that is left and thus the representations at the deeper layers degrade. This does not appear to be the case.We speculate that the reason for the discrepancy with the results previously reported on unconditional diffusion models lies in the complexity of the SD models, which were trained on a diverse dataset with various modes. Here, the learned representation might not admit simple linear extraction of concepts.A natural question to ask is whether different human concepts are represented at different levels of depth in SD models, for example, more abstract concepts being more salient in deeper layers. To investigate this question, we make use of the VICE dimensions (Muttenthaler et al., 2022) , which model the human similarity space using a human-interpretable positive orthogonal basis. Using VICE, each image of the THINGS dataset can be decomposed into 45 dimensions. We use the labeling of the dimensions from (Muttenthaler et al., 2023a) , noting that it is only a post-hoc interpretation of their semantics.We follow the experimental protocol of Muttenthaler et al. (2023a) and train a multinomial ridge regression to predict the VICE dimensions from the extracted representations. The results were obtained using 5-fold cross-validation, where, within each fold, the regularization parameter was chosen from {10 i } 5 i=-2 using leave-one-out CV. In Fig. 3 the regression metric, measured by R 2 is computed for distinct concepts at varying layer depths. Qualitatively, it can be observed that except for the colors red, green, and yellow, which follow the same pattern of correlation, there is little differentiation of concepts across layers. Most concepts are best decodable from the second up-sampling block. See Appendix C.2 for additional concept-wise results across noise levels. Most concepts remain stable up to about 40% noise and degrade beyond that.Diffusion models are often trained and used with textual prompts to guide generation. In this section, we investigate the effect of textual conditioning of SD models on their alignment. In particular, we condition the reconstruction of x from z t on 'a photo of a 〈OBJ〉', where 〈OBJ〉 is replaced by the name of the object depicted in the image, as per the image file name.We observe that textual conditioning stabilizes alignment across noise levels, keeping the variability across layers intact but reducing the variability across noise levels to a low level. At very high levels of noise, where the denoiser has to rely almost exclusively on the text conditioning, there may even be improvements to the OOOA, stemming from the relatively higher text-embedding OOOA (see Appx. D). For SD2.1, especially the bottleneck and adjacent blocks benefit from text conditioning beyond their unconditional maximum values, although only at higher noise levels. Improvements are less localized in SD1.5. We refer to Appx. D for the full set of results as well as a comparison with conditioning on the output of a text captioning model. blocks of the denoising network, we find the intermediate up-sampling blocks yield the most aligned representations. Furthermore, we observe that conditioning the denoising on textual object labels improves alignment at high levels of noise.The presented results open several lines of future investigations. Does the residual structure of the U-Net architecture itself affect the alignment of its individual components? Is the visual reconstruction objective of generative models orthogonal to human alignment of representations? Perhaps the way the representations are structured even requires a different measure of alignment (e.g. evaluating the triplet task with a similarity measure other than cosine similarity). As the representation space might be highly non-linear, alignment-increasing transformations may need to allow for non-linearity.Denoising diffusion models have emerged as effective generative models for a variety of tasks, including unconditional image generation (Sohl-Dickstein et al., 2015; Ho et al., 2020; Dhariwal & Nichol, 2021 ), text-to-image synthesis (Ho & Salimans, 2021; Saharia et al., 2022; Rombach et al., 2022) , and inverse problems (Song et al., 2021; Chung et al., 2022) . As these models gain widespread adoption, understanding their internal representations becomes crucial. Their text-toimage synthesis capabilities suggest semantic knowledge, which has proven useful for classification (Li et al., 2023a; Jaini et al., 2024) and learning representations for downstream tasks (Mittal et al., 2023) . Analyzing the representation space facilitates the identification of failure modes (Liu et al., 2024) and semantic directions (Haas et al., 2023; Park et al., 2023) . Such analysis, akin to work on GANs (Härkönen et al., 2020) , also allows for the manipulation at the bottleneck layer of U-Net (Kwon et al., 2022) . A parallel line of inquiry attempts to train diffusion models specifically for representation learning (Hudson et al., 2023; Mittal et al., 2023) or to infuse their representations with concepts (Ismail et al., 2023) .The comparison of behavior between neural networks and humans has been approached from different angles: the majority consider error consistency in image classification (Geirhos et al., 2020; 2021; Rajalingham et al., 2018) , others focus on semantic similarity judgments (Jozwik et al., 2023; Peterson et al., 2018; Aminoff et al., 2022; Marjieh et al., 2022) , or analyse perceptual similarity (Zhang et al., 2018; Jagadeesh & Gardner, 2022) . We build upon an analysis of human and neural network similarity judgments Muttenthaler et al. (2023a) to assess the alignment of representations extracted from pretrained diffusion models.Noisy Image 5% 10% 20% 30% Noise Level 40% 50% 60% 70% 80% 90% Fig. 4 shows both the noisy latent and its x reconstruction for Stable Diffusion 2.1. The reconstruction quality remains good up to 60% noise, while from 80% noise on, the image is barely identifiable. This matches the decrease in alignment observed in representation space.In this section we report the OOOA results for unconditioned representations, using all evaluated SD models. The patterns discernable in Fig. 5 follow a similar pattern as described in Sec. 3.1, but in SD1.5 OOOA is almost as high at the middle layer as it is at the second up-sampling block.The complete OOOA results for affine transformed representations, using all models, are reported in Fig. 6 . The general pattern is consistent across models and similar to the one observed for the original representations, albeit at a generally higher level of alignment. Specifically, we see that the Up 1 block yields the most aligned representations, with slightly lower values at its symmetric counterpart, Down 2. For SD1.5, the layers between those two layers are more aligned than in SD2.1 and SDT. Noise level (%) 38.3 39.8 43.5 43.8 44.1 42.1 45.2 40.7 38.4 38.3 39.7 43.7 44.0 44.2 42.3 45.3 40.9 38.6 38.3 39.5 43.9 44.2 44.3 42.6 45.3 41.2 38.9 38.2 39.4 43.7 44.3 44.3 42.8 45.2 41.4 39.3 38.0 39.1 43.2 44.2 44.2 43.0 44.8 41.6 39.6 37.8 38.7 42.3 43.7 43.7 42.8 44.0 41.4 39.7 37.5 38.2 41.1 42.5 42.5 41.8 42.8 40.7 39.5 37.2 37.8 39.8 40.9 40.5 40.0 41.0 39.8 38.8 37.0 37.4 38.5 39.0 38.1 37.7 38.6 38.2 37.4 36.9 37.1 37.6 37.6 36.4 36.1 36.7 36.6 38.6 40.0 42.0 38.3 36.4 35.8 42.8 40.8 38.7 38.6 40.3 42.7 39.2 37.2 36.0 43.2 41.2 38.6 38.3 40.4 42.7 39.8 37.9 36.3 43.2 41.2 38.4 38.2 40.2 42.3 39.9 38.3 36.6 42.9 41.0 38.1 38.0 39.9 41.8 39.7 38.6 36.8 42.3 40.7 38.0 37.8 39.6 41.1 39.4 38.5 36.8 41.4 40.2 37.9 37.6 39.1 40.4 38.8 38.1 36.5 40.4 39.6 37.8 37.5 38.5 39.3 38.1 37.4 35.9 39.1 38.7 37.7 37.3 37.9 38.2 37.2 36.7 35.5 37.7 37.7 37.5 37.2 37.5 37.1 36.3 35.8 34.8 36.5 36.5 38.4 40.4 45.0 39.9 38.1 36.0 45.5 43.9 39.8 38.2 40.2 45.1 40.2 38.6 36.2 45.5 44.3 40.0 38.1 40.0 45.0 40.6 39.5 36.7 45.4 44.8 40.5 37.9 39.9 44.7 40.9 40.0 37.0 45.3 45.1 40.9 37.7 39.7 44.2 41.0 40.5 37.1 44.8 45.0 41.1 37.4 39.3 43.3 41.0 40.5 37.3 44.1 44.3 40.9 37.2 38.8 42.0 40.5 39.9 37.1 42.6 42.6 40.0 36.9 38.1 40.4 39.7 39.0 37.0 40.8 40.7 38.8 36.8 37.6 38.6 38.0 37.4 36.4 38.6 38.6 37.4 36.6 37.0 37.0 36.3 36.0 35.7 36.7 36.9 4 38.3 47.8 51.8 53.8 54.1 53.9 55.4 53.3 45.5 37.4 46.3 50.3 52.0 52.2 52.7 54.3 52.3 44.6 36.2 44.3 48.0 49.4 48.2 49.8 51.1 49.7 42.9 35.3 41.5 45.7 45.8 43.9 45.2 47.2 45.6 38.6 34.1 38.9 42.8 42.8 41.8 41.9 43.2 41.9 36.3 28.7 35.7 39.7 40.7 40.2 39.0 39.7 38.8 35.5 Transformed SD1.5 ( = 0.1) Transformed SDT ( = 0.1) Figure 6 : Odd-one-out accuracy for transformed representations without text conditioning. The observed alignment is greatly improved over zero-shot representations (Fig. 5 ). Noise level (%) 38.4 39.8 43.4 43.8 43.7 41.7 45.1 40.7 38.4 38.3 39.7 43.7 44.0 43.8 41.9 45.3 40.9 38.6 38.3 39.6 43.8 44.2 43.9 42.1 45.4 41.2 38.9 38.3 39.6 43.7 44.3 43.9 42.3 45.3 41.6 39.3 38.3 39.6 43.6 44.4 43.9 42.4 45.1 41.9 39.6 38.2 39.6 43.5 44.6 44.0 42.5 44.9 42.0 39.8 38.1 39.5 43.4 44.8 44.0 42.4 44.7 42.1 40.0 37.9 39.5 43.4 45.1 44.0 42.3 44.5 42.1 40.1 37.7 39.3 43.1 45.5 44.3 42.4 44.5 42.5 40.2 37.5 39.3 43.0 45.8 44.6 42.5 44.5 42.9 39.9 Label-conditional SD1.5 Noise level (%) 38.4 39.9 43.5 43.7 43.6 41.6 45.1 40.7 38.4 38.4 39.8 43.7 43.9 43.7 41.8 45.3 40.9 38.6 38.4 39.7 43.8 44.0 43.7 42.0 45.4 41.3 38.9 38.5 39.8 43.7 44.1 43.7 42.1 45.3 41.6 39.3 38.5 39.8 43.6 44.2 43.8 42.3 45.1 41.9 39.6 38.5 40.0 43.5 44.4 43.8 42.3 44.9 42.0 39.9 38.5 40.1 43.7 44.7 44.0 42.4 44.9 42.2 40.1 38.5 40.4 44.0 45.0 44.2 42.5 44.9 42.3 40.2 38.5 40.7 44.3 45.4 44.5 42.9 45.1 42.8 40.3 38.2 41.3 44.6 45.6 44.6 43.4 45.3 43.0 39.6 Caption-conditional SD1.5 8 38.6 36.5 35.4 43.3 41.0 38.7 38.6 40.5 43.4 39.4 37.2 35.9 43.8 41.5 38.6 38.4 40.8 43.5 40.0 37.8 36.3 44.0 41.7 38.4 38.3 40.9 43.4 40.2 38.6 36.7 43.9 41.8 38.2 38.2 40.9 43.4 40.3 38.9 36.9 43.8 41.8 38.1 38.0 41.1 43.2 40.5 39.5 37.0 43.5 41.9 38.1 38.0 41.1 43.2 40.6 39.6 37.0 43.3 42.0 38.1 37.8 41.0 43.1 41.0 40.1 37.3 42.9 42.1 38.1 37.8 40.9 42.9 41.2 40.3 37.4 42.7 41.9 38.1 37.8 41.1 42.8 41.7 40.8 37.8 42.1 41.4 38.2 Caption-conditional SD2.1 2 37.3 44.5 44.8 41.2 38.4 41.4 44.6 41.6 41.8 37.7 44.4 44.4 41.1 38.8 42.1 44.7 42.1 42.5 38.1 44.2 43.9 40.8 39.2 43.3 45.1 42.6 43.1 38.6 44.2 43.6 40.5 39.8 44.9 45.4 43.0 43.6 39.2 44.3 43.6 40.3 Caption-conditional SDT At Down 0 with high noise, OOOA values can get below the random-guessing level of 1 3 . This is due to counting a triplet-task solution as wrong if more than one pair shares the highest similarity value and thus the representations do not unambiguously yield an odd-one-out.We use the default scheduler for each model from the diffusers library https://github.com/ huggingface/diffusers.https://huggingface.co/runwayml/stable-diffusion-v1-5https://huggingface.co/stabilityai/stable-diffusion-2-1https://huggingface.co/stabilityai/sd-turboCONCLUSIONDespite previous work uncovering semantic directions in smaller diffusion models and the outstanding capabilities of stable diffusion models, we show that internal representations of the latter are not exceedingly aligned with the similarity space extracted from human behavioral experiments. While an affine transformation improves alignment significantly, the gap to contrastive image-text models trained on large amounts of data remains unclosed. This suggests that diffusion models trained on large multi-modal datasets do not have a linearly decodable representation space. Of the various 
 
 Appendix:  



Reviews: review: title: title: Good paper but limited novelty
main_review: main_review: This work investigates the alignment between internal representations in Stable Diffusion models and the similarity space derived from human behavioral experiments. Despite the effectiveness of stable diffusion models, the authors found that their internal representations did not closely match the human behavioral similarity space.

**Clarity** and **Correctness**: The paper is well-written, and the pipeline illustration is straightforward and intuitive. However, Section 2.4, *Alignment by Affine Probing* could be improved. It is unclear what the optimization's objective is, requiring reading the referenced papers for a better understanding.

**Novelty**: The novelty of the paper is somewhat limited. It concentrates on analyzing Stable SD models within the same setting used by other papers studying different models (e.g., CLIP-based and ResNet). Despite the distinct nature of these models—given that existing works rely on classifiers rather than diffusion models—the method for extracting representations from the diffusion models is not novel but rather follows an existing approach.

**The paper is well-suited for the workshop.**
strength_weakness: strength_weakness: The paper is well-written and easy to follow. The authors proposed several future investigations that could be interesting to explore.
title: title: Interesting paper with many surprising findings
main_review: main_review: The authors investigate the representation alignment between humans and diffusion models. I found this paper particularly interesting because it has several surprising, unexpected observations: (1) stable diffusion is not as aligned as other less well performing generative models. This seems to imply that stable diffusion develops a strategy to represent images that is orthogonal to humans. (2) The bottleneck is not the informative/aligned layer, but rather some up-sampling layers are more aligned with human representations. (3) representations are not linearly decodable for complex datasets; (4) text conditions can improve representation alignment even in the high noise regime. This may imply that human's visual representations are also closely related to language?

This paper is clearly written and sound. My only nitpick is that the study is not very mechanistic, i.e., making a bunch of observations without making very mechanistic explanations.
strength_weakness: strength_weakness: I actually ride on fence whether to give a 2 or a 3. My reluctance was coming from the fact that this paper is basically an experimental paper without mechanistic explanations (i.e., does not provide constructive suggestions on e.g., how to construct better generative models, or more human-aligned models). Said that, I ended up giving a 3 since I was probably asking too much for a short workshop paper.
title: title: Adding human-alignment angle to diffusion models
main_review: main_review: # Summary
The paper analyzes the alignment of latent diffusion models with human responses on a triplet odd-one-out task. It identifies that the most aligned layers of the denoiser U-Net are intermediate upsampling layers, not the bottleneck, as suggested by prior works. They show that text-conditioning improves alignment at higher noise-levels.
The paper highlights that despite the outstanding capabilities of diffusion models, their internal representations are not highly aligned with human similarity space. This suggests that diffusion models may not have a linearly decodable representation space.
# Strong points
-	Well written and easy to follow, motivation and line of experiments is very clear
-	Interesting and novel results
# Weak points
-	Some of the results are hard to understand (e.g. in Figure 2, zero-shot and probing are nor clearly defined before)
-  The link between plot labels and caption is missing (intermediate up-sampling block = up1?)
-  Context if achieved accuracies are high or low is missing (what is baseline, max accuracy?)
# Clarity
Clear structure and writing, easy to follow. Results are well structured and follow three main experiments, with clear goals. 
# Correctness
Methods and experimental design seem well-grounded in previous works, while still adding a novel research angle. Everything is well described. 
# Novelty
No novel methods, but novel research question adding a human-alignment angle to the semantic latent space discovery/analysis in diffusion models. Results challenge previous works and suggest not only looking at the bottleneck layer but also upsampling layers for semantic spaces. 
# Interest to community
Potential interest for machine learning researchers interested in diffusion models or interpretability of models. Limited interest for neuroscientists and cognitive scientists. 
# Recommendation
I recommend to accept this paper, as it is fitting to the workshop and of potential interest for the community. The findings are novel and expand on previous works by adding an human-alignment angle. 
# Recommendations for improvement
-	Main results and conclusion can be formulated clearer and provide some context and potential future implications.
-	Clearer figure captions (e.g. intermediate layer = up1,2), it takes some time to know where to look
-	For the future, it might be interesting to look at different human-alignment tasks
strength_weakness: strength_weakness: Work is well-suited for the workshop

Comments: title: Reply to reviewers
comment: We thank the reviewers for the positive feedback as well as for the suggested improvements.
 
Following reviewer $\textit{tzfZ}$, we added clarifications to the probing optimization's objective to convey the underlying intuition.
Furthermore, we tried to address the ambiguities pointed out by reviewer $\textit{veLK}$, in particular unclear elements in the figure captions, and emphasized the min/max achievable accuracy.
We also thank reviewer $\textit{vf3j}$ for their constructive review; we hope to soon be able to extend our findings with explanations of the mechanisms underlying our observations.