Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners
Authors: Xuehai He
Weixi Feng
Tsu-Jui Fu
Varun Jampani
Arjun Reddy Akula
Pradyumna Narayana
S Basu
William Yang Wang
Xin Eric Wang
Venue: Accepted by TMLR ()

Abstract: Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation. Since text-to-image generation often requires models to generate visual concepts with fine-grained details and attributes specified in text prompts, can we leverage the powerful representations learned by pre-trained diffusion models for discriminative tasks such as image-text matching? To answer this question, we propose a novel approach, Discriminative Stable Diffusion (Discffusion), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information and fine-tune the model via a new attention-based prompt learning to perform image-text matching. By comparing Discffusion with state-of-the-art methods on several benchmark datasets, we demonstrate the potential of using pre-trained diffusion models for discriminative tasks with superior results on few-shot image-text matching.

Published in Transactions on Machine Learning Research (09/2024)
Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners
Xuehai He' Weixi Feng² Tsu-Jui Fu²
Varun Jampani Arjun Akula* Pradyumna Narayana*
Sugato Basu* William Yang Wang? Xin Eric Wang
UCC Santa Cruz, UC Santa Barbara, "Stability AI, 'Google (xhe89, xwang366))uccccceeer
Reviewed on OpenReview:
Abstract
Diffusion models, such as Stable Diffusion (Rombach et al. 2022a), have shown incredible performance on text-to-image generation. Since titttto-image generation often requires models to generate visual concepts with fine-grained details and attributes specified in lext prompis, can leverage the powerful representations learned by pre-trained diffusion models for discriminative tasks such image-text matching? To answer this question, propose novel approach, Discriminative Stable Diffusion (Discffusion), which turns pre- trained text-to-image diffusion models into few-shot discriminative learners. Our approach uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information and fine-tune the model via a new attention- based prompt learning to perform image-lexi matching. By comparing Discffusion with stale- of-the-art methods on several benchmark datasets, we demonstrate the potential of using pre-trained diffusion models for discriminative tasks with superior results few-shot image- text matching.
1 Introduction
"What I Cannot Create, I Do Not Understand."
Richard Feymmon
This quote by Richard Feynman perfectly captures the essence of human learning techniques. In the context of machine learning, it can be interpreted as the ability to generate images given text prompts is a strong indicator of understanding and matching between visual and textual information. Despite the success of various methods in the image-text matching task (Karpathy & Fei-Fei 2015 Lee et al. 2018). there is still a need for more advanced models that can better capture the fine-grained details, spatial relat ionships, and compositionality. Mcanwhile, diffusion models (Sohl-Dickstein et al.) 2015a Rombach et al.] 2022a) have been shown produce high-quality and diverse images from text descriptions. Therefore, in this paper, we investigate the idea of leveraging the power of pre-trained Diffusion Models, specifically the state-of-the-an text-to-image generative model-Stable Diffusion (Rombach et al. 2022a), for the discriminative imago-text matching task, as shown in Figure D The success of Stable Diffusion in generative tasks suggests that it has a strong understanding of the relationship between visual and textual information, and we aim to harness this understanding for image-text matching tasks.
The key advantages of using Stable Diffusion for image-text matching are two folds: first, Stable Diffusion
VQVAE (Kingma & Welling 2013: Van DenOord et al. 2017) and cross-attention layers in its

---

Published in Transactions on Machine Learning Research (09/2024)
Dark brown horse with long straight black tail standing in a field.
Stable
Diffusion
Score computation Prompt learning
Dark brown horse with long straight black tail standing up in a field.
Discriminative Stable Diffusion
Mattchigg Score:
0.691
Discriminative Stable Diffusion
Matching Score: 0.103
A commerciat stainless kitchen with a pot of food cooking.
x
Figure 1: The upper subfigure in the teaser image illustrates the ability of Stable Diffusion to generate realistic images given text prompt. The bottom subfigure illustrates the process of our proposed method, Discriminative Stable Diffusion (Discffusion), for utilizing Stable Diffusion for the image-text matching task. Discffusion can output a matching score for a given text prompt and image, with a higher score indicating a stronger match.
architecture, which provides strong compressed representations and shed information about the alignment of data from different modalities. Second, Stable Diffusion, through proper adaptations, demonstrates a
capacity to comprehend spatial relations Wu et al. 2023 Derakhshani et al. 2023) and to discern fine- tne-grained,
disentangled concepts (Matsunaga et al. 2022 Rambhatla & Misra 2023 It can generate images that closely
align with the specifies of textual prompts, while traditional vision and language model such as CLIP (Radford
et al. 2021), which are pre-trained on discriminative (asks, predominantly facilitate coarse-grained, contextual image-text text alignment. Such models lack the capability to perform compositional matching at finer gramularity, specifically failing to achieve detailed cross-modal alignment at the region-word level (Jiang et al. 2022).
However, to efficiently adapt Stable Diffusion to the image-text matching task, two key challenges need to be addressed: (1) how to disentangle the degree of aligmment between the image and text from the latent space of Stable Diffusion? In text-to-image gencration, the model is trained to generate an image that is semantically consistent with given text prompt. However, in image-text matching, the task is to determine the degree alignment between a given image and text. Therefore, it is important to disentangle the degree of alignment between the image and text in the latent space of Stable Diffusion, to effectively use it for image-text matching: (2) How to efficiently adapt the model in few-shot setting. Adapting text-to-image generation model such as Stable Diffusion for image-text. matching involves transitioning the model from generative to discriminative task. This shift presents significant challenges due to the differences in task requirements and underlying model architectures.
To address these challenges, we propose the Discriminative Stable Diffusion (Discffusion) method, which includes two key ideas: (1) identifying and leveraging attention scores from the selocted cross-attention maps as the matching score and (2) employing attention-based prompt learning for model fine-tuning.
Our contributions in this paper are threefold:
We conduct a pioneering study that repurposes latent text-to-image generation diffusion mod- els origiinllllly designed for generative tasks for discriminative tasks such image-text matching.
We introduce a novel method that leverages cross-attention maps from Stable Diffusion across multiple layers, combined with allention-based prompt learning, to address the image-texi matching task.
We demonstrate the effectiveness of our approach through experimental evaluation on Compositional Visual Genome (Jiang al.) 2022), RefCOCOg (Yu et al. 2016). Winoground (Thrush et al.) 2022). and VT-checklist (Zhao et al. 2022) datasets for image-text matching. We further demonstrate the
2

---

Published in Transactions on Machine Learning Research (09/2024)
adaptability of our method by applying it to the visual question answering task, where it excels the VQAv2 (Antol et al. 2015) dataset.
2 Related Work
Diffusion Probabilistic Models (DPMs) Diffusion probabilistic models (DPMs) have been widely used generative models for images in recent years. These models, which include diffusion (Sohl-Dickstein et al. 2015b) and score-based generative models (Song & Ermon 2019), have been shown to outperform generalive adversarial networks (GANs) Goooffelow et al. 2014) in many cases (Baranchuk et al.) 2021). In the past two years, significant progress has been made in the development of DPMs, with a focus on improving sampling techniques such as classifier-free guidance (Ho & Salimans, 2021 p. DPMs are typically implemented using convolutional U-Net architectures (Ronneberger al. 2015a) which contain pross-attention layers. Hertz et al. (022a) finds that replacing attention maps in the cross-attention module of text-to- sxxt-to-image generation diffusion models can edit image attributes. Just scaling the attention maps of the respective word can adjust the effect of a particular word in the prompt. Feng et al. (2022) demonstrates that one can retain the compositional semantics in the generated image by manipulating the cross-attention. Kumari et al. (022) proposes to fine-tune the key and value mapping from text to latent features in the cross-attention layers of text-to-image diffusion model to compose multiple new concepts in the image. In the context of image-text matching, the attention scores between the text and image representations the DPMs can reflect the degree of aligument between them.
Few-shot Learning for Vision and Language Tasks Vision and Language discriminative models pre-trained on large-scale image-lext pairs have demonstrated great potential in multimodal representation
learning (Jia et al. 2021 Yao et al. 2021 Yuan et al. 2021 Radford et al. 2021 Liu et al. 2024 Among them, CLIP (Radford et al. 2021 benefits from 400M curated data and defines various prompt templates to carry out zcro-shot image classification. Like CLIP, several different few-shot learners were proposed. GPT (Brown el al.] 2020), as a strong few-shot learner, capable of performing a new language task by learning from only a few training instances. Frozen (Tsimpoukelli al.) (021]) is developed based on GPT and made into a multimodal few-shot learner by expanding the soft prompting to include a collection of images and text. The concept of prompt learning (Schick & Schütze 2020) has been widely explored in natural language processing (NLP) and computer vision. It allows pre-trained models to adapt to various downstream tasks with minimal number of data by introducing a small prompt layer (Schick & Schütze 2020 Liu et al.) 2021). In the context of image-text matching, prompt learning has been used to fine-tune pre-trained models for the task (He et al. 2022b). In our work, instead of adding learnable prompts over the inputs or between transformer layers (Jia et al. 2022), we introduce Icarnable prompts over the attention layers. In our paper, our primary research question is the adaptation of pre-trained generative diffusion models into discriminative models for specific tasks. This focus is driven by the challenges and opportunities presented by utilizing diffusion-based processes in discriminative sctting, specifically for the image-text matching task, which has distinct characteristics compared to the modeling approaches mentioned above.
Generative Models for Discriminative Tasks There has been a significant amount of research on using generative models for discriminative tasks in the past decades. Ng & Jordan (2001) compare the discriminative classifier with generative classifier. Ranzato et al. (2011 apply deep generative models to the recognition task. For diffusion models, recently, Li et al. (2023a): Clark & Jaini (2023) propose to use
pre-trained diffusion models for zero-shot classification. Wei et al. (2023) formulate diffusion models
masked autoencoders and achieves state-of-the the-art classification accuracy video tasks. Different from these works, we are the first to explore the use of cross-attention maps in pro-trained diffusion models for discriminative (asks, specifically the image-text matching lask. Another line of works use diffusion models as data source and then training a discriminative model on the synthetic data generated from it (He et al.) 2022a Jahanian et al., 20221: Zhang al., 2021). Differs from these works, our approach emphasizes the direct adaptation of generative diffusion models, leveraging their pre-existing structures and knowledge without the need to generate synthetic data.
3

---

Published in Transactions on Machine Learning Research (09/2024)
Score
LSE Pooling
man wearing white T-shirt dropping racket on court.
CLIP text encoder
XT
steps
VAE encoder
Latent Vector Zz
Cross- attentio
attention
Cross- atlention
ettention
Latent Vector Zo
K
Noise
T: total number of sampling steps
m: number of n: number of
Learnable
text tokens
image feature tokens
W
W
+
p
prompt
Figure The overview of our Discriminative Stable Diffusion framework, which measures how much the given images and texts matched use the cross-attention mechanism the Stable Diffusion. Discriminative Stable Diffusion added learnable prompt over attention matrices (red boxes). The learnable prompt will receive gradients during training and updated, while the pretrained weights are fixed. The layer index in and n is dropped for simplicity.
3 Preliminaries on Diffusion Models
In this section, we provide a brief overview of the concepts and techniques in denoising diffusion models that are necessary to understand our proposed method. Diffusion models are a class of generalive models that are particularly effective at generating high-quality images (Sohl-Dickstein et al. 2015b Nichol et al. 2021 Ramesh et al., 2022: Saharia al., 2022: Rombach al.] 2022b). They aim model distribution Po (To) that approximates the data distribution (xo) and is casy to sample from. DPMs model a "forward process" in the space of xp from data noise by adding noise to real data, and reverse process that tries to reconstruct the original data from the noisy version. The forward process is described by the equation
g = N( v Varao, (1 TI,
(1)
where F1.r defines a set of noisy images and 2o is the initial image. W denotes a Gaussian distribution, and Og are hyper-parameters. The reverse process is modeled by a Gaussian distribution
pe(xx-1/2z) (z)( Ez())))
(2)
where neural networks are used to predict the mean and covariance of the distribution. The parameters of the model, 0, are learned by optimizing a variational lower bound on the log-likelihood of the real data. Once trained, new images can be generated by starting from noise sample and iteratively sampling from the reverse process distribution until reaching the final time step. In latent diffusson probabilistic models such as Stable Diffussion, these two processes are similar, while they proceeds in the latent space: To is encoded into 2p in an efficient, low-dimensional latent space first and then do the diffusion process. And the case where DPM conditioned on additional information, such as text information c, the reverse process becomes po p= v), where y is the input text.
4 Discriminative Diffusion Models
4.1 Problem Formulation
The problem of image-text matching is formalized as follows: given a texl prompt y E y and a set of images X, we aim to find the image x" E X that is most semantically aligned with the given text prompt y. Formally, we define the imago-text matching problem as finding the function f x X -> [0, 1] that assigns score each image-text pair (y. x) indicating the degree of semantic alignment between the text and image. The goal is to find the image that maximizes the score for given text prompt y, ie., x" = arg max ((),
4

---

Published in Transactions on Machine Learning Research (09/2024)
4.2 Method Overview
To learn the function f. the main idea is to leverage the powerful representations learned by pre-trained Stable Diffusion model to perform image-text matching. There are three key modules in Discffusion, cross- attention score computation, LogSumEap pooling, and attention-bosed prompt learning, as shown in Figure 2 The cross-attention score computation module extracts the mutual influence between visual and textual information by computing the attention scores from the cross-altention matrix in U-Nets of the Stable Diffusion model. The LogSumExp pooling module pools these attention scores over all tokens in the text description to obtain single matching score. Finally, the attention-based prompt learning module fine-tunes the model by updating the key and value mappings from text to latent features in the cross- attention layers under a few-shot setting. This allows the model to learn new image-text concepts while retaining the ability to capture complex and nuanced relationships between images and text. The model outputs a score that measures the alignment, between the image and text, which can be used to adapt the model from a text-to- -ontae generation task to an image-text matching task. Note that Discffusion a general pipeline that can adopt other faster sampling strategies as well.
4.3
Score Computation
Cross-attention scores can be a measure of the relevance of an image and text to each other (Chen et al., 2020: Li et al., 2019). Prior research (Hertz et al. (022b) shows that the cross-attention within diffusion models governs the layout of generated images and the scores in cross-attention maps represent the amount of information flows from text token to a latent pixel. They are calculated by taking the dot product of the representations of the image and text in a latent space, and normalizing by the product of their norms. We propose to adapt cross-attention scores and text in the image-lext matching task. In the sequel, we elaborate on our strategy in depth.
way to better capture the complex relationships between images
Stable Diffusion (Rombach et al., 2022a) is trained to generate images from text prompts, and as such, it has learned strong compressed ropresentations of both text and images. This enables us to use these representations to learn the function f for image-text matching.
More specifically, a text prompt y is first encoded into an intermediate representation ry re(y) Rudd, using domain-speeifie encoder Te, where represents the number of text tokens. We then encode cach image x E & where x E Hx3 in RGB space into a latent image representation E(r), where E(x) is the image encoder. The noisy version of the input is obtained by:
00 VI āyc for c~ (O,I).
(3)
where constants ā₂ are saper--prameters inherited from Song al. (000)). which will control the level of noises applied to the latent. representation. The encoder 68 in the U-Net (Ronneberger et al.) 2015b) of the pre-trained text-to-ine text-to-image generation model then encode 2 into r= = po (z),, where 4; (22) E R"xd. n and denote the number of image feature tokens and index of layers respectively (we dropped the superscript for n for notation simplicity). This forms a (flattened) intermediate depiction within the U-Net that uses ca, which are subsequently integrated into intermediate layers of the U-Net via cross-attention coss-attention mechanism
defined as Attention( Q, K, V)- = softmax
[QK")
V, with Q = ( K = =Ty Wkk" V Tg Wo())
Vd
Here, Wh R xd W=4(9) E Rd.. xd Wn( E Idd xd are learnable projection matrices (Jaegle et al. 2021 Vaswani et al., 2017). These matrices map the inputs to query, key, and value matricos, respectively, where d is the output dimension of the projection operations in the attention computation.
4.4 LogSumExp Pooling (LSE)
To compute the function and quantitatively evaluate the degree of semantic alignment between an image and a text prompt, we leverage LogSumExp (LSE) pooling (Blanchard et al. 2021] as a means of aggregating the attention maps generated by the cross-attention mechanism in our model. By using LSE pooling, we are able to take into account the relative importance of different image and text tokens in the attention map, rather than simply averaging or summing all elements in the map. This has several benefits. Firstly, LSE
5

---

Published in Transactions on Machine Learning Research (09/2024)
pooling is able to handle large values and outliers in the attention map more robustly than other pooling methods, such as average or sum pooling. Secondly, LSE pooling has high numerical stability during training. Thirdly, LSE pooling is able to better preserve the ordering of values in the attention map, allowing for more interpretable and accurate matching scores.
For notation simplicity, we drop the batch and attention head dimension, the attention map matrix is denoted as A R=xm, where n represents the number of image tokens (height width) in the latent space and represents the number of text tokens. To compute the image-text pair matching score, utilize the LogSumExp (LSE) pooling operator, which is applied directly to each row of the attention matrix A, resulting in vector of length m, and then averaged to produce the final score:
m
S(A)
log
Σ
exp(AAt))
(4)
Here, A.; represents the i-th row the matrix A, Ave is the average operator, and À is a scaling factor that magnifies the importance of the most relevant pairs of image region features and attended text sentence vectors. By default, we set À 1. This operation computes the log-sum-cxponential over each row, summarizing the contribution of each text loken to the image features, and then averages these values across different timesteps to produce a single scalar score for the image-text pair (v,z). We further enhance the model's capability for image-text matching through attention-based prompt learning, which we will introduce in the next section.
4.5 Attention-basel Prompt Learning for Stable Diffusion
We aim to adapt the latent diffusion probabilistic model to the image-text matching task leveraging only a few examples, that is, under the few-shot sctting. The task of fine-tuning aims at updating the mapping from the given text to the aligned image distribution, and the text features are only input to W and W
projection matrix in the cross-attention block. Inspired by Liu et al. (2021): Lester et al. (2021 ; Li & Liang (2021), propose the use of learnable prompts, which are added to the attention matrices in our model. Specifically, as shown in Figure 2 we introduce learnable prompt embedding matrices, which are added element-wise to the key and value attention ion matrices at the identified layer of the Stable Diffusion model. We keep the original W and Th frozen, and only update the learnable prompt weights addod to them. As our addition operation applies to all layers and sampled timesteps, we will omit superscripts I and layer for notational clarity and obtains:
(5) Both W and Wo are frozen and I² and p" receive gradients and are updated during training, which is inspired by LoRA (Hu al.) 20211) and can further improve the training efficiency. This allows the model adapt new instances by attending to relevant information in the intermediate representation of the text inputs, Ta (v).. With the learned prompt embeddings in the few-shot scenario, we effectively adapt the Stable Diffusion to image-text matching performance. The overall algorithm is shown in Algorithm 2 For optimization, we use the margin- based triplet loss function between the predicted match score and the true match score. Let L be the loss, we have:
W = W D W W"
L = Σ max (0, (ZEogg 8) - fI u) + M),
(6)
where f(..) represents the score function defined in Eq. (4) computed from the cross-attention map with image x. and text y, i denotes the sample index, Tpos is the groundtruth image corresponding to the i-th text. y, Tneg is the negative image, and M is a predefined margin where use 0.2 in our experiments.
Dynamic Attention Head Weighting We propose a method for adjusting the weights of different attention heads in the cross-attention of our model. We compute the gradient of the output of each attention head with respect to the input, and use this gradient to weight the contribution of each head to the final
6

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm Discflusion Training
Image, y: Text
Noisc
Latent representation, zy: Noisy latent representation
4: E: Encoder
5: т: Domain-specific encoder
6: yo: Intermediate representation of the U-Net
for (x, y) the batch do
Image latent representation t 8(x)
Noisy image latent representation 2/
Eq.
10:
Text latent representation ry + (()
11:
Intermediate representation
+- (p())
12:
Compute attention maps A < TT,
13:
Update attention maps with attributed attention maps A < Attr Compute matching score (A)
Eq
14:
Eq.
15:
Compute loss f- y
Eq.
16:
Update Wk' W W W
Eq.
17: end for
A Algorithm Diseffusion Inference
1: 0: Image, y: Text.
2: E: Noise
3: 2: Latent representation, z;: Noisy latent representation
E: Encoder
Domain-specific encoder
yo: Intermediate representation of the U-Net
7: for (z, in the batch do
8: Image latent representation 2 + E(x)
9: Noisy image latent representation 2t < 2,
Eq.
3
10: Text latent representation ry + T(y)
11: Intermediate representation Fg f ((z))
12:
Compute attention maps A Typ Fs
13:
Compute matching score fAA
Eq.
14: end for
output of the model follows:
H 0f(A)
Attra(A) Ad ①
Σ
(7)
km aA₁
where @ is element-wiet multiplication, A, E Raxm denotes the h-th head's attention weight matrix, and (f(4) computes the gradient along A₁. The (i.j)-th element of (ttr,(A) computes the interaction in terms BA,, of the 1-th allention head. The gradient weights are detached and utilized solely scalar multipliers during training. During training, we then update the attention map with Attry(A), By adjusting these weights, we are able to estimate the gradient flow, keep the most essential token alignment, guide the matching process between text and image and achieve better performance in our task. The inference time of Discffusion is introduced in the next section.
4.6 Inference
During inference, Discffusion leverages the latent representations of both images and text to compute a matching score that determines how well the image corresponds to the text prompt, as outlied in Algorithm P Specifically, for each pair of image x and text y in the batch, Discffusion first obtains the noisy latent representation 2, Then, the text prompt y is encoded into its latent representation ry using the domain-
7

---

Published in Transactions on Machine Learning Research (09/2024)
specific encoder The U-Net is used to obtain the intermediate image representation ro from the noisy latent, representation The attention maps A are computed using the latent representations ry and Tr- Finally, the matching score fAA is computed from the attention maps from different sampling timesteps, where Discffusion can adopt various sampling strategies designed for diffusion-baset generative models [ This process integrates both high-level and low-level features from the image and lext during the diffusion process, effectively leveraging the strong representation of the diffusion model to perform image-text matching.
5 Experiments
5.1
Datasets
We use the Compositional Visual Genome (ComVG) (Krishna et al. 2017) and RefCOCOg Yu et al. 2016 datasets to do image-texl matching, which requires model's ability to understand fine-grained details, spatial relationships, and compositionality of image and text pairs. Additionally, we include the VQAv2 dataset, which we adapted for image-text matching by concatenating questions with their corresponding answers form unique text prompts for matching with images, to demonstrate the versatility and robustness of Diseffusion across different vision and language tasks. Apart from these, Winoground (Thrush et al. 2022) and VL-checklist (Zhao et al. 2022 are also included for evaluating the understanding ability of compositionality.
Compositional Visual Genome (ComVG) (Krishna et al. 2017) is a reconstructed dataset of the Visual Genome (Krishna et al. 2017) dataset, which contains 108,007 images annotated with 2.3 million relationships. These relationships are reprosented as and spatial relationships. ComVG was created by selecting a subset of 542 images from Visual Genome that. contain clear relationships, resulting in a total of 5400 data points. mainly contains three subcategories: subject, predicate, and object. We perform evaluation on the entire datasct well on the 'Subject', 'Predicate', and 'Object' subeategories, respectively.
triplets and include both action
RefCOCOg ( Yu et al., 2016) is a reconstructed dataset of the MS-COCO (Lin et al. 2014) dataset, including 85,474 referring expressions for 54,822 objects in 26,711 images, with a focus on images containing between 2 and 4 objects of the same category.
VQAv2 (Goyal et al., 2017) The VQAv2 dataset (Goyal et al., 2017 contains questions such as 'Binary', 'Other', 'Numbers', it commonly converted a classification task with 3,129 answer classes with frequency large than 9. In our setting, modify the candidate text to be the concatenation of question and answer pair for each question and perform matching with images. We perform evaluation on all the modified dataset. as well as on the 'Binary' and 'Other' subcategories, respectively.
5.2 Experimental Setup
We use Stable Diffusion v2.1-base with the xFormer (Lefaudeux et al. 2022 and flash attention (Dao et al. 2022] implementation, which utilizes the LAION (Schuhmann et al. 2022) datasct for pre-training. On the RefCOCOg dataset, we sample 10 text prompts from the pool each time, and the model is asked to do the correct matching given the image and the 10 sampled text prompts. We first evaluate our method under the zero-shot setting and select the variant with the best performance (using attention maps with dynamic attention head weighting, averaged across all U-Net layers, and using the LogSumExp, see the ablation studies section for details). We then test Discffusion under the setting where we train the model with only 5% of the datasct (Yoo et al. 2021 demonstrating its adaptation capability using limited data.
5.3 Baselines
We mainly compare Discffusion with several baseline models, including CLIP, BLIP2, and Diffusion Classifier. For CLIP, we use the CLIP-ViT/HH11 model with the OpenCLIP variant as the backbone for the fair comparison.
By default the paper, we use the DDIM (ong al. 2020 sampling strategy.
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Comparison of accuracy (%) under the fine-tuning setting with 5% training data and zero-shot sctting (Average of three runs). Discffusion can beat CLIP and Diffusion Classificr by large margin consistently across all these datasets under the fine-tuning setting, demonstrating the superiority of our approach compared with traditional vision and language models pre-trained for discriminative tasks. Note that OFA's pre-training datasets include Visual Genome (VG), RefCOCOg, and VQAv2, and BLIP2's pre-training datasets also include Visual Genome (VG) and COCO (Lin et al. 2014 (whose images are the sources images for RefCOCOg and VQAv2). Therefore, for a fair comparison, CLIP should be the main baseline to evaluate
the effectiveness of Discffusion.
Compocitional Visual Genome
RefCOCOg
VQAr2
Method
Subjects
Objects
Predicate
All
Top-1 Acc.
Top-5 Acc.
Binary
Other
All
CLIP
78.87
80.59
58.60
74.20
68.10
82.79
66.31
4.90
17.55
BLIP2
80,96
81.07
61.14
80.65
75.28
89.20
67.73
6.73
20.49
Zero-shot
OFA*
80,79
82.49
60.89
76.28
72.05
88.52
66.01
6.37
19.57
Diffusion Classifier
79.85
82.41
63.28
77.05
73.02
88.60
67.10
5.95
19.64
Diseffusion
80.62
81.74
63.27
78,11
73.07
89.16
67.51
6.82
20.43
CLIP (Fine-tuning)
80,77
82.49
60.50
76.10
69.88
84.57
66.94
5.10
17.86
CLIP (Prompt Learning)
78.88
79.51
60.41
74.24
69.40
84.48
67.32
5.16
18.03
BLIP2*
81.12
84.23
61.30
80.81
76.43
90.35
71.23
7.23
20.99
Fine-tuned
OFA*
80.80
82.50
60.90
76.29
73:31
80.78
66.45
6.81
20.01
Diffusion Classifier
80.48
82.52
63.42
77.50
73.40
89.83
68.94
6.78
19.88
Discffusion
80.78
84.90
63.13
78.27
75.87
89.96
70.60
20.52
Diseffusion (w/ pre-training)
80.98
84.97
65.17
76.91
71.04
20.83
CLIP (Fine-tuning) Radford et al. 2021 : We fine-tune the CLIP model, adjusting only the last
layer.
CLIP (Prompt Learning) Zhou et al. 2022): Inspired by prompt learning strategies, we incorporate learnable prompts textual inputs conditioned on individual images.
BLIP2 (Li et al. 2023b): A vision-language pre-training model that bridges the modality gap using lightweight Querying Transformer.
Diffusion Classifier [Li et al. 2023a Extracts standard classifiers from class- condiconnai diffusion
models by choosing the conditioning that best predicts the noise added to the input image.
5.4 Experiments: Comparison under the Fine-tuned Setting
In order to facilitate a fair comparison, we adapt the use of the Discffusion model two distinct settings ComVG, given the different resolutions of Stable Diffusion and CLIP. The first setting involves resizing all images in the dataset to resolution of 224x224 first, and then upsampling and resizing to 512x512 for the use of Discffusion, so that Discffusion will not take advantage of its higher input resolution. The results are shown in Table [ The other setting involves directly resizing all images to 224x224 and 512x512 base resolution, which is employed by CLIP and Stable Diffusion, with the results shown in the Appendix.
We present a comprehensive comparison with other state-of-the-art methods: BLIP2 (Li et al., 2023b) ) and OFA (base) (Wang al. 2022), specifically under the few-shot scenario using only 5% of the data in Table | Our results show that our method is competitive, trailing BLIP2 closely, which is the state-of-the-art across numerous vision and language tasks. It's worth noting that OFA's 's pre-training datasets include Visual Genome (VG), RefCOCOg, and VQAv2, and BLIP2's pre-training datascts also include Visual Genome (VG) and COCO (Lin et al.) 2014) (whose images are the sources images for ReICOCOg and VQAv2). Therefore, for a fair comparison, CLIP should be the baseline to evaluate the effectiveness of our approach for few-shot lcarning. As shown in Table Discffusion can beat CLIP consistently across all these datascts. In addition, with pre-training on the corresponding datasets, Discffusion outperforms all other methods, including the state-of-the-art BLIP2 on Compositional Visual Genome, RefCOCOg, and the 'Other' category VQAv2.

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Comparison of accuracy (%) on Winoground and VT.-checklist across three runs for CLIP, BLIP2, and Discffusion, both zero-shot and fine-tuned on MS-COCO. For BLIP2's evaluation on Winoground, is approached an image-text matching lask to ensure fair comparison, rather than prompling the model to answer the rewritten question from captions (Yarom et al. 2024 BLIP2 gains little from fine-tuning MS-COCO because BLIP2's pre-training datasets already include MS-COCO (Lin et al. 2014), and Winoground is quite different from MS-COCO as it requires compositional understanding ability. Similarly, Diffusion Classifier, designed as a zero-shot approach, does not benefit much from fine-tuning. As can be seen, under the fine-tuned setting, Discffusion outperforms all other approaches by a large margin across both datasets in each category. The detailed results on Winoground with 95% confidence intervals across different categories are presented in Table 10
Winoground
VL-checklist
Method
Text
Image
Group
Attribute
Object
Relation
Average
CLIP
30.68
11.91
8.36
67.82
75.67
67.11
70.18
BLIP2
31.78
12.37
9.02
79.00
84.05
73.55
78.86
Zero-shot
Diffusion Classifier
37.75
12.25
9.00
64.19
75.98
66.99
68.23
Discffusion (Ours)
34.11
13.38
11.04
73.12
79.58
69.48
74.06
CLIP
31.50
12.00
9.25
67.90
75.75
68.15
70.73
MS-COCO
BLIP2
28.25
13.00
9.09
79.64
84.70
73.95
79.97
Fine-tuned
Diffusion Classifier
26.76
10.00
8.75
65.90
76.81
67.14
68.94
Discffusion (Ours)
35.75
14.50
12.75
79.80
85.90
74.58
80.52
5.5 Experiments: Zero-shot Transfer from Training on the MS-COCO Dataset
We assess the zero-shot transfer capabilities of the Discffusion model and its comparison with notable baselines such as CLIP, BLIP2, Diffusion Classifier (Li et al. 2023a We fine-tuned all these models on the MS-COCO dataset, and then performed the evaluation on downstream datasets. Given that BLIP2's pre-training datasets include Visual Genome (VG) and COCO (Lin et al. (014)), and considering the overlap of MS-COCO images with those in ComVG and RefCOCOg, we selected two commonly used evaluation benchmarks: Winoground (Thrush et al. 2022 and VL-chocklist (Zhao et al. 2022). The evaluation results are depicted in Table 2 Discffusion outperforms all others Winoground and VL-checklist, showcasing its superior ability to generalize the learned visual-linguistic representations to datasets beyond the ones used during fine-tuning.
5.6 Experiments: Zero-shot Evaluation on the Winoground and VL-checklist Dataset
We conducted additional experiments to evaluate the potential of Discffusion for zero-shot inference with 50 sampling steps on the Winoground (Thushh et al. 2022) and VT--cheklistt datasets, which require compositional understanding ability. The top Table 2 shows a comparison of Discffusion with other zero-shot approaches. Discffusion outperforms CLIP and competitive with Diffusion Classifier under the zero-shot setting on both the two datasets, even though Discffusion is primarily designed as a fine-tuned method. The detailed performance across categories <relation, object, both in the format (text score, image score, group score) is shown in Table 10 in the Appendix. As shown, Discffusion excels notably in the 'relation' calegory, demonstrating proficiency in discerning shuffled elements like verbs, adjectives, prepositions, and pre-training and our adaptation
adverbs. This suggests that Discffusion, with its wewpaative-ta---rriittte strategy, offors enhanced capability for handling fino-grained nuances, spatial interrelationship, and intricate composilionallttie compared to conventional discriminative vision and language models.
5.7 Discussion about Inference Efficiency of Discffusion
The inference memory cost of Discffusion is approximately 9.8GB when using the batch size of one. We use DDIM sampling get the sampling timesteps. The inference speed heavily dependent on the number of sampling timesteps. We demonstrate the trade-off between the number of sampling timesteps and the
10

---

Published in Transactions on Machine Learning Research (09/2024)
75.4
90.0
Average selected hea (zero-shot)
87.5
75.2
Accuracy (%)
85.0
Accuracy (%)
Dynamic Attention Head Weighting (few-shot)
75.0
K2.5
Average across selected heads (fewishot)
80.0
77.5
SD 2.1-v (768x768)
75.0
74.6
SD 2.1 1-base (51 (512x512)
72.5
2
16
70.0
Number of Layers from U-Net
Number of Heads
Figure 3: Ablation study on the number of attention maps used from layers of the U-Net (x-axis). The y-axis represents the accuracy. Tests two variants of Stable-Diffusion v2: trained as standard noise- prediction model on 512x512 and 768x768 images. The results demonstrate that using attention maps from all layers yields the best performance, showing the importance of using attention maps from
Figure 4: Ablation study the mumber of attention heads (five in total within the Stable Diffusion) in the U-Net (x-axis) with few-shot performance on the ComVG dataset (y-axis) under the two scenarios: using the average all attention maps and using our dynamic attention head weighting method. The results illustrate the superiority of our weighting method.
layers.
Table 3: Ablation study on the ComVG dataset across three runs using cosine similarity, maximum value from each column of the attention map, and the smoothed maximum (LogSumExp pooling); and the amount of noise added during the diffusion process: using consistent noise levels of 0.4, 0.8, and using ensembling. The accuracy numbers are in (%).
Method/Noise Level
Subjects
Objects
Predicate
All
Cosine similarity
50.7
42.5
30.5
43.0
Maximum (column-wise)
58.8
59.5
40.4
57.7
Method
Maximum (row-wise)
57.8
59.0
40.1
57.0
LogSumExp
78.9
80.1
61.2
75.4
Noise level 0.4
75.5
77.5
58.4
72.5
Noise Level
Noise level 0.8
75.7
77.7
58.5
72.7
Ensembling
76.1
78.0
59.2
73.1
inference time on the RefCOCOg dataset with a batch size 16. The inference was executed in a distributed manner on a NVIDIA workstation equipped with A6000 GPUs, leveraging the capabilities of the Accelerate library 2 as shown in Figure We observe that the Top-1 Accuracy tends stabilize when the number of sampling timesteps approaches approximately 100.
5.8 Ablation Studies
In this section, we delve deeper into the nuances of our experimental findings in zero-shot setting the ComVG dataset. ComVG's complexity allows us to examine the finer-graines cross-modal alignment at the entity level, showcasing how each design aspect of our method impacts the compositional matching of discntanglod concepts.
Effect of Attention Maps from Different Sets of U-Net Layers This study examines the effect of different layer configurations in the U-Net architecture of Stable Diffusion v2 on the computation of attention maps. We use two variants of Stable Diffusion v2 and take the average of the attention maps from different.
11

---

Published in Transactions on Machine Learning Research (09/2024)
78.27%
78.13%
8.30h
7-
77.81%
78.0
Infercnce (hours)
tep Accur acy (96
6-
76.98%
77.5
5.94h
5-
Inference pine
Top-1 Accuracy
77.0
4-
4.15h
76.5
2-
75.87%
2.53h
0.55
10
70
100
Number of Sampled Time Steps
Figure 5: Inference time and accuracy versus sampling steps on ReiCOCOg with Discffusion, conducted on a GPU setup.
layers of U-Net. Following insights from Kwon et al. (002 Tang et al. (2022) which suggest that deeper layers capture more semantically relevant. information, we consider the last one; the last two, the last eight, and all layers. The results, shown in Figure g demonstrate that incorporating all layers yiclds the highest accuracy, indicate that comprehensive integration of both high-level and low-level features is crucial for tasks like image-text matching, maintaining both coarse and detailed aspects of image representation. Interestingly, focusing solely on the last two layers also results in performance closely mirroring that of using just the final layer, indicating a potential redundancy in the task-specific information contained within these later stages.
Cosine Similarity us. Maximum es. LogSumExp Pooling for Score Computation We compare the overall accuracy of using Cosine Similarity, Maximum value from the attention map, and LogSumExp Pooling for score computation in Table 3 For cosine similarity, the dimensionality of the latent representations for images and texts differs. Therefore, we employ projection layers to project the representations from both modalities into common dimensionality. For the maximum value method, we evaluate two variants: (i) extracting the maximum value from each column of the attention map (column-wise), and (ii) extracting the maximum value from each row of the attention map (row-wise). The results indicate that LogSumExp Pooling achieves the highest accuracy, followed by the maximum value method, with Cosine Similarity yielding the lowest performance. This may be because LogSumExp can smooth out the influonce of individual noisy elements, resulting in more robust and accurate matching scores. Additionally, LogSumExp can effectively capture the overall importance of each element in the attention map, rather than relying solely on the maximum value. Overall, these results highlight the effectiveness of LogSumExp a method for computing the matching score in image-text matching tasks.
Effect of Dynamic Attention Head Weighting We investigale the elfect of using varying numbers of attention heads in Stable Diffusion on image-text matching performance, specifically, the attention heads in the cross-attention layer of the spatial transformer. To do this, we randomly sample a subset of heads and leave the rest unused. The results, as shown in Figure 4 indicate that using all heads (five in total in our evaluated model version) from the Spatial Transformer (Jaderberg et al.) 2015)) located in the U-Net of Stable Diffusion performs the best. Additionally, we compare the results of using dynamic attention head weighting versus averaging across selected heads (few-shot) to evaluate the impact of attention head selection performance, and dynamic attention head weighting performs better.
Ensembling over Noise Levels In diffusion modcls, the level of noise controls the variance of the Gaussian noise added during the diffusion process, which can affect the degree of change in the generated image. Our study, illustrated in Table 3, explores the effects of varying noise levels in the diffusion process. To further improve the performance of our mothod, we use an ensemble technique inspired by Wolleb al. (2022) by averaging the scores over four relative different noise levels: f0.2, 0.4, 0.6, 0.8}. This is done by first obtaining the score under each noise level scenario and then averaging them. The results of this comparison
12

---

Published in Transactions on Machine Learning Research (09/2024)
are shown in Table 3 Our experimental results demonstrate that this ensemble technique leads to noticeable improvement in performance.
6 Conclusion
In this paper, we proposed a method for matching text and images in the latent space of Stable Diffusion. We fine-tuned the U-Net part of the model by focusing on the cross-allention between lext. features and latent image features, which reflects the alignment between text and image. Our results show that this finc-tuning approach improves the alignment between text and image, leading to better performance in image-lext matching lasks. Overall, our approach is pioneering work that leverages the inherent flexibility of diffusion-based visual generative models, demonstrating improved explainability and an enhanced ability to capture fine-grained details, spatial relationships, and compositionality. Our results can motivate research on simpler alternatives to adapt Stable Diffusion models, as well as on future methods for better utilization of them.
Broader Impact Statement
The Discriminative Stable Diffusion (Discffusion) approach proposed in this paper dependent on a pre- trained Stable Diffusion model, which may be challenging to obtaín in certain scenarios where the model has yet to be publicly released or where the computational resources required for training not available. Additionally, the quality of the pre-training the need for further research to investigate methods for improving the pre-training process. There are other tochniques such as multi-task learning and meta-learning that can be possibly incorporated to improve the performance of DiscITusion. It worth noting that real-world scemarios, there often a limited amount of labeled data, available, and collecting more data can be costly and time-consuming. Therefore, the ability of Discffusion to perform well under few-shot setting an important aspect of its potential utility in practical applications.
greatly impact the performance of Disclfusion, highlighting
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffuusio models. arXio preprint arXim:2112 11N:0112003120 2021.
Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXim:1801.01975 2018.
Pierre Blanchard, Desmond J Higham, and Nicholas J Higham. Accurately computing the log-sum-exp and softmax functions. IMA Journal of Numerical Analysis, 41(4):2311 2330, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 1461122005.4465, 2020.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020.
Kevin Clark and Priyank Jaini. Texl-lo-inage diffusson models are zero-shot classifiers. arXio preprinl arXiv:2303.15233, 2023.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Mohammad Mahdi Derakhshani, Menglin Xia, Harkirat Behl, Cees GM Snoek, and Victor Rühle. Unlocking spatial comprehension in text-to-image diffusion models. arXiv preprint Xix:23311 1. 17937, 2023.
Weixi Feng, Xuehai He, Juu-Jui Fu, Varun Jampani, Arjun Akula, Pradyunna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional lext-to-image synthesis. Xie preprint urx:v22212 05032, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672 2680, 2014.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh, Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint Xxxx22210.07574 2022a.
Xuehai He, Diji Yang, Weixi Feng, TsuJJui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. Cpl: Counterfactual prompt learning for vision and language models. arXiv preprint arXiv:2210. 10362, 2022b.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208. 01t22200000666, 2022a.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2008. 01626, 2022b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NearIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106. 09685 fes), October 2021.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651-4664. PMI.R, 2021.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models data source for multiview representation learning. arXiv preprint arXiv:2166. 05258, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text. supervision. In International Conference on Machine Learning, pp. 4904-491 PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint ar Nto:2203.12110 2022.
Kenan Jiang, Xuchai He, Ruize Xu, and Xin Eric Wang. Comelip: Training- free compositional image and text matching. arXiv preprint ar Xiv:22211. 19854, 2022.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the TEEE conference on computer vision and pollern recognilion, pp.
2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint ar xiw::13122.6114 2013.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connocting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32 73, 2017.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customiza- tion of text-to-image diffusion. orXiv preprinl ar Nrxiv:2212 84442222.044888 2022.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXit:2910.10900 2022.
Kuang- Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 201-216, 2018.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and 2022.
hackable transformer modelling library. https://githe wttp///fcccbboeeeearraaaatteeere
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parametter-effiieel prompt luning. arXiv preprint orXiv:2104.08691, 2021.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. arXic preprint arx::23033. 11299933..2200 16203, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023b.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint vrxii:1900 1eer.03085557 2019.
Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. a7X1v:2101.00190 [cs), January 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingvang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. areie preprint arXiv:2110.00m 10.07602 2021.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXim:2310.044 04978, 2023.
Naoki Matsunaga, Masato Ishii, Akio Hayakawa, Kenji Suzuki, and Takuya Narihira. Fine-grained image editing by pixel-wise guidance using diffusion models. arxiv preprint 2tt.:2222002022,, 2022.
Andrew Ng and Michael Jordan. On discriminative generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. Xiv preprint or Xin221 222.211111122 10741, 2021.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00000 2021.
Sai Saketh Rambhatla and Ishan Misra. Selfeval: Leveraging the discriminative nature of generative models for evaluation. aixe preprinl arXiu:2311.10708, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.060 061225, 2022.
Marc' Aurelio Ranzato, Joshua Susskind, Volodymyr Mnih, and Geoffrey Hinton. On deep generative models with applications to recognition. In CVPR 2011, pp. 2857-2864. IEEE, 2011.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-000000 10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022b.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. pp. 234-241, 2015a.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234 241. Springer, 2015b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. wrXiv:2205.11487 2022.
Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676, 2020.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Thoo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open largo-scale dataset for training next generation image-text models. arXie preprinl arXiv:2210.08// 08402, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thernodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015a.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. Tn ICML, 2015b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ar arXiv:2010.0250 2020.
arXiv preprint
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NewrIPS, 2019.
Raphael Tang, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Jimmy Lin, and Ferhan Ture. What the daam: Interpreting stable diffusion using cross attention. arXiv preprint arXiv:2210.0448 2022.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for lisiu-iin compositionality. In Proceedings of the IRFF/CVF Conference on Computer Vision and Patlern Recognition, pp. 5238 5248, 2022.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Newral Information Processing Systems, 34: 200 212, 2021.
Aaron Van DenOord, Oriol Vinyals, and etal. Neural discrete representation learning. volume 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-lo--sqqeeee learning framework. CoRR,
2022.
Chen Wei, Karttikeya Mangalam, Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. arXiv preprint arXiv:2304.0322 2023.
Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin. Diffusion models for implicit image segmentation ensembles. In International Conference on Medical Imaging with Deep Learning, pp. 1336-1348, PMLR, 2022.
Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang. Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7766-7776, 2023.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-imase pre-training, orXiv preprint arXiv:2111.0700 2021.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roce Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. Advances in Neural Information Processing Systems, 36, 2024.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Woo Lee, and Woomyeong Park. Gpt3mix: Leveraging large-scale language models for text augmentation. arXiv preprint 100:2210400886, 2021.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference Computer Vision, pp. 85. Springer, 2016.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new loundation model for computer vision. arXiv preprint xrXiv22111.1142, 2021.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Fraacoy, Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10145-10155, 2021.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprial XrX:::2207.0022 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. arXiv preprint XrX::220330055 05557, 2022.
17

---

Published in Transactions on Machine Learning Research (09/2024)
A Ablations on Input Images Resolutions
In our experimental setup, CLIP accepts image resolutions of 22Ax224 and Stable Diffusion accepts 512x512. Considering that image resolutions within the ComVG datasct vary, we adopted two distinct resizing strategies to maintain balanced comparison. In the first approach, initially resize all inpul images to 224x224 resolution, followed by subsequent resizing to 512x512 specifically for Discffusion. In this way, there is no additional information introduced when comparing with using CLIP. In the second approach, we directly resize the input image resolutions to 512x512 and 224x224 for Discffusion and CLIP respectively. The results of these strategies are illustrated in Table | It evident that Discffusion surpasses both the fine-tuned CL.IP and the CLIP with prompt learning basclines by large margin. These outcomes suggest that the Discffusion model can adeptly harness the benefits of larger input image resolutions.
B Training Efficiency of Discffusion
The training efficiency of the Discriminative Stable Diffusion (Discffusion) model is noteworthy. Remarkably, it requires only single NVIDIA V100 GPU for training, with memory footprint of approximately 15.4 GB with the Accelerate library and FlashAttention (Dao et al. 2022 using the implementation available in HuggingFace Diffusers + The model demonstrates efficient training times across different datasets: it completes training on the Compositional Visual Genome (ConVG) datasct 7 hours, on the RefCOCOg dataset in 11 hours, and on the more extensive Visual Question Answering version (VQAv2) dataset in 17 hours. These training durations highlight the model's practicality and feasibility for use in various research and application scenarios, especially considering the moderate computational resources required.
C Adapting Discffusion to Latent Consistency Models (Luo et al., 2023
This section investigates the adaptability of Discriminative Stable Diffusion (Diseffusion) to Latent Consistency Models (LCMs) a class of faster diffusion models that require fewer steps for both training and inference (four steps). We evaluate the performance of Discffusion extended with LCMs two experimental setups: training on the MS-COCO dataset followed by zero-shot testing on Winoground, and few-shot training on ConVG with subsequent lesting on the same.
The empirical outcomes, detailed in Table reveal that Discffusion when augmented with LCMs surpasses the original Discffusion on Winoground. However, it underperforms on the ComVG dataset. Closer examination of the training dynamics for Discffusion (L.Css) on ComVG indicates instability, as depicted by the fluctuating loss curve in Figure ( This instability may due to the small number of sampling steps and account for the diminished results on ComVG. Nonetheless, the preliminary success on Winoground suggests that, with further refinement and larger dataset fine-tuning, LCMs could enhance the capabilities of Disctfusion. This finding demonstrates the potential of Discffusion to excel when integrated with more advanced diffusion models, paving the way for future research in this direction.
D Visualization of Learned Prompt Weights
We present a visualization of the prompt weights learned after training on the MS-COCO dataset in Figure 7 The distribution weights is predominantly near zero, showcasing sparsity in the update made to the original Stable Diffusion weights. This sparsity suggests that only minor updates are required for Stable Diffusson to adapt to discriminative tasks effectively, which underscores the model's potential for such applications and strong latent representations gained during pre-training.
18

---

Published in Transactions on Machine Learning Research (09/2024)
train_loss
0.4
0.1
3cep
5k
10k
15k
20k
25k
30k
Figure 6: Training loss curve for Diseffusion with LCMs on the ComVG dataset. The observed fluctuations point to training instability, which potentially impacts performance on the few-shot setting.
-2.0015
00:0010
00,0005
0,0000
0,0005
0,0010
0.0015
Value
Figure 7: Visualization of learned prompt weights after training on MS-COCO.
E Image-text Retrieval Directly Using Discffusion Embeddings
In this section, we study the quality of the representation space learned in Discffusion. We directly using Diseffusion's 's embedding to do retrieval. Specifically, after fine-tuning Discffusion on the Compositional Visual Genome (ComVG) and RefCOCOG dataset, we use the latent space features after the VAE to the retrieval. This process involves using cosine similarity match these features with corresponding text embeddings. We then compare the performance of Diseffusion's embeddings with those from the CLIP model. The results, shown in Table 6 illustrate the capability of Discffusion embeddings for retrieval tasks, underscoring their potential for future rescarch and applications in this area.
F Additional Experimental Results
F.1 Extreme Few-shot Setting
We have expanded our experimentation to extreme few-shot learning by conducting tests with only 0.5% of training data (27 examples from ComVG), with the results shown in Table 7 These results illustrate the benefits of ours with such small data, particularly when compared to CL.IP's performance.
19

---

Published in Transactions on Machine Learning Research (09/2024)
Table Comparison of accuracy (%) on Compositional Visual Genome (ComVG) using CLIP, Discriminative Stable Diffusion under the few-shot sctting with different input image resolutions.
Compositional Visual Genome
Method
Resolution
Subjects
Objects
Predicate
Average
CLIP (Fine-tuning)
224x224
80.77
82.49
60.50
76.10
CLIP (Prompt Learning)
224x224
78.88
79.51
60.41
74.24
Discffusion
224x224
80.81
83.17
63.51
78.11
DiscItusion
512x512
80.97
84.32
63.53
78.19
Table 5: Comparison of accuracy (%) on Winoground and VL-checklist CLIP, BLIP2, and Discffusion
fine-tuned on MS-COCO.
Winoground
ComVG
Method
Text
Image
Group
Subjects
Objects
Predicate
Average
Discffusion
34.00
13.75
10.50
80.78
84.90
63.43 60.08
78.27
Discffusion (LCM)
34.75
15.75
11.00
79.84
83.33
73.47
F.2 Masking Out Objects in RefCOCOg
We conduct additional experiments by masking objects with the provided bounding boxes to observe their impact on the RefCOCOg dataset using 300 images. When elements are masked, the Top-1 accuracy of Discriminative Stable Diffusion (Discffusion) drops significantly from 81.7% 43.6%, as shown in Table S This substantial decrease in accuracy demonstrates that Diseffusion can capture key objects in images for the correct matching.
F.3 Ablation Study on Time Steps Sampling in Discffusion
As presented in Table we explored the effectiveness of using cross-attention maps from different time steps in the Discriminative Stable Diffussin (Diseflusion) compared to variants that employ single time step's cross-attention maps (the sampled first time step and the sampled last time step). The comparative results demonstrate the enhanced performance of the original Discffusion method. Leveraging all time steps' information can effectively integrate and process the coarse-to- to-fine information throughout the diffusion steps to do matching.
F.4 Ablation Study on Loss Functions in Discffusion
Vision and language models trained with contrastive objective may not preserve all the visual infor- mation. In this section, we study the impact of using different loss functions in training Diseffu- sion on image-text text matching performance on the ComVG dataset. We test the cross-entropy loss: LCE - D' (yn log In + (1 - yn) log (1 - 9n)) where N the number of samples the training datasot and yn and In are the true and predicted match scores for the n-th sample, respectively; and the standard denoising diffuuion training objective: DDiifusion = Eu,,i [|kc- cp (z, t) The results, as presented in Table II indicate that Discffusion, employing the original margin- based triplet loss function outperforms the alternative approaches. This superiority is primarily attributable to the original diffusion training loss's inefficacy in adapting Stable Diffusion for discriminative tasks and the relative inadequacy of the cross-entropy loss in managing the nuances of image-text matching tasks.
20

---

Published in Transactions on Machine Learning Research (09/2024)
Table Accuracy comparison (%) between directly using CT.P-embedding and Diseffusion-embeedingg to do retrieval on the Compositional Visual Genome (ComVG) and RefCOCOg datascts.
Compositional Visual Genome
RefCOCOg
Method
Subjects
Objects
Predicate
Average
Top-1 Acc.
Top-5 Acc.
CI.I-embeddiing
80.77
82.49 76.25
60.50
76.10
69.88
84.57
Discffusion-embeedding
78.43
57.68
73.27
66.14
84.61
Table 7: Comparison of Few-shot and Extreme Few-shot Learning in Compositional Visual Genome.
Few-shot (ComVG)
Extreme Few-shot (ComVG)
Method
Subjects
Objects
Predicate
Average
Subjects
Objects
Predicate
Average
CLIP (Fine-tuning)
80.77
82.49
60.50
76.10
74.40
75.10
57.90
69.80
CL.IP (Prompt Learning)
78.88
79.51
60.41
74.24
74.55
75.15
57.79
69.83
Discffusion (Ours)
80.78
84.90
63.43
78.27
78.65
81.40
66.68
75.91
F.5
Additional Zero-shot results on the Winoground dataset
In Table 10 we show additional results on the Winoground dataset in accuracy with 95% confidence intervals across the three different categories. Discffusion excels notably in the 'relation' category, demonstrating proficiency in discerning shuffled elements like verbs, adjectives, prepositions, and adverbs.
G Results on Text-to-lmage Generation
In addition to the existing evaluations in the main paper, we also evaluate the capability of fine-tuned Discffusion model on the task of text-to-image generation. We use two scores evaluate the quality of generated images:
Frechet Inception Distance (FID) (Heusel et al., 2017) measures the similarity between two sets of images. It considers both the quality and diversity of the generated images. Tt. calculates the distance between the distributions of the real and generated images in the feature space of a pre-trained Inception model. A lower FID score suggests a higher similarity to the real image dataset, implying better image quality and variety.
Inception Score (IS) (Barratt & Sharma 2018), on the other hand, was proposed to evaluate the quality and diversity of images generated by GANs. It leverages the Inception model, originally used for image classification, to score generated images. The Inception Score is calculated based the assumption that good models should generate diverse images (high entropy) but still be able to classify these images correctly (low conditional entropy). Hence, a higher IS indicates better performance of the generative model.
We detail the performance of these metrics on both the MS-COCO (Lin et al. 2014 and Compositional Visual Genome (ComVG) datasets in Table 12 The results show that the Discriminative Stable Diffusion (Discffusion) model, when finc-tuned a few-shot setting for discriminative tasks on the ConVG datasct does not compromise the generative capability of the model, which remains on par with the performance of the original Stable Diffusion model.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Table 8: Comparison of Top-1 and Top-5 accuracy (%) on RefCOCOg using Discriminative Stable Diffusion (Discffusion) under the few-shot setting, and Discriminative Stable Diffusion (Discffusion) with the impact of masking elements. This table highlights the effect of element masking on the Discffusion approach in the RefCOCOg dataset.
Method
Top-1 Acc.
Top-5 Acc.
Discffusion
81.74
91.96
Discffusion (Masking Elements)
43.60
54.78
Table 9: Comparison of accuracy (%) on Compositional Visual Genome (ComVG). The table shows the effectiveness of utilizing cross-attention maps from different time steps compared to using only single time step's 's cross-attention map.
Method
Subjects
Objects
Prodicate
Averago
Discffusion
80.78
84.90
63.43
78.27
Diseffusion (First Time Step Sampling)
59.93
54.24
51.68
55.45
Discffusion (Last Time Step Sampling)
76.78
80.90
59.43
74.27
H Qualitative Results
H.1
Visualization of Generate Samples with Discffusion
In this section, study the generative abilities after adapting Stable Diffusion a discriminative one. We present visualization of generated samples from the Visual Genome dataset with both Discffusion and Stable Diffusion in Figure The image quality of Discffusion is comparable to Stable Diffusion.
H.2 Visualization of Cross-attention Maps
We present visualizations of averaged cross-attention maps prior to the integration of the Iearned prompt in Figure g These representations elucidate the Discffusion model's reasoning strategy for ground-truth image-text pairs. During the diffusion process, the correspondence between the text and image is reflected token-level scores on the attention map. Tokens receiving higher scores become predominant decision-making process for matching. It clear from the visualizations that the regions of the attention maps imbued with informative elements tend correspond with higher scores. The higher attention implies stronger similarity between token in the given text and elements of the image.
the
Error Analysis
Here, present example of failure rllustrated in Figure T0 where the Disclfusion model failed to successfully discriminate. As can be observed, Discffusion struggles to recognize the word 'white' in the image, leading to incorrect matching. This could potentially be attributed to the lack of ability to understand compositionality inherent the original stable diffusion model.
the Compositional Visual Genome (ComVG) dataset,
J Hyper-parameter Settings
In this section, we will introduce the detailed
settings in the paper.
Datasets: We used the Compositional Visual Genome and RefCOCOg datasets for the image-text matching fask, where we used the split. introduced in Jiang al. (2022) and Yu et al. (2016).
22

---

Published in Transactions on Machine Learning Research (09/2024)
Stable Diffusion
Discffusion
A man catching the football
A man is catching a dog
woman eating the food
A dog is smiling
A commercial stainless kitchen with pot of food cooking
Dark brown horse with long straight black tail standing up in a field
Figure 8: The visualizations of generated samples from the Visual Genome dataset using the prompts man is catching the football, A man is catching a dog, A woman is eating the food, dog is smiling, A commercial stainless kitchen with a pot of food cooking, and Dark brown horse with long straight black tail standing up The left two columns are samples generated by the original Stable Diffusion. The right two columns are samples generated by Discffusion. The image quality of Discffusion is comparable to Stable Diffusion, suggesting that the few-shot fine-tuning discriminative tasks in Discffusion does not hinder text-to-image generation performance.
field, employing DDIM sampler with 50 steps.
23

---

Published in Transactions on Machine Learning Research (09/2024)
Table 10: Zero-shot results on the Winoground dataset in accuracy with 95% confidence intervals. Discffusion excels notably in the 'relation' category, demonstrating proficiency in discerning shuffled clements like verbs, adjectives, prepositions, and adverbs.
Relation
Object
Both
Category
Text
Image
Group
Text
Image
Group
Text
Image
Group
CLIP
0.22_ _o.02
0.08 1.00
1.05
0.36
0.12_o.zz o.02
0.081o.oz
0.81 10.00
0.451o.a
0.42 10.04
BLIP2
0.22..o.1
0.05
1.0
0.35
0.20 10.00
0.1310.0
0.621o.0
0.37
0.37
Discffusion
0.25
0.10
0.07
0.40
0.13=0.01
0.1220.02
0.80
0.4620.02
0.42
Table 11: Comparison of accuracy (%) on Compositional Visual Genome (ComVG) trained with different loss functions.
Method
Subjects
Objects
Predicate
Average
Discffusion
80.78
84.90
63.43
78.27
Diseffusion
Loss)
75.77
81.42
60.05
72.01
Diseffusion (Diffusion Loss)
65.58
59.83
54.42
60.96
Model architecture: We used the Stable Diffusion v2.1 model with a U-Net architecture and VAE, with resolutions of 768x768 and 512x512, and CLIP-VIT/H as the text encoder. The sampling was carried out using the DDIM (Son el al. 2020) method with total of 50 steps.
Pre-training: The model was pre-trained from scratch using the LAION-5B (Schuhmann et al. 2022 dataset. The U-Net architecture consists of 16 layers with input blocks, middle blocks, and output blocks. The input blocks contain 6-layer spatial transformer, the middle block contains 1-layer spatial transformer, and the output block contains 9-layer spatial transformer. Each spatial transformer contains two attention layers, one self-attention layer and one cross-attention layer.
Evaluation metrics: We evaluated the performance of our approach using the Acc@ metric, with (=11,5), measuring the accuracy of the correct match of image-text pairs.
Ablations: In the ablation study on the number of attention heads used, we experimented with dynamic attention head weighting. To obtain the weights, we first trained the model and used the learned weights the subsequent inference time. For the other two scttings, where we averaged the attention head weights, used consistently averaged weights across all the experiments. This allowed to compare the effect of the number of attention heads on the final performance of the model, while controlling for the potential confounding factor of different attention head weights; In the ablation study of using cosine similarity, the dimension of the latent representation of images and texts are not the same, we compute the cosine similarity on the shared dimensions between the image and text representation. For the experiments the maximum value, we take the maximum value from each column of the attention map; In the ablation study on performing ensembling over noise levels, we set the noise level to (0.2, 0.4, 0.6, 0.8} and obtain the prediction under each scenario. We then average the predictions to obtain the final score.
using
24

---

Published in Transactions on Machine Learning Research (09/2024)
Table 12: Comparison of Frechet Inception Distance (FID) and Inception Score (IS) for Text-Condital
Image Synthesis the RefCOCOg and Compositional Visual Genome (ComVG) datasets. We use DDIM sampler and the total number of sampling steps is 50.
Text-Conditional Image Synthesis
Method
MS-COCO
ComVG
FID 4
IS 1
FID
IS f
Stable Diffusson
23.3
19.7
26.9
17.5
Discffusion
24.1
19.5
26.0
18.1
5-
15
20
Figure The visualization averaged attention maps in Discffusion for the prompt: table with plate of food, glass of water, and napkin.
A man in front of large hill wearing skis and ho holding poles.
Discriminative Stable Diffusion
Matching
Score:
0.705
Discriminative Stable Diffusion
Matching Score: X
A roowboarder in white pants and a b black jacket.
0.731
Figure 10: The failure case in Discffusion for the prompt: A man in front of a large hill wearing skis and holding poles.
25


Reviews:
[{"review_id": "s50BNOBJdG", "reviewer": "Reviewer_NNvx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The proposed method demonstrates impressive performance across various benchmark datasets, highlighting its effectiveness.\n\n2. The methodology is well-explained and straightforward, making it easy to understand and implement.\n\nWeakness:\n\n1. The ablation study lacks thoroughness, which leaves some aspects of the method's performance unexplored.\n\n2.  The paper does not clearly define or compare its results against established baselines. Some of the baselines are not discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel method, Discriminative Stable Diffusion (Discffusion), which repurposes pre-trained text-to-image diffusion models for few-shot discriminative tasks, specifically image-text matching. The method leverages the cross-attention scores from pretrained diffusion models to evaluate the alignment between visual and textual information. By using additional learnable prompts, the model is fine-tuned to perform image-text matching with limited new data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. While the authors introduce several techniques such as learnable prompts and log-sum-exponential pooling, it is challenging to evaluate how each component contributes to the overall performance. Additionally, the paper does not adequately address how these techniques generalize across different datasets. Moreover, the utility of the zero-shot setting in model selection is unclear.\n\n2. The baseline methods used in this work are not clearly defined. For instance, the type of CLIP model utilized is unspecified. In comparison, OpenCLIP has been shown to provide better results in other studies [1]. It is unclear why OpenCLIP was not used as a baseline.\n\n3. Table 3 shows that well-pretrained models like BLIP already achieve strong performance. It remains unclear how the proposed method can close the performance gap. For example, it would be insightful to understand how the proposed method would perform if it were also pretrained on datasets like RefCOCO.\n\n\n\n[1] Li, Alexander C., et al. \"Your diffusion model is secretly a zero-shot classifier.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "broader_impact_concerns": "Based on the information provided, it seems the authors have already addressed any potential broader impact concerns within their work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gs5f7J0OIM", "reviewer": "Reviewer_2hXW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- creative utilization of the pre-trained diffusion model as a discriminative model.\n- strong experiments across several benchmarks and baselines to evaluate the proposed method.\n- detailed ablation studies to justify all design decisions for the proposed method.\n- Many additional experiments including ablations about image generation abilities and the method combined with latent Consistency models\n\nWeaknesses\n- the role of the noise level as input for the method in the method section is not well explained\n- the proposed method is slow during inference and requires multiple hours, which is a significant drawback compared the the CLIP baselines. This weakness should be discussed concerning the used baselines. \n- some statements regarding classifier guidance are not well explained\n- the conclusion mentions \"a new pathway for innovation where traditional methods fall short\", but it does not discuss which pathways would be opened by the proposed method. Without a concrete example, this sounds exaggerated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- introduce Discffusion, a novel discriminative model that uses the cross-attention scores of a pre-trained stable diffusion generative model for few-shot discrimination of text2image pairs\n- Propose an attention-based prompt learning module to finetune the cross-attention weights of diffusion models for discriminative tasks\n- Several experiments and ablation studies to verify the effectiveness of the proposed method on several benchmarks in few-shot, and zero-shot settings", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Relevant for acceptance:\n- The statement in Section Dynamic Attention Head Weighting about Classifier Guidance is wrong: \"This is done by adding a term to the loss function that is proportional to the gradient of the log probability of the class given the image.\" The Classifier Guidance is not used during training but only during inference in the denoising process, where the gradient guides the denoising process toward regions of high probability given the relevant prompt. This needs to be corrected. The following method is motivated by this statement. Thus, this motivation needs to be revised accordingly.\n- The requirement to use several noise levels is not well explained in the method section of the paper. Algorithm 1 also does not mention noise as input for the proposed method. I was confused, if the model adds noise to the images for the comparison or not during the method description. Only in the ablations the relevance of multiple noise levels for the method is discussed and evaluated. Some additional clarity about this aspect in the method part would be helpful and improve readability.\n- Revise the conclusion with the statement about possible new pathways with concrete examples.\n\nNot relevant for acceptance\n- how is the performance compared to current Sota CLIP models such as SigLiP?\n- How easy is it to generalize the proposed method to Diffusion Transformer architectures? \n- Sometimes the best result in a table is marked with bold text, other times it's not used. This should be consistent for bette readability.", "broader_impact_concerns": "Broader impact statement in the paper is sufficient and I do not have any other concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "33N6dtvonX", "reviewer": "Reviewer_yyhJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- Method is novel and tries to take advantage of the benefits of a pretrained text-to-image diffusion model. \n- Careful ablations on each component of their method. \n- Discussion of the inference time limitations of their method. \n\n**Weaknesses**: Overall, I thought that this paper has significantly more weaknesses than strengths. \n- **Method is not well-explained**\n  - Figure 2 should be self-contained. What does $T$ mean? How are the learnable prompts used? What are $m$ and $n$? Why is this repeated $T-1$ times? Does this indicate that this is happening over the course of sampling?\n   - Section 4.3: \n      - $\\varphi_i(z)$: shouldn't $n$ depend on the layer index $i$ as well? \n      - $A$ includes multiplication with $V$ as written. Is that correct?    \n      -  The equations for Q, K, and V have shape errors in the matrix multiplication. \n   - Section 4.4: why can't Eq. 4 be folded into Eq. 3? \n   - Section 4.5: \n      - The concept of a \"learnable prompt\" appears to be a complicated way to say \"we fine-tune $W^k$ and $W^v$.\" \n      - Eq. 6 should try to stick to previous notation and not introduce a new $d$ function that is only used here. \n      - Eq. 7: $Attr_h(A)$ is not used anywhere else in the paper. And what is $f(A)$? The section uses it to denote the model output, but the diffusion model has a vector-valued output. This makes it unclear how to calculate this scaling term. \n- Experiments are not convincing.\n   - Section 5.1: why are only ComVG and RefCOCOg mentioned in the first line, and not VQAv2?\n   - Section 5.2: \n      - SD details: Its not clear which SD is being used. Is it 2.0 or 2.1? And is it base or the v-objective model? Description of the SD training data is also not correct. As written, it sounds like LAION-5B is separate from LAION-2B-en and LAION-aesthetics v2. \n      - \"Few-shot\": **I would definitely not consider training on 5% of large datasets as \"few-shot.\"** Please remove this description. In general, there's no justification why the proposed method would be particularly good at few-shot learning, or why there isn't evaluations using all of the training data. \n   - Section 5.3: Which CLIP model is the baseline here?\n   - Table 1: Why aren't methods like Diffusion Classifier and BLIP2 included here? You could add them (both zero-shot and few-shot), as well as Discffusion (zero-shot). \n      - To clarify, e.g. on VQAv2, CLIP and Discffusion are fine-tuned on 5% of VQAv2 training data here?\n   - Section 5.5/Table 2: Are all methods (CLIP, BLIP2, Diffusion Classifier, and Discffusion) fine-tuned on MS-COCO? Why can't they be evaluated zero-shot without any fine-tuning. In particular, the fine-tuning seems to hurt Diffusion Classifier performance, which was reported as 38.5 text score on Winoground, but seems to decrease here to 26.76? Also, is the \"26.76\" a typo? There are only 400 examples in Winoground, so the score should end in 0, 0.25, 0.5, or 0.75. Something seems generally off in the evaluation -- BLIP2 scores are also too low. \n   - Table 3: \"Note that evaluation datasets during pre-training\" is duplicated. \n   - Section 5.7/Table 4: \n      - Where is Diffusion Classifier?\n      - Can you show 95% confidence intervals? Winoground has very few eval samples for some of these categories, so some of these results are likely not statistically significant. \n      -  Misleading bolding. Only the \"Relation\" column is bolded, and the highest number is not consistently bolded. For example, CLIP's Relation Group score is the same as Discffusion, but only Discffusion is bolded. \n   - Section 5.8: \n      - Up until now, there has been no discussion of what sampling timesteps are. Why isn't this explained in the method section?\n      - How much time does it take per example with 100 sampling timesteps?\n      - How much time do the baselines take?\n      - \"NVIDIA A6000 equipped with 4 GPUs\" -- do you mean a workstation with 4 A6000 GPUs?\n    - Section 5.9: \n      - Cosine Similarity and Maximum are not explained anywhere in the paper. \n      - \"we implement the cosine similarity by only comparing the shared dimensions of the image and text feature vectors\" -- what does this mean?\n- Other issues with writing: \n  - Citations: \n     - Section 4.3: wrong citation for Stable Diffusion. Should be \"High-Resolution Image Synthesis with Latent Diffusion Models\"\n     - Section 4.5: wrong citation for classifier guidance. Should be \"Diffusion Models Beat GANs on Image Synthesis\"\n     - Incorrect usage of citep vs citet", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Discffusion, a method to do image-text matching with the cross-attention scores of a Stable Diffusion v2 model. They introduce a log-sum-exp pooling method to combine these cross-attention maps into a single metric, and propose a triplet loss function for fine-tuning SD to do better on these matching tasks. Finally, they introduce a \"dynamic attention head weighting\" method to upweight certain attention heads. They show that their method is competitive on image-text matching tasks like Winoground, VL-checklist, ComVG, RefCOCOg, and VQAv2.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors need to carefully address every issue I brought up in the \"Strengths and Weaknesses\" section. They need to rewrite the method section (including the algorithm block) to fully explain how training and inference work. They need to make the results section more clear and consolidate their main results without picking and choosing baselines. It seems like there are three main settings: zero-shot, zero-shot transfer (fine-tune on a different dataset than the eval dataset), and standard (fine-tune on the eval dataset if it has a training set). There should be a fair comparison between the proposed Discffusion method and the relevant baselines (CLIP, BLIP2, OFA, Diffusion Classifier). Details are above. \n\nThese changes are critical for me to recommend this paper for acceptance.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The major concern from the reviewers is on the clarity of the paper presentation (see comments from Reviewers 2hXW and yyhJ). Additional experiments were added and the authors also included revised presentation of the algorithm lists for clarification. The revision addressed most of the concerns, although I would suggest in final revision the authors should consider comments from yyhJ and make the descriptions of the method precise. Also providing intuitions on each of the components of the method will be much helpful, especially if the explanations can be backed by the ablation studies.
Decision: Accept (poster)