Text-to-Image Generation Via Energy-Based CLIP
Authors: Roy Ganz
Michael Elad
Venue: Accepted by TMLR ()

Abstract: Joint Energy Models (JEMs), while drawing significant research attention, have not been successfully scaled to real-world, high-resolution datasets. We present CLIP-JEM, a novel approach extending JEMs to the multimodal vision-language domain using CLIP, integrating both generative and discriminative objectives.  For the generative one, we introduce an image-text joint-energy function based on Cosine similarity in the CLIP space, training CLIP to assign low energy to real image-caption pairs and high energy otherwise. For the discriminative one, we employ contrastive adversarial loss, extending the adversarial training objective to the multimodal domain.  CLIP-JEM not only generates realistic images from text but also achieves competitive results on the compositionality benchmark, outperforming leading methods with fewer parameters. Additionally, we demonstrate the superior guidance capability of CLIP-JEM by enhancing CLIP-based generative frameworks and converting unconditional diffusion models to text-based ones. Lastly, we show that our model can serve as a more robust evaluation metric for text-to-image generative tasks than CLIP.

Published in Transactions on Machine Learning Research (07/2025)
Text-to-lmage Generation Via Energy-Based CLIP
Roy Ganz
voug275522@gmail.com
Electrical Engineering Degartment Technion
Michael Elad
elad tec.nion.cccccc..
Computer Science Department
Technion
Reviewed on OpenReview: https: // openreview. net/ forum? id- F
Abstract
Joint Energy Models (JEMs), while drawing significant research attention, have not been successfully scaled to real-world, high-resolution datasets. We present CLIP-JEM, novel approach extending JEMs to the multimodal vision-language domain using CLIP, integrating both generative and discriminative objectives. For the generative one, we introduce an image-lext joint-energy function based on Cosine similarity the CLIP space, training CLIP assign low energy real image-caption pairs and high energy otherwise. For the discriminative one, we employ contrastive adversarial loss, extending the adversarial training objective to the multimodal domain. CLIP-JEM not only generates realistic images from text but also achieves competitive results on the compositionality benchmark, outperforming leading methods with fewer parameters. Additionally, we demonstrate the superior guidance capability of CLIP-JEM by enhancing CLIP-basod generative frameworks and converting unconditions ional diffusion models to text-based ones. Lastly, we show that our model can serve as a more robust evaluation metric for text-to-image generative tasks than CLIP.
1 Introduction
Energy-based models (EBMs) (LeCun et al., 2006)
are a class of models that define a probability dis- tribution over data points using an energy function, where lower energy values correspond higher prob- abilities. These models are trained by adjusting the learned function to minimize the energy of observed data points and maximize the energy of synthetic ones, effectively aligning the energy landscape with the true data distribution. Joint Energy Models (JEMs) (Grathwohl et al., 2019) extend EBMs by utilizing a classifier's logits to also model a joint en- ergy function. JEMS are trained with both discrim- inative and generative objectives, namely, aiming to classify data points and to model the joint energy function, respectively. However, both EBMs and JEMs face significant scalability challenges. Their training processes can be unstable and computa- tionally intensive, restricting their applicability smaller datasets and making them unsuitable for real-world, high-resolution image datasets.
CLIP
CLIP-JEM
Acorgidag
An Egyplian
Figure 1: CLIP-JEM gradients. Demonstration of the meaningful input gradients CLIP-JEM compared to "vanilla" CLIP model with respect different. tcxtual prompts.

---

Published in Transactions on Machine Learning Research (07/2025)
Adversarial training (Goodfellow et al., 2015; Madry et al., 2018) is a technique designed to enhance models' robust against adversarial examples, which are small, imperceptible perturbations added to inputs to mislead classifiers. By training models to correctly classify these adversarial examples, such training results in perceptually aligned gradients (PAG) (Tsipras et al., 2019). PAG refers to the phenomenon where the model's input gradients' are semantically meaningful and aligned with human perception indicating that the features learned by the model are more human-aligned. Recently, the concept of PAG was extended to the multimodal image-text domain with CLIPAG (Ganz& & Elad, 2024), which applies adversarial training to the visual part of CLIP (Radford et al., 2021). This approach enables text-based generation through pixel-space optimization. However, while CLIPAG can produce good-looking images, the results are often non-realistic and heavily reliant on multiview augmentation. The need for such augmentation suggests that the gradients themselves are not sufficiently informative lo generate realistic images from single views. These limitations highlight the need for more advanced techniques to achieve realistic generation using CLIP models.
In this work, we propose CLIP-JEM, a novel approach that extends JEMs to the multimodal vision-language domain using CLIP by combining it. with adversarial training. This combination leverages the strengths of both techniques to address their limitations: mitigating the scalability and stability issues of JEMs and enabling high-resolution text-based generation while overcoming the non-rcalistic outputs typical of CLIPAG. Inspired by unimodal JEMs, we fine-tune CLIP using two objectives: generative and discriminative. For the generative objective, we introduce an image-text energy function based on Cosine similarity in the CLIP space, We train CLIP to assign low-energy values to real image-text pairs and high values to others. More specifically, inspired by EBMs, we utilize the model draw text-based generated samples and train it to assign these with high energy values. This is done by an iterative pixel-space optimization following the model's gradients, starting from random sample. We formulate this contrastive loss to align with CLIP. For the discriminative objective, we follow the path set by CLIPAG to define a contrastive adversarial loss. By combining these two objectives, we train the visual encoder of CLIP, resulting in CLIP-JEM, a model with semantically meaningful gradients (fig. 1) capable of generating realistic samples through simple pixel-space optimization (fig. 2).
We establish the effectiveness of CLIP-JEM across three key domains: text-to-image generation, guidance capabilities, and as evaluation metric. First, CLIP-JEM enables in text-to-image generation through pixel-space optimization, producing realistic images, significantly surpassing CLIPAG by more than 20 FID points, without applying any augmentation. Despite its relatively small size, this model achieves results competitive with much larger models on the challenging compositionality benchmark, CompBench (Huang et al., 2023). Specifically, it surpasses Stable diffusion v2 (Rombach et al., 2022a) and methods tailored for compositionality (Liu et al., 2023b; Feng el al., 2023). We attribute this to the discriminat nature of the model, enabling it to better align with the provided prompts. Furthermore, CLIPP-JMM significantly enhances text-based guiding capabilities. To this end, we illustrate that incorporating CLIP-JEM for guidance converts unconditional diffusion models (Dhariwal & Nichol, 2021; Ahn et al., 2024) into text-guided ones with just 25 diffusion steps. Additionally, replacing CLIP with CL.P--JM in based generative frameworks markedly boosts their performance. Finally, CLIP-JEM proves its utility as an evaluation metric (a.k.a. CLIP-Score) for text-based image editing. It shows robustness to adversarial examples and enhanced sensitivity to image quality compared to the "vanilla" " counterpart. This indicates that CLIP-JEM is a more reliable and precise tool for asscssing the quality and integrity of generated images. To summarize,
We introduce CLIP-JEM, novel approach extending Joint Energy Models to the vision-language domain using CLIP.
CLIP-JEM enables high-resolution text-lo-image generation through pixel-space optimization, achieving competitive results on a challenging compositionality benchmark.
CLIP-JEM enhances text-based guidance capabilitics, boosting CLIP-based generative frameworks and converting unconditional diffusion models into text-guided ones.
We demonstrate that CLIP-JMM can serve as an improved CLIP-Score evaluation metric for basod image editing.
This gradient computed as the derivative of the chosen output logit w.r.t. the input. image.
2

---

Published in Transactions on Machine Learning Research (07/202) 2025)
bench in the shade on grassy area.
lovelyy living decorated for the holidays.
baseball game being played city park.
stuffed bear with others wearing crowns
A bed with lot of pillows and window
CRARO
A beautiful bird sitting on a branch.
beautiful boat docked in a bay.
A banana tree in forest of trees.
A small marina with boats docked there
A street sign for the street named Crapo.
Figure 2: CLIP-JEM qualitative results. Images generated using CLIP-JEM with ConvNext-XNN XXLL
2
Related Work
2.1 Energy-Based Models
EBMs (LeCun et al., 2006) define a probability distribution over data points using an energy function:
po(x) exp(-Fo(x))
(1)
Z(0)
where Ea(x) assigns scalar values to data points, and 2(0) is normalizing constant. Lower energy values indicate higher probabilities. Training a model with parameters involves minimizing the energy assigned to "positive" data samples x+ ~ p(x) and maximizing the energy of "negative" samples x ~ pe px().. In this framework, the model's ability to "distinguish" between samples is operationalized by assigning lower energy (i..., higher likelihood) samples drawn from the data distribution, and higher energy (f.e. lower likelihood) to those drawn from the model distribution. Convergence is achieved when the model can no longer reliably separate positive and negative samples based their energy values. To sample from po(-), we employ Stochastic Gradient Langevin Dynamics (SGLD), which begins from predefined initial distribution and iteratively updates the samples using step size
a (x((xx))
X = x, 2 0x, te, c~,
(2)
2.2 Joint-Energy Models
Recently, Grathwohl al. (2019) observed that one parameterized pe(x, y) and px(x) using the logits of classifiers. Given a label and an image x, the joint distribution can be expressed as
(x y) exp(fo(x)y)
(3)
Z(0)
where fo(x), the logit corresponding with the th class label. Thus, the joint energy function is Eg (x. v) - -fo(xxy Marginalizing over y results in an unconditional distribution po(.). Training JEMs involve with optimizing both discriminative and generative objectives. Despite having several merits
3

---

Published in Transactions on Machine Learning Research (07/2025)
(e.g., generative capabilities and adversarial robustness), training such models often suffers from instability and even divergence. Despite recent, advancements, JFMs perform well for relatively small datasets (mainly SVHN (Netzer et al., 2011), CIFAR (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015)) but are not competitive when brought to real-world visual content (Yang et al., 2023; Zhu et al., 2021; Yang & Ji, 2021). In this work, we aim to extend JEMs into the most challenging selup texl-to-image generation using a CLIP- model.
2.3 CLIP for Text-to-lmage Generation
CLIP (Radford et al., 2021) is a vision-language model, pretrained to align a massive corpus of image-text pairs. The outstanding performance of CLIP visual and textual encoders has propelled great advancements in various fields. In Large Vision Language Modeling (Li al., 2022; Zhu et al., 2023; Liu et al., 2023a; et al., 2023; Ganz et al., 2023; 2024), CLIP vision encoder serves as the primary visual backbone, leading to umprecedented performance. In text-to-image generation, two main lines of work harness CLIP: (i) Utilizing CLIP image-text alignment to guide the visual results to be aligned with the textual description (Frans et al., 2022; Crowson et al., 2022; Patashnik et al., 2021; Gal et al., 2022; Kwon & Ye, 2022; Vinker et al., 2022); and (ii) Using CLIP's text encoder to condition generative models (Kang et al., 2023a; Nichol et al., 2022; Ramesh et al., 2022; Rombach et al., 2022b). Unlike these works, which utilize CLIP along with a generative model, we aim to cast CL.IP into an energy-based model, capable of performing text-to-image generation without an additional generative model.
2.4 Perceptually Aligned Gradients
Adversarially robust models (Carlini & Wagner, 2017; Madry et al., 2018) are designed to withstand adversarial attacks (Saegedy et al., 2014; Goodfellow et al., 2015), which are small, imperceptible perturbations ions aimed at mislcading classifiers. has been observed that such models exhibit phenomenon known as Perceptually Aligned Gradients (PAG), which is absent in their non-robust counterparts. PAG refers to the model's imput gradients being semantically meaningful and aligned with human perception, indicating that the features learned are more aligned with human vision (Ilyas et al., 2019; Engstrom et al., 2019; Salman et al., 2020). PAG has been harnessed for generative tasks, such as image generation and image-to-image translation (Santurkar et al., 2019), thereby improving slate-of-the-art image synthesis results (Ganz & Elad, 2022). PAG has also been explored for enhanced robust classification (Blau et al., 2023). Recently, the study of PAG has been extended to the multimodal domain using CLIPAG (Ganz & Elad, 2024), which applies adversarial training to the multinodal texl-to-image domain using CLIP (Radford et al., 2021). This approach enables text-based generation through pixel-space optimization. However, while CLIPAG produce good-looking images, the results are often non-realistic and heavily reliant on multiview augmentation. The need for such augmentation suggests that the gradients themselves are not sufficiently informative to generate realistic images from single views. These limitations highlight the need for more ad advanced techniques for realistic text-to-image generation using CLIP models.
3 Method
In this work, we aim to extend Joint Energy Models to the challenging text-to-image generation setting using a CLIP model. Our overall framework, illustrated in Figure 3, consists of two main objectives: a contrastive energy loss and an adversarial loss. In section 3.1, we first define the joint image-text energy function, forming a mcasure of the faithfulness and alignment of the given pair, and claborate on its training procedure. Next, in section 3.2, we detail the contrastive adversarial loss, extending CLIP's loss to the adversarial case. Lastly, in section 3.3, describe the overall training procedure of CLIP-JEM, resulting in multimodal energy model capable of text-to-image generation.
3.1 Joint Image- Text Energy Via CLIP
We extend pretrained CLIP model the joint energy of image-text pairs. We denote its vision and textual towers as f/ and f6, respectively. Given these notations, we propose to utilize CLIP to formulate
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
image-text joint distribution,
Pe II. T) exppCosineSSmmlaattyy//l/ (T)))
(4)
(
where I and T are visual and textual inpuls, CosineSimilarite larity is the well-known Cosine similarity measure used by CLIP, and 2()) is an unknown normalizing factor. Thus, the induced joint image-text energy function is (I, = - CosineSimilarit((ff (T), (T)), where a higher degree of image-text similarity results in lower energy values and higher probability and vere-versa.
Similar to energy-based model training, adapting CLIP to form energy mcasure requires both "positive" and "negative" image-text pairs, denoted T) and T),, respectively. As CL.IP is contrastively-trained model, we utilize such pairs to formulate a contrastive energy objective, illustrated on the upper-left side of Figure 3. This rectangular matrix contains the negative joint-energy values of textual and visual inputs. Specifically, the upper part contains the energy of the "positive" image inputs and the lower one the "negatives". Given this con- trastive energy matrix, we train the model weights of (.) using the cross-entropy loss with the main diagonal, marked in blue, as the ground-truth annota- tions. Namely, the objoctive is obtaín high values in this main diagonal (low energy, high probability) and low values elsewhere. Focusing tual input Tk, minimizing the proposed contrastive energy loss results in high Cosine similarity values of the "positive" " pair (Ig, Tx) and low ones for the "negative" pair (Iz, Tx). This formulated loss has two additional advantages: First, minimizing results in providing low Cosine similarities and hence high energy and low probability for unmatching image- text pairs (I, Tÿ) for i + enhancing the model's discriminative ability. Second, it enables the utiliza- tion of different realizations of negative samples per each text prompt, which we empirically find to stabilize the training.
Contrastive Energy Loss
hT
hT.
CLIP Score
ET
KT
LT.
hT
1.T.
bT.
X
...
X
Con aettaasive Adversarial Loss
Perceptual Quality
T
certain tex-
h
liTi
-.
I-T
bTi
I.T
IT.
X
Xy
Figure 3: CLIP-JEM method. Illustration of the contrastive losses in our method and their effect on cnabling realistic text-based image synthesis via pixel- space optimization.
The remaining question how obtain the "negative" samples. Similar to the methodology the JEM line of work, we craft such samples in an iterative process based on the model's gradients, starting from simple canonical distribution (e.g., Unifrom distribution). Current JEM works utilize SGLD (Welling & Teh, 2011), which requires hundreds of iterations to obtain good samples. To mitigate the computation overhead involved in generating such samples, such works utilize a replay buffer. This data structure stores thousands of generated "negative" samples and updates them throughout the training process, enabling a relatively small number of SGLD steps. However, in the image-text context, employing such a solution not feasible, as, unlike multiclass datasets such as CIFAR-10, which have a fixed amount of classes, the text-to-image setting is open-vocabulary in nature with practically infinite number of possible captions. Thus, in our setting, maintaining replay buffer and updating its samples is not practical solution, as the probability of having the exact two captions in dataset is very low, eliminating the usefulness of such structure. To enable fast "negative" sampling, we proposc to avoid employing SGLD and utilize a momentum- based optimization with an adaptive learning rate instead. In practice, we use Adamm optimizer (Loshchilov & Hutter, 2017) to update the image in the iterative process, which enables drawing samples within a relatively small number of steps. Overall, we first draw the initial sample from canonical uniform distribution, fr= Next, we compute the Cosine similarity between the visual and textual encodings and calculate its input-gradients. Lastly, we update the visual inputs accordingly and repeat the process for TIEM iterations. Notably, we calculate the
5

---

Published in Transactions on Machine Learning Research (07/2025)
gradients slightly noisy version of the visual imput, introducing randomness to the sampling mechanism. In the supplementary materials, we discuss the connection of our sampling procedure to SGLD.
3.2 Contrastive Adversarial Training
Similar to CLIPAG (Ganz & Elad, 2024), we extend adversarial training to the contrastive loss, forming a contrastive adversarial loss. Given an input batch of image-lext pairs, we perform a two-step procedure (i) crafting visual adversarial examples that maximizes the similarity w.r.t the matching texts; (ii) optimizing the vision encoder's weight minimize the contrastive adversarial loss. The adversarial contrastive loss matrix is illustrated on the lower-left side of Figure 3, and in eq. (5). In this matrix, the vertical green vector represents the encodings of adversarial visual input, while the horizontal pink vector represents the textual encodings. This approach ensures that the model learns to be robust against adversarial perturbations and equips it with semantically meaningful gradients (d... PAG), which enhance the iterative generation process.
min 01 (x.))D
Σ
max CosineSimilarity(f. (x (f, 8), f6 (t))) (t))).
(5)
SEa
3.3 Training Procedure
CLIP-JEM's objective is the combination of contrastive adversarial and contrastive energy losses, and the overall training protocol is described in the supplementary materials (algorithm 1). The goal of CLIP-JEM is to obtain image-lett energy model capable of generating images based on textual descriptions. On the right side of Figure 3, we illustrate this generation process (piece-wise linear arrow), starting from random sample from a canonical distribution (markod in gray). In this figure, any image-text pair is represented by two values: CLIP score and percept quality. The former measures the alignment of the pair according lo our CLIP model, while the latter conceptual measure of the image's visual quality. We aim to generate samples with high CLIP scores (low energy) and good perceptual quality. We analyze the contribution of both objectives to this goal. The contrastive energy loss prevents CLIP-JEM from assigning high CLIP scores to low-quality images, throughout. the energy training, the model is trained to lower the CL.IP scores of "negative" samples and provide high scores to the "positive" ones, which have high perceptual quality. The contrastive adversarial objective trains the model against adversarial inputs, namely, imperceptible visual changes that significantly decrease the CLIP score (these samples reside in the bottom right corner of the figure's center part). Thus, this loss prevents CLIP-JEM from assigning low CLIP scores to high-quality inputs. Overall, combining the two objectives eliminates the red modes in the upper-right panel of Figure 3, preventing the iterative generation from drawing such samples. This, in return, leads to drawing samples of the green mode, which are realistically looking images that align with their corresponding text. We demonstrate the contribution of combining these two objectives in the appendix D.
4 Experiments
We train different variants of CLIP using Algorithm 1 including VIT-B/32 and ConvNext in base, large, and XXL configurations on the extensive imago-caption DataComp datasct (Gadre et al., 2024) for 20, 000 steps. Throughout the training process, we keep the text encoder frozen and update solely the vision encoder. Implementation and training details are provided in the supplementary materials. To analyze the performance of CLIP-JEM, first evaluate it in the text-to-image generation setting Section 4.1). Next, we demonstrate its effectiveness as a guiding model (Section 4.2). Lastly, we show that CLIP-JEM can serve as an improved evaluation metric compared to the "vanilla" " CL.IP, attributed to its robustness and awareness of perceptual quality.
4.1 Text- To-lmage Generation
Similar to the training procedure, we perform pixel-space optimization to generate samples. Given a target prompt, initialize the image random sample from uniform distribution and perform 50 steps to
6

---

Published in Transactions on Machine Learning Research (07/2025)
Table MS-COCO text-to-image generation results. Frechet Inception Distance (FID, lower is better) and CLIPSIM (higher is better) results, along with model sizes. "ZS" indicates whether the model was trained on MS-COCO.
Method
# Params.
ZS
FID4
CLIPSIM
Stack-GAN
X
74.1
AttnGAN
230M
X
35.5
27.7
CogV View
4,000M
X
27.1
33.2
DALL-E
12,000M
V
27.5
GLIDE
6,000M
12.2
LDM-KL-8
1,450M
23.3
LDM-KT-8-G
1,450M
12.6
LAFITE
226M
26.9
StyleGAN-T
~1100M
13.9
NÜWA
870M
12.9
34.3
GigaGAN
1034M
9.1
CLIPAG
200M
82.0
30.3
CLIPAG
200M
V
47.6
33.4
CLIP-JEMvEr
88M
V
68.3
34.5
CLIP-JEMa
88M
V
34.8
31.6
CLIP-JEML
200M
26.7
31.7
CLIP-JEMXNL
846M
23.4
33.5
maximize the cosine similarity with respect the text, using an AdamW optimizer with no momentum. We evaluade the performance of CLIP-JEM in two main setups: image quality and compositionality.
Quality and Fidelity We use CLIP-JEM to generate 30, 000 samples from the MS-COCO datasel (Lin et al., 2015) and report the results FID? and CLIPSIM using VIT-B/32 in Table We compare CLIP-JEM to various GAN-based, diffusion-based, and autoregressive text-to-image models (see more details in the supplementary materials). We report the number of parameters of the basclines and the size of the vision encoder used for CLIP-JFM As shown in Table 1, scaling up the model size significantly benefits our method, substantially improving the FID scores. Specifically, in the XXL case, CLIP-JEM performs similarly to the unguided Latent Diffusion Model (LDM-KL-8) and outperforms DALL-E and CogView despite being smaller. Additionally, we train a CLIPAG baseline using the same training configuration and architecture to better demonstrate the effectiveness of our image-text energy objective. We report the results of two variants of CLIPAG using Next Large with and without multiview augmentations, denoted CLIPAGL and CLIPAGL', respectively. As can be seen in CLIPAG results, the multiview augmentation pipeline is crucial (improves the FID from 82.0 to 47.6), highlighting the unsatisfying quality of its gradients. Interestingly, using the same model with CLIP-JEM lcads to a much-improved FID score (26.7) without applying any augmentation. This strongly indicates the eflectiveness of introducing our cont cont rastive energy loss and our method's improved capability of generating realistic samples. In the CLIPSIM metric, CLIPAG, leads to a better result than CLIP-JEM We attribute this to the fact that the multiview augmentations lead unrealistic images (which impair the FID) that highly aligu with the text (increasing the CLIPSIM), Overall, these results indicate that CLIP-JEM achieves both of its goals extending JEM training into text-to-image generation and improving CLIPAG's photorealism.
Compositionallity We compare CLIP-JEM with other generative models using T21 CompBench (Huang et al., 2023), which evaluates open-world compositional text-to-image generation across attribute binding (color, shape and texture), object relationship (spatial and non-spatial) and complex. Despite advances in text-to-image generation, models still struggle to compose objects with different characteristics and
We the evaluation codes with DM-GAN, which available https://github. htt.g/ttww.comm/iinegggng//////e

---

Published in Transactions on Machine Learning Research (07/2025)
Table 2: T2l-CompBench results. CLIP-JEM performance on the text-to-image generation compositional-
ity benchmark.
Attribute Binding
Object Relationship
Model
Complex
Average
Color
Shape
Texture f
Spatial 1
Non-spatial f
SD144
0.3765
0.3576
0,4156
0.1246
0.3079
0.3080
0.3150
SD2
0.5065
0.4221
0.4922
0.1342
0.3127
0.3386
0.3677
Composable (SD2)
0,4063
0.3299
0.3644
0.0800
0,2980
0.2898
0.2947
Structured (SD2)
0,4990
0.4218
0,4900
0,1386
0,3111
0.3355
0.3660
Attn-Exet (SD2)
0.6400
0.4517
0.5963
0.1455
0.3109
0.3401
0.4141
CLIP-JEMvir
0.5305
0.5159
0.5566
0.0262
0.3343
0,2900
0.3756
CLIP JB
CLIP-JEMB
0.5799
0.5122
0.6154
0.0708
0.3145
0.2938
0.3978
CLIP-JEMr
0.5715
0.5202
0.6072
0.0768
0.3152
0.3020
0.3988
CLIP-JEMXXL 0.5670
0.5021
0.6132
0.0841
0.3205
0.3129
0.4000
relationships into a coherent image. Following CompBench procedure, we generate 10 samples per prompt and average the results on the validation sets each category, using the same evaluation metrics as in the original paper (B-VQA, UniDet, CLIP, and 3-in-1 for the attribute binding, spatial, non-spatial, and complex calegories). Table 2 reports CLIP-JEM results compared to top-performing models, including an average score across six categories. CLIP-JEM excels in attribute binding, associating attributes with corresponding objects in generated images, and performs well in the non-spatial relationship category (e.g., "speak to" and "look at"). However, it scores lower in the spatial relationship category due to CLIP's known limitations in spatial compositionality (Yuksekgonul et al., 2022; Lewis et al., 2022). In the complex category, involving multiple objects and attributes, CLIP-JEM performs well despite containing spatial relationships, due to its strengths in attribute binding and non-spatial understanding. Overall, CLIP-JEM outperforms most basclines in compositional generation, despite being smaller. Specifically, surpasses methods deliberately designed tackle compositionality and rely a much stronger generative model (Stable Diffusion v2). We attribute this success to the generative and discriminative objectives combination, enabling effective alignment with compositional prompts.
4.2 Text Guidance Using CLIP-JEM
ted--condllooee samples
Unconditional samples
cute red A living room with cow on the beach fireplace
As shown in fig. 1, CLIP-JEM possesses semanti- cally meaningful gradients with respect to given text. In this section, demonstrate the guidance capability of our method in two main settings: diffu- sion guidance and improving CLiP-based generative frameworks, utilizing a ConvNext Large model.
bar surnounded by green field
A Agldennbie bird
A beach in Thailan
Diffusion guidance We utilize CLIP-JEM as guiding technique transform unconditional diffu- sion models trained on ImageNet (Dhariwal & Nichol, 2021; Ahn et al., 2024) into text-conditioned ones using only 25 DDIM steps 25 (Song et al., 2020). In each DDIM step 4, we update the estimations for the clean image &n using the gradients of CLIP-JEM (eq. (6)) and use it to compute Xt-1 and continue the reverse DDIM process.
Figure 4: Diffusion guidance using CLIP-JEM. Converting unconditional diffusion model into text-based one with CLIP-JEM. In each row, plot the unconditional alongside the guidance results using the same seed.
8n En 8 Vx, CosineSimillaittyyf (T))
(6)
8

---

Published in Transactions on Machine Learning Research (07/2025)
drewing of
Horse ealing cupake
3D rendering temple
Family vecalion
Self
A diglial Ilkatiaiion of fatrrstic spaceship
A wood carsing depicling a somee from Norse nythology
A sculphare imspired by the works of Conolantin Bräncugi
An impressinnsl painting af blooming folld flowers
A watercolor painting ol dit
Woll
CLIP
CLIP CLIP-JEN JEM
Bagus Rolale S.LFF
CUP-JEM
65.25
Hore 17.03
Koyhh 19.66
SAI 15.215
(a) CLi results
(b) VQGAN+CLIP results
Figure 5: Improving CLIP-Bases Generative Frameworks via CLIP-JEM. Qualitative results of CLIP-JEM compared to CLIP using CLIPDraw and VQAGAN+CLIP in a zero-shot setting.
We provide qualitative results of text-based diffusion guidance in Figure 4, showcasing the capability of CLIP-JEM to convert unconditional diffusion models to text-based ones, thereby enhancing their utility.
Improving CLIP-Based Generative Frameworks CLIP is widely used in text-to-image generation frameworks to update the resulting image to better align with the target text in CLIP's space. This is typically done using the imput gradients of the CLIP's vision encoder with respect to the text (maximizing the cosine similarity). However, these gradients tend to be semantically meaningless, as demonstrated in Figure 1. To mitigate this, a multiview augmentation pipcline is often employed to acquire scmantically meaningful gradients. Our method, on the other hand, inherently produces gradients that convey rich semantic information. Consequently, using CLIP-JEM can eliminate the need for multiview augmentations, thereby improving computational efficiency. To demonstrate this, we experiment with two such frameworks: CLIPDraw (Frans et al., 2022) and VQGAN+CLIP (Crowson et al., 2022). CLIPDraw generates drawings by optimizing the parameters of Bézier curves using CLIP, and VQGAN+CLIPP updates the latent code of a VQGAN to enable text-to-image generation. In both cases, we do not perform multiview augmentations and report the results with augmentations in the supplementary matcrials. In Figure 5a, we present the results of CLIPDraw on target texts from the original paper, along with CLIP's top prediction for each prompt. As shown, using CL.IP-JMM leads to significantly better visual results, which are more aligned with the text than those generated by the "vanilla" CLIP. The high percentages in the top predictions imply that the images generated by CLIP have an adversarial nature, maximizing the score without performing significant modifications. Next, we evaluate the effect of using our approach to guide VQGAN. To this end, we prompted ChatGPT to provide artistic target texts, aligning with the original paper's domain. As seen in Figure 5b, removing the augmentation pipeline results in non-meaningful outpuls, whereas CLIP-JEM generates semantically meaningful images. These results strongly attest to the improved guidance capabilities of CLIP-JEM.
4.3 CLIP-JEM as an Evaluation Metric
CLIP often used to evaluate text-to-image generative tasks by measuring cosine similarity between textual descriptions and images in its embedding space, known CLIP-T. We compare CLIP-JEM with the standard CLIP model, focusing on the ViT-B/32 commonly used for this purpose. Using TEdBench (Kawar et al., 2023), evaluate CLIP-T scores for various inputs, including outputs from top-performing generative model (Imagic), source images ("No Edit"), and "Noise" images. Results are shown in table 3 under "Vanilla". We also assess robustness under adversarial attacks with a low perturbation budgel (e = 2/256). These al lacks aim to increase scores for "bad" images and decrease scores for "good" ones. Our findings indicate that CLIP highly suscoptible to adversarial attacks, resulting in higher CLIP-T scores for non-odited and noise images than for Imagic outputs. In contrast, CLIP-JEM remains robust, maintaining higher scores for Imagic outputs even under adversarial attacks.
9

---

Published in Transactions on Machine Learning Research (07 /025)
Table 3: Robustness To Adversarial Perturba- tion.
Gf
100
Input images
CLIP
CLIP-JEM
Vanilla
Attack
Vanilla
Attack
Imagic
0.3031
0.2053
0.2016
0.1951
No Edit
0.2740
0.3547
0.1811
0.1866
Noise 0.2033
0.3037
0.0905
0.0959
000
Figure 6: Sensitivity to perceptual quality.
We further analyze the effect of perceptual quality on CLIP-T scores by blending Imagic images (Xtmagic) with uniform noise: Am/magic + (1 - X)u where u 10, 1]. We plot CLIP-T scores for varying À values in Figure 6, normalizing scores to 1.0 at À = 0. While the standard CLIP model prefers noisier versions up to À = 0.45, CLIP-JEM 's scores decrease with increasing noise, indicating greater sensitivity to image quality, attributed to the contrastive energy objective making the model to assign high energy to non-real images.
5 Discussion and Conclusion
In this work, we introduce CLIP-JEM, a novel approach that extends Joint Energy Models to the multimodal domain using CLIP. Through extensive evaluations, CLIP-JEM demonstrates its ability generate high- quality, compositionally coherent images, achieving competitive results the MS-COCO dataset and excelling in the T21 CompBench benchmark. Moreover, CLIP-JEM showcases strong guiding capabilities, significantly improving the performance of CLIP-based generative frameworks and converting unconditional diffusion models to text-based ones. Additionally, CLIP-JEM proves to be a robust and perceptually aware evaluation metric, maintaining high scores under adversarial attacks and showing greater sensitivity to image quality than the standard CLIP model. We hope that the insights and findings presented this paper will inspire further exploration and advancements in multimodal JEM research.
References
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. Xie preprint arXiv:2433. 17977, 2024.
Tsachi Blau, Roy Ganz, Chaim Baskin, Michael Elad, and Alex Bronstein. Classifier robustness cuhancement via test-time transformation, 2023.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Symposium on Security and Privacy (SP), 2017.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architocturr,, training and dataset, 2025. URL https://arxis: wereii.orr//ae wwt.oorr///../000///0.3 09568
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. VQGAN-CI.IP: open domain image generation and editing with natural language guidance. In ECCV 2022 17th European Conference, volume 13697, pp. 88 105, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
10

---

Published in Transactions on Machine Learning Research (07 2025))
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers, 2021.
Logan Eugstrom, Andrew Dyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Learning perceptually-alignee representations via adversarial robustness. arXiv preprint arv: 9006000045 2019.
Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- free structured diffusion guidance for compositional text-to-image synthesis, 2023.
Kevin Frans, Lisa B. Soros, and Olaf Witkowski. Clipdraw: Exploring texl-to-drawing synthesis through language-iman -mage encoders. In NeurIPS, 2022.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datascts. Advances in Neural Information Processing Systems, 36, 2024.
Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan- nada: Clip-guided domain adaptation of image gencrators. ACM Trans. Graph., 4114:11111111 2022.
Roy Ganz and Michael Elad. BIGRoc: Boosting image generation via a robust classifier. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Roy Ganz and Michael Elad. Clipag: Towards generator-free text-to-image generation. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 19843-3854 2024.
Roy Ganz, Oren Nuriel, Aviad Aberdam, Yair Kittenplon, Shai Mazor, and Ron Litman. Towards models that can see and read. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 21718 21728, 2023.
Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avraham, Oren Nuriel, Shai Mazor, and Ron Litman. Question aware vision transformer for multimodal reasoning. arXiv preprint orXiv:2402.054 05772, 2024.
I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations, ICLR, 2015.
Will Grathwohl, Kuan-Chiol Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprinl arXiv:1912.03260 2019.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint rrXio:2307.06330 2023.
Andrew Tlyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, 2019.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-imape synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
Minguk Kang, Jun- Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis, 2023b.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosscri, and Michal Irani. Imagic: Text- real image editing with diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation, 2022. URL
wwwpwiwwwwwwww.////////a.p..aaet 2110.02711.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with single text condition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), pp. 18062 18071, June 2022.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Martha Lewis, Nihal V Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H Bach, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:211.110597 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping eaaguage-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730 19742. PMLR, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitmick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URL https://arxiv http://axxxv...g/bb...aa/ 1405.405000 0312.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models, 2023b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Compuler Vision (ICCV), December 2015.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXio preprint arXiv:1711.05101, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. Tn International Conference Learning Representations, ICLR, 2018.
Yuval Netzor, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 7, 2011.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text- guided diffusion models. In International Conference on Machine Learning, ICML 2022, volume 162, pp. 16784 16804, 2022.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In 2021 TEEE/CVF International Conference on Computer Vision, ICCV 2021, pp. 2063-2074, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763, 2021.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-to-image generation, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/ 200/2204.06125,, 2022. doi: 00448500/a2iiv22244.. URL https://doi.orge wt......///////............ 2204. 220.206124......
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognition (CVPR), pp. 10684 10695, June 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022c.
Hadi Salman, Andrew Hyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Newral Information Processing Systems, 33:3533 3545, 2020.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Iyas, Logan Engstrom, and Aleksander Madry.
Image Synthesis with a Single (Robust) Classifier. 2019.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis, 2023.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2rXiv:2010.008: 0.02502, 2020.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruma, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference Learning Representations, ICLR, 2014.
D. Tsipras, S. Santurkar, Engstrom, A. Turner, and A. Madry. Robustness May Be Odds with Accuracy. In International Conference on Learning Representations, ICLR, 2019.
Yael Vinker, Ehsan Pajouheshgar, Jessica Y. Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: semantically-aware object sketching. ACM Trans. Graph., 41 41(4):86:1-880) 16:11,, 2022.
Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICMI-111, pp. 681 688. Citeseer, 2011.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation, 2021. URL https: http://aaawvv..o/aa///s///ss wttg/bs//gg1aaao..//spgg/ 12417.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks, 2017.
Xiulong Yang and Shihao Ji. +: Improved techniques for training jem. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) pp. 6494-6503, 2021.
Xiulong Yang, Qing Su, and Shihao Ji. Towards bridging the performance gaps of joint energy-based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15732 15741, 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of- words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks, 2017.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Yao Zhu, Jiacheng Ma, Jiacheng Sun, Zewei Chen, Rongxin Jiang, Yaowu Chen, and Zhenguo Li. Towards understanding the generative capability of adversarially robust classifiers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 7728-7737, 2021.
14

---

Published in Transactions on Machine Learning Research (07 (025)
A Detailed background on energy-based models
Energy based models Energy-based models (EBMs) (LeCun et al., 2006) are a class of probabilistic models that define probability distribution over the data points via an energy function. Specifically, EBMs utilize the fact that any probability density p(x) for x E De can be written
- exp(-Foole
po(
((0)
where (-) the energy function, which assigns scalar values data points (Ei RDD RR and Z()) is a normalizing constant. In EBMs, the probability of given data point is determined by its energy, where lower energy indicates higher probability, as can be seen in Equation (7).
Training such a model involves learning the parameters 0 of the energy function, typically modeled by a neural network, to minimize the energy of observed "positive" data points, x ~ p(x), while maximizing the energy "negative" samples, x ~ po (x), generated by the model during training. The training converges when the model cannot detect whether a sample "positive" or "negative", attesting that po and p have a similar distribution.
EBMs require sampling from pe, both during training to acquire "negative" samples and during inference to synthesize new images. In practice, drawing such samples is done via Stochastic Gradient Langevin Dynamics (SGLD), initialized with Xo ~ Px() from a canonical distribution (e.g. Uniform over the input domain), and updated iteratively by
a (FFx((x;)
X++1 = X, 2 0x;
te, e~~N
(8)
where a is the sampler's step sizc.
Joint energy models Recently, Grathwohl et al. (2019) observed that one utilize classifier to model energy function and train it accordingly. Given a classifier fe which maps inputs into K values, known as logits (fg : R + k),, one can parameterize a conditional distribution using the Sof tmax function:
exp([o(x),)
po (o( x) Eg exp(fo(x)))
(9)
where fo(x) y is the logit corresponding with the 4th class label. The key observation is that one can construct expressions for po(x.y) y) and (x()) using the classifier's logits. The joint distribution of data point and label given via
po v) exp(fo(x)y)
(10)
2(0)
where Z(0) = [2 exp(fo(x), is an intractable normalizing factor. Accordingly, we can define a joint energy function Fo(x, v) -x)(x)y- Using marginalization over y, obtain the unconditional distribution,
Ey Exp(((((()))
po(x, po(x)= = Epox y)
(11)
Z(0)
Thus, in the unconditional case, the energy function is given Ex(x) log E, exp(fo(x)y)
With this interpretation, one can train joint model for both discriminative and generative modeling, optimizing both classification and EBM objectives. Such models lead to good classification capabilities, showcasing impressive adversarial robustness while being able to generate new data samples. Nevertheless, training such models often suffers from instability and even divergence, making applicable mainly for small datasets but not for real-world ones.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Arch.
BS disc.
BS gen.
#Sleps
LR
WD
Sched.
Warmup
Adv.
Tadv
ThE
7
VIT-B/32
256
32
10
ConvNext-BB
128
32
20K
10-5
10
Cusine
200
3.0
0.1
1.5
0.025
CouyNext-L
128
16
2 x 10
ConVNext-XXX
32
8
2 10-6
Table 4: Implementation details. We provide the training hyperparameterss of CLIP-JEM for the different. architectures (BS disc. and gen. stands for the discriminative and generative batch sizes, respectively).
Arch.
Time [Sec.]
Memory [M]
ViT-B/32
1.4
2787
ConvNext-B
2.6
3009
ConvNext-L.
2.4
4523
ConwNext- XXLL
4.5
11837
Table 5: Sampling time and memory, The time per-sample in seconds and memory consumption of the different considered models.
B Implementation details
Training hyperparameters We implement our method upon the OpenClip codebase". We consider the following model architectures from the model zoo convnext convnext_xxlarge, convnext large d, convnext base and ViT-B-32 with the following pretrained weights, respectively laion2h s34b b82k augreg_sool _aainn2__s44b______
laion2b_sst s2bb_______________________ s26b_b102k augreg, laion2b s13b__b222 b__kk_aakreegggg_______a augreg and openai. In table 4, report the training hyperparameters for CLIP-JEM We use these hyperparameters to train the different architectures on DataComp. To train our models, we use 8 A40 GPUs for training. Training the largest variant (ConvNext- XXL) for 20K itcrations takes 10 days. During training, in the generation process of the "negative" samples we employ a momentum 0.9 to the Adamn optimizer. However, throughout the inference phase, we do not. utilize momentum at all. We aim to make our code and pretrained models publicly available upon acceptance.
Experimental settings To measure the quality and fidelity of the generated images of our method, we compare it to strong baselines using MS-COCO dataset (table 1). Specifically, compare CLIP-JEM to text- to-image generative models of different types: (i) GAN-based Stack-Gat.... (Zhang et al., 2017), AttnGAN (Xu el al., 2017), LAFITE (Zhou et al., 2022), StyleGAN-T (Sauer et al., 2023), and GigaGAN (Kang el al., 2023b) (ii) Diffusion- baased GLIDE (Nichol et al., 2022), and LDM (Rombach et al., 2022c), and Autoregressive ones DALL-E (Ramesh et al., 2021), CogVien (Ding et al., 2021), and NÜWA (Wu et al., 2021). Our reported results and model sizes originate from the respective papers. The CLIPAG baseline results were obtained by architecture and hyperparameters. As for
us, by removing the contrastive energy loss term, using the the CompBench results (table 2), report the one from the benchmark's 's paper.
As for the improving CLIP-based generative frameworks experiments, we use an ADM-based model (Dhariwal & Nichol, 2021) with perturbed-attentin guidance (Ahn et al., 2024) which strongly improves the unconditional generation as our baseline for diffusion guidance.
Sampling time and memory We train 4 different architectures of CLIP-JEM. As different model size and structure alfects the runtime complexity, we report the time of generation process using batch size of using an Nvidia A40 GPU in table 5. As expected, increasing the model size leads to more memory consumption and increases the time per sample. However, the Conv-Nett- generation time is slightly larger than the Conv This is due to the fact the we utilize the wide variant of the ConvNext-B, which not available in ConvNext-L.
16

---

Published in Transactions on Machine Learning Research (07/2025)
A Algorithm CLIP-JEM Training. Given CLIP image and lext encoders f)() (+) and fø (:); image-lexi. dataset D, adversarial budget e, adversarial and energy step-sizes a1 02, energy loss coefficient 7, and number of adversarial and generation iterations Tady; TIEM:
while not converged do Sample (1,T) from dataset D /* Contrastive adversarial loss
dg +- for f from 0 to Tadu do
5141 (6, 01 (f (I+ (T))
end
Iadv = =I STAd
Ladv = ClipLoss (f)(a))) (T))
/* Contrastive energy loss
*/
Sample initial "negative" sample I.
Optimizer + AdamW AdamW(params = [,,1 = 02) for t from 0 to TJEM do
CJEM = ClipLoss (f (I fn), (T)
/* n ~ I), B is small scalar
Calculate @Coes//1 and perform an optimizer step end
LJEM = ClipLoss (Concat I)), (T))
/* Update the vision encoder
L = Ladv + * LJEM
Calculate 84/00 and update CLIP image encoder (-)
end
C Training protocol
In algorithm 1, we detail overall training procedure of CLIP-JEM. First, we draw image-text batches from dataset D and apply Tadv iterations to obtain the adversarial visual impputs Lady which we use to calculate the adversarial loss Ladv- Next, we perform an iterative pixel-space optimization to obtain "negative" samples i. Specifically, this is done using THEM iterations using AdamW optimizer. We form our joint energy based loss using the "positive" and "negative" samples. Lastly, we combine these two terms to formulate our overall objective, which we use to update the vision encoder weights. We repeat this process until convergence.
D Ablation study
Architecture In table 1, we report the results of both ViT-B-32 and ConvNext base using CLIP-JEM. Notably, despite the two architectures share a similar capacity, the ConvNext FID score is significantly better (by 33.5 points). We hypothesize that this stems from the improved prior that CNN-based architectures serves. Additionally, generated images from ViT contain grid artifact from the patch procossing mechanism. Thus, we mainly focus on ConvNext-based models.
Objectives contribution To highlight the importance of combining the adversarial and energy contrastive losses, we train the same model for 1, 1,00 iterations using (i) contrastive energy loss; (ii) contrastive adversarial loss; and (iii) cont rastive energy loss + contrastive adversarial loss. We plot the results of 16 text prompts from MS-COCO for the resulting models in fig. The results indicate that using only the contrastive energy loss does not produce meaningful outputs. In contrast, employing solely the contrastive adversarial loss results in meaningful but unrealistic content. Remarkably, combining the two objectives (CLIP-JEM's approach) leverages the strengths of both CLIPAG and JEMs, yielding superior outcomes.
17

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Contraction Energy Lore
Contractive Advertarial Lote
Contractivn Energy Lore Contractive Advercarial Loce
Figure 7: Objectives ablation study. CLIP-JEM employs a two objectives: Contrastive Adversarial Loss and a Contrastive Energy Loss, In this figure we ablate the effect of each objective, highlighting the importance of cach objective. We plot 16 images based on different textual prompts from MS-COCO after 500 training steps.
E
Connection to Stochastic Gradient Langevin Dynamics
SGLD enables sampling from distribution p(x) given its score function Vx log The stochasticity in this algorithm is important, as applying a simple gradient method would result in sampling the distribution peaks rather than truly sampling the distribution. The iterative update of SGLD in its discrete form is as follows:
Lt+1 Ig V logp(x;; +e,
where a is the step size and ~~ NO af) is Gaussian noise. From eq. (7), the above equation can be also written in terms of energy,
a Vn, Fo(x)) +
E++1 = Et
and in the joint energy case, the update rule becomes
a Ea, te. 2
D1+1 = It
Note that our algorithm we add small Gaussian perturbation to the input, prior the calculation of the gradient (algorithm 1). This introduces similar, yet different, stochasticity in approach, compared with SGLD, To better understand this, expand the following Taylor series:
VxEo(x+e,1) R Ex + Eo
+[[E
As our noise is of small variance, the higher order term can be neglected,
Ex VxEa(x.v)) v) + E y) "
making it similar to the SGLD noisy update step, the term Eo T) is linear transformation of the Gaussian noise, resulting in colored version of a Gaussian distribution, governed by the Hessian of the energy function.
We demonstrate the randomness introduced by this mechanism using our sampling process in fig. 8. Unlike SGLD, we utilize an AdamW optimizer for the update step for the generated sample. Thus, we do not use the raw gradient as in SGLD since our optimizer has an adaptive learning rate mechanism. This results in different effective step size per iteration (a in SGLD). AdamW also enables momentum term, which
18

---

Published in Transactions on Machine Learning Research (07 (075)
A cozy cabin in the woods with smoke coming from the chimney
medieval knight standing in front of a castle
WONDER
group of astronauts exploring distant planet
Figure & Stochasticity demonstration of CLIP-JEM sampling.
roduces bias to the gradients. During CLIP-JEM training, we utilize the momentum as it significantly stabilizes the training. However, at inference time, we do not employ momentum, resulting in an unbiased version of the gradients.
F Additional results
We provide additional qualitative results for both improving CLIP-based generative frameworks and text- to-image generation. For the former, we provide the results using the same prompts with augmentations (figs. 10 and 11). As can be seen, seamlessly replacing CLIP with CLIP-JEM leads to improved results with and without augmentations. Notably, CLIP-JEM results without augmentations of CLIP with augmentation, offering substantial reduce of computational overhead. For the latter, we provide more generated image (fig. 9) and demonstrate the stochasticity of CLIP-JEM(fig 8).
comparable to the ones
G Positioning CLIP-JEM within the Landscape of Generative Models
Comparison with Diffusson Modcls While CLIP-JEM does not achieve state-of-tho-art FID scores in lexi-to-image generation (see lable 1), it offers several notable advantages. exhibits superior compositional generalization, outperforming Stable Diffussin v1.4 and v2 on the T21-CompBench benchmark (Gadre et al., 2024), as well as surpassing composiiionallyy-rrinndd diffusion-based methods. This performance stems from its joint energy-conee contrastive adversarial training, which enhances the alignment between textual prompts and generated images. Unlike diffusion pipelines that rely multi-step denoising in pixel latent. spaces
19

---

Published in Transactions on Machine Learning Research (07 (225))
necessitating learned decoder and intricate noise scheduling CLIP-JEM samples directly in pixel space through a straightforward iterative gradient-following process. This approach yields energy values proportional to sample likelihoods, facilitating explicit probabilistic ranking and selection of outputs. We leverage this property to develop a robust text-to-image alignment metric (see section 4.3) and to convert unconditional diffusion models inlo lext-conditioned generators. Furthermore, CLIP-JEM energy-based formulation allows for seamless 'plug-and-play" integration into CLIP-based synthesis workflows; for example, replacing the CLIP encoder in CLIPDraw with CLIP-JEM results in more semantically faithful renderings under the same optimization loop (Frans et al., 2022).
Comparison with Architectures Employing CLIP Image Encoders Leading texl-to-image systems incorporate CLIP's image encoder alongside distinct generative backbones rather than utilizing it to generate pixels directly. DALI - (Ramesh et al., 2022) employs a diffusion prior to map CLIP text embeddings to CLIP image embeddings, followed by a modified GLIDE decoder for pixel syuthesis. DiffusionCLIP (Kim et al., 2022) integrates a CLIP-based contrastive loss into the reverse diff'usion trajeclory of pretrained diffusion models to enable zero-shot, text-guided image manipulation. VQGAN-CLIP (Crowson et al., 2022) pairs pretrained VQGAN generator with CLIP's image encoder to score and steer pixel-space updates via gradient descent. Recently, BLIP3-o (Chen et al., 2025) utilizes the CLIP image encoder in conjunction with large language model and a diffusion transformer to generate images. In contrast, CLIP-JEM constructs a generative model directly from CLIP itself, eliminating the need for an additional generative model.
H Limitation and Futrue Work
Our current work exhibits two main limitations. First, CLIP-JEM shows limited diversity in its generated samples. The primary source of stochasticity comes from the random initialization process, and while we experimented with noise injection during optimization steps, we maintained a small noise coefficient for stability. This resulted generated images sharing similar visual characteristics for the same prompt, evident in fig. 8. Future work could explore the enhancement of sample diversity through increased noise levels during optimization alternative initialization strategies. Second, CLIP-JEM underperforms tasks involving spatial relationships, as demonstrated in table 2. This limitation stems from CLIP's inherent constraints in spatial compositionality, as our model initializes from CLIP weights. This aligns with the well-documented limitations of CLIP in handling spatial relationships. Future research could focus on addressing these spatial understanding capabilities, potentially through architectural modifications or specialized training objectives that better capture spatial information.
20

---

Published in Transactions on Machine Learning Research (07/2025)
A beautiful garden and dog in the middle
A vintage train traveling through countryside.
A bustling market street in an ancient city.
bed with some kind of blankets on
A couple of bikes and a umbrella in a yard.
A ballet dancer performing on grand stage
bear in the woods standing on a log.
A small marina with boats docked there
Figure 9: Additional qualitative results. Samples generated by CLIP-JEM using ConvNext XXL.
21

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
drawing of a cat
Horse eating cupcake
3D rendering of temple
Family vacation to Wolt Disney World
Self
r
CLIP
Cat 73.76%
Cupcake 62.87%
Temple 14%
Vacation 11%
Self 99.22%
Sel
CLIP-JEM
Wot Poodt
CAT
Relaile
2006
Sell LeL
Cat 65.88%
Horse 37.07%
Temple 94.96%
Vacation 45.54%
Self 95.21%
Aug+CLIP
Sel
Cat 47.57%
Horze 26.06%
Temple 93.98%
Vacation 52.00%
Self 98.79%
SeLf
Aug-CLIP-JEM
abtt
T.S. S. MTTCOL
A
6x 7
Cat 18.60%
Horse 15.41%
Temple 98.10%
Vacation 51.59%
Self 61.02%
Figure 10: CLIPDraw results. We provide the results of CLIP and CLIP-JEM with and without augmentations.
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
A digital illustration futuristic spaceship
A wood carving depicting a scene from Norse mythology
sculpture inspired by the works of Constantin Brâncuşi
An impressionist painting of blooming field of flowers
A watercolor painting of a city view
CLIP
CLIP-JEM
Aug+CLIP
Aug+CLIP-JEM
Figure 11: CQGAN+CLIP results. We provide the results of CLIP and CLIP-JEM with and without augmentations.
23


Reviews:
[{"review_id": "GPsLaDRSg8", "reviewer": "Reviewer_9mgW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Creatively combines the strengths of EBMs (generative flexibility) and CLIP (multimodal alignment).\n\n2. CLIP-JEM achieves impressive results across multiple evaluation benchmarks.\n\n3. The authors have validated that the model's versatility across multiple applications: direct text-to-image generation, guidance for other models, and as an evaluation metric.\n\nWeaknesses:\n\n1. The model seems to struggle with spatial relationships compared to other methods. Is there any explanation of this situation?\n\n2. Quantitative evaluation of guidance: The guidance capabilities are primarily demonstrated through qualitative examples, but more quantitative evaluation would strengthen this claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces CLIP-JEM, a novel approach that extends Joint Energy Models (JEMs) to the multimodal vision-language domain using CLIP. The key contributions are:\n\n1. CLIP-JEM is a novel approach extending Joint Energy Models to the vision-language domain using CLIP.\n\n2. Improving Text-to-Image Generation via pixel-space optimization\n\n3. Enhancing text-based guidance capabilities and converting unconditional diffusion models into text-guided ones with just 25 DDIM steps\n\n4. CLIP-JEM can serve as an improved CLIP-Score evaluation metric for text-based image editing.\n\nThe method achieves competitive results on MS-COCO and T2I-CompBench, demonstrating strong compositional understanding despite its smaller size compared to models like SD v2.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide some explanations about why the model performs unsatisfying in the spatial tasks.\n\n2. Some quantitative evaluation results of the guidance are expected.", "broader_impact_concerns": "No Broader Impact Concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tC7y424qgc", "reviewer": "Reviewer_2vtY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1) The idea of revesting JEM to build a modern generative model is interesting.\n2) The idea of combining adversarial training and generative objective is novel.\n3) Empirical results are promising.\n\n### Weaknesses\n1) Section 2.1 is not clear:\n- \"The model converges when it cannot distinguish between positive and negative samples.\": The model assigns energy to data points. How does it distinguish between positive and negative samples?\n2) It would be interesting to discuss the advantage of energy model over diffusion model (or flow-based generative models), which has a similar sampling function and has strong performance.\n3) Some previous/concurrent works also leverage CLIP feature to build a generative model, e.g., DALLE-2 or BLIP3-o. It would be helpful to discuss the relationship of this work w.r.t. them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces CLIP-JEM, a novel joint energy-based model extending to text-to-image generation by combining both generative and discriminative training objectives. It defines an energy function using the cosine similarity within CLIP's embedding space, which assigns low energy to real image-text pairs and high energy to mismatched pairs. CLIP-JEM significantly outperforms prior methods like CLIPAG in generating realistic images without relying on extensive augmentations, demonstrating superior performance on challenging compositional benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See previous section point 2 and 3.", "broader_impact_concerns": "Since this paper is proposing a generative model, it would be useful to discuss how to prevent misuse of the proposed method.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hUWCXv7ip5", "reviewer": "Reviewer_3D78", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The paper is generally well-structured and easy to follow.\n\nS2: The methodology design is well-motivated and shows valid improvement compared to related baselines such as CLIPAG.\n\nS3: Experiments somewhat demonstrate and emphasize the potential versatile usage of the proposed CLIP-JEM, such as the compositionality and the usage as an evaluation metric.\n\nW1: The existing evaluations lack the discussion on the diversity of generated samples. While the authors emphasize the general quality and alignment w.r.t. the text prompt, the ability to generate diverse images based on a single prompt seems to be excluded from current discussions. From the existing qualitative samples from Fig.8, the generated images given a specific prompt appear to be visually similar.\n\nW2: The exact formats of the contrastive adversarial objective are missing from Sec. 3.2. It is recommended to explicitly write down the equations as in Sec. 3.1 to improve the clarity.\n\nW3: The paper lacks specific details on the computational cost for training, while Tab. 4 and Tab. 5 show some training implementation details and sampling cost, what is the actual training time cost for those variants?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes CLIP-JEM to extend Joint-Energy-Models (JEMs) to visual-language domain using CLIP. Specifically, this is achieved by a combination of one generative and one discriminative objective. The generative objective extends the JEMs to a slightly different visual-language version; the discriminative one adapts adversarial training to the contrastive loss similar to the previous CLIPAG. The authors perform text-to-image generation experiments on MS-COCO, and test compositionality on T2I CompBench. They also used CLIP-JEM to improve the existing  CLIP-based generative frameworks, and propose to use CLIP-JEM as an evaluation metric with better sensitivity to perceptual quality with added noises.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to the W1, W2, and W3 above, the reviewer is also curious about one specific setting of ablation experiments. Specifically, the authors mention that CILPAG is heavily reliant on multiview augmentation, claiming that the proposed CLIP-JEM does not, and show a comparison with the case w/o multiview augmentation. However, does further adding multiview augmentation will help CLIP-JEM? Why or why not?", "broader_impact_concerns": "A Broader Impact Statement would probably be helpful despite the methodology focus, as this is a work on the generative models, and may produce unlicensed and potentially harmful content.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)