Whitened CLIP as a Likelihood Surrogate of Images and Captions
Authors: Roy Betser
Meir Yossef Levi
Guy Gilboa
Venue: ICML 2025 poster (2025)
Keywords: CLIP
latent space
likelihood approximation
multi-modal representation
Primary area: deep_learning->foundation_models
TL;DR: Direct likelihood approximation of images and captions based on CLIP's learned distribution.

Abstract: Likelihood approximations for images are not trivial to compute and can be useful in many applications. We examine the use of Contrastive Language-Image Pre-training (CLIP) to assess the likelihood of images and captions. We introduce Whitened CLIP, a novel transformation of the CLIP latent space via an invertible linear operation. This transformation ensures that each feature in the embedding space has zero mean, unit standard deviation, and no correlation with all other features, resulting in an identity covariance matrix. We show that the whitened embedding statistics can be well approximated by a standard normal distribution, allowing log-likelihood to be estimated using the squared Euclidean norm in the whitened space. The whitening procedure is completely training-free and uses a precomputed whitening matrix, making it extremely fast. We present several preliminary experiments demonstrating the properties and applicability of these likelihood scores to images and captions. Our code is available at github.com/rbetser/W_CLIP/tree/main.

Whitened CLIP as a Likelihood Surrogate of Images and Captions
Roy Betser' Meir Yossef Levi
Guy Gilboa
Abstract
Rare
Common
Likelihood approximations for images are not trivial to compute and can be useful in many ap- plications. We examine the use of Contrastive Language-Image Pre-training (CLIP) to assess the likelihood of images and captions. We intro- duce Whitened CLIP, a novel transformation of the CLIP latent space via an invertible linear oper- ation. This transformation ensures that each fea- ture in the embedding space has zero mean, unit standard deviation, and no correlation with all other features, resulting in an identity covariance matrix. We show that the whitened embeddings standard
Likely
Raw
Cencered
Whitened
Figure I. Raw, centered and whitened CLIP geometry. The whitened CLIP space is isotropic, transforming the original ellip- soid shaped space into hypersphere. In this space, the embed- ding norm reflects likelihood level. Higher norms correspond to lower probabilities.
statistics can be well approximated normal distribution, thus, the log-likelihood is estimated simply by the square Euclidean norm in the whitened embedding space. The whiten- ing procedure is completely training-free and per- formed using a pre-computed whitening matrix, hence, is very fast. We present several preliminary experiments demonstrating the properties and ap- plicability of these likelihood scores to images and captions. Our code is available here.
dual text-image semantics. However, its potential as a likeli- hood surrogate remains unexplored. This paper introduces Whitened CLIP (W-CLIP), linear whitening transformation of the CLIP latent space, where each feature standard- ized to have zero mean and identity covariance. In this whitened space, validate by statistical tests that the em- beddings approximate normal distribution, hence negative ligeliiel estimations are function of the Euclidean norm in the transformed space.
1. Introduction
To the best of our knowledge, this represents the first direct computation of likelihood functions for images and text prompts under the CLIP-learmed distribution.
Computing likelihoods for images is a challenging yet valu- able task with numerous applications in computer vision, such as image generation (Ramesh et al., 2022) and editing
Our main contributions follows:
(Kawar et al., 2023). Traditional approaches, including dif- fusion models, primarily rely the likelihood gradient score function, limiting direct likelihood computation (Ho et al., 2020; Song et al., 2020).
We propose Whitened CLIP (W-CLIP), based on an invertible lincar operation, allowing likelihood assess- ments while retaining the generative and semantic ca- pabilities of CLIP.
Contrastive Sagguage-Imagee Pre-training (CLIP) (Radford ct al., 2021) has become a widely adopted embedding for
We perform quantitative statistical experiments using Anderson-Darlin and D' Ccossino-PParsson tests, indi- cating the features in the whitened space can be well approximated by normal distribution.
Viierrbi Faculty of Electrical and Computer Engineering, Technion Israel Institute of Technology, Haifa. Israel.. Cor- respondence to: Roy Betser <roybe@cannee watpps.tecceoionnnn www.tionn.c..ciii Meir Yossef Levi www.lee@@aampus <guy. <guy.gilboa@e ec pe.technion.ac.l>>
@wwmuus wwweeeawwmscsssmaaaa..a. wwc.nion.ac.il...... Guy Gilboa
3. We are the first to propose direct computation of likelihood functions for images and text prompts, under the CLIP learned distribution. For images, to the best of our knowledge, this the first direct likelihood computation with semantic capabilities.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Real
AI with artifacts
Log likelihood
-1048.68
-2202.13
-2449.25
-2625.80
Log- likelihood
-1036.38
1226.23
-1220.85
-2536.27
Figure 2. log-likellhood of real and generated images with artifacts. Real images of a hand and a dog (left) and three similar Al generated images with artifacts. Real images have higher log-likelihood than generated images with artifacts.
4. We show W-CLIP can be used to estimate probability drifts in generative models, discover artifacts im- age generation and rank statistical deviation out-of- distribution (OOD) benchmarks, such as ImageNet-C and ImageNet-R, compared to in-distribution (ID) sets.
In natural language processing (NLP), large language mod- els (LLMs) estimate probabilities directly (Devlin, 2018; Brown et al., 2020), while vision-language models (VLMs), including CLIP (Radford al., 2021) and other recent mod- els (Desai al., 2023; Chou & Alam, 2024), embed images and text into shared space. Despite its success in enabling applications such as captioning (Mokady et al., 2021) and image manipulation (Kawar et al., 2023), CLIP's latent space remains underexplored. Known phenomena include the Narrow Cone Effect, where embeddings occupy limited angular space (Schrodi al., 2024) and the Modality Gap. where image and text distributions disjoint (Liang et al., 2022; Shi et al., 2023; Levi & Gilboa, 2025). Mokady et al. (2021) introduce mapping network to bridge the modality gap for image captioning.
5. For image manipulation, we use W-CLIP to extend Spherical Linear Interpolation (SLERP) by introduc- ing full-circle SLERP, enabling both interpolation and extrapolation between two given images.
2. Related Work
Estimating the likelihood of images, P(X), a fundamen- tal task with numerous downstream applications, including super-resolution (Li et al., 2022a: Gao al., 2023), denois- ing (Tian et al., 2020; Goyal al., 2020), and inpainting (Yu et al., 2018; Elharrouss et al., 2020). Early approaches relied on assumptions about natural image smoothness (Ge- man & Geman, 1984; Ruderman& & Bialek, 1993) and patch distribution (Zoran& & Weiss, 2011). Generative models such as Generative Adversarial Networks (GANs) (Goodfellow al., 2020). Autoencoders (AEs) (Hinton Salakhutdinov, 2006; Kingma, 2013), Energy-Based Models (EBMs) (Du & Mordatch, 2019; Ou al., 2024) and Diffusion models (Ho et al., 2020; Song et al., 2020) have further advanced image synthesis by implicitly estimating P(X). However, these methods do not provide explicit access to P(X); for instance, diffusion models approximate the score function,
To the best of our knowledge. this work is the first to analyze CLIP embeddings from probabilistic perspective and to propose leveraging its latent space as probability estimator, partieularly for the challenging domain of images.
3. Method: CLIP Likelihoods
3.1. Notations
Let X = {xi, f. xN be a set of N random vectors of dimension d, x, E Rd where μ = EE x, is the mean vector. We denote by 2, Z; f the centered vector, where X = (6)," f... Let Σ Rddd be the empirical
Vr log P(X), rather than P(X) itself.
2

---

Whitened CLIP Likelihood Surrogate of Images and Captions
0.08
ImageNet-A
ImageNet-C
Impuise Noise
0.06
0.2
Histogram
Histogramn
0.15
0.04
0.1
0.02
ImageNet
ImageNet
0.05
Norm
20
30
40
50
70
Norm
ImageNet-R
0.15
Embroidery
Painting
Origami
0.4
Graffiti
Sketch
VideoGame
.05
ImageNet
30
Norm
60
Figure 3. Norm histograms of ImageNet variations. Top left: ImageNet-A, comprising of natural adversarial examples, closely aligns with clean ImageNet due to their natural origins. Top right: ImageNet-C. histograms under varying impulse noise levels of severity display significantly larger norms than clean ImageNet, indicating distributional deviations. Bottom: ImageNet-R comparison shows that different styles cause varying likelihood shifts, with graffiti closest to real images and video game renditions exhibiting the largest shifts.
covariance matrix of X:
0, E = Il) (see App. D.2). The inverse transform from the whitened space to the original space is performed simply by x = W W-1y yy + μ and in matrix notation = W- + 4- 1, where E R X is a row vector of 1's.
Σ=
N XXT
We recall that the covariance matrix is symmetric, positive semi definite and that the diagonal contains the variance of each feature in the vector.
Given a of raw CLIP embeddings, The whitening proce- dure offers three key advantages:
1.
W is obtained in purely data-driven process, without additional meta- meta-parameters.
3.2. Whitening transform
Given a set of random vectors with a non-singular covari- ance matrix E, let W E dd be a matrix that satisfies WTW 1 We note that W is not unique. common way to obtain by principal component analysis (PCA). Let diagonalize the covariance matrix Σ VAVT, where A E Rdxd is diagonal matrix of the eigenvalues, A, and V E ddd consists of the corresponding eigenvectors. Then, the whitening matrix W can be defined as:
Since the transform is invertible, all existing applica- tions developed in the raw embedding space can be seamlessly integrated with this approach.
The computation of W is performed only once and priori, based on representative dataset. Memory and computational requirements are very mild, allowing efficient use also in low-resource settings.
W = A
(2)
It is known that the CLIP latent spaces of images and cap- tions are disjoint (Liang et al., 2022; Levi & Gilboa, 2025). Therefore, we treat the distribution of each modality in- dependently. Additional implementation details and the complete whitening algorithm are App. D.1, Alg. 1.
Note that W is an invertible matrix. A single vector x is whitened by y = W and the whitened matrix E MdxN
corresponding to the raw measurement matrix X is
= W
3.3. Whitened CLIP embeddings
Y consists of isotropic random vectors, that is, each vector has zero mean and an identity covariance matrix (jty =
Our likelihood estimation relies on modeling the whitened CLIP space as following approximately independent and
3

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Independent
Identically Distributed
Uniformity
Diagonal Score: 0.02
Diagonal Score: 0.98
leb.
400
Raw CLIP
Contened
400
1.5
Whirosed
Frequency
200
200
0.00
0.75
(a) Raw CLIP Covarlance Matrix
(b) W-CLIP Covarlance Matrix
(C) Raw CLIP Features
(d) W-CLIP
(e) Cosine Simllarities
Features
Figure 4. Raw CLIP and W-CLIP analytic comparison. The covariance matrices of raw CLIP (a) and W-CLIP (b) demonstrate the effectiveness of the whitening transformation achieving unit variance and zero correlation among features. Histograms four CLIP features (c) vary in mean and variance, whereas four W-CLIP features (d) exhibit zero mean and unit variance. Cosine similarity histograms for image pairs (e) across raw, centered, and W-CLIP embeddings reveal that W-CLIP'ss cosine similarity is concentrated around zero, indicating significantly improved uniformity compared to the centered and raw CLIP spaces.
identically distributed (i.i.d) standard normal distribution. We thus first examine the validity of this approximation.
measuring the proximity of matrix being diagonal (in the range [0, 1] with 1 being exactly diagonal) is
Normal distribution tests. To assess how well the whitened embeddings approximate normal distribution, we employ two statistical tests: Anderson-Darlingg (Anderson & Dar- ling, 1954) and D' Agostino-Pearson Pearson (D' "aggsstno & Pearson, 1973). The Anderson- -arliin test evaluates how well the empirical cumulative distribution function (CDF) matches the expected CDF of normal distribution, placing higher weight on the tails to detect deviations. The D' Agostino- Pearson test combines skewness and shape characteristies measures to assess normality, offering sensitivity to both symmetric and asymmetric deviations. See App. D.3 for additional details regarding these tests, specifically Eq. (12), Eq. (13). For stability, the 5000 embeddings of MS-COCO validation set (Lin et al., 2014) are divided into 20 equal groups of 250 samples each. As shown in Tab. 1, the results validate that normal distribution is good approximation for both image and text embeddings. Specifically, in both tests, more than 90% of the text features, and more than 98% of the image features conform to a normal distribution, with average scores that satisfy the test criteria by a large margin. Additional details regarding these tests, empirical statistics, and plots are provided in App. D.3.
Σ, Edd Ee, Σ
Diagonal Score
(4)
where Σi,j is an element at row i and column j of the covari- ance matrix. Scores of the full matrices verify that, provided the normal distribution model is valid, the independence assumption holds as well. Regarding the latter, in Fig. 4 the CLIP features exhibit varied mean and variance values (c), while W-CLIP features have all zero mean and unit variance (d), see further results in Fig. 18, App. D.3. Consequently, the whitened embeddings can be approximated reasonably well as i.i.d. features.
Table f. Anderson-Darlinne and D' Agestino-Pearson scores for image and text embeddings. The Avg. column contains the average score for all features, and Normal Features represents the percentage of features passing the normal distribution test based on their average score. The threshold score (in brackets) indicates the required condition for normal distribution, with the sign showing whether higher or lower results imply normal distribution.
Avg. Normal Features
Anderson-Darlinn ( <<] 0.752)
Independent and identically distributed (i.id). Let us break this assumption into independence and identical dis- tributed conditions. For the former, in normal distribution, non-correlation is sufficient condition for independence. In Fig. 4, the 20 first features of the covariance matrices of raw CLIP embeddings (a) and W-CLIP embeddings (b) are presented. While the CLIP embeddings exhibit correlations between features, the covariance matrix of the whitened embeddings is almost exactly diagonal, indicating that the features are uncorrelated. This is expected, since the whiten- ing transform designed for exactly this purpose. A metric
Image
0.4890
98.3%
Text
0.5926
90.1% %
D' D'Agostino-Pearsee
>> 0.05 )
Image
0.3624
99.3%
Text
0.2568
99.2%
3.4. Log probabilities using W-CLIP
Embedding likelihood. The explicit likelihood of a d- dimensional random vector, x, with i.id standard normal variables is:

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Log- likeiihood
L likethoodd
"Donald Duck talking to Minnie Mouse at Disney World at Christmas."
Jenny points hair drier like it is gun."
-2334 -233.001
Donald DUCK talking to Mouse Disney Word at Chrisemas.
"A bride points nair drier like it is a gun."
-2151.33
-2125.64
DucK calking Minnie Mouse Disney World at Christmas."
Likelihood
"An old woman points a hair drier like it gun.
2586.75
Norm
-2101.06
-2473.99
Donald Duck talking Minnie Mouse Christmas."
"A woman points nair drier like it is gun."
-2387.06
-1934.67
Figure 5. Likelihood variation for different levels of details. The original MS-COCO caption is marked with with deviations underlined. Left: Removing details, such as character names or locations, increases likelihood. Right: Adding specificity, such as replacing "woman" with "bride" "Jenny", decreases likelihood.
space. We denote the norm of x by S V likeliho of is:
27. The
1 (2T)²
exp (-21le1p)
P(x)
(5)
log(P(S)) C(d) + (
2 - log(52) - $2 (7)
where ||x||| x x. The log-likelihood is:
2
where C(d) = - log (2())-1(())) and I is the Gamma
(x))= = log P(x)
(d + ||x]1))
(6)
function. The expected value and standard deviation of are:
Thus, we propose ((x) to be an approximation of the log likelihood of image or caption instances, based on the W- CLIP embedding. To the best of our knowledge, this is the first method to directly obtain a probability score for image text embeddings using CLIP and the first probability com- putation for images which is not based on low-level patch statistics but on high-level semantics. In contrast, natural language processing (NLP) language models can directly approximate the negative log-likelihood (NLL) for text prompt. The relationship between our log-likeliihe likelihood mea- sure and those of language models is discussed in Sec. 4.4.
r d++
E|S] = v2
Std(s) = V 2gz.
(8)
T (
For large d, [E[S] + v d The comparison between theoretical and empirical measurements in Tab. 2, based on MS-COCO, reaffirms the assumed framework of normal dis- tribution. The mean and standard-deviation of the whitened image embeddings closely align with the expected values, while the text embeddings exhibit slightly greater deviation; trend consistent with the results in Tab.
Norm distribution in W-CLIP. According to Eq. (6), the norm is directly related to the log- likelihoodoo We thus high- light some consequences and recall the distribution of norms under standard normal statistics. We first note that the most probable sample resides at the center the whitened em- bedding space, nevertheless, the likelihood of sampling this singular point out of the entire space zero in practice. In general, high- dithnnmmenii normal distributions have close to zero mass near the origin. This follows phenomenon called Thin Shell (App. D.4), which reveals that the major- ity of the distribution is concentrated near the surface a sphere of radius Vd. The chi distribution (xa), is the appro- priate model for the distribution of norms in the whitened
Table 2. Empirical and theoretical measurements, For d 768; relative deviation of the empirical (Emp.) from the theoretical (Theo., Eq. (8)) values are shown in brackets.
Mean (Emp. / Theo.)
Std (Emp. / Theo.)
Image Text
27.43 27. 22443/27.702 27.7
0091/3.9600.555)
28.49 28.49/27.7(2.855))
5.72/6.60 (13.24%))))))))1
4. Experiments
All the experiments in this section employ the CLIP VIT-
I/14 model and utilize the MS-COCO validation set to
5

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Step O
Step 5
Step 10
Step 25
Step 20
Step 30
5000
Original likelhood
Frequency
Raw
2500
Norm
-1000
LieeLiielldoodd Difference
(a) Image
(b) Iterative Generation Bias
Generation Bias
Figure 6. Bias in image generation. Left: Using CLIP-encoded images in MS-COCO validation set as a condition for generating new images. The histogram shows a bias towards lower likelihoods in generated images. Right: Iteratively using UnCL.IP to generate images encoded by CLIP with fixed sced. The raw process gradually becomes noisy (Top), whereas with normalization (to vd at cach encoding step). the content drifts but remains within the natural reasonable image space.
compute the whitening matrix W.
(Deng et al., 2009), as presented in Kan et al. (2018), in comparison to ImageNet-A (Hendrycks et al., 2021b), ImageNet-C (Hendrycks & Dietterich, 2019), and ImageNet- R (Hendrycks et al., 2021a). Here we show the distribution of norms of each set, instead of the likelihood estimation. Following Eq. (6), we have
4.1. Attributes of W-CLIP
Text complexity. Our observation is that more complex and specific words, such as names, are expected to yield lower likelihood scores. In Fig. 5 we show results of caption editing. Words are replaced with either generic or specific terms, for example, by adding removing names. The likelihood scores adjust accordingly, decreasing for more specific terms and increasing for more generic ones. Addi- tional examples are available in Figs. 25, 26, App. H.
||>||| V 22(x()) d log()
where {(x) is the log- likeliillhood estimation of x. Thus is simple monotonic transformation, which in some cases may serve alternative, more convenient, visualization. One should notice that a higher norm indicates lower likelihood. ImageNet-A A consists of natural adversarial images. Since the images are natural, their norm distribution is similar to that of ImageNet, apart from a slight shift toward higher values. ImageNet-C introduces common corruptions (e.g.. impulse noise), with higher noise levels corresponding to lower likelihoods and distributions consistently below Im- ageNet. ImageNet-R assesses robustness to domain shifts with renditions like art, graffiti, and video games. Rendi- tions closer to real images, like graffiti, have lower norms than video games, but all renditions exhibit higher norms than ImageNet. For additional Imagenet-C corruptions see Fig. 11, App. B.3.
Uniformity enhancement. An additional desirable prop- erty promoted naturally in the whitened space is uniformity (Wang & Isola, 2020). Fig. 4.e presents a histogram of cosine similarities between all possible image pairs (pre- dominantly composed of negative examples) in the raw, centered and whitened spaces. In the whitened space, co- sine similarities are concentrated near zero with smaller variance. contrast, the centered space exhibits higher variance, while the raw CLIP space has similarities centered around 0.5 with high variance. These results indicate that the whitened CLIP distribution is more uniform.
4.2. Data Analysis using W-CLIP
Artifact detection. An important attribute of any image likelihood function is its capacity to discriminate between authentic and synthetic images, with particular emphasis on identifying artifacts present in synthetic counterparts. In Fig. 2, we compare the likelihood of real images, and Al-generated ones from the SynArtifact dataset (Cao et al., 2024) containing notable artifacts. All generated images have lower likelihoods than their real counterparts. Addi- tional examples provided in Fig. 9, App. B.1.
4.3. Image manipulations
Image generation bias and variance. Generative models may produce outputs which are more likely or less likely than intended. Here we give an example how this can be quantified, allowing to obtain likelihood- and likelihood- variance of a generator, as shown in Fig. 6. Image generation was performed using UnCLIP (Ramesh et al., 2022), condi- tioned by a CLIP embedding. In this experiment, each image from MS-COCO validation set was encoded using CLIP and subsequently ten images were generated by UnCLIP with
Domain shift. Fig. 3 evaluates a subset of ImageNet
6

---

Whitened CLIP Likelihood Surrogate of Images and Captions
w/o nours
A group Of men standing around à table OF food in
Text Data
pouring a Into a*
Caption
Frequency
Ours Frequency
"Baseball players in action, one person dressed in green running while the others in white are looking and reaching toward the ball."
wooden room.
-d5000
Loe-LInellhodd
1000
7000
Los-L
-1000
Iiveiinod D
Frequency
man pouring drink into a wine glass."
we nours
with nouns
200
"Tha''s only one ciny piece OF the puzzie."
Caption Data
Text Data
Normallized Log: L
GPT2
Frequency
OPT
.4
Log Likethhood (a) Removing Nouns
Log Lkeeloodd
10
Words
25
(D) Text Type
(C) Caption Length
Figure 7. Differences between likelihood functions. Our proposed likelihood estimation is highly sensitive grammatical errors (a), demonstrated by the removal of all nouns from the captions, and text type (b), where Text Data refers to a general text dataset (OpenWebTex)) and Caption Data refers MS-COCO captions. However, it remains less sensitive caption length (c). In contrast, language models are highly sensitive to caption length and treat captions as being within the distribution of general text. The removal of nouns from the captions causes only negligible changes the overall distribution of the model's likelihood. Histograms of all language models are in Figs. 12, 13, App. C.
different random seeds. A histogram showing the likelihood differences between the original and the generated embed- dings is provided in Fig. 6.a. The results demonstrate clear bias towards lower likelihoods. This result indicates that our likelihood approximation method can potentially be lever- aged generated image detector. Thorough investigation of this task will be conducted in future work.
tween them (calculated the normalized dot product), and [0, 1] is the interpolation step. SLERP assumes em- beddings lie on a hypersphere (Liang et al., 2022; Wang & Isola, 2020) and mathematically valid for beyond [0, 1]. Fig. 20 (App. F) illustrates SLERR on a 2D circle. When one point off the circle, interpolation forms an ellipse the perimeter; if shifted from the origin, the ellipse deviates significantly further. Full-circle SLERP uses an interpolation degree with CLIP space, full-circle SLERP often produces noise, with reasonable images only near and between the original em- beddings. In the whitened space, it generates consistent images across all angles, with semantic diversity, indicating embeddings remain within the distribution. Images from full circle SLERP examples are in Fig. 8 and App. F (Fig. 21). In Fig. 8, at 300 degrees, which is extrapolation of the source embedding to the reverse direction from the destination em- bedding, the dog with a bottle of bear (source embedding) becomes a man sitting next to bottles of bear. This is an interesting extrapolation result, not specifically guided. In order to further evaluate this phenomena, and quantify to quantitative measures perform an additional exper- iment. Using MS-COCO validation set, for each image, we performed full-circle SLERP in both the raw CLIP and W-CLIP embedding spaces. In this process, a source image is interpolated toward a destination image along circu- lar path within the embedding space. Crucially, the image generated at the 180" position from the source-referred to the "opposite image" (generated from the "opposite
To further understand this phenomena, we implemented iterative sequence, starting with real image encoded by CLIP. The resulting embedding is used to generate a new image, which is re-encoded to CLIP. This iterative process caused the generated images to drift in content, quickly de- grading into noise. According the chi distribution, an embedding is most likely to have norm of approximately vd (Eq. (8)). We use this to normalize each embedding in W-CLIP to norm of Vd and project back to CLIP space, mitigating this issue. While the iterative process still has a semantic drift, reasonable images were consis- tently produced. Fig. 6.b illustrates this process. Additional experiments are provided in App. (Figs. 23, 24).
in Eq. (9). In the raw
Full circle SLERP. Ramesh al. (2022) propose spheri- cal interpolation (SLERP) on image CLIP embeddings to interpolate between images. SLERP is defined as:
SLERP((; E,, E2) sin (1 t)0) sin( E,
t)0)
sin(to) (i)
Es, (9)
where Ej and Eg are embeddings, 0 the angle be-

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Source
Source
30
330
300
59.6
Destination
Destination
200
220
200
120
(b) W-CLIP
(a) CLIP
Figure 8. Full cirele SLERP example. The full circle SI.RR is performed in both the raw CLIP space (a) and in the W-CLIP space (b). The different angle between embeddings in hoth space is presented. In the raw CLIP space the full circle SL.RR results with noise for most of the degrees not between the source and destination embeddings. In the W-CLIP space for all degrees real images are generated.
umbedding")-13 invariant to the chosen destination and de- termined solely by the source. While other positions along the path are influenced by the destination embedding, the opposite embedding is a fixed, symmetric counterpart.
sequence likelihood under the model's learned distribution (Bishop & Nasrabadi, 2006; Murphy, 2012). Our proposed log-likelihood score (Eq. (6)) approximates the likelihood of text prompts based on single embedding vector for the entire prompt. We evaluated MS-COCO validation set cap- tions using our method and various language models. Both LLMS (GPT-2 (Radford et al., 2019), NEO (Black et al., 2021). OPT (Zhang et al., 2022)) and VLMs (BLIP (Li et al., 2022b), GIT (Wang et al., 2022)) were tested. Our method computes log-likelihood values in a different range of values, with correlation values between 0.33 and 0.48 with all language models. See Fig. 14, App. C, for full de- tails. In Fig. 7, we highlight three main differences between our likelihood score and those of language models:
We generate these opposite images using both CLIP and W-CLIP embeddings and observe a stark contrast: in the CLIP space, opposite images degrade into structured noise, whereas in the W-CLIP space, they remain visually natural and semantically meaningful, as shown in Fig. 8. The struc- tured noise produced by CLIP exhibits 4x4 pixel blocks and restricted color palette, suggesting synthetic artifacts. We provide a large visual example in App. F (Fig. 22).
To quantify these differences, we compute Total Variation (TV), Entropy, and the percentage of extreme saturation values (top or bottom 1% of the pixel range). All metrics are computed per channel and averaged per image across three sets: original MSCOCO images, CLIP opposites, and W-CLIP opposites. The results summarized in Tab. 4. These findings confirm that W-CLIP opposites are statis- tically similar to natural images, whereas CLIP opposites exhibit significantly reduced entropy and variation and much higher percentage of saturation values, indicating a lack of natural structure.
Caption length. All language models approximate lower mean likelihood scores as caption length in- creases. In contrast, our likelihood score is less sen- sitive to caption length, Fig. 7.c. Levy et al. (2024) recently demonstrated degradation in LLM perfor- mance on long inputs, particularly in reasoning tasks.
2. Text type. While text models are trained on general text, CLIP is trained specifically on captions of im- ages. We sampled 5,000 sentences from OpenWeb- Text (Gokaslan et al., 2019), a general text dataset, ensuring that their lengths are comparable to those of MS MS-COCO captions, and compared both likelihood histograms. For L.MMs captions align with the general text distribution, whereas VLMs and our method result in separable distributions (Fig. 7.b).
4.4. Relations to language model probabilities
In natural language processing (NLP), large language mod- els (LLMs) minimize the negative log-likelihood (NLL) during training learn a probability distribution over se- quences. At inference, the NLL is computed by summing the negative log probabilities of each token in the prompt, conditioned on previous tokens. The final NLL is averaged over all tokens, with lower NLL scores indicating higher
3. Grammatical errors. Wu et al. (2023) demonstrates that ChatGPT performs poorly on datasets containing grammatical errors, particularly on long sentences. On
8

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Table Cross dataset comparison. COCO: MS-COCO. F8k: Flickr8k. Data T: dataset used tests, Data dataset used calculate the whitening transform. Avg. AD, DP: the average Laderson-Darling, D' - Pearson p-value test scores (threshold is under 0.752
above 0.05, respectively). Correlation is calculated between likelihood scores on test data using different whitening matrices.
Image
Text
Data T
Data W
Avg. AD
Avg. DP Correlation
Avg. AD
Avg. DP
Correlation
COCO
0.489
0.362
0.592
0.257
COCO
0.69
0.74
F8k
0.466
0.380
0.574
0.282
COCO
0.641
0.317
0.735
0.226
F8k
0.77
0.88
F8k
0.522
0.329
0.626
0.242
4.5. Data generalization
Table Opposite image comparison. Total Variation (TV), tropy and percentage of saturation pixels (SAT[&]) for natural images (MS-COCO) and the opposite images generated using CLIP and W-CLIP embedding spaces. W-CLIP represents better the statistics of natural images.
As W-CLIP is completely data-driven we test its generaliza- tion capabilities. Flickr8k (Hodosh et al., 2013), similarly to MS-COCO, is benchmark for image- that emphasizes real-world imagery and descriptive diver- sity. In Tab. 3. we compare results using MS-COCO and Flickr8k as both the whitening and testing datasets. We eval- uate the normal distribution test scores Landerson-Darling and D' Agostino-Pearson) as in Sec. 3.3, and the correlation of likelihoods computed for the same data, using different datasets for whitening. The results show that whitening with one dataset and testing on another yields similar nor- mal distribution test scores for features, and moderate to high correlations between likelihoods. Additional ablation studies, including different dataset size and utilizing a dif- ferent CLIP model, are provided in App. E. These findings confirm that, although W-CLIP is data-driven, it generalizes well across datasets within the same domain. However, as shown in Fig. 3, W-CLIP is sensitive to domain shifts.
tasks
Method
TV
Entropy
Sat [ [%]
MS-COCO
222.3
7.3
4.2
CLIP
156.7
4.8
55.5
W-CLIP
215.9
7.2
6.4
the other hand, we noticed our method sensitive to grammatical errors and nonsensical inputs. To test this, remove all the nouns from the MS-COCO captions and compare likelihood before and after. Our likeli- hood score is significantly affected, while the language models demonstrate less sensitivity, Fig. 7.a.
In Figs. 12, 13, App. C, histograms of all other language models are available. In Table 5, we quantify the separation between the likelihoods of different data types and captions with and without nouns. We employ the AUC metric, as defined in Eq. (10) (App. C), to evaluate the separation be- tween distributions. Additional text examples are provided in Fig. 15, App. C. It is shown that the likelihood approx- imated using W-CLIP positively correlates with language model likelihoods but contains unique information derived from CLIP's learned distribution.
5. Conclusion
This paper introduces Whitened CLIP, transforming the raw CLIP latent space into an isotropic space. Whitened CLIP is statistically verified approximate well normal distri- bution with independent and identically distributed (IID) components, and exhibits enhanced uniformity. The key contribution of this work is the proposal of direct compu- tation of likelihood functions for images and text prompts within the CLIP-leamed distribution. Embeddings in the whitened space approximately follow the standard normal distribution, enabling the use of the squared Euclidean norm to estimate log-likelihood. These likelihood functions ef- fectively identify artifacts, domain shifts, and demonstrate sensitivity to the complexity of details in text captions. Bi- ases in generative models can be detected by comparing the likelihood of generated images to those of real images. Furthermore, the introduction of full-circle SLERP in the whitened space facilitates both interpolation and extrapola- tion between images. We believe the results of this research can further benefit numerous applications.
Table 5. Likelihood separation with grammatical errors and different text types. AUC values indicating the separation be- tween likelihood distributions. Type compares the separation be- tween captions and general text prompts, while Nouns compares the separation hetween original captions and the same captions with nouns removed. Vision-language models (VLMs) show a high separation for different text types and slightly higher separation when removing nouns compared to language models (LLMs). Our method yields the best separation, especially for Nouns.
LLMs
VLMs
Ours
GPT2
OPT
NEO
BLIP
GIT
Type
0.8
0.8
0.77
0.92
0.97
0.999
Nouns
0.43
0.58
0.58
0.66
0.69
0.94
9

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Acknowledgements
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 2018.
We would like acknowledge support by the Israel Science Foundation (Grant 1472/23) and by the Ministry of Science and Technology (Grant No. 5074/22).
Du, Y. and Mordatch, I. Implicit generation and modeling with energy based models. Advances in Neural Informa- lion Processing Systems, 32, 2019.
Impact Statement
Elharrouss, O., Almaadeed, N., Al-Maadeed, S., and Akbari, Y. Image inpainting: A review. Neural Processing Letters, 51:2007-2028, 2020.
This work advances Machine Learning by enhancing the understanding foundation model (CLIP) and utilizing it derive direct likelihood functions for images and captions. Potential societal consequences of our work are related to downstream tasks that may rely on our findings. None of these consequences need to be specifically highlighted here.
Gao, S., Liu, X., Zeng, B., Xu, S., Li, Y., Luo, X., Liu, J., Zhen, X., and Zhang, B. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10021-10030, 2023.
References
Anderson, T. W. and Darling, D. A. A test of goodness of fit. Journal of the American statistical association, 49 (268):765-769, 1954.
Geman, S. and Geman, D. Stochastic relaxation, gibbs dis- tributions, and the bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelli- gence, 6:721-741, 1984.
Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006.
Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Open- webtext corpus. http://sky ion007.goo gith.b.. Lo/openneebTextcorpuss 2019.
Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. you use this software, please cite it using these metudata, 58(2), 2021.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neclakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:
Goyal, B., Dogra, A., Agrawal, S., Sohi, S., and Sharma, A. Image denoising review: From classical state-of-the art approaches. Information fusion, 55:220- 2020.
1877-1901, 2020.
Groeneveld, R. A. and Meeden, G. Measuring skewness and kurtosis. Journal of the Royal Statistical Society Series D: The Statistician, 33(4):391 1-399, 1984.
Cao, B., Yuan, J., Liu, Y., Li, J.. Sun, S., Liu, J., and Zhao, B. Synartifact: Classifying and alleviating artifacts in syn- thetic images via vision-language model. arXiv preprint arXiv:2402. 18068, 2024.
Hendryeks, D. and Dietterich, T. Benchmarking neural network robustness common corruptions and perturba- tions. arXiv preprint arXiv:1903.122611 2019.
Chou, C.-C. and Alam, N. Embedding geometries of contrastive language-image pre-training. arXiv preprint arXiv:2409. 13079, 2024.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of dittibtiintt generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3340-8349, 2021a
D' "agostino,, R. and Pearson, E. S. Tests for departure from normality, empirical results for the distributions of i and Vb. Biometrika, (013):61-622. 1973.
Deng, J., Dong, W. Socher, R., Li, L.JJ,, Li, K., and Fei-Fei, L. Imagenet: A lcage-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Iece, 2009.
Hendrycks. D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
Desai, K., Nickel, M., Rajpurohit, T., Johnson, J., and Vedantam, S. R. Hyperbolic image-text representations. In International Conference on Machine Learning, pp. PMLR, 2023.
Hinton, G. E. and Salakhutdinov, R. R. Reducing the di- mensionality of data with neural networks. science, 313 (5786):504-50) 2006.
10

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Mokady, R., Hertz, A., and Bermano, A. H. Clip-
cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 2021.
Hodosh, M., Young, P., and Hockenmaier, J. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47: 853-899, 2013.
Murphy, K. P. Machine learning: perspeii MIT press, 2012.
Ou, Z. et al. Energy-based models with applications to speech and language processing. Foundations and Trends in Signal Processing, 18(1-2):1-199, 2024.
Kan, w., Howard, A., and Park, E. Im- agenet object localization challenge. In
Paouris, G. Concentration of mass on convex bodies. Geo- metric & Functional Analysis GAFA, 16(5):1021-1049, 2006.
objertlocalicatinn-calllleees 2018.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagie: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pullern Recognition, pp. 6007-6017, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are un- supervised multitask learners. OpenAl Blog, I (8), 2019. URL https://cdn. com/ better- htw.:-aannggge--mddlll/////agnoo models aree_are____ _et_aarrr__u___ee learners.pdf.
Kingma, D. P. Auco-ingg variational bayes. arXiv preprint 13/2.61111211///1 14,66/11120 2013.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Levi, M. Y. and Gilboa, G. The doublc-ellipsoid geometry of clip. International Conference on Machine Learning. PMLR, 2025.
Levy, M., Jacoby, A., and Goldberg, Y. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14844 2024.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Li, H., Yang. Y., Chang, M., Chen, S., Feng, H., Xu, Z., Li, Q., and Chen, Y. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59,
Ruderman, D. and Bialek, W. Statistics of natural images: Scaling in the woods. Advances in neural information processing systems, 6, 1993.
Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., and Brox, T. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive languuel representation learning. arXiv preprint arXiv:2404.07983, 2024.
Li, J. Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-iman image pre-training for unified vasion-language understanding and generation. International confer- ence on machine learning. pp. 12888-12900. PMLR, 2022b.
Shi, P.. Welle, M. C., Björkman, M., and Kragic, D. Towards understanding the modality gap in clip. ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Lin, T.-Y. Maire, M., Belongie, S., Hays, Perona, P., Ra- manan. D.,, Dollár, P.. and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Tian, C., Fei, I.., Zheng, W., Xu, Y., Zuo, W., and Lin, C.-W. Deep learning on image denoising: An overview, Neural Networks, 131:251-275. 2020.
Vershynin, R. High-Dimensie
Probability: An Intro-
duction with Applications in Data Science. Cambridge University Press, 2018.
Mardia, K. and Jupp, Directional Statistics. Wiley, 2000.

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. Git: A generative image-to- text transformer for vision and language. arXiv preprint arXiv:2205.141000 2022.
Wang, T. and Isola, Understanding contrastive represen- tation learning through alignment and uniformity the hypersphere. In International conference machine learning, pp. 9929 9939. PMLR, 2020.
Wu, H., Wang, W. Wan, Y., Jiao, W., and Lyu, M. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arxiv, 2023.
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. S. Generative image impainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint ar arXiv:2205.01005 2022.
Zoran, and Weiss, From learning models of natural image patches to whole image restoration. In 201 I in- ternational conference on computer vision, pp. 479-486. IEEE, 2011.
12

---

Whitened CLIP Likelihood Surrogate of Images and Captions
A. Reproducibility
Our code along detailed instructions is available HERE. The repository includes: 1) Implementation of our method, reproducing most of our experiments, including simple demo notebooks: Whitening matrices; 3) Additional examples beyond those in the paper and in the appendix, specifically video demonstrations the full circle SLERP.
B. Artifacts and Domain Shifts Examples
B.1. Image artifacts
In Fig. 9 we offer additional examples of real images compared to similar generated images with artifacts, as presented in Fig. 2 in Sec. 4.2.
B.2. Text artifacts
Trying to generate artifacts in text captions we remove the first or last words from a caption, or one of the middle words. Examples in Fig. 10. all cases the original caption has the highest log-likelihood score.
B.3. ImageNet datasets
In Fig. 11 we provide histograms using different corruptions from ImageNet-C. All corruptions have a lower log-likelihood compared to ImageNet.
Real
AI with artifacts
Log likelihood
-865.48
-948.86
-962.55
-1009.45
Log likelihood
-925.92
936.91
-1043.53
11205.59
Figure 9. Log-likelihood of real and generated images with artifacts. Real images of zebras and a surfer (left) and three similar AI generated images with artifacts. Real images have higher likelioosos than AI generated images with artifacts.
13

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Caption
Log likelihood
Caption
Log likelihood
"A young Child and cat in living room."
"A teddy bear leaning against a tree next to the road."
-994.03
-975.17
Young Child and Cat in living room."
-1052.18
"A teddy bear leaning against a next to the road."
-1001.55
"A young Child and Cat in a_ _
1138.67
"A teddy bear leaning against tree next to the _
1039.10
"A young Child and in a living room."
"Teddy bear leaning against a tree next to the road."
-1228.79
-2046.25
Figure 10. Log-likelihood of real captions and captions with artifacts. Real captions, framed with blue frame and atrifacted captions, where we removed the first or last words from a caption, or one of the middle words. In all cases the original caption has the highest ligeliill score.
C. Comparison with Language Models Examples
In Fig. 14, we present the log-likelihood values for the MS-COCO validation set using language models, along with the correlation between these log-likelihoods and those computed using our method. Our approach yields log-likell eg-likklihhood scores with larger absolute values and greater variance, aligning with its intended design.
Figs. 12, 13 display histograms replicating the experiments from Figs. 7.a,b for additional models. The LLMs (OPT, NEO) exhibit behavior very similar to GPT-2. The VLMs (BLIP, GIT) also show behavior similar GPT-2, but with some deviations trending toward method's likelihood. This observation is reasonable, these models, like CLIP, trained (or fine-tuned) caption data rather than general text data. In Fig. 15, we present additional examples of captions with varying relative likelihood scores. We sort the likelihood scores of 5,000 captions from MS-COCO in ascending order and examine the sorted index for different captions. The comparison includes our method, an LLM (GPT2), and a VLM (BLIP). Each set of examples demonstrates one of the three differences discussed in Sec. 4.4 (e.g. text type, grammatical errors and caption length).
The Area Under the Receiver Operating Characteristic Curve (AUC-ROC, AUC for simplicity) evaluates the ability of model to distinguish between two classes. It measures the trade-off between the positive rate (TPR) and the false positive rate (FPR) at various threshold levels. The AUC score in Tab. 5 is mathematically defined as:
AUC = / TPP(FPR (FF(RFFR))))))) dx
TP
FP
TPR
FPR
(10)
TP FN
FP+
where TP are true positives, FP are false positives, TN are true negatives and FN are false negatives. the context of Tab. the MS-COCO captions are defined positives and the general text or captions without nouns are defined as negatives.
14

---

Whitened CLIP a Likelihood Surrogate of Images and Captions
0.1 ImageNet-C
Brightnegs
ImageNet-C
Contrast
0.1
Histogram
Histogram
0.05
0.05
ImageNet
ImageNet
20
30
40
50
60
70
20
30
40
50
60
70
Norm
Norm
ImageNet-C
Defocus Blur
0.1
ImageNet-C
Transf orm
Elastic
Histogram
0.1
Histogram
0.05
ImageNet
0.05
ImtgeNet
20
30
40
50
60
70
20
Norm
Norm
0.1
Nete
Fog
0.1
ImageNet-C
Frost
Histogram
Histogram
ImageNet
0.05
TetgeNet
0.05
20
20
30
40
60
80
Norm
Norm
ImageNet-C
Glass Blur
0.15 ImageNet-C
JPEG
0.1
Compression
Histogram
Histogram
0.1
NmtgeNet
0.05
0.05
IaageNet
20
30
50
60
80
30
40
50
60
Norm
Norm
ImageNNtt-C
Motion Blur
0.1
ImageNet-C
Pizelate
Histogram
Histogram
0.05
NmageNet
0.05
ImageNet
0
20
30
40
50
60
70
20
30
40
50
60
70
Norm
Norm
ImageNet-C
Snow
0.1
ImageNet-c
Zoom Blu
Hislogram
Histogram
ImageNet
0.05
0.05
Net
0
20
30
40
50
60
7C
20
30
40
50
60
70
80
Norm
Norm
Figure 11. ImageNet-C histograms all corruptions. All corruptions have significantly higher (lower log-likelihood) than ImageNet. For most corruptions, as level of corruption increases the norm increases. Some corruptions do not show this monotonic behavior (motion/glass blur for example) different levels of corruption.
15

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Text Data
Text Data
Frequency
Caption Data
200
Caption Data
200
Caption
LLMs
930
930
Log-Likeiihood (a) OPT
Log-Llkeilhood (b) NEO
-00
Log-Likellhood (C) GPT2
Text Data Caption Data
300
VLMs Frequency
200
Caption Data
200
Caption Data
LikeLikeidoood
Log-Likelihood (e) BLIP
7000
Log-Likelihood (f) Ours
-1000
(d) GIT
Figure 12. Likelihood for different text types. Comparing likelihood values computed for MS-COCO captions and OpenWeTText general text sentences. The sentences from OpenWebText filtered to have similar lengths to MS-CMCS captions. LLMs (OPT, NEO, GPT2) treat captions similarly to general text while VLMs (BLIP, GIT) show some separation. No model shows strong separation like our likelihood does (Tab. 5).
mouns with nouns
WIO nouns
with nouns
200
with
LLMs Frequency
-120
Log-Likelihood (a) OPT
Log-Likellhood (b) NEO
Log-Likeilhood (C) GPT2
wio nouns
VLMS Frequency
200
with nouns
with
with nouns
Likelhooo
Log-Likelihood (e) BLIP
Log-Likellhood (f) Ours
(d) GIT
Figure Likelihood drift when removing nouns Comparing likelihood values computed MS-COCO captions with without nouns. None of the models show a drift like our likelihood (Tab. 5).
16

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Log-Likethood
Model
Correlation
GPT2
0.33
12500
OPT
GPT2
OPT
0.35
NEO
0.37
GIT
-2000
GIT
0.42
Ours
BLIP
0.48
1000
2000
3000
4000
5000
Index
Figure Log-likelihood values and correlations. liglliiklihoods are computed on 5,000 captions from MS-COCO validation set, with correlations measuring the alignment of each model's ' log-like likelihond and ours.
Cut-off Caption
General text captions
Sorted Index
Corted Index
Caption
GPT2
BLIP
Ours
Caption
GPT2
BLIP
Ours
A very large elephant Standing next a baby elephant."
"Nothing is quite soft as a young puppy."
4256
1042
51
3690
3404 3250
I am unable to an image above." There is no image describe for this question."
4512
1602
37
A very large elephant standing next to a."
3126
3984 766
4507
2953
15
Long captions
Sorted Index
Caption
GPT2
BLIP
Ours
Baseball players action, one person dressed green running while the others in white are looking and reaching toward the ball.
5
30
2722
A metal poje with a traffic light attaChed, red right arrow illuminated, with green grass, green trees, and white sky in background.
2
25
2628
Figure 15. Examples of differences between language models and method. The sorted index represents the position of 5000 captions from MS-COCO, ranked from low to high likelihood values. The relative likelihood index compared among GPT2 (LLM). BLIP (VLM), and method. Top left: Our method shows significant drop in relative likelihood compared the language models when the sentence is cut-off. Top right: Captions that do not describe images receive the lowest relative likelihood from our method, the highest from GPT2, and intermediate scores from BLIP Bottom: Long captions are assigned low relative likelihoods by language models, while our method assigns them average rolative likelihood scores.
D. Implementation and Theoretical Details
D.I. Implementation details
As explained in Sec. 3.2 if æ is random vector Rd with a non-singular covariance matrix (and with zero mean), then W satisfying WTW E-1 called the whitening matrix. One common approach to achieve whitening through
17

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Principal Component Analysis (PCA), although other methods like Zero-Phasn Component Analysis (ZCA) whitening and Singular Value Decomposition (SVD) whitening exist. PCA whitening transforms the data into coordinate system
defined by the principal components of the covariance matrix. It rescales each component to have unit variance, effectively "whitening" the data. Steps of PCA whitening:
Compute the covariance matrix of the data.
Perform eigenvalue decomposition to obtain eigenvalues (A) and eigenvectors (V).
Transform the data:
whitened = A A- X.
(11)
The main advantages of PCA whitening are that it ensures that the resulting features are uncorrelated and transforms data along principal axes, which often correspond to meaningful directions in the dataset. It can be efficient for dimensionality reduction, something we do not use our work. The main limitation is the loss of original geometry (ZCA whitening for instance maintains the original geometry).
When the features in the original data highly correlated, the matrix W becomes unstable and may not be invertible. To address this issue, we remove one of the highly correlated features and replace it with random noise. In our experiments, this situation occurs only when whitening text embeddings and not with image embeddings. While this introduces some randommess into the process, it has minimal impact on the empirical results. Our full whitening code, together with scripts repeating our experiments is available here.
Algorithm Whitening Process
Input: Dataset X N correlation threshold
Output: Whitening matrix W
Step 1:
Compute Correlation Matrix.
Calculate the correlation matrix:
Cov(Xi, X;)
C
0:05
Step 2: Remove Highly Correlated Features. Identify feature pairs (i.j) where CCty T. For each pair, remove one feature (e.g.. j) and replace it with random noise r, Denote the updated dataset as X':
r F~ 0.1).
Step 3: Compute Covariance Matrix. Calculate the covariance matrix:
N xx"
Step 4: Perform Eigenvalue Decomposition.
Decompose E into eigenvalues A and eigenvectors V:
= VAV
Step 5: Compute Whitening Matrix and Transform Data. Calculate the whitening matrix:
W = A T
where A 11/2 is a diagonal matrix with elements given by the inverse square root of the eigenvalues:
A
18

---

Whitened CLIP Likelihood Surrogate of Images and Captions
D.2. Isotropic random vectors
An isotropic random vector one where all components are identically distributed and statistically independent, with zero mean and unit variance, e.g it's covariance matrix Σ is the unit matrix. In other words, isotropic vector is uniformly distributed across the space, exhibiting directional bias. Such vectors often arise in high-dimensional statistical models and machine learning applications, where isotropy ensures that the data's 's statistical properties are invariant to rotation or translation (Mardia & Jupp, 2000). Isotropic distributions are particularly relevant in contexts such as embedding spaces, where uniformity and independence across features simplify analysis and facilitate probabilistic modeling (Vershynin, 2018).
D.3. Normal distribution tests
Normality tests assess whether a dataset follows a Normal distribution, critical assumption in many statistical methods. As discussed above Sec. 3.3 the Anderson- Darling test evaluates how well the empirical cumulative distribution function (CDF) matches the expected CDF of normal distribution, placing higher weight on the tails to detect deviations (Anderson Darling, 1954). The 'gosstno--Paaaae test combines skewness and shape characteristics measures assess normality, offering sensitivity both symmetric and asymmetric deviations (D' agostino & Pearson, 1973). The Anderson- test statistic is defined as:
1
A -
[22 1) (In F(y) + In (1 F(yn+1-i)))
(12)
71 i=1
Where is the sample size, // S U2 S ... S Yn are the ordered data samples and F(y) is the Cumulative Distribution Function (CDF) of the hypothesized distribution. The D' Agostino-Pearson test statistic combines skewness and kurtosis:
k²= 2i + 22
91
22
92
3
(13)
V
24
where X2 the ''AAgsstno-Pearson test statistic, #1, 22 are the standardized skewness and kurtosis. gis 92 are the sample skewness and kurtosis and is the sample size. For details regarding skewness and kurtosis please refer Groeneveld Meeden (1984).
These tests are well-suited for high-dimensional data they are robust various types of distributional departures, making them effective for validating Normal approximations in the context of our proposed whitened embedding spaces. Bellow we present statistics of image (Fig. 16) and text (Fig. 17) embeddings, on both tests. Mean values (on all groups of data) with standard deviation and histograms of mean values are presented. In addition, in Fig. 18 we present the mean and variance of all the whitened features, demonstrating minor deviation from the expected values (zero mean and unit variance).
19

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Anderson-Darlin
D'Agostino-Pearson
Waee Std Desiation
Moan Volue Stt Deviatiio
94
92
382
Features
Features
175
150
Frequancy
125
Moan Value of Features
Mean Value Features
Figure 16. Normal distribution tests on image embeddings. Top row mean value and standard deviation per feature, over all groups of embeddings. Bottom histogram of mean values of each feature. plots line represents test threshold. Left Anderson Darllinga test, threshold is 0.752, lower is better. Right D' Peostino-Pearseee test, threshold is 0.05, higher is better.
Anderson-Darling
D'Agostino-Pearson
Std Devention
Mean Value Stdl Devilation
08
0.
Wean
6.0
Features
Features
Frequncy
8.5
3.75
0.0
Moan Value Features
Moan Value
Features
Figure 17. Normal distribution tests on text embeddings. Top row mean value and standard deviation feature, over all groups of embeddings. Bottom histogram of mean values of each feature. In plots the red line represents the test threshold. Left Anderson- Anderson-Darling test, threshold is 0.752, lower is better. Right D' Agostino-Pearson test, threshold is 0.05, higher is better.
20

---

Whitened CLIP Likelihood Surrogate of Images and Captions
1055
1.5
1.010
Value
Value
0.0
-1.
0.998
Feature Index
Feature Index
(a) Mean Values
(b) Variance Values
Figure 18. Mean and variance of all whitened features. We show the mean and variance of all the 768 features of the whitened embeddings. There are minor deviations from (for mean) and (for variance). mean values the deviation is up to 0.0015% the case of the variance the deviations are up to 1%.
D.4. Thin shell theory
The thin shell theory (Paouris, 2006) is a concept in high-diment nigh-dimensional geometry. According to the thin shell theory most of the volume of high dimensional convex space is concentered near the surface of the space. Specifically, for a convex space K 12d, the majority of points in K lie at an approximated distance from the origin. This can be formally described in terms of the concentration of measure, where the typical distance of random point from the origin is concentrated around specific radius:
P([ (r- C(d)
(14)
where æ is random sample from the space K, r the typical radius of the space, i small deviation (e 1) and C' is constant that depends the dimensions d. This result indicates that the dimension d grows, the concentration near the thin shell becomes sharper. The thin shell phenomenon is closely related to the chi distribution described above (Eq.
(7)). Specifically relating Eq. (8) to Eq. (14) we get V 1.2 Combining both phenomena, as d increases, the space expands and concentration near the surface emerges because the majority of the space's mass resides near its boundary. Consequently, most points are located near the surface, even as the overall space grows.
E. Ablation Study with Different Data and CLIP Model
We apply the whitening transform to embeddings of MS- MS-COCO validation set using second CLIP model CLIP ViT-B/32, which encodes embeddings with 512 features (compared to embeddings with 768 features encoded by CLIP ViT-L/14). Results are Tab. 6. Results are very similar to CLIP ViT-L-14, used in the paper. verifying that our method is general for different CLIP models.
Table Normal distribution tests using a different CLIP model Avg. AD, DP the average Anderson- Darrlnng D' A'oossino-Peaason p-value test scores (threshold is under 0.752 and above 0.05, respectively). MS-COCO validation set tested using CLIP ViT-B/32, which has 512 features in each embedding.
Avg. AD
Avg. DP
Image
0.65
0.31
Text
0.61
0.25
We conduct ablation study, examining the influence of the size of the data used for computing the whitening matrix W. For each size (Ik, 2k, 3k, 4k) we randomly sampled 5 subsets of MS-COCO validation set. The average scores with standard deviation are plotted in Fig. 19. The tests are performed on the full MS-COCO validation set (5k images). For the D' A'Aogsttnn-Paana Pearson test, for all data sizes the tested embeddings comply with the normal distribution. For the Anderson- Darling test using samples for computing W results with tested embeddings that do not comply with the
21

---

Whitened CLIP Likelihood Surrogate of Images and Captions
normal distribution. Using least 2k results with tested embeddings that comply with the normal distribution. For both tests, using more data compute W results with improved results. As the whitening transform is completely data-driven it is expected that using additional data improves the results. However note that the improvement between using 4k and 5k samples is small.
0.36
0.75
AD Score
DP p-value
0.28
0.50
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Number of Samples
Number of Samples
(a) Anderson-Darling Test
(b) D'Agostino-Pearson Test
Figure 19. Normal distribution tests with different data sizes Anderson-Darling average scores and standard deviation (a) and Aggstino-Pearson p-value average scores and standard deviation (b). Threshold is under 0.752 (marked with black line) and above 0.05, respectively. Computing the whitening matrix W with different data sizes. For each size (Ik, 2k, 3k, 4k) we randomly sampled 5 subsets MS-COCO validation set and present the average score with standard deviation. The tests are performed on the full MS-COCO validation set (5k images).
F. Full circle SLERP Examples
In Fig. 20 simple 2D scenario full circle SLERP demonstrated. The main observation is that if the source and destination points
circle around the origin (allowing small deviations) the full circle SLERP points (blue) remain (or near) the original circle (orange). However, if the circle is skewed from the origin the SLERP points deviate far from the original circle.
We present an additional example of sets images from a full circle SLERP, as discussed in Sec. 4.3, in Fig. 21. As in Fig. 8, also in this case it clear that a full circle SLERP not practical in the raw CLIP space, while resulting with real images throughout the full circle the W-CLIP space.
22

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Circle SLERP points
PI (start)
P2 (end)
(a)
(D)
(C)
Figure 20. 2D full cirele SLERP example. The SLERP points are in blue and the circle perimeter is in orange. Examples of simple 2D case of full circle SLERP. When both points the circle (a) the SLERP points follow the circle perimeter perfectly. If one of the points deviates from the circle (b) the SLERP points form an ellipse, but remain close to the circle perimeter. If the circle is skewed from the origin (c) the SLERP points form large ellipse, that distances far from the circle perimeter.
Cource
Source
35*
290
71.5
Destination
Deseination
200
220
(a) CLIP
(b) W-CLIP
Figure 21. Full circle SLERP example. The full circle SLERP is performed in both the raw CLIP space (a) and in the W-CLIP space (b). The different angle between embeddings in both space is presented. the raw CLIP space the full circle SLERP results with noise for most of the degrees not between the source and destination embeddings. In the W-CLIP space for all degrees real images are generated.
23

---

Whitened CLIP a Likelihood Surrogate of Images and Captions
Figure 22. Opposite image generated in the raw CLIP space. The structured noise produced by CLIP exhibits 4x4 pixel blocks restricted color palette (black ('O' in all color channels), white C11 in all color channels), red, green, blue, magenta ('1' in red and blue channels), eyan ('1' green blue channels), and yellow ('1' in red green channels)), suggesting synthetic artifacts.
24

---

Whitened CLIP Likelihood Surrogate of Images and Captions
G. Image Generation Bias Examples
In Figs. 23, 24 additional examples the bias image generation models are presented. Different random seeds lead to different results due to the image generation model. all cases the images become noise when no normalization is applied in the whitened space. Normalizing embeddings in the whitened space have a norm of v in each iteration results with reasonable images, with varying content.
Step o
Step
Step 10
Step 15
Step 20
Step 25
Step 30
Step 40
Step 50
Raw Şeed #1
Norm
Seed #1
Raw Seed #2
Norm
Şeed #2
Figure 23, Generation bias. Iteratively using UnCLIP to generate images encoded by CLIP with two fixed seeds. The raw process gradually becomes noisy, whereas with normalization (to v each encoding step), the content drifts but remains within natural and reasonable image space.
25

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Step o
Step
Step 10
Step 25
Step 20
Step
Step
Step 40
Step
Raw
Seed #1
Norm
Seed #1
Raw Seed #2
Norm
Seed #2
Figure 24, Generation bias. Iteratively using UnCLIP generate images encoded by CLIP with two fixed seeds. The raw process gradually becomes noisy, whereas with normalization (to v each encoding step), the content drifts but remains within natural and reasonable image space.
H. Text Complexity Examples
In Figs. 25, 26 we repeat the experiment presented in Fig. 5, showing additional examples how adding and removing details from concepts (not the concepts themselves) decreases/inereases the likelihood respectively.
Caption
Log likelihood
Caption
Log jikelihood
Caption
Log iikeihhodd
"A group men standing the side street."
man sicting front of plate food."
"A living room filled with furniture and table."
-890.43
991591
9334997
'A group of men standing the pavement
-923.46
"A man sicting front of
10661.26
*A living room filled with antique furniture and table."
992-82
street."
food."
"A group of suited men standing on the side of a street."
1060.98 "A man sitting frone of plate of pasta."
-1097.99
"A living room filled with chairs and table."
995.23
"Seven men standing on side street."
-2132.51
"A teenager sitting in
-1024.05
living room filled with furniture and consoje."
-1015.49
Front of plate of food."
"A group men standing on the side of Broadway."
"John sicting fronc of place of food."
"A Boudoir fitled with furnicure and a table."
-1339.71
-2096-61
-1299-67
Figure 25, Adding details to concepts. The original caption from MS-COCO is framed blue. Adding details decreases the likelihood.
26

---

Whitened CLIP Likelihood Surrogate of Images and Captions
Caption
Log likellhood
Caption
Log likelhhood
"A hybrid diesel electric commuter bus on the street
1519.25
"A hot dog Cart across from the Hall.
-1367.05
"A commuter bus on the Street in England." "
-1777-36
"A hot dog cart across from the Radio Hall.
-1378.41
"A hybrid Commuter bus on the Street in England."
-1868.65
"A hot dog cart across from the Radio City Hall.
1475.37
"A hybrid diesel bus on the street
1932.08
in England." hybrid diesel electric commuter bus on the street England."
"A hot dog Cart across from the Radio City Music Hall."
-1654.94
19770.31
Figure 26. Removing details from concepts. The original caption from MS-COCO is framed frame. Removing different details increases the likelihood.
27


Reviews:
[{"review_id": "4fHZcjvuEa", "reviewer": "Reviewer_Sqox", "summary": "This paper studies the CLIP features to approximate the likelihood of images and captions. The presented method, Whitened CLIP (W-CLIP), uses an invertible linear operation to convert the CLIP features into a zero-mean, unit standard deviation space. This normalized embedding space can be used in many cases, including detecting artifacts in synthetic images, analyzing the domain drifts for different datasets, and enhancing image manipulation of two images.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's detailed and thoughtful feedback. Below, we address each point raised.\n\n### **Methods and Evaluation Criteria**\n\n#### **First Point, first part**\n\n> *\"While we have qualitative examples in Fig 2 and Fig 8 to show that W-CLIP can detect artifacts in synthetic images, we don't have a quantitative metric to know how good it is.\"*\n\nTo address this, we conducted a large-scale experiment on zero-shot detection of generated images. Full details are provided in our response to Reviewer 531Y. A summary of the comparative results is given below:\n\n| **Method**| **AUC** | **AP** | **F1** | **Acc** |\n|--|-|-|-|-|\n| AEROBLADE| 0.52   | 0.48   | 0.64   | 0.53 |\n| RIGID | 0.51| 0.53 | 0.28 | 0.52|\n| ZED| 0.69| 0.66| 0.69   | 0.62 |\n| Manifold-Bias     | 0.85| 0.88   | 0.76 | 0.78|\n| **Ours**| **0.89** | **0.89** | **0.82** | **0.81** |\n\n#### **First Point, second part**\n> *\"Moreover, can W-CLIP embedding be used to tell the aesthetics or quality of images?\"*\n\nWe refer to Fig. 3 (top right), where an ImageNet-C experiment shows that norms of whitened embeddings respond to image corruptions (e.g., impulse noise). Additional results appear in Fig. 10 (Appendix B), covering 12 corruptions including blur, defocus, and low contrast. In all cases, corrupted images exhibit higher norms (in the whitened space) than the original images. As explained in Sec. 4.2 (lines 316–322), higher norms in the whitened space correspond to lower likelihood values, indicating reduced image quality. These results suggest that W-CLIP embeddings capture information related to image quality.\n\n#### **Second Point**\n\n> *\"When using full circle SLERP to do image manipulation, we have qualitative examples in Fig 20 and Fig 21, but again, we don't have any large-scale evaluation to show the superiority of W-CLIP compared with CLIP.\"*\n\nTo provide quantitative insight, we conducted a full-circle SLERP experiment on the MSCOCO validation set (5k images). For each image, we performed full-circle SLERP in both the CLIP and W-CLIP embedding spaces. In this process, a source image is interpolated toward a destination image along a circular path within the embedding space. Crucially, the image generated at the 180° position from the source—referred to as the **“opposite image”** (generated from the “opposite embedding”)—is invariant to the chosen destination and determined solely by the source. While other positions along the path are influenced by the destination embedding, the 180° embedding is a fixed, symmetric counterpart.\n\nWe generate these opposite images using both CLIP and W-CLIP embeddings and observe a stark contrast: in the CLIP space, opposite images degrade into structured noise, whereas in the W-CLIP space, they remain visually natural and semantically meaningful, as shown in Figs. 20, 21.\n\nThe structured noise produced by CLIP exhibits 4×4 pixel blocks and a restricted color palette (black ('0' in all color channels), white ('1' in all color channels), red, green, blue, magenta ('1' in red and blue channels), cyan ('1' in green and blue channels), and yellow ('1' in red and green channels)), suggesting synthetic artifacts. We provide visual examples (20 opposite images) in the following link:  \n[https://drive.google.com/drive/folders/1Q85pz8y-36K2eHXDHsRcciblihvHgwgX?usp=drive_link](https://drive.google.com/drive/folders/1Q85pz8y-36K2eHXDHsRcciblihvHgwgX?usp=drive_link)\n\nTo quantify these differences, we compute Total Variation (TV), Entropy, and the percentage of extreme saturation values (top or bottom 1% of the pixel range). All metrics are computed per channel and averaged per image across three sets: original MSCOCO images, CLIP opposites, and W-CLIP opposites. The results are summarized below:\n\n| **Method** | **TV** | **Entropy** |Saturation Values [%] |\n|-|-|-|-|\n|MSCOCO| 222.3| 7.3|4.2|\n|CLIP Opposite| 156.7| 4.8|55.5|\n|W-CLIP Opposite| 215.9| 7.2|6.4|\n\nThese findings confirm that W-CLIP opposites are statistically similar to natural images, whereas CLIP opposites exhibit significantly reduced entropy and variation and much higher percentage of saturation values, indicating a lack of natural structure.\n\n### **Other Comments or Suggestions**\n\n> *\"The figures in this paper are vague and have low resolution. For instance, subfigures in Figure 4 have small x-y axis labels and legends. Figure 3 is a good example, I highly recommend the authors redraw all the figures like Figure 3.\"*\n\nWe thank the reviewer for this helpful suggestion. We will improve the visual quality of all figures in the final version. Specifically, we will enlarge the legends and the axes ticks in Figs. 4, 6, and 7, and reformat histograms for clearer presentation, following the style of Fig. 3.\n\nWe hope our additional experiments and clarifications address the concerns raised and further strengthen the validity and impact of our proposed approach. We would appreciate positively considering increasing the final rating of the paper.", "claims_and_evidence": "This paper is clearly motivated and proposes a simple yet effective method with abundant experiments.", "methods_and_evaluation_criteria": "Some key results lack quantitative metrics to validate the effectiveness of the proposed method.\n- While we have qualitative examples in Fig 2 and Fig 8 to show that W-CLIP can detect artifacts in synthetic images, we don't have a quantitative metric to know how good it is. Moreover, can W-CLIP embedding be used to tell the aesthetics or quality of images?\n- When using full circle SLERP to do image manipulation, we have qualitative examples in Fig 20 and Fig 21, but again, we don't have any large-scale evaluation to show the superiority of W-CLIP compared with CLIP.", "theoretical_claims": "This paper doesn't have any theoretical contributions.", "experimental_designs_or_analyses": "The experiment design is holistic including the image and text domain. However, this paper lacks some quantitative evaluation for some experiments. (See Methods And Evaluation Criteria part)", "relation_to_broader_scientific_literature": "This paper claims they are the first to whiten the CLIP features for image and text likelihood analysis. I haven't tracked this domain closely, so I cannot verify their claim. I'll look at other reviewers' comment on this.", "essential_references_not_discussed": "I haven't tracked this domain closely, so I cannot verify their claim. I'll look at other reviewers' comment on this.", "other_strengths_and_weaknesses": "See other parts.", "other_comments_or_suggestions": "The figures in this paper are vague and have low resolution. For instance, subfigures in Figure 4 have small x-y axis labels and legends. Figure 3 is a good example, I highly recommend the authors redraw all the figures like Figure 3.", "questions_for_authors": "I am open to discuss and would be happy to reaccess my rating during rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhfBR6zM9a", "reviewer": "Reviewer_wApW", "summary": "The authors propose the use of a whitening transform in the CLIP space, which offers an efficient solution in closed-form via the SVD. Using “WCLIP” they explore a wide number of practical downstream tasks one can tackle using the now-quantifiable likelihood (OOD image detection, caption complexity, quantifying artifacts, etc).\n\n## update after rebuttal\nThe authors provide useful clarifications during the rebuttal, and additional evidence of the usefulness of the proposed methodology. However, I maintain that work is needed on the experimental section to clarify the authors' key contributions--ultimately, I now lean towards a weak acceptance based on these considerations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the reviewer’s critical insights and suggestions. Below, we provide detailed clarifications and supporting evidence addressing the concerns raised.\n\n### **Experimental Designs or Analyses**\n\n#### **Text Complexity**\n\nOne example in Fig. 5 does show higher likelihood for a shorter sentence (left), but this does not imply a bias toward shorter inputs. In Fig. 7c, our method's likelihood is shown to be unaffected by caption length, unlike the other LLMs and VLMs. Thus, our findings are contrary to the reviewer’s claim. In fact, in Fig. 7a and Tab. 4, we present a surprising finding: LLMs and VLMs show similar distributions for captions with and without nouns. As illustrated, the noun-free captions are semantically illogical. Since LLMs and VLMs are highly biased toward shorter inputs, the distribution remains unchanged. In contrast, our method—agnostic to caption length—assigns significantly lower likelihood values to the noun-free captions.\n\nRegarding cosine similarity as an alternative to likelihood: the two serve fundamentally different roles. Cosine similarity compares *two* inputs and outputs a scalar based on angular distance, whereas likelihood evaluates a *single* input. Therefore, cosine similarity is not suitable as a substitute for likelihood in this context.\n\n#### **Uniformity**\n\n- **Unit Variance and Zero Correlation**: As the reviewer notes, these properties are directly achieved through the whitening transform. While this is given in theory, this experiment empirically confirms that the transformation results in an isotropic space where features have zero mean, unit variance, and are uncorrelated. These characteristics, along with the normality verified in Sec. 3.3, are essential for computing likelihood using a multivariate Gaussian model (Sec. 3.4).\n\n- **Uniformity**: We agree that we did not elaborate on uniformity in the main paper, due to space constraints. Here we supply a more detailed explanation - uniformity in latent representations ensures that embeddings are evenly distributed across the latent space, preventing representation collapse where different inputs become indistinguishable due to overly similar embeddings [1]. A uniform space enhances discrimination between inputs, improving downstream tasks such as classification [2]. It also promotes generalization and avoids overfitting to specific latent regions [3]. In contrastive learning, uniformity complements alignment by pushing dissimilar inputs apart [1], resulting in a well-structured, robust, and generalizable representation space. We will add a more detailed explanation to the appendix and refer to it from the main paper.\n\n#### **Data Analysis**\n\nWe refer the reviewer to our new experiment on zero-shot generated image detection (see response to Reviewer 531Y). Specifically in relation to cosine similarity, we note that both RIGID and Manifold-Bias use cosine similarity as part of their detection pipelines. However, our likelihood-based method is simpler, significantly faster (see table below), and outperforms both baselines. Moreover, other cosine-based methods [4,5] for this task are not applicable in a zero-shot setting. This experiment highlights both the effectiveness and practical advantages of our approach.  To demonstrate the efficiency of our method, we report per-image inference times on a single A100 GPU for the zero-shot detection task:\n\n| **Method**| **Running time [sec per image]** |\n|------------------|------------------------------|\n| AEROBLADE| 4.66|\n| RIGID| 0.59|\n| ZED | 0.26|\n| Manifold-Bias| 1.66|\n| **Ours** | **0.05**  |\n\n\n### **Other Strengths and Weaknesses**\n#### **Not \"Training-Free\"**\nThe reviewer raises a valid point: PCA in the whitening transform can be seen as unsupervised training. However, this is analogous to CLIP pretraining—performed once on unlabeled data and reused across downstream tasks. In our case, this step is extremely lightweight (under 1 second). As demonstrated above, leveraging the likelihood for a downstream task results with a very efficient method.\n\nWe hope these clarifications and new experiment address the reviewer’s concerns and underscore the practical strengths of our approach. We would appreciate positively considering increasing the final rating of the paper.\n\n\n[1] Wang, Tongzhou, and Phillip Isola. \"Understanding contrastive representation learning through alignment and uniformity on the hypersphere.\" ICML 2020  \n[2] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" ICML 2020  \n[3] Balestriero, Randall, and Yann LeCun. \"Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods.” NeurIPS 2022  \n[4] Cozzolino, Davide, et al. \"Raising the Bar of AI-generated Image Detection with CLIP.\" CVPR 2024  \n[5] Sha, Zeyang, et al. \"De-fake: Detection and attribution of fake images generated by text-to-image generation models.\" ACM SIGSAC 2023", "claims_and_evidence": "Yes — the authors do not make many claims in the paper, and the few small claims they do make are reasonable and sound.", "methods_and_evaluation_criteria": "The method is well-formulated and makes sense theoretically. The absence of a need to train a mapping end-to-end (and tune the hyperparameters that come with such approaches) is a key strength of the authors' proposal to use the whitening transform. However, there is a lack of appropriate experimental results comparing the proposed method to alternative baselines for the tasks the authors explore (please see my comments in the “experimental design” section for more on this).", "theoretical_claims": "N/a — no theoretical claims are made in the paper.", "experimental_designs_or_analyses": "The authors explore multiple properties and downstream use cases of WCLIP. However, I am not convinced by the majority of the experiments.\n\nThe authors did an insufficient job for motivating the practical benefits of the properties induced by the method, and the downstream tasks explored lack any comparisons to baselines to confirm that the proposed method confers any advantages over existing/simpler techniques. Concretely: \n\n**Text complexity**: the authors show that longer captions yield lower likelihood scores. However, I am confused about what insights this finding adds — CLIP is (presumably) trained with short image-caption pairs. Thus, I would argue the decreasing likelihood as a function of caption complexity likely predictably arises from the data itself, and likelihood here offers no unique insights. I fully expect to see the same relationship with CLIP’s original cosine similarity for short vs long/specific captions. What value does this analysis provide over the same analysis with the cosine similarity metric? The authors could perform the same experiments with the original CLIP cosine similarity to show why the likelihood is more informative, or why this is useful.\n\n**Uniformity:** the authors show in Figure 4: “the effectiveness of the whitening transform in achieving unit variance and zero correlation among features”. Isn’t this just by definition of using the whitening transform? There is no motivation for *why* this is a desirable property or the unique benefits of it. Furthermore, the authors next mention uniformity as a “desirable” property too, but do not elaborate at all on this point.\n\n**Data analysis**: In the authors’ experiments showing the use of likelihood to distinguish between real/fake/OOD data, they once again fail to benchmark against even a simple baseline to show why one would use this method over existing techniques—for example, what about simple k-means anomaly detection in the original CLIP space?\n\nI am not convinced there is any practical benefit of WCLIP over much simpler analyses currently.", "relation_to_broader_scientific_literature": "I am not familiar with the wider literature for analysis of CLIP. However, even without specific knowledge of the literature, I am confident in my assessment of the paper as needing experiments comparing their method to simpler alternative baseline approaches to justify and motivate why one would use the whitening transform.", "essential_references_not_discussed": "The paper appears to do quite a good job in the literature review. I don’t know the literature well enough to identify whether they have missed important works.", "other_strengths_and_weaknesses": "I like the authors’ ideally conceptually — whilst i wouldn’t totally agree with the authors' characterization that it is “training-free” (perhaps one could say instead it offers a closed-form solution, based on the SVD), I do think the efficiency of the method is a key strength of the paper.\n\nHowever, as discussed above, the experiments in the paper need a lot of work. I would encourage the authors to prioritize depth over breadth here, and polish the motivation for the whitening analysis.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kpaIkhQiiN", "reviewer": "Reviewer_531Y", "summary": "## update after rebuttal\nI will keep my score as is (3)\n\nThis paper proposes Whitened CLIP (W-CLIP) that transforms the CLIP latent space, providing direct access through log-likelyhood function. W is computer only once, based on data, a-priories.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and comments. Below, we address key concerns regarding the applicability of our approach to zero-shot settings, and address the reviewers questions.\n\n### **Methods and Evaluation Criteria**\n\nTo address the reviewer’s comment regarding zero-shot transfer to a downstream task, we conducted a large-scale experiment on the generated image detection task. The generated images were synthesized from text, taken from three benchmarks [1,2,3]. Images are generated using 20 generative models, totaling 100k generated images. Since these datasets include fewer real images, we supplemented real samples from the MSCOCO training set.\n\nHere, zero-shot refers to having no exposure to generated content and no task-specific training. We use only real images (from MSCOCO validation set) to compute the whitening matrix and do not fine-tune on this task. Thus, we compare against other zero-shot image detection baselines.\n\nWe benchmark against four zero-shot detection methods: AEROBLADE [4] (CVPR 2024), RIGID [5] (arXiv 2024), ZED [6] (ECCV 2024), and Manifold-Bias [7] (ICLR 2025). We use official implementations for [4] and [7], and self implement [5] and [6] based on their papers and available code (of other models used in these methods).\n\nEach method outputs a continuous criterion score, which we binarize using a threshold from a small calibration set of 1k real images:  \n**th = mean(C) + std(C)**, where C denotes criterion values. This calibration set is disjoint from the evaluation set.\n\nWe use four metrics: AUC and AP (Average Precision) as separation metrics, and F1-score and Accuracy as classification metrics. Evaluation is based on 100k generated and 100k real images. As shown below, our method outperforms all baselines across all metrics. This setting follows the protocol used in [7], the most comprehensive of the compared works. Our results for baseline methods align with those reported in [7]. We believe this validates our method’s ability to distinguish real from generated images in a zero-shot setting.\n\n#### Performance Comparison\n\n| Method| AUC|AP|F1|Acc|\n|-|-|-|-|-|\n| AEROBLADE| 0.52| 0.48   | 0.64| 0.53|\n| RIGID| 0.51| 0.53| 0.28   | 0.52|\n| ZED| 0.69| 0.66| 0.69| 0.62|\n| Manifold-Bias| 0.85| 0.88| 0.76| 0.78|\n| Ours| **0.89** | **0.89** | **0.82** | **0.81** |\n\nWhile on some generative models methods [6] or [7] occasionally perform competitively, ours is more consistent across all generative models, whereas others experience significant drops on specific ones. Full per-model results are available here:  \n[https://drive.google.com/file/d/1hQFwAqpo3opByTqC70mb4-fDz0z8TkVQ/view?usp=drive_link](https://drive.google.com/file/d/1hQFwAqpo3opByTqC70mb4-fDz0z8TkVQ/view?usp=drive_link)\n\n### **Theoretical Claims**\n\nWhen CLIP embedding features are highly correlated, the covariance matrix can become ill-conditioned, yielding near-zero eigenvalues. During whitening, eigenvectors are scaled by the inverse square root of these values, so eigenvalues close to zero can result with a numerically unstable and non-invertible whitening matrix W.\n\n### **Questions for Authors**\n\n1. We thank the reviewer for highlighting two relevant papers, which we will cite in the final version. These works propose alternative, hierarchical embedding spaces to CLIP. The whitening transform is agnostic to the embedding space, as long as the data adequately represents it. In the context of these embedding spaces we assume this requires diverse samples, including many complex examples. For our likelihood approximation, a key factor is the distribution of each feature. As shown in Fig. 4.c, CLIP features exhibit a Gaussian-like distribution, which becomes standard normal after whitening, as validated in Tab. 1. If the original features of these embedders follow a different distribution, whitening may not yield standard normal features, undermining the assumptions of our likelihood model.\n\n2. This question is addressed in the zero-shot detection experiment discussed above.\n\nWe hope these additional results and clarifications further support our proposed approach.\n\n[1] Wang, Sheng-Yu, et al. \"CNN-generated images are surprisingly easy to spot... for now.\" CVPR 2020.‏  \n[2] Ojha, Utkarsh, et al. \"Towards universal fake image detectors that generalize across generative models.\" CVPR 2023.‏  \n[3] Zhu, Mingjian, et al. \"Genimage: A million-scale benchmark for detecting AI-generated image.\" NeurIPS 2023  \n[4] Ricker, Jonas, et al. \"AEROBLADE: Training-free detection of latent diffusion images using autoencoder reconstruction error.\" CVPR 2024  \n[5] He, Zhiyuan, et al. \"RIGID: A training-free and model-agnostic framework for robust AI-generated image detection.\" arXiv 2024  \n[6] Cozzolino, Davide, et al. \"Zero-shot detection of AI-generated images.\" ECCV 2024.‏  \n[7] Brokman, Jonathan, et al. \"Manifold induced biases for zero-shot and few-shot detection of generated images.\" ICLR 2025", "claims_and_evidence": "Yes. The whitening transform in section 3.2 and whitening CLIP embedding in section 3.3 contains convincing evidence. The authors did quantitative statistical experiments using Anderson-Darling and D’Agostino-Pearson tests, indicating the features in the whitened space can be well approximated by a normal distribution.", "methods_and_evaluation_criteria": "Since this work is on CLIP embedding transformation - the paper would be much more stronger if the authors did evaluation on zero shot transfer, classification, OCR etc. downstream tasks similar to the original CLIP paper https://arxiv.org/pdf/2103.00020", "theoretical_claims": "The output log-likelyhood is consistent with intuitive understanding - specific words have less log likelyhood vs common words, synthetic image with artifacts has less log likelyhood vs real image. Sound claims except it is not clear in line 897-900 why W matrix becomes unstable (and may not be invertible) if the features are highly correlated specially for text embedding", "experimental_designs_or_analyses": "The experiments are sound as in section 4. But I think this paper could have experiments to see how W-CLIP impacts the performance of downstream tasks such as image classification, OCR etc.", "relation_to_broader_scientific_literature": "This work expands the original CLIP paper https://arxiv.org/pdf/2103.00020 . It transforms the CLIP latent space, ensuring each feature in the embedding space has unit mean, zero standard deviation and no correlation with other features. As a result, one can express this embedding as a direct log-likelyhood function which is a novel idea", "essential_references_not_discussed": "I think \"Hyperbolic Image-Text Representation\" ( https://arxiv.org/abs/2304.09172) and \"Embedding Geometries of Contrastive Language-Image Pre-Training\" ( https://www.arxiv.org/abs/2409.13079) are two related work for this paper. They do not do what this paper suggests but seem to be a good reference for the related work since they show alternative representation of CLIP embedding. I do not see those two papers referenced anywhere unless I am missing something.", "other_strengths_and_weaknesses": "Strength:\n\t1. Transforms the CLIP latent space, ensuring each feature in the embedding space has unit mean, zero standard deviation and no correlation with other features\n\t2. Formulating the CLIP method in log likelyhood\n\t3. This log-likelyhood representation helps separate real image vs synthetic image with artifacts. The later having less log likelyhood which is intuitive\n       4. Very well written paper with clear instruction on reproduction. \n\nWeaknesses:\n    1. Lack of evaluation of W-CLIP on downstream tasks such as image classification, similarity search, OCR etc.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. I think \"Hyperbolic Image-Text Representation\" ( https://arxiv.org/abs/2304.09172) and \"Embedding Geometries of Contrastive Language-Image Pre-Training\" ( https://www.arxiv.org/abs/2409.13079) are two related work for this paper. I do not see them referenced anywhere unless I am missing something. Wondering what do the authors think about these two work? What will happen if Whitened  technique is applied to those two representation?\n2. log-likelyhood representation helps separate real image vs synthetic image is interesting. Can it help separate synthetic image if there is no artifact?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces W-CLIP, a simple way to normalize CLIP embeddings so they can be used like likelihood scores. It’s fast, training-free, and works across tasks like artifact detection and image editing.

After a strong rebuttal with new experiments, including solid zero-shot detection results, most concerns were addressed, and all reviewers lean towards weak accept. The idea is clear and useful, though some analyses could be better motivated. The ACs hope the authors can revise and improve the method based on the feedback from the reviewers.
Decision: Accept (poster)