Implicit Inversion turns CLIP into a Decoder
Authors: Antonio D'Orazio
Maria Rosaria Briglia
Donato Crisostomi
Dario Loi
Emanuele Rodolà
Iacopo Masi
Venue: ICLR 2026 Poster (2026)
Keywords: Model Inversion
Text To Image
CLIP
Implicit Neural Representations
Primary area: optimization
TL;DR: We turn the discriminative CLIP model into a generator capable of addressing multiple tasks—such as text-to-image synthesis, style transfer, and image editing—without fine-tuning

Abstract: CLIP is a discriminative model trained to align images and text in a shared embedding space. Due to its multimodal structure, it serves as the backbone of many generative pipelines, where a decoder is trained to map from the shared space back to images. We show that image synthesis is nevertheless possible using CLIP alone—without a pre-trained generative decoder or CLIP tuning. Our approach optimizes a frequency-aware implicit neural representation that encourages coarse-to-fine generation by stratifying frequencies across network layers. To stabilize this inverse mapping, we introduce adversarially robust initialization, a lightweight Orthogonal Procrustes projection to align local text and image embeddings, and a blending loss that anchors outputs to natural image statistics. With CLIP frozen, this framework unlocks capabilities such as text-to-image generation, style transfer, and image reconstruction. Our findings suggest that discriminative models may hold untapped generative potential, hidden in plain sight.

Code: https://github.com/OmnAI-Lab/implicit-inversion

Published as a conference paper at ICLR 2026
IMPLICIT INVERSION TURNS CLIP INTO A DECODER
Antonio D'Orazio'
Maria Rosaria Briglia'
Donato Crisostomi 2.1
Dario Loi
Emanuele Rodola
lacopo Masi"
Sapienza University of Rome, Italy
AM Lab &
GLADIA
Paradiigma
Pinatubo
Tibetan monastery on cliff
Prague castle
Machu Picchu shrouded in mist
Grand Canyon at sunaet
Photo of
eruption
A group of people sitting around table
Kept retra 1969s looking living
pastry store with cupcakes roon set on display up in boul
Sone oranges are stacked
A man standing near a blue and cabinets and stuff brown bed on the table
kitchen with
Figure I: Text-to- synthesis by CLIP inversion. CLIP inverts CLIP's image encoder using implicit neural representations, enabling text-lo-image synthesis without any fine-tuning of CLIP dedicated gener- ative decoder. samples are generated with CLIP ViT-B/32 Radford al. (2021), with top rows showing generic scene prompts and bottom rows illustrating complex captions from MS-COCO Lin et al. (2014).
ABSTRACT
CLIP is a discriminative model trained to align images and text in a shared em- bedding space. Due to its multimodal structure, it serves the backbone of many generative pipelines, where a decoder is trained to map from the shared space back images. We show that image synthesis is nevertheless possible us- ing CLIP alone-withowe without a pre-trained generative decoder or CLIP tuning. Our approach optimizes frequency-await implicit neural representation that encour- ages coarse-to-fine generation by stratifying frequencies across network layers. To stabilize this inverse mapping. we introduce adversarially robust initialization, a lightweight Orthogonal Procrustes projection to align local text and image beddings, and a blending loss that anchors outputs to natural image statistics. With CLIP frozen, this framework unlocks capabilities such as rext-to-image gen- eration, style transfer, and image reconstruction. Our findings suggest that dis- criminative models may hold untapped generative potential, hidden plain sight. Code: https: / / thub. com//mnAIIem Lab/ impl i iiit inversiiiinn
INTRODUCTION
Text-to-image generation has progressed from a challenge to a widely accessible technology, with recent models achieving photorealistic results and remarkable creative abilities (Betker et al., 2023; Saharia et al., 2022; Nichol et al., 2022; Chang et al., 2023). Among these, latent diffusion mod- els (Rombach et al., 2022) currently define the of-te-of-theart by relying architecture, where the encoder maps the input text into a latent representation, and the decoder re- constructs image from it. Foundational vision-language models like CLIP (Radford et al., 2021)
cecoder-decodee

---

Published as a conference paper at ICLR 2026
often used text encoders (Betker et al., 2023; Wang et al., 2022; Tao et al., 2023), whereas the decoder is typically a diffusion model - by far one of the most computationally demanding stages.
In this paper, we show that CLIP alone can perform text-to-image generation, without pretrained generative decoder. We achieve this by inverting the CLIP vision encoder: rather than mapping an image its latent embedding,
start from CLIP embedding and reconstruct the corresponding image. Although prior work has attempted this through direct pixel space optimization (Kazemi al., 2024; Ganz & Elad, 2023; 2024), such approaches produce low-quality output with visible artifacts (Kazemi et al., 2024) or require additional training (Ganz & Elad, 2023; 2024). In contrast, and CLiP-tuning-fre solution based on Implicit Neural Representations (INRs) (Liu et al., 2024). Fig. 1 offers our results on generic prompts and longer, more complex prompts from MS-COCO (Lin et al., 2014).
we introduce CLIP-1 ! pretrained-decoddr-ffee
We first retrieve a low-frequency seed INR whose caption is most similar to the prompt. Because this INR was pre-trained on a blurred version of its image with Adversarial Weight Perturbations (AWP) (Wu et al., 2020), its low-frequency weights are stable and provide robust anchor. We then refine the INR layer by layer: a peaked learning-rate schedule moves from low- to high-frequency layers, progressively adding details in a coarse-to-fine manner. During refinement, we encourage on-manifold solutions using cosine losses against a style prompt and a few retrieved natural images, eventually resulting in more aesthetic generations. Without pixel-space optimization, approach avoids structural artifacts and results in a coarse-to-fine generation akin to diffusion models.
Despite the stability of the INR backbone, inversion still suffers from CLIP's modality gap (Liang et al., 2022b; Mistretta et al., 2025; Zhang et al., 2024): text and image embeddings fall on slightly offset sub- tobbmanifolds, making the raw prompt embedding an unreliable target. We bridge this gap by seeking an optimal orthogonal transformation via Procrustes analysis (Wang & Mahadevan, 2008; Maiorca et al., 2023), using the k nearest caption- -mmag pairs to align the prompt embedding with the image manifold and produce a coll-conditioned target for optimization.
We benchmark our pipeline on 10k MS-COCO (Lin et al., 2014) captions, reporting Fréchet In- ception Distance (FID) (Heusel et al., 2017), Inception Score (IS) (Salimans et al., 2016), and CLIPSIM metric (Hessel al., 2021). Compared with DAS (Fort & Whitaker, 2025), a concur- rent inversion- based approach that similarly requires no pretrained decoder or CLIP fine-tuning, our method achieves half the FID while almost doubling IS, producing crisper, more faithful im- ages. The same frozen model transfers zero-shot to image reconstruction, controlled image edits, and neural style transfer, confirming the versatility of the INR backbone. Ablations show that (i) adversarially robust INRs boost quality, (ii) Procrustes alignment yiclds sharper, more semantically aligned images, and (iii) frequency-steeres optimization suppresses high-frequency artifacts.
In summary, our contributions are: (1) Text-to-image synthesis by inversion, without pretrained decoders or CLIP tuning. We repurpose a frozen CLIP as a generator by optimizing a frequency- aware INR. retrieved via blur-initialized anchor and refined through driven only by CLIP losses, without external decoders or re-training. (2) Modality-gap reduction We reduce the mismatch between CLIP's text and image sub-manifolds through orthogonal Pro- crustes transformation estimated from the k nearest cuption-image pairs. (3) Extensive empirical validation. We evaluate reconstruction, controlled edits, style transfer, and interpretability tasks. We release all code and models to support future research.
to-se-tt--fine schedule
We emphasize that our goal not to compete with diffusion or autoregressive decoders fidelity, but to show that sufficiently realistic images can be generated using only CLIP's latent space, without any external decoder. This also gives the method analytical value: it reveals how much visual structure is already encoded in a frozen CLIP and how the model reacts to challenging or atypical textual inputs, offering a practical tool for interpretability, stress-testing. and debugging multimodal systems before CLIP's errors and/or biases propagate into larger pipelines.
RELATED WORK
Image generative models. Modern image generators can be categorized into four broad families: GANs, diffusion models, normalizing flows, and autoregressive (AR) models. GANs train gener- ator to fool discriminator with realistic samples, a strategy refined from early DCGANs (Radford et al., 2015) to StyleGAN-T (Sauer et al., 2023) and BigGAN (Brock et al.). Diffusion models be-
2

---

Published as a conference paper at ICLR 2026
gin with pure noise and iteratively denoise it back to an image; latent diffusion (Rombach et al., 2022), GLIDE (Nichol et al., 2022), DALL-E 2 (Ramesh et al., 2022) and DALL-E (Betker et al,, 2023) differ mainly in how they compress the signal and guide it with text. Normalizing flows (c.g., (Kingma & Dhariwal, 2018; Esser et al., 2024)) learn a chain of invertible transforms so that sam- pling the base Gaussian and running the reverse pass yields data in few steps. AR models play sig- nificant role in the natural language processing domain, with a decoder-only transformer ( (Vaswani al., 2017) powering tools such as LLMs. Although AR models have been primarily applied in discrete text domains, Tian et al. (2024) demonstrates that they also yield state-of-the-art generation performance in the image domain, competing with diffusion models. Despite these distinct me- chanics, every approach relies on latent-to-image decoder: the generator in GANs, the denoising network in diffusion, or the reverse flow normalizing-foo flow models, along with the internal repre- sentation in transformers in AR models. Our work bypasses the need for pretrained latent-to-image decoder by directly inverting a frozen discriminative encoder (CLIP).
Model inversion. Several recent works attempt to repurpose CLIP for image generation by inverting its embeddings. The earliest approach (Kazemi et al., 2024) directly optimises randomly initialised pixels minimise the cosine distance between the CLIP embedding of the image and that of target text prompt. GALIP (Tao et al., 2023) introduces a CLIP-condition ing both a generator and discriminator enable fast, controllable synthesis with fewer parameters and less data than large-scale diffusion models. CLIPAG (Ganz & Elad, 2023) does not rely on pretrained decoder but applies adversarial fine-tuning CLIP itself improve generation qual- ity. EB-CLIP (Ganz & Elad, 2024) further extends this idea, where the generation is formulated as energy minimisation in CLIP's joint image-text space. Concurrently with our work, DAS (Fort & Whitaker, 2025) shows that frozen CLIP can be inverted by optimising at multiple spatial res- olutions in a coarse-to-fing manner. Like us, DAS reveals generative priors within discriminative models, but differs by operating directly in pixel space rather than through a frequency-awan im- plicit representation.
GAN framework, train-
Adversarial robustness. Adversarial training has become core strategy for improving model robust- ness, with (Madry et al., 2018) introducing the first widely adopted method using input perturbations. TRADES (Zhang et al., 2019) extended this by leveraging KL divergence to balance accuracy and robustness, later refined by Cui et al. (2023) to address its asymmetry. Beyond input-space attacks, AWP (Wu et al., 2020) proposed perturbing model weights during training, improving generalisa- tion by flattening the loss landscape. While primarily used for classification, recent work (Mirza al., 2024) suggests robust models also encode stronger generative priors.
Image modeling with INRs. Implicit Neural Representations model images as continuous functions that map spatial coordinates (f, to RGB values via neural network, typically an MLP. To cap- ture fine detail, they rely on frequency-aware components such as positional encodings (Tancik al., 2020) or periodic activation functions like SIREN (Sitzmann al., 2020). However, fixed- frequency activations limit adaptability, motivating FINER (Liu et al., 2024), which introduces seriable-periodic activations that dynamically adjust to local frequency content. We adopt FINER for its efficient representation, well-suited to our inversion task, and to mitigate the spectral bias that hampers high-fidelity reconstruction in standard INRs.
METHOD
Given an input text prompt. our approach aims to generate images by inverting the corresponding
CLIP embeddings. The pipeline consists of three main stages: (i) data preparation step (performed once offline), (ii) an initialization step leveraging preprocessed data to retrieve a suitable starting point, (iii) an optimization procedure refining the initial image match the target text.
3.1 PRELIMINARIES
Implicit Neural Representations. INRs represent images as functions mapping spatial coordinates (i,j) to RGB values (i.j) = (r.g. b), where ó are parameters of neural network. The architec- ture is typically a Multilayer Perceptron (MLP), using positional encoding or frequency-awan acti- vations for compact, efficient representation, crucial for image synthesis via iterative optimization. We adopt FINER (Liu et al., 2024), which enhances fine-detail modeling using variable periodic
3

---

Published as a conference paper at ICLR 2026
Forward Pass
Encode CLIP
Loss
Modality Gap Crosiig
Plants gridl 4)
Prodict gisel colors
A Angaentation
image
Model Iuversion
Layer schetullng
Input Text
M
NW
Chattccnne in a hallvays
0T
Toa Enibodinn
Backward Pass
Decode by inzersioo
Figure 2: CLIP-1 text-to-image inversion pipeline. The image represented with Implicit Neural Representation (INR) fo(i,), optimizing weights to match an input text prompt. The inversion starts from a robust INR trained with Adversarial Weight Perturbation (AWP). The optimization updates the INR layer- wise so that the embedding of its rendering aligns with the text prompt embedding. The procedure includes augmentations, CLIP embeddings averaging. and projection onto the unit sphere. To align text and image embeddings, apply orthogonal Procrustes transformation address the modality gap in CLIP.
activations. It is based on SIREN (Sitzmann et al., 2020), which uses fixed- fixe-frequency activation z, sin (w (Wizzi 1 b,)). FINER improves adaptability by introducing an additional coeffi- cient:
2, = sin (war; (W, b:)), a; = --: 22 +
where a, dynamically sets local frequency based input magnitude. We leverage FINER's bias initialization, which stratifies frequencies across layers, capturing low frequencies early and high frequencies deeper, leading to better convergence and reconstruction. See Fig. 6 for illustration.
3.2 DATA PREPARATION
The computation of the CLIP embedding of both images and captions of the chosen dataset is per- formed offline once and stored in lightweight index using FAISS (Johnson al., 2019), which is then used to retrieve the dataset sample closest to the input prompt. We used images from LAION Aesthetics, subset of LAION-5B by Schuhmann al. (2022). This data serves two purposes: (i) training INRs for initializing text-to-image optimization and (ii) computing natural image CLIP embeddings to act as anchor points the latent space.
Each image is first blurred using Gaussian filter to suppress high frequencies, providing a smoother starting point for inversion. An INR then trained to reconstruct the blurred image, and its weights stored. For the i-th image, the predicted pixel values from the INR fai. with weights øi. The INR images are then encoded via CLIP to obtain visual embeddings 0, (f The CLIP text embeddings Or (y*) of the corresponding captions are also stored. This results dataset D (Jr())
( Sas. Or (y²)), containing CLIP image embeddings, INRs, and CLIP text embeddings for each training sample.
Robust INR initialization. The INR weights øf can be viewed as representations the i-th image- each uniquely capturing its content, like the RGB representation pixels. However, small pertur- bations in these weights can significantly alter the reconstructed image, making them sensitive and potentially unstable for downstream tasks. To address this, propose training method that im- proves INR weights robustness by incorporating adversarial perturbations during training. Given an INR with weights the perturbation range. The training objective is to make the model resilient to such perturbations by solving the following min-max optimization:
define an adversarial weight perturbation (AWP) Ao Ω. where Ω bounds
min max L (fo-ss. blur(x))
AgEn
4

---

Published as a conference paper at ICLR 2026
where fo+20 is the perturbed INR output and L is the reconstruction loss function w.r.t the blurred target image x. Unlike standard adversarial training in the input space (Wu et al., 2020), our ap- proach perturbs only the model weights. The perturbation set is constrained by a relative norm bound Ω = {A ||]| S >]|@[]], with y controlling the allowed perturbation magnitude. This robust training is applied only once, during the construction of the initial INR the inversion step n = O. It ensures that early optimization steps do not cause the weights to drift too far from the frequency content of the initialization, improving stability during inversion. The supplementary ma- terial details the training procedure using the AWP algorithm ($A.3). Since training a single INR is fast, can be fit on the fly, eliminating the need store weights for the entire dataset. The resulting weights can be cached and reused for reproducibility and faster consecutive generations with the same initialization, or discarded to encourage more diverse samples.
3.3 INITIALIZATION AND MODALITY GAP HANDLING
Inversion initialization. Text-to-image generation begins with a text prompt y. which is encoded using the CLIP text encoder to obtain e, = Or (y) E R To initialize the inversion, we search our dataset for the INR whose associated caption has the highest cosine similarity to ez. This serves as the starting point for the optimization.
Bridging the modality gap. CLIP aligns text and image embeddings globally, projecting them onto the unit sphere. However, local differences between modalities persist: text embeddings tend to encode abstract semanties, while image embeddings reflect concrete visual features. Directly optimizing an image to match text embedding often causes artifacts lextual hallucinations - the model overfits to abstract concepts and produces unrealistic visuals (Liang et al., 2022a).
To address this, we learn local transformation to align text and image embeddings more precisely. We retrieve the nearest neighbors of the input text embedding e, from our dataset D, forming two matrices: ET E _d encoding the k closest text embeddings, and E, E dgd* encoding their corresponding image embeddings. Solving the orthogonal Procrustes problem we find an orthogonal matrix R that best aligns these two sets:
min REE E,llF R
s.t. R R
(3)
where I1e is the Frobenius norm. This is performed for each input prompt. The resulting or- thogonal matrix R aligns the local structure of text embeddings with that of image embeddings. We then transform the input text embedding into the image modality as et2 Re,, which becomes the target embedding for the CLIP inversion process.
3.4 INVERTING CLIP WITH IMPLICIT NEURAL REPRESENTATIONS
Text-To- Imxa-To- via CLIP Inversion. Our pipeline, shown in Fig. 2, inverts CLIP text embedding to synthesize an image using an INR. Given text prompt y, we obtain the projected image-space embedding e:2; RØr (y). The INR fo is optimized so that its output image matches e:2 when passed through the frozen CLIP image encoder O,. This is formalized as:
o = arg min
where
Gr2 = -ROT (y).
(4)
Here, is the cosine distance, and gradients flow from CLIP back to the INR parameters Φb
Layer-wise frequency optimization. Instead of optimizing pixel values, we update the INR weights, leveraging FINER's 's property that network layers correspond to different frequency bands. The INR is structured as an I--laer MIP, with each layer representing specific frequency range. To guide the optimization process, we apply Gaussian learning rate scheduling: at each iteration, we focus the optimization a specific layer by assigning a peak learning rate, while attenuating the rates of neighboring layers according to a Gaussian curve (see Fig. 7 in the Appendix). This helps reconstruct coarse features before fine details, improving stability and fidelity.
Augmentations for stable optimization. Following CLIPDraw (Frans et al., 2022) and CLIPAG (Ganz & Elad, 2023), we apply color, scale, and shear augmentations during optimization. Augmentations are CLiP-encoder averaged, and projected onto the unit sphere: ea = Σi= 0, (augment(ff enforcing robustness to distortions.
5

---

Published as a conference paper at ICLR 2026
Table I: MS-COCO text-to-image generation results. FID (lower better), CLIPSIM and IS (both higher are better), along with model sizes.
Betker al. Nichal et al. Rombocc al. Gana & Elad Ganz & Elad Gunz & Elad
Kazemi et al. Fort Whinkke Fort & Whituker
(2023)
(2022)
(2022)
(2003)
(2024)
(2024)
(2024)
(2025)
(2025)
Ours
DALLE
GLIDE
LDM KL.
CLIPAG
EB CLIP
EB CLIP
CLIP lav
DAS VIT
DAS
CLIP
VIT
VIT
XXL.
VI
Ensemble
Synthesis inversion
Tuning-feee
params (M)
88
perams
12000
150
150
9x1.0
RIX
27.5
41.5
161
121.6
72.5
(LIPSIM(1)
34.7
34.5
33,5
61.4
22.7
36,9
38.5
18(1)
17.9
2003
15.7
4.3
5.7
8.36
9.5
Blending natural image priors. To further guide generation toward realistic outputs, we incorporate information from natural images. For given prompt y, retrieve the k most similar image embeddings from a reference dataset using cosine similarity. These are linearly combined into a blended target embedding fmg with weights given by the softmax of the similarity scores. A blending loss Liiend then encourages the output embedding of to remain close to the manifold.
Final optimization formulation. The complete CLIP-1 optimization pipeline updates the INR pa- rameters e to generate realistic images, leveraging both augmented embeddings and natural image priors, with no CLIP retraining modification. The full procedure is described in Eq. (5)
(a) e = - Σ 0, 0 (augment(fax))
(b) Φ = ming maxaşen (fo+as. blur(x))
(5)
(c) on Φn-1 V [(e; e:2i) ) + 3Lstend (e) (gng)
Step (a) computes the CLIP embedding from the augmented INR outputs; step (b) initializes the INR with adversarial weight perturbations enhance robustness, and (c) updates the INR weights via backpropagation using both alignment and blending losses.
4 EXPERIMENTS
We evaluate our inversion pipeline across a range of tasks. We begin with text-to-image genera- tion on MS-COCO (Lin et al., 2014), comparing against both standard generative models and prior inversion- approaches, along with experiments to prove robustness out-of-distribution sam- ples 1). Next, we demonstrate the generality of our method through zero-shot transfer to down- stream tasks, including reconstruction, controlled modification, and style transfer ($4.2). Lastly, we quantify the effect of each component through an ablation study ($4.3). Additional experiments in the Appendix ($A.1) further validate the generative behavior of CLIP-1, showing consistent align- ment while capturing natural pixel-level variability across runs.
4.1 TEXT-TO-IMAGE SYNTHESIS BY INVERSION
Setting. The goal is to generate visually realistic images that are semantically aligned with a natural language description. To assess the visual fidelity of our method, we compute the Fréchet Inception Distance (FID) (Heusel et al., 2017) and the Inception Score (IS) (Salimans et al., 2016) over a subset of 10, 000 captions from MS-COCO (Lin et al., 2014). Since these metrics do not capture semantic alignment with the prompt, we also report CLIPSIM (Hessel et al., 2021), which measures the cosine similarity between the CLIP embeddings of generated images and their corresponding captions. We compare our method against prior and concurrent inversion-based approaches, including those that require CLIP fine-tuning (Ganz Flad, 2023; Ganz al., 2023) and those that do not (Fort & Whitaker, 2025; Kazemi et al., 2024). As our method uses frozen CLIP and pretrained decoder, we regard tuning-free free baselines as the most relevant points of comparison. For completeness, we also report results from tfate-of-the-ar generative models that rely both training and dedicated decoder (Betker et al., 2023; Nichol et al., 2022; Rombach et al., 2022). The implementation details can be found in the supplementary material.
Results. Table reports our results along with model sizes and architectural requirements. Among inversion-based methods without pretrained decoders CLIP tuning, CLIP-1 achieves the lowest

---

Published as a conference paper at ICLR 2026
DAS Fost Whitaker (2025)
CLIP-Iav et all (2024)
CLIP w/Ganz Blad (2024)
CLIPAG Ganz & ald Game Blad (2023)
CLIP-JEM
CLIP-1 (VT--//3))
CLIP-1 (REeNTT)
CLIP w/ Gana blad (2023)
(2024)
(XLL)
eM streaking through the night s4ya
<A mi solse-d field at daybrea' with wildf owers g aattennng in early rays.
Figure 3: Qualitative comparison of prior pixel-based methods against different CLIP configurations.
original
Rec.
Rec.
2
Original Prompt
Prompt 2 Original Reference Reault
Reconstruction
(b) Controlled Modification
Neural Style Transfer
Figure 4: Downstream Tasks. (a) Recreates the input image from its CLIP encoding. (b) Alters the input image based on a specified prompt. Prompt 1: <Snowy peaceful landscapes Prrompt 2: *Torrential rainfall, lightinn boltso, (c) Applies the visual style of reference image the input.
FID (72.5 vs. 161.8 for DAS-VIT (Fort & Whitaker, 2025)) and highest (9.5 vs. 5.7), marking substantial improvement in visual quality. Although diffusion- based models still attain lower FID, they require orders of magnitude more parameters and full training pipelines, whereas our method uses frozen backbone and lightweight INR. Finally, method achieves CLIPSIM of 38.6, outperforming both training-free free and fine-tuned exselites-xxcept for CLIPInvert (Kazemi et al., 2024), whose higher CLIPSIM can be attributed overfitting the target embedding. evidenced by its elevated FID and low IS. Taken together, these results indicate that Procrustes alignment and frequency-aware INR optimization effectively improve text-image consistency without modifying CLIP's weights Figure 3 further provides qualitative examples: compared to other training- free baselines, our generations exhibit fewer structural artifacts and sharper details, and visually ap- proach the quality of tuned approaches (Ganz & Elad, 2023; 2024). We also apply our pipeline in a plug-and and-play fashion tuned CLIP variants, such as CLIPAG (Ganz Elad, 2023) and CLIP- JEM (Ganz & Flad, 2024), showing broader compatibility of our method with discriminatively trained models. Additional results are available the Appendix in Figure 8.
Robustness to distribution shift. We ablate initialization, Procrustes alignment, and blending loss, forcing the Plain CLIP-1 variant to start from random INR weights. Despite this, it consistently outperforms the DAS baseline MS-COCO and Flickr30k (Table 3a). Since all init/anchor embed- dings derive from -esthhetss,,,,s these benchmarks constitute a strict OOD evaluation. Both Plain and Full CLIP-1 retain strong performance under this shift, with initialization serving only as an optimization accelerator. Cross-datases Fréchet distances (Table 2) confirm COCO and Flickr lie far from LAION in CLIP space, underscoring the OOD nature of these benchmarks.

---

Published as a conference paper at ICLR 2026
Abstract Prompts
Negations
CLIP
Plain CLIP
CLIP
Plain CLIP
Init
Final
Init
Final
Init
Final
Init
Final
<A visually incoherent composition.
<his is not photo of a dog.
structure with impossible geometry..*
This not a photo food.>
<An abstract and unrecognizable object.
This not photo car.
AA fictional entity that does not resemble anything real.-
<TTis is not photo of a building.*
<An object without delinuble atiributes.e
<'Tiis is not photo of an animal.o
Figure 5: Explainability of CLIP. Exploring CI.IP's behavior under out-of-distrihution concepts.
4.2 ZERO-SHOT TASK GENERALIZATION
To assess the versatility of our approach, we explore sev- eral downstream tasks, demonstrating that the same in- version framework can successfully generate images in different settings without requiring task-specific modifi- cations or additional optimization.
Table 2: Cross-dataset Fréchet Distances (mean std, runs) on CLIP embed- dings of 30k images and captions. High distances confirm COCO and Flickr lie out- side LATON's embedding space. Numbers
in 10
Image reconstruction. The goal to reconstruct a given input image using inversion pipeline. We treat the image as target whose CLIP embedding is known, and optimize INR produce output that matches this embedding. The INR is initialized from a blurred version of the image, and refined align with the embedding of the full-resolution input, effectively operating as decoder recovering semantic content from latent space. Unlike text-to-image generation, this task provides well-defined ground truth and serves controlled setting to evaluate inversion precision. Figure 4a shows qualitative results on both artistic and photographic inputs. Across all examples. high-level semantic content, such as facial identity or scene composition, is consistently preserved. Fine-graine spatial details, especially in structured regions like faces or buildings, are approximated with some distortion or shift, reflecting the inherent ambiguity of CLIP's embedding space.
Metric
Images
Caption
Bascline (LAION split)
3.4 3.1
6,8 1 0.1
Baseline (COCO split)
4.2 0.1
0.1
Buseline (Flickr split)
0.1
4.0 0.1
COCO LAION Flickr vs. LAION
365.3 0.7
463.9
366.9 - 0.9
503.4
Controlled image modification. The goal is to modify an input image according to natural language prompt that specifies targeted change in content style. The image is first encoded via an INR fitted to its original form. text prompt is then provided to guide the modification (e.g.. "snowy landscape" or "torrential rainfall"). The INR is optimized to align the CLIP embedding of the generated image with that of the prompt, while starting from the original image representation. This setup encourages localized, semantically consistent transformations without disrupting the broader
8

---

Published as a conference paper at ICLR 2026
structure or identity of the scene. Figure 4b shows three examples for the task. In each row, the left- most column is the original image; the next two columns show the edits for "snow" and "storm". The road, the Great Wall, and the city keep their geometry and colour palette, while only the requested weather effects (snow cover, rain streaks, lightning) are added. This confirms that CLIPP- can act as prompt-driven image editor, producing targeted edits without explicit masks or additional training.
Neural style transfer. We supply two images: a content photo and a style reference. The content image is represented by an INR initialized exactly reproduce the original photo; the style image is fed only through the frozen CLIP encoder. Optimization minimizes a weighted sum of two CLIP- based losses: (i) a style loss that pulls the INR's embedding towards that of the reference painting, and (ii) content loss (weight 0.5) that keeps the embedding close to the original photo. Figure 4c illustrates the outcome. In each case, the brush-stroke texture and overall palette of the reference painting are transferred, while object layout and scene geometry remain intact. The method therefore separates appearance from semantics without hand-crafted losses or additional training. indicating that the inversion pipeline can exploit CLIP's latent space to disentangle style from content.
Explainability. Prior work shows that negative prompts and out of distribution prompts can reveal how CLIP structures semantic space and separates in distribution concepts from everything else Li et al. (2024); Nie et al. (2024). We offer a complementary view: we use our pipeline to directly visualize how CLIP interprets such texts. We evaluate two types of out of distribution targets: negative prompts such as "this is not a photo of a", and generic out of distribution prompts such as Man abstract and unrecogni zable object". These reconstructions allow us to inspect how CLIP responds to these prompts and to identify which visual cues it treats as discriminative or unstable outside its training distribution. Results are shown in Fig. 5.
4.3
ABLATION STUDY
Model
MS-COCO
FlickrS0k
Variant
FIDI
CSIM:
IS1
HDD
CSIM)
iSt
PDD
CSIM 1S1
CLIP 1
107.1
38.8
7.7
Steps
40
400
DAS (VIT)
161.8
22.7
5.7
220.5
38.1
6.5
w/o Freq, Ope (EO.)
30.5
7.8
FIDI
107.1
72.5
DAS (Enn.)
121.6
36.9
8.3
161.1
39.3
7.1
iit. w/o AWP
121.0
43.0
7.3
CSIM
17.8
38.6
Plain CLIP-
we F.O. Proc.
111.3
46.4
9.1
CLIP
72.5
38.6
9.5 86.4
41.1 7.3
R. wio Litend
119.7
49.5
7.9
1St
10.6
9.5
(a) Out of distribution evaluation
(b) Quantitative ablation study
(c) Eval. vs steps
Table 3: (a) Out of distribution evaluation on MS-COCC/Flick330k Even without initialization. CLIP-1 surpasses DAS across all metrics. (b) Quantitative ablation study on on 1,000 MS-COCO captions; legend- i. full model - ii. frequency scheduling -ü. AWP iv. F.O. Procrustes W. F.O. + blending loss. (c) Ablations wrt to the number of steps in the inversion.
Ablation for the proposed components. We now perform controlled ablation over the four key components: layerwise frequency scheduling, adversarial weight perturbation (AWP), orthogonal Procrustes alignment, and the natural- -mage blending loss; the corresponding ablated variants referred to as (i), (ii), (iii). (iv). We run every variant in the same 1000 captions from MS-COCO and report FID (Heusel et al., 2017), CLIPSIM (Hessel et al., 2021), and IS (Salimans et al., 2016) in Ta- ble 3b. Better FID/IS and higher CLIPSIM show better perceptual realism and stronger text-image agreement, respectively. In parallel, we visualize representative generations so that the numerical shifts can be linked to visual outcomes. More results can be found in the supplementary material.
The ablation shows that each proposed component plays a distinct, complementary role (see Fig. 9). When the layer-wise learning-rate schedule is removed (ii) the INR is forced to optimize all fre- quency bands simultaneously; high-frequency layers overfit first, so fine textures emerge before the coarse layout has stabilized; the premature detail introduces stripe-like artifacts and drives FID to its worst value. Dropping AWP (iii) preserves the coarse-to- to-fine dynamic but does not constrain weights to remain on the manifold defined by the robust anchor, introducing neural artifacts; this allows the unconstrained result to align more closely with the caption, increasing CLIPSIM at the expense of realism (FID 1). Similarly, eliminating the orthogonal Procrustes projection (iv) pushes the optimization toward the raw text embedding. i.e. slightly outside the image sub-man mannmadifodd: CLIP rewards the closer alignment (CLIPSIM 7), but the outputs become noticeably busier, with sharper outlines, cluttered details, and occasional duplicated elements. Finally, disabling the blending loss (v) stops the optimizer from referencing real-photo statisties; colors turn harsher and small objects

---

Published as a conference paper at ICLR 2026
50
= 100
= 150
= 200
250
f
300
i 400
CLIP
wVo AWP
wif Freq. Opt.
w/o both
<Panoranic
landscape with dramatic Lighting detailed
duness
Figure 6: Qualitative ablation. Text-to Text-to-image synthesis of desert landscape over 400 iterations, comparing CLIP with ablations: without AWP, without frequency-base based optimization, and without both.
appear in duplicate, which degrades FID yet inflates CLIPSIM the model over-expresses caption tokens. In the full model (i), frequency scheduling suppresses glitches, AWP keeps the solution near stable anchor, and both Procrustes and the blending loss guide the search along the natural-image manifold, trading few points of raw caption similarity for substantially higher visual quality. The qualitative grid mirrors the numbers: (i) the only variant that is simultaneously aesthetic, coher- ent, and semantically faithful. Figure traces prompt through 400 inversion steps to show how the most critical components influence the optimization. Two broad patterns emerge. (i) Frequency scheduling governs the refinement path: when it is present (top two rows) the image is generated in a coarse-to-fine order where color appears first, then shapes, then texture; without (rows 4) high-frequence stripes appear almost immediately and persist. (ii) AWP mitigates cumulative drift: when it present (rows 1 & 3) the global scene layout stays stable throughout optimization, whereas its absence (rows and 4) lets distortions and noise grow with every iteration. The variant lacking both safeguards shows the combined failure modes, underscoring their complementary roles.
Ablation for number of inversion steps. Table 3c shows instead the performance varying the number of inversion steps. Similar to the recurrence of flow-based model, we also have a recurrence when tough increase the steps 400 our models does not overfit likes the other; also, if we decrease the inversion to 40 steps for faster synthesis, the performance drops but still higher than current competing inversion methods.
inverting. Here show that
CONCLUSION
We present CLIP-1 inversion-based approach that frozen CLIP image encoder and no pre- trained decoder for text-to-image synthesis through implicit neural representations (INRs). Instead of relying on generative decoder, we show that CLIP, combined with frequency-aware INR and lightweight alignment step, can guide image synthesis directly from text prompts. Our aim is not to match state-of-the-and generators, but to highlight an underexplored capability: a frozen discrimina- tive model can still produce coherent, semantically aligned images without any additional training. The same setup also handles zero-shot reconstruction, controlled edits, and neural style transfer, and offers an interpretabilityy tool: CLIP inversion reveals how the encoder responds to negations and out-of-disee d-distribution prompts, helping expose biases or misalignments before they propagate into larger multimodal systems. These results suggest directions for repurposing pretrained models and broader implications for robustness and interpretability. Current limitations reflect the nature of CLIP inversion: fine details remain under-constrained and may introduce local artifacts, since all regularization comes from the INR and the CLIP-based losses. A lightweight natural-image prior or projection step onto the manifold could further improve fidelity.
10

---

Published as a conference paper at ICLR 2026
ETHICS, AND REPRODUCIBILITY
Ethics. We assert that this work does not raise identifiable ethical concerns or foreseeable negative societal consequences. Rather, our contributions point toward improving the explainability of clas- sifier models and their hidden biases, and toward future extensions enabling image generation on commodity hardware and controllable image editing. Our generator, however, guided by a frozen CLIP model and therefore inherits its known societal biases, potential misuse risks (Kazemi et al., 2024), and conceptual limits. To mitigate these issues, we performed preliminary experiments with Safe-CLIP (Poppi al., 2024), which successfully blocked harmful content, providing encouraging evidence that safety-awane integrations can be effective.
Reproducibility. For reproducibility, we carefully document our full pipeline in ($A.4), along with the complete AWP algoritm ($A.3), providing a step-by-ster description of the inversion process. Detailed implementation settings, including all hyperparameters used in our experiments, are further reported in ($.))..
Acknowledgment. This work supported by projects PNRR MUR PE0000013-FAIR under the MUR National Recovery and Resilience Plan funded by the European Union NextGeneration PRIN 2022 project 20227YET9B "AdVVentt CUP code B53D23012830000 and MUR FIS2 grant n. FIS-2023-00943 "NEXUS" (cup B53C250010300011, It was also partially supported by Sapienza research projects D2QNeT and BEAT (Better dEep leAming securiTy) bandi per la ricerea di Ateneo 2024 and Seed of ERC grant "MINT.AI". Computing was supported by CINECA cluster under project Ge-Di HPIOCRPUVC, RDM HP10C7YYL2 and the Sapienza Computer Science Department cluster. The authors would like to thank Andrea Salinetti for his initial work and bachelor's thesis on CLIPP inversion.
REFERENCES
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall-e-3 pdf, 2(3):8, 2023. 1, 3, 6
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations. 2
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming- Yang, Kevin Patrick Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to- image generation via masked generative transformers. In International Conference on Machine Learning, pp. 4055-4075. PMLR, 2023.
Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Decoupled kullback-leibler divergence loss. arXiv preprint arXiv:2305. 2023. 3
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, pp. 12606- 12633, PMLR, 2024.
Stanislav Fort and Jonathan Whitaker. Direct ascent synthesis: Revealing hidden generative capa- bilities in discriminative models. arXiv preprint arXiv:2502.07755 2025. 2, 3, 6, 19
Kevin Frans, L. B. Soros, and Olaf Witkowski. Clipdraw: exploring text-to-drawing synthesis through eagguage-image encoders. In Proceedings the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
ISBN 978171387101
Roy Ganz and Michael Elad. Clipag: Towards generator- free text-to- ext-to-image generation, 2023. URL https: / axxiv owwooraaaaooooo 6306. 6805. 2, 3, 5, 6, 19
Roy Ganz and Michael Elad. Text-to -imtat generation via energy- based clip, 2024. URL https: 17046. 6. 2, 6, 19

---

Published as a conference paper at ICLR 2026
Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply robustness? In ICML,, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.0871 08718, 2021. 2, 6.9
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPS, volume 30, 2017. 6,
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Car- lini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, et al. Measuring forgetting memorized training examples. In ICLR, 2023. 18
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3):535-544 547, 2019. 4
Hamid Kazemi, Atoosa Chegini, Jonas Geiping, Soheil Feizi, and Tom Goldstein. What do learn from inverting clip models? In Neurips Safe Generative Al Workshop 2024, 2024. 2, 3, 6, 7, 11, 19
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018. 3
Tianqi Li, Guansong Pang, Xiao Bai. Wenjun Miao, and Jin Zheng. Learning transferable nega- tive prompts for out-of-distrio dittr--diitiitttion detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17612-17625. Cuman Associates, Inc., 2022a. URL //ptocedig/ wew.eeeeeddnnn....neeeeeeeeeeeee cc wttps/ffleessseeeeeeeeeee #1lee/7224db75433e
Paper Conferenceeecceee pdf.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung. and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances Neural Information Processing Systems, 35:17612-17625, 2022b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 2,
Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodis activa- tion functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2,3
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. Advances in Neural Informa- fion Processing Systems, 46:55394-55414 2023. 2
Mujtaba Hussain Mirza, Maria Rosaria Briglia, Senad Beadini, and Iacopo Masi. Shedding more light on robust classifiers under the lens of energy- based models. In European Conference on Computer Vision, pp. 451 451-468. Springer, 2024. 3
Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci. Marco Bertini, and Andrew D Bagdanov. Cross the gap: Exposing the intra-modal misalignment clip via modality inversion. ICLR 2025, 2025. 2
12

---

Published as a conference paper at ICLR 2026
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning, pp- 16784-16804. PMLR, 2022.
Jun Nie, Yonggang Zhang, Zhen Fang, Tongliang Liu, Bo Han, and Xinmei Tian.
detection with negative prompts. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki. M. Khan, and Y. Sun (eds.), International Conference on Representation Learning, volume 2024, pp. 46394- 46413, 2024. URL www.t:ttps//ccmmmmmm http::///poooeeedeeeeeoo iclr ec wwwpp/ppppp.../.eeeeeeeeeeeeeeee www.f//aaeeeeeeeeeeeeeea http://98877998888 7tt.://87777..111000030767777763
pdf. 9
Samuele Poppi, Tobia Poppi. Federico Cocchi, Marcella Comia,, Lorenzo Baraldi, and Rita Cuc- chiara. Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models. In Proceed- ings of the European Conference on Computer Vision, 2024. 11
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1.06434, 2015. 2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 22Xi::2204.06125, 2022. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022. 3,6
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-imaae diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 2, 6,
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale -o-tgee synthesis. In International conference on machine learning, pp. 30105-30118 PMLR, 2023. 2
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural formation processing systems, 2022. 4
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473. 2020. 3,4
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains, 2020. URL https:
//arxiv. org/ abs /2006.00.00 10066..7730 10739.
Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthesis, 2023. URL https: //axiii org/ fabb/22311 12959. 2,
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024. 3
13

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Chang Wang and Sridhar Mahadevan. Manifold alignment using procrustes analysis. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pp. 1120-1 127, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605582054, doi: 10.1145/
10001566.39097... URL https: htt.//ii.cctte orq/ 10. 1145/13902 1390297. 2
Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. Clip-gen: Language-free training of text-to-image generator with clip. arXiv preprini arXiv:2203.00388 2022.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener-
alization. Advances in neural information processing systems, 33:2958-2966 2020. 2, 3,5
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019. 3
Yuhui Zhang, Elaine Sui, and Serena Yeung-Levy. Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data. ICLR 2024, 2024. 2
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
This appendix expands on key aspects of our work by providing additional technical details and extended qualitative results. It is organized into eight sections, each addressing a specific area that complements the main paper. Section Section A.I examines the generative capability of CLIP-1 highlighting its ability to synthesize diverse yet semantically consistent images. Section Section A.2 analizes runtime and resource efficiency. comparing memory usage against prior baselines. Sec- tion A.3 offers a detailed explanation of the adversarial weight perturbation (AWP) training proce- dure, including how the perturbation is computed and integrated into the overall training pipeline. Section A.4 outlines the complete algorithm used for the text-to-image task. Section A.5 describes the implementation setup, including model configurations, training parameters, and data preprocess- ing choices. Section A. 10 presents the best and worst 500 generations from MS-COCO prompts, the same samples used in the paper's evaluation. Section A.6 provides qualitative examples generated with different pre-trained VIT-B/32 models, illustrating the variability introduced by different back- bone initializations. Finally, Section A.7 includes additional qualitative results from the ablation study, offering visual comparisons that highlight the contributions of individual components. This appendix is intended support reproducibility and provide deeper insight into our methodology and experimental findings.
A.1 GENERATIVE CAPABILITY AND INHERENT STOCHASTICITY
While our full pipeline leverages an informative initialization to improve convergence and quality, the generative capability of CLIP does not depend on it. The inversion framework can synthesize images directly from a randomly initialized INR, which effectively acts as a noise input.
In generative models (GMs), GANs sample (0 1] and diffusion models _N 1]. In our
case, fixing the FINER INR's inductive bias, each weight W, is drawn from 21 6/n, v 6/n and p). each bias h; ~ (1-p, p), where n is the neuron input count, the initial high-frequency capacity.
This defines tractable high-dimensional prior (d >> k k) over the INR weight space, refined by inversion on the text prompt. Different produce distinct optimizations and outputs. This is novel in our setting but similar the multivariate Gaussian in DDPM with d k x k sampling in GAN where d < k x k. Thus, our noise sources are tractable and sufficient for diversity.
To further assess generative behavior, we fixed a text prompt and generated multiple samples. We measured (i) pixel-space variance across runs and (ii) the mean + std of CLIP similarity between each generated image and the prompt. Results in Table indicate that CLIP- achieves substantially higher pixel-space variance than DAS while maintaining comparable CLIP alignment. This confirms that our method produces semantically consistent yet visually diverse outputs. The per-channel variances closely match those of natural images, further validating the realism of the diversity.
Table 4: Intra-prompt stochasticity. CLIP-1 achieves higher pixel variance while keeping semantic align-
ment stable.
Metric
CLIP
DAS
Variance the pixel space
0.0447
0.0060
Variance & the pixel space (per channel)
[0.0432, 0.0442, 0.0467]
[0.0052, 0.0070, 0.0057]
Alignment to prompt (mean + std)
0.3151 + 0.0058
0.3033 + 0.0078
15

---

Published as a conference paper at ICLR 2026
A.2 RUNTIME AND RESOURCE ANALYSIS
We report the resource usage in Table 6, compared to DAS. The higher VRAM of full CLIP is due to training the Init INR on-the-fly, an overhead removed when cached INRs are reused.
Table 5: Hardware usage comparison Average inference time and peak VRAM. (NVIDIA RTX A6000)
Model
Time (s)
Peak VRAM
DAS Ensemble
44.88
5.3 GB
CLIP (cached INR, 400 steps)
22.88
3.2 GB
CLIP-1
(on-the-fly, 400 steps)
29.19
4.3 GB
To evaluate whether the pipeline RTX 4060 (8GB VRAM). We use 40 optimization steps, since Table 3c shows that this setting does not significantly affect image quality. Compared to GLIDE and LDM-KL-8, CLIP-I is notice- ably more efficient: GLIDE requires more time and memory, while LDM-KL-8 goes out of memory on the same hardware. CLIP-1 therefore remains the method in this comparison that runs most comfortably on lower-end GPUs.
mid-range consumer GPU, we tested it
Table 6: Hardware usage comparison consumer GPU Average inference time and peak VRAM. (NVIDIA RTX 4060, 8GB VRAM)
Model
Time
Peak VRAM
DAS Ensemble
Im 26s
5.3 GB
LDM-KL-8
00M
GLIDE (Tex12lmg)
19.2s
3.7 GB
GLIDE (CLIP-Guided)
9.8s
3.9 GB
CLIP 1
(40 steps)
7.8s
3.2 GB
A.3 FURTHER DETAILS ON THE AWP ALGORITHM
We provide a detailed breakdown of the Adversarial Weight Perturbation (AWP) procedure and its integration into the INR training loop. Algorithm 1 outlines the core AWP mechanism: given the weights of the INR model and the input coordinates (i.j) and a temporary clone & optimized to maximize the negative structural similarity index (SSIM) loss between the predicted output and blurred version of the ground-truth image x. The resulting adversarial perturbation Áo is computed, normalized, and applied to the original weights & to obtain the perturbed weights Oadv-
Algorithm 1: Adversarial Weight Perturbation (COMPUTE
1 Inputs: INR mudel weights Φ) input coordinates (i,j), larget image
1 $ t- clone(6)
// proxy model initialization // maximize the loss
Lump -EsuuM (Jz f)), blur(x))
4 Optimize w.r.l. Luup
Ad f-
// compute perturbation // normalize and scale perturbation
Ad 7 1o Ad
Return: AΦ
Algorithm illustrates the incorporation of AWP into INR training. At each iteration, adversarial perturbations are computed using Algorithm and then applied to the network. The overall training loss is a weighted combination of mean squared error (MSE). SSIM, and & loss. This adversarial training scheme improves the robustness and generalization of the INR by encouraging consistency under weight-level perturbations, which are the gradients received by inverting CLIP when generat- ing.
16

---

Published as a conference paper at ICLR 2026
Algorithm 2: INR Training with AWP
Inputs:
target image initial weights doo input coordinates (i,j) Hyperparameters: learning y, perturbation scale iterations N for k I...., N do
Aø +- COMPUTE_ (t,
@hdl 9s AΦ
11 apply perturbation
Sor model with weights Oudv
[ ((i,))
C an LmSE (R, blur(x)) + azz ( blur(x)) - asCu1( blur(x))
9 Update Paer via optimizer step minimizing // restore original weights for next iteration
10 @l++1 +- theer AΦ
Return: trained weights ON
A.4 TEXT-TO-IMAGE FULL PIPELINE
We optimize an implicit neural representation (INR) to synthesize image that aligns with a given text prompt using CLIP. The procedure includes text and image retrieval, feature alignment, and iter- ative gradient-bass based optimization. Procrustes alignment and natural image constraints are enabled. Algorithm shows the detailed pipeline.
Algorithm Text to Lext-to-Imape Image Synthesis
Input: Text prompt y.
Output:
Synthesized image x faN ((1,j))
/+ Input pre pre-processing and alignment
Encode the input prompt e; 0,
Select top-k matches to e, in D to build Ey, E Rdik Compute on the fly orthogonal Procrustes rolation matrix
s.t. R R =I Project e, visual domain: e:2 R0, (y)
R = minR ||RE Eylle
/. Retrieve Initialization
From dataset D. retrieve image embedding 0y(x) of & with caption closest e,2i 10 Initialize INR weights do +- INIT INIT_INR LN__INR__A (WW(())-aa Algorithm
/* Natural Image Constraints:
Retrieve top-k natural images {xx} near e; in CLIP space
13 Encode them to features {emggs)
14 Store the similarity to input prompt Wy CLIPSIM(X3, ex)
Compute weighted average:
eimng Σj Mygkngg where vi are normalized w/ softmax
16
/ Opt imiizer zer Setup:
Initialize layer-wise optimizers with Gaussian learning rates (peak is 7) over INR depth. Schedule shifting of Gaussian center every steps-see see Fig.
19
20 for
I1
if learning rate schedule triggers then Shift Gaussian center layer
22
Encode via CLIP the augmentations of the rendered INR:
Or (augmen((fa:))
Compute total loss:C(e,, ev2c) + B BLaiond (e,, e)nng) Update di
24
25
Φn Φ V L(ej, ezzi) BLesend (ef eimg)
Return: Final image
fax ((b,5))
17

---

Published as a conference paper at ICLR 2026
Gaussian attenuation
M
MN
R
G
Ir
B
(c) peak learning rate
Figure 7: Gaussian Scheduling, Each layer represents a frequency interval (b) for fa. The learning rate is centered specific layer and gradually shifted (c), decreasing with Gaussian attenuation across neighboring layers (a).
A.5 IMPLEMENTATION DETAILS
A.5.1 INR PARAMETERS AND INITIALIZATION
We initialize implicit neural representations (INRs) with in_ features and out_features = 3, using five hidden layers of 256 units each. Sinusoidal pa- rameterization is applied with first_omega enable high-frequency signal modeling. Training is performed using the Adam op- timizer with learning rate of 1 10-4, and a cosine annealing schedule torch, opt imm l rr_acheduler. Cosi neAnnea ingWarmResartss with restart period of 100 iterations.
25 and hidden_cmega
25 to
We apply Adversarial Weight Perturbation (AWP) using proxy optimizer with the same learning rate and a perturbation strength of a 0.01. The ground-truth image x is preprocessed using Gaussian blur with cernel_size = 101 and e (10.0, 20.0) to provide a smoother supervision signal.
The training loss combines mean squared enror (MSE), structural similarity (SSIM), and l recon- struction loss, weighted, respectively, by a = 0.85, a2 = 0.25, and 0g = 0.25.
A.5.2 TEXT-TO-IMAGE INVERSION PARAMETERS
Our method built upon a VTT-B/32 backbone initialized with the default OpenAI weights. We perform 400 inversion steps using the AdamW optimizer (without AMSGrad) and a learning rate of x 10-4, During INR optimization, we employ Gaussian scheduling strategy focused on layers [0, 1, with gradient norm clipping thresholds set to [1.0, 0.5, 0.2) respectively. This schedule is refreshed every 70 iterations to preserve both stability and optimization efficiency over time.
The loss function incorporates hyperparameters 3 0.5 and k 8, balancing the trade-off's be- tween reconstruction fidelity and robustness. To promote generalization, we apply data augmenta- tion by generating 32 variations per input sample. For spatial alignment, we use Orthogonal Pro- crustes analysis over the nearest = 256 elements. Following Stable Diffusion (Jagielski et al., 2023), guide the CLIP inversion process by appending auxiliary textual prompts that explicitly describe desired image characteristics. This strategy improves the fidelity and perceptual quality of the generated outputs.
18

---

Published as a conference paper at ICLR 2026
A.6 QUALITATIVE SAMPLES UNDER DIFFERENT CLIP MODELS
CI-F-JJMM Gana Blad Gana Blad (2024)
CTTPAA
DAS Fort Nhitakes (2025)
CLIP-InV Kazemi 6l. (2024)
CLIP w/Ganz Blad 12023)
CLIP w/Genz slad (2024) [XC1
CLIP (ViT
CLIP-! (RESNET)
wSleigh Ride Sunny
<ML1ky Kay Over Street>
<Man steading cattlew
<Sunset from the creeke
evie of lake from hiking trails
BO
Arbnnd
wAbondoned old barna
H
#S:zeets of London in winters
Chrilee
Filiton
<View of the Chateau de Chillens
Figure 8: Qualitative comparison additional samples extending Figure the main paper.
19

---

Published as a conference paper at ICLR 2026
A.7 QUALITATIVE SAMPLES OF THE ABLATION STUDY
Figure 9: Quantitative ablation study. (a) Results on 1,000 MS-COCO captions. (b) Samples for each case within the same prompt; columns show: i. full model ti. frequency scheduling ii AWP iv. F.O. Procrustes v. F.O. blending loss.
20

---

Published as a conference paper at ICLR 2026
w/o Freq.
w/o EO.
w/o FO.
w/o Freq. Opt
w/o F.O+
w/o EOO, + Blending
CLIP
w/o AWP
Blending
CLIP
w/o AWP
Left: CLIP-1 Ablation Rows 1-10
Right: CLIP-1 Ablation Rows 11-20
Figure 10: Ablation Study additional samples of the ablation study shown in Figure 6 of the main paper.
21

---

Published as a conference paper at ICLR 2026
A.8 HIGHER RESOLUTIONS
Images are synthesized at 448 x 448, which is 2x the original CLIP input resolution. Below we compare the original image, its upscaled version, and the corresponding native synthesis. Generating at higher resolution also induces the model to introduce additional content, as seen in the monastery- on-a-cliff image, where a second building appears in the reconstruction.
22

---

Published as a conference paper at ICLR 2026
Original (upsealed)
Native 448xx488
Figure 11: Native text-to-image higher resolution. Left: images generated at 224 x 224 and upscaled. Right: images generated natively at 448 x 448.
23

---

Published as a conference paper at ICLR 2026
A.9 ROBUSTNESS TO WRONG AWP INITIALIZATIONS
We assess robustness by introducing three types of wrong AWP Initializations that may happens when implementing Section 3.2.
We fixed an initial robust AWP seed matching a prompt and then we invert the correct prompt. This corresponds to the generation with the correct retrieved initialization related to a lake. Then do the following: we kept fixed the initialization of the correct prompt, but invert four different prompts uncorrelated with current initialization. The uncorrelated prompts are:
(a) Some oranges stacked up in bowl
(b) A pastry store with cupcakes on display
(c) Tibetan monastery on a cliff
(d) A group people sitting around a table
We show the result above for three different starting seeds in Fig. 12
The first (i) is a spacious and uncluttered image, such as a lake, used when the target scene is highly cluttered. The second (ii) corresponds seed built from a complex out-of-distribute concept. The third (iit) is a seed whose colors and overall appearance do not match the concepts described the other prompts.
The results show that the semantics remain largely unchanged, which indicates robustness. The seed mainly affects the spatial arrangement of objects and, more strongly, the overall color palette. Even when the palettes differ, the coloring scheme remains semantically consistent across the objects in the scene. In case (iti), however, the initialization departs more from what is expected, due to nature of the adversarial seed itself (hard edges, city-like structures, neon lights), and this leads to more visible artifacts.
Wrong oranges are atokk up how
Wrong Pastzy store with cupcakes display
Wrong Tibetan monastery on elifr
Wrong group people sitting aruun table
Init
Correct Prompt
(i):
Correct seed (ii)z dog wearing sunglasses>
Correct seed (sét): <& dense city st night with neon lightex
Figure 12: Behavior with the wrong initialization We show how generation performs when the initialization does not match the inverted prompt.
24

---

Published as a conference paper at ICLR 2026
A.10 BEST vs. WORST MS-COCO GENERATIONS USING CLIP SCORE AS METRIC
To qualitatively assess model's performance, this section presents the 500 highest and 500 lowest scoring generations based CLIP Score, using prompts from the MS-COCO captions dataset. Fig. 13 shows the best generations that achieved CLIP scores between 45.5 and 53.9 (mean: 47.2). Fig. 14 shows the worst, ranging from 23.0 to 32.2 (mean: 30.6). These examples illustrate the range of output quality, from strong semantic alignment to notable failure cases.
Figure 13: Best generations on MS-COCO prompts CLIP Score ranging from 45.5 to 53.9 (mean: 47.2)
25

---

Published as a conference paper at ICLR 2026
Figure 14: Worst generations on MS-COCO prompts. CLIP Score ranging from 23.0 to 32,2 (mean: 30.6)
A.11 LIMITATIONS
Our method's quality is inherently tied to the representations encoded in CLIP: performance may degrade for prompts that abstract, rare in CLIP's training data, require fine compositional detail (e.g., "a woman wearing planet-shaped earrings"). These challenges are intrinsic to classifier- inversion approaches, and plan to address them more systematically in future work.
26

---

Published as a conference paper at ICLR 2026
A.12 LLM USAGE
Large language models were used exclusively for text polishing and minor exposition refinements. All substantive research content, methodology, and scientific conclusions were developed entirely by the authors.
27


Reviews:
[{"review_id": "qfnfwJildY", "reviewer": "Reviewer_JCC3", "summary": "In this paper, the author introduces CLIP−1, a novel approach for text-to-image synthesis by inverting CLIP embeddings without relying on a pretrained generative decoder or fine-tuning CLIP. The method leverages Implicit Neural Representations (INRs) optimized layer-by-layer in a frequency-aware manner, enabling coarse-to-fine image generation. Overall, this inversion method sheds light on how encoder models can potentially be used as image generations and also on the inner workings of CLIP. However, I believe that there's still a way to go for such methods to be used as pure image generators.", "questions": "Overall, the paper makes a significant contribution by repurposing a frozen discriminative model for generative tasks, showcasing its potential for image synthesis. However, limitations in handling complex prompts which suggest areas for improvement and lack of comparisons to pure image generators draw me towards a reject. The work however opens promising directions for leveraging pretrained models in generative applications. \n\nI urge the authors to answer questions about the practical utility of inversions with encoder models? How do they see such methods replacing pure image generators or how can such methods be used to improve the encoder capabilities itself. I believe these are unanswered questions.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a decoder-free and tuning-free approach for text-to-image synthesis, which is computationally efficient. \n\n- The method supports zero-shot generalization to multiple tasks, including reconstruction, style transfer, and controlled edits, within a unified framework\n\n- Extensive experiments on MS-COCO and Flickr30k demonstrate strong performance and also ablations showing improvements over other inversion methods.", "weaknesses": "- While the generated images are semantically aligned, fine-grained spatial details and textures occasionally exhibit distortions or artifacts. I understand that the authors are not competing with SoTA image generation methods, but a discussion on it would be fruitful. \n\n- I am a bit apprehensive on the practical applications of the method beyond just understanding the CLIP space. How far is the performance (quantitatively) from pure image generators?\n\n- While FID, IS, and CLIP-SIM are reported, additional perceptual metrics or user studies could strengthen the evaluation of visual quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRYaprhUrL", "reviewer": "Reviewer_YbCx", "summary": "This paper propose a new method of inversing CLIP encoder to achieve text-to-image geneartion using $\\textnormal{CLIP}^{-1}$.\nThis is achieved thorugh a training-free approach, leveraging INR to achieve image embeddings similar to target text embeddings.", "questions": "How do the authors think about the scalability of this approach, given the per-image optimization loop?\n\nAlso, would this method extend to higher-resolution outputs?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The technical contributions of this work are interesting. The authors propose techniques to: (1) enable a training-free approach that generates images without training any decoder, (2) mitigate the CLIP modality gap, and (3) achieve a robust, generalizable framework by leveraging AWP.\n\nThe comparison to existing work is solid, and the reported FID and other metrics are reasonable for this class of technique.", "weaknesses": "The primary concern with this work is its relevance and timeliness. The idea of inverting CLIP for image generation was novel and exciting a few years ago; however, the community has since shifted toward more scalable and expressive generative models.\n\nMoreover, the outputs here are not yet high-quality enough, which is a significant shortcoming for practical use. Therefore, the work’s practical impact seems limited to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYAgqoNr05", "reviewer": "Reviewer_kSFm", "summary": "This paper investigates image synthesis through CLIP inversion. Rather than optimizing directly in the pixel space, they leverage implicit neural representations (INRs) to perform the inversion. To stabilize the inverse mapping, they incorporate techniques such as adversarial training, orthogonal Procrustes projection, and a blending loss.", "questions": "-", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a well-rounded review of related works across multiple relevant areas, helping readers grasp the broader context and motivation behind the proposed approach.\n- It integrates several complementary techniques to tackle distinct challenges—for instance, using Adversarial Weight Perturbation (AWP) for robust INR initialization and incorporating natural image priors through blending.\n- Unlike prior CLIP inversion methods that rely on direct pixel-space optimization and often produce noticeable artifacts, the proposed approach demonstrates improved visual quality in its qualitative results.\n- The paper includes comprehensive ablation studies, analyzing the contribution of each introduced component both quantitatively and qualitatively.", "weaknesses": "- There are some minor issues in how certain points are presented. For instance, in the related work section, the statement “Modern image generators fall into three…” overlooks autoregressive models, which are also widely used and should be acknowledged.\n- From a practical perspective, while the approach is conceptually interesting, it’s not particularly applicable in real-world scenarios. As shown in the results, the generated images still lag far behind other methods. However, the work remains valuable from an interpretability and analytical standpoint.\n- The visualization and presentation could be improved. Specifically, Figures 5 and Tables 2 and 3 are rendered too small, which hinders readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a9oLeO7n9U", "reviewer": "Reviewer_fJkf", "summary": "This paper introduces $CLIP^{-1}$, a novel framework for text-to-image synthesis that demonstrates the latent generative capabilities of a frozen, pre-trained discriminative model (CLIP) . The central claim is that a dedicated, trained generative decoder is not strictly necessary. Instead, the authors reframe image synthesis as an inversion problem: finding an image that, when encoded by CLIP's image encoder, matches a target text embedding .The paper's core contribution is a set of techniques to solve this notoriously ill-posed problem, which typically yields unstable, non-naturalistic images (adversarial examples) . The authors' solution avoids direct pixel optimization and instead optimizes the weights of a frequency-aware Implicit Neural Representation (INR), which inherently enforces smoothness and a structured, coarse-to-fine generation process .To further stabilize this inversion, the $CLIP^{-1}$ pipeline introduces three key components:An adversarially robust initialization using Adversarial Weight Perturbation (AWP) on a blurred seed image, which anchors the optimization in a stable \"flat\" region of the loss landscape .A lightweight, geometric solution to the modality gap using Orthogonal Procrustes Analysis to project the target text embedding onto the image embedding manifold, providing a more stable target .A blending loss that acts as a natural image prior, regularizing the output by encouraging it to remain close to the statistics of real image embeddings .Empirically, $CLIP^{-1}$ substantially outperforms other training-free inversion methods like DAS, achieving a Fréchet Inception Distance (FID) that is less than half of its competitor . The framework also demonstrates zero-shot versatility on downstream tasks, including image reconstruction, controlled text-guided editing, and neural style transfer.", "questions": "Following up on the main weakness: \n1) Could the authors provide a direct comparison of the inference time (time per image on a standard GPU like an A100 or RTX 4090) and peak VRAM usage for $CLIP^{-1}$ against the SOTA generative baselines listed in Table 1 (e.g., LDM-KL-8 or GLIDE)? This would be invaluable for understanding the practical trade-off between the method's training cost and its inference cost.\n2) The optimization process is set to 400 steps. Have the authors experimented with this hyperparameter? How does image quality (FID) degrade as the number of steps is reduced? Is there a \"sweet spot\" that balances quality and speed? \n3) This question is about the robustness of the initialization strategy. The paper's AWP-trained seed image is described as a \"robust anchor\". We are interested in how strongly this anchor constrains the final generation.\n- First, the paper shows a \"Plain $CLIP^{-1}$\" variant that starts from random INR weights. Could the authors provide a qualitative comparison for this variant? Does it simply converge slower, or does it suffer from more significant artifacts, confirming the necessity of the AWP anchor?\n- Second, what happens in an adversarial scenario where the retrieved seed image is semantically incorrect (e.g., the prompt is \"a photo of a dog\" but the retrieved seed is \"a photo of a car\")? Is the optimization pipeline, guided by the Procrustes-corrected text embedding and the blending loss, strong enough to correct this completely wrong starting point and still generate a \"dog\"? Or does the robust anchor effectively trap the optimization near the (incorrect) seed, leading to a failed generation?\n4) The use of Orthogonal Procrustes is a very clever and simple solution for the modality gap. Did the authors experiment with any alternative methods, such as learning a small, lightweight mapping network (e.g., a simple MLP) to bridge the gap? Was Procrustes chosen primarily for its simplicity and training-free nature, or did it also empirically outperform other simple alternatives?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) The paper's core idea, unlocking the generative potential hidden within a frozen discriminative model, is highly original and significant. It moves beyond simply using CLIP as an encoder and shows it can be \"run in reverse\" as a generator, challenging the conventional wisdom that separates discriminative and generative models.\n\n2) The key innovation is reframing the inversion problem from pixel space to the weight space of an Implicit Neural Representation (INR). This choice enforces smoothness, helps in preventing high-frequency artifacts and enables a structured coarse-to-fine optimization by leveraging the frequency-stratified architecture of the FINER network.\n\n3) The paper introduces simple, effective, and computationally cheap solutions to complex problems. I like the use of Orthogonal Procrustes Analysis to geometrically correct the modality gap, as it requires no training and is demonstrably effective.\n\n4) The combination of Adversarial Weight Perturbation (AWP) for initialization and the blending loss creates a more stable optimization process. These components successfully anchor the search and prevent it from converging to the \"adversarial examples\" (perceptually meaningless noise) that have plagued previous inversion attempts. \n5) The ablation study is thorough and provides a clear justification for each component of the $CLIP^{-1}$ pipeline.", "weaknesses": "1) The paper's primary weakness is its discussion of the computational cost. The method is \"training-free\", but it shifts the entire computational load to inference time, which requires an iterative optimization process (400 steps). The runtime analysis in the appendix compares $CLIP^{-1}$ only to DAS, another inversion-based method. The main quantitative comparison compares quality (FID/IS) against SOTA generative models like LDM and GLIDE but critically omits a comparison of inference time and VRAM usage. A single forward pass through a diffusion decoder is likely much faster than the 1+ minute optimization required by $CLIP^{-1}$. To fairly evaluate this as a generative \"decoder\" alternative, this trade-off must be made explicit. \n2) As the authors rightly acknowledge, a significant gap in photorealism and fine-detail fidelity remains when compared to SOTA diffusion models. The generated images, while coherent, often have a \"painterly\" or slightly artificial quality and lack crisp, high-frequency textures. This limits the method's immediate practical application for photorealistic generation.Initialization.\n3) The full, best-performing model relies on retrieving a semantically relevant seed image from the LAION dataset to perform its AWP-based initialization. While the out-of-distribution evaluation shows that a \"Plain $CLIP^{-1}$\" variant (starting from random weights) can still function, the main method's output is still influenced by this seed. This could constrain the creative novelty of the generations, potentially biasing them towards common compositions found in the retrieval database.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)