Implicit Inversion turns CLIP into a Decoder

## Abstract

Abstract CLIP is a discriminative model trained to align images and text in a shared embedding space. Due to its multimodal structure, it serves as the backbone of many generative pipelines, where a decoder is trained to map from the shared space back to images. In this work, we show that image synthesis is nevertheless possible using CLIP alone—without any decoder, training, or fine-tuning. Our approach optimizes a frequency-aware implicit neural representation that encourages coarse-to-fine generation by stratifying frequencies across network layers. To stabilize this inverse mapping, we introduce adversarially robust initialization, a lightweight Orthogonal Procrustes projection to align local text and image embeddings, and a blending loss that anchors outputs to natural image statistics. Without altering CLIP’s weights, this framework unlocks capabilities such as text-to-image generation, style transfer, and image reconstruction. These findings suggest that discriminative models may hold untapped generative potential, hidden in plain sight.

## 1 Introduction

Text-to-image generation has progressed from a challenging research problem to a widely accessible technology, with recent models achieving photorealistic results and demonstrating remarkable creative abilities  [1, 29, 24, 3].
Among these, latent diffusion models [28] currently define the state-of-the-art by relying on an encoder–decoder architecture, where the encoder maps the input text into a latent representation, and the decoder reconstructs an image from it. Foundational vision-language models like CLIP [26] are frequently utilized as text encoders [1, 37, 35], whereas the decoder is typically a diffusion model – by far the most computationally demanding stage of the pipeline.

In this paper, *we show that CLIP alone can perform text-to-image generation, without requiring a dedicated decoder.*
We achieve this by inverting the CLIP vision encoder: rather than mapping an image to its latent embedding, we start from a CLIP embedding and reconstruct the corresponding image. Although prior work has attempted this through direct pixel space optimization [14, 8, 9], such approaches produce low-quality output with visible artifacts [14] or require additional training [8, 9].
In contrast, we introduce CLIP^-1, a *decoder-free and training-free* solution based on Implicit Neural Representations (INRs) [19]. [Fig. 1](https://arxiv.org/html/2505.23161v2#S0.F1) offers our results on generic scenes prompts and longer, more complex prompts from MS-COCO [18].
In practice, we first retrieve a seed, low-frequency INR whose caption is most similar to the prompt. Because this INR was pre-trained on a blurred version of its image with Adversarial Weight Perturbations (AWP) [38], its low-frequency weights are stable and provide a robust anchor.
We then refine the INR layer by layer: a peaked learning-rate schedule moves from low- to high-frequency layers, progressively adding details in a coarse-to-fine manner. During refinement, we encourage on-manifold solutions by employing simple cosine losses against a style prompt and a small set of retrieved natural images, eventually resulting in more aesthetic generations.
Avoiding direct pixel-space optimization, our approach naturally avoids structural artifacts and elegantly results in a coarse-to-fine generation akin to that of diffusion models.

Despite the stability of the INR backbone, inversion still suffers from CLIP’s modality gap [17, 23, 40]: text and image embeddings fall on slightly offset sub-manifolds, making the raw prompt embedding an unreliable target.
We bridge this gap by seeking an optimal orthogonal transformation via Procrustes analysis [36, 21], using the $k$ nearest caption–image pairs to align the prompt embedding with the image manifold and produce a well-conditioned target for optimization.

We benchmark our decoder-free, training-free pipeline on 10k MS-COCO [18] captions, reporting Fréchet Inception Distance (FID) [12], Inception Score (IS) [30], and CLIPSIM metric [11]. Compared with DAS [6], a concurrent decoder-free and training-free approach, our method achieves half the FID while almost doubling IS, producing noticeably crisper, more faithful images. The very same frozen model transfers zero-shot to image reconstruction, controlled image edits, and neural style transfer, confirming the versatility of the INR backbone. Ablations show that (i) adversarially robust INRs boost quality, (ii) Procrustes alignment yields sharper, more semantically aligned images, and (iii) frequency-steered optimization suppresses high-frequency artifacts.

In summary, our contribution is 3-fold.

- 1.
Decoder–free, training–free CLIP inversion.
We repurpose a *frozen* CLIP as a text-to-image generator by optimizing a frequency-aware Implicit Neural Representation instead of generating images directly. Our pipeline (i) retrieves an adversarial-robust, blur-initialized INR that anchors low-frequency content, and (ii) refines its weights with a coarse-to-fine, layer-wise schedule guided solely by CLIP losses—eliminating the need for external decoders or re-training.
- 2.
Modality-gap reduction with orthogonal Procrustes.
We bridge the mismatch between CLIP’s text and image sub-manifolds with a lightweight orthogonal transformation on the $k$ nearest caption–image pairs, projecting the prompt embedding into the image manifold to stabilize inversion.
- 3.
Extensive empirical validation.
We extensively evaluate our pipeline, outperforming comparable prior work while demonstrating zero-shot transfer to image reconstruction, controlled edits, and neural style transfer, with ablations confirming the benefit of each component.

We publicly release all code and models to foster future research(^1^11Code is available at [github.com/omnai-lab/implicit-inversion](https://github.com/omnai-lab/implicit-inversion).).

## 2 Method

Figure: Figure 2: CLIP^-1 text-to-image inversion pipeline. The image is initially represented using an Implicit Neural Representation (INR) $f_{\phi}(i,j)$, optimizing the INR weights to match an input text prompt. The starting point for the inversion is a robust INR trained with Adversarial Weight Perturbation (AWP). The optimization updates the INR layer-wise, so that the embedding of its rendering aligns with the input text prompt embedding. The procedure includes image augmentations and CLIP embeddings averaging and projection onto the unit embedding sphere. In order to align text and image embeddings, we also apply an orthogonal Procrustes transformation to address the modality gap present in CLIP.
Refer to caption: x2.png

Given an input text prompt, our approach aims to generate images by inverting the corresponding CLIP embeddings.
The pipeline consists of three main stages:
(i) a *data preparation* step (performed once offline),
(ii) an *initialization* step leveraging preprocessed data to retrieve a suitable starting point,
(iii) an *optimization* procedure refining the initial image to match the target text.

### 2.1 Preliminaries

Implicit Neural Representations.
INRs represent images as functions that map spatial coordinates $(i,j)$ to RGB values $f_{\phi}(i,j)=(\text{r},\text{g},\text{b})$, where $\phi$ are the parameters of a neural network. The architecture is typically a Multilayer Perceptron (MLP), using positional encoding or frequency-aware activations for compact and efficient representation, crucial for image synthesis via iterative optimization. We adopt FINER [19], which enhances fine-detail modeling using variable periodic activations. It is based on SIREN [33], which uses the fixed-frequency activation function $\mathbf{z}_{i}=\sin\left(\omega(\mathbf{W}_{i}\mathbf{z}_{i-1}+\mathbf{b}_{i})\right)$. FINER improves adaptability by introducing an additional coefficient:

$$ $\displaystyle\begin{gathered}\mathbf{z}_{i}=\sin\big{(}\omega\alpha_{i}( \mathbf{W}_{i}\mathbf{z}_{i-1}+\mathbf{b}_{i})\big{)},\quad\alpha_{i}=\left| \mathbf{W}_{i}\mathbf{z}_{i-1}+\mathbf{b}_{i}\right|+1\,,\end{gathered}$ (2) $$

where $\alpha_{i}$ dynamically adjusts local frequency based on input magnitude. We also leverage FINER’s bias initialization, which stratifies frequencies across layers, capturing low frequencies early and high frequencies deeper, leading to better convergence and reconstruction. See [Fig. 6](https://arxiv.org/html/2505.23161v2#S3.F6) for an illustration.

### 2.2 Data preparation

The data preparation stage is performed offline once and sets up the foundation for inversion initialization. In practice, we used images from LAION Aesthetics (a subset of LAION-5B [32]). This data serves two purposes: (i) training INRs for initializing text-to-image optimization and (ii) computing natural image CLIP embeddings to act as anchor points in the latent space.

Each image is first blurred using a Gaussian filter to suppress high frequencies, providing a smoother starting point for inversion. An INR is then trained to reconstruct the blurred image, and its weights are stored.
For the $i$-th image, the predicted pixel values from the INR are $f_{\phi^{i}}$, with weights $\phi^{i}$. The INR images are then encoded via CLIP to obtain visual embeddings $\boldsymbol{\theta}_{I}\big{(}f_{\phi^{i}}\big{)}$. The CLIP text embeddings $\boldsymbol{\theta}_{T}\left(\mathbf{y}^{i}\right)$ of the corresponding captions are also stored. This results in a dataset $\mathcal{D}=\{\boldsymbol{\theta}_{I}\big{(}f_{\phi^{i}}\big{)},f_{\phi^{i}},
\boldsymbol{\theta}_{T}\left(\mathbf{y}^{i}\right)\}$, containing CLIP image embeddings, INRs, and CLIP text embeddings for each training sample.

Robust INR initialization.
The INR weights $\phi^{i}$ can be viewed as compact embeddings of the $i$-th image – each uniquely capturing its content. However, small perturbations in these weights can significantly alter the reconstructed image, making them sensitive and potentially unstable for downstream tasks.

To address this, we propose a training method that improves INR weights robustness by incorporating adversarial perturbations during training. Given an INR with weights $\phi$, we define an adversarial weight perturbation (AWP) $\Delta\phi\in\Omega$, where $\Omega$ bounds the perturbation range. The training objective is to make the model resilient to such perturbations by solving the following min-max optimization:

$$ $\min_{\phi}\max_{\Delta{\phi}\in\Omega}\mathcal{L}\left(f_{{\phi}+\Delta{\phi} },\text{blur}(\mathbf{x})\right)\,,$ (3) $$

where $f_{{\phi}+\Delta{\phi}}$ is the perturbed INR output and $\mathcal{L}$ is the reconstruction loss function w.r.t. the blurred target image $\mathbf{x}$.
Unlike standard adversarial training in the input space [38], our approach perturbs only the model weights. The perturbation set is constrained by a relative norm bound $\Omega=\{\Delta:\big{\|}\Delta\big{\|}\leq\gamma\big{\|}{\phi}\big{\|}\}$, with $\gamma$ controlling the allowed perturbation magnitude.
This robust training is applied only once, during the construction of the initial INR at the inversion step $n=0$. It ensures that early optimization steps do not cause the weights to drift too far from the frequency content of the initialization, improving stability during inversion. The supplementary material details the training procedure using the AWP algorithm.

### 2.3 Initialization and modality gap handling

Inversion initialization.
In text-to-image generation, the process begins with a text prompt $\mathbf{y}$, which is encoded using the CLIP text encoder to obtain $\mathbf{e}_{t}=\boldsymbol{\theta}_{T}\left(\mathbf{y}\right)\in\mathbb{R}^{d}$. To initialize the inversion, we search our dataset $\mathcal{D}$ for the INR whose associated caption has the highest cosine similarity to $\mathbf{e}_{t}$. This serves as the starting point for the optimization.

Bridging the modality gap.
CLIP aligns text and image embeddings globally, projecting them onto the unit sphere. However, local differences between modalities persist: text embeddings tend to encode abstract semantics, while image embeddings reflect concrete visual features.
Directly optimizing an image to match a text embedding often causes artifacts or textual hallucinations – the model overfits to abstract concepts and produces unrealistic visuals [16].

To address this, we learn a local transformation to align text and image embeddings more precisely. We retrieve the $k$ nearest neighbors of the input text embedding $\mathbf{e}_{t}$ from our dataset $\mathcal{D}$, forming two matrices: $\mathbf{E}_{T}\in\mathbb{R}^{d\times k}$, encoding the $k$ closest text embeddings, and $\mathbf{E}_{I}\in\mathbb{R}^{d\times k}$ encoding their corresponding image embeddings.
Solving the orthogonal Procrustes problem we find an orthogonal matrix $\mathbf{R}$ that best aligns these two sets:

$$ $\displaystyle\begin{gathered}\min_{\mathbf{R}}||\mathbf{R}\mathbf{E}_{T}- \mathbf{E}_{I}||_{F}\qquad\text{s.t.}\ \mathbf{R}^{\top}\mathbf{R}=\mathbf{I} \,,\end{gathered}$ (5) $$

where $\|\cdot\|_{F}$ is the Frobenius norm. This procedure is performed individually for each input prompt. The resulting orthogonal matrix $\mathbf{R}$ aligns the local structure of text embeddings with that of image embeddings.
We then transform the input text embedding into the image modality as $\mathbf{e}_{t2i}=\mathbf{R}\mathbf{e}_{t}$, which becomes the target embedding for the CLIP inversion process.

### 2.4 Inverting CLIP with Implicit Neural Representations

Text-To-Image via CLIP Inversion.
Our pipeline, shown in [Fig. 2](https://arxiv.org/html/2505.23161v2#S2.F2), inverts a CLIP text embedding to synthesize an image using an INR. Given a text prompt $\mathbf{y}$, we obtain the projected image-space embedding $\mathbf{e}_{t2i}=\mathbf{R}\boldsymbol{\theta}_{T}\left(\mathbf{y}\right)$. The INR $f_{\phi}$ is optimized so that its output image matches $\mathbf{e}_{t2i}$ when passed through the frozen CLIP image encoder $\boldsymbol{\theta}_{I}$. This is formalized as:

$$ $\phi=\arg\min_{\phi}\mathcal{L}(\boldsymbol{\theta}_{I}\big{(}f_{\phi}\big{)}, \mathbf{e}_{t2i})\quad\text{where}\quad\mathbf{e}_{t2i}=\mathbf{R}\boldsymbol{ \theta}_{T}\left(\mathbf{y}\right).$ (6) $$

Here, $\mathcal{L}$ is the cosine distance, and gradients flow from CLIP back to the INR parameters $\phi$.

Layer-wise frequency optimization.
Rather than optimizing pixel values directly, we update the INR weights, leveraging FINER’s property that network layers correspond to different frequency bands. The INR is structured as an $L$-layer MLP, with each layer representing a specific frequency range.
To guide the optimization process, we apply Gaussian learning rate scheduling: at each iteration, we focus the optimization on a specific layer by assigning it a peak learning rate, while attenuating the rates of neighboring layers according to a Gaussian curve (see [Fig. 3](https://arxiv.org/html/2505.23161v2#S2.F3)). This helps reconstruct coarse features before fine details, improving stability and fidelity.

Figure: Figure 3: Gaussian Scheduling. Each layer represents a frequency interval (b) for $f_{\phi}$. The learning rate is centered on a specific layer and gradually shifted (c), decreasing with a Gaussian attenuation across neighboring layers (a).
Refer to caption: x3.png

Augmentations for stable optimization.
Inspired by CLIPDraw [7] and CLIPAG [8], we apply image augmentations such as color shifts, scaling, and shearing, while optimizing. Each augmented image is encoded with CLIP, and their embeddings are averaged and projected back onto the unit sphere: $\mathbf{e}_{i}^{\star}=\frac{1}{n}\sum_{k=1}^{n}\boldsymbol{\theta}_{I}\big{(}
\text{augment}(f_{\phi^{k}})\big{)}$, strongly enforcing synthesized image robustness to visual distortions.

Blending natural image priors.
To further guide generation toward realistic outputs, we incorporate information from natural images. For a given prompt $\mathbf{y}$, we retrieve the $k$ most similar image embeddings from a reference dataset using cosine similarity. These are linearly combined into a blended target embedding $\mathbf{e}^{\star}_{img}$, with weights given by the softmax of the similarity scores. A blending loss $\mathcal{L}_{\text{blend}}$ then encourages the output embedding $\mathbf{e}_{i}^{\star}$ to remain close to the manifold.

Final optimization formulation.
The complete CLIP^-1 optimization pipeline updates the INR parameters $\phi$ to generate realistic images, leveraging both augmented embeddings and natural image priors, with no CLIP retraining or modification. The full procedure is:

$$ $\displaystyle\begin{split}\begin{cases}\begin{array}[]{l}\vspace{6pt}\text{(a) }\quad\mathbf{e}_{i}^{\star}=\frac{1}{n}\sum_{k=1}^{n}\boldsymbol{\theta}_{I} \big{(}\text{augment}(f_{\phi^{k}})\big{)}\\ \text{(b)}\quad\phi_{0}=\min_{\phi}\max_{\Delta{\phi}\in\Omega}\mathcal{L}\big {(}f_{{\phi}+\Delta{\phi}},\text{blur}(\mathbf{x})\big{)}\\ \text{(c)}\quad\phi_{n}=\phi_{n-1}-\nabla_{\phi}\Big{[}\mathcal{L}(\mathbf{e}_ {i}^{\star},\mathbf{e}_{t2i})+\beta\mathcal{L}_{blend}\left(\mathbf{e}_{i}^{ \star},\mathbf{e}^{\star}_{img}\right)\Big{]}\end{array}\end{cases}\end{split}$ (7) $$

Step (a) computes the CLIP embedding from the augmented INR outputs; step (b) initializes the INR with adversarial weight perturbations to enhance robustness, and (c) updates the INR weights via backpropagation using both alignment and blending losses.

## 3 Experiments

We evaluate our decoder-free, training-free inversion pipeline across a range of tasks. We begin with text-to-image generation on MS-COCO [18], comparing against both standard generative models and prior inversion-based approaches (§[3.1](https://arxiv.org/html/2505.23161v2#S3.SS1)). Next, we demonstrate the generality of our method through zero-shot transfer to downstream tasks, including reconstruction, controlled modification, and style transfer (§[3.2](https://arxiv.org/html/2505.23161v2#S3.SS2)). Lastly, we quantify the effect of each component through an ablation study (§[3.3](https://arxiv.org/html/2505.23161v2#S3.SS3)).

**Table 1: MS-COCO text-to-image generation results. FID (lower is better), CLIPSIM and IS (both higher are better), along with model sizes.**
|  | [1] | [24] | [28] | [8] | [9] | [9] | [14] | [6] | [6] |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | DALL-E | GLIDE | LDM-KL-8 | CLIPAG | EB-CLIP | EB-CLIP | CLIP-Inv | DAS | DAS | CLIP^-1 |
|  |  |  |  | ViT | ViT | XXL | ViT | ViT | Ensemble |  |
| Decoder-free | × | × | × | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Tuning-free | × | × | × | × | × | × | ✓ | ✓ | ✓ | ✓ |
| # Train params (M) | 12000 | 6000 | 1450 | 88 | 88 | 1200 | 0 | 0 | 0 | 0 |
| # Tot params (M) | 12000 | 6000 | 1450 | 150 | 150 | 846 | 150 | 150 | 3x150 | 150 |
| \cdashline1-12<br>FID$(\downarrow)$ | 27.5 | 12.2 | 23.3 | 42.3 | 68.3 | 23.4 | 140.1 | 161.8 | 121.6 | 72.5 |
| CLIPSIM$(\uparrow)$ | – | – | – | 34.7 | 34.5 | 33.5 | 61.4 | 22.7 | 36.9 | 38.6 |
| IS$(\uparrow)$ | 17.9 | – | 20.03 | 18.7 | – | – | 4.8 | 5.7 | 8.36 | 9.5 |

### 3.1 Generator-free text-to-image synthesis

Setting.
In text-to-image synthesis, the goal is to generate visually realistic images that are semantically aligned with a natural language description. To assess the visual fidelity of our method, we compute the Fréchet Inception Distance (FID) [12] and the Inception Score (IS) [30] over a subset of $10,000$ captions from MS-COCO [18]. Since these metrics do not capture semantic alignment with the prompt, we also report CLIPSIM [11], which measures the cosine similarity between the CLIP embeddings of generated images and their corresponding captions.
We compare our method against prior and concurrent decoder-free approaches, including those that require fine-tuning [8, 10] and those that do not [6, 14]. As our method requires no training or decoder, we regard tuning-free baselines as the most relevant points of comparison. For completeness, we also report results from state-of-the-art generative models that rely on both training and a dedicated decoder [1, 24, 28]. The implementation details can be found in the supplementary material.

Results.
[Section 3](https://arxiv.org/html/2505.23161v2#S3) reports our results alongside model sizes and architectural requirements. Among decoder-free, training-free methods, CLIP^-1 achieves the lowest FID (72.5 vs. 161.8 for DAS-ViT [6]) and highest IS (9.5 vs. 5.7), marking a substantial improvement in visual quality. Although diffusion-based models still attain lower FID, they require orders of magnitude more parameters and full training pipelines, whereas our method uses a frozen backbone and a lightweight INR.
Finally, our method achieves a CLIPSIM of 38.6, outperforming both training-free and fine-tuned baselines—except for CLIPInvert [14], whose higher CLIPSIM can be attributed to overfitting to the target embedding, as evidenced by its elevated FID and low IS. Taken together, these results indicate that Procrustes alignment and frequency-aware INR optimization effectively improve text-image consistency without modifying CLIP’s weights.
[Figure 4](https://arxiv.org/html/2505.23161v2#S3.F4) further provides qualitative examples: compared to other training-free baselines, our generations exhibit fewer structural artifacts and sharper details, and visually approach the quality of tuned approaches [8, 9].
We also apply our pipeline in a plug-and-play fashion to tuned CLIP variants, such as CLIPAG [8] and CLIP-JEM [9], showing broader compatibility of our method with discriminatively trained models.

Figure: (a) Reconstruction
Refer to caption: extracted/6511808/results/dstream_task/reconstruction/henry_cropped.jpg

### 3.2 Zero-shot task generalization

To assess the versatility of our decoder-free approach, we explore several downstream tasks, demonstrating that the same inversion framework can successfully generate images in different settings without requiring task-specific modifications or additional optimization.

Image reconstruction.
In this task, the goal is to reconstruct a given input image using our inversion pipeline. We treat the image as a target whose CLIP embedding is known, and optimize an INR to produce an output that matches this embedding. The INR is initialized from a blurred version of the image, and refined to align with the embedding of the full-resolution input, effectively operating as a decoder that recovers semantic content from latent space.
Unlike text-to-image generation, this task provides a well-defined ground truth and serves as a controlled setting to evaluate inversion precision. [Figure 4(a)](https://arxiv.org/html/2505.23161v2#S3.F4.sf1) shows qualitative results on both artistic and photographic inputs. Across all examples, high-level semantic content, such as facial identity or scene composition, is consistently preserved. Fine-grained spatial details, especially in structured regions like faces or buildings, are approximated with some distortion or shift, reflecting the inherent ambiguity of CLIP’s embedding space.

Controlled image modification.
In this task, the goal is to modify an input image according to a natural language prompt that specifies a targeted change in content or style. The image is first encoded via an INR fitted to its original form. A text prompt is then provided to guide the modification (e.g., “snowy landscape” or “torrential rainfall”).
The INR is optimized to align the CLIP embedding of the generated image with that of the prompt, while starting from the original image representation. This setup encourages localized, semantically consistent transformations without disrupting the broader structure or identity of the scene.
[Figure 4(b)](https://arxiv.org/html/2505.23161v2#S3.F4.sf2) shows three examples for the task. In each row, the left-most column is the original image; the next two columns show the edits for “snow” and “storm”. The road, the Great Wall, and the city keep their geometry and colour palette, while only the requested weather effects (snow cover, rain streaks, lightning) are added. This confirms that CLIP^-1 can act as a prompt-driven image editor, producing targeted edits without explicit masks or additional training.

Neural style transfer.
For style transfer we supply two images: a *content* photo and a *style* reference. The content image is represented by an INR initialized to exactly reproduce the original photo; the style image is fed only through the frozen CLIP encoder. Optimization minimizes a weighted sum of two CLIP-based losses:
(i) a style loss that pulls the INR’s embedding towards that of the reference painting, and
(ii) a content loss (weight 0.5) that keeps the embedding close to the original photo.
[Figure 4(c)](https://arxiv.org/html/2505.23161v2#S3.F4.sf3) illustrates the outcome. In each case, the brush-stroke texture and overall palette of the reference painting are transferred, while object layout and scene geometry remain intact. The method therefore separates appearance from semantics without hand-crafted losses or additional training, indicating that the inversion pipeline can exploit CLIP’s latent space to disentangle style from content.

### 3.3 Ablation study

Figure: (a)
Refer to caption: extracted/6511808/fig_ablation/table2_figs/24.png

## 4 Related work

Image generative models.
Modern image generators fall into three broad families: GANs, diffusion models, and normalizing flows.
GANs train a generator to fool a discriminator with realistic samples, a strategy refined from early DCGANs [25] to StyleGAN-T [31] and BigGAN [2]. Diffusion models begin with pure noise and iteratively denoise it back to an image; latent diffusion [28], GLIDE [24], DALL-E 2
[27] and DALL-E 3 [1] differ mainly in how they compress the signal and guide it with text.
Normalizing flows (e.g., [15, 5]) learn a chain of invertible transforms so that sampling the base Gaussian and running the reverse pass yields data in a few steps.
Despite these distinct mechanics, every approach relies on a latent-to-image decoder: the generator in GANs, the denoising network in diffusion, or the reverse flow in normalizing-flow models. Our work removes this requirement altogether by directly inverting a frozen discriminative encoder (CLIP).

Model inversion.
Several recent works attempt to repurpose CLIP for image generation by inverting its embeddings. The earliest approach [14] directly optimises randomly initialised pixels to minimise the cosine distance between the CLIP embedding of the image and that of a target text prompt. GALIP [35] introduces a CLIP-conditioned GAN framework, training both a generator and discriminator to enable fast, controllable synthesis with fewer parameters and less data than large-scale diffusion models. CLIPAG [8] follows a decoder-free design but applies adversarial fine-tuning to CLIP itself to improve generation quality. This idea is further extended in EB-CLIP [9], where the generation is formulated as energy minimisation in CLIP’s joint image-text space. Concurrently with our work, DAS [6] shows that a frozen CLIP can be inverted without training by optimising at multiple spatial resolutions. Like us, DAS reveals generative priors within discriminative models, but differs by operating directly in pixel space rather than through a frequency-aware implicit representation.

Adversarial robustness. Adversarial training has become a core strategy for improving model robustness, with [20] introducing the first widely adopted method using input perturbations. TRADES [39] extended this by leveraging KL divergence to balance accuracy and robustness, later refined by [4] to address its asymmetry. Beyond input-space attacks, AWP [38] proposed perturbing model weights during training, improving generalisation by flattening the loss landscape. While primarily used for classification, recent work [22] suggests robust models also encode stronger generative priors.

Image modeling with INRs.
Implicit Neural Representations model images as continuous functions that map spatial coordinates $(i,j)$ to RGB values via a neural network, typically an MLP. To capture fine detail, they rely on frequency-aware components such as positional encodings [34] or periodic activation functions like SIREN [33]. However, fixed-frequency activations limit adaptability, motivating FINER [19], which introduces variable-periodic activations that dynamically adjust to local frequency content. We adopt FINER for its efficient representation, well-suited to our inversion task, and to mitigate the spectral bias that hampers high-fidelity reconstruction in standard INRs.

## 5 Conclusion

This work presents CLIP^-1, a decoder-free and training-free approach that inverts the CLIP image encoder for text-to-image synthesis using implicit neural representations (INRs). Rather than requiring a generative decoder, we show that CLIP, when combined with a frequency-aware INR and a lightweight alignment mechanism, can guide image synthesis directly from text prompts.
While our results are not intended to compete with state-of-the-art generative models in absolute quality metrics, they highlight an underexplored and surprising capability: a frozen discriminative model like CLIP can be coaxed into producing coherent, semantically aligned images with no additional training or generative backbone.
Moreover, the same framework supports zero-shot applications such as image reconstruction, controlled edits, and neural style transfer–all within a single unified setup. These findings open new directions for repurposing pretrained models for generation and suggest broader implications for model interpretability and robustness.
That said, the current approach relies entirely on the INR and associated losses to stay close to the natural image manifold; no explicit mechanism enforces this constraint. Incorporating a projection step back onto the manifold could improve fidelity and regularisation, and presents a promising direction for future work–especially toward understanding and interpreting the structure of CLIP’s embedding space.

Acknowledgment. This work was supported by projects PNRR MUR PE0000013-FAIR under the MUR National Recovery and Resilience Plan funded by the European Union - NextGenerationEU and PRIN 2022 project 20227YET9B “AdVVent” CUP code B53D23012830006. It was also partially supported by Sapienza research projects “Prebunking”, “Adagio”, “Risk and Resilience factors in disadvantaged young people: a multi-method study in ecological and virtual environments”, “BEAT (Better dEep leArning securiTy)” and Seed of ERC grant “MINT.AI”.
Computing was supported by CINECA cluster under project Ge-Di HP10CRPUVC, RDM HP10C7YYL2 and the Sapienza Computer Science Department cluster. The authors would like to thank Andrea Salinetti for his initial work and bachelor’s thesis on CLIP inversion.

## References

- Betker et al. [2023]
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.
Improving image generation with better captions.
*Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf*, 2(3):8, 2023.
- [2]
Andrew Brock, Jeff Donahue, and Karen Simonyan.
Large scale gan training for high fidelity natural image synthesis.
In *International Conference on Learning Representations*.
- Chang et al. [2023]
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T Freeman, Michael Rubinstein, et al.
Muse: Text-to-image generation via masked generative transformers.
In *International Conference on Machine Learning*, pages 4055–4075. PMLR, 2023.
- Cui et al. [2023]
Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang.
Decoupled kullback-leibler divergence loss.
*arXiv preprint arXiv:2305.13948*, 2023.
- Esser et al. [2024]
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *International Conference on Machine Learning*, pages 12606–12633. PMLR, 2024.
- Fort and Whitaker [2025]
Stanislav Fort and Jonathan Whitaker.
Direct ascent synthesis: Revealing hidden generative capabilities in discriminative models.
*arXiv preprint arXiv:2502.07753*, 2025.
- Frans et al. [2022]
Kevin Frans, L. B. Soros, and Olaf Witkowski.
Clipdraw: exploring text-to-drawing synthesis through language-image encoders.
In *Proceedings of the 36th International Conference on Neural Information Processing Systems*, Red Hook, NY, USA, 2022. Curran Associates Inc.
- Ganz and Elad [2023]
Roy Ganz and Michael Elad.
Clipag: Towards generator-free text-to-image generation, 2023.
- Ganz and Elad [2024]
Roy Ganz and Michael Elad.
Text-to-image generation via energy-based clip, 2024.
- Ganz et al. [2023]
Roy Ganz, Bahjat Kawar, and Michael Elad.
Do perceptually aligned gradients imply robustness?
In *ICML*, 2023.
- Hessel et al. [2021]
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
Clipscore: A reference-free evaluation metric for image captioning.
*arXiv preprint arXiv:2104.08718*, 2021.
- Heusel et al. [2017]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
In *NeurIPS*, 2017.
- Jagielski et al. [2023]
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, et al.
Measuring forgetting of memorized training examples.
In *ICLR*, 2023.
- [14]
Hamid Kazemi, Atoosa Chegini, Jonas Geiping, Soheil Feizi, and Tom Goldstein.
What do we learn from inverting clip models?
In *Neurips Safe Generative AI Workshop 2024*.
- Kingma and Dhariwal [2018]
Durk P Kingma and Prafulla Dhariwal.
Glow: Generative flow with invertible 1x1 convolutions.
*Advances in neural information processing systems*, 31, 2018.
- Liang et al. [2022a]
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou.
Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.
In *Advances in Neural Information Processing Systems*, pages 17612–17625. Curran Associates, Inc., 2022a.
- Liang et al. [2022b]
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou.
Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.
*Advances in Neural Information Processing Systems*, 35:17612–17625, 2022b.
- Lin et al. [2014]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In *ECCV*, 2014.
- Liu et al. [2024]
Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao.
Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024.
- Madry et al. [2018]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks.
In *ICLR*, 2018.
- Maiorca et al. [2023]
Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà.
Latent space translation via semantic alignment.
*Advances in Neural Information Processing Systems*, 36:55394–55414, 2023.
- Mirza et al. [2024]
Mujtaba Hussain Mirza, Maria Rosaria Briglia, Senad Beadini, and Iacopo Masi.
Shedding more light on robust classifiers under the lens of energy-based models.
In *European Conference on Computer Vision*, pages 451–468. Springer, 2024.
- Mistretta et al. [2025]
Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D Bagdanov.
Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion.
*ICLR 2025*, 2025.
- Nichol et al. [2022]
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models.
In *International Conference on Machine Learning*, pages 16784–16804. PMLR, 2022.
- Radford et al. [2015]
Alec Radford, Luke Metz, and Soumith Chintala.
Unsupervised representation learning with deep convolutional generative adversarial networks.
*arXiv preprint arXiv:1511.06434*, 2015.
- Radford et al. [2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pages 8748–8763. PMLR, 2021.
- Ramesh et al. [2022]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
*arXiv preprint arXiv:2204.06125*, 2022.
- Rombach et al. [2022]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10684–10695, 2022.
- Saharia et al. [2022]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.
Photorealistic text-to-image diffusion models with deep language understanding.
*Advances in neural information processing systems*, 35:36479–36494, 2022.
- Salimans et al. [2016]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.
Improved techniques for training gans.
*Advances in neural information processing systems*, 29, 2016.
- Sauer et al. [2023]
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.
Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis.
In *International conference on machine learning*, pages 30105–30118. PMLR, 2023.
- Schuhmann et al. [2022]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation image-text models.
*Advances in neural information processing systems*, 2022.
- Sitzmann et al. [2020]
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.
Implicit neural representations with periodic activation functions.
*Advances in neural information processing systems*, 33:7462–7473, 2020.
- Tancik et al. [2020]
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.
Fourier features let networks learn high frequency functions in low dimensional domains, 2020.
- Tao et al. [2023]
Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu.
Galip: Generative adversarial clips for text-to-image synthesis, 2023.
- Wang and Mahadevan [2008]
Chang Wang and Sridhar Mahadevan.
Manifold alignment using procrustes analysis.
In *Proceedings of the 25th International Conference on Machine Learning*, page 1120–1127, New York, NY, USA, 2008. Association for Computing Machinery.
- Wang et al. [2022]
Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi.
Clip-gen: Language-free training of a text-to-image generator with clip.
*arXiv preprint arXiv:2203.00386*, 2022.
- Wu et al. [2020]
Dongxian Wu, Shu-Tao Xia, and Yisen Wang.
Adversarial weight perturbation helps robust generalization.
*Advances in neural information processing systems*, 33:2958–2969, 2020.
- Zhang et al. [2019]
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan.
Theoretically principled trade-off between robustness and accuracy.
In *ICML*, 2019.
- Zhang et al. [2024]
Yuhui Zhang, Elaine Sui, and Serena Yeung-Levy.
Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data.
*ICLR 2024*, 2024.

## 6 Supplementary Material

This supplementary document expands on key aspects of our work by providing additional technical details and extended qualitative results. It is organized into six sections, each addressing a specific area that complements the main paper. [Section 6.1](https://arxiv.org/html/2505.23161v2#S6.SS1) offers a detailed explanation of the adversarial weight perturbation (AWP) training procedure, including how the perturbation is computed and integrated into the overall training pipeline. [Section 6.2](https://arxiv.org/html/2505.23161v2#S6.SS2) outlines the complete algorithm used for the text-to-image task. [Section 6.3](https://arxiv.org/html/2505.23161v2#S6.SS3) describes the implementation setup, including model configurations, training parameters, and data preprocessing choices. [Section 6.4](https://arxiv.org/html/2505.23161v2#S6.SS4) presents the best and worst 500 generations from MS-COCO prompts, the same samples used in the paper’s evaluation. [Section 6.5](https://arxiv.org/html/2505.23161v2#S6.SS5) provides qualitative examples generated with different pre-trained ViT-B/32 models, illustrating the variability introduced by different backbone initializations. Finally, [Section 6.6](https://arxiv.org/html/2505.23161v2#S6.SS6) includes additional qualitative results from the ablation study, offering visual comparisons that highlight the contributions of individual components. This supplementary material is intended to support reproducibility and provide a deeper insight into our methodology and experimental findings.

### 6.1 Further details on the AWP algorithm

We provide a detailed breakdown of the Adversarial Weight Perturbation (AWP) procedure and its integration into the INR training loop. [Algorithm 1](https://arxiv.org/html/2505.23161v2#alg1) outlines the core AWP mechanism: given the weights of the INR model $\phi$ and the input coordinates $(i,j)$ and a temporary clone $\widehat{\phi}$ is optimized to maximize the negative structural similarity index (SSIM) loss between the predicted output and a blurred version of the ground-truth image $\mathbf{x}$. The resulting adversarial perturbation $\Delta\phi$ is computed, normalized, and applied to the original weights $\phi$ to obtain the perturbed weights $\phi_{\text{adv}}$.

Figure: Algorithm 1 Adversarial Weight Perturbation (compute_awp)

[Algorithm 2](https://arxiv.org/html/2505.23161v2#alg2) illustrates the incorporation of AWP into INR training. At each iteration, adversarial perturbations are computed using Algorithm 1 and then applied to the network. The overall training loss is a weighted combination of mean squared error (MSE), SSIM, and $\ell_{1}$ loss. This adversarial training scheme improves the robustness and generalization of the INR by encouraging consistency under weight-level perturbations, which are the gradients received by inverting CLIP when generating.

Figure: Algorithm 2 INR Training with AWP

### 6.2 Text-to-Image full pipeline

We optimize an implicit neural representation (INR) to synthesize an image that aligns with a given text prompt using CLIP. The procedure includes text and image retrieval, feature alignment, and iterative gradient-based optimization. Procrustes alignment and natural image constraints are enabled. [Algorithm 3](https://arxiv.org/html/2505.23161v2#alg3) shows the detailed pipeline.

Figure: Algorithm 3 Text-to-Image Synthesis

### 6.3 Implementation details

#### 6.3.1 INR parameters and initialization

We initialize our implicit neural representations (INRs) with in_features = 2 and out_features = 3, using five hidden layers of 256 units each. Sinusoidal parameterization is applied with first_omega = 25 and hidden_omega = 25 to enable high-frequency signal modeling. Training is performed using the Adam optimizer with a learning rate of $1\times 10^{-4}$, and a cosine annealing schedule via torch.optim.lr_scheduler.CosineAnnealingWarmRestarts with a restart period of 100 iterations.

We apply Adversarial Weight Perturbation (AWP) using a proxy optimizer with the same learning rate and a perturbation strength of $\alpha=0.01$. The ground-truth image $\mathbf{x}$ is preprocessed using a Gaussian blur with kernel_size = 101 and $\sigma\in(10.0,20.0)$ to provide a smoother supervision signal.

The training loss combines mean squared error (MSE), structural similarity (SSIM), and $\ell_{1}$ reconstruction loss, weighted, respectively, by $\alpha_{1}=0.85$, $\alpha_{2}=0.25$, and $\alpha_{3}=0.25$.

#### 6.3.2 Text-to-image inversion parameters

Our method is built upon a ViT-B/32 backbone initialized with the default OpenAI weights. We perform 400 inversion steps using the AdamW optimizer (without AMSGrad) and a learning rate of $2\times 10^{-4}$. During INR optimization, we employ a Gaussian scheduling strategy focused on layers $[0,1,2]$, with gradient norm clipping thresholds set to $[1.0,0.5,0.2]$ respectively. This schedule is refreshed every 70 iterations to preserve both stability and optimization efficiency over time.

The loss function incorporates hyperparameters $\beta=0.5$ and $k=8$, balancing the trade-offs between reconstruction fidelity and robustness.
To promote generalization, we apply data augmentation by generating 32 variations per input sample.
For spatial alignment, we use Orthogonal Procrustes analysis over the nearest $p=256$ elements.
Following Stable Diffusion [13], we guide the CLIP inversion process by appending auxiliary textual prompts that explicitly describe desired image characteristics. This strategy improves the fidelity and perceptual quality of the generated outputs.

The complete text-to-image synthesis pipeline executes in approximately 1 minute and 18 seconds on a single NVIDIA RTX 4060. The entire process requires around 3200MiB of VRAM.

### 6.4 Best vs. Worst MS-COCO generations

To qualitatively assess model’s performance, this section presents the 500 highest and 500 lowest scoring generations based on CLIP Score, using prompts from the MS-COCO captions dataset. [Fig. G](https://arxiv.org/html/2505.23161v2#S6.F7) shows the best generations that achieved CLIP scores between 45.5 and 53.9 (mean: 47.2). [Fig. H](https://arxiv.org/html/2505.23161v2#S6.F8) shows the worst, ranging from 23.0 to 32.2 (mean: 30.6). These examples illustrate the range of output quality, from strong semantic alignment to notable failure cases.

Figure: Figure G: Best generations on MS-COCO prompts. CLIP Score ranging from 45.5 to 53.9 (mean: 47.2)
Refer to caption: extracted/6511808/figs_suppmat/best.png

Figure: Figure H: Worst generations on MS-COCO prompts. CLIP Score ranging from 23.0 to 32.2 (mean: 30.6)
Refer to caption: extracted/6511808/figs_suppmat/worst.png

### 6.5 Qualitative samples under different CLIP models

Figure: Figure I: Qualitative comparison of additional samples extending Figure 4 in the main paper.
Refer to caption: extracted/6511808/figs_suppmat/supp_results/1_clipag.png

### 6.6 Qualitative samples of the ablation study

Figure: Left: CLIP^-1 Ablation Rows 1-10
Refer to caption: extracted/6511808/figs_suppmat/supp_ablation/1.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 2
presentation: 3
questions: Overall, the paper makes a significant contribution by repurposing a frozen discriminative model for generative tasks, showcasing its potential for image synthesis. However, limitations in handling complex prompts which suggest areas for improvement and lack of comparisons to pure image generators draw me towards a reject. The work however opens promising directions for leveraging pretrained models in generative applications. 

I urge the authors to answer questions about the practical utility of inversions with encoder models? How do they see such methods replacing pure image generators or how can such methods be used to improve the encoder capabilities itself. I believe these are unanswered questions.
soundness: 3
strengths: - The paper proposes a decoder-free and tuning-free approach for text-to-image synthesis, which is computationally efficient. 

- The method supports zero-shot generalization to multiple tasks, including reconstruction, style transfer, and controlled edits, within a unified framework

- Extensive experiments on MS-COCO and Flickr30k demonstrate strong performance and also ablations showing improvements over other inversion methods.
summary: In this paper, the author introduces CLIP−1, a novel approach for text-to-image synthesis by inverting CLIP embeddings without relying on a pretrained generative decoder or fine-tuning CLIP. The method leverages Implicit Neural Representations (INRs) optimized layer-by-layer in a frequency-aware manner, enabling coarse-to-fine image generation. Overall, this inversion method sheds light on how encoder models can potentially be used as image generations and also on the inner workings of CLIP. However, I believe that there's still a way to go for such methods to be used as pure image generators.
weaknesses: - While the generated images are semantically aligned, fine-grained spatial details and textures occasionally exhibit distortions or artifacts. I understand that the authors are not competing with SoTA image generation methods, but a discussion on it would be fruitful. 

- I am a bit apprehensive on the practical applications of the method beyond just understanding the CLIP space. How far is the performance (quantitatively) from pure image generators?

- While FID, IS, and CLIP-SIM are reported, additional perceptual metrics or user studies could strengthen the evaluation of visual quality.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: How do the authors think about the scalability of this approach, given the per-image optimization loop?

Also, would this method extend to higher-resolution outputs?
soundness: 3
strengths: The technical contributions of this work are interesting. The authors propose techniques to: (1) enable a training-free approach that generates images without training any decoder, (2) mitigate the CLIP modality gap, and (3) achieve a robust, generalizable framework by leveraging AWP.

The comparison to existing work is solid, and the reported FID and other metrics are reasonable for this class of technique.
summary: This paper propose a new method of inversing CLIP encoder to achieve text-to-image geneartion using $\textnormal{CLIP}^{-1}$.
This is achieved thorugh a training-free approach, leveraging INR to achieve image embeddings similar to target text embeddings.
weaknesses: The primary concern with this work is its relevance and timeliness. The idea of inverting CLIP for image generation was novel and exciting a few years ago; however, the community has since shifted toward more scalable and expressive generative models.

Moreover, the outputs here are not yet high-quality enough, which is a significant shortcoming for practical use. Therefore, the work’s practical impact seems limited to me.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: -
soundness: 3
strengths: - The paper provides a well-rounded review of related works across multiple relevant areas, helping readers grasp the broader context and motivation behind the proposed approach.
- It integrates several complementary techniques to tackle distinct challenges—for instance, using Adversarial Weight Perturbation (AWP) for robust INR initialization and incorporating natural image priors through blending.
- Unlike prior CLIP inversion methods that rely on direct pixel-space optimization and often produce noticeable artifacts, the proposed approach demonstrates improved visual quality in its qualitative results.
- The paper includes comprehensive ablation studies, analyzing the contribution of each introduced component both quantitatively and qualitatively.
summary: This paper investigates image synthesis through CLIP inversion. Rather than optimizing directly in the pixel space, they leverage implicit neural representations (INRs) to perform the inversion. To stabilize the inverse mapping, they incorporate techniques such as adversarial training, orthogonal Procrustes projection, and a blending loss.
weaknesses: - There are some minor issues in how certain points are presented. For instance, in the related work section, the statement “Modern image generators fall into three…” overlooks autoregressive models, which are also widely used and should be acknowledged.
- From a practical perspective, while the approach is conceptually interesting, it’s not particularly applicable in real-world scenarios. As shown in the results, the generated images still lag far behind other methods. However, the work remains valuable from an interpretability and analytical standpoint.
- The visualization and presentation could be improved. Specifically, Figures 5 and Tables 2 and 3 are rendered too small, which hinders readability.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: Following up on the main weakness: 
1) Could the authors provide a direct comparison of the inference time (time per image on a standard GPU like an A100 or RTX 4090) and peak VRAM usage for $CLIP^{-1}$ against the SOTA generative baselines listed in Table 1 (e.g., LDM-KL-8 or GLIDE)? This would be invaluable for understanding the practical trade-off between the method's training cost and its inference cost.
2) The optimization process is set to 400 steps. Have the authors experimented with this hyperparameter? How does image quality (FID) degrade as the number of steps is reduced? Is there a "sweet spot" that balances quality and speed? 
3) This question is about the robustness of the initialization strategy. The paper's AWP-trained seed image is described as a "robust anchor". We are interested in how strongly this anchor constrains the final generation.
- First, the paper shows a "Plain $CLIP^{-1}$" variant that starts from random INR weights. Could the authors provide a qualitative comparison for this variant? Does it simply converge slower, or does it suffer from more significant artifacts, confirming the necessity of the AWP anchor?
- Second, what happens in an adversarial scenario where the retrieved seed image is semantically incorrect (e.g., the prompt is "a photo of a dog" but the retrieved seed is "a photo of a car")? Is the optimization pipeline, guided by the Procrustes-corrected text embedding and the blending loss, strong enough to correct this completely wrong starting point and still generate a "dog"? Or does the robust anchor effectively trap the optimization near the (incorrect) seed, leading to a failed generation?
4) The use of Orthogonal Procrustes is a very clever and simple solution for the modality gap. Did the authors experiment with any alternative methods, such as learning a small, lightweight mapping network (e.g., a simple MLP) to bridge the gap? Was Procrustes chosen primarily for its simplicity and training-free nature, or did it also empirically outperform other simple alternatives?
soundness: 3
strengths: 1) The paper's core idea, unlocking the generative potential hidden within a frozen discriminative model, is highly original and significant. It moves beyond simply using CLIP as an encoder and shows it can be "run in reverse" as a generator, challenging the conventional wisdom that separates discriminative and generative models.

2) The key innovation is reframing the inversion problem from pixel space to the weight space of an Implicit Neural Representation (INR). This choice enforces smoothness, helps in preventing high-frequency artifacts and enables a structured coarse-to-fine optimization by leveraging the frequency-stratified architecture of the FINER network.

3) The paper introduces simple, effective, and computationally cheap solutions to complex problems. I like the use of Orthogonal Procrustes Analysis to geometrically correct the modality gap, as it requires no training and is demonstrably effective.

4) The combination of Adversarial Weight Perturbation (AWP) for initialization and the blending loss creates a more stable optimization process. These components successfully anchor the search and prevent it from converging to the "adversarial examples" (perceptually meaningless noise) that have plagued previous inversion attempts. 
5) The ablation study is thorough and provides a clear justification for each component of the $CLIP^{-1}$ pipeline.
summary: This paper introduces $CLIP^{-1}$, a novel framework for text-to-image synthesis that demonstrates the latent generative capabilities of a frozen, pre-trained discriminative model (CLIP) . The central claim is that a dedicated, trained generative decoder is not strictly necessary. Instead, the authors reframe image synthesis as an inversion problem: finding an image that, when encoded by CLIP's image encoder, matches a target text embedding .The paper's core contribution is a set of techniques to solve this notoriously ill-posed problem, which typically yields unstable, non-naturalistic images (adversarial examples) . The authors' solution avoids direct pixel optimization and instead optimizes the weights of a frequency-aware Implicit Neural Representation (INR), which inherently enforces smoothness and a structured, coarse-to-fine generation process .To further stabilize this inversion, the $CLIP^{-1}$ pipeline introduces three key components:An adversarially robust initialization using Adversarial Weight Perturbation (AWP) on a blurred seed image, which anchors the optimization in a stable "flat" region of the loss landscape .A lightweight, geometric solution to the modality gap using Orthogonal Procrustes Analysis to project the target text embedding onto the image embedding manifold, providing a more stable target .A blending loss that acts as a natural image prior, regularizing the output by encouraging it to remain close to the statistics of real image embeddings .Empirically, $CLIP^{-1}$ substantially outperforms other training-free inversion methods like DAS, achieving a Fréchet Inception Distance (FID) that is less than half of its competitor . The framework also demonstrates zero-shot versatility on downstream tasks, including image reconstruction, controlled text-guided editing, and neural style transfer.
weaknesses: 1) The paper's primary weakness is its discussion of the computational cost. The method is "training-free", but it shifts the entire computational load to inference time, which requires an iterative optimization process (400 steps). The runtime analysis in the appendix compares $CLIP^{-1}$ only to DAS, another inversion-based method. The main quantitative comparison compares quality (FID/IS) against SOTA generative models like LDM and GLIDE but critically omits a comparison of inference time and VRAM usage. A single forward pass through a diffusion decoder is likely much faster than the 1+ minute optimization required by $CLIP^{-1}$. To fairly evaluate this as a generative "decoder" alternative, this trade-off must be made explicit. 
2) As the authors rightly acknowledge, a significant gap in photorealism and fine-detail fidelity remains when compared to SOTA diffusion models. The generated images, while coherent, often have a "painterly" or slightly artificial quality and lack crisp, high-frequency textures. This limits the method's immediate practical application for photorealistic generation.Initialization.
3) The full, best-performing model relies on retrieving a semantically relevant seed image from the LAION dataset to perform its AWP-based initialization. While the out-of-distribution evaluation shows that a "Plain $CLIP^{-1}$" variant (starting from random weights) can still function, the main method's output is still influenced by this seed. This could constrain the creative novelty of the generations, potentially biasing them towards common compositions found in the retrieval database.


Decision: Accept (Poster)