Optimal Generative Cyclic Transport between Image and Text

Authors: Haoming Liu, Yuanhe Guo, Yijia Cao, Shengjie Wang, Hongyi Wen
Primary area: generative models
Keywords: Cross-modal Information Transformation, Generative Models, Optimal Transport

Abstract:
Deep generative models, such as vision-language models (VLMs) and diffusion models (DMs), have achieved remarkable success in cross-modality generation tasks. However, the cyclic transformation of text $\rightarrow$ image $\rightarrow$ text often fails to secure an exact match between the original and the reconstructed content. In this work, we attempt to address this challenge by utilizing a deterministic function to guide the reconstruction of precise information via generative models. Using a color histogram as guidance, we first identify a soft prompt to generate the desired text using a language model and map the soft prompt to a target histogram. We then utilize the target color histogram as a constraint for the diffusion model and formulate the intervention as an optimal transport problem. As a result, the generated image has the exact color histogram as the target, which can be converted to a soft prompt deterministically for reconstructing the text. This allows the generated images to entail arbitrary forms of text (e.g., natural text, code, URLs, etc.) while ensuring the visual content is as natural as possible. Our method offers significant potential for applications on histogram-constrained generation, such as steganography and conditional generation in latent space with semantic meanings.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730546991969

thread_type:
official_review

<thread_object>
creator: reviewer_dagu

<content>
summary:
This work proposes a novel text-image-text reconstruction method in cross-modality generation by guiding generative models with a deterministic function. The authors use color histograms as constraints to perform precise reconstruction, and this method can be applied for data protection.

strengths:
1. The authors propose a novel and effective method for the cyclic transformation of text → image → text.

2. The manuscript is well-written and easy to understand.

3. Both the visualization and quantitative results seem promising.

4. The use of color histogram guidance is interesting.

weaknesses:
1. Additional robustness analysis is needed to evaluate the effects of noise, blur, JPEG compression, and other transmission channel distortions.

2. The first section should ideally begin with a discussion on common applications of text → image → text translation.

3. There are already numerous generative methods for data protection, so the contribution stated in line 442 seems somewhat overstated. The authors should discuss their differences from existing methods in more detail, such as:

    [r1] StegaStyleGAN: Towards Generic and Practical Generative Image Steganography (AAAI 24)

    [r2] Cross: Diffusion Model Makes Controllable, Robust, and Secure Image Steganography (NeurIPS 24)

    [r3] StegaDDPM: Generative Image Steganography based on Denoising Diffusion Probabilistic Model (MM 23)

    [r4] Generative Steganography via Auto-Generation of Semantic Object Contours (TIFS 24)

    [r5] Secret-to-Image Reversible Transformation for Generative Steganography (TDSC 23)

4. There are a few typo errors throughout the manuscript. For example, in line 391, “a100” should be corrected to “A100.”

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730820682942

thread_type:
official_review

<thread_object>
creator: reviewer_fk7m

<content>
summary:
This paper propose a method to generate images more aligned with the text prompt using optimal generative cycle consistency between image and text. They ensure the image consistency, using color histogram matching using optimal transport and show the generated images are natural looking for various tasks.

strengths:
1. The idea of using optimal transport for color histogram matching is interesting.

2. The generated results looks good, to some extent.

weaknesses:
1. Why matching color histogram is enough to generate consistent images? E.g., it might happen that two images have exact same color histogram, but totally differnet content. Color domain discripency makes sense if the diffusion model is conditioned on images, but for text-to-image diffusion model, it is highly unlikely that the prompt can capture the exact details. Please clarify this.

2. There are works which address the issue of (image->text->image) consitency using pretrained caption and diffusion models, comparing with those methods need to be done. 

3. Exact prompt and seed might vary across machines, implementations, how the authors justify this? E.g., they attempt to retreive the specific diffusion model and seeds from the images, which I am not convinced to be extracted using these details. Please clarify.

4. Instead of just visualization, the authors might quantify their approach for some recognition tasks, e.g., captioning, classififcation etc.

5. The paper lacks coherence sometimes, e.g., in the abstract they mentioned about steganography, but it is not explored anywhere. 

6. Overall writing and flow need to be improved.

[1] Roy et al. "Cap2aug: Caption guided image to image data augmentation". WACV 2025 

[2] Kondapaneni et al. "Text-image Alignment for Diffusion-based Perception", CVPR 2024

questions:
Please justify the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730912230249

thread_type:
official_review

<thread_object>
creator: reviewer_g8lh

<content>
summary:
The paper presents a novel approach, Optimal Generative Cyclic Transport (OGCT), aimed at achieving exact, cyclic transformations between text and image modalities. The core of this work involves encoding text into images using a soft embedding that aligns with a color histogram, which can be recovered deterministically. The technique leverages diffusion models and optimal transport to generate images that encapsulate the encoded text as a recoverable soft prompt. The paper proposes and evaluates multiple binning strategies, with the random binning having perceptually indistinguishable outputs from the unencoded images

strengths:
- The paper addresses an important challenge in multimodal generative models, where information is typically lost during cyclic transformations. This application could be instrumental for secure communication and data integrity.

- The introduction of a reversible soft embedding that maps text to color histograms, recoverable through a deterministic optimal transport algorithm, represents a novel approach in the field.

- The method is described in a structured manner, with an explicit algorithm provided for the entire OGCT process. This detailed exposition facilitates understanding and reproducibility.

- The random binning algorithm effectively enables the storage of a large quantity of text information within images while keeping the visual impact minimal, showing potential in practical scenarios.

weaknesses:
- The method's reliance on color histograms raises concerns about robustness. While the authors have demonstrated resilience to rescaling, the algorithm may be sensitive to other common augmentations like color jitter, cropping, rotation, blur, and Gaussian noise. Testing against these transformations is necessary to confirm its applicability in realistic, potentially hostile scenarios.

- The experiments presented do not cover a comprehensive analysis of the algorithm. Including experiments such as variable perturbation time steps and perturbation strengths would provide deeper insights into the algorithm’s robustness and limitations.

questions:
- In Section 2.4, the authors introduce the $\varphi$ function for histogram matching in the diffusion model but do not provide clarity on its calculation through the optimal transport approach. It would be beneficial if the authors included pseudo-code or an explanation of how the "OT-histogram-matching" function is implemented.

- Could the authors clarify the selection of perturbation steps, $\tau$, used in the experiments? Additionally, has there been any investigation into how using fewer or different sets of perturbation steps impacts the performance, especially if perturbations are applied only at the last generation step ($t=1$)?

- If the method is inherently unsuited to handle augmentations such as color jitter, what alternative solutions do the authors suggest? Could other functions besides color histograms be incorporated into the algorithm? The paper would gain greater value by incorporating a more generalized set of transformations alongside color histograms to enhance the robustness of the method.

- The suggested algorithm performs VAE-decoding on $z_0^t$ values calculated during the generation process. Can the authors provide a study or an experiment they performed to show the reliability of using VAE-decoding on these approximate values of $z_0$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731077503194

thread_type:
official_review

<thread_object>
creator: reviewer_vkg7

<content>
summary:
The paper presents the Optimal Generative Cyclic Transport (OGCT) framework, aimed at achieving precise, lossless cyclic transformations across image and text modalities. Leveraging color histograms as a guidance vector, the proposed framework optimizes "soft prompts" within a language model and constrains the diffusion process in a generative model to encode text-based information directly into image color histograms. The encoded images are then decodable to the original text sequence using the same histogram vector, making this framework robust and flexible for applications in secure communication and generative encryption.

strengths:
The paper is well-organized and detailed.

weaknesses:
1. The paper lacks an evaluation of OGCT’s robustness against common image compression and information degradation techniques, as demonstrated in Table 2 of the paper [1]. In real-world scenarios, images often undergo transformations that introduce minor information loss, such as JPEG compression, added noise, and other alterations, which could potentially affect the integrity of OGCT’s color histogram-based decoding. Without experiments that test OGCT’s performance under these conditions, it is unclear how well the framework would perform in practical applications where images are subject to compression or other modifications. 

2. The robustness of the OGCT under different types of input text or image complexity is not thoroughly tested. The framework may encounter difficulties when encoding highly complex images or diverse language styles, particularly with non-standard characters, which could affect accuracy in real-world scenarios.

[1] Yu, Jiwen, et al. "Cross: Diffusion model makes controllable, robust and secure image steganography." Advances in Neural Information Processing Systems 36 (2024).

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732539667698

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Comments by Authors

comment:
**We thank all reviewers for their invaluable feedback and have revised the paper and appendix accordingly. The revised parts in the original text and the newly added sections in the Appendix have been noted in blue.**

The major revisions include:

[1] The big picture about the proposed framework for histogram-constrained image generation (Sec.1, Sec.4.1, Sec.6).

[2] Related work discussion on Steganography and its relation to the proposed framework (Sec. 5).

[3] Robustness evaluation of OGCT against complex data, wrong histogram, and Gaussian noise. (Sec. B)

[4] Qualitative results for OGCT in latent space as well as its robustness against JPEG compression. (Sec. C)

[5] Further details about diffusion process intervention: choice of perturbation time steps, visualization of intermediate predictions, and histogram matching implementation. (Sec. D)

Meanwhile, we wish to further clarify a common question on the relationship between steganography and our framework:

1) **We position our framework to be a histogram-constrained generation approach, where steganography is one of its prominent applications (done by relating the histogram to the soft prompt).**

2) Unlike many existing steganography approaches that are restricted to dataset-specific scenarios, the steganography enabled by our approach is widely applicable to any text-to-image model and any form of text in a plug-and-play manner.

3) Typical steganography works tend to hide information by encoding bits or encrypted/transformed bits, whereas we are the first approach that encodes the information using an off-the-shelf generative model.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this work, the authors presents the Optimal Generative Cyclic Transport (OGCT) framework to address the issue of securing an exact match between the original and reconstructed contents for the cyclic transformation of text -> image -> text.
Overall, its main weakness is insufficient evaluations, as commonly mentioned by reviewers as follows:
Reviewer G8LH
``...I meant assessing performance when augmentations like color jitter, blurring, or Gaussian noise are applied to the images in pixel-space. The paper lacks sufficient experiments on this aspect, and the evaluation remains incomplete...including robustness tests, especially since pixel-space augmentations may affect the method's reliance on color histograms.''

Reviewer fK7m
``... concerns are not well addressed. E.g., if the content is different, but the color histogram is same, how the decoded information is same? In that case the essence of generative models are not used. Also, I believe that comparing with the above-mentioned works needs to be done for generalization...''

Reviewer dAGu
``... agree with the weaknesses pointed out by Reviewer fK7m and I believe fK7m's concerns are not well addressed....''