FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 18:37:49

Abstract: Large-scale text-to-image diffusion models have been a revolutionary milestone in the evolution of generative AI and multimodal technology, allowing extraordinary image generation with natural-language text prompts. However, the issue of lacking controllability of such models restricts their practical applicability for real-life content creation, for which attention has been focused on leveraging a reference image to control text-to-image synthesis. This paper contributes a concise and efficient approach that adapts the pre-trained text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plug-and-play manner, realizing high-quality and versatile text-driven I2I translation without any model training, model fine-tuning, or online optimization. To guide T2I generation with a reference image, we propose to model diverse guiding factors with different frequency bands of diffusion features in DCT spectral space, and accordingly devise a novel frequency band substitution layer that dynamically substitutes a certain DCT frequency band of diffusion features with the corresponding counterpart of the reference image along the reverse sampling process. We demonstrate that our method flexibly enables highly controllable text-driven I2I translation both in the guiding factor and guiding intensity of the reference image, simply by adjusting the type and bandwidth of the substituted frequency band, respectively. Extensive experiments verify the superiority of our approach over related methods in image translation visual quality, versatility, and efficiency.

Main paper: 
 introduction: 
 Text-driven I2I translation is an appealing computer vision problem that aims to translate a reference image with open-domain text prompts, and is also a typical application of the booming multimodal technology. Since the advent of CLIP [29] bridging vision and language with large-scale contrastive pre-training, attempts have been made to instruct image manipulation with text by combining CLIP with generative models. VQGAN-CLIP [6] pioneers Unpublished working draft. Not for distribution.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ACM MM, 2024, Melbourne, Australia © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn Reference "young man" "little girl" "elderly lady" "robot" "painting of a park" "Rome ruins" "space station" "market" "autumn" "winter" "desert" "iceberg" text-driven image translation by optimizing VQGAN [9] image embedding with CLIP image-text similarity loss. DiffusionCLIP [16] fine-tunes diffusion model [12] under CLIP loss to manipulate an image with a text. DiffuseIT [17] combines VIT-based structure loss [39] and CLIP-based semantic loss to guide the diffusion sampling process via manifold constrained gradient [5] , synthesizing translated image that complies with the target text while maintaining the structure of the reference image. However, these methods are not competitive in generation quality due to limited model capacity and training data of the backbone generative model. To promote image translation visual quality, efforts have been made to train large models on massive data. InstructPix2Pix [2] employs GPT-3 [3] and Stable Diffusion [31] to synthesize huge amounts of paired training data, based on which trains a supervised text-driven I2I mapping for general image manipulation tasks. Design Booster [37] trains a latent diffusion model [31] conditioned on a joint representation that fuses both text embedding and image embedding, realizing layout-preserved text-driven I2I translation. Nevertheless, these methods are remarkably computationally intensive due to the need for training large models on immense data.To circumvent formidable training costs, research has been focused on leveraging off-the-shelf large-scale T2I diffusion models for text-driven I2I translation. This type of methods further divide into fine-tuning-based methods and inversion-based methods.The former type of fine-tuning-based methods represented by SINE [45] and Imagic [15] fine-tune the pre-trained T2I diffusion model to reconstruct the reference image before manipulating it with a target text. These methods require separate fine-tuning of the entire model for each time of image editing, which is less efficient and prone to underfitting or overfitting to the reference image.The latter type of inversion-based methods invert the reference image into diffusion model Gaussian noise space and then generate the translated image via diffusion sampling process guided by the target text. A pivotal challenge of this pipeline is that the sampling trajectory may severely deviate from the inversion trajectory due to the error accumulation caused by the classifier-free guidance technique [13] , which impairs the correlation between the reference and the translated image. To remedy this issue, Null-text Inversion [23] optimizes unconditional null-text embedding to align the sampling trajectory to the inversion trajectory. Prompt Tuning Inversion [8] and StyleDiffusion [18] minimize trajectory divergence by learning to encode the information of the reference image into learnable conditional embedding. Pix2Pix-zero [26] penalizes trajectory deviation by matching cross-attention maps between the two trajectories with least-square loss. These methods apply online optimization at each diffusion time step to calibrate the whole sampling trajectory, introducing additional time overhead. Moreover, most of these methods rely on cross-attention control introduced by P2P [11] for structure preservation, requiring paired source text of the reference image, which is not available in most cases. PAP [40] maintains image structure by extracting and injecting the internal feature maps and self-attention maps of the denoising U-Net into the reverse sampling trajectory, realizing optimization-free text-driven I2I translation, though the designed feature extraction and manipulation pipeline is heuristic, cumbersome, and time-consuming.In this paper, we propose a concise and efficient approach termed FBSDiff, realizing plug-and-play and highly controllable text-driven I2I translation from a frequency-domain perspective. To guide T2I generation with a reference image, a key missing ingredient of existing methods is the mechanism to control the guiding factor (e.g., image appearance, layout, contour) and guiding intensity. Since the guiding factors of the reference image are difficult to isolate in the spatial domain but are decomposable in the frequency domain, we consider modeling different guiding factors with the corresponding frequency bands of diffusion features in the Discrete Cosine Transform (DCT) spectral space. Based on this motivation, we propose an inversion-based text-driven I2I framework characterized by a novel frequency band substitution mechanism, which realizes plugand-play and controllable reference image guidance by dynamically substituting a certain DCT frequency band of diffusion features with the corresponding counterpart of the reference image along the reverse sampling process. As displayed in Fig. 1 , T2I image synthesis with appearance and layout control, pure layout control, and contour control of the reference image to the generated image can be realized by substituting the low-frequency band, mid-frequency band, and high-frequency band, respectively, and thus allowing highly controllable and versatile text-driven I2I translation.The strengths of our approach are fourfold: (I) dynamic reference image control at inference time, realizing plug-and-play text-driven I2I translation; (II) conciseness and efficiency, our method dispenses with the need for paired source text as well as cumbersome attention modulations as compared with existing methods, while still achieving leading I2I performance; (III) more generic methodology, our method applies frequency band transplantation on the denoised features along the reverse sampling trajectory, requiring no access to any internal feature embedding of the denoising network, and thus decouples with the specific diffusion model backbone as contrasted with existing methods; (IV) our method allows to flexibly control the guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band. The effectiveness of our method is fully demonstrated with both qualitative and quantitative evaluations. To summarize, we make the following key contributions:• We provide new insights about controllable diffusion process from a novel frequency-domain perspective. • We propose a novel frequency band substitution technology, realizing plug-and-play text-driven I2I translation without any model training, fine-tuning, or online optimization. • We contribute a concise and efficient text-driven I2I framework that is free from source text and cumbersome attention modulations, highly controllable in both guiding factor and guiding intensity of the reference image, and invariant to the used diffusion model backbone, all while achieving superior I2I translation performance among existing methods. 
related work 2.1 diffusion model: 
 Since the advent of DDPM [12] , diffusion model has soon dominated the family of generative models [7] . DDIM [36] and its variants [20, 20] substantially accelerate diffusion model sampling process. Palette [32] extends diffusion model to the realm of conditional image synthesis. Large-scale T2I diffusion models [25, 30, 33] bring image creation to an unprecedented level, whose computation overhead is significantly reduced by LDM [31] by training diffusion model in low-dimensional feature space. ControlNet [43] and T2iadapter [24] add spatial control to T2I diffusion models by training a control module conditioned on certain image priors. SDXL [28] and DiTs [27] improve diffusion model backbone to larger capacity. Now, diffusion model has been applied to a wide variety of vision tasks with noticeable performance gains [1, 19, 21, 22, 34, 38, 42] , and is still making rapid progress in theory and application.Neural networks are mostly used to tackle vision tasks in the spatial or temporal domain, some research improves model performance from a frequency-domain perspective. For example, Ghosh et al. [10] introduce DCT to CNN to accelerate network convergence. Xie et al. [41] propose a frequency-aware dynamic network for lightweight image super-resolution. Cai et al. [4] impose Fourier frequency spectrum consistency to image translation tasks for better identity preservation. FreeU [35] improves image generation quality by selectively enhancing or depressing different frequency components of diffusion model U-Net features.  
method: 
 In this section, we first describe the overall model architecture, then elaborate on the frequency band substitution mechanism, and finally summarize the algorithm and show implementation details.For the diffusion model background, please refer to the appendix.Built on the pre-trained Latent Diffusion Model (LDM), FBSDiff adapts it from T2I generation to text-driven I2I translation by proposing a plug-and-play reference image guidance mechanism, realizing controllable guiding factor and guiding intensity of the reference image via dynamic frequency band substitution. As Fig. 2 shows, FBSDiff comprises three diffusion trajectories: (i) inversion trajectory (𝑧 0 → 𝑧 𝑇 𝑖𝑛𝑣 ); (ii) reconstruction trajectory (𝑧 𝑇 𝑖𝑛𝑣 = ẑ𝑇 → ẑ0 ≈ 𝑧 0 ); (iii) sampling trajectory ( z𝑇 → z0 ). Starting from the initial feature 𝑧 0 = 𝐸 (𝑥) extracted from the reference image 𝑥 by the encoder 𝐸, a 𝑇 𝑖𝑛𝑣 -step DDIM inversion is employed to project 𝑧 0 into the Gaussian noise latent space conditioned on the null-text embedding 𝑣 ∅ , based on the assumption that the ODE process can be reversed in the limit of small steps:EQUATIONEQUATIONwhere { ᾱ𝑡 } are schedule parameters that follows the same setting as DDPM [12] , 𝜖 𝜃 is the denoising U-Net of the pre-trained LDM. The Gaussian noise 𝑧 𝑇 𝑖𝑛𝑣 obtained after the 𝑇 𝑖𝑛𝑣 -step DDIM inversion is directly used as the initial noise feature of the subsequent reconstruction trajectory, which is a 𝑇 -step DDIM sampling process that reconstructs ẑ0 ≈ 𝑧 0 from the inverted noise feature ẑ𝑇 = 𝑧 𝑇 𝑖𝑛𝑣 :EQUATIONin which 𝑓 𝜃 ( ẑ𝑡 , 𝑡, 𝑣 ∅ ) follows the same form as Eq. 5. The length of the reconstruction trajectory could be much smaller than that of the inversion trajectory (i.e., 𝑇 ≪ 𝑇 𝑖𝑛𝑣 ) to save inference time. The same null-text embedding 𝑣 ∅ is conditioned in the reconstruction trajectory to ensure feature reconstructability (i.e., ẑ0 ≈ 𝑧 0 ). Meanwhile, an equal-length sampling trajectory is applied in parallel with the reconstruction trajectory to synthesize the target image. The sampling trajectory is also a 𝑇 -step DDIM sampling that progressively denoises a randomly initialized Gaussian noise feature x𝑇 ∼ N (0, 𝐼 ) into x0 conditioned on the target-text embedding 𝑣. To amplify the effect of text guidance, classifier-free guidance technique [13] is utilized which interpolates conditional (target-text) and unconditional (null-text) noise prediction at each time step with a guidance scale 𝜔 during the sampling trajectory:EQUATIONEQUATIONEQUATION) Due to the inherent property of DDIM inversion and DDIM sampling, the reconstruction trajectory forms a deterministic denoising mapping towards the reference image. Therefore, the intermediate denoising results { ẑ𝑡 } along the reconstruction trajectory can be used as pivotal guidance features to calibrate the corresponding counterpart { z𝑡 } along the sampling trajectory to establish the correlation between the reference image and the generated image, and thus enables text-driven I2I translation. Specifically, we implement feature calibration by inserting a plug-and-play frequency band substitution (FBS) layer in between the reconstruction trajectory and the sampling trajectory. FBS layer substitutes a certain frequency band of z𝑡 with the same frequency band of ẑ𝑡 along the sampling process to impose a certain guiding effect of the reference image, where both the guiding factor (e.g., appearance, layout, contour) and guiding intensity are flexibly controllable by tuning the type and bandwidth of the substituted frequency band, respectively.To balance image guidance and generation quality, we partition the sampling trajectory into a calibration phase and a noncalibration phase separated by the time step 𝜆𝑇 . In the former calibration phase ( z𝑇 → z𝜆𝑇 ), dynamic frequency band substitution is applied at each time step for stable calibration of the sampling process; in the latter non-calibration phase ( z𝜆𝑇 -1 → z0 ), we remove FBS layer to avoid over-constrained sampling results, fully unleashing the generative power of the diffusion model to improve image generation quality. Here 𝜆 denotes the ratio of the length of the non-calibration phase to that of the entire sampling trajectory.At last, the final result z0 of the sampling trajectory is converted to the translated image x via the decoder 𝐷, i.e., x = 𝐷 ( z0 ).As Fig. 3 illustrates, the FBS layer takes in a pair of diffusion features ẑ𝑡 and z𝑡 , converts them from the spatial domain into the frequency domain with 2D-DCT, then transplants a certain frequency band in the DCT spectrum of ẑ𝑡 to the same position in the DCT spectrum of z𝑡 . Finally, 2D-IDCT is applied to transform the fused DCT spectrum back into the spatial domain as the calibrated z𝑡 .In 2D DCT spectrum, elements with smaller coordinates (nearer to the top-left origin) encode lower-frequency image information, larger-coordinate elements correspond to higher-frequency image components, and most of the DCT spectral energy is occupied by a small proportion of low-frequency elements.In the FBS layer, the sum of 2D coordinates is used as thresholds to extract DCT frequency bands of different types and bandwidths through binary masking. Specifically, we design three types of binary masks which are respectively termed the low-pass mask (𝑀𝑎𝑠𝑘 𝑙𝑝 ), high-pass mask (𝑀𝑎𝑠𝑘 ℎ𝑝 ), and mid-pass mask (𝑀𝑎𝑠𝑘 𝑚𝑝 ):       𝑀𝑎𝑠𝑘 𝑙𝑝 (𝑥, 𝑦) = 1 𝑖 𝑓 𝑥 + 𝑦 ≤ 𝑡ℎ 𝑙𝑝 𝑒𝑙𝑠𝑒 0, 𝑀𝑎𝑠𝑘 ℎ𝑝 (𝑥, 𝑦) = 1 𝑖 𝑓 𝑥 + 𝑦 > 𝑡ℎ ℎ𝑝 𝑒𝑙𝑠𝑒 0, 𝑀𝑎𝑠𝑘 𝑚𝑝 (𝑥, 𝑦) = 1 𝑖 𝑓 𝑡ℎ 𝑚𝑝1 < 𝑥 + 𝑦 ≤ 𝑡ℎ 𝑚𝑝2 𝑒𝑙𝑠𝑒 0,where 𝑡ℎ 𝑙𝑝 is the threshold of the low-pass filtering; 𝑡ℎ ℎ𝑝 is the threshold of the high-pass filtering; 𝑡ℎ 𝑚𝑝1 and 𝑡ℎ 𝑚𝑝2 are respectively the lower and upper bound of the mid-pass filtering. Given a binary mask 𝑀𝑎𝑠𝑘 * ∈ {𝑀𝑎𝑠𝑘 𝑙𝑝 , 𝑀𝑎𝑠𝑘 ℎ𝑝 , 𝑀𝑎𝑠𝑘 𝑚𝑝 }, the frequency band substitution operation in the FBS layer can be formulated as: (7) where 𝐷𝐶𝑇 and 𝐼 𝐷𝐶𝑇 refers to the 2D-DCT and 2D-IDCT transformations respectively, which are described in detail in the Supplementary Materials. The usage of the low-pass mask 𝑀𝑎𝑠𝑘 𝑙𝑝 , high-pass mask 𝑀𝑎𝑠𝑘 ℎ𝑝 , and mid-pass mask 𝑀𝑎𝑠𝑘 𝑚𝑝 respectively corresponds to the extraction and substitution of the low-frequencyz𝑡 = 𝐼 𝐷𝐶𝑇 (𝐷𝐶𝑇 ( ẑ𝑡 ) • 𝑀𝑎𝑠𝑘 * + 𝐷𝐶𝑇 ( z𝑡 ) • (1 -𝑀𝑎𝑠𝑘 * )),Input: the reference image 𝑥 and the target text. Output: the translated image x.1: Extract the initial latent feature 𝑧 0 = 𝐸 (𝑥). compute z𝑡 -1 from z via Eq. 4; 13: end for{DDIM sampling in the non-calibration phase} 14: Obtain z0 and the final translated image x = 𝐷 ( z0 ).band, high-frequency band, and mid-frequency band, which controls different guiding factors of the reference image to the finally generated image:• Low-frequency band substitution enables low-frequency information guidance of 𝑥, realizing appearance (e.g., color, luminance) and layout control over the generated image x. • High-frequency band substitution enables high-frequency information guidance of the reference image 𝑥, realizing contour control over the generated image x. • Mid-frequency band substitution enables mid-frequency information guidance of the reference image 𝑥. By filtering out higher-frequency contour information and lowerfrequency appearance information in the DCT spectrum, it realizes pure layout control over the generated image x. The DCT masking type and the corresponding thresholds used in the FBS layer are algorithm hyper-parameters, which could be flexibly modulated to enable diverse guiding factors and continuous guiding intensity of the reference image to the generated image.We use the pre-trained Stable Diffusion v1.5 as backbone diffusion model and set the classifier-free guidance scale 𝜔 = 7.5. We use 1000-step DDIM inversion to ensure high-quality reconstruction, i.e., 𝑇 𝑖𝑛𝑣 =1000, and use 50-step DDIM sampling for the reconstruction and sampling trajectory, i.e., 𝑇 =50. Along the sampling trajectory, we allocate 55% time steps to the calibration phase and the remaining 45% steps for the non-calibration phase, i.e., 𝜆=0.45. For the default DCT masking thresholds used in the FBS layer, we set 𝑡ℎ 𝑙𝑝 =80 for the low-frequency band substitution (low-FBS); 𝑡ℎ ℎ𝑝 =5 for the high-frequency band substitution (high-FBS); 𝑡ℎ 𝑚𝑝1 =5, 𝑡ℎ 𝑚𝑝2 =80 for the mid-frequency band substitution (mid-FBS). The complete algorithm of FBSDiff is described in Alg. 1. 
experiments: 
 In this section, we first present and analyze the qualitative results of our method; then delve into the frequency band substitution with ablation studies; and finally show quantitative evaluations. is not restricted. The mid-FBS mainly enforces image layout control by filtering out lower-frequency appearance information and higher-frequency contour information of the reference image in the DCT domain. For all three types of frequency band substitution, the image translation results exhibit high visual quality and high fidelity to the text prompts, both for real-world and artistic reference images. The effect of guiding factor decomposition and control is more clearly demonstrated in Fig. 5 . Given a target text for image manipulation, FBSDiff preserves image appearance with low-FBS, maintains image contour while allowing appearance change with high-FBS, and constrains only image layout with mid-FBS. Pursue appearance divergence In figure 7 , we qualitatively compare our method with SOTA text-driven I2I translation methods. Results in the top panel show that our method with low-FBS achieves better appearance consistency between the reference and the translated image than related approaches, and is thus more suitable to image creation scenario where we want to largely borrow the appearance and style from an existing image. Results in the bottom panel show that existing SOTA inversion-based methods struggle at producing text-driven I2I results that largely deviate from the reference images in visual appearance, while our method with high-FBS enables to generate the translated images with noticeably different appearance, and is thus more suitable to image creation scenario where appearance divergence is pursued. Besides, an advantage of our approach over𝑡ℎ #$ ＝30 𝑡ℎ #$ ＝40 𝑡ℎ #$ ＝50 𝑡ℎ #$ ＝60 𝑡ℎ #$ ＝70 𝑡ℎ #$ ＝80 𝑡ℎ #$ ＝90 𝑡ℎ #$ ＝100Figure 10 : Demonstration of our method in controlling the low-frequency information guiding intensity by varying the 𝑡ℎ 𝑙𝑝 in the low-FBS. related methods is sampling diversity. As displayed in Fig. 6 , our FB-SDiff can produce diverse I2I results for fixed reference image and target text due to randomly sampling x𝑇 from isotropic Gaussian distribution, while other inversion-based methods [8, 18, 23, 26, 40] lack such sampling diversity for directly initializing x𝑇 with the inverted image embedding. The reference image control functionality of FBS is clearly shown in Fig. 8 , in which we see that removing FBS leads to the sampled results with no correlation with the reference image. Moreover, as Fig. 9 displays, our method can well adapt to the text prompts with varying semantic discrepancies with the reference image, producing I2I results that comply with the target text even in the cases of large semantic discrepancy.𝑡ℎ #$% =5 𝑡ℎ #$& =60 𝑡ℎ #$% =5 𝑡ℎ #$& =70 𝑡ℎ #$% =5 𝑡ℎ #$& =80 𝑡ℎ #$% =5 𝑡ℎ #$& =90Besides the controllability in the guiding factors of the reference image, the guiding intensity is also controllable by modulating the bandwidth of the substituted frequency band. Results displayed in Fig. 10 demonstrate the appearance consistency control of our method by adjusting the low-pass filtering threshold 𝑡ℎ 𝑙𝑝 in the low-FBS. In this case, larger value of 𝑡ℎ 𝑙𝑝 corresponds to wider bandwidth (more information) of the transplanted frequency band, leading to I2I results with more resemblance to the reference image, while lower value of 𝑡ℎ 𝑙𝑝 brings more variations of the generated results to the reference images. Likewise, results in Fig. 11 demonstrate the structure consistency control of our method by tuning the mid-pass filtering upper bound threshold 𝑡ℎ 𝑚𝑝2 in the mid-FBS. When increasing the value of 𝑡ℎ 𝑚𝑝2 , more high-frequency information of the reference image is included and transplanted, leading to more accurate contours of the reference image transferred into the generated result. Decreasing the value of 𝑡ℎ 𝑚𝑝2 , on the contrary, shrinks the transplanted high-frequency information, and thus leads to weaker structure consistency of the generated images.We also explore other designs of frequency band substitution, including substituting the frequency band only once at 𝜆𝑇 time step rather than along the whole calibration phase (which we denote as Once Substitution) and substituting the full DCT spectrum rather other a partial frequency band of it (which we refer to as Full Substitution). Results in Fig. 12 show that Once Substitution fails to produce reasonable images, indicating that step-by-step FBS along the whole calibration phase is of crucial importance for smooth information injection and stable information fusion. Since image content is basically formed in the early stage of the diffusion sampling process, removing feature calibration of FBS in the early Therefore, the content of the generated image has already been formed to be basically the same as the reference image after the calibration phase, making it difficult to manipulate image semantics in the latter non-calibration phase.For quantitative evaluation, we separately evaluate all the methods on the text-driven I2I translation task pursuing image appearance consistency (favoring appearance preservation) and the task pursuing image appearance divergence (favoring large modification of image appearance). For the former task, we assess model performance by measuring structure similarity (↑), perceptual similarity (↑), and style distance (↓) between each pair of the reference and translated image. For the latter task, we assess models' appearance modification and structure-preserving abilities by measuring structure similarity (↑) and style distance (↑) between the reference and the translated image pairs. Besides, CLIP similarity (↑) is evaluated for both two tasks to measure semantic consistency between the target text and the translated image, i.e., the text fidelity of the I2I translation results. Finally, we evaluate the aesthetic score of the translated images with the pre-trained LAION Aesthetics Predictor V2 model. We use the DINO-ViT self-similarity distance [39] as the metric for structure similarity, use LPIPS [44] metric for perceptual similarity, and use AdaIN style loss [14] to measure style discrepancy between the reference and translated image. We perform all the quantitative evaluations on the LAION Aesthetics 6.5+ dataset, we sample 300 evaluation images for each task and manually design 2 editing texts for each image, resulting in 600 evaluation samples per task. The average values of all the metrics are shown in Tab. 1. For evaluation of our method, we use the low-FBS for the I2I task pursuing appearance consistency, and use the high-FBS for the task pursuing appearance divergence. Results shown in the table show that our method achieves top rankings for all the metrics, demonstrating the superiority of our method over related approaches. Among all the compared methods, our method is the only one allowing versatile control over the guiding factors of the reference image to the generated image, and also takes advantage in continuous control over the guiding intensity of the reference image. 
conclusion: 
 This paper proposes a novel plug-and-play module adapting the pretrained text-to-image diffusion model to versatile I2I applications. At the heart of our method is a training-free frequency band substitution layer, which dynamically calibrates the denoising diffusion process of the target image by substituting certain DCT frequency band extracted from the reference image into the reverse sampling diffusion process. We demonstrate that versatile I2I applications can be unified by our method simply by switching among different modes and ranges of the substituted frequency band, realizing effective, flexible, and comprehensive control over the translated images.  
 
 Appendix:  


Decision: This paper proposes a method for high-quality and versatile text-driven image translation which at generation time is done without any model training, model fine-tuning, or online optimization. Reviewers indicate that there is a comprehensive set of experiments and that both quantitative and qualitative results are very good. There were some initial doubts, but after the rebuttal all reviewers indicate this as a borderline or weak accept.

Reviews: review: paper_summary: paper_summary: This work proposes a plug-and-play and controllable text-driven image2image translation method, FBSDiff, from a frequency-domain perspective. FBSDiff first encodes the reference image and computes the inversion trajectory. Then, it computes the latent maps for both the reconstruction trajectory and the sampling trajectory. By leveraging the frequency band substitution layer, FBSDiff calibrates sampling latent maps with different frequency band extractions corresponding to different controls of appearance, layout, and contour. Comprehensive experiments show the effectiveness of the method.
strength_weakness: strength_weakness: - The method is training or finetuning free.
- Comprehensive experiments have been done.
- FBSDiff can do large semantic gap translation.
limitations: limitations: - Though the method is training free, it requires computing multiple trajectories. The computing cost is not really low. 
- Authors claim different frequency bands correspond to different controls, i.e. low frequency to appearance and layout, high frequency to contour, and mid frequency to layout. The correspondence is not very clear. More results should be shown to make it convincing. 
- Due to the aforementioned shortages, it seems several trials on each image should be conducted to achieve the best translation result, which limits the application of this method. 
- The performance of FBSDiss is not obviously superior to other methods.
paper_summary: paper_summary: This paper introduces a novel approach that utilizes various frequency bands of diffusion features in the DCT spectral space, enabling dynamic frequency band substitution to guide image generation. This technique realizes highly controllable and versatile text-driven image-to-image (I2I) translation.
strength_weakness: strength_weakness: 1. The introduction of the Non-Calibration phase and FBS based on the PAP method further enhances controllability.
2. The proposal to use different frequency bands to distinguish different types of features is innovative.
3. Both quantitative and qualitative comparisons demonstrate impressive results.
limitations: limitations: 1. The quantitative comparison lacks the FID metric, which is commonly used to measure image fidelity in text-to-image methods.
2. The introduction of the Non-Calibration phase adds an additional tunable hyperparameter \(\lambda\) compared to the PAP method, which seems to reduce ease of use during generation.
3. Line 430 claims that the low-frequency band can control color, but the experiments in Figure 5 show that the color did not change in the Low-FBS.
paper_summary: paper_summary: This  paper proposes a simple training-free method to advance controllability in Text-to-Image (T2I) diffusion models, by substituting frequency bands in the DCT spectral space, enabling the dynamic replacement of frequency features with those of a reference image. Specifically, the reference image undergoes inverse diffusion and diffusion processes to obtain the frequency features for substitution. Experimental comparisons demonstrate superior performance in image translation quality and versatility.
strength_weakness: strength_weakness: 1. The proposed frequency band substitution framework is novel for text-conditioned I2I translation. 
2. The authors provide extensive qualitative and quantitative experiment results to support the effectiveness of their method. 
3. The writing is clear and easy-to-follow.
limitations: limitations: 1. Could the authors provide analysis and empirical results of the capability limit of modifying frequency bands? While frequency domain is useful for handling structural level information, it may be inferior in terms of semantic information, which seems to me the major limitation of this work,. 
2. The inversion and reconstruction processes of the reference image seem to introduce extra computation burden. Could the authors also provide comparison of training and inference costs with other methods? 
3. It would be more convincing to see more quantitative ablation results on the hyperparameters to define low, mid, and high level frequency bands. Is the performance of FBSDiff robust to different frequency band settings?