One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting
Authors: Haipeng Liu
Yang Wang
Meng Wang
Venue: NeurIPS 2025 poster ()
Keywords: text-guided image inpainting
image editing
diffusion models
Primary area: deep_learning

Abstract: Text-guided image inpainting aims at reconstructing the masked regions as per text prompts, where the longstanding challenges lie in the preservation for unmasked regions, while achieving the semantics consistency between unmasked and inpainted masked regions. Previous arts failed to address both of them, always with either of them to be remedied. Such facts, as we observed, stem from the entanglement of the hybrid (e.g., mid-and-low) frequency bands that encode varied image properties, which exhibit different robustness to text prompts during the denoising process. In this paper, we propose a null-text-null frequency-aware diffusion models, dubbed NTN-Diff, for text-guided image inpainting, by decomposing the semantics consistency across masked and unmasked regions into the consistencies as per each frequency band, while preserving the unmasked regions, to circumvent two challenges in a row. Based on the diffusion process, we further divide the denoising process into early (high-level noise) and late (low-level noise) stages, where the mid-and-low frequency bands are disentangled during the denoising process. As observed, the stable mid-frequency band is progressively denoised to be semantically aligned during text-guided denoising process, which, meanwhile, serves as the guidance to the null-text denoising process to denoise low-frequency band for the masked regions, followed by a subsequent text-guided denoising process at late stage, to achieve the semantics consistency for mid-and-low frequency bands across masked and unmasked regions, while preserve the unmasked regions. Extensive experiments validate the superiority of NTN-Diff over the state-of-the-art diffusion models to text-guided diffusion models. Our code can be accessed from https://github.com/htyjers/NTN-Diff.

One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting
Haipeng Liu' Yang Wang Meng Wang
School of Computer Science and Information Engineering Hefei University of Technology, China
hpliu._fuu@hhommil... com
yangwangöhfut. edu. rzic.mengrangggmail.comm
Abstract
Text-guided image inpainting aims at reconstructing the masked regions as per text prompts, where the longstanding challenges lie in the preservation for un- masked regions, while achieving the semantics consistency between unmasked and inpainted masked regions. Previous arts failed to address both of them, always with either them to remedied. Such facts, as we observed, stem from the entangle- ment of the hybrid (e.g.. mid-and-low frequency bands that encode varied image properties, which exhibit different robustness to text prompts during the denois- ing process. In this paper, propose null-text-null frequency-aware diffusion models, dubbed NTN-Diff, for text-guided image inpainting, by decomposing the semantics consistency across masked and unmasked regions into the consistencies as per each frequency band, while preserving the unmasked regions, to circumvent challenges in row. Based on the diffusion process, we further divide the denoising process into early (high-level noise) and late (low-level noise) stages, where the mid-and-low frequency bands are disentangled during the denoising process. As observed, the stable mid-frequee fri-ffreuency band is progressively denoised to semantically aligned during text-guided denoising process, which, meanwhile, serves as the guidance to the null-text denoising process to denoise low-frequency band for the masked regions, followed by a subsequent text- text-guided denoising pro- cess at late stage, to achieve the semantics consistency for mid-and-low frequency bands across masked and unmasked regions, while preserve the unmasked regions. Extensive experiments validate the superiority of NTN-Diff over the state-of-the-art diffusion models to text-guided diffusion models. Our code can be accessed from bttpps:/gitte cattuub..om//ttt// www/ttyjerss...ee.o NTN- cot.e///r/r/eee////DDtD
Introduction
Traditional image inpainting methods 23
46
19
primarily rely on unmasked regions
as guidance to recover masked areas, which are widely applied to object removal [8
45 28
and
photo restoration [2] 2 (7). Based on that, text-guided image inpainting 35 47
36 has
achieved remarkable progress, offering more flexible and diverse approach. These methods focus on reconstructing user-specified objects within masked regions given text prompts.
The success of text- eext-g image inpainting is largely attributed to the powerful generative capabilities of Denoising Diffusion Probabilistic Models (DDPMs) [9] e.g., Stable Diffusion [3). As a critical
Equal contribution
Yang Wang is the corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

with ungin
foxt tecege
boling alowr
get witk
Soxasio boonittayy nasked and seansked II
wiits visu"
Medied kaag
Exeunded reginss deepte wiggeeiiiggnnneooooo priprepe
Macked llegge and Pronet
NTW DeT (Dha
(C Cueguraes of www.erccsssescccccedacceeeoeeeee with Dh ualiodiehears
Figure I: (a-c) 1-SNE visualization of CLIP latent space evolution during text-guided denoising. At each step, the Euclidean distance between the text prompt and the denoised image (Denoised Image-Texs Text Distance) reflects semantic consistency across masked and unmasked regions- smaller value indicates better alignment. As a reference, we include the distance between the ground truth and the text prompt (red dashed line). To assess unmasked region preservation, we also compute the distance between the denoised image and ground truth (Denoised Image-GT Distance). (d) Comparison between our NTN-Diff (Fig 3 and -tate-of-theearts
12
for text- text-guided inpainting;
subtask in the broader field of image editing 20 8 distinguishes itself from other subtasks by requiring the masked regions to be generated as per text prompt, while preserve the unmasked regions. Upon the alignment between the generated content for masked regions and the text prompts, the longstanding challenges lie in the preservation for unmasked regions, while achieving the semantics consistency between unmasked and masked regions as inpainted.
15 22
text-guided image inpainting
To circumvent the above two challenges, series of text-guided diffusion models 42
35
have been proposed for image inpainting. Specifically, BLD [T] implements a blending operation during the denoising process at each step to preserve the unmasked regions, while Smartbrush [42 adopts multi-task based denoising process conditioned on both text and shape for each step, where the denoised unmasked region is substituted by unmasked regions from ground-truth. Despite the unmasked preservation, as illustrated in Fig [a), they still suffer from the semantics inconsistency between with masked and unmasked regions, owing to the discrepancy from the diffusion process and the inpainted maksed regions from the text-guided denoising process.
26 49 on preserving the unmasked regions while ensuring the semantics consistency between masked and unmasked regions. For instance, CAT 4 utilizes the text prompts and unmasked regions in the CLIP latent space to form latent code-level semantics consistency constraints for image inpainting, which, however, suffer from the information loss for unmasked regions. BrushNet 12 generates dense texture feature map during the denoising process with no cross-attention in UNet 34 to inpaint masked regions by unmasked information for consistency. However, illustrated in Fig [b),
Orthogonally, great efforts have been spent 31
17
[b) the unmasked regions fail to be preserved, which is incurred by the other text-guided denoising process to inpaint masked regions.
So far, the prior arts still fail to simultaneously overcome the above two challenges. Such failure, as investigated in 2ig stems from the following hurdle: as disclosed by (29, the low-frequency band fluctuates more significantly during the early stage of the denoising process with high-level noise than the mid-and-high frequency bands, since diffusion models typically recover the low-frequency band first and progressively refine the mid-and-high frequency band?[ more results in Sec|A of the Appendix). Therefore, the low-frequency band for both masked and unmasked regions are easy to be modulated by text prompts, as illustrated in Fig F mid-frequency band across all regions is robust to the text prompts while aligns well with text prompts, which may better preserve the unmasked regions than low-frequency band upon text prompts. The above observations further motivate us to decompose the semantics consistency across masked and unmasked regions into the individual task of achieving that as per each frequency band across masked
a). To be contrary, as seen in Fig 2b). the
Coomprred to the mid frequency band, the gigh-frequency band is sparser and more easy to be disrupted by high-level noise in the early stages, resulting in much less contribution on guiding the low- frequency band than mid- frequency band. Therefore, we focus on mid-and-a lnd-low frequency bands.

---

Text Promet "laa.. business surt"
Earty Stage
Lake Singe
Step
Stop
Stap
Mtop 55
Stop
Stip
Step
Sego
Trepaced
Lee
Masked Enaage
Expainted Ensage
Treeolod low frequeery restiguited donelinn
OCT
Frog no
Layoer
Eepaiited lmsage
NTN-DI
Figure 2: We investigate the text-guided denoising process for both the (a) low- www-rrquenccy and (b) mid- frequeeyyynyy bands. For each step, we employ the red bounding boxes to highlight the variations for the now-frequency band during the late stage in (a) and the blue bounding boxes to visualize the variations for layout information (b). For DCT spectrum, the top-left region represents low frequencies, with the bottom-right region corresponds to high frequencies. The dark red and yellow indicate the highest and lowest value.
and unmasked regions, while preserving the unmasked regions, to tackle two challenges in a row. To this end, we formally delve into the following questions: How disentangle different frequency bands, particularly the early stage of the denoising process with high-level noise? and How to exploit the hybrid frequency bands for diffusion models simultaneously achieve the above two goals?
Inspired by the recent progress on frequency- domain based diffusion models 39 14 10, we answer the above questions by proposing frequency- null-text-null diffusion models, named NTN- Diff (see Fig.),, which comprises early and late stages for text-guided image inpainting. For the early stage, 1) first propose null-text denoising process 2. to avoid the low-frequency band is influenced by text prompts under the high-level noise, then replace its denoised result with unmasked regions from the forward diffused results. parallel, 2) we perform the second texi-guided denoising process (Sec 2.3.2) to denoise the masked regions, especially its mid-frequency band to be aligned with text prompts across both masked and unmasked regions, while replace the low-frequency band from the denoised output with that from the above null-text denoising process, so that the low-frequency can be preserved without being disturbed by text prompts; 3) we further exploit the above denoised mid- mid-ffreuencc to guide the last null-text denoising process (Sec2.3.3) by substituting mid-frequency band from this null-text denoising process, while denoising the low-frequency band especially for masked regions, to attempt the complete alignment with text prompts.
Based on the above, during the late stage, we perform the text-guided denoising process (Sec (Sec2.4p logether with the unmasked regions substitution from the early stage of the diffusion process for unmasked regions preservations for each step, while achieve the semantics consistency between mid-and-low frequency bands across both masked and unmasked regions conditioned on text prompts, to well resolve the two challenges, leading to final inpainted output in Fig(T(c). Fig [d) further validates the intuition of NTN-Diff, which not only can achieve better semantics consistency between masked and unmasked regions than BLD [ but also preserves the unmasked regions to outperform BrushNet [1], upon the alignment with text prompts.
The above observations further validate the intuitions of NTN-Diff by resolving the above two challenges, i.e., semantics consistency between masked and unmasked regions and unmasked regions preservation, for the text-guided image inpainting via diffusion models. The extensive experimental results validate its superiority over the state-of-the-arts for text-guided image inpainting.
2 Methodology
Central to our proposed NTN-Diff lies in the following aspects: 1) the motivation of hybrid frequency aware diffusion models to text-guided image inpainting (Sec(2.2); 2) how to disentangle different frequency bands and exploit the hybrid frequency bands for diffusion models the early stage (Sec 3) how to achieve unmasked regions preservation and semantics consistency with masked regions in the late stage (See 2.4g before shedding light on NTN-Diff, we elaborate the preliminaries regarding DDPMs, which plays pivotal role of text-guided image inpainting.
3

---

Unail Low-
Deasiling Presees
Drwwnding Preres
CHIP
Mike
Figure 3: Illustration of our proposed NTN-Dii pipeline, which comprises a (I) null-text denoising process (2.3T] to avoid being influenced by text prompts, and a (II) text- rext-guided denoising process (S.22.3.2 to denoise the masked regions, while replacing the ow-frequency band from the denoised output with that from the above null-text denoising process. Building on this, we further utilize the denoised mid- uid-freeuency to guide another (III) null-text denoising process (Sec(2.3.3) by substituting the mid- fererrruyency band from this process. Additionally,
(IV) late-stage text-guided denoising process (See 2.4) is performed, along with the substitution of unmasked regions from the early stage of the diffusion process, to preserve unmasked regions at each step.
2.1 Preliminaries
Given a completed image Let E R text prompt c, and a binary mask M E 13 such that 0 denotes masked regions and denotes unmasked regions, The goal of text-guided image inpainting is to restore the missing regions in a masked image Lm = Lgt C M, so that the reconstructed content aligns with the text prompts to precisely indicate where and what to be inpainted. To this end, state-of-the- tftt------s--sssee-ee----------s predominantly leverage pre- preitreined text-to-image latent diffusion frameworks, mainly for Stable Diffusion [33 (SD) as their foundation, where Variational Auto-Encoder (VAE) [33 is employed via encoder E to transform the image Tgt from pixel space to latent space 2 RAX # W to reduce computational complexity without compromising visual quality. The model then performs the forward diffusion process and denoising process in the latent space:
Forward Diffusion Process. In the forward diffusion process with T timesteps, following the typical Denoising Diffusion Probabilistic Model (DDPM) (9), Stable Diffusion adds Gaussian noise
e ~ (0, f) to convert the clean sample 2g = gt into noisy sample zT. For any timestep E [0, the forward process 2t is then defined as:
ze(20, c) = Vōezo + v1 ãee,
(1)
where ar denotes the corresponding noise level.
Reverse Denoising Process. During the reverse denoising process, Stable Diffusion is trained to estimate the noise added to the noisy image, conditioned on c, and progressively removes it over T timesteps. Given an input noise 2T sampled from a random Gaussian distribution, the training objective of the denoising network (UNet (34) ca at the t-th timestep is formulated as:
(2) text encoder, and |la denotes the la norm. After T timesteps, the denoised latent representation z/ is reconstructed into pixel space via the decoder D to generate the final result Igen-
LLDM 0200) []oo 4, To - =l12.
where (-)-) is the CLIP 30
2.2 Motivation: Hybrid Frequency Aware Diffusion Models to Text-Guided Image Inpainting
Before shedding light on our pipeline, upon the diffusion process as per [q. we investigate the text- guided denoising process for both the denoised low-frequency (dense texture [7] and mid-frequency (extracted from the Discrete Cosine Transform spectrum via mid-pass mask 55) information. The compared results are shown in Fig while observe the followings:
a), (1) As shown in Fig a), the low-frequency band can be easily changed by text prompts, especially the early stage with high-level noise, such trend keeps on even into nearly the middle (e.g., 45-th step) of the late stage.

---

(2) Fig b) illustrates the text- sext-guided denoising results for mid- mid-frequency band. To validate its stability, we visualize the layout information as bounding box for each step (2nd row), which, as indicated by S
is closely related to the mid-frequency band. Akin to Fa) the denoised mid mid-freuuency band (3rd row) also changes during the initial early stage owing to text prompts with high-level noise, yet quickly converges by the end (e.g., 65-th step) of the early stage, which further leads to the stable mid-frequency band with nearly no influence by text prompt across the whole late stage with low-level noise.
6 4
Capitalizing on the discoveries, instead of low-frequency, we propose to exploit the mid-frequency band during denoising process, which plays the pivotal role of achieving the semantics consistency upon text prompts, while can better achieve the semantics consistency between masked and unmasked regions, while preserve its own frequency band well, than low-frequency band, owing its robustness to the text prompt during the denoising process. The above observations further motivate us with novel frequency-aware null-text-null diffusion models, named NTN-Diff, for text-guided image inpainting. Our pipeline is illustrated in Fig.3] where we discuss NTN-Diff within the early and late stages, demarcated by the critical step X7 with as the total steps for diffusion models; the detailed discussions for X can be seen in Sec 3.3.2 (See more intuitions on hybrid frequency aware diffusion models text-guided image inpainting in Sf the Appendix)
2.3 Early Stage for Null-Text-Null Frequency-Aware Diffusion Models
Based on the diffusion process regarding Eq. we discuss the early stage of text-guided denoising process. Unlike the typical models [4] I with text-guided denoising process, 1) we propose a null-text denoising process (Sec conditioned on null-text prompt at the t-th step avoid being influenced by text prompts even under the high-level noise, focusing primarily on low-frequency band, we then replace its denoised result with the unmasked regions from the forward diffused results within the (T' t)-th step. In parallel, 2) we perform the second text guided denoising process (Sec (Sec(.3..2) for f-th step. to denoise the masked regions, while replace the low-frequency band (Fig a)) from the denoised output with that from the above null-text denoising process, so that low- low-rrequency band (e.g., color and illumination) especially for unmasked regions can preserved even under text prompts, while the mid-frequend frequency band over both masked and unmasked regions varies owing to text prompts, yet struggling to be semantically consistent to its low-frequency band across the whole regions, resulting in semantic inconsistency between masked and unmasked regions (see 6
The denoised mid-frequency band well aligns with the text prompts especially for masked regions, while also encodes the information related to low-frequency band from the above null-text denoising process (See ).3.3). which is further exploited to guide 3) the last null-text denoising process, 3.1) achieve semantics consistency between lnd-low frequency bands across masked and unmasked regions, by denoising the low-frequency band throughout the path to be semantically consistent to mid-frequency band, with no influence from text prompts (see Fig[[a)): 3.2) such intuition can further restore low-frequency band especially for masked regions, to be aligned with text prompts, while ready to preserve both mid-and-low frequency bands for unmasked regions, along with its semantics consistency to masked regions via a late stage text-guided denoising process in See 2.4
2.3.1 Null-Text Now-Frequency Aware Denoising Process
Formally, we initialize the denoising process with random Gaussian noise 27Mh E T and null-text cg as the input to the denoising network c0(27m T, To (cg)) for null-text denoising process. To preserve the unmasked regions, at each step t, we replace the unmasked regions of denoising result =;n with those from the diffusion result 2qt_, at the (T f)-th step to yield sun, as per the following:
m. + 2." ① (1 m=))
(3)
where m. E 12Ixx 11 is obtained by downsampling the mask M, we replicaate it along four channels for 2qth, and = @ is the element-wise multiplication. Following the above, we reformulate the self-attention mechanism of the denoising network by incorporating a mask to suppress the attention scores of the masked regions, to ensure that the inpainting process primarily relies on the unmasked regions. Given the denoised latent image feature Fi E x x at the 1-th layer of denoising UNet architecture e0 =e(" ,t, (ca)) at the t-th timesteps, the modified self-attention mechanism is
5

---

IRJEET
ge
(3) Brashaed Low- Frequency Rund liyer
(b) Densked Mid Frequency Riad loyer
Figure 4: Illustration of (a) denoised uww-freueeccy band layer and (b) mri-frequency band layer.
defined as follows:
(FiWa;) (Ei O m)
Fi+1 = softmax
F.Wyy
(4)
ddi
where my E Rhixxu is the downsampled result of the mask M, replicate it along di channels for Fú, Wo; E Rdxxd-,, Wx; E IR4 and W Rdixdi+1 are learnable linear projection matrices, with di representing the embedding dimensions.
2.3.2 Text-Guided Denoising Process
To inpaint the masked regions per text prompts, resort to the text- sat-guuied denoising process, to- gether with wwwffreuency band substitution from null-text denoising process (See12.3.I). Specifically, we initialize the denoising process with random Gaussian noise 2/ E R2x x
and text prompts the input to the denoising network eø e0(2tert, T, (a(c)) for text-guided denoising process for masked regions. As inspired by Fig we preserve the unmasked regions as per their low-frequency band, since, instead of mid-frequency or even high-frequency band, such information can be easily modulated by text prompt. To this end, we propose plug-and-play Denoised I. Low-Frequency Band layer (Fig a)) to substitute the latent representation zfeat within the text-guided denoising process, by 2t from the null-text denoising process (see Sec 2.3.1 for the 1-th step to yield gteat, which is formulated as:
(5) where DCT(-) represents 2D Discrete Cosine Transform, IDCT(.) represents 2D Inverse DCT, and Mime denotes the low-pass mask to isolate low-frequency band, the denoised frequeeu band layer employs a binary mask defined by the sum of the 2D coordinates as threshold, as follows:
Jezt IDCT (DCT(>?")) mio DCT(2
Miow)) ))
1, i y thip
Molo (a. w) =
(6)
0. otherwise
where the, is the threshold for low-pass filtering. To adaptively extract the low-frequency band from masked images, set thi follows:
||
thip Xop + Xip HW
(7)
where Mip and Xrp are sapee-pprammters.. The & norm || represents the number of entry with value M. We set thip to correlate with the ratio of unmasked regions, since the larger unmasked regions require more low-frequency band from the null-text denoising process to substitute the low-frequency band within the text- text-guided denoising process.
Remark 1. During this process, the low-fie wwwfrequency band across both masked and unmasked regions are preserved even under text prompts owing to the substitutions from the null-text denoising process, with only mid-frequency aligned with text prompts for both regions, hence still failed to achieve the consistency between masked and unmasked regions. To address this, we propose another concurrent null-text denoising process, guided by above denoised mid-level frequency, to help denoise the low-frequency band for both regions, especially for masked regions.
2.3.3 Null-Text Mid-Frequency Aware Denoising Process
We resort to another null-text denoising process guided by the denoised mid-frequency band from the second eoxt-guided denoising process 2.3.2) for the t-th step of the early stage. Particularly, initialize the denoising process with random Gaussian noise zir [B4x x and null-text og as input to the denoising network (2 T, 7a (co)). We further exploit the denoised mid-fre hid-frequency result from 2 to substitute the denoised mid-frequency band of in from the last null-text denoising

---

process, throughout a plug-and-pe and-play Denoised Mid- Mid-Frequency Band layer (as illustrated in Fig to yield ztr, which is formulated below:
b))
IDCT (DCT(E4"")) ① Meid (C" о (1 mmmis)
(8)
where miiid denotes the mid mid-pass mask. Analogous to Eq. (6), we define:
1,
if Eltmpl <x + S thmp2
Mamil (x, y)
(9)
(0
otherwise
where l/mpp and thmp are the thresholds for mid-pass filtering to extract mid- mid-frequency band. To adaptively extract the mid-frequency band from masked images, we set thmmpl and thmp2 as follows:
MM
MM HW
thmpl =
Xmaps Xipp
HW thmpp = Xinya + Xinpa
(10)
where Xm Ampt, N and Nayz are arper-parameters, such that thmy2 > thmpp1 Similar as Eq. (7), we also set the lhmpl and thmp? to correlate with the ratio of unmasked regions. Since the larger unmasked regions require more mid-frequency band to encode the information from the low-frequency band substituted from the first null-text denoising process.
Remark 2. The above null-text mid-frequene band for the masked regions to be aligned with text prompt, owing the denoised mid middfrreuency from text-guided process; besides, the denoised mid-and-low frequency bands from unmasked regions also help denoising process for masked regions, which, however, cannot be served as the final text- rext-guided inpainting output for masked regions, as the denoised and--nndloww frequency band is not semantically strong text prompt. To address that, a text- suided denoising process is performed the late stage of denoising process, as discussed below.
guided denoised process can denoise the low- frequeeyy
2.4 Late Stage of Text-Guided Denoising Process
Following the discussion above, for final late stage of denoising process, we perform the text- guided denoising process based on the early null-text mid-frequency guided denoising process (Sec 2.3.3). the low-frequency band within the masked regions is denoised throughout the path to text prompts alignment, together with the stable mid- mud-frequency band (see Fig b)) from the early text- text-guided guided denoising process (Sec(23.22) the masked regions are well inpainted. However, the low-frequency band for unmasked regions biases lot from the ground truth from the early null-text denoising process (..3),.3.3) and augmented by the influence from the text prompts for late stage. To this end, we substitute the unmasked regions from early stage of the diffusion process into the denoising results 2 for the t-th step during the late stage to yield 2im which formulated as:
= m, 2 @ (1 -ms);
(11)
where = donates the forward diffused results with the (T - t)-th step. Building upon the noisy sample 2/" the text prompt along with the step t. we perform the denoising network eø ( f, (c)) to get the denoising result = in the latent space for the next (t 1)-th step. After XT steps, the denoised latent representation zi" is reconstructed into the pixel space via the VAE decoder D to generate the final image inpainting output. (See the pseudo algorithm in Sec.Clof the Appendix) Remark 3 : Last question on Semanties Consistency is answered! Although the alignment between mid-and-low frequency bands from masked regions and text prompts, together with un- masked regions preservation, can be achieved, one last question is how the denoised results within this late stage can achieve semantics consistency between masked and unmasked regions? The crucial intuition lies in the denoised mid miid-freuency band from the early text-guided denoising process (Sec (.322.3.2). which, based on Remark 2, also encodes the information from the low- wwwffreqency band substituted from the null-text denoising process to match diffusion process (ground truth) including both masked and unmasked regions, hence the fow-frequency band under mid- nreuueryy guidance for masked regions also achieves the consistency to the substituted unmasked regions from the diffusion process, since the mid-frequency band is stable during the late stage of the denosing process even under text prompts (see Fig bi)).
With all frequency bands to be semantically consistent across all regions conditioned on text prompts, the semantics consistency between masked and unmasked are well achieved, together with the success of the preservation for unmasked regions!
7

---

Table I: Quantitative comparisons between NIN- (*) and other diffusion- based inpainting models over BrushBench for inpainting and outpainting are shown, where all models use Stable Diffusion V1.5 as the baseline model. 1: higher is better; J: lower is better, red and blue stand for the best and second best result. * with blending operation of BrushNet. NTN-Diff (+) achieves the best result.
Quality
Provrvatio
x200"
x197
BED
TOO1 35
9.75
25.81
21:33
0.76
10.33
26.15
ChifEs
100V 25
75
12:35
13.18
1..47 243
BreskSr
17.40
31.55
9,31
26.45
Icudee
K2R 15
11:08
76.90
23.63
4.55
43.50
36.1
NEN-Dit
12:45
27.53
25.55
0.50
01.35
26.54
CNT (2)
20C9 05
25.32
24.33
25.12
NEN-DUP (Quar)
12.09
40.70
0.11
0.50
26.49
100 25
181
26.33
15,85
35.55
21.40
26.35
CN
10CV 25
9.26
27.55
11:30
53.03
56.15
27.29
P0Y5-1 31
75.01
15.01
11.18
15:13
36.3
Dasdde
K218
15.01
22:48
15.33
26.56
Iepiieding
11.51
28.33
18.17
20.41
18.46
27.54
UNP
17:00
31.0
3012
NEN-UHP (Ureo
26.36
31,85
27.30
3 Experiments
3.1 Implementation Details
We evaluate NTN-Diff on two benchmarks tailored for text-guided image inpainting in diffusion models: BrushBench [2] comprises 600 images with human- annotated masks and captions. The dataset balances natural and artificial images (e.g., paintings) and equally distributes across categories (humans, animals, indoor/outdoor scenarios), ensuring fair and equitable evaluation, BrushBench refines the task by considering two specific scenarios: segmentation mask inside-inpainting and segmentation mask outside-inpainting: EditBench [35] features 240 images with an equal split between natural and generated images, each accompanied by mask and caption annotations. Following [12], we utilize a fine-tuned version of Stable Diffusion v1.5 as the base model, configured with 100 sampling steps to ensure high-quality outputs. All experiments are implemented in Py PyTorch and run on a single NVIDIA RTX 3090.
We evaluate inpainted results using six metrics across three aspects. For image generation quality, we employ Image Reward (IR) 44 to model human preferences for image quality and HPS V2 40 that combines perceptual studies with deep learning for comprehensive visual and semantic assessment. Both metrics measure the semantics consistency of the high-quality image generation aligned with human standards; for masked region preservation, we use PSNR and MSE to measure pixel-wise differences from ground truth, along with LPIPS [48) which evaluates perceptual similarity through deep feature distances; finally, for text alignment, CLIP Score (38] quantifies text-image consistency by comparing their embeddings in CLIP's s shared space.
3.2 Comparison with State-of-the-arts
To validate the superiority of NTN-Diff, we conduct a comprehensive comparison with state-of-the-art text-guided image inpainting diffusion models, including HDP (25] and BLD [I] which implement a blending operation during the denoising process at each step to preserve the unmasked regions, while suffer from the semantics inconsistency between with masked and unmasked regions, owing to the discrepancy between the diffusion process for unmasked regions substitution and the denoising process for masked region alignment with text prompt. CNI (4). PP [49 and BrushNet (12) focus on the ensuring the semantics consistency between masked and unmasked regions. Particularly, BrushNet [12 generates dense texture feature map during the denoising process with cross- attention in UNet [34), to inpaint masked regions by unmasked information for consistency, while unmasked regions are disrupted owing to another text-guided denoising process.
Quantitative and Qualitative analysis. The quantitative results in Table D summarriz our findings below: NTN-Diff enjoys larger IR, HPS v2, PSNR and CLIP Score, together with smaller MSE and LPIPS than the competitors. Notably, BrushNet remains the large performance margins (at most 0.73% for IR, 0.62% for HPS, 8.59% for PSNR, 30. 18% for MSE, 15.51% for LPIPS and 0.23% for CLIP Score) compared to NTN- in Table We also present the quantitative results of NTN- with the pixel-level blending operation of 12 named NTN-DifY to preserve the unmasked regions,
8

---

parnding
panding of leainng on mechine
cartion cat sliting tue af luaking ciy
pakding Square lrtter clas Blar
welwan piui jacket and fivee sakking on Janveg pask
the paintig of mniiive wannley serving fee of concering
witmaan with ving drowee hair poiing for portrott
pingglie on dack with few gadee the Beskaround
Taagg 31.0111
(13)
Own
Masiod Teagge
BraskBeach
(k) EditBbach
Figure 5: Comparison of the text-guided inpainted results with the sfate-of-the-art on BrushBench 12
and EditBench [35]. NTN-Diff delivers the superior inpainted results over others.
Table 2: Ablation studies on three denoising pro- cesses: Case A (Sec2.3.1) D. Case B (Sec2.3.2) and Case C (Sec(2.3.3) for the early stage, based on the same text-guided denoising pro- cess (Sec (.4) for the late stage. red and blue stand for the best and second best result.
Table 3: Ablation study about the hyperparame- ter sensitivity analysis of A. red and blue stand for the best and second best result. With À 0.6 balancing early and late stage, the best perfor- mance of image quality, unmasked region preser- vation and text alignment are achieved.
Menic
Ciee
Ciee #
Cae
Oves
Cae Cae
Case
Oars
Mark
BrankBoohs AT
46
1R
2441
52.5g
214
25.5
38.0
12.31
10.0
241
243
276
10.0
B1A3 9.55
34TH 8:58
11:3
13.75
LE3PS
25.00
2832
23.44
26.11
6434
43/85
4482
44.09
2810
28.6
4645
36:38
26.45
46.66
4508
CLIP Sooeet
36.0
36.33
38.00
3645
25.55
3578
36.00
36.09
35.99
28.06
29:00
3658
26A8
22.97
2601
1500
2006
25:91
which demonstrates the ability to tame the hybrid frequency issue, the results further verifies the intuition in S achieves the semantics consistency between mid-and-low frequency bands across masked and unmasked regions, while preserving unmasked regions. We illustrate the above intuitions in 5 (See more analysis of computational efficiency in De the Appendix, along with more compared results for Editbench [35] can be seen in Sec[E the Appendix)
3.3 Ablation Studies
3.3.1
Discussion on Different Denoising Processes of the Early Stage
To validate of our NTN-Diff (Sec we perform the ablation study on BrushBench and EditBench datasets with three variants: Case A: removing masked self- atffntttnnion in the first null-text denoising process (See 2.3.1.11 Case B: removing both null-text and text- text-guided denoising process (Sec 2.3.2). Case C: removing the null-text denoising process (Sec 2.3.3; Table Saalerreprrttt the results for all three cases, which suggests that NTN-Diff outperforms Case A. confirming that the null-text denoising process conditioned on null-texi prompt can avoid being influenced by text prompts even under the high-level noise, focusing primarily on tow-frequency band, which in line with See 1.3.. Case B exhibits a suboptimal performance degradation, despite the null-text denoising process in the early stage, together with the text-guided denoising process, which verifies that the null-text mid- mid-frequency guided denoised process can denoise the low-frequency band for the masked regions to be aligned with text prompt, owing to the denoised mid-frequent from text-guides process, hence validating the important of text- (ext-guided process (Sec 2.3.2); NTN-Diff maintains achieves the large performance gain over Case C (at most 44.8%), implying that only mid-free hid-frequency aligned with text prompts for masked and unmasked regions, still failed to achieve the consistency across the whole regions, which is in line with See 2.3.3 and illustrated in Figl6] (See more generation results for the impact of the text and null-text prompts on low-and-mid frequency bands in Fo the Appendix).
3.3.2 Hyperparameter Sensitivity Analysis of X for the Length of the Early and Late Stage
We analyze the parameter À 2.22 that divides the denoising process into early and late stages at step XT, we set the X values to (0.9; 0.8; 0.7: 0.6; 0.5). As reported in Table when À = 0.9, the performance the worst with the shortest early stage. When À = 0.6 with the balance of early and late stage, the best performance are achieved, confirming that the desirable length of carly stage

---

close up boking to
the painting of yellow dag next table with cae
rabbit with white and brown. face aitting the grosnd
the sketck tlax stage singing, standing it back prns.
Image
Imagy
Figure 6: The inpainted output about the different denoising process for the early stage, NTN-Diff delivers the better inpainted results (marked as the red box) than others. Case A fails to preserve the unmasked region in the null-text denoising process (Sec 2.3.1. such as the background of dog (4th row); without text-guided denoising process (Sec /Secc2.3.2 (2..2) Case cannot inpaint the masked region as per the text prompts, such as the disappeared dog in the 3rd row of Case B; specifically, Case C cannot achieve the semantics consistency between masked and unmasked region, owing to the null-text denoising process (Sec 2.3.3), such the tiny rabbit within the rabbit head-shaped mask.
dovll strawberriss and Hutberriis striped toble
two tents in the desert with car parked in the background
cate ow! sitting franch with leaves
with big smuls its fnce
Masked
Masked
Image
Case
(NTN-Dim
Image
(NTN-DiM)
Figure 7: The inpainted output about the impact of adaptively extracting the low-and-mid frequency bands, NTN-Diff can achieve the better inpainted results (marked as the box) than others.
can make the low-frequency band under the mid-frequency guidance for masked regions achieve the consistency to the substituted unmasked regions for ground truth, which is consistent Sec 2.4 (See more generation results in Gf the Appendix).
3.3.3 More Ablation Studies on the Impact of Adaptively Extracting the Low-and-Mid Frequency Bands
As mentioned in Sec 2.3.2 and See 2.3.3 we set thip Eq. and l/mpp and thmy2 in Eq. to correlate with the ratio of unmasked regions. We provide ablation studies on the impact of adaptively extracting low-and-mid frequency bands on the BrushBench and EditBench datasets using three variants: Case adopts fixed thresholds to extract both mid-and-low frequency bands for different images; Case adaptively extracts cow-frequency band while using fixed thresholds for mid- mid-frequency bands; Case III adaptively extracts mid-frequency band while using a fixed threshold for low-frequency extraction. As shown in Fig 7, due to the use of fixed thresholds for extracting mid- uid-frequeney band, Case and Case II inevitably generate inconsistent content between masked and unmasked regions (e.g., the smiling cat in the second row of Fig. [ which can be attributed to the fact that the larger unmasked regions require more mid-frequency band to encode the information from the low-frequency band substituted from the first null-text denoising process, as explained in Sec. 2.4 In contrast, Case III shows substantial performance degradation (c.g.. the light yellow color blocks in the bowl in the first row of 7) confirming that larger unmasked regions demand more low-frequency information from the null-text denoising process to replace the low-frequency bands in the text-guided denoising process, which is consistent with the findings in See 2.3.2
4 Conclusion
In this paper, we propose a null-text-null frequency-aware diffusion models, named NIf-Diff. for text-guided image inpainting, by decomposing the semantics consistency across masked and unmasked regions into the consistencies as per each frequency band, while preserving the unmasked regions, to simultaneously address two challenges of unmasked regions preservations, along with its semanties consistency with inpainted masked regions. The extensive experimental results validate the advantages of NTN-Diff over ofat-oof-hhe-rt diffusion models for text-guided image inpainting.
10

---

Acknowledgments This research is supported by National Natural Science Foundation of China (U21A20470, 62172136, 72188101); Institute of Advanced Medicine and Frontier Technology (2023HH001800) and sponsored by CCF-NetEase ThunderFire Innovation Research Funding (NO. CCF-Nettase 202513); The computation is completed on the HPC Platform of Hefei University of Technology.
References
[1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics (TOG), 42(4):1-11, 2023.
[2] Chenjie Cao, Qiaole Dong, and Yanwei Fu. 7its++: Image inpainting by improving the incremental transformer structural priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[3] Haiwei Chen and Yajie Zhao. Don't look into the dark: Latent codes for pluralistic image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7591-7600, 2024.
[4] Yifu Chen, Jingwen Chen, Yingwei Pan. Yehao Li, Ting Yao, Zhineng Chen, and Tao Mei. Improving text-guided object inpainting with semantic pre-inpainting. In European Conference Computer Vision, pages 110-126. Springer, 2024.
[5] Xiang Gao and Jiaying Liu. Fbsdiff: Plug-and-play frequency band substitution of diffusion features for highly controllable text-driven image translation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 4101-4109, 2024.
[6] Xiang Gao, Zhengho Xu, Junhan Zhao, and Jiaying Liu. Frequency-controlle diffusion model for versatile text-guided image-to-image translation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
volume 38, pages
[7] Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and munti-insstruction image editing by attention modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6986-6996 099669962222 2024.
[8] Amir Hertz, Ron Mokady, Jay Tenenhaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to- prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations.
[9] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NIPS), 33:6840-688 1851,6811111 2020.
[10] Linjiang Huang, Rongyao Fang, Aiping Zhang. Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-fre free high-resolution image synthesis. In European Conference on Computer Vision, pages 196 212. Springer, 2024.
[11] Yuzhou Huang, Lianghin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction based image editing with multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8362-8371, 2024.
[12] Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-pil -lay image inpainting model with decomposed dual-branch diffusion. In European Conference on Computer Vision, pages 150-168. Springer, 2024.
[13] Diederik Kingma and Max Welling, auto-eneoding variational bayes. arXiv preprint 2013.
[14] Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, and Chang D Yoo. Flexiedit: Frequency- latent refinement for enhanced non-rigid editing. In European Conference on Computer Vision, pages 363-379. Springer, 2024.
[15] Shanglin Li, Bohan Zeng, Yutang Feng, Sicheng Gao, Xiuhui Liu, Jiaming Liu, Lin Li, Xu Tang, Yao Hu, Jianzhuang Liu, et al. Zone: Zero-shot instruction-guiia guided local editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6254-6266 2024.
[16] Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi. Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 10758-1018 2022.
11

---

[17] Yaowei Li, Yuxuan Bian, Xuan Ju. Zhaoyang Zhang, Junhao Zhuang, Ying Shan. Yuexian Zou, and Qiang Xu. Brushedit: All-in-one image inpainting and editing. arXiv preprint XXiv:2412.10316, 2024.
[18] Haipeng Liu, Yang Wang, Biao Qian, Meng Wang, and Yong Rui. Structure matters: Tackling the semantic discrepancy in diffusion models for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8038-8047, June 2024.
[19] Haipeng Liu, Yang Wang, Meng Wang, and Yong Rui. Delving globally into texture and structure for image inpainting, In Proceedings of the 30th ACM International Conference on Multimedia, pages 1270-1278, 2022.
[20] Heng Liu, Guanghui Li, Mingqi Gao, Xiantong Zhen, Feng Zheng, and Yang Wang. Few-shot referring video single-and multi-object segmentation via cross-modal aflinity with instance sequence matching. International Journal of Computer Vision, 133:5610-5628 2025.
[21] Qiankun Liu, Yuqi Jiang. Zhentao Tan, Dongdong Chen, Ying Fu, Qi Chu, Gang Hua, and Nenghai Yu. Transformer based pluralistic image completion with reduced information loss. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10);6652 6668, 2024.
[22] Jingyi Lu, Xinghui Li, and Kai Han. Regiondrag: Fast region- based image editing with diffusion models. In European Conference on Computer Vision, pages 231-246. Springer, 2024.
[23] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. RePaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11461-11471, 2022.
[24] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mean-reverting stochastic differential equations, arXiv preprint 111,, 2023.
[25] Hayk Mamukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Hd-painter: High-rcsolution and faitpt-faithful text-guided image inpainting with diffusion models. arXiv preprint 12211, 2023.
[26] Yotam Nitzan, Zongze Wu, Richard Zhang, Eli Shechtman, Daniel Cohen-Or, Taesung Park, and Michael Gharbi. Lazy diffusion transformer for interactive image editing. In European Conference on Computer Vision, pages 55-72. Springer, 2024.
[27] Biao Qian, Yang Wang, Richang Hong, and Meng Wang. Adaptive data-free quantization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7960-7968, 2023.
[28] Biao Qian, Yang Wang, Hongzhi Yin, Richang Hong, and Meng Wang. Switchable online knowledge distillation. In European Conference on Computer Vision, pages 449-4666 Springer, 2022.
[29] Yurui Qian, Qi Cai, Yingwei Pan, Yehao Li, Ting Yao, Qibin Sun, and Tao Mei. Boosting diffusion models with moving average sampling in frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8911 19200, 2024.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages $748-8763. PmLR. 2021.
[31] Yuxi Ren, Jie Wu, Yanzuo Lu, Huafeng Kuang, Xin Xia, Xionghui Wang. Qianqian Wang. Yixing Zhu, Pan Xie, Shiyin Wang, et al. Bytcedit: Boost, comply and accelerate generative image editing. In European Conference on Computer Vision, pages 184 200. Springer. 2024.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2022.
[34] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assion assistedd intervention-Miie MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages Springer, 2015.
12

---

[35] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy. Stefano Pellegrini. Yasumasa Onoe, Sarah Laszlo, David Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating lext-guided image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), pages 1939911369, 2023.
[36] Yang Wang, Biao Qian, Haipeng Liu, Yong Rui, and Meng Wang. Unpacking the gap box against data-free knowledge distillation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 46(9):6280-622 1229, 2024.
[37] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13(4):600-61 2004.
[38] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain vidcos from natural descriptions. arXiv preprint arXiv:2104.1488 2021.
[39] Wei Wu, Qingnan Fan. Shuai Qin. Hong Gu. Ruoyu Zhao, and Antoni B Chan. Freediff: Progressive frequency truncation for image editing with diffusion models. In European Conference on Computer Vision, pages 194-209. Springer, 2024.
[40] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: solid benchmark evaluating human preferences text-to- text-to-image synthesis. arXiv preprint arXi::2306.0. 09341, 2023.
[41] Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding, Image inpainting with Iearnable bidirectional attention maps. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8858-8867, 2019.
[42] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smarthrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Partern Recognition (CVPR), pages 22428-22437, 2023.
[43] Wenbin Xie, Dehua Song, Chang Xu, Chunjing Xu, Hui Zhang, and Yunhe Wang. Learning frequency- aware dynamic network for efficient super-resolution. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4308-4317, 2021.
[44] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for texl-to-image generation. Advances Neural Information Processing Systems, 36: 15:05591155935, 2023.
[45] Yongsheng Yu, Dawei Du, Libo Zhang, and Tiejian Luo. Unbiased multi-modality guidance for image inpainting. Computer Vision Eocoo-ECCV222 2022: 17th European Conference, Tel Aviv, Isruel, October 23-27, 2022, Proceedings, Part XVI. pages 668-68 684. Springer, 2022.
[46] Zeng, Zhe Lin, Huchuan Lu, and Vishal Patel. Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14164-14173, 2021.
[47] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pages 3836-3847, 2023.
[48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 586-595, 2018.
[49] Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high- versatile image inpainting. arXiv preprint arXiv:2312.03594, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We confirm the main claims reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper have conducted a discussion of limiations in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
We did not include theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have described the model details in the implementation details in Sec 3.] and carefully described the experimental metries the experimental chapter. The informa- tion we provide is sufficient and detailed for replication purposes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We make detailed description of the implementation details to ensure that they are repeatable and use publicly available text-guided image inpainting datasets. We provide our code in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides PoodeSubmissiooooliee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips wttps/cccccccccgcuucdeooocccecee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have carried out detailed narration in the implementation details in
Sec 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA].
Justification: Based involved in this work is high, with results that are replicable and stable, rather than simply reporting the highest outcomes. Additionally, previous related work [12 49 47 25 also not reported error bars. We thus do not run the statistical significance test.
experimental experience, the reproducibility of the experiments
has
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We state this detailed information of computer resources in Sec 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:/p///eee...p....aaaaa/aaaaa
Answer: [Yes]
Justification: We confirm that the research involved in the article complies with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper have conducted discussion of broader impacts in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We utilize the publicly available text- -utd-guiddd image inpainting dataset 12 35
and the pretrained generation models Stable Diffusion V1.5, there no relevant description we will set up safeguards when we release the code of NTN-Diff.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We utilize the publicly available text-guided image inpainting dataset [22 35
and the pretrained generation models Stable Diffusion V1.5, both of them state the dataset and model license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.ppperssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not involve LLM as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) hutoree//eerrppccc/oooffeeeeeeee /c///oofeeeeeees//225////.. for what should or should not be described.
20

---

Appendix Overview
Due to page limitation of the mainbody, as indicated by our submission, the supplementary material offers further discussion on the motivation of mid-frequent freuereue band and more visual results with higher resolution, which are summarized below:
More analysis of the high-and-low frequency components in the denoising process, as mentioned in Sec. I of the mainbody (Sec
More intuitions on hybrid frequency aware diffusion models to text-guided image inpainting, mentioned in Sec.2.2 of the mainbody (SecIB)
The algorithm of NTN-Diff, mentioned in Sec.2.4 of the mainbody [Sec[C
More analysis of computational efficiency for the comparison with state-of-the-arts, as mentioned in Sec.3.2 of the mainbody (SeclD).
Additional quantitative results and qualitative analysis with higher resolution for the com- parison with state-of-the-arts, as mentioned in Sec.3.2 of the mainbody (Sec
More ablation studies on the impact text and null-text prompts on low-and-mid frequency bands during denoising process, mentioned in Sec.3.3.1 of the mainbody (Sec
Additional generation results for the ablation study about hyperparameter sensitivity analysis of for the length of the early and late stage, as mentioned in Sec.3.3.2 of the mainbody
(See G
We list the limitations and broader impacts (Secl) H
A
More Analysis of the High-and-Low Frequency Components in the Denoising Process
Due to the page limitation, we further conduct experiments to verify the conclusion stated in Sec. 1 of the mainbody: "the low-frequency band fluctuates more significantly during the early stage of the denoising process with high-level noise than the mid- and high-frequency bands." " This observation aligns with the common understanding that diffusion models first recover low-frequency components and progressively refine mid- and high-frequency details. Motivated by this, we experimentally validate the denoising process using several metrics to measure variations in the low-and-high frequency bands different timesteps, as reported in Table.
We adopt two groups of metrics to analyze the frequency behavior during the denoising process:
Low-frequency information: We adopt Mean Gray, Mean HSV-V, and Mean Luma, which primarily capture the global brightness and luminance structure of the denoised image. These metrics evaluate how well the low-frequency bands are maintained throughout the denoising process.
High-frequency information: We use Average Gradient, Variance, and LBP Variance, which are sensitive to edge sharpness, local contrast, and texture complexity, respectively. These metrics reflect the preservation and recovery of fine details during denoising.
The results show that fre-freuy metrics (Mean Gray, HSV-V, V, Luma) remain nearly unchanged from the beginning, indicating early stabilization. In contrast, high-frequency metrics (Avg Gradient, Variance, LBP Variance) increase steadily in the later timesteps, confirming that high-frequency details are refined gradually during denoising, which is consistent with the statement in Sec. 1 of the mainbody.
B
More Intuition on Hybrid Frequency Aware Diffusion Models to Text-Guided Image Inpainting
Due to page limitation, we offer more visual results of the mid-frequency band of the denoising process. Fig illustrates the text-guided denoising results for mid-frequency band. To validate its stability, we visualize the layout information bounding box for each step, which, as indicated

---

Table 4: Metries measuring the low-and-higg -igh frequency band variations during the denoising process, confirming that diffusion models typically recover the fwwfrreqenccy band first and progressively refine the mid-and-higg frequency bands.
Timesteps
Mean Gray
Mean HSV.V
Mean Luma
Avg Gradient
Variance
LBP Variance
0
120.9
136.0
120.9
59,01
3955.0
4,45
5
120.8
134.6
120.8
75.07
4098.3
3.71
10
120.7
133.5
120.7
101.58
4378.9
3,25
15
120.7
133.7
120.7
133.26
4790.8
3.07
20
120.9
135.1
120.9
164.23
5274.8
3.07
25
121.2
137.3
121.2
$743.9
30
1214
139.2
1214
209.84
6099.5
3.25
35
121.1
140.0
121.1
222.03
6291.1
3.32
40
120.5
140.1
120.5
228.75
6363.9
3.35
45
119.9
139.8
119.9
232.19
6394.8
3.38
50
119.5
139.5
119.5
233.88
6419.5
3,40
and llueberries"
98
Visualization Mid-frqquene
"a chreese cake"!
Masked Imsage
"a cake
"a indester cake"
The mawe
(b) More Results on Mid-frequency Band for NTN-Dur During the Deneising Process
Figure 8: We investigate the text-guided denoising process for the mid-freeuency bands. For each step, we employ the bounding boxes visualize the variations for layout information (a) and the changes of layout information for the same image based on three similar text prompts. the denoised mid-frequency band changes during the initial early stage owing to text prompts with high-level noise, yet quickly converges by the end of the early stage, which further leads to the stable states with nearly no influence by text prompt across the whole late stage.
by 5 43 is closely related to the mid-frequency band. The denoised mid-frequency band also changes during the initial early stage owing to text prompts with high-level noise, yet quickly converges by the end of the early stage e.g., 60-th step, which further leads to the stable mid-frequency band with nearly no influence by text prompt across the whole late stage with low-level noise. We further exhibit additional visual results by performing the experiments on three similar text prompts. Despite the different text prompts, the uud-frequency can quickly converge by the end of the early stage, e.g., 60-th. Following the above, propose exploit the mid-frequence band during denoising process, which plays the pivotal role achieving the semantics consistency upon text prompts, while better achieve the semanties consistency between masked and unmasked regions, to further preserve its own frequency band well, owing to its robustness to the text prompt during the denoising process.
the same image based
C The Algorithm of NTN-Diff
Our proposed NTN-Diff pipeline consists of a null-text denoising process (Sec. 2.3.1 that avoids the influence text prompts, and lext-guided denoising process (See. 2.3.2) that reconstructs masked regions while replacing the low-frequency band of its denoised output with that from the null-text denoising process. Building upon this design, we further utilize the denoised mid-frequency information to guide another null-text denoising process (Sec. 2.3.3) by substituting the mid-freque hid-frequency

---

band accordingly. In addition, a late-stage text- text-guided demoising process (Sec. 2.4 is conducted, during which the unmasked regions from the early stage of the diffusion process are progressively substituted to preserve structural consistency. Based on the above, we summarize the complete algorithm of our proposed null-text-null frequency-aware diffusion model in Algorithm below:
Algorithm 1 NTN-Diff
Input: masked image binary mask M, text prompt C, null-text prompt Cą Output: inpainted image I'
I. Diffusion Process for Unmasked Regions
1: Extract the initial latent feature of the unmasked region z8¹ = E(T).
2: for t = 0 to T Finy do
3: Compute 2++1 from 2
4: end for inversion]
II. Early Stage
ztxt 1: Initialize 2 27 27 (O,
2: for = T XT + do
3: Substitute the unmasked regions of 2t with the corresponding counterpart of 24 via Eq.3;
4: Compute 2t-1 an
from 2t conditioned on C via Eq.4;
5:
Substitute low-frequency band of stezt with the corresponding counterpart of atm via 2t
Eq.5-7;
6:
Compute teet from 2fext, conditioned on C:
2t-1
7:
Substitute mid-frequency band of z(" with the corresponding counterpart of ztent via
Eq.8-10:
8:
Compute znn from " conditioned on Co:
9: end for
III. Late Stage
1: for t = XI' to 1 do
2: Substitute unmasked regions of zi" with the corresponding counterpart of 2f via Eq.11;
3: Compute = from z;"h, conditioned on C;
4: end for
5: Output the final inpainted image I' = (2)gr).
D More Analysis of Computational Efficiency in the Sec.3.2 of the Mainbody
As indiacted the mainbody, we further provide the inference time and VRAM consumption of our proposed NTN-Diff model, and compared it with several representative baselines, which are reported in Table, Our inference time comparable to BrushNet [12 and faster than recent methods such as HDP 25 and PP [49]. Moreover, our model consumes less VRAM than other text-guided inpainting models, owing to the fact that our method performs low- and mid-frequency substitution across different branches after each denoising step. The key idea is that, being decoupled from the resource- eesource-intensive denoising process itself, our approach introduces only negligible additional memory cost compared to the single denoising process in BLD [T]
Table 5: Inference time and VRAM consumption comparison among different models. NTN-Diff achieves a comparable inference time while exhibiting the lowest memory cost among all methods.
Methods (Venue)
Inference Time (s) 1
VRAM Consumption (GB)
BLD [1] (TOG'23
4.78
2.11
PP [49] (ECCV*24)
7.24
4.09
BrushNet [2] (ECCV'24)
6.59
3.19
HDP 25 (ICLR'25) 25)
10.57
11.75
NTN-Diff (Ours)
6.96
2.11
3

---

Table 6: Quantitative comparisons between NTN-Diff and other diffusion-basee inpainting models over EditBench for inpainting are shown, where all models use Stable Diffusion V1.5 as the baseline model. red and blue stand for the best and second best result. NTN-Diff achieves the best result.
Metrics
Masked Region Preservation
Text Alignment
Models
Venue
PSNR1
MSE.
LPIPS x100
SSIM x10
CLIP Scoret
SDI
32
CVPR' 22
23.25
6,94
24,30
90.13
28,00
BLD
TOG' 23
20.89
10.93
31,90
85.09
28.62
CNT 47
ICCV* 23
12.71
69.42
159.71
79.16
28.16
CNT
ICCV' 23
22.61
35.93
26.14
94.05
27.74
PP 49
ECCV' 24
23.34
20.12
24.12
91.49
27.80
BrushNet* 112
BCCV" 24
33.66
0.63
10.12
98.13
28.87
BrushEdit* [17
ArXiv' 24
32.97
0,70
7.24
98.60
29.62
HDP [25]
ICLR' 25
23.07
6.70
24.32
92.56
28.34
NTN-Diff* (Ours)
36.83
0.28
5.39
99.46
29.72
with blending operation of BrushNet
E Additional Quantitative and Qualitative analysis in the Sec.3.2 of the Mainbody
Evaluation metric. We adopt five metrics to evaluate the inpainted results below: Peak Signal-to- Noise Ratio (PSNR), structural similarity index (SSIM) (37] and Mean Squared Error (MSE) evaluate low-level pixel-wise differences between generated images and their ground truth counterparts. Additionally, Learned Perceptual Image Patch Similarity (LPIPS) (88 measures perceptual similarity by computing the distance between deep features extracted from a pre-trained neural network, offering a robust metric for assessing perceptual alignment; CLIP Similarity (CLIP Score) [38] measures text-image consistency by projecting both the generated images and their corresponding text prompts into a shared embedding space using the CLIP model. It then evaluates the similarity between their embeddings.
As indiacted in the mainbody, we further exhibit additional quantitative results by performing the experiments the EditBench [35;; see Table It is observed that NTN-Diff enjoys larger PSNR, SSIM and CLIP Score, together with smaller MSE and LPIPS than the competitors. Notably, BrushEdit 17) remains the large performance margins (at most, 3.86% for PSNR. 0.42% for MSE 1.85% for LPIPS, 0.86% for SSIM, and 0.1% for CLIP Score) compared to NTN-Diff in Table We also present the quantitative results NTN-Diff with the pixel-level blending operation of [12]. named NTN-DifT* to preserve the unmasked regions, which demonstrates the ability to tame the hybrid frequency issue, the results further verifies the intuition in Sec.1 of the mainbody -TN-Diff can achieve the semantics consistency between mid-and-lov frequency bands across masked and unmasked regions, while preserving unmasked regions.
To shed more light on the advantages of our method, we further perform the visual analysis on the inpainted results with the higher resolution. Fig delivers the following: due the discrepancy between the diffusion process for unmasked regions substitution and the denoising process for masked region alignment upon text prompt. BLD 1 inevitably generate the content of the masked regions (the first column of the Fig (a)) Note that, PP [49 and BrushNet 12 often exhibits the unreasonable inpainted result, egg. The extra woman(the (the 4th row of PP Fig Fa) and the disappeared dinosaur (the 1st row of BrushNet in F owing to the unmasked regions are disrupted, thus fails reconstruct the masked region per text prompt, which attributes to the fact that the low-frequency bands for both masked and unmasked regions are easily to be changed by text prompts, especially under the early stage of the text- text- guided denoising process with high-level noise. As opposed to that, mid-frequency band can better achieve the semantics consistency between masked and unmasked regions, while preserve its own frequencyy band well, than low-frequency band, owing to its robusiness to the text prompt during the denoising process, while the previous arts fail to disentangle all frequency bands during the denoising process especially for its early stage with high-level noise (see 1 of the mainbody).

---

Figure 9: Comparison of the text-guided inpainted results with the state-of-the-ard on BrushBench 112 and EditBench [35]. NTN-Diff delivers the superior inpainted results over others, which can simultaneously preserve the unmasked regions while achieve the semantics consistency between paintiin of a wowan with colorful hand
the all painting of dinosaur leaning on coín motchine maxt to streec
oil painting squars letter bathroom Roor cartteow can sitting om toy of rock looking up at the sky
wolaan plenk Jacket gloves walking di snowy path
painting monkay ceremong
brows. pasing
pangolin dack with few gackts the background
Emsage
(20) BruskNet (12]
Iusaze
BraakNi [12]
(a) BrushBench
(b) EditBench
unmasked and inpainted masked regions.
Table 7: Ablation studies the impact of text and null-text prompts on low-and-mid frequency bands during denoising process: TTN-Diff, NTT-Diff and TTT-Diff for the early stage, based on same text-guided denoising process (Sec.2.4 of the mainbody) for the late stage. red and blue stand for the best and second best result.
Dataset
Metric
ITN-Diff
NTT-Dif
TTT-Diff
Owrs(NTN-Dir)
IR
1.57
2.12
1.34
3.10
PSNRT
22.57
22.58
22,51
22.65
EditBench
LPIPS x103
CLIP Scoret
28.87
28.89
28.81
28.95
1R PSNRT
28.06
28.08
28.01
28.10
BrushBench
LPIPS LPIPS x003
44.63
44.55
44.88
44.09
Scoret
F More Ablation Studies on the Impact of Text and Null-Text Prompts on Low-and-Mid Frequency Bands during Denoising Process
As mentioned in Sec.3.3.1 of the mainbody, due to page limitation, we further provide more ablation studies the impact of text and null-text prompts low-and-mid frequency bands during denoising process on BrushBench and EditBench datasets with three variants: TTN-Diff: replacing the first
null-text denoising process with the text-guided denoising process; NTT-Diff: replacing the last null-text denoising process with the text-guided denoising process; TTT-Diff: replacing both the first and last null-text denoising processes with the text-guided denoising processes; Table 7 suggests that our NTN-Diff outperforms NTTT-Diff, despite the denoised mid-frequency band well aligns with the text prompts especially for masked regions, while also encodes the information related to frequencqncy band from the first null-text denoising process, which verifies that the last null-text denoising process can achieve semantics consistency between aid-and-low frequency bands across masked and unmasked regions, by denoising the low-frequent semantically consistent mid-free mid-frequency band, with no influence from text prompts, which is in line with 3.3.2.3.3 of the mainbody; TTN-Diff exhibits a substantial performance degradation, confirming that the null-text denwising process conditioned on null-text prompt can avoid being influenced by text prompts even under the high-level noise, focusing primurily low-frequency band, hence validating the important the first null-text denoising process 2Se..2.3.1 of the mainbody). We illustrate the above intuitions in Fig [0
band throughout the path to be
5

---

a bird with a yellow and white face sitting on a tree branch
a painting of fish swimming a coral reef
a teacup and saucer with a spoon and a spoon
a bee shown in this poster
Masked Image
Ours (NTN-Diff)
TTN-Diff
NTT-Dif
TTT-Dif
The inpainted output about the impact of text and null-text prompts on low-and-mid frequency bands during denoising process, with the first null-text denoising process (Sec.2.3. of the mainbody), the second text-guided denoising process (Sec.2.3.2 of the mainbody) and the last null-text denoising process 23.c.2.3.3 of the mainbody), NTN-Diff can achieve the better inpainted results than others.
Figure 10:
G Additional Generation Results for the Ablation Study in Sec.3.3.2 of the mainbody
As mentioned in S.c.3.3.2 of the mainbody, due to page limitation, we further provide more visual results to analyse the parameter À in the denoising process (Sec.2.2 the mainbody), which is utilized divide the denoising process into early and late stages, demarcated by the critical step 0.9, the performance is the worst with the shortest early stage, such as blue background in the 4th row. When À = 0.6 with the balance of early and late stage, the best performance of image quality, unmasked region preservation and text alignment are achieved, such the face of the cat in the first rows, confirming the rational that the denoised low now-frequency band for the masked regions is guided by the denoised mid mid-frequency band within the whole early stage, which is guided by text prompts for a few number of steps from text- rext-guided late denoising process to avoid the large influence from text prompts to be inconsistent for unmasked regions, thus the desirable length of early stage can make the low-frequency band under the mid mid-frequency guidance for masked regions achieve the consistency to the substituted unmasked regions for ground truth, which is consistent to Sec.2.4 of the mainbody.
XT. see Fig 11 When
H Limitations and Broader Impacts
Unlike previous state-of-the-art methods that fine-tune Stable Diffusion (v1.5) on large-scale datasets for text-guided image inpainting, which typically require extensive time and computational resources for training, our method introduces plug-and-play frequency-aware null- text-null diffusion framework. This framework substitutes mid-and-low frequency bands during the
Limitations.

---

close up cat looking to the
rabbit with white and brown. face sitting on the ground
the oil painting of yellow dog table with under
the sketch of a blue dog on stage singing, standing on its back paws.
Masked Image
2=0.9
2=0.8
2-0.7
2.=0.6
2==0.5
Figure 11:
The inpainted output about hyperparameter sensitivity analysis of X for the length of the early and late stage, when À = 0.6 with the balance of early and late stage, NTN-Diff can achieve the better inpainted results than others.
early stage of the denoising process and, result, involves three parallel null-text-nun denoising processes the early stage, resulting in a larger computational cost than single process. Nevertheless, it shares the same order of magnitude as the previous methods.
Broader Impacts. Our NTN-Diff can achieve the masked regions to be generated according to the text prompt, while preserve the unmasked regions, our method may raise certain ethical concerns. The inpainted image could potentially be misused in the creation of misleading information. We strongly advocate for the establishment of clear accountability in the use of such technologies, along with enhanced legal and technical oversight, to ensure they are applied responsibly.


Reviews:
[{"review_id": "aGhA3T36zj", "reviewer": "Reviewer_hGay", "summary": "This paper aims to address a core contradiction in text-guided image inpainting: the need to generate semantically consistent content in the masked region according to the text prompt while perfectly preserving the original appearance of the unmasked region. The authors observe that previous methods struggle to balance these two requirements because different frequency components of an image have varying robustness to text prompts during the denoising process of diffusion models. Specifically, low-frequency components (such as color and lighting) are easily influenced by text prompts and can change, leading to damage in the unmasked region; while mid-frequency components (such as structure and layout) are more stable against text prompts and better suited for guiding semantic alignment.​\nBased on this finding, the paper proposes a new method called NTN-Diff. The core idea of this method is to decouple and process image information in the frequency domain. It divides the denoising process into early and late stages. In the early stage, NTN-Diff meticulously processes different frequencies through three parallel denoising sub-processes:​\nA null-text process, used to preserve low-frequency information in the unmasked region without text interference.​\nA text-guided process, used to generate content aligned with the text in the masked region, while its low-frequency part is replaced by the result of the first process to initially protect the background.\nAnother null-text process, utilizing the mid-frequency information already aligned with text from the second process as guidance, to generate consistent low-frequency information, thereby addressing the semantic gap between masked and unmasked regions.\nIn the late stage, the model performs the final text-guided denoising while forcibly restoring the unmasked region at each step to ensure it remains unchanged. Through this complex, multi-stage, frequency-specific processing pipeline, NTN-Diff successfully achieves both protection of the unmasked region and semantic consistency between the generated region and the background.", "questions": "1. Question on Computational Efficiency\n\nYour proposed NTN-Diff method runs three denoising processes in parallel during the early stage, which appears to incur significant computational overhead. To better assess the practicality of this approach, could you provide comparative data on inference time and VRAM consumption against major baseline methods (such as BrushNet)? This data is crucial for understanding the trade-off between performance improvement and resource consumption in your method.\n\n2. Question on Robustness to Extreme Mask Ratios\n\nThe adaptive threshold for frequency-domain masking (Eq. 7 & 10) is correlated with the area ratio of the masked region. How does the method perform when processing very small or very large masked regions? Does a critical point exist where the effectiveness of the frequency-domain separation strategy diminishes? For instance, when the masked region is extremely large (i.e., the unmasked region is very small), is the low-frequency information from the unmasked region still sufficient to effectively guide the overall inpainting process?\n\n3. Regarding the experiments on EditBench\n\nYour paper only reports the results of your proposed method on this benchmark. The results of other methods such as BrushNet are not included. We strongly recommend supplementing this section with comparative performance data.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Originality & Significance\n\nThe core contribution of this paper lies in providing a novel perspective—frequency-domain analysis—for text-guided image inpainting. Identifying the distinct behaviors of low- and mid-frequency components during the diffusion denoising process, and establishing this as the theoretical foundation for resolving the \"content generation vs. background preservation\" conflict, constitutes a very novel and insightful perspective.\n\n2. Quality\n\n  Technical Depth: The method is elegantly designed and technically sound. NTN-Diff is not a simple model but a meticulously crafted multi-stage, multi-process workflow.\n\n  Strong Results: The authors evaluated their method on two mainstream benchmarks (BrushBench, EditBench) using six distinct metrics covering image quality, region preservation, and text alignment. NTN-Diff surpasses existing SOTA methods on BrushBench. Ablation studies demonstrate the effectiveness of each component within its complex design. Furthermore, sensitivity analysis of the key hyperparameter λ reflects the rigor of the research.\n\n3. Clarity\n\nThe introduction clearly articulates the problem and challenges. Figures 1 and 2 play a crucial role in explaining the root cause of the problem and the motivation behind the method, providing excellent intuition. Despite the inherent complexity of the method, the authors make it understandable through step-by-step descriptions and flowcharts.\n\n\n**Weaknesses**\n1. Quality\n\n  Computational Complexity: The most significant potential drawback is computational complexity. The requirement to run three independent denoising processes in parallel during the early stage could result in substantially increased inference time and VRAM consumption compared to single-process baselines (potentially up to threefold). The paper lacks quantitative analysis and comparison of this computational overhead, making it difficult to assess the method's practical feasibility.\n  Potential for Design Simplification: While ablation studies confirm the utility of each component, whether this complex three-process architecture represents the only or optimal solution remains debatable.\n\n2. Originality\n\nThe originality of this work manifests more in the clever integration and sophisticated application of existing ideas (e.g., frequency-domain decomposition, text-free inversion) than in proposing fundamentally new theories or techniques. However, this is only a minor weakness, as the integration itself is highly non-trivial and demonstrably effective.", "was_revised": true, "final_justification": "The authors' rebuttal partially resolved my concerns. After reviewing all feedback, I will raise my rating to 5.", "rebuttal": "# To Reviewer hGay\nThank you very much for your valuable comments that helped us significantly improve this submission. Below are detailed point-to-point responses about how we address your questions.\n\n---\n# [Q1]: Question on Computational Efficiency\n\n**Response**: Thanks for your valuable comments. As per your suggestion, we made up the\ninference time and Peak VRAM consumption of our model, comparing it with other baseline models, along with the detailed explanations, as shown below:\n\n| Methods (Venue)       | Inference Time (s) ↓ | VRAM Consumption (GB) ↓ |\n|----------------------|:------------------:|:----------------------:|\n| BLD [1] (TOG’ 23)  |       **4.78**         |        **2.11**            |\n| PP [2]  (ECCV’ 24)          |       7.24         |        4.09            |\n| BrushNet [3]  (ECCV’ 24)    |       6.59         |        3.19            |\n| HDP [4] (ICLR’ 25)         |      10.57         |       11.75            |\n| **NTN-Diff (Ours)** |       6.96         |        **2.11**            |\n|||\n\n* **Analysis of computational efficiency**\n\n*\"Our inference time is comparable to BrushNet [3] and faster than recent methods such as HDP [4] and PP [2]. Moreover, our model consumes less VRAM than other text-guided inpainting models, **owing to the fact that our method performs low-and-mid frequency substitution across different branches after each denoising step**. The basic idea is to be decoupled from the resource-intensive denoising process itself, it incurs only negligible additional memory cost than the single denosing process BLD [1].\"*\n\n\n---\n# [Q2]: Question on Robustness to Extreme Mask Ratios\n\n**Response**: Thanks for your valuable comments. According to **Line 246- 254, Remark 3 of the mainbody**, We kindly clarify that *due to the denoised mid-frequency band encodes information from the low-frequency band substituted from the null-text denoising process for ground truth including both masked and unmasked regions, **hence the denoised mid-frequency band not only aligns with the text prompts for both regions, but also achieving the consistency to the substituted unmasked regions for ground truth***, \n\nBased on the above, large masked regions make it easier to keep consistency between masked and unmasked regions. Our method adaptively extracts low-frequency bands. **So when the masked region is very large, only a small amount of low-frequency information is needed to maintain consistency**. \n\nWe also provide quantitative comparisons on BrushBench to support this claim. As indicated in **Lines 261–265, Sec. 3.1 of the mainbody**, BrushBench refines the task by considering two specific scenarios: segmentation mask inside-inpainting and segmentation mask outside-inpainting. The outside-inpainting setting focuses on cases with large masked ratios, as shown in **Table. 1 (Outside-Inpainting) of the mainbody, we copy that as follow**: \n\n* **Table. 1 (Outside-Inpainting) of the mainbody**\n\n|Method (Venue)| $IR_{×10}$ ↑ | $HPS_{×10²}$ ↑ |  PSNR ↑  | $MSE_{×10³}$ ↓ | $LPIPS_{×10³}$ ↓ | CLIP Score ↑ |\n|--------------------------|:-------:|:------------:|:------:|:---------:|:-----------:|:------------:|\n| BLD [1] (TOG’ 23)              | 7.81    | 26.77        | 15.85  | 35.86     | 21.40       | 26.73        |\n| CNI [6] (ICCV’ 23)             | 9.26    | 27.68        | 11.91  | 83.03     | 58.16       | 27.29        |\n| PP [2] (ECCV’ 24)              | 7.45    | 28.01        | 18.04  | 31.78     | 15.13       | 26.72        |\n| BrushNet [3] (ECCV’ 24)        | 10.82| 28.02 | 18.06 | 22.86  | 15.08  | 27.33    |\n| HDP [4] (ICLR’ 25)             | 9.66    | 27.79        | 18.03  | 22.99     | 15.22       | 26.96        |\n| **NTN-Diff (Ours)**            | **11.54** | **28.22** | **18.47** | **20.44** | **14.46**   | **27.54**    |\n|||\n\n* **Analysis on the quantitative results of the large mask**\n\n*\"The quantitative results summarize our findings below: **NTN-Diff** enjoys larger IR, HPS v2, PSNR and CLIP Score, together with smaller MSE and LPIPS than the competitors. Notably, BrushNet remains the large performance margins (at most 6.65% for IR, 0.71% for HPS, 2.27% for PSNR, 10.59% for MSE, 4.11% for LPIPS and 0.77% for CLIP Score) compared to **NTN-Diff**, which demonstrates the ability to tame the hybrid frequency issue, the results further verifies the intuition in **Sec.1 of the mainbody** -- **NTN-Diff** achieves the semantics consistency between mid-and-low frequency bands across masked and unmasked regions, while preserving the unmasked regions. We also illustrate the above intuitions in the **last two rows of Fig.1 (d) and the last row of Fig. 5(a) in the mainbody** \"*\n\n---\n# [Q3]: Regarding the experiments on EditBench\n\n**Response**: Thanks.We mildly remind you that we do present a comparison of text-guided inpainting results with state-of-the-art methods on **EditBench** in **the right part of Fig. 5 in the mainbody**. In addition, due to the page limitation for mainbody, we include further quantitative comparisons on **EditBench** with state-of-the-art approaches in the **Line 45-54, Sec. B of the Appendix**, *i.e.*, **Table. 1**, as indicated by **Line 298, Sec. 3.2 of the mainbody, we copy that as follow:**\n\n\n* **Table. 1 of the Appendix**\n\n|      Method (Venue)       | $IR_{×10}$ ↑ | $HPS_{×10²}$ ↑ |  PSNR ↑  | $MSE_{×10³}$ ↓ | $LPIPS_{×10³}$ ↓ | $SSIM_{×10²}$ ↑ | CLIP Score ↑ |\n|-------------------------|:-------:|:------------:|:--------:|:---------:|:-----------:|:----------:|:------------:|\n| SDI [5] (CVPR' 22)            |  1.86   |    24.24     |  23.25   |   6.94    |    24.30    |   90.13    |    28.00     |\n| BLD [1] (TOG' 23)             |  0.90   |    23.81     |  20.89   |  10.93    |    31.90    |   85.09    |    28.62     |\n| CNI [6] (ICCV' 23)            |  0.90   |    23.79     |  22.61   |  35.93    |    26.14    |   94.05    |    27.74     |\n| PP [2] (ECCV' 24)             |  1.24   |    24.50     |  23.34   |  20.12    |    24.12    |   91.49    |    27.80     |\n| BrushNet [3] (ECCV' 24)       |  4.46   |    25.24     |  33.66   |   0.63    |    10.12    |   98.13    |    28.87     |\n| BrushEdit [7] (ArXiv' 24)     |   -     |      -       |  32.97   |   0.70    |     7.24    |   98.60    |    29.62     |\n| HDP [4] (ICLR' 25)            |  1.74   |    24.20     |  23.07   |   6.70    |    24.32    |   92.56    |    28.34     |\n| **NTN-Diff (Ours)**          |**4.61**|**25.83**| **36.83**| **0.28**  |  **5.39**   | **99.46**  |  **29.72**   |\n|||\n\n* **Analysis of the quantitative results on EditBench**\n\n*\"In particular, we further exhibit additional quantitative results by performing the experiments on the **EditBench**; It is observed that **NTN-Diff** enjoys larger PSNR, SSIM and CLIP Score, together with smaller MSE and LPIPS than the competitors. Notably, BrushEdit remains the large performance margins (at most, 3.86% for PSNR, 0.42% for MSE, 1.85% for LPIPS, 0.86% for SSIM, and 0.1% for CLIP Score) compared to **NTN-Diff** in table. The results further verifies the intuition in **Sec.1 of the mainbody** -- **NTN-Diff** can achieve the semantics consistency between mid-and-low frequency bands across masked and unmasked regions, while preserving unmasked regions.\"*\n\n\n---\n# References\n\n- [1] Avrahami O, Fried O, Lischinski D. Blended latent diffusion[J]. ACM transactions on graphics (TOG), 2023, 42(4): 1-11.\n\n- [2] Zhuang J, Zeng Y, Liu W, et al. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 195-211.\n\n- [3] Ju X, Liu X, Wang X, et al. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 150-168.\n\n- [4] Manukyan H, Sargsyan A, Atanyan B, et al. Hd-painter: high-resolution and prompt-faithful text-guided image inpainting with diffusion models[C]//The Thirteenth International Conference on Learning Representations. 2025.\n\n- [5] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.\n\n- [6] Zhang L, Rao A, Agrawala M. Adding conditional control to text-to-image diffusion models[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2023: 3836-3847.\n\n- [7] Li Y, Bian Y, Ju X, et al. Brushedit: All-in-one image inpainting and editing[J]. arXiv preprint arXiv:2412.10316, 2024."}, {"review_id": "AYM7Uii6qN", "reviewer": "Reviewer_uzVF", "summary": "The robustness of diffusion models to text prompting varies during the course of the denoising process. At the start of the reverse process (when the image is mostly noise), the reversal is very susceptible perturbations from the text prompting (low frequency changes are common). In order to fix this, the authors propose using several reverse processes that are each coupled together. The main reversal is “null-text” at large noise levels, and text-guided at low noise levels. A separate but coupled text-guided reversal is used to fill in the mid-frequencies of the null-text process at large noise levels, and a third null-text process is used to fill in the low frequencies of the text-guided reversal used above. At low noise levels, the forward process from the masked ground truth itself is coupled into the reversal. \n\nNote: Overall I recommend accept because this seems like a decent effort to solve an interesting problem with a novel idea. I hesitate to say this because I think this paper needs an important major rewrite, albeit for superficial reasons.", "questions": "1. In Figure 1, what is the red dashed line? It says “Diffusion result” in the figure, but the caption says that it plots the difference between the ground truth and the text prompt in CLIP latent space. Why is that changing? Is Figure 1(d) just a completely independent figure that depicts a comparison with other methods? \n2. It is very difficult to see Figure 2. It seems like there are depictions of the low frequency band and mid-frequency band of the latent space. Does that mean that you pass the latent representation through a low/band pass filter? Can this be explained formally somewhere? This seems like an interesting idea but I have never seen it before and I feel a reader could use a more gentle introduction to this topic.\n3. Is there some formal way of saying that when you do a diffusion in pixel space the lower frequency components get decided first? It seems like a simple model of this should be possible - for instance the mean brightness of the noise you sample at the beginning might be correlated with the mean brightness of the resulting image and is determined already at the start of the process. It seems natural that in pixel space you cannot change lower frequency components as easily as higher frequency components.\n4. Is it possible to just use the low frequency and high frequency components of a null-text model with mid-frequency components of a text-guided model? In other words, why are there two null-text processes?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThis is a good idea. If indeed it is the case that we would like to avoid text conditioning at large noise levels, this seems like a reasonable way to do it, while still maintaining the mid-frequency learned by the text prompting. Basically I think the idea is to use a text-conditioned reversal and a null-text reversal and couple them in such a way as to get the conditioning from the text-guided process and the robustness of the null-text process.\n\nWeaknesses: \nI found the writing and presentation difficult to follow. I dont mean to be unnecessarily picky about this, but it should be easy enough to use simpler writing choices, especially with today’s tools.\n1. What does “One stone with two birds” have to do with the paper? \n2. There is a >70 word sentence in the abstract that can likely be broken into at least 3 smaller sentences.\n3. There is no pseudo code, or any other such consolidated description of the algorithm. What I expressed in the summary is based off of staring at Figure 3 for a while.\n4. All of the figures are too small, or contain titles, axis descriptions that are difficult to see even when you zoom in.\n5. Line 33, preserve -> preserving. Possibly many more examples of this.", "was_revised": true, "final_justification": "Thank you for your response. I will keep my score.", "rebuttal": "# To Reviewer uzVF\nThank you very much for your valuable comments that helped us significantly improve this submission. Below are detailed point-to-point responses about how we address your questions.\n\n---\n# [W1]: The Meaning of “One stone with two birds”\n\n**Response**: Thanks for your valuable comments. We would like to clarify that, as mentioned in **Line 32 - 52, Sec.1 of the mainbody**, *the longstanding challenges of text-guided image inpainting lie in **the preservation for unmasked regions**, while **achieving the semantics consistency between unmasked and masked regions** as inpainted. Previous arts failed to address both of them, always with either of them to be remedied.* In this paper, we propose a null-text-null frequency-aware diffusion models, named **NTN-Diff**, for text-guided image inpainting, **to simultaneously address two challenges** of **unmasked regions preservation**, along with its **semantics consistency with inpainted masked regions**. **Hence, we title one stone with two birds.**  \n\n\n---\n# [W2]: Pseudo Code of the Algorithm  \n\n**Response**: As you suggested, we summarize the pseudo code of the algorithm below, which is consistent to our code in **Sec.G of the Appendix** for reproducibility.\n\n> ### **Algorithm: NTN-Diff**\n>\n> ---\n> **Input**: masked image $I$, binary mask $M$, text prompt $C$, null-text prompt $C_{∅}$  \n> **Output**: inpainted image $I'$\n>\n> ---\n> #### **I. Diffusion Process for Unmasked Regions**\n> 01: Extract the initial latent feature of the unmasked region $z_{0}^{gt} = E(I)$.  \n> 02: **for** $t = 0$ to $T_{inv} - 1$ **do**  \n> 03:  Compute $z_{t+1}^{gt}$ from $z_{t}^{gt}$;  \n> 04: **end for** {DDIM inversion}\n>\n> ---\n> #### **II. Early Stage**\n> 05: Initialize $z_{T}^{un}$, $z_{T}^{text}$, $z_{T}^{mid} \\sim N(0,1)$.  \n> 06: **for** $t = T$ to $\\lambda T + 1$ **do**  \n> 07:  Substitute the **unmasked regions** of $z_{t}^{un}$ with the corresponding counterpart of $z_{t}^{gt}$ via **Eq.3**;  \n> 08:  Compute $z_{t-1}^{un}$ from $z_{t}^{un}$, conditioned on $C_{∅}$ via **Eq.4**;  \n> 09:  Substitute **low-frequency band** of $z_{t}^{text}$ with the corresponding counterpart of $z_{t}^{un}$ via **Eq.5–7**;  \n> 10:  Compute $z_{t-1}^{text}$ from $z_{t}^{text}$, conditioned on $C$;  \n> 11:  Substitute **mid-frequency band** of $z_{t}^{mid}$ with the corresponding counterpart of $z_{t}^{text}$ via **Eq.8–10**;  \n> 12:  Compute $z_{t-1}^{mid}$ from $z_{t}^{mid}$, conditioned on $C_{∅}$;  \n> 13: **end for**\n>\n> ---\n> #### **III. Late Stage**\n> 14: **for** $t = \\lambda T$ to $1$ **do**  \n> 15:  Substitute **unmasked regions** of $z_{t}^{mid}$ with the corresponding counterpart of $z_{t}^{gt}$ via **Eq.11**;  \n> 16:  Compute $z_{t-1}^{mid}$ from $z_{t}^{mid}$, conditioned on $C$;  \n> 17: **end for**  \n> 18: Output the final inpainted image $I' = D(z_{0}^{mid})$. \n\n---\n# [W3]: Writing and Presentation\n\n**Response**: Thanks for the comments, we will fix all your suggestions regarding writings issues, such as long sentences, small figure and typos etc. \n\n---\n# [Q1]: The Explanation of Fig.1\n\n**Response**: Thanks for the valuable comments. To better understand **red dashed lines**, we kindly clarify that t-SNE visualization of the CLIP latent space is adopted to illustrate the behavior of the text-guided denoising process in **Fig.1 (a–c) of the mainbody**, which aims to examine whether our method can **preserve the unmasked regions** and **achieve semantic consistency between masked and unmasked regions** across timesteps in the denoising process.\n\n* Instead of computing the direct distance between the noise-free ground truth and the text prompt, **red dashed line** in the Fig.1 highlights the trace the denoising trajectory of the ground truth at different timesteps and connect them to serve as a reference path.\n\n* Besides, we compare our NTN-Diff (**Fig.1(c)**) with state-of-the-arts for text-guided inpainting: The BLD result in **Fig.1(d)** corresponds to the type represented in **Fig.1(a)**, while the BrushNet results in **Fig.1(d)** align more closely with the type in **Fig.1(b)**, which provides qualitative support for our analysis. \n\n---\n# [Q2]: The Explanation of Fig.2\n\n**Response**: Thanks the comments. We visualize the low- and mid-frequency bands in the latent space during the denoising process to better illustrate their differences. \n\n* At each timestep, we apply the 2D Discrete Cosine Transform to the latent representation and use the masks defined in **Eq. (6) (Line 195 - 196) and Eq. (9) (Line 216 - 217) of the mainbody** to extract the corresponding low- and mid-frequency components. \n\n* We then apply the 2D Inverse Discrete Cosine Transform to project these frequency-filtered components back into the latent space, yielding the mid-frequency latent representation and the low-frequency latent representation.\n\n* To further facilitate visual comparison, we compute the corresponding timestep-0 representations by inverting the noise addition step defined in **Eq. (1) (Line 114 - 115) of the mainbody**—i.e., applying the reverse operation of the noise injection without running the full denoising process, to facilitate the observation on the influence of each frequency band on the final output in a more isolated and interpretable manner. \n\n* We then decode these latent representations into the image space via the pre-trained VAE decoder for visualization.\n\n---\n# [Q3]: Changes in Low-and-High Frequency Components During Denoising Process\n\n**Response**: As you suggested, we conducted the experiments to verify the conclusion in **Lines 54–57, Sec. 1 of the mainbody**: *the low-frequency band fluctuates more significantly during the early stage of the denoising process with high-level noise than the mid-and-high frequency bands*. This observation is consistent with the fact that diffusion models typically recover the low-frequency components first and progressively refine the mid- and high-frequency details.\n\nMotivated by the above, we experimentally validate the denoising process as per several metrics to measure changes in the low-and-high frequency bands at different timesteps, as reported below:\n\n| Timesteps | Mean Gray | Mean HSV-V | Mean Luma | Avg Gradient | Variance | LBP Variance |\n|-----------|:---------:|:----------:|:---------:|:------------:|:--------:|:------------:|\n|**0**|**120.9**|**136.0**|**120.9**|**59.01**|**3955.0**|**4.45**|\n|5|120.8|134.6|120.8|75.07|4098.3|3.71|\n|10|120.7|133.5|120.7|101.58|4378.9|3.25|\n|15|120.7|133.7|120.7|133.26|4790.8|3.07|\n|20|120.9|135.1|120.9|164.23|5274.8|3.07|\n|25|121.2|137.3|121.2|190.49|5743.9|3.15|\n|30|121.4|139.2|121.4|209.84|6099.5|3.25|\n|35|121.1|140.0|121.1|222.03|6291.1|3.32|\n|40|120.5|140.1|120.5|228.75|6363.9|3.35|\n|45|119.9|139.8|119.9|232.19|6394.8|3.38|\n|50|119.5|139.5|119.5|233.88|6419.5|3.40|\n|||\n\n* **Analysis of the high-and-low frequency in the denoising process**\n\n\"*We adopt two groups of metrics to analyze the frequency components of the denoising process:* \n\n* *For low-frequency information, we adopt **Mean Gray**, **Mean HSV-V**, and **Mean Luma**, which primarily capture the global brightness and luminance structure of the denoised image. These metrics help evaluate how well the low-frequency bands are maintained throughout the denoising process.*\n\n* *For high-frequency information, we use **Average Gradient**, **Variance**, and **LBP Variance**, which are sensitive to edge sharpness, local contrast, and texture complexity, respectively. These metrics reflect the preservation and recovery of fine details during denoising.*\n\n*The results show that low-frequency metrics (Mean Gray, HSV-V, Luma) remain nearly unchanged from the beginning, indicating early stabilization. In contrast, high-frequency metrics (Avg Gradient, Variance, LBP Variance) increase steadily in the late timesteps, **confirming that high-frequency details are refined gradually during denoising**, which is in line with **Lines 54–57, Sec. 1 of the mainbody**.\"*\n\n---\n# [Q4]: Discussion on the Second Null-Text Denoising Process of the Early Stage\n\n**Response**: Thanks. as mentioned in **Line 202 - 207, Sec.2.3.2 of the mainbody**. We mildly clarify that *the low-frequency band across both masked and unmasked regions are preserved even under text prompts owing to the substitutions from **the first null-text denoising process**, with only mid-frequency aligned with text prompts for both regions, hence still failed to achieve the consistency between masked and unmasked regions. To address this, we propose another concurrent **null-text denoising process**, guided by above denoised mid-level frequency, to help denoise the low-frequency band for both regions, especially for masked regions*,  To verify that, we also conduct the ablation studies in **Line 301 - 313, Sec.3.3.1 of the mainbody, **i.e.**, Table. 2, we copy that below:**\n\n\n* **Table. 2 of the mainbody**\n\n|Method| $IR_{×10}$ ↑ | PSNR ↑ | $LPIPS_{×10³}$ ↓ | CLIP Score ↑ |\n|---------------------|:-------:|:------:|:-----------:|:------------:|\n| EditBench - Case C  | 2.14    | 22.53  | 25.24       | 28.60        |\n| EditBench - Ours   | **3.10**| **22.65** | **24.21**  | **28.95**    |\n||||||\n| BrushBench - Case C | 10.02   | 28.06  | 44.92       | 26.03        |\n| BrushBench - Ours  | **11.12**| **28.10**| **44.09**  | **26.09**    |\n||||||\n\n\n* **Analysis of the second null-text denoising process**\n\n*“...Case C: removing the null-text denoising process (Sec.2.3.3); ... Case C exhibits a substantial performance degradation (at most 44.8%), confirming that only mid-frequency aligned with text prompts for masked and unmasked regions, still failed to achieve the consistency across the whole regions, which is in line with Sec.2.3.3 and illustrated in Fig.6.”*"}, {"review_id": "R0bL6f0M6z", "reviewer": "Reviewer_zpgQ", "summary": "This paper proposes NTN-Diff, a frequency-aware diffusion model for text-guided image inpainting. The core idea is to improve consistency between masked and unmasked regions by aligning their low- and mid-frequency components during denoising. The method uses a three-stage early denoising process to disentangle and coordinate frequency bands, followed by standard text-guided refinement in the late stage.", "questions": "1. The authors note that “compared to the mid-frequency band, the high-frequency band is sparser and more easily overwhelmed by high-level noise in the early stages,” and thus focus primarily on the mid- and low-frequency bands. Does this imply that the proposed method is currently unable to handle or preserve high-frequency content in the un-masked regions? Given that high-frequency cues (e.g., edges and fine textures) are critical for human perception, this design choice seems somewhat counterintuitive. Could the authors provide an analysis or discussion comparing image quality with and without accurate high-frequency information? Such insights would be valuable for the community to better understand how different frequency bands contribute to perceptual quality in text-guided image generation.\n\n2. Equations (6) and (9) suggest that the low-pass and mid-pass masks are predefined based on coordinate thresholds, and thus are data-independent. However, it is intuitive that the distribution of low-, mid-, and high-frequency components can vary significantly across different images. Would it not be more appropriate to adopt data-dependent frequency masks that adapt to the specific frequency characteristics of each input? Could the authors clarify this design decision, and potentially discuss whether a learned or adaptive masking strategy might further improve performance?\n\n3. The paper lacks experimental evidence to support the necessity of the proposed three-stage denoising process in the early phase (null-text → text-guided → null-text), which likely incurs significant inference overhead. To better justify this design, it would be helpful for the authors to provide comparisons against simplified versions of the pipeline—such as single-stage or two-stage denoising—in terms of both performance (e.g., semantic consistency, fidelity) and computational efficiency (e.g., runtime, FLOPs). Quantifying how much performance degrades when reducing the number of stages would help clarify the necessity and cost-effectiveness of the current design.\n\n4. The visualizations and SOTA comparisons are not particularly convincing. For instance, in Figure 5, several examples (e.g., the woman with a colorful headscarf) suggest that BrushNet already preserves the unmasked regions reasonably well. This observation seems to contradict the claim in the introduction that existing methods \"fail to preserve unmasked regions.\" Given this discrepancy, could the authors clarify whether the metrics shown in Figure 1, particularly those based on CLIP distance, reliably reflect unmasked region preservation? Additional qualitative or user-based validation may be needed to support this point.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel frequency-aware perspective for text-guided image inpainting. Specifically, it proposes to retain low- and mid-frequency components from the unmasked regions to guide the inpainting process. This spectrum-based formulation of guidance is intuitive and technically interesting.\n\n2. This paper includes statistical observations on how different frequency bands behave during early-stage denoising, which may be helpful to the community.\n\nWeaknesses:\n1. While the method is grounded in frequency-domain intuition, the exploration and justification of frequency-related design choices lack depth.\n\n2. The multi-stage denoising structure increases complexity, but its necessity and efficiency are not adequately quantified.\n\n3. Some visual results do not clearly support the paper’s core claims, raising concerns about the reliability of the evaluation metrics used.", "was_revised": true, "final_justification": "The authors have addressed all of my concerns. This is a solid paper. Although it didn’t particularly appeal to me, it indeed presents a complete piece of work. Therefore, I am raising my rating to borderline accept.", "rebuttal": "# To Reviewer zpgQ\nThanks very much for your valuable comments that helped us significantly improve this submission. Below are detailed point-to-point responses about how we address your questions.\n\n---\n# [Q1]: Discussion on the High-Frequency Band\n\n**Response**: Thanks. As elaborated in **Remark 1 (Lines 202–207, Sec. 2.3.2) of the mainbody**, to ensure consistency between masked and unmasked regions, we introduce a concurrent null-text denoising process guided by the previously denoised mid-frequency components. This process helps to denoise the low-frequency band across both masked and unmasked regions. Meanwhile, the **high-frequency band** is also addressed in the final null-text denoising step of the early stage. \n\nFollowing your suggestions, we further conduct the ablation studies on exploiting the denoised high-frequency result from the second text-guided denoising process, to substitute the denoised high-frequency band from the last null-text denoising process, along with the detailed explanations, as shown below:\n\n| Method| $IR_{×10}$ ↑ | PSNR ↑ | $LPIPS_{×10³}$ ↓ | CLIP Score ↑ |\n|---------------------|:-------:|:------:|:-----------:|:------------:|\n|EditBench-$Case_{HF}$|2.36|22.49|25.03|28.63|\n|**EditBench-Ours**|**3.10**|**22.65**|**24.21**|**28.95**|\n|||\n|BrushBench-$Case_{HF}$|10.55|28.06|44.71|25.98|\n|**BrushBench-Ours**|**11.12**|**28.10**|**44.09**|**26.09**|\n|||\n\n* **Analysis of the impact of substituting the denoised high-frequency components**\n\n*\"The results show that $Case_{HF}$ exhibits a substantial performance degradation (at most 23.87% for IR), confirming that **the high-frequency band is inherently sparser and more susceptible to being overwhelmed by high-level noise during early denoising, which significantly limits its effectiveness in guiding low-frequency reconstruction**, which is in line with **footnote in Sec. 1**, as well as **the Line 53 - 61 of mainbody**.\"*\n\n* We experimentally validate the denoising process as per several metrics to measure changes in the low-and-high frequency bands at different timesteps, we adopt two groups of metrics to analyze the frequency components of the denoising process below:\n\n| Timesteps | Mean Gray | Mean HSV-V | Mean Luma | Avg Gradient | Variance | LBP Variance |\n|-----------|:---------:|:----------:|:---------:|:------------:|:--------:|:------------:|\n|**0**|**120.9**|**136.0**|**120.9**|**59.01**|**3955.0**|**4.45**|\n|5|120.8|134.6|120.8|75.07|4098.3|3.71|\n|10|120.7|133.5|120.7|101.58|4378.9|3.25|\n|15|120.7|133.7|120.7|133.26|4790.8|3.07|\n|20|120.9|135.1|120.9|164.23|5274.8|3.07|\n|25|121.2|137.3|121.2|190.49|5743.9|3.15|\n|30|121.4|139.2|121.4|209.84|6099.5|3.25|\n|35|121.1|140.0|121.1|222.03|6291.1|3.32|\n|40|120.5|140.1|120.5|228.75|6363.9|3.35|\n|45|119.9|139.8|119.9|232.19|6394.8|3.38|\n|50|119.5|139.5|119.5|233.88|6419.5|3.40|\n||\n\n* **Analysis of the high-and-low frequency in the denoising process**\n\n*The results show that low-frequency metrics (Mean Gray, HSV-V, Luma) remain nearly unchanged from the beginning, indicating early stabilization. In contrast, high-frequency metrics (Avg Gradient, Variance, LBP Variance) increase steadily in the late timesteps, **confirming that high-frequency details are refined gradually during denoising**, which is in line with **Lines 54–57, Sec. 1 of the mainbody**.\"*\n\n---\n# [Q2]: Discussion on Adaptively Extracting the Low-and-Mid Frequency Bands\n\n**Response**: Thanks for your valuable comments. We kindly remind that we truly considered to **adaptively** extract the low-and-mid frequency bands from masked images based on **Eq.(7) and Eq.(10) of the mainbody, we copy that as follow:**\n\n* **Line 197 - 198, Eq.(7) of the mainbody**\n\n$$\nth_{lp} = λ_{lp}^{f} + λ^{r}_{lp} \\cdot \\frac{\\|M\\|_1}{HW}\n$$  \n\n*where $λ_{lp}^{f}$ and $λ_{lp}^{r}$ are hyper-parameters. We set $th_{lp}$ to adaptively correlate with the ratio of unmasked regions, since the larger unmasked regions require more **low-frequency band** from the null-text denoising process to substitute the low-frequency band within the text-guided denoising process.*\n* **Line 219 - 220, Eq.(10) of the mainbody**\n\n$$\nth_{mp1} = λ_{mp1}^{f} - λ_{mp1}^{r} \\cdot \\frac{\\|M\\|_1}{HW}\n$$\n\n$$\nth_{mp2} = λ_{mp2}^{f} + λ_{mp2}^{r} \\cdot \\frac{\\|M\\|_1}{HW}\n$$\n\n*where $\\lambda_{mp1}^{f}$, $\\lambda_{mp1}^{r}$, $\\lambda_{mp2}^{f}$, and $\\lambda_{mp2}^{r}$ are hyper-parameters.  We also set $th_{mp1}$ and $th_{mp2}$ to correlate with the ratio of unmasked regions.  This is because larger nmasked regions require more **mid-frequency band** to encode the information that originates from the **low-frequency band** substituted during the first null-text denoising process.*\n\nWe provide further ablation studies on the impact of adaptively extracting low-and-mid frequency bands in the **Line 70 - 89, Sec.C of the Appendix, i.e., Table. 2, we copy that below:**\n\n* **Table. 2 of the Appendix**\n\n|Method|  $IR_{×10}$ ↑ |   PSNR ↑  | $LPIPS_{×10³}$ ↓ | CLIP Score ↑ |\n| ----------------------- | :-------: | :-------: | :----------: | :----------: |\n|EditBench-CaseI|2.98|22.53|25.12|28.91|\n|EditBench-CaseII|3.02|22.50|25.20|28.87|\n|EditBench-CaseIII|3.07|22.62|24.55|28.93|\n|**EditBench-Ours**|**3.10**|**22.65**|**24.21**|**28.95**|\n|||\n|BrushBench-CaseI|10.04|28.08|44.18|25.91|\n|BrushBench-CaseII|10.10|**28.12**|44.25|25.89|\n|BrushBench-CaseIII|10.95|28.08|44.15|26.07|\n|**BrushBench-Ours**|**11.12**|28.10|**44.09**|**26.09**|\n|||\n\n* **Analysis of adaptively extracting the low-and-mid frequency bands**\n\n*\"As mentioned in **Sec.2.3.2 and Sec.2.3.3 of the mainbody**, we set $th_{lp}$ in Eq.(7), and $th_{mp1}$ and $th_{mp2}$ in Eq.(10) to correlate with the ratio of unmasked regions. Due to page limitations, we provide further ablation studies on the impact of adaptively extracting low-and-mid frequency bands ... **Case III** shows substantial performance degradation, confirming *that larger unmasked regions demand more low-frequency information from the null-text denoising process to replace the low-frequency bands in the text-guided denoising process*, which is consistent with the findings in **Sec.2.3.2 of the mainbody**. These observations are also illustrated in Table 2.\"*\n\n---\n# [Q3]: Discussion on the Performance and Computational Efficiency\n**Response**: Following your suggestions, we extend the experiments in **Table 2 and Line 300 - 313, Sec. 3.3.1 of the mainbody** to more results regarding the inference time and VRAM consumption during the early-stage denoising process, as shown below:\n\n* **Table. 2 of the mainbody**\n\n| Method              | $IR_{×10}$ ↑ | $LPIPS_{×10³}$ ↓ | CLIP Score ↑ | Inference Time (s) ↓ | VRAM Consumption (MB) ↓ |\n|---------------------|:------------:|:-----------------:|:-------------:|:----------------------:|:-------------------------:|\n|EditBench-CaseB|1.24(–60.00%)|24.82(–2.52%)|28.53(–1.45%)|**4.78**(–2.18s)|**2159.35**(–3.94MB)|\n|EditBench-CaseC|2.14(–30.97%)|25.24(–4.25%)|28.60(–1.21%)|5.51(–1.45s)|2163.17(–0.12MB)|\n|**EditBench-Ours**|**3.10**|**24.21**|**28.95**|6.96|2163.29|\n|||\n|BrushBench-CaseB|9.59(–13.77%)|47.08(–6.78%)|25.78(–1.19%)|**4.78**(–2.18s)|**2159.35**(–3.94MB)|\n|BrushBench-CaseC|10.02(–9.89%)|44.92(–1.88%)|26.03(–0.23%)|5.51(–1.45s)|2163.17(–0.12MB)|\n|**BrushBench-Ours**|**11.12**|**44.09**|**26.09**|6.96|2163.29|\n|||\n\n* **Analysis of the balance of performance and computational efficiency**\n\n*\"The results demonstrates that our method significantly improves key performance metrics across both EditBench and BrushBench datasets. For example, IR increases by up to 44.86% and 10.99%, while LPIPS and CLIP Score also show consistent improvements, indicating better image quality and semantic alignment. Although inference time increases by 2.18s, this trade-off is fully justified given the substantial gains in generation quality and consistency. **Overall, the performance boost far outweighs the moderate increase in computational cost.** \"*\n\n---\n# [Q4]: Discussion on the Unmasked Region Preservation\n\n**Response**: Thanks. We kindly clarify that the CLIP-based metric shown in **Fig.1 of the mainbody** is used as a complementary metric, for measuring the preservation of unmasked regions. While CLIP distance primarily captures semantic-level similarity, it can help reveal whether the denoised image has undergone undesirable semantic shifts—including in the unmasked areas.\n\nTo ensure a more comprehensive evaluation, we have included multiple metrics (*e.g.*, LPIPS, PSNR, SSIM and MSE). These jointly verify that our method better preserves the unmasked regions, which demonstrates the ability to tame the hybrid frequency issue, the results further verify the intuition in **Sec.1 of the mainbody** - NTN-Diff achieves the semantics consistency between mid-and-low frequency bands across masked and unmasked regions, while preserving unmasked regions. To facilitate the validation, **we copy the quantitative results** of unmasked region preservation between BrushNet [1] and ours **in Table 1 of the mainbody and Appendix below**:\n\n\n* **Table. 1 of the mainbody** \n\n| Method | PSNR ↑ | $MSE_{×10³}$ ↓ | $LPIPS_{×10³}$ ↓ |\n|--------------------------|:------:|:---------:|:-----------:|\n|BrushBench(Insides)-BrushNet [1]|21.65|9.31|48.28|\n|**BrushBench(Insides)-NTN-Diff(Ours)**|**23.51**|**6.50**|**40.79**|\n|BrushBench(Outsides)-BrushNet [1]|18.06|22.86|15.08|\n|**BrushBench(Outsides)-NTN-Diff(Ours)**|**18.47**|**20.44**|**14.46**|\n|||\n\n* **Table. 1 of the Appendix**\n\n| Method| PSNR ↑ | $SSIM_{×10²}$↑ | $MSE_{×10³}$ ↓ | $LPIPS_{×10³}$ ↓ |\n|--------------------------|:------:|:------:|:---------:|:-----------:|\n|EditBench-BrushNet [1]|33.66|98.13|0.63|10.12|\n|**EditBench-NTN-Diff(Ours)**|**36.83**|**99.46**|**0.28**|**5.39**|\n|||\n\n---\n# References\n\n- [1] Ju X, Liu X, et al. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 150-168."}, {"review_id": "BEtwAwb3lO", "reviewer": "Reviewer_7rxu", "summary": "This paper addresses the problem of image inpainting within text-to-image (T2I) diffusion models. Specifically, the authors aim to tackle two key challenges: preserving the fidelity of unmasked regions and ensuring semantic consistency between the unmasked and inpainted (masked) regions. To this end, they propose a framework called Null-Text-Null Frequency-Aware Diffusion (NTN-Diff). The core idea is to decompose semantic consistency across masked and unmasked regions into frequency bands and handle them separately at different denoising stages. Experiments are conducted on the BruchBench and EditBench benchmarks, using Stable Diffusion v1.5 as the base model.", "questions": "Please see the weaknesses specified above.", "limitations": "The limitations are discussed in the appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**S1:** The idea is well-motivated and logically structured. The core motivation to disentangle the varying influence of different frequency components across denoising stages is supported by both analytical insights and empirical evidence.\n\n**S2:** The paper is well-written, with the key technical components clearly explained and easy to follow.\n\n**S3:** The experimental results demonstrate consistent performance improvements over multiple baseline methods.\n\n\n**W1:** The iterative sampling process is likely to extend the denoising procedure and introduce additional inference overhead. A discussion of the computational cost and a comparison with existing methods would provide a more complete picture of the proposed framework’s practicality.\n\n**W2:** Since the proposed method relies on the structure of DDPMs, its generalizability to more recent T2I models, such as DiTs or flow-based models, may be limited. Moreover, the current experiments are conducted solely on Stable Diffusion v1.5.\n\nMinor: There appear to be some noticeable formatting adjustments made to accommodate long text sequences.", "was_revised": true, "final_justification": "The rebuttal answered my questions in terms of computational complexity and generalization ability to other diffusion based models.", "rebuttal": "# To Reviewer 7rxu\nThanks very much for your valuable comments that helped us significantly improve this submission. Below are detailed point-to-point responses about how we address your questions.\n\n---\n\n# [W1]: Question on Computational Efficiency\n\n**Response**:Thanks, we kindly suggest you to refer to [Q1] from **Reviewer hGay**, to facilitate your reading, we made up the inference time and VRAM consumption of our model, comparing it with other baseline models, along with the detailed explanations below:\n\n\n| Methods (Venue)       | Inference Time (s) ↓ | VRAM Consumption (GB) ↓ |\n|----------------------|:------------------:|:----------------------:|\n| BLD [1] (TOG’ 23)  |       **4.78**         |        **2.11**            |\n| PP [2]  (ECCV’ 24)          |       7.24         |        4.09            |\n| BrushNet [3]  (ECCV’ 24)    |       6.59         |        3.19            |\n| HDP [4] (ICLR’ 25)         |      10.57         |       11.75            |\n| **NTN-Diff (Ours)** |       6.96         |        **2.11**            |\n|||\n\n* **Analysis of computational efficiency**\n\n*\"Our inference time is comparable to BrushNet [3] and faster than recent methods such as HDP [4] and PP [2]. Moreover, our model consumes less VRAM than other text-guided inpainting models, **owing to the fact that our method performs low-and-mid frequency substitution across different branches after each denoising step**. The basic idea is to be decoupled from the resource-intensive denoising process itself, it incurs only negligible additional memory cost than the single denosing process BLD [1].\"*\n\n\n---\n\n# [W2]: Results on More T2I Models\n\n**Response**:Thanks for the valuable comments. We kindly clarify that we adopt **Stable Diffusion v1.5 (SD v1.5)** since most existing text-guided image inpainting methods fine-tune their models based on SD v1.5. To ensure a fair comparison, we mainly present the inpainted results based on SD v1.5 in the mainbody. \n\nFollowing your suggestion, we additionally conducted the inpainted results of other representative text-to-image (T2I) models, including the DiT-based method **HunyuanDiT** and **Stable Diffusion XL (SDXL)**, which is an upgraded version of SD v1.5. We report the quantitative results on **BrushBench**, covering both **inside** and **outside** inpainting tasks, along with detailed explanations, as shown below:\n\n| Method| $IR_{×10}$ ↑ | $HPS_{×10²}$ ↑ | CLIP Score ↑ | \n|---------------------|:-------:|:-----------:|:------------:|\n|BrushBench (Inside) - **SD v1.5**|9.61|27.52|26.90|\n|BrushBench (Inside) - **SDXL**| 11.85 |28.10 |27.92 |\n|BrushBench (Inside) - **HunyuanDiT**| 12.07| 28.36 | 28.06 |\n||||||\n|BrushBench (Outside) - **SD v1.5**| 5.29 | 25.31  | 26.12 |\n|BrushBench (Outside) - **SDXL**|7.43 | 26.68 |26.84|\n|BrushBench (Outside) - **HunyuanDiT**| 7.69 | 26.82 |26.98|\n||||||\n\n* **Analysis of the improvements achieved on powerful T2I models**\n\n*\"We adopt the IR and HPS to evaluate the image generation quality, and use the CLIP score to assess the text-image alignment, which is in line with **Lines 270–277, Sec. 3.1 of the mainbody**. The results show that both HunyuanDiT and SDXL outperform SD v1.5 on BrushBench. This improvement can be attributed to their significantly larger parameter scales and training on more diverse and extensive datasets. As a result, in the latent space, the features of the unmasked regions retain more information after downsampling. This enables our frequency-aware null-text-null diffusion framework to exchange richer mid-and-low frequency components during the early stage of the denoising process, which is consistent to the motivation in **Sec. 2.3 of the mainbody**.\"*\n\n---\n\n# [W3]: Writing and Presentation\n\n**Response**: Thanks for the comments, we will fix all your suggestions regarding writings issues, such as long sentences, small figure and typos etc. \n\n\n---\n# References\n \n- [1] Avrahami O, Fried O, Lischinski D. Blended latent diffusion[J]. ACM transactions on graphics (TOG), 2023, 42(4): 1-11.\n \n- [2] Zhuang J, Zeng Y, Liu W, et al. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 195-211.\n \n- [3] Ju X, Liu X, Wang X, et al. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 150-168.\n \n- [4] Manukyan H, Sargsyan A, Atanyan B, et al. Hd-painter: high-resolution and prompt-faithful text-guided image inpainting with diffusion models[C]//The Thirteenth International Conference on Learning Representations. 2025."}]

Author rebuttal:

Meta-review: After a comprehensive review of all reviewers' comments, we have decided to accept this paper. The paper's strengths, including its novelty, technical intuition and depth, as well as superior experimental results, have been generally appreciated by all reviewers.​

Regarding the thoughtful questions raised by the reviewers, the authors have provided clear clarifications and validations, which have received positive confirmation from all reviewers. It is noted that most of the answers were already available in the mainbody and Appendix of the manuscript, as gently reminded. Specific examples include clarifications on experiments related to large masks and EditBench, discussions on the second null-text denoising process at the early stage, and explanations on the adaptive extraction of low-and-mid frequency bands and unmasked region preservation, all of which have been addressed with references to relevant sections and tables.​
Furthermore, the authors have supplemented more consistent experimental results as suggested by the reviewers, covering aspects such as computational efficiency, algorithm pseudo-code, experiments on changes in low-and-high frequency, and experiments on more T2I models. These supplements have been well-received, with three reviewers upgrading their ratings and the remaining one maintaining their initial rating.​

We sincerely appreciate the authors' efforts in addressing the reviewers' concerns. This paper is thus recommended for acceptance.
Decision: Accept (poster)