TD-Paint: Faster Diffusion Inpainting Through Time Aware Pixel Conditioning

## Abstract

Abstract Diffusion models have emerged as highly effective techniques for inpainting, however, they remain constrained by slow sampling rates. While recent advances have enhanced generation quality, they have also increased sampling time, thereby limiting scalability in real-world applications. We investigate the generative sampling process of diffusion-based inpainting models and observe that these models make minimal use of the input condition during the initial sampling steps. As a result, the sampling trajectory deviates from the data manifold, requiring complex synchronization mechanisms to realign the generation process. To address this, we propose Time-aware Diffusion Paint (TD-Paint), a novel approach that adapts the diffusion process by modeling variable noise levels at the pixel level. This technique allows the model to efficiently use known pixel values from the start, guiding the generation process toward the target manifold. By embedding this information early in the diffusion process, TD-Paint significantly accelerates sampling without compromising image quality. Unlike conventional diffusion-based inpainting models, which require a dedicated architecture or an expensive generation loop, TD-Paint achieves faster sampling times without architectural modifications. Experimental results 1 1 1 Our code is provided in supplementary materials. across three datasets show that TD-Paint outperforms state-of-the-art diffusion models while maintaining lower complexity.

## 1 Introduction

Figure: Figure 1: (left) Illustration of the RePaint [] generation process. The inpainting generation process of RePaint applies a cycle of backward ($q(x_{t}|x_{t-1})$ purple lines —) and forward ($p_{\theta}(x_{t-1}|x_{t})$ green lines —) diffusion steps to the samples. Several cycles are needed to synchronize the condition to which noise is added and the current generation. It can be observed that the intermediate steps of the generation process lack consistency, changing from a man with dark hair at $\hat{x}_{0,T-1}$ to a man with blond hair at $\hat{x}_{0,t}$ to a woman at $\hat{x}_{0,t+k}$. These changes occur due to the synchronization process, where the initially predicted images $\hat{x}_{0,t}$ are not well aligned with the given condition.(right) Illustration of the TD-Paint generation process.In comparison, TD-Paint can use a clean condition from the beginning of the inpainting process, resulting in a faster and more stable process. Note how the intermediate TD-Paint steps are consistent from one to another.
Refer to caption: x1.png

Given an image and a binary mask, image inpainting aims to generate the missing region while preserving the semantics of the visible region. This task is challenging because the generated content must not only be coherent with the existing parts of the image but also appear realistic. Additionally, the generation process should be stochastic to produce diverse outputs. An effective inpainting model must also address variations in mask shape and size. Generalizing to unseen masks during training and accurately filling large missing regions further complicates the task.

Diffusion models have shown significant success as generative models [, ], by approximating the distribution of real images through a fixed Markov chain that transforms Gaussian noise into the real image distribution. During training, a forward diffusion process gradually adds noise to an image, and the model is trained to reverse this process, learning to denoise and recover the original image distribution.
During generation, the backward diffusion iteratively removes noise from an initial Gaussian noise image. The trained model predicts and removes noise at each step, gradually refining the image until a photorealistic result is achieved.

Methods proposed in the literature have investigated using the standard diffusion model [, ] for inpainting by combining the noisy condition with the current generation at each step.
This technique has its limitations. While the textures match, it creates disharmony between the conditioning and generation parts.
This disharmony comes from the fact that, during the early steps of the generation process, the condition contains a lot of noise that the model cannot leverage. Therefore, the generation moves away from the intended semantics and produces unsatisfactory results.
An illustration of such inpainting can be found in Figure 1(left).

To address this limitation, RePaint [] introduced a resampling mechanism that repeats the diffusion steps multiple times, enabling the model to synchronize condition and generation better. Although RePaint produces highly faithful images, this resampling significantly slows the generation process. Indeed, RePaint requires approximately 5k steps to generate a single image, increasing time complexity. An illustration of RePaint’s generative process is shown in Figure 1 (left).
Other approaches introduce additional constraints at each diffusion step[, ]. For example, [] integrates a correction mechanism that encourages the diffusion path to remain close to the data manifold. This is achieved by minimizing the reconstruction error of the known image region relative to the unknown region.

In contrast to models that use a noisy condition[, ], our Time-aware Diffusion Paint (TD-Paint) approach integrates the currently generated sample with a clean condition. Our method uses semantic information from the beginning of the generation process, resulting in a more efficient and cost-effective approach.
To achieve this, TD-Paint uses time conditioning derived from the standard formulation of diffusion models. Instead of using a single scalar $t$ for the entire image, TD-Paint assigns a unique $t$ value to each pixel. The known image region is assigned a smaller $t$, indicating lower noise levels. In contrast, the region to be generated is assigned a $t$ value proportional to the current step in the generation process. An illustration of TD-Paint’s generative process is provided in Figure 1(right).

Our main contributions are summarized as follows:
•  We propose a novel noise modeling paradigm for diffusion models that allows for integrating varying noise levels into the input of diffusion models. TD-Paint exploits time conditioning in diffusion models to achieve faster and higher-quality generation.
• An extensive set of experiments on the challenging CelebA-HQ, ImageNet1K and Places2 datasets demonstrates that our TD-Paint, not only outperforms state-of-the-art diffusion-based models but also surpasses other inpainting methods, including those based on CNNs and transformers. Additionally, the results indicate TD-Paint is a more cost-effective solution for diffusion models.

## 2 Related Works

Inpainting aims to fill in a missing part of the image realistically. Traditional inpainting methods try to combine techniques to propagate texture and structural information onto the missing parts []. Some algorithms [] use large image datasets and assume that the possible semantic space for missing regions is limited.
In recent years, deep learning models for inpainting have made impressive progress using two types of generative models, VAEs [] and GANs [, ].

(a) Single-Stage Inpainting:
Most single-stage methods use the context encoding setting introduced by Pathak et al. [], with an encoder-decoder setup. A reconstruction loss (L2) ensures global structure consistency, while an adversarial loss ensures the reconstruction is realistic.
Global consistency is an important consideration. CNNs are limited by a receptive field that grows slowly with network layers. Many layers are needed for information to travel from one side of the image to the other.
Dilated convolution [] has been used by [] to increase the receptive field.
Partial convolution [] uses mask information to attend only the visible regions.
The pyramid context encoder [] learns an affinity map between regions in a pyramidal fashion.
Fourier convolution [] aims to provide a global receptive field to both the inpainting network and the loss function. Fast Fourier Convolutions have an image-wide receptive field, which helps with large missing areas.
Mask-Aware Transformer (MAT) [] is a transformer-based architecture that allows the processing of high-resolution images. A customized transformer block considers only valid tokens, and a style manipulation module updates convolution weights with noise to produce diverse outputs.

(b) Progressive Image Inpainting:
These methods seek to address global consistency by using coarse-to-fine multi-stage approaches.
Multiple generations are possible for a large missing region in single-stage training. Some may have a large pixel-to-pixel distance from the original ground truth, which can be misleading when training models with pixel-wise distance losses.
To mitigate this problem, Yun et al. [] proposed a two-stage generative approach, where the first stage optimizes a coarse result using L1 loss with spatial discount, and the second stage further refines the result using global and local critics.
Gated convolution [] has been used in a coarse-to-fine network to learn valid pixels [].

(c) Prior Knowledge Inpainting:
These methods leverage and mine information from generative models.
Prior Guided GAN (PGG) [] uses the latent space of a pre-trained GAN and learns to map masked images to this space using an encoder. A masked image can be mapped to a latent code during inference, and the generator can produce a corresponding inpainted image.
Deep Generative Prior [] relaxes the frozen generator assumption of GAN inversion methods and proposes progressively refining each layer.
PSP [] uses StyleGAN [] latent space to encode an image into its latent space. Inpainting is formulated as a domain translation task performed in the latent space of StyleGAN, removing adversarial components from the training process.

(d) Diffusion Model Inpainting:
While GANs have recently shown impressive results, most applications are limited to generating a specific domain. In contrast, Diffusion models have gained traction for image generation; Denoising Diffusion Probabilistic Model (DDPM) [] and Denoising Diffusion Implicit Models (DDIM) [] can generate very diverse and high-quality images.
Some unconditional diffusion models have shown the ability to perform zero-shot inpainting [, ] but provide only qualitative results.
The Pixel Spread Model (PSM) [] uses a decoupled probabilistic model that combines the efficiency of GAN optimization with the prediction traceability of diffusion models.
It also employs a specific design architecture.
Latent Diffusion Models (LDM) [] decouple two tasks: image processing and compression, and the denoising process is learned in latent space instead of pixel space.
Inpainting is performed by encoding the masked input image, downsampling the inpainting mask, and concatenating them as additional conditions to the denoising model.
Designing an image and mask conditional diffusion model requires a special architecture to accept additional input for inpainting, as done in [, ], and often treats inpainting as a domain translation task.
In contrast, TD-Paint does not require any architecture modification by directly combining the masked input and current generation with different noise levels.
The Manifold Constrained Gradient (MCG) [] adds a correction term to ensure each sample step remains close to the data manifold, allowing for more stable inpainting.
RePaint [] is close to our work. It combines the noisy condition region with the current generation. The diffusion model updates the missing pixel based on the surrounding context.
We observe that during the early inpainting step of RePaint, the condition is dominated by noise and does not contain any semantic information.
This causes the model prediction to deviate from the target manifold (see Figure 1(left)).
A resampling mechanism is needed to synchronize the condition and generation regions, allowing semantically corrected images to be produced at the cost of significantly increasing computation time.

Instead of degrading the condition to the same level as the generation, we propose keeping it clean and conditioning the missing part on the known pixels from the beginning of the generation process (see Figure 1(right)).
Although it requires fine-tuning the model, our approach provides much faster sampling by avoiding resampling [] or additional constraint [].
We use temporal information to model the amount of detail in the image.
The condition, which contains clean data, is assigned a low noise level while the generation region starts with maximum noise that gradually decreases during the process. This approach allows for a direct combination of the clean and generated regions in the same space without changing the model architecture.

## 3 Background On Inpainting Diffusion Models

Diffusion Models:
Diffusion models learn a data distribution from a training dataset by inverting a noise process.
During training, the forward diffusion process transforms a data point $x_{0}$ into Gaussian noise $x_{T}\sim\mathcal{N}(0,\mathbf{I})$ in $T$ steps by creating a series of latent variables $x_{1},...,x_{T}$ using:

$$ $q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{ I}),$ (1) $$

where $\beta_{t}$ represents the predefined variance schedule. Given $\alpha_{t}=1-\beta_{t}$, $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$, and $\epsilon\sim\mathcal{N}(0,\mathbf{I})$, $x_{t}$ at step $t$ can be marginalized from $x_{0}$ using the reparametrization trick as follows:

$$ $x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon.$ (2) $$

The reverse denoising process $p_{\theta}(x_{t-1}|x_{t},t)$ allows to generate from the data distribution by first sampling from $x_{T}\sim\mathcal{N}(0,\mathbf{I})$ and iteratively reducing the noise in the sequence $x_{T},...,x_{0}$.
The model $\epsilon_{\theta}(x_{t},t)$ is trained to predict the added noise $\epsilon$ to produce the sample $x_{t}$ at time step $t$. The model is trained using mean square error (MSE):

$$ $\mathcal{L}=\mathbb{E}_{\epsilon\sim\mathcal{N}(0,\mathbf{I}),x_{0},t}\| \epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}} \epsilon,t)-\epsilon\|^{2}_{2}.$ (3) $$

RePaint Methodology:
Given an image $x$ and a binary mask $m$, the goal of inpainting is to generate the missing region $x^{\text{$\ominus$}}$ specified by $x\odot(1-m)$ conditioned on the known region $x^{\text{$\oplus$}}$ specified by $x\odot m$.
For this, RePaint combines a noisy version of the condition $x_{0}\odot m$ with the previous output of the generation process:

$$ $x^{\text{$\oplus$}}_{t-1}\sim\mathcal{N}\left(\sqrt{\bar{\alpha}_{t}}x_{0},(1- \bar{\alpha}_{t})\mathbf{I}\right)$ (4) $$

$$ $x^{\text{$\ominus$}}_{t-1}\sim\mathcal{N}\left(\mu_{\theta}(x_{t},t),\Sigma_{ \theta}(x_{t},t)\right)$ (5) $$

$$ $x_{t-1}=x^{\text{$\oplus$}}_{t-1}\odot m+x^{\text{$\ominus$}}_{t-1}\odot(1-m)$ (6) $$

RePaint uses the known pixels as a noisy condition to guide the generation of the unknown pixels.
In the first inpainting step, inputs contain a high noise level and limited information about the condition. This leads to samples that deviate from the intended semantics of the condition, often resulting in artifacts. To address this, RePaint introduces a resampling mechanism that harmonizes the two semantics by applying forward diffusion on the output $x_{t-1}$ back to $x_{t+j}$. The denoising and re-noising process involves executing the same diffusion step multiple times during the generation phase, sacrificing computational efficiency to achieve higher image quality.

## 4 The Proposed TD-Paint Method

In TD-Paint, the model is conditioned on known and unknown regions using the time step $t$, already present in diffusion models.
Instead of using $x^{\text{$\oplus$}}_{t-1}$, $x_{0}$ is combined directly with $x_{t}$ without forward diffusion. This results in a faster diffusion process without changing the model architecture.

### 4.1 Noise Modeling for Fast Inpainting

Training Process: The objective of TD-Paint is to enable the diffusion model to discern the information content of each input region. This allows the model to differentiate between conditioned regions and those needing to be painted. By maintaining the known regions free of additional noise, TD-Paint preserves the maximum amount of information in these areas.
Using the time step $t$ allows for smooth and continuous modeling of information content. Regions from the known part of the image are given a $t$ value close to zero, indicating minimal noise. On the other hand, regions that need to be inpainted start with a $t$ value close to $T$, which progressively decreases during the generation process.
To do so, $t\in\{0,...,T\}$ is transformed from a scalar to a tensor $\tau\in\{0,...,T\}^{h\times w}$.
Similarly, other variables are accommodated to perform a pixel-wise diffusion process. With one $t$ per pixel, $\alpha_{t}$ (resp. $\bar{\alpha}_{t}$, $\beta_{t}$) becomes $\alpha_{\tau}$ (resp. $\bar{\alpha}_{\tau}$, $\beta\tau$).
This innovative modification can be integrated into most diffusion model training pipelines. Figure 2 illustrates the intermediate images used for training.
During training, each input image pixel $x_{i,j}$ receives an amount of noise controlled by $\tau_{i,j}$. The forward diffusion process is then applied to $x$ on a pixel-wise basis, as illustrated in Figure 2 and can be formulated as:

$$ $x_{\tau}=\sqrt{\bar{\alpha}_{\tau}}x_{0}+\sqrt{1-\bar{\alpha}_{\tau}}\epsilon,$ (7) $$

in which $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, $\tau\sim\phi_{\text{train}}$ and $\phi_{\text{train}}$ is a training strategy to sample different noise per pixel.
The diffusion network then predicts $\epsilon$, using less noisy regions to reconstruct more noisy regions by optimizing the loss:

$$ $\mathcal{L}=\left\|\epsilon-\epsilon_{\theta}(x_{\tau},\tau)\right\|^{2}_{2}.$ (8) $$

The final component of TD-Paint training is the strategy used for $\phi_{\text{train}}$. A random patch size and a proportion of patches are sampled to define a condition. The input image is then divided into known and unknown regions based on this sampled proportion. The possible patch sizes are defined as powers of two, i.e., ${2^{i}|i\in\mathbb{N},2^{i}\leq w}$, up to the maximum size of the image.
The fraction of pixels designated as the known region is represented by a real value within the interval $[0,1]$, ensuring that at least one patch remains in the unknown region. For example, as illustrated in Figure 2 for a patch size of $128$, $25\%$ of the pixels are assigned to the known region.

Generation Process: Inpainting an image with TD-Paint involves sampling a time $t$ for the conditioning region $x^{\text{$\oplus$}}$ and a time $t$ for the inpainted region $x^{\text{$\ominus$}}$ using $\phi$.
Unless otherwise specified, we set $\phi^{\text{$\oplus$}}_{t}=0$ for the condition and $\phi^{\text{$\ominus$}}_{t}=t$ for the region to inpaint for training and generation.
For generation, the known region $x^{\text{$\oplus$}}$ is merged with the current unknown region $x^{\text{$\ominus$}}_{t}$, and one reverse step can be expressed as:

$$ $x_{\tau}=x^{\text{$\ominus$}}_{t}\odot(1-m)+x^{\text{$\oplus$}}_{0}\odot m$ (9) $$

$$ $x^{\text{$\ominus$}}_{t-1}\sim\mathcal{N}\left(\mu_{\theta}(x_{\tau},\tau), \Sigma_{\theta}(x_{\tau},\tau)\right).$ (10) $$

This approach allows the use of input-known regions from the beginning of the diffusion process. Consequently, we can eliminate RePaint’s resampling mechanism, resulting in faster inpainting.
The general algorithm for inpainting an image with an arbitrary $\phi$ function is given by Algorithm 1.

Figure: Algorithm 1 TD-Paint Generation Process.

Figure: Figure 2: Illustration of the TD-Paint patch-wise training procedure. Each region of the input $x_{0}$ receives a different level of noise controlled by $\tau$. The network uses less noisy regions to reconstruct more noisy regions.
Refer to caption: x3.png

### 4.2 Time-aware Diffusion Architecture

Diffusion models using U-Net architectures [] can incorporate a time map $\tau$ without requiring architectural modifications. This adaptation is achieved by applying pixel-wise time conditioning to each pixel in the feature maps.

Classical time conditioning first uses a position embedding layer to obtain a time embedding $\gamma\in\mathbb{R}^{d}$ from the time step $t$:

$$ $\gamma=L\left[E(t)\times\sigma(L(E(t)))\right],$ (11) $$

where $L$ denotes linear layers, $\sigma$ is the sigmoid function, and $E$ is a sinusoidal timestep embedding.
Additionally, at each layer of the U-Net, the vector $\gamma$ is used to perform time conditioning with scale-shift normalization and can be written as:

$$ $h^{l+1}_{i,j}=GN(h^{l})_{i,j}\times(1+L_{\text{scale}}(\gamma))+L_{\text{shift }}(\gamma),$ (12) $$

where $h^{l}\in\mathbb{R}^{c_{l}\times h_{l}\times w_{l}}$ are the current features for layer $l$, $GN$ is a group normalization layer, while $L_{\text{scale}}$ and $L_{\text{shift}}$ are linear layers that change the dimension of $\gamma$ from $d$ to $c_{l}$.
We apply the pixel-wise time conditioning across each spatial dimension of $h^{l}$ by using $\tau_{i,j}$ instead of $t$ as follows:

$$ $\Gamma_{i,j}=L(E(D(\tau)_{i,j})\times\sigma(L(E(D(\tau)_{i,j}))))$ (13) $$

$$ $h^{l+1}_{i,j}=GN(h^{l})_{i,j}\times(1+L_{\text{scale}}(\Gamma_{i,j}))+L_{\text {shift}}(\Gamma_{i,j}),$ (14) $$

where $\Gamma\in\mathbb{R}^{d\times h\times w}$ is the embedding of time $\tau$, and $D$ is a downscaling operation function which rescales $\tau$ to $h_{l}\times w_{l}$(^2^22We use bilinear interpolation, but min-pool or other techniques could be used with similar effect.).

## 5 Results and Discussion

This section empirically shows that TD-Paint:
(a) produces high-quality inpainting with a clean condition on various mask sizes and shapes, on par or better than other inpainting models;
(b) provides more efficient sampling, making it faster than other diffusion-based models without requiring a dedicated architecture;
(c) generates diverse, high-quality images.
Details on the training masks and their corresponding ablation study are presented in Appendix A, and additional qualitative results are provided in Appendix C.

Figure: Figure 4: Qualitative results: TD-Paint against state-of-the-art inpainting methods on CelebA-HQ. Zoom in for better details. Additional examples can be found in Appendix C.
Refer to caption: x5.png

Figure: Figure 5: Qualitative results: TD-Paint against state-of-the-art inpainting methods on ImageNet1K. Additional examples can be found in Appendix C.
Refer to caption: x6.png

Figure: Figure 6: Qualitative results: TD-Paint against state-of-the-art inpainting methods on Places2. Additional examples can be found in Appendix C.
Refer to caption: x7.png

### 5.1 Experimental Methodology

Baselines:
We evaluate TD-Paint against state-of-the-art diffusion-based inpainting methods: RePaint [], which conditions the generative process using noisy inputs and synchronizes them with the output through resampling. MCG [], which adds a correction term to keep the generation closer to the data manifold, and
CoPaint [] that utilizes Tweedie’s formula for better generation.
Recent studies [, ] have shown that CoPaint and MCG are among the best-performing inpainting methods.
We denote the RePaint-20 model as using 20 resampling steps, while RePaint-1 refers to the model with 1 resampling step, matching the number of steps used by TD-Paint.
Additionally, TD-Paint is compared with state-of-the-art mask and image conditioned inpainting models: the CNN-based LaMa [] and transformer-based MAT [].

Implementation Details:
Our approach is validated using the CelebA-HQ [] dataset, the ImageNet1K [] dataset, and the Places2 dataset [] at 256x256 resolution.
We modify the implementation of [], maintaining all their hyperparameters. Training on CelebA-HQ is conducted for approximately 150K steps with batch size 64 on 4 A100, for ImageNet1K and Places2 for about 200K steps with batch size 128 on 8 A100.
For baselines, we utilize existing code and pre-trained models when available.
For ImageNet1K, we train LaMa for 1M steps on batch size 5 using their implementation, and MAT for 300K steps on batch size 32 using their implementation.

Evaluation Metrics:
Image quality is evaluated using established metrics from the inpainting literature: the Learned Perceptual Image Patch Similarity [] (LPIPS), the Structural Similarity Index Measure [] (SSIM), and the Kernel Inception Distance [] (KID) (using the TorchMetrics [] implementation).
The number of diffusion steps (NBS) and the mean time to inpaint an image (Runtime) are used to evaluate the computational efficiency of TD-Paint.
For evaluation, we use 2,824 images from the CelebA-HQ test set, 5,000 images from ImageNet1K, and 2,000 images from Places2.

### 5.2 Comparison with Diffusion-Based Models

Wide and Narrow masks
In the standard image inpainting scenario, TD-Paint is compared using Wide and Narrow masks following LaMa [] protocol.
Tables 1 and 3 shows that TD-Paint consistently outperforms other diffusion-based models, improving by 20% RePaint-20 LPIPS’s on the Wide mask on CelebA-HQ, ImageNet1K and Places2, and by 30% on the Narrow masks.
MCG lacks global consistency, resulting in significant artifacts on Wide masks, as seen in Figures 4 and 5 where it produces eyes of different colors, strange textures, and inpainting artifacts. RePaint produces high-quality images at the cost of significant inference time. Our approach can produce high-quality images while requiring much less processing time.

Super-Resolve 2x and Altern. Lines masks
In the Super-Resolve 2x setting, every other pixel is removed from the image, while the Altern. Lines setting removes every other line. All diffusion-based baselines achieve low LPIPS scores and produce high-quality images for both types of masks (see Figures 4, 5 and 6).
TD-Paint outperforms all considered baselines on CelebA-HQ and comes a close second to CoPaint on ImageNet1K and Places2, as shown in Tables 1 and 3.
The third-best performing model is RePaint. Compared to RePaint, TD-Paint improves LPIPS by 115% in the Super-Resolve 2x setting and by 69% in the Altern. Lines setting.

Half and Expand masks
The Half setting removes the right part of the images, and the Expand setting keeps only the central 64$\times$64 part of a 256$\times$256 pixel image.
Both LPIPS and SSIM metrics are less suitable when a significant portion of the image is missing, as they rely on a single ground truth. This penalizes methods that generate realistic images with semantics different from the ground truth [].
In this case, the KID, which measures the distance between distributions, is more reliable for assessing image quality.
Applying Half and Expand masks is particularly challenging, as they remove substantial portions of the images. task where an important part of the images is removed. This complexity is shown both visually in Figures 4, 5 and 6 and by the quantitative results in Tables 1 and 3.
Our model achieves the best performance on both the Half and Expand masks across all datasets, as measured by the KID, while significantly reducing the inference time.
Figure 4 shows that TD-Paint can produce high-quality images in this challenging setting where RePaint lacks global consistency.
On ImageNet1K and Places2 (see Figures 5 and 6), we observe that MCG and CoPaint often mirror the images from the Half mask, resulting in symmetrical outputs.
Similar behavior is observed on Expand masks, with significant texture blending.

Table: Table 1: Quantitative results: LPIPS and SSIM evaluation of diffusion models for inpainting on the CelebA-HQ and ImageNet1K datasets.

Table: Table 2: Quantitative results: evaluation of CNN- and transformer-based models for inpainting on the CelebA-HQ and ImageNet1K datasets.

Table: Table 3: Quantitative results: KID evaluation of diffusion models for inpainting on the CelebA-HQ and ImageNet1K datasets.

### 5.3 Comparison with CNN- and Transformer-Based Models

Figure: Figure 7: Examples of diverse generations using TD-Paint on the CelebA and ImageNet, with the same input image and different initial noise. Additional examples are available in Appendix C.
Refer to caption: x8.png

We compare TD-Paint with LaMa (CNN-based) and MAT (transformer-based) models in Table 2.

Wide and Narrow masks
Our approach closely matches the performance of LaMa and MAT in the Wide setting on CelebA-HQ and even surpasses them in the Narrow setting on CelebA-HQ, as well as in both the Wide and Narrow settings on ImageNet1K. As shown in Figure 4, LaMa tends to generate pupils of different sizes when one eye is hidden in the Wide and different eye colors in the Narrow settings.

Super-Resolve 2x and Altern. Lines masks
Table 2 shows that TD-Paint outperforms the baselines by a wide margin. Particularly MAT struggles with this task and often produces images with significant artifacts and blurring (see Figures 4, 5 and 6). In contrast, TD-Paint consistently achieves lower LPIPS scores and higher SSIM values, reflecting superior image quality and performance.

Half and Expand masks
On CelebA-HQ, TD-Paint achieves the best results across all datasets, as indicated by the KID metrics (see Table 3).
As shown in Figures 4, 5 and 6, LaMa generates blurry artifacts in both the Half and Expand settings, whereas our proposed model consistently produces high-quality images.
This behavior of LaMa may be due to overfitting to the training mask distribution, as suggested by previous studies [].

Table: Table 4: Inpainting speed for different methods.

### 5.4 Quality vs. Efficiency

We compare the time efficiency of different approaches by computing the average time to sample 100 images consecutively on a single V100 and the results are reported in Table 4.
State-of-the-art approaches require over $6\times$ longer to sample compared to TD-Paint. The increased time in RePaint is due to the resampling mechanism needed to synchronize condition and generation, while MCG requires an additional backward pass and more steps to optimize the image.
In contrast, our model reduces inference time by trading fine-tuning costs, enabling faster generation of high-quality images than other diffusion-based inpainting models.

### 5.5 Diversity of Generated Images

While TD-Paint performs fast and high-quality inpainting, we must ask whether this comes at the cost of diversity. To evaluate this, we compute the Diversity Score [] by generating 10 different images for 100 inputs. The quantitative results reported in Table 5.
The most diverse model is RePaint-1, which also has a high LPIPS score. In contrast, TD-Paint achieves a high Diversity Score across most masks while consistently producing high-quality images (see Figure 7) with lower LPIPS scores.

Table: Table 5: Diversity Score on CelebA-HQ with 10 random generations across 100 images.

## 6 Conclusions

In this paper, we introduce Time-aware Diffusion Paint (TD-Paint), a method that accelerates inpainting by modeling multiple noise levels through time conditioning in the diffusion process. Unlike other diffusion-based models, TD-Paint does not require any special architecture for inpainting, and generates high-quality, diverse images more quickly. This efficiency makes TD-Paint highly practical for real-world applications and usable for resource-constrained devices.

## Acknowledgments

This work was financially supported by the ANR Labcom LLisa ANR-20-LCV1-0009.
We thank our colleagues at CRIANN, who provided us with the computation resources necessary for our experiments.
This work was performed using HPC resources from GENCI–IDRIS (Grant 2024-AD011013862R1).
We thank the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Digital Research Alliance of Canada.

## Appendix A Ablations: Training Mask

We examine the contribution of masks during training on the CelebA-HQ dataset using our mask strategy (Ours) described in Section 4.1, LaMa masks (LAMA), and a random mix of the two (Ours+LAMA). Unless stated otherwise, all results are reported with Ours+LAMA.
Table 6 shows that using LAMA over Ours reduces the error for every test mask except Super-Resolve 2x and Altern. Lines, which is explained by the more complicated design of LAMA, which produces train masks closer to the Wide and Narrow test masks and of more diverse shapes.
Notably, using Ours+LAMA masks still allows for very low Super-Resolve 2x and Altern. Lines errors compared to LaMa errors on the same masks in Table 2.
Using Ours+LAMA allows the benefits of Ours masks to be retrained while having low Super-Resolve 2x and Altern. Lines errors.

Table: Table 6: Ablation study for the types of training time map. The metrics show how the use of Ours focuses more on very fine inpainting masks. The distribution of LAMA masks is closer to larger inpainting masks, such as Wide. Combining the two allows for strong performance across the range of test masks considered.

## Appendix B Image Quality and Diffusion Steps

We compare the LPIPS metrics over the diffusion step for TD-Paint, RePaint and MCG in Figures 8 and 9 averaged over 100 images from the CelebA-HQ dataset.
TD-Paint can produce high-quality images in a fraction of the steps required by RePaint because it takes advantage of the state since the early step of the diffusion process.

Figure: Figure 8: Image quality at different time step for 100 images on CelebA-HQ dataset for Wide and Half masks.
Refer to caption: x9.png

Figure: Figure 9: Image quality at different time step for 100 images on CelebA-HQ dataset for Super-Resolve 2x mask.
Refer to caption: x10.png

## Appendix C Additional Qualitative Results

We provide additional qualitative inpainting results compared to the state-of-the-art models described in Section 5.

For CelebA-HQ on
Wide and Narrow masks in Figure 10,
Super-Resolve 2x and Altern. Lines masks in Figure 11,
Half and Expand in Figure 12.

For ImageNet1K on
Wide and Narrow masks in Figure 13,
Super-Resolve 2x and Altern. Lines masks in Figure 14,
Half and Expand in Figure 15.

For Places2 on
Wide and Narrow masks in Figure 16,
Super-Resolve 2x and Altern. Lines masks in Figure 17,
Half and Expand in Figure 18.

Additional diversity results on CelebA-HQ and ImageNet1K can be found in Figure 19, and for ImageNet1K with different conditional classes in Figure 20.
Through class conditioning, TD-Paint can generate diverse images based on different target classes. This mechanism allows TD-Paint to guide the inpainting process toward specific semantic categories, resulting in more controlled and contextually relevant image completions. This feature is especially useful when the inpainted region must match a particular class, increasing TD-Paint’s flexibility and effectiveness across a range of inpainting tasks.

Figure: Figure 10: CelebA-HQ qualitative results
Refer to caption: x11.png

Figure: Figure 11: CelebA-HQ qualitative results
Refer to caption: x12.png

Figure: Figure 12: CelebA-HQ qualitative results
Refer to caption: x13.png

Figure: Figure 13: ImageNet1K qualitative results
Refer to caption: x14.png

Figure: Figure 14: ImageNet1K qualitative results
Refer to caption: x15.png

Figure: Figure 15: ImageNet1K qualitative results
Refer to caption: x16.png

Figure: Figure 16: Places2 qualitative results
Refer to caption: x17.png

Figure: Figure 17: Places2 qualitative results
Refer to caption: x18.png

Figure: Figure 18: Places2 qualitative results
Refer to caption: x19.png

Figure: Figure 19: Example of divers generation using TD-Paint on CelebA-HQ and ImageNet1K using the same input image and different initial noise.
Refer to caption: x20.png

Figure: Figure 20: ImageNet1K TD-Paint diversity qualitative results using different class conditioning. For a line, TD-Paint is prompted with the same input image and mask but, with different classes.
Refer to caption: x21.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: How can the proposed method be used to scale up to higher resolution images? For instance, is there a straightforward way to extend to LDMs, as some of the more recent inpainting approaches have been doing?
soundness: 3
strengths: The proposed approach is quite simple but at the same time intuitive and effective. The idea of varying the diffusion at a pixel level is a promising one, both for novel applications (such as inpainting) as well as for a more general approach to accelerating diffusion inference.
summary: This paper proposes a way to perform inpainting by leveraging pre-trained pixel diffusion model. The authors fine-tune a diffusion model such that instead of inputting a single scalar that represents the tilmestep of the noise in the diffusion process, the model inputs a map, representing per-pixel tilmesteps. Then, at inference, this allows them to always input the fully denoised image in the known regions while iteratively denoising the masked areas. Unlike the main baseline, RePaint, this process is much faster, as it does not require multiple diffusion passes to synchronize the known and unknown regions.
weaknesses: My main concern has to do with lack of comparison to or discussion of some more recent and relevant SOTA diffusion-based inpainting approaches. For instance, LatentPaint [Corneanu et al., WACV 2024] also a propose an approach that requires fine-tuning with a fast inference pass (compared to RePaint). Additionally, "Blended Latent Diffusion" [Avrahami et al., SIGGRAPH 2023] and "Uni-paint" [Yang et al., MM 2023] leverage latent diffusion models for an inpainting task (while the former does not explicitly target unconditional inapinting, it ca be used for that purpose).  It would be very helpful to see comparisons to some of these recent works as well as a high-level discussion of their differences in order to evaluate the proposed method.

On a more minor note, the presentation and exposition could be improved. Figure 1, which illustrates the proposed approach in contrast to RePaint, is somewhat difficult to follow and includes notation that is not defined until much later. In contrast, the idea behind RePaint is explained multiple times in various sections throughout the paper and could be truncated.

I would also encourage the authors to cite and discuss "Differential Diffusion: Giving Each Pixel Its Strength" [Levin and Fried, 2023]. Given that this work does not appear in a peer-reviewed publication, I am not considering it in my rating, but given its close relevance to the topic, it would be good to mention.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Training Inference Gap: The training is done on patch-wise known regions. But inference of the known region doesn’t necessarily be axis-aligned patches. How do you solve this gap? Will training on randomly shaped known regions further boost the model’s performance?
2. I wonder how well the proposed method performs when adapted to larger diffusion models like stable diffusion.
3. The ControlNet-Inpaint model also provides fast and high-quality diffusion-based inpainting. How does the proposed method perform compared to that model?

I would raise my score if these questions were answered.
soundness: 3
strengths: 1. Training the diffusion model to directly use clean known parts as the diffusion condition, avoiding complex modifications to the diffusion process. This approach significantly speeds up the method compared to existing approaches, enhancing efficiency without compromising quality.
2. Introduction of a novel spatial-varying timestep condition for injecting the known region mask. This innovative technique allows for more precise control over the inpainting process, resulting in improved output quality.
3. Comprehensive testing on multiple diverse datasets, demonstrating superior performance (for both quality and diversity) across various scenarios. The extensive evaluation provides strong evidence for the method's robustness and effectiveness in different contexts.
summary: The paper presents a novel method for speeding up diffusion-based inpainting. It changes the noised known region condition in the RePaint paper into a clean one for condition. To support this change, it introduces a trained diffusion model that supports a spatial-varying timestep condition to mark the known and missing part of the image. This new design also allows the method to avoid the repeated sampling process in RePaint, greatly increasing the speed. Quantitative results also demonstrate the proposed method performs on par or better with existing methods.
weaknesses: The paper is in general good, I only have concerns on its generality: 

Since the proposed method requires training and is not trained with data besides CelebA-HQ, ImageNet1K, and Places2, it probably cannot generalize to open-world images like LaMa.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - During training, the authors use noise levels close to zero to generate noises in known region, while uses zero during inference. Any specific reasons of adding noises during training? Is it possible to simply set them as zero?
- How did the authors design the time step conditions input to the network? It is not clearly stated in the paper.
- How is the inpainting performance of the model compared with state-of-the-art commercial inpainter like Photoshop Generative Fill, MidJourney Editor, and DALL-E inpainter? No need to do detailed comparison, but a few figures / images to show the advantages or gaps will be sufficient. 
- The pixel-wise time embedding is actually a region-wise time embedding? Is it possible to formulate it as only two time embeddings + one binary mask, so as to reduce the learning complexity of the model. The reviewer is not sure about the learning stability of the model. Also pixel-wise embedding can be well generalized to SDEdit, but it can be very hard to train.
soundness: 3
strengths: - The design of spatial-variant time embedding for inpainting tasks bring two major advantages: (1) no architecture change form T2I, and (2) relative faster inpainting process. 
- The authors provide extensive and scientific evaluation on different masks, and the proposed method achieved good inpainting quality.
summary: The authors mainly proposed to use the same T2I architecture to do inpainting. To properly feed in the known image conditions, they propose to use a pixel-wise noisy injection with different levels, and a spatial-variant timesteps map. Therefore, during training and inference, it allows to add minimum levels of noises to known image regions, while only diffuse the unknown regions following the normal diffusion process. The idea is simple, effective and easy to implement. Extensive experiments also demonstrate its effectiveness in maintaining good inpainting quality while reducing inpainting complexity compared to RePaint.
weaknesses: - Repaint seems not the state-of-the-art inpainting model, and recently models based on latest latent diffusion model or DiT framework achieve much better results, for instance, powerpaint(https://powerpaint.github.io/). Some comparison needs to be updated. 
- Since the model is trained on diffusion-based framework, is it still necessary to list ConvNet-based or transformer-based model in the comparison? They will be not comparable given the inference speed and model capacity. The comparisons should focus on latest diffusion-based model in quality and speed. 
- To the best of the reviewer's knowledge, recent diffusion-based inpainting is no longer using the way of RePaint to do inference. The problems mentioned by the authors like 'consistency' are no longer an issue. It can be simply achieved by adding an additional conditions concatenated with the noisy input in UNet, or adding additional tokens for DiT-based framework. Some controlnet-based method do have the composition and harmonization problems.
- It is not clear how the framework can be generalized to image editing, when the mask is soft or we need some levels of content preservation under the mask region. Without having the original image as a condition, it will be very hard to generalize the model to other editing tasks, like color changing. Therefore, the proposed framework will be limited as a foundation model, or used to finetuned for other tasks. 
- The masks used in inpainting literature are mostly not suitable for real practical application because they are not used by the real inpainter users. Better to try some object-shaped masks and use those for user study. 
- The proposed method may lack mask precision in a latent diffusion framework, especially for tasks like background replacement. For latent diffusion, the masks are usually applied to the pixel space, and the masked image will be encoded using the latent encoder for compression, and server as the condition. In this case, the latent diffusion model can still learn to reconstruct the high-precision mask boundary to preserve the details around the boundary. If we apply the low-res mask in the latent and apply different t in the precision of latent mask, it won't be able to leverage the advantages of the high-quality encoding.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: The paper's contribution to the community is not that great, as time improvement is not that significant.
soundness: 2
strengths: 1. It can generate high-quality inpainting on various mask sizes and shapes.
2. It is faster compared to other stare-of-art methods.
3. It has extensive evaluations
summary: This paper proposes a method that can do faster inpainting in diffusion models by introducing time-aware denoising process.
weaknesses: 1. The visual result is not much better than other baselines
2. Even though they have many evaluation results, the improvement in image quality with scores in not much.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. The training masks are always square. How can you applied the trained model on random masks during the inference?
2. Is it possible that this method can be extended to the latent diffsusion?
soundness: 3
strengths: 1. The training strategy is novel and the performance is good.
summary: The authors proposed a new training scheme of diffusion-based inpainting model that can perfectly maintain the regions inside the mask.
weaknesses: 1. Any evaluations about how good the unchanged regions are maintained?
2. I admit that the training scheme is novel, but I am not sure if the novelty is enough for publishing. Since the entire method is a slight modification of existing diffusion model.

Meta-review: The paper introduces TD-Paint, a novel approach for faster diffusion-based inpainting by leveraging time-aware pixel conditioning. It introduces a spatially varying timestep map that allows the model to maintain known pixel regions while efficiently diffusing unknown areas, eliminating the need for repeated sampling as in RePaint. TD-Paint achieves significant speed improvements without architectural changes, maintaining or exceeding state-of-the-art performance across datasets. The approach is simple, effective, and demonstrated through extensive experiments.

Strengths:
Innovative Method: Introduces spatially varying timesteps for precise, efficient inpainting without architectural changes.
Faster Performance: Speeds up inpainting significantly compared to existing methods like RePaint.
Robust Results: Achieves high-quality inpainting across diverse datasets, masks, and scenarios.
Simple and Practical: Intuitive, easy-to-implement approach with broad applicability.

Weaknesses:
Limited Comparisons: Missing evaluations against recent state-of-the-art diffusion-based methods like LatentPaint and PowerPaint.
Generalization Issues: Trained on limited datasets, with unclear applicability to open-world scenarios or broader editing tasks.
Mask Handling: Struggles with precise masks and lacks evaluation on realistic, user-oriented masks.
Modest Gains: Visual improvements over baselines are minor, raising concerns about the method’s impact.

We are happy to accept the paper, as all reviewers gave positive scores.

Decision: Accept (Poster)