Restoration based Generative Models
Authors: Jaemoo Choi
Yesom Park
Myungjoo Kang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Diffusion Generative Models
Image Restoration
Maximum a Posteriori

Abstract: Denoising generative models (DGMs) have recently attracted increasing attention by showing impressive synthesis quality. DGMs are built on a diffusion process that pushes data to the noise distribution and the models learn to denoise. In this paper, we establish the interpretation of DGMs in terms of image restoration (IR). Integrating IR literature allows us to use an alternative objective and diverse forward processes, not confining to the diffusion process. By imposing prior knowledge on the loss function grounded on MAP estimation, we eliminate the need for the expensive sampling of DGMs. Also, we propose a multi-scale training, which alleviates the latent inefficiency of DGMs, by taking advantage of the flexibility of the forward process. Our model improves the quality and efficiency of both training and inference, achieving state-of-the-art performance when the number of forward steps is limited. Furthermore, we show the applicability of our model to inverse problems. We believe that our framework paves the way for designing a new type of flexible general generative model.

Under review as a conference paper at ICLR 2023
RESTORATION BASED GENERATIVE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Denoising diffusion models (DDMs) have recently attracted increasing attention by showing impressive synthesis quality. DDMs built diffusion process that pushes data to the noise distribution and the models learn to denoise, In this paper, we establish the interpretation of DDMs terms of image restoration (IR). Integrating IR literature allows us to use an alternative objective and diverse forward processes, not confining to the diffusion process. By imposing prior knowledge on the loss function grounded on MAP estimation, we eliminate the need for the expensive sampling DDMs. Also, we propose a multi-scale training, which improves the performance compared to the diffusion process, by taking advantage the flexibility of the forward process. Our model improves the quality and efficiency of both training and inference, Furthermore, our model to inverse problems. We believe that our framework paves the way for designing new type of flexible general generative model.
show the applicability
INTRODUCTION
Generative modeling is prolific machine learning task that the models learn to describe how a dataset is distributed and gen-
Generative Modeling
Image Restoration
MUMSE
erate new samples from the distribution. The most widely used generative models primarily differ in their choice of bridging the data distribution
tractable latent distribution (Good- fellow et af., 2020; Kingma & Welling, 2014; |Rezende et al. Rezende Mohamed| 2 Dickstitttteinn et al., (0155] Chen et al. 20) In recent years, denoising diffusion models (DDMs) (Ho et al., 2020 Song & Ermon 2019; Song al..
RGM (Ours)
MAP
RGMs
DDMs
2020b Dockhorn et al. 2021) have drawn considerable at- tention by demonstrating remarkable results terms of both high sample quality and likelihood. DDMs rely forward diffusion process that progressively transforms the data into Gaussian noise, and they learn to reverse the noising process. Albeit their enormous successes, their forward process is fixed diffusion process, which gives rise to few limitations. To pull latent variables back to the data distribution, the denoising process requires thousands of network evaluations to sample single instance. Many follow-up studies consider enhancing inference speed (Song et al. [020aa] Salicceur-Martincar et al., [2021] Tachibana et al. 2021 or grafting with other
Forward Process
Ax + (noising, blurring, downsampling.. .)
X - (noising)
Objective
MAP
MMSE
Figure 1: Comparison of DDMs and RGMs.
generative models (iiao al. 2021a] Vahdat et al. [021] Zhang & Chen 202 Pandey al. 2022).
In this study, we focus on a different perspective. We interpret the DDMs through the lens of image restoration (IR), which is a family of inverse problems for recovering the original images from corrupted ones (Castleman, 1996. Gunturk & Li| 2018). The corruption arises in various forms, including noising (Buades et al. 2005) Rudin et al. 1992), blurring (Biemond et al. 1990). and downsampling (Farsiu al. 2004)) IR has been
-tandinnn problem because of its high practical value in various applications (Besag et al.) 1991 Banham & Katsaggelos 1997 Lehtinen et al. 2018 Ma et al. 2011). From an IR point of view, DDMs can be considered as IR models based on minimum mean square error (MMSE) estimation (Zervakis & Venetsanopoulos 1991 Laumont et al. 2022). focusing only on the denoising task. Mathematically., IR is an ill- ilo-poee inverse problem in the sense that it does not admit a unique solution and hence. leads to instability in reconstruction
(Hadamard, 1902). Owing the ill-posedness of IR, MMSE which only measures data fidelity

---

Under review as a conference paper at ICLR 2023
produces impertinent results. DDMs alleviate this problem by leveraging costly Langevin sampling, and this inefficient inference scheme has been regarded as an indispensable tool in the literature of DDMs. By casting DRMs IR models, however, the forward process need not be restricted to Gaussian noising, and ill-posedness can be detoured in ways other than Langevin dynamics.
Inspired by this observation, propose new flexible family of generative models, that refer to as restoration-basel generative models (RGMs). First, we adopt an alternative objective; a maximum a posteriori (MAP) (Trussell 1980, Hunt 1977). which is predominantly used in IR. The MAP-based estimator compensates the ill-posedness by regularizing the data fidelity loss by a prior term. Many advent hand-crafted regularization schemes (Tikhonov) 1963, Donoho 1995 Mallat 1999; Baraniuk] 2007) encourage solutions to satisfy certain properties, such as smoothness and sparsity. However, for the purpose of density estimation, we implicitly parameterize prior term as variational regularization via GAN (Goodfellow et (022)) with newly introduced random auxiliary variable. Our MAP approach retains the density estimating capability of DDMs at a much smaller computation cost. Secondly, unlike DDMs, which are buried in a Gaussian noising process, RGMs can be combined with other general degradation processes. As one instantiation, we design a multi-scale training that resolves the latent inefficiency of DDMs. Because the behavior of generative models is significantly affected by how the data distribution is transformed into a simple distribution, our approach opens the way for designing more flexible generative models. Our comprehensive empirical studies on image generation and inverse problems demonstrate that RGMs generate samples rivaling the quality of DDMs. Also, the inference of our model is several orders of magnitude faster than DDMs. In particular, our model achieve FID 2.47 on CIFARIO 10, with only seven number of network function evaluations.
BACKGROUND
Image Restoration A common inverse problem arising in image processing, including denoising, deblurring, super-resolution, and inpainting, is the estimation of an image x given a corrupted image
Ax +
(1)
where A is matrix that models the degradation process, including blurring and downsampling kernels, and & ~ N' NN(0, Σ) is an additive noise. A family of such problems are known as image restoration (IR). The inference of the image from the noised one y is typically ill-posed, in the sense that the inverse problem (T) has multiple valid explanations (Hadamard 1902). In other words, the noisy y does not have exactly one restoration x. This is further exacerbated when the noise level is large. To produce consistent results, most methods use the maximum a posteriori (MAP) estimator:
MAP argmax, logp
p (x yy argmin f (x Ag (x),
where I (x. = log (y x) = (E)) 1 (Ax- | is the data fidelity term with the pseu- doinverse (Moore 1920) Et, g is the prior term that encourages the reconstruction to satisfy some prior assumptions on x, and a scalar À 0 controls the strength of the regularization. The reg- ularization term is essential because it relieves the ill-posedness nature of the inverse problem by imposing the assumption about the desirable solution. Therefore, many researchers have been devoted to designing proper g (Rudin et al. 1992
Mallat 1999 Lunz et al. 2018).
Denoising Generative Models Denoising diffusion models (DDMs), such as DDPM (Ho et al.. 2020), and score matching with Langevin dynamics (Song et al., 2020b), have recently emerged as the forefront of image synthesis research. Starting from the data distribution, they gradually corrupt Pdat into Gaussian noise over time through forward Markovian diffusion process;
the image Xg
T-1
q (XieT xo)
II t=
(2) (x++1
xx), X0 ~ Pdata-
They pose the data generation as an iterative denoising procedure pa (x: Xe41), the reverse of the forward diffusion process:
T
(Xo.T) = (2)) (XT)
II
Pa
(x
Xc+1), XT ~ N
(4)
1=0
2

---

Under review as a conference paper at ICLR 2023
As they use linear diffusion process whose diffusion distributions q(*) (xt+1 xx) are modeled with conditional Gaussian distributions, we can deduces a tractable evidence lower bound (ELBO) (Sohl
Dickstein et al. 2013). The ELBO can be further simplified to the following objective (Ho et al. 2020 Song et al. 2020b:
(0)
>T 1xx -opon (xx: (x)) (A |[g (xx, t) X
(5)
where Gg neural network parametrized by 0 that learns the noise by minimizing 5). and À (1) 2 0.
METHOD
3.1
DDMS ARE RESTORATION MODELS
We open this section by drawing the interpretation of DDMs in terms of restoration. DDMs use forward process, and these
VPSDE VESDE [Song et al.] 2020b)
known to be exchangeable with each other (Kim et al. 2022). Therefore, the rest of the paper focuses on VESDE. For given noise level de the forward process of the VESDE formulated
Xo + E, { ~ N' 10)
X
which is the forward process ( with identity degradation matrix A As Vx. log yr. (x+ xo) (x: Xo) /a? the loss (5) for each forward step can be rewritten as the following minimum mean square error (MMSE) objective:
(()) = E
[[[Ga(x, t) x
(6)
Therefore, DDMs are IR models which seek denoiser Gg for each a, that minimizes the MMSE ( MMSE loss is simple and straightforward to train, however, it confronts some apparent draw backs. Since the MMSE only contains the recovery term, the solution only optimized ensure accordance with the degradation process. Therefore, it is affected by the ill- pol-posedness. To be precise, when de large, ( becomes a highly ill- ill-posed and possess many solutions for given observation. this case, MMSE solution averages all these candidate solutions, resulting in an atypical reconstruction. Formally, the solution to the MMSE for a corrupted data x; is
MMSSE
XPdata (x) Po. (xe x) dx.
Recent works (Laumont et al. 2022 Kawar et al. 2021) have endeavored to resolve this problem by stochastic sampling, however, they suffer from notoriously low efficiency as they roll out thousands of trajectories. In a similar manner, DDMs utilize sampling scheme that requires thousands of steps. In summary, there are two limitations of DDMs from the IR perspective:
1. The degradation process is restricted to Gaussian noising,
2. The inference efficiency is intrinsically low due the MMSE estimator.
In the following sections, we show how to cope with these two limitations.
3.2 MAP- BASED ESTIMATION FOR GENERATION
As alluded in Section 3.1 DDMs can be regarded as MMSE grounded IR models, specialized in denoising. This observation brings us a new perspective on the design of family of flexible generative models. As an alternative MMSE, we propose new generative model based on the MAP
Ex NoN (1)
20 |Go(y)
1
y1l2 + >g(Go(y)))
where the second term delivers the prior knowledge of data distribution. MAP has been adopted as standard approach for high-dimensional imaging problems and is known to be more relevant than MMSE in many applications (Saha et al. 2009: Bigdeli et al.] 2019 Chen 2016). By leveraging prior information on the solution, MAP-ban based approaches alleviate the ill poseddesssss of the inverse problem (I) without use of costly sampling of MMSE estimation. Therefore, carefully crafting the relevant prior term is crucial. We now show how one can execute an appropriate prior term for density estimation while alleviating the ill ill-posedness.
3

---

Under review as a conference paper at ICLR 2023
Alleviation of ill ill-posedness Unlike the general denoising task, it is necessary to bridge the image to the Gaussian noise to learn the data distribution. As the noise level increases, a single distorted observation has several solutions, which indicates that the ill-posedness deepens. This in turn degrades the expressiveness of the model for estimating the distribution. Therefore, it is difficult for the regularization term to remedy all ill-posedness its own. We further offload the ill- ill-posedness by imposing a priori knowledge by introducing a random auxiliary variable z ~ N (z O, I). Importantly, z enables us to obtain several solutions for a heavily degraded x, through the guidance provided by z. This eventually helps to pull the noised observations back to the data distribution, allowing for a rich representation for the density.
Implicit Prior Knowledge For density estimation, the knowledge about the data distribution should be properly encoded in the prior term g of (7). The explicit density function with determined marginal distribution is intractable, but it can be learned with the aid of well-developed generative models. That is, we can design by integrating various existing generative models. In this paper, we learn an implicit representation of the data density by adopting generative adversarial network (GAN), which has shown promising results in many generative tasks, and use generator loss as a relevant prior term g- For each forward step, our MAP-based objective for our generator in conjunction with the GAN prior is given by:
G(G))
(G 2)
y112 )go(Go z)))
(8)
20
where the first term is the data fidelity term, (x(x) log D& (x)) log Do (x) is learnable prior term that is trained coupled with a discriminator Da, and X 2 0 is a hyperparameter. As conventional, we train the discriminator Da to minimize Censen-Shannon divergence. Contrary to conventional MAP whose prior term is pre-defined, our approach tries to learn the prior term by coordinating with the denoiser Gg through the discriminator. This end-to-end training allows our discriminative learning method to deliver promising performance. Note that when the discriminator
is optimal, i.e. D²(x)= (o))+ the loss g with À agrees with
p(ar)
E Exx-pax [QK (xly) |lp(x + H(g)).
where H denotes an entropy, which corresponds to training the model to learn the posterior distribution. The overall training procedure combined with all a E {or is provided in Appendix B.2
Small Denoising Steps A major downside of DDMs is their slow sampling procedure, which requires hundreds to thousands of denoising steps to obtain a single image. By adopting MAP approach and parametrizing the prior distribution through GAN, our model provides an avenue to offload the time-consuming sampling scheme and enables significantly small denoising steps. For small degradation we can obtain a restored image in one shot. But, as our restoration starts from the Gaussian noise, the data distribution is not completely estimated. Therefore, we perform the generation iteratively. In our experiments on CIFAR10. we generate a high-quality sample in four denoising steps, whereas most DDMs use hundreds to thousands steps.
3.3 EXTENSION TO GENERAL RESTORATION
In Section 3.2
we proposed a denoising generative model based on MAP-like objective. However, from the IR perspective, it is not necessary to restrict it to denoising (A = D) and it can be generalized to any family of degradation matrices A and noise factors Σ in ( Utilizing the general forward process, we learn the generative model by generalizing the loss function (8) as follows:
[1
(I)) 1/ (A Goly 2) - ->> (go 2))) 2
((G)) = (-
(10)
Therefore, RGM has an flexible structure that can permeate any forward process, and aids in designing a new generative model. Here, we propose a new model established upon super-resolution (SR).
Multi-scale RGM Most DDMs maintain the image size during the diffusion process by adding noise to individual pixels. Consequently, they are very inefficient because they require a latent as much as dimension of pixel space that is much larger than the submanifold of the image space. Motivated by
4

---

Under review as a conference paper at ICLR 2023
Figure 2: Generated samples on LSUN Church (left) and CelebA-HQ (right).
this, we take as block averaging filter that averages out pixel values. Halving the image size at each coarsening step allows us more expressive generative model with lower-dimensional latent distribution. Moreover, multi-scale training has proven to be an effective strategy for synthesizing large scale images (Denton al. 2015 Karras et al. 2017b] Reed et al. (017). Therefore, our model produces strikingly realistic images by progressively extracting spatial information.
4 EXPERIMENTS
This section evaluates the performance of the proposed RGMs on several benchmark datasets. including CIFAR 10 (Krizhevsky et al. 2009) (32 x 32 unconditional), CelebA-HQ (Liu et al. 2015) (256 x 256), and LSUN Church Yu et al. 2015) (256 x 256). We also show the capability of RGMs for solving inverse problems. We parametrize our generator Go based the UNet-like structure (Ronneberger et al. 2013) which was successfully used in NCSN++ architecture (Xiao et al. 2021a).
The internal details of the implementation can be found in Appendix B
Setup We implement
models: RGM-D is model trained with the diffusion process, which is mainly used by DDMs. We also consider multi-scale model whose degradation matrix is a 2 x 2 averaging filter. In this case, unlike the diffusion process, the image corrupted by a downsampling filter together with additive noise. Therefore, the model (termed by RGM-SR (naīve)) is demanded to conduct upsampling and denoising at the same time. To make more effective, we explore another forward schedule that separates the downsampling and noising process and performs them alternatively. RGM-SR refers to the model to which this schedule is applied. (See Appendix B. for details).
4.1 2D TOY EXAMPLE
We first employ a two-dimensional example to vali- date the effectiveness of prior knowledge of our MAP framework. We adopt mixture of Gaussian with eight components (Grathwohl et al. 2018) target distri- bution. In Figure 3 we depict the benefits of our MAP approach over the MMSE approach. As illustrated in
Pdata
Pnoise
the top row, we diffuse the data distribution through four different noise levels. Each row from the second row to the bottom represents the learned distribution of our RGM, RGM without the auxiliary variable z, and MMSE. First, the bottom row shows the failure of
MAP wo'
MMSE, where the modes of Paata are connected and then missed. This tendency exacerbates as the noise level increases. Since the MMSE fails to reconstruct the data distribution even with small rise in the noise level, the MMSE does not yield a satisfactory gener- ative model with small number of diffusion steps. Consequently, MMSE approaches, such as DDMs, require a large number of steps to stably recover the data distribution. On the other hand, by leveraging the prior term, our model generates samples
MMSE
Figure Comparison of recovering density by MMSE versus MAP- based objective.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Results on unconditional generation of CIFARIO.
Class
Model
FID (U)
IS (T)
NFE (L)
RGM
RGM-D RGM-SR
3.08
9.14
4
2.47
9.68
7
DDM
DDPM (Ho al.
2020
3.21
9.46
1000
NCSN (Song & Ermon 2019
25.3
8.87
1000
Score SDE SDE(VE) (Sog
2020b
2.20
9.89
2000
Score SDE (VP) (Sonp
2020b
2.41
9.68
2000
Probability Flow (VP) (Song al.
2020b
3,08
9.83
140
L.GGM (Vahdar al. 202T
2.10
9.87
147
DDIM (50 steps) (Song
2020a
4.67
8.78
50
FastDDPM (T=50) Kong & Ping 2021
3,41
8.98
50
Recovery EBM (Gao
2020
9.58
8.30
180
VDM (Kingma al. 202T
4.00
1000
UDM (Kim al. 2021
2.33
10.1
2000
Gotta Go Fast (Alexia Jolicouu- Martincau
2021
2.44
1000
Subspace Dilluston (Jing cl al. 2022
2.17
9.94
1000
CLD (Dockhorn el al.
2.25
2000
DEIS Zhang & Chen
3.37
9.74
15
DDGAN (Aiao al.
3.75
9.63
4
BoyleGAN2+ES-DDRR Lyu
2022
5.52
101
Progressive Distillation (Sahmans
Ho 2022
3,00
4
GAN
SNGAN+DGflow (Ansari et al.
2020
9.62
9.35
25
AutoGAN (Gong cl al. 2019
12.4
8.60
TransGAN (Jiang et al. 2021
9.26
9.02
StyleGAN2 w/o ADA Karras cl al.
2020
8.32
9.18
StyleGAN2 w/ ADA Karras et al.
2020
2.92
9.83
VAE
NVAE (ahddat Kautz 2020
23.5
7.18
Glow (ingma Dharwall 2018
48.9
3.92
1
PixeICNN (Van Oord 2016
65.9
4.60
1024
VAEBM Xiáo
2020
12.2
8.43
16
from the multimodal distribution significantly better. By imposing the prior knowledge, the MAP- based objective accurately estimates the density, which allows distribution recovery with much smaller number of forward processes than the MMSE approach. Furthermore, we can observe the effect of the auxiliary variable by comparing the second and third rows Figure] MAP with z has higher sample quality. The effect of z is further amplified for more complex distributions, such as image data (See Figure and conjunction with the random auxiliary variable z enables accurate and efficient generative modeling.
and Table This synthetic experiment validates that our MAP approach in
4.2 IMAGE GENERATION
We compare the performance our method with several existing baselines. For quantitative compari- son, we use Fréchet Inception Distance (FID) and Inception Score (IS) as the evaluation metrics. We also report the number of network function evaluations (NFE). For DDMs and RGMs, NFE value and real inference time are proportional. Following Song et al. (2020b); Dockhorn al. 2021 focus on the widely used CIFARIO 1O unconditional image generation benchmark and also validate the performance of RGMs large-scale CelebA-HQ-255 images. Table I summarizes the quan- titative evaluations on CIFAR10, and the results on CelebA-HQ reported in Table 2 Qualitative performance depicted in Figures 2 and
Results We see that our model achieves the state-of-the-art FID score on CelebA-HQ-255 among restoraion-based models. On CIFAR10, our RGMs are comparable to the best existing DDMs and GAN models. Although the best denoising models obtain better results than ours on CIFARIO they use much larger number of denoising steps (c.g. ScoreSDE with VESDE requires 2000 steps). Progressive distillation achieves an FID score comparable to ours with quite reduced sampling cost, but they require considerable additional cost at training time due to their distillation process. Notably, our RGM-SR achieves FID 2.47 and IS 9.68 with only seven steps, which state-of-the-art sampling FID performance when NFE is limited. The overall results confirm that our MAP-based estimation immediately eliminates the need for an expensive sampling scheme while still maintaining the density estimating capability of DDMs. Interestingly. RGM-SR outperforms RGM-D by large margin even with far fewer latent variables than RGM-D. This improved performance may be attributed to the increase in NFE: however, the FID of RGM-D with T reported in Table 7 confirms that is

---

Under review as a conference paper at ICLR 2023
Table 2: Results on generation of CelcbA-HQ-256.
Class
Model
FID (4) NFE (L)
RGM
RGM-D
7.15
DDM Scure SDE (VP) Song el al.
2020b
7.23
4000
Probability Flow (Song cl al
20206
128.13
335
LSGGM Vahdat et al. 2021
7.22
UDM Kim el al.
7.16
2000
DDGAN
X1a0
ZU21
7.64
GAN
PGGAN (Karas et
201 7:
8.03
Adv. LAE Pidhorskyi
2020
19.2
VQ-GAN (Esser al.
20
10.2
DC-AF Parmar el al
2021
15.8
StyleSwin Zhang ot a
2022
3.25
VAE
NVAR Vahddat & Kautz
2020
29.7
VAEEM
Xiao ct al.
2020
20.4
NCP- VAE
Anga ot al.
202]
24.8
Figure 4: CIFAR10 generated samples.
not. The overall results indicate that the MAP approach of RGMs is a promising way for generating high-quality samples in limited steps. More uncurated images can be founded in Appendix C.6
4.3 ABLATION STUDIES
This section is devoted to ablation analyses which show that all parts of our objective, the data fidelity term, and the prior term together coupled with the auxiliary variable, and the regularization parameter, each play an important role in our performance of density estimation.
On the role of We include experimental results LSUN, which demonstrate how the auxilliary variable z alleviates the ill- posedness of the inverse problem. By noising the upper-left image Xp, we obtain the forward trajectory {xx (==) The figures on the right are restored images of x2 by RGM-D together with four different z. We can see that a reconstruction is almost unique when the noise level is small. But, as the noise level increases, a single X has various reconstructions. It is evident that assigning z helps generate different denoised images from heavily degraded X₃ through the guidance provided by z. However, one might think that the ill-posedness detoured by multi-step training using multiple on rather than through This claim can be refuted using the result of RGM-D without z reported in Table 3 We observe the significant difference in FIDs of RGM-D with and without under the same number of denoising steps, which indicates the effectiveness of z.
X1
X
Figure 5: Study on effect of z.
On the effect of Varying À We investigate the sensitivity of the regularization parameter À in (8). Since it controls the relative importance between the fidelity term and the prior term, À is trade-off hyperparameter that determines how much regularizes present FID
RGM5-3
the joint distribution of pı and Pk++1 In Figure scores measured on CIFARIO with the same number of degrada- tion steps (T' 4) and varying X. We can see that our models are quite robust with respect to X. An empirically observed sweet spot of À d/10 < À < d for the image size d, which FID is no longer improved outside this threshold. For small X, the models put a lot of effort recover the degradation, which hinders estimating data distribution. Choosing large À also results in performance degeneration. There is another point that draws our attention. When = oo, that is, when there is no fidelity term on the objective, FID scores completely deteriorate. In this case, the models are trained by the vanilla GAN loss. From the perspective of GAN, our MaP-based objective adds the fidelity term to the GAN loss function. We further observe that with the help of the fidelity term, model enhances the mode-collapsing resiliency of GAN (See Figure I6). These results validate that the performance of RGMs owes to both fidelity and prior term, and the reliable regularization parameter is determined balance these two terms.
KGM-KK
18
Figure 6: Study on effect of X.
7

---

Under review as a conference paper at ICLR 2023
Colorization
Super-Resolution
Original
Grayscalo
Ours
GAN baseline
DDRM
Downsampled
Ours
Bicubic
GAN baseline
DDRM
Figure 7: Colorization (left) and super-resolution (right) results on LSUN and CelebA-HQ datasets.
On the forward process schedule Since the forward process de- termines the way of connecting the data and latent distributions, it
Table 3: Ablation Studies.
significantly affects the performance of models. The first important factor is the number of forward steps T, which is directly related to NFE. In Table 3 we ablate the effect  When T 1, may be difficult for the model to directly approximate the data distribution from the Gaussian noise. This is reflected in the poor FID score. We also study the forward process schedule of the SR model. We can ob- serve that the separation of the same forward process into two steps makes the model easier to learn, and this brings the performance enhancement of RGM-SR compared to RGM-SR (naive).
Model
FID (L)
RGM-D (A 00)
32.5
RGM-D (T 1)
14.6
RGM-D (T=4) 4)
3.04
RGM-D (wo /
3.87
RGM-SR (naive)
3.17
RGM-SR
2.47
4.4 INVERSE PROBLEMS
While our model was originally devised generate images, we further show the applicability of RGMs to inverse problems. Recently, a promising approach in imaging inverse problems is to leverage learned denoiser as an alternative to the proximal operator of splitting algorithms (Romano) et al.] 2017 Hurault et al. (021). Such methodology is referred to as Plug-and-Play (PnP) algorithms (Venkatakrishnan et al., 2013). a similar spirit, we utilize the trained RGMs as a modular part of the PnP algorithms to solve various inverse problems. In this section, we testify our RGM-D for two inverse problems; super-ress resouution and colorization, by plugging our model into Douglas-Rasd Rachford Splitting algorithm (Lions & Mercier 1979). Details can be found in Appendix B.3
Results We compare the performance of our model with current-leading models: We compare our model with DDRM (Kawar al. 2022). which solves inverse problems with pre-trained DDPM by posterior sampling scheme. As a GAN baseline, we adopt StyleSwin (Zhang et al.) 2022) and reconstruct the image by optimizing over the latent vector (Pan et al. 2021). We also consider bicubic interpolation as baseline for super-resolution. We observe that our model is capable of reconstructing faithful and realistic images, as evident in Figure 7 Compared with basclines, our model produces high-quality reconstructions across all the datasets. In particular, promising performance for colorization. These results show the applicability of RGMs to PnP prior, and this will bring range of potential applications, including image segmentation, conditional generation, and other imaging inverse problems. Additional quantitative and qualitative results are provided in Appendix C.4
model shows
Comparison of RGM-D and RGM-SR We investigate the effect of the degradation process used during training on the performance of solving inverse problems. We compare the reconstruction performance of RGM-D and RGM-SR that are trained on different degradation processes by applying both models to denoising and super-resolution (SR) tasks on CIFARIO. Quantitative results are presented in Table 4 We can see that the RGM-SR that is trained based on SR actually performs the SR task better. Also, we can observe a similar tendency for denoising. The results confirm that the degradation process used in training actually helps in solving the corresponding inverse problem.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Quantitative comparison of RGM-D and RGM-SR on image reconstruction.
Model
Super-Resolnne
Denoising
(x2)
(x4)
(a
10/255)
(a
20/255)
(a 40/255)
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
RGM-I D
26.63
0.88
20.84
0.58
30.11
0.93
26.57
0.86
24.23
0.80
RGM-SR
27.42
0.90
21.14
0.59
29.41
0.92
25.87
0.84
23.53
0.77
5
RELATED WORK
In recent years, denoising-based generative models (Ho et al., 2020] Song Ermon 2019; Song
2020b) have emerged as a class of density estimation models, first sparked by (Sohl-Dickster 2015). They define a sampling process as the reverse of a forward diffusion process that maps data to Gaussian noise by consecutively adding small portion of the noise to the input data. As they faithfully estimate the data distribution and generate high-fidelity samples, they have rapidly been applied to various domains such as conditional generation (Lee et al., 2022; Ho et al. 2022a). audio synthesis(Kong et al. 2021; Popov et al. 2021). medical imaging (Song ct al., 2021 Chung & Ye| 2022), video generation (Ho al.) 2022b Yang et al. 2022), and 31) point cloud generation (Lyu et al.) 2021). Their major drawback is slow and expensive inference. Many studies have been dedicated to circumventing this downside by developing fast numerical solver Dolicoeur-Martineau et al. 2021; Zhang & Chen 2022 Tachibana et al. 2021; Liu et al. (022)) or using an alternative noising process such as non- non-Markowe Markovian (Song et al. 2020a). ) second-order Langevin dynamics (Dockhorn et al. 202I), and non-lineat diffusion processes (De Bortoli et al.] 2021] Chen et al. 2021b). Another line of work improves sampling efficiency by incorporating into other generative models, including GAN (Xiao et al.) 2021a: Lyu et al. 2022). , and VAE (Vahdat & Kautz, 2020). Xiao
it al.
al.
ct al. (201aa) which enjoys small sampling steps by using GAN is one of our related work. However, they did not introduce GAN from MAP perspective and our model requires less training iteration to achieve the same performance, (See Appendix A for details.) Moreover, there have been distillation
approaches (Salimans & Ho 2022 Meng et al. 2022 On a side note, all the aforementioned models use the Gaussian noising process as the forward process.
Recently, the literature has begun to replace the additive Gaussian noising process with other transforms. Breaking away from the diffusion process, (Rissanen et al. (022)) proposed a forward blurring process inspired by heat dissipation. They suggest new generation process, but they specialize in the proposed blurring process and cannot be incompatible with other degradation processes. Possibly the closest study to our work is Cold Diffusion (Bansal et al., (022) which generalizes the diffusion process to arbitrary image transformations. It seems to use a general transform similar to our models, but Cold Diffusion only uses deterministic degradation processes by entirely removing additive Gaussian noise, which hinders its density estimation performance. Also, they use the MMSE objective, still requiring an array of several forward steps. We include a comparison with these related works in Appendix
CONCLUSION AND FUTURE WORK
In this study, presented general framework for modeling efficient generative models through the lens of IR. Compared to DDMs whose both forward and reverse processes are fixed to thousands of Gaussian steps, our approach provides more flexible models that eliminate expensive sampling and can enjoy versatile forward processes. We eliminated the usage of slow sampling by taking on MAP approach and incorporating implicit prior information through GAN. In addition, propose multi-scale method as an example of the usability of various forward processes. The experimental results showed that the image quality obtained was on par with the leading DDMs, and we achieved state-or-the-an art performance using a limited number of forward steps. We hope that this work provides a broad view of modeling useful generative models.
Our model has two degrees of freedom: One is how to parametrize the prior knowledge, and the other is the choice of the forward process. This opens up interesting directions for future research. In addition to the GAN we used, the prior term can be constructed in different ways. It could also be interesting to explore other degradation transformations. Moreover, analyzing the effect of the random auxiliary variable would be an worthwhile direction for future research. Future work could include the comprehensive design of a convergence guaranteed PnP algorithm for application to various inverse problems. We leave these further extensions future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rémi Jicé--Tailefee Tal Kachman Ioannis Mitliagkas Alexia Policoeur-Martines Ke Li. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14008 2021.
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480-493, 2021.
Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discrimi- nator gradient flow. arXiv preprint arXiv:20/2.00780, 2020.
Mark Banham and Aggelos K Katsaggelos. Digital image restoration. JEEE signal processing magazine, 14(2)22441, 1997.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint urXiv:2208.099 09392, 2022.
Richard Baraniuk. Compressive sensing [lecture notes]. IEEE signal processing magazine, 24(4): 118-121, 2007.
Julian Besag, Jeremy York, and Annie Mollié. Bayesian image restoration, with two applications in spatial statistics. Annals of the institute of statistical mathematics, 43(1):1-20, 1991.
Jan Biemond, Reginald L. Lagendijk, and Russell M Mersereau. Iterative methods for image deblurring. Proceedings of the IEEE, 78(5):8568883, 1990.
Siavash Bigdeli, David Honzátko, Sabine Süsstrunk, and I. Andrea Dunbar. Image restoration using plug-and-phe enn map denoisers. arXiv preprint arXiv: /9/.99922.999..//. 2019.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1-122, 2011.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttss://ppeneeeiiee
net/ piet/forum?ice 9ttt/formmidd-1xqq:0000 wwx.iiidddxxxjjj0000000 m
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising, In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 005). volume 2, pp. 60-65. leee, 2005.
Kenneth R Castleman. Digital image processing. Prentice Hall Press, 1996.
Tianrong Chen, Guan- Liu, and Evangelos Theodorou. Likelihood training of schr " odinger bridge using barward-backwerr sdes theory. arXiv preprint arXiv:2110.11291 2021a.
Tianrong Chen, Guan-Horng Liu, and Evangelos A Theodorou. Likelihood training of schrodinger bridge using harward-backwerr sdes theory. arXiv preprint arXiv:2110.1 12.1110..111111111
Yunjin Chen. Higher-order mrfs based image super resolution: why not map? IET Image Processing, 10(4):297-303, 2016.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri Medical Image Analysis, pp. 102479, 2022.
Giamnis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint 2xXiv:2209.05442 2022.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Amaud Doucet. Diffusion schrödinger bridge with applications to bcore-based generative modeling. Advances Neural Information Processing Systems, 14957699-77709, 2021.
10

---

Under review as a conference paper at ICLR 2023
Emily L Denton, Soumith Chintala, Rob Fergus, al. Deep generative image models using laplacian pyramid adversarial networks. Advances in neural information processing systems, 28, 2015.
Tim Dockhor, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. arXiv preprint arXiv:2112.07068. 2021.
David L Donoho. De-noising by soft-thresholding. IEEE transactions information theory, 41(3): 613-627, 1995.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.0m 03906, 2015.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar, Fast and robust multiframe super resolution. IEEE transactions on image processing, 13(10):1327-1344 2004.
Ruiqi Gao. Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. arXiv preprint arXiv:2012.08125, 2020.
Donald Geman and Chengda Yang. Nonlinear image recovery with half-quadratic regularization. IEEE transactions on Image Processing, 4(7):932-944 1995.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139 144, 2020.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint 1rXiv:1810.01367, 2018.
Bahadir Gunturk and Xin Li. Image restoration. CRC Press, 2018.
Jacques Hadamard. Sur les problèmes aux dérivées partielles et leur signification physique. Princeton university bulletin, pp. 49-52, 1902.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47-1, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv.220..03458, 2022b.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint 270.57220 2022.
Bobby R. Hunt. Bayesian methods in nonlinear digital image restoration. IEEE Transactions Computers, 26(03):219-229, 1977.
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Gradient step denoiser for convergent plug-and-play arXiv preprint arXiv:21 2021.
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Proximal denoiser for convergent plug-and-
play optimization with nonconvex regularization. arXiv preprint arXiv:2201. 13255, 2022.
11

---

Under review as a conference paper at ICLR 2023
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers make strong gan. arXiv preprint arXiv:2102.07011 1(3,, 2021.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. arXiv preprint arXiv:2205.0/ 2022.
Alexia ooliceeur-aattneen Ke Li, Rémi Fiché-Taillefes Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 140,00 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. 2017a.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017b.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12 1212-1142114 2020.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21766 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201.1179. 2022.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527, 2021.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In International Conference on Machine Learning, pp. 1120--11223 PMLR, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707,, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint crXiv:1412.6900 6980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. International conference on machine learning, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.
Advances in neural information processing systems, 31, 2018.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint 201::2/00000022. 2021.
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. International Conference on Learning Representations, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, 15(2):701-737, 2022. 2022.
Sang-gil Lee, Heeseung Kim. Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. International Conference on Learning Representations, 2022.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv: 1803 0819, 2018.
12

---

Under review as a conference paper at ICLR 2023
Chun-Liang Li, Wei- Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference on machine learning, pp. 1718-1727. PMLR, 2015.
Pierre-Louis Lions and Bertrand Mercier. Splitting algorithms for the sum of two nonlinear operators.
SIAM Journal on Numerical Analysis, 16(6):94--979, 1979.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. arXiv preprint arXiv:2202.099720 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp- 3730-3738, 2015.
Sebastian Lunz, Ozan Öktem, and Carola-Bibiane Schönlieb. Adversarial regularizers in inverse problems. Advances in neural information processing systems, 31, 2018.
Zhaoyang Lyu, Zhifeng Kong. Xudong Xu, Liang Pan, and Dahua Lin. A conditional point diffusion- refinement paradigm for point cloud completion, arXiv preprint arXiv:21 2021.
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang. Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205.12552 2022.
Jianhua Ma, Jing Huang, Qianjin Feng, Hua Zhang, Hongbing Lu, Zhengrong Liang, and Wufan Chen. Low-dose computed tomography image restoration using previous normal-dose sean. Medical physics, 38(10):5713-573) 2011.
Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999.
Chenlin Meng, Yang Song. Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint 0rXiv:2108.0107. 2021.
Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho,, and Tim Salimans. On distillation of guided diffusion models. arXiv preprini arXiv:2210.03142, 2022.
Eliakim H Moore. On the reciprocal of the general algebraic matrix. Bull. Am. Math. Soc., 26: 394-395, 1920.
Radford M Neal. Probabilistic inference using Markov chain Monte Carlo methods. Department of Computer Science, University of Toronto Toronto, ON, Canada, 1993.
Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional slised-wasserstein and applications to generative modeling. arXiv preprint arXiv:2002.07367, 2020.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Effi cient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.00300 2022.
Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends@ Optimization, (3):127-239, 2014.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 823-832, 2021.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14104-14113, 2020.
13

---

Under review as a conference paper at ICLR 2023
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speehh. In International Conference on Machine Learning. pp. 8599 8599-86088. PMLR, 2021.
Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In Interna- fional conference machine learning, pp. 2912 2921. PMLR, 2017.
Edward T Reehorst and Philip Schniter. Regularization by denoising: Clarifications and new interpretations. IEEE transactions on computational imaging, 5(1):52-67, 2018.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipa- tion. arXiv preprint arXiv:2206. 2022.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal Imaging Sciences, (0(4):8804-188 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241 Springer, 2015.
Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, (0(1-4):259-10 268, 1992.
Saikat Saha, Yvo Boers, Hans Driessen, Pranab Kumar Mandal, and Arunabha Bagchi. Particle based map state estimation: A comparison. In 2009 12th International Conference on Information Fusion, pp. 278-283. IEEE, 2009.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 2XXv:2202.005//2. 2022.
Jascha Schl-Dickster Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Jamming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song. Jascha Dich--iicksten, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcore-based generative modeling through stochastic differential equations. arXiv preprint 1.Xii::0111133550 2020b.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score- based generative models. arXiv preprint arXiv:2111.08005, 2021.
Hideyuki Tachibana, Mocho Go, Muneyoshi Inahara, Yotaro Katayama, and Yotaro Watanabe. It', -tayloo sampling scheme for denoising diffusion probabilistic models using ideal derivatives. 2021.
arXiv preprint 12.
Andrei Nikolaevich Tikhonov. On the regularization of ill ill-posed problems. In Doklady Akademii Nauk, volume 153, pp. 49-52. Russian Academy of Sciences, 1963.
H Trussell. The relationship between image restoration by the maximum posteriori method and maximum entropy method. IEEE Transactions on Acoustics, Speech, and Signal Processing. 28 (1):114-117, 1980.
14

---

Under review as a conference paper at ICLR 2023
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667-19679, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 14:112877 1287-1171111 11302 2021.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747-1756. PMLR, 2016.
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pp- 945-944--90 944. IEEE, 2013.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from eπror visibility to structural similarity. IEEE transactions on image processing. 13(4):600-612, 2004.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: symbiosis between variational autoencoders and energy-based models. arXiv preprint arXiv:2010.0065: 2020.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.078044 2021a.
Zhisheng Xiao, Qing Yan, and Yali Amit. Ebms trained with maximum likelihood are generator models trained with self-adverserial loss. arXiv preprint arXiv:2102.11757, 2021b.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203.0944 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv: /50. 506600365, 2015.
ME Zervakis and AN Venetsanopoolos. A class of noniterative estimators for nonlinear image restoration. IEEE transactions on circuits and systems, 38(7):731-744, 1991.
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer-based gan for high-resolution image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 11304-11314, 2022.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Information Processing Systems, 14:16200-11291, 2021.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arX1v:220.113902 2022.
15

---

Under review as a conference paper at ICLR 2023
A MORE RELATED WORKS
DDMs have been pertinent generative models by showing promising results on various generation tasks. DDMs degrade the data with a reference diffusion process and learn the data distribution by restoring it. We have arranged DDMs in the context of restoration, and DDMs can be interpreted as an MMSE estimator for a denoising task.
Energy-based models (EBMs) are another line of generative models that learn the unnormalized data distribution by giving low energy to high-density regions in the data space. As DDMs have demonstrated that recovery of sequence of noisy data more effective than directly approximating the data density, Gao et al., (2020) recently proposed recovery energy- based model (REBM) by using a diffusion process, Inspired by DDMs, REBM learns a sequence of energy functions for the marginal distributions of the diffusion process. More precisely, from the noisy observation (x(()))) by
X = x + E. & ~~ ~~N (0, o2 they estimate the conditional likelihood po (x x) x exp
learning the energy function
1 20
E.
|x
x
x1
fo (x).
(11)
They indeed learn the marginal density fe and inference the data through the recovery likelihood.
The marginal density fo is adversarially trained by assigning low energy to high-probabillty regions in the data space and high energy values outside these regions. Since direct sampling from po (x x) is intractable, samples are usually drawn by leveraging Langevin dynamics (LD) (Neal) 1993).
which is a conventional sampling method of EBMs. Therefore, REBM trains marginal density Te using a kind of adversarial loss, but REBM is actually a MAP estimator implicitly defined by the sampling dynamics. In other words, REBM learns the posterior distribution using the reference diffusion process, but it does not deviate from the traditional sampling method of EBM, still generating samples through inefficient L.D There are two difficulties of such a Markov Chain Monte Carlo (MCMC) sampling: Applying MCMC in pixel space sample one instance from the model is impractical due to the high dimensionality and long inference time, As reported in (Xiao al. 22021b), the estimated density of EBMs can sometimes differ significantly from the data distribution, even if the model with the short-run LD produces relevant samples. It is also known that the convergence of LD very difficult when the energy function complicated.
Another related work is a denoising diffusion GAN (DDGAN) (Xiao et al. 2021a), which enjoys small sampling steps by using GAN. DDGAN focuses on improving the sampling efficiency while maintaining the sample quality and mode coverage of DDMs. The reason why DDMs adhere to the heavy sampling scheme is their common assumption that the true posterior is approximated by Gaussian distributions. This assumption holds only with small denoising steps. When the number of denoising steps is reduced, the denoising distribution is no longer Gaussian distribution, but non non-Gaussian multi-modal, which is usually intractable. DDGAN breaks the Gaussian assumption by reducing the number of denoising steps, and then approximates the -an-Gassiin multimodal posterior distribution with the help of GAN. DDGAN enhances the sampling efficiency of DDMs and also resolves the mode collapse problem of GANs by using a couple of denoising steps from the perspective of GAN literature. The architecture of DDGAN is somewhat similar to that of our RGM-D. However, there difference a way of estimating MAP. DDGAN assigns all responsibility for MAP estimation to the discriminator. On the other hand, our models learn the MAP-based estimator by separating the posterior distribution into the fidelity term and the prior term. Therefore, the model is much easier to learn than DDGAN. As a consequence, RGM-D obtains substantial savings in terms of training iterations than DDGAN, Specifically, in CIFARIO experiments, DDGAN takes 400K iterations to achieve FID of 3.75. In comparison, our RGM-D only uses 150K iterations to achieve the same performance as DDGAN, and takes 200K iterations for FID of 3.08. For the CelebA dataset, DDGAN requires 750K iterations attain FID 7.64, while RGM-D obtains the same FID score using only 450K iterations and FID 7.15 even with much less 500K iterations.
As such, there have been various density estimation models based denoising. Diffusion models, such as DDPM and score matching with Langevin dynamics and its variants, are HAMSE-based estimators. The model of REBM itself approximates the marginal density is trained with MAP- loss, whereas REBM generates samples from the posterior distribution through the sampling method. Diffusion models and REBM train different estimators, but both
do, but model
16

---

Under review as a conference paper at ICLR 2023
models a Langevin sampling scheme that requires thousands of network evaluations. On the other hand, DDGAN a model that can perform one-shot sampling with the help of GAN (away from the Langevin sampling), just like our RGMs. However, since DDGAN learns the whole posterior density through the discriminator, is more inefficient in terms of learning than our models, which separate the fidelity and the prior term. Consequently, our RGMs achieve better performance than DDGAN with much fewer iterations. All these models are restricted to the diffusion process. Otherwise, our RGMs can enjoy flexible forward processes and are also given degree of freedom in how to parametrize the prior term. In other words, our approach does not need to restrict to the diffusion process and unlike DDGAN, which is limited to the GAN structure, it is possible design the prior term by leveraging different generation models. This is further discussed in Appendix
B IMPLEMENTATION DETAILS
B. DEGRADATION SCHEDULE
Let A and Σ; be a degradation matrix and a noise variance on the k--th degradation step, respectively, Then, given data sampled from the real data distribution Pdatas degraded data yı the k-th forward step sampled from
(y&s x) =N (yx: A Ex).
We denote marginal distribution at the T-th degradation step as pr. Because our primary goal to bridge Pdata to easy to sample distribution pr, (especially to zero mean Gaussian distribution), we gradually decrease the norm A₁ to zero as k increases. In Section 4 we introduced two families of models based on the degradation schedule {AA,, ) with the corner cases: RGM-D for A I and RGM-SR for A Px 2 2 averaging filter. Roughly speaking, consider three models based on different forward processes designed as follows:
RGM-D: noise noise noise noise
RGM-SR (naive): downsample + noise + downsample + noise
RGM-SR: noise - downsample + noise + downsample
With the following notations
k:
2
B6 = 4 (Bmax Smin)
+
3min
(12)
T
2
T-'
k:
2
3x (Bmax Bmin)
mmi
(13)
T
where 9max = 20 and 8min 0.1. Table details the explicit form of the forward processes used for each models.
Table 5: The choice schedule A and Eg and the corresponding latent distribution pr for RGM-D, RGM-SR (naive), and RGM-SR. P& is projection matrix that downscale the images by block averaging in a factor of 2h For RGM-SR, we set 7 (12) be a half of the total steps added by one.
RGM-D
RGM-SR (naive)
RGM-SR
A
e -
P
e 0in/2 P P1a/2]
(1 (1-e-200 e1---226)21 286
11-e
28
P.P
(2 (1 e 2814/21) )) *P P Pla22
PT
(O.I)
N (0.
(0,41)
The noise schedule of RGM-D follows the Variance Preserving SDE provided by Song et al. (2020b). and others are implemented with a slight modification of them.
When the degradation matrix A the averaging filter, the corresponding forward process downsamples the image while adding Gaussian noise. RGM according to this forward process, referred to RGM-SR (naive), is demanded to super-resolve the degraded data while simultaneously
17

---

D. Under review as a conference paper at ICLR 2023
denoising it. It is considerably more difficult than the denoising task when the noise level is the same. To address this difficulty, we consider a newly scheduled degradation scheme that decomposes the forward process into downsampling and noising operations. We name the RGM designed in conjunction with this forward schedule as RGM-SR. As provided in 5 when the step is odd, the difference from the (I= + 1)-th step is only the projection matrix. Namley, only downsample is performed when sampling the (A: + 1)-th degraded data from the k-th degraded observation. Conversely, when & an even number, the forward process produces the (k + 1)-th degraded image by adding the Gaussian noise. In summary. RGM-SR focuses on denoising the data in odd steps and super-resolving the data in even steps. Provably due the difficulty of performing super-resolution and denoising simultaneouly, RGM-SR (naive) has the worst performance. Whereas RGM-SR, which uses the decomposed forward process, outperforms both RGM-D and RGM-SR by large margin as reported in Section
B.2 TRAINING RGMS
In this section, we unambiguously elucidate how we train our RGMs. In Algorithm and 2 we summarize the two training procedures that are suited different situations. Moreover, the generation process is provided in Algorithm
Training As proposed in Section 3.2 RGMs learn the data distribution Pas through the process of degrading the image through a forward process and then restoring it using the MAP-based objective (8) However, since is too difficult restore the image directly from the Gaussian distribution in one shot, we use handful of forward steps and train RGMs with the MAP estimation that recovers the distribution between each step. (We also include an ablation study on this Appendix (C.I) In other words, at each step k. we first sample a degraded image Y& of a given image ~ Paata- The generator Go generates the restored image X, and then, we degrade it by the posterior distribution ÿk-1 (ýx-1 Yk, x). We train our MAP-hased loss function so that ÿk-1 becomes restoration of yk. The discriminator loss is also imposed on the (k - 1)-th step. Through the overall process, we ultimately learn the model that restores the distribution of the previous (k 1)-th step at each k-th step. The training procedure is articulated in Algorithm
Algorithm Training of RGMs with Posterior sampling
Input: Dataset D, degradation schedule {(Ar, Σx) with (Ao, En) = (I, 0), posterior distribu- tion Pajk-1 (yk-1,ys) = N (ALYR: Στ). generator Go, discriminator D&, and regularization parameter 1 0.
1: for 0, 1, do
2: Sample data
3: Sample k ~ Uniform (0niform(11,,,) (. TT)).
4:
Sample z ~ / (0, I).
5:
Sample degraded data yı N (A₂x, mx) and Yk-1 ~ N ((A Ex-1).
6:
Generate an image x = Go (yk, k. z).
7:
Degrade data by posterior sampling ÿk-1 ppppp1))1 X).
8: Update & by the following loss:
log (1 D' (fk-1, 1)) log De (yk-1, 1).
9: Update 0 by the following loss:
(E)) & (hin-1-xh)) 2X
1
log (1 Do (8--1, - 1)) - -log Do (fx-1, -1) --
10:
end for
However, we can exactly formulate the posterior distribution only when the forward process satisfies certain conditions. For all k 1.... T. if there exists (A,, Σg} satisfying
A₁ A Σ = A > 0,
(14)
18

---

Under review as a conference paper at ICLR 2023
explicitly construct conditional distribution Palk-1 (y V N((AAyk- Σx) and posterior distribution (Ho et al. 2020 Kingma et al. [021 Xiao et al. 2021a). For example, the forward process of RGM-D falls under this condition (1)), but that of RGM-SR does not. Therefore, the Algorithm [ does not fit with RGM-SR. To unravel such a problem, we propose a prevalent algorithm that is applicable to forward processes that are in discord with the condition (14). See Algorithm 2 The only difference from the Algorithm[[] i the replacement of the posterior sampling by the prior sampling in Line 5 and the data fidelity term in Line 9. When the posterior distribution is unavailable, we corrupt the image & restored by the generator Gg to the (k - 1)-th degraded distribution using the k-th forward process rather than posterior sampling. Moreover, since the conditional distribution between k and (A: 1) steps is unknown, we adopt the fidelity term of the image & reconstructed by the generator: This algorithm universally applicable to general forward processes. One notable fact is that RGM-D, whose posterior distribution is tractable, learns the data distribution better when using this algorithm than Algorithm| This is discussed in detail in Appendix C.I.
Algorithm 2 Relaxed training algorithm of RGMs
Input: Dataset D, degradation schedule {AA, Σx)) with (Ag, Eo) = OI, discriminator Dø- generator Gg. and regularization parameter À 2
1: for i 0, 1, 2.... do
2: Sample x E 'D
3: Sample k ~ Uniform((1,,...... TT).
4: Sample z ~ N (0,I). I).
5: Sample degraded data yı N (ARx, yod Eg-
6: Generate an image & = Go ki z).
7: Degrade & by ýx-1 ~ N (A 1x-1).
8: Update e by the following loss:
log (1 - Do (ýk-1, 1)) + log Do (yk-1, 1).
9:
Update by the following loss:
|=/)² (Axt y&) 21
1
log (1 D& (fx-1, 1)) - log Do (ýK-1, k 1)+
10: end for
The sampling algorithm is summarized in Algorithm 3 Start-
Sampling
Algorithm 3 Sampling Procedure of RGMs
ing from a latent variable yr PT, the trained Go generates the restored image x Gg (yk+1, k, z) with randomly selected auxiliary variable from the (k + -he degraded image Yk+1, and then corrupt it by passing
Input: Trained generator Go and degradation schedule (Ak, Σx] [A,,Ex]>
1: Sample initial state ~ N (0, Er).
2:
for k= T 1,7 ......... do
3: Sample z ~ (0,
4: Restore image X; by X Go (y k + 1,2).
the k-th forward process. Continue this procedure until A O. When we train our model with Algorithm II the line 5 should be replaced by the pos- terior sampling.
5: Sample yk ~ N (ARăs, Ex).
6: end for
7:
return Xo
To optimize our RGMs, we are mostly following the previous literature (Xiao al. 2021a), including network architectures, R1 regularization, and optimizer settings. Note that our code is largely built on top of DDGAN (MIT License). We vary the discriminator by simply changing input channels into three. Moreover, we use learning rate of 10 for generator update in all experiments and a learning rate of 10-4 for discriminator update. We use A = 10-3 10 for image size of 256. The models are trained with
Hyperparameters
for image size of 32, and A-
https://github.... NV1 w / si ffusion-ga on-gan
19

---

Under review as a conference paper at ICLR 2023
Adam (Kingma & Ba 2014) in all experiments. In CIFARIO experiments, we train RGM-D and RGM-SR (naive) for 200K iterations and RGM-SR for 230K iterations. For CelebA-HQ, we train RGM-D for 500K iterations, and use 300K iterations in LSUN experiments. In the implementation of the two-dimensional Gaussian Mixture, we use 3-layered ML. of 32 hidden dimension for both generator and discriminator with Tanh activation. We concatenated all the inputs and passed through the network. They are trained for 100K iterations with a learning rate of 10 batch size of 1000.
Other details We train our models on CIFAR-10 using 4 V100 GPUs. The training takes approx- imately 40 hours on CIFAR-10. Moreover, the sampling 100 samples takes approximately 0.25 seconds for RGM-D on single V100 GPUs. For evaluation CIFAR1O, we use 50K generated samples to measure IS and FID. For CelebA-HO-256, we use 30K samples to compute FID.
B.3 SOLVING INVERSE PROBLEMS
Modern image processing algorithms reconstruct the groundtruth image by solving the following minimization problem:
minimize fy (x) + Ag (x).
x
where ff measures the fidelity to a corrupted observation y. and 9 constrains the solution space by measuring the complexity or noisiness of the image. Many imaging inverse problems. such as colorization, super-resolution (SR), and deblurring, fall under this form. Since the above optimization problem does not have closed-form solution in general, first-order proximal splitting algorithms, including half-quadratic splitting (HQS) (Geman & Yang 1995), alternating direction method of multipliers (ADMM) (Boyd et al.) 2011), solve the problem by operating individually on f and via the proximal operator (Parikh et al., 2014). With the aid of the emergence of deep learning, Plug-and-Plas (PnP) algorithms (Venkatakrishnan et al.) 2013) have recently begun to connect proximal splitting algorithms and deep neural networks by replacing the proximity operator of the regularization term with generic denoiser (Romano et al. 2017 Reehorst & Schniter (018).
Similarly, trained RGMs can be used as PnP priors. In Section 4.4 we solved two inverse problems, colorization and super-rese rusoutiesolution, by plugging the trained RGMs into Douglas- Rachford Splitting (DRS) algorithm (Lions& & Mercier 1979), following (Hurault al. (022). This is summarized in Algorithm 4 Starting from the degraded observation y, the DRS algorithm updates the solution by alternatively utilizing proximal operations for both f and g. By iteratively updating the solution, the solution lies far outside the distribution on which our denoiser Ga trained. For this out-distribution data, Gg cannot recover the original image distribution, which in turn prevents the DRS algorithm from convergence. To remedy this problem, the input of Gø should always be within the trained distribution. Therefore, we push the updated solution into the learned distribution through the forward process. Note that the proximal operation is calculated by utilizing efficient singular value decomposition proposed in Kawar et al. 2022).
Algorithm 4 Solving Inverse Problems by RGM
Input: A degraded observation y, fidelity loss function fy. repeat number M, update rate E (0, 1). regularization parameter À 2 0, trained generator Gg, and degradation schedule {Aki Σg 17
1: XK y
2: for 0, 1, M do
3: for i
K, K 1, 1,....; , do
4:
Sample ÿ ~ N (Ax;, E.) and z ~ A I).
5:
+ Go(Y;i- i 1,z).
6:
X + (1 - a)x; aax.
7:
Ax t prox A,y (2x- x2) X.
8:
X;-1 x; + Ax.
9: end for
end for
11:
return Xo
Settings & Hyperparameters In SR experiments, we downscale images by using a block averaging filter by in each axis. The filter is applied for stride of r. We experiment on = 4 and r = 8 for
20

---

Under review as a conference paper at ICLR 2023
LSUN and CelebA-HQ datasets. In CIFARIO experiment, experiments, we simply degrade color images to gray by averaging images along channels of each pixel. All tasks are evaluated on hundred samples which are sampled from evaluation dataset.
= 2 and r 4. In colorization
Table reports the exact set of hyperparameters that we used in our experiments. We set K = 2 for colorization and K 1 for denoising and SR tasks.
On CIFARIO experiments, to fairly compare RGM-D and vanilla version of RGM-SR, we train both models with same degradation steps of three (T = 3). For RGM-D, we used A = 6 B and
2
Σn (1 --e- e 22x I. For RGM-SR, we used A 9. Pk and Σn 11-e 23, P
Table 6: Hyperparameters used for solving inverse problems.
CIFARIO
LSUN/CelebA-HO
SR2
SR4
10/255
20/255
40/255
SR4
SR8
Color
5
10
10
10
40
20
0.2
0.1
0.01
5
10
10
5
0.2
0.2
0.2
0.1
0.1
0.05
0.05
0.5
Baselines We employed two main comparison models, namely DDRM (Kawar et al. 2022 and GAN baseline, which is close to our work. Similar to our method, both comparison models assume that degradation matrix is given and they iteratively update degraded images by using their knowl- edge obtained from pretrained network and degradation matrix. Moreover, our model and these comparisons does not require heavy additional training. The implementation DDRM follows its original implementation. The implementation of GAN baseline mainly follows the implementation of DGP (Pan al., 202Ip. however, instead of using BigGAN (Brock et al. 2019), we replaced with pretrained model of StyleSwin (Zhang et al. 2022). which is one of the state-of- fate-offthee-art For discriminator loss of DGP, we used last feature vector of StyleSwin discriminator. We additionally adjusted the weights of the losses. For experiments in SR, we use MSE loss weight of 1.0 and discriminator loss weight of 1.0. For colorization, we use MSE loss weight of 1.0 and discriminator loss of 1.0 for previous 400 iterations and 0.1 after that. Other hyperparameters of GAN baseline implementation follows Pan et al. (202I). We also compare our model with SDEdit Meng et al. (202I, stroke-based diffusion model. In the implementation of SDEdit, use total denoising steps of 200 with the number of repeats of three.
C ADDITIONAL RESULTS
C.I ADDITIONAL ABLATION STUDIES
In this section, we include additional ablation studies on our training procedure and the forward process schedule. All experiments are conducted on the CIFARIO dataset and focused on RGM-D.
Directly restoring the data distribution Given k-th de- graded image yk. the generator is trained to restore the original train RGMs directly
Table 7: Additional ablation studies on CIFARIO experiments.
image in one shot. Therefore, restore the real image distribution from each degraded step k. RGM-D trained in this say denoted by Directly matching data in the Table Z This model was trained in the same forward process as RGM-D (T = 4). The FID score shows that the model has difficulties in learning the data distribution, falling short of FID score by 21.2. seems that it is still difficult to directly restore the image of the real data distribution from severely degraded image ya (I T) even with the help of auxiliary variable z.
Model
FID (4)
Directly matching data RGM-D w/ posterior
21.2
3.52
RGM-D (T 8)
6.50
RGM-D (T - 4)
3.04
Training with posterior sampling As introduced in B.2 there are two ways to push the image, reconstructed by the generator, to the (k - 1) degraded distribution; prior sampling and posterior sampling. The posterior sampling is theoretically well- rell-grounded since it minimizes the statistical MAP loss of the posterior distribution. However, to obtain an explicit form of posterior sampling. the forward process should be constrained to satisfy the conditions (14). Since the noising forward
21

---

Under review as a conference paper at ICLR 2023
process of RGM-D satisfies these conditions, trained RGM-D with both posterior sampling (Algorithm D and prior sampling (Algorithm{2 2 under the same setup. In Table 7 RGM-D (T = 4) and RGM-D D w/ posterior refer the model trained with prior and posterior sampling, respectively. As shown in Table 7 both models achieve similar results in terms of FID score, where RGM-D with prior sampling slightly precedes posterior sampling. This verifies that two training objectives of Algorithm I and [ are somewhat consistent. Because the performance is a bit better, we adopt the prior sampling in all our experimental studies.
Effeec of the number of forward steps The number of forward steps is one of the important factors affecting the performance of the model. We investigated this in Section 4.3 by comparing a four-step model RGM-D (I' = 4) with the RGM-D (T' 1), where we use only one degradation step. As reported in Table 3 RGM-D (T' = 1) struggles to learn the data distribution because it needs to recover the real data distribution directly from Gaussian noise with one chance. On the other hand, RGM-D (T' = 4) estimates the data density well. Besides, what happens when we use more steps? Since our RGMs learn the data distribution in way that restores the distribution of the previous degradation step (k- - 1) distribution from the k-th degraded distribution, one may expect that the models will be easier to estimate the density as the distribution between the two steps is closer by dividing the forward process with more steps. However, the opposite results presented in the Table 7 The results show that RGM-D (T' = 8) attains a higher FID score. In other words, dividing the forward process into smaller pieces does not enhance the model performance. In addition, this phenomenon is also observed for Directly matching data. RGM-D (T = 1) can actually be regarded as Directly matching data (T = 1). whereas the Directly matching data presented in the table uses T = 4. Comparing these two, we can observe that that the model using fewer degradation steps performs better. Xiao al. (2021a) reported similar tendency. Choosing appropriate T is crucial for algorithmic performance, but not straightforward how many steps are optimal.
Reducing mode collapse using data fidelity Lastly, we examine the influence of the data fidelity term in our MAP-based estimation. To quantify the contribution of the fidelity term, trained RGM-D by the loss function without the data fidelity loss (termed by RGM-D (A = oo)) in Section 4.3 and we reached an FID score of 32.5 (See Table 3. This result clearly motivates our objective. Moreover, we observe the mode collapse for RGM-D (A = oc), which is the one of common failure modes of GAN. As evidence, generated samples are presented in Figure Comparing samples generated by our RGM-D (see Figure 2 to Figure 16 it is clear that images generated by RGM-D have higher diversity and better quality. The results verify that it is beneficial to train our RGMs together with the data fidelity term.
C.2 EXPERIMENTS ON THE FLEXIBILITY OF THE PRIOR TERM
Without being tied by the GAN structure presented in Section 3.2 our RGMs have the freedom to parametrize the prior term g of regularizer S in any other way. To demonstrate that RGM framework
universally works for variously parametrized prior terms, we design the prior term in two additional ways: maximum mean discrepancy (MMD) Dziugaite et al. 2015) and distributed sliced Wasserstein
distance (DSWD) (Nguyen et 2020
Different to GAN, the MMD-based generative model replaces the discriminator in GAN with a two-sample test based on kernel maximum mean discrepancy (MMD) (Li et al. 2017 For given two sets of data X = {aI, 22,,,,,2.,,,,1 and Y = {31 1/2 ......u }. the MMD prior g(X: g(X,)) which estimates the MMD distance, is defined as follows;
(X,Y)
Σ k (11, -2
2 k (2i11)) Wy) + Σ k (vi, Us)
(15)
where A is positive definite kernel. Following the prior works Dziugaite et al. 2015 Li et al.
2015 2017), we use a mixture of RBF kernels k(x,z')= = Σi= Ko: (a, x²) where Ko is a Gaussian kernel with bandwidth parameter of a.
To measure the distance of two datasets X = {01, x2 ... 2N3 } and Y {yi, U2 ... MM }. sliced Wasserstein- framework (SW) projects the data into a one dimensional vector then explicitly calculates the Wasserstein distance on the projected space. In such an explicit calculation, SW
22

---

Under review as a conference paper at ICLR 2023
Pdata
RGMGAN
RGMMM
RGMnswd
MMSE
Phoisc
Figure 8: Efficiency of MAP-basee approach compared to MMSE. The prior term of our objective is variously parametrized with GAN, MMD, and DSWD. All three MAP-based approaches much more efficient than MMSE approach, which shows the flexibility of how to parameterize the prior term of the RGMs.
be freed from unstable adversarial framework. Recently, Nguyen et al. (2220) has proposed a novel and efficient method to obtain the useful projection samples, hence, we followed the implementation of this prior work in our experiments.
The results on the 2D syntehtic example discussed in Section are depicted in Figure X The results
validate that RGMs parametrized in three different ways show consistent performance, where they are all more efficient than the MMSE estimator. In particular, MMD measures the distance between kernel. Because the kernel is not trained, the prior term in the objective of RGM is fixed rather than learned like GAN. Also, despite this simple structure, Figure 3 confirms that our RGM with MMD is more efficient than MMSE.
two distributions based
Furthermore, we also carry out the experiment of RGM-D with the DSWD prior, termed RGM- D-DSWD, on CIFARIO. RGM-D-DSWD achieves an FID score of 3.14 retaining comparable performance with RGM-D with GAN prior. The overall results verify that our MAP approach works universally well even when the prior term is parametrized way other than the GAN structure.
Implementation details For both 2D toy and CIFARIO experiments, we use the same architecture and hyperparameters with RGM-D unless stated. We use the number of iterations of 150K, the number of projections of 1000, 10 DSW iterations and Ac 10. For RGM-D-DSWD implementation on feature vector. For
CIFARIO, we use the output of fifth convolutional layer of the discriminator the DSWD experiment on the 2D data, we use the number of iterations of 100K, the number of projections of 10, 10 DSW iterations and Àc = 10. We refer to (Nguyen et al.) 2020 for the precise definition of hyperparameters. For the MMD experiment, we applied kernel bandwidths of 0. 0.5,
2, and 10.
C.3 COMPARISON WITH EXISTING MODELS USING VARIOUS DESTRUCTION
Recently, several works introduce various degradation processes as an alternative the diffusion process. Rissanen et al. 2022) proposed an inverse heat dissipation model (IHDM) with forward blurring process inspired by heat equation. Afterwards, Hoogeboom & Salimans 2022) established theoretical bridge between diffusion models and IHDM using Fourier transform. Based on this
insight, they built blurring diffusion model. Daras et al. (202 proposed a general framework for
learning the score function for any linear corruption process. Moreover, Cold Diffusion (Bansal jet al. 2022) proposed a new family of models using deterministic degradation processes. Similarly, the proposed RGMs can leverage general linear degradation processes. Therefore, we compare the
23

---

Under review as a conference paper at ICLR 2023
performance of RGMs with the aforementioned related works in Table g In comparison with model itself, the change in the forward process brings FID improvement. But compared to other models, we can observe how efficiently our MAP-based approach produces high-quality images.
Table 8: Comparison with restoration- based models with various forward processes. Sample quality CIFARIO is measured by FID score.
Model
FID (L)
NFE
Cold Diffusion (SR) Bansal ct
2022
152.76
Cold Diffusion (Blur) Bansal ct
2022
80.08
50
IHDM (Rissanen et al. 2022
18.96
200
Soft Diffusion (Daras al. 2022
3.86
100
Soft Diffusion (Blur) (Daras et al.
2022
4.64
Σ 100
Blurring Diffusion (Hoogeboom & Salımans
2022
3.17
1000
RGM-D RGM-SR
3.04
2.47
C.4 ADDITIONAL RESULTS ON INVERSE PROBLEMS
To quantify the performance our RGM, we report signal-to-noke ratio (PSNR), which measures faithfulness the ground-truth image. Also, as perceptual metric, include structural similarity index measure (SSIM) (Wang et al. 2004) that quantifies image. Table (aaleeesumriizze the PSNR and SSIM performances of colorization and super-resolution (SR) on CelebA-HQ and LSUN datasets. Since the primary goal of SDEdit is to generate realistic and faithful image in the absence of paired data, we did not make quantitative comparison with SDEdit. But we include qualitative comparisons.
The goal of image colorization to restore a gray-scale image to a colorful image with RGB channels. We present more colorization results on CelebA AHH and LSUN church in Figure 10 and II respectively. Results reported in Table 9 show that our RGM achieves comparable and sometimes even better performance than baselines. From the qualitative results, we observe that our RGM is able to reconstruct more faithful and realistic images than other models.
Colorization
Super-resolution
Super-resolution aims at recovering high high-resolution image the corresponding to given low-resolution image. We consider downsampled images with two scale scale factors 4 and We also compare SR results with bicubic interpolation. Figure 12 and 13 present the qualitative comparisons. Compared against bicubic upsampling, bicubic attains higher PSNR and SSIM values. However, we can observe from Figure and [ that bicubic interpolation results in blurry images and RGM super-resolves more plausible images, Also, visual differences between RGM and DDRM are qualitatively not large.
Table 9: Colorization and super-resolution results different methods.
Model
Colorization
Super Super-Resolution
LSUN
CelebA-HQ
LSUN
CelebA-HQ
(x4)
(x8)
(x4)
(x8)
PSNR
SSIM
PSNR SSIM
PSNR
SSIM
PSNR
SSIM
PSNR SSIM PSNR SSIM
RGM
23.78
0.93
25.57
0.93
22.74
0.65
19.96
0.48
28.51
0.81
24.86
0.70
DDRM
23.68
0.94
23.94
0.93
23.22
0.67
20.61
0.51
29.32
0.83
26.23
0.73
GAN baseline
20.02
0.81
24.79
0.88
20.32
0.48
18.06
0.34
26.77
0.71
23.92
0.59
24

---

Under review as a conference paper at ICLR 2023
C.5 ADDITIONAL RESULTS OF VARYING
We investigated the influence of the auxiliary variable z in Section 4.3 observations in Figure 9
Here, we include more
C.6 ADDITIONAL QUALITATIVE RESULTS GENERATION
We present more generated image samples in Figure 14
and 18
25

---

Under review as a conference paper at ICLR 2023
AC
O
AJIA
AC
AC
AC
AC
X1
UJA
X2
L
X3
X4
Xo
X1
X2
X3
X4
Figure 9: Illustration of the effect of varying z on CelebA-HQ (top) and LSUN (bottom). The images in the leftmost column depict the selected trajectory (x+ 14 degraded from an image X Each row on the right presents restored images of x, using four different random auxiliary values z. When the noise level is small, they generate almost identical images, which means that the restoration problem is almost well-posed. As the noise level increases, however, each degraded observation Xy estimates diverse images depending on the z. other words, the larger the noise, the more severe the ill-posedness, and the results validate that a much wider restoration is possible through the introduction z.
26

---

Under review as a conference paper at ICLR 2023
Original
Grayscule
Ours
GAN
baseline
DO
DDRM
SDEdit
Figure 10: Colorization. Qualitative comparison on CelebA-HQ.
Original
Grayscale
Ours
GAN
baseline
SDEdit
Figure 11: Colorization. Qualitative comparison on LSUN church.
27

---

Under review as a conference paper at ICLR 2023
Original (x 8)
Down Sampled (x 4)
Ours
DO
Bisubic
GAN
baseline
DDRM
SDEdit
Down Sampled
Ours
Bicubic
GAN baseline
DDRM
SDEdit
Figure 12: Super-resolution. Qualitative comparison on CelebA-HQ.
28

---

Under review as a conference paper at ICLR 2023
Original
Down Sampled (x 4)
Ours
Bisubic
GAN baseline
DDRM
SDEdit
Down Sampled (x 8)
Ours
Bicubic
GAN baseline
DDRM
SDEdit
Figure 13: Super-resolution. Qualitative comparison on LSUN church.
29

---

Sampled images of Under review as a conference paper at ICLR 2023
Figure 14: Generated samples of RGM-D on CIFAR1O.
Figure 15: Generated samples of RGM-SR on CIFARIO.
Figure 16: Mode collapse of RGM-D trained without the data fidelity term.
RGM-D (X 20) seem repetitive.
30

---

Under review as a conference paper at ICLR 2023
99
Figure 17: Additional qualitative results of RGM-D trained on CelebA-HQ-256.
31

---

Under review as a conference paper at ICLR 2023
Figure 18: More qualitative results of RGM-D trained on LSUN Church.
32


Reviews:
[{"review_id": "J_ZnzLifkjW", "reviewer": "Reviewer_VqGo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to map from Gaussian noise to images by successively \"removing'' a series of degradations. In one proposed variation, these degradations are the addition of Gaussian noise, as in the diffusion model framework, and in another variation they include downsampling. The authors train a conditional GAN to stochastically \"undo\" each degradation.", "summary_of_the_review": "This paper presents shows that a combination of diffusion models with the GAN literature can obtain competitive results, which I believe is a good contribution. The weaknesses I listed were generally related to presentation and I am recommending acceptance as long as this is improved.", "strength_and_weaknesses": "Strengths:\n- The method is an interesting combination of GANs with diffusion modelling which shines some light on the benefits of each.\n- Impressive empirical results, including state-of-the-art on CelebA-HQ at 256 resolution.\n\nWeaknesses:\n- The output of $G_\\theta(y,z)$ is repeatedly referred to as a \"MAP estimate\". This is not true; it is the stochastic output of a conditional GAN. Calling it a MAP estimate is misleading and should be changed.\n- I cannot find the training times listed anywhere, only the number of training iterations. I imagine the proposed method takes more time per iteration than a diffusion model due to the more complex GAN objective, and this should be reported. Similarly, it's not clear to me if each function evaluation at test-time is slower than for typical diffusion models. A statement on this would be interesting. \n- Progressive distillation ([Salimans & Ho, 2022](https://arxiv.org/abs/2202.00512), [Meng et al., 2022](https://arxiv.org/abs/2210.03142)) would be relevant work to comment on, since it speeds up sampling from diffusion models. [Salimans & Ho, 2022](https://arxiv.org/abs/2202.00512) achieve a FID of 3.0 on CIFAR10 with 4 NFEs and should probably be added to Table 1.\n\nMinor:\n- ~~It is not clear what metric is being reported in Table 4.~~", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H91mx3E4Lj", "reviewer": "Reviewer_58kp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a generative model in terms of image restoration, named restoration-based generative model (RGM). This paper eliminates expensive sampling by performing MAP estimation and incorporating implicit prior information via GAN. Furthermore, a multi-scale training is proposed to alleviate the latent inefficiency of DGM. The ablation study demonstrates the effect of all parts of the MAP objective. A series of experiments on different datasets demonstrate that the RGM achieves state-of-the-art performance when the number of forward steps is limited.", "summary_of_the_review": "This paper proposes a new generative model, named restoration-based generative models (RGMs). RGM uses MAP estimation and incorporates implicit prior information via GAN. Furthermore, RGM is not limited to denoising and can be effectively extended to other IR tasks, like image SR. The overall writing and organization are good. It is a novel and meaningful work.", "strength_and_weaknesses": "**strength**\n\n-  The authors propose  a new flexible family of generative models, named restoration-based generative models (RGMs). RGMs achieve state-of-the-art performance using a limited number of forward steps.\n- The proposed MAP estimation is efficacy, and a 2D example gives a visual validation.\n- The ablation study results are extensive and demonstrate the effects of all parts of the MAP. \n- The main paper and supplementary file are well prepared. The motivation is clear and reasonable. The paper is carefully organized.\n- The authors extend RGM to general restoration and propose a new model established upon super-resolution (SR).\n- The authors also provide code, which further shows the solidness of the work.\n\n**weaknesses**\n\n- The multi-scale training is introduced to alleviate the latent inefficiency of DGMs, where experiments are needed to prove it.\n- Figure 4 is not mentioned in the article, and some introduction should be added.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EFPmDJS41M", "reviewer": "Reviewer_3aDJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposed a variant of generative models called restoration based generative models (RGM). The key idea is based on a new interpretation of denoting generative models (DGMs) from an image restoration (IR)  perspective. By replacing the MMSE denoiser with MAP denoiser and introducing a regularized prior imposed by another generative models such as GAN, RGM apparently reduces the number of sampling steps. RGM has also been applied to inverse problems. ", "summary_of_the_review": "Overall the idea of RGM is interesting. From the IR perspective, RGM is designed by  imposing an additional generative model such as GAN as an implicit prior. My  main concerns are about the essential need of an additional generative model such as GAN, which might affect the training, stability, flexibility, and efficiency of the proposed RGM as described above. \n\nUpdate after rebuttal:\n\nI raised the score accordingly. ", "strength_and_weaknesses": "Strengths:\n1. A new perspective of RGM from IR is provides, which is interesting and opens up possibilities for improvements from denoiser design perspective.\n\n2. Using MAP with a regularized prior imposed by GAN, RGM achieves very good performance on standard datasets, and in particular, with a very few  (tens of) samplings steps.\n\n3. Extended applications in inverse problems are enabled. \n\nWeaknesses:\n\n1. The training process of RGM is much difficult than previous RGM, as shown in Algorithm 1.  In particular, a prior generator such as GAN is needed. In other words, two generative models are actually needed. Suppose that there is no GAN available at hand, then, to train a RGM, we have to first train a GAN, which itself is not an easy task. This kind of double-training is not preferred. \n\n2. Also related to the use of GAN as implicit prior. Intuitively, it is unsurprising that the performance would improve if an additional generative model is imposed as a prior. The authors claim that the use of MMSE loss is not as good as MAP. The authors verified the benefits of MAP in Figure 3 in the toy model.  I am a bit suspicious of this point. How can we confirm that the improvement comes from the MAP, rather than the implicit prior imposed by the additional generative model GAN in RGM? \n\n3. How is the performance of RGM for out-of-distribution datasets for inverse problems? It seems that by imposing the additional GAN prior, the obtained RGM is difficult to generalize to OOD datasets. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9Uf-tk0x-Fr", "reviewer": "Reviewer_otH9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes three changes to vanilla diffusion models: i) a regularization term in the training objective, ii) a source of randomness in the training objective to account for the fact that there is no single solution to the denoising problem iii) a class of more general corruption processes. The paper demonstrates that the resulting models are faster to sample from and they achieve better or on par with the baselines.\n", "summary_of_the_review": "The paper presents some interesting ideas: GAN regularization in the training objective, different forward processes and a stochastic knob in the input of the generative model. However, there are several weaknesses (detailed above) that prevent me from recommending acceptance. The most important issue for me right now is that the main benefits seem to be coming from the change of the forward process and not from the regularization. If that's the case, I think the presentation of the paper needs a lot of restructuring and the baselines for the comparisons should be other methods that generalize diffusion.\n\nUpdate, Nov. 13: The authors addressed some of my concerns in the rebuttal and I am increasing the score from 3 to 5. I believe that there are still some concerns about the novelty of this work, but I am open to more discussion and perhaps a further increase of the score if these issues are addressed. ", "strength_and_weaknesses": "Strengths:\n* The ideas presented in the paper are interesting. It is intuitive that better priors, introduced by a different forward model and a regularization term, can lead to better models.\n* Experimental results are quite convincing.\n* The observation that optimizing for super-resolution leads to better results for super-resolution is interesting. There are many recent papers that generalize diffusion to non-gaussian corruptions (e.g. Cold Diffusion, Soft Diffusion, Blurring Diffusion Models, etc) and it would be interesting to evaluate whether these models also perform better on the inverse problem they train for.\n\n\nWeaknesses:\n* The paper is poorly written. See analytical comments below.\n* The classical objective *is* a MAP method. As long as there is no stochasticity in the solution, you are maximizing something -- the only thing that is changing is your prior (imposed by the regularization term).\n* It is interesting that you introduce stochasticity to account for the ill-posedness nature of the inverse problems. However, what prevents the model from dismissing entirely the randomness? It is just an additional input, right? The model can always ignore it. Why is it best to not ignore it for minimizing the objective?\n* I am struggling to understand the added regularization. It seems that it is a trainable Discriminator (similar to the Discriminator in GANs). If that's the case, then you should also feed to the discriminator the real images, right? Eq. (8) doesn't make sense to me -- if the Discriminator doesn't see real images, how is it regularizing? How is the generator not fooling the discriminator? Now, even if we assume that there is a typo and the Discriminator actually sees real images, doesn't that introduce the training instabilities of GANs? Or mode collapse and less diversity? Please clarify these points in the rebuttal.\n* The theoretical underpinnings of the proposed objective are not discussed/explained. When there is no regularization term, it has been shown that the DSM objective learns the score function (and minimizes the KL between the true and the sampling process). Can similar guarantees be achieved for this new objective?\n* How is the sampling rule derived by using this objective? Since there is no guarantee that you are learning the score-function, can you still solve for the reverse SDE and get reasonable samples? Can we compute likelihoods? If so, how? Would be interesting to add likelihood evaluations to the paper.\n* The paper has several inaccurate and/or unsupported claims. It is not true that all diffusion models diffuse in the pixel space, e.g. see Latent Diffusion Models. It is also not true that Diffusion models typically use thousands of steps. The is a huge line of work for accelerating diffusion models, including Progressive Distillation. The state-of-the-art claim for CelebA-HQ-256 is not exactly accurate. From a quick search, it seems that StyleSwin (which is used as a baseline for the inverse problems) achieves FID 3.25, which is 2x better.\n* From the experimental evaluation of the paper, it seems that the main contribution to the improvement in the FID scores is the change of the forward process. RGM-D performs *worse* than the simple VE SDE. There are several recent works that use different distributions for corrupting images, e.g. Cold Diffusion, Soft Diffusion, Blurring Diffusion Models, Generative Modelling With Inverse Heat Dissipation, etc. Since the improvement is coming from the change of corruption, these other works should be discussed more extensively. Ideally, a performance comparison with these works should be made.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject