Learning multi-scale local conditional probability models of images
Authors: Zahra Kadkhodaie
Florentin Guth
Stéphane Mallat
Eero P Simoncelli
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Image priors
Markov wavelet conditional models
multi-scale score-based image synthesis
denoising
super-resolution

Abstract: Deep neural networks can learn powerful prior probability models for images, as evidenced by the high-quality generations obtained with recent score-based diffusion methods. But the means by which these networks capture complex global statistical structure, apparently without suffering from the curse of dimensionality, remain a mystery. To study this, we incorporate diffusion methods into a multi-scale decomposition, reducing dimensionality by assuming a stationary local Markov model for wavelet coefficients conditioned on coarser-scale coefficients. We instantiate this model using convolutional neural networks (CNNs) with local receptive fields, which enforce both the stationarity and Markov properties. Global structures are captured using a CNN with receptive fields covering the entire (but small) low-pass image. We test this model on a dataset of face images, which are highly non-stationary and contain large-scale geometric structures.
Remarkably, denoising, super-resolution, and image synthesis results all demonstrate that these structures can be captured with significantly smaller conditioning neighborhoods than required by a Markov model implemented in the pixel domain. Our results show that score estimation for large complex images can be reduced to low-dimensional Markov conditional models across scales,  alleviating the curse of dimensionality. 

Published as a conference paper at ICLR 2023
LEARNING MULTI-SCALE LOCAL CONDITIONAL PROBABILITY MODELS OF IMAGES
Zahra Kadkhodaie
Florentin Guth
CDS, New York University zk388@nyu.
DI, ENS, CNRS, PSL University fl florentin, www.iiinnntttttss...m.mmmmm fr
Stéphane Mallat
Eero P. Simoncelli
Collège de France
CNS, Courant, and CDS, New York University Flatiron Institute, Simons Foundation eero. www.ncellimmyu.........m edu
Flatiron Institute. Simons Foundation st stephane mallat (ens. fr
ABSTRACT
Deep neural networks can learn powerful prior probability models for images, as evidenced by the high-quality generations obtained with recent score-based diffusion methods. But the means by which these networks capture complex global statistical structure, apparently without suffering from the curse of dimension- ality, remain mystery. To study this, incorporate diffusion methods into a multi-scale decomposition. reducing dimensionality by assuming a stationary local Markov model for wavelet coefficients conditioned on coarser-scale coefficients. We instantiate this model using convolutional neural networks (CNNs) with local receptive fields, which enforce both the stationarity and Markov properties. Global structures are captured using a CNN with receptive fields covering the entire (but small) low-pass image. We test this model on dataset of face images, which are highly non-stationary and contain large-scale geometric structures. Remark- ably, denoising, super-resolution, and image synthesis results all demonstrate that these structures can be captured with significantly smaller conditioning neighbor- hoods than required by a Markov model implemented in the pixel domain. Our results show that score estimation for large complex images can be reduced to low-dimensional Markov conditional models across scales, alleviating the curse dimensionality.
Deep neural networks (DNNs) have produced dramatic advances in synthesizing complex images and solving inverse problems, all of which rely (at least implicitly) on prior probability models. Of particular note is the recent development of "diffusion methods" (Sohl-Dickstein et al., 2015), in which network trained for image denoising incorporated into iterative algorithm to draw samples from the prior (Song & Ermon, 2019: Ho et al., 2020; Song et al., 2021). or to solve inverse problems by sampling from the posterior (Kadkhodaie & Simoncelli, 2020; Cohen et al., 2021; Kawar et al., 2021; Daras et al., 2022). The prior these procedures is implicitly defined by the learned denoising function, which depends on the prior through the score (the gradient of the log density). But density or score estimation is notoriously difficult for high-dimensional signals because of the curse of dimensionality: worst-case data requirements grow exponentially with the data dimension. How do neural network models manage to avoid this curse?
Traditionally, density estimation is made tractable by assuming simple low-dimensional models, or structural properties that allow factorization into products of such models. For example, the classical Gaussian spectral model for images or sounds rests on an assumption of translation-invarinn (stationarity), which guarantees factorization in the Fourier domain. Markov random fields (Geman & Geman, 1984) assume localized conditional dependencies, which guarantees that the density can be factorized into terms acting on local, typically overlapping neighborhoods (Clifford & Hammersley, 1971). In the context of images, these models have been effective in capturing local properties, but not sufficiently powerful to capture long-range dependencies. Multi-scale image decompositions offered a mathematical and algorithmic framework better suited for the structural properties of images (Burt& & Adelson, 1983; Mallat, 2008). The multi-scale representation facilitates handling of larger

---

Published as a conference paper at ICLR 2023
structures, and local (Markov) models have captured these probabilistically (e.g.. Chambolle et al. (1998); Malfait & Roose (1997): Crouse et al. (1998); Buccigrossi & Simoncelli (1999); Paget & Longstaff' (1998); Mihçak et al. (1999); Wainwright et al. (2001); Şendur & Selesnick (2002); Portilla et al. (2003); Cui & Wang (2005); Lyu & Simoncelli (2009)). Recent work, inspired by renormalization group theory in physics, has shown that probability distributions with long-range dependencies can be factorized as a product of Markov conditional probabilities over wavelet coefficients (Marchand et al., 2022). Although the performance of these models is eclipsed by recent DNN models, the concepts on which they rest sttt-tonaioottt. locality and multi-scale conditioning are still of fundamental importance. Here, we use these tools to constrain and study score-based diffusion model.
number of recent DNN image synthesis methods-includiing variational auto-encoders (Chen et al., 201 8), generative adversarial networks (Gal et al., 2021) normalizing flow models (Yu et al., 2020; Li, 2021)), )) and diffusion models (Ho et al., 2022; Guth et al., 2022)- usee coarse-to-fine strategies, generating sequence of images of increasing resolution, each seeded by its predecessor. With the exception of the last, these do not make explicit the underlying conditional densities, and none impose locality restrictions on their computation. On the contrary, the stage-wise conditional sampling is typically accomplished with huge DNNs (up to billions of parameters), with global receptive fields.
-use
Here, we develop low- -iw-dimeniional probability model for images decomposed into multi-scale wavelet sub-bands. Following the renormalization group approach, the image probability distribution is factorized as a product of conditional probabilities of its wavelet coefficients conditioned by coarser scale coefficients. We assume that these conditional probabilities are local and stationary, and hence can be captured with low-dimensional Markov models. Each conditional score can thus be estimated with conditional CNN (cCNN) with a small receptive field (RF). The score of the coarse-scale low- pass band (a low-resolution version of the image) modeled using CNN with a global RF, enabling representation of large-scale image structures and organization. We test this model on dataset of face images, which present a challenging example because of their global geometric structure. Using coarse-to-fine fine anti-diffusion strategy for drawing samples from the posterior (Kadkhodaie & Simoncelli, 2021), we evaluate the model on denoising, super-resolution, and synthesis, and show that locality and stationarity assumptions hold for conditional RF sizes small as 9 x 9 without harming performance. In comparison, the performance of CNNs restricted to a fixed RF size in the pixel domain dramatically degrades when the RF is reduced to such sizes. Thus, high-dimensional score estimation for images can be reduced to cww-dimeniional Markov conditional models, alleviating the curse of dimensionality, A software implementation is available at https: / / writthe thub. . com/ onn LabFo r inna Vis www.iiiiiinnnnnnoi walssionnooccc /lcplllcllllll wwrobbaalalaaaaaaao probabi i 1 mity-modell o-oo wetoffimmmm images
MARKOV WAVELET CONDITIONAL MODELS
Images are high-dimensional vectors. Estimating an image probability distribution or its score therefore suffer from the curse of dimensionality, unless one limits the estimation to relatively low-dimensional model class. This section introduces such a model class as a product of Markov conditional probabilities over multi-scale wavelet coefficients.
Markov random fields (Dobrushin, 1968; Sherrington & Kirkpatrick, 1975) define low-dimensional models by assuming that the probability distribution has local conditional dependencies over graph, which is known a priori. One can then factorize the probability density into product of conditional probabilities, each defined over small number of variables (Clifford & Hammersley, 1971). Markov random fields have been used to model stationary texture images, with conditional dependencies within small spatial regions of the pixel lattice. At a location u, such Markov model assumes that the pixel value (=)), conditioned on pixel values (=))) for u in neighborhood of is independent from all pixels outside this neighborhood. Beyond stationary textures, however, the chaining of short- range dependencies in pixel domain has proven insufficient to capture the complexity of long-range geometrical structures. Many variants of Markov models have been proposed (c.g., Geman & Geman (1984); Malfait & Roose (1997); Cui & Wang (2005)), but none have demonstrated performance comparable to recent deep networks while retaining a local dependency structure.
Based on the renormalization group approach in statistical physics (Wilson, 1971), new probability models are introduced in Marchand et al. (2022), structured as a product of probabilities of wavelet coefficients conditioned on coarser-scale values, with spatially local dependencies. These Markoy
2

---

Published as a conference paper at ICLR 2023
W
W
W
Figure 1: Markov wavelet conditional model structure. At each scale j. orthogonal wavelet transform W decomposes an image -; into three wavelet channels, Ty. containing vertical, horizontal, and diagonal details, and a low-pass channel æ; containing coarse approximation of the image, all subsampled by a factor of two. At each scale j. we assume a Markov wavelet conditional model, in which the probability distribution of any wavelet coefficient of žj (here, centered on the left eye), conditioned on values of , and 3, in a local spatial neighborhood (red squares), is independent of all coefficients of v outside this neighborhood.
conditional models have been applied to ergodic stationary physical fields, with simple conditional Gibbs energies that are parameterized linearly. Here, we generalize such models by parameterizing conditional Gibbs energy gradients with deep conditional convolutional neural networks having a local RF This yields a class of Markov wavelet conditional models that can generate complex structured images, while explicitly relying local dependencies to reduce the model dimensionality.
An orthonormal wavelet transform uses a convolutional and subsampling operator W defined with conjugate mirror filters (Mallat, 2008), to iteratively compute wavelet coefficients (see Figure 1). Let 2p be an image of N x N pixels. For each scale j > 1, the operator W decomposes σ5-1 into:
Wæ (z), ;.));
where z, is lower-resolutig image and æ, is an array of three wavelet coefficient images, each with dimensions N/2 x N/22, as illustrated in Figure The inverse wavelet transform iteratively computes ---1 W 1.))))
We now introduce the wavelet conditional factorization of probability models. Since W is orthog- onal, the probability density of i is also the joint density of (z)) It can be factorized by conditioning on æy:
p(a, p(a,, 2By) = p(z,))((z(zz))
This is performed J times, so that the lowest resolution image æy is small enough, which yields:
p(x) = p(æ))
II j=1
p ((,))
The conditional distributions p(ε,|az;) (x;;) specify the dependencies of image details at scale j condi- tioned on the coarser scale values, and may be expressed in terms of a conditional Gibbs energy:
p(,,1æ;;) (z,)) -E, (#,
where Z(x,)) is the normalization constant for each æ,. The conditional Gibbs energies (2) have been used in the wavelet conditional renormalization group approach to obtain stable parameterization of the probability model even at critical phase transitions, when the parameterization of the global Gibbs energy becomes singular (Marchand et al., 2022).
Local wavelet conditional renormalization group models (Marchand et al., 2022) further impose that (x))) is a conditional Markov random field. That is, the probability distribution of wavelet coefficient of ă; conditioned on values of x, and ãú in a restricted spatial neighborhood independent of all coefficients of 3; and a outside this neighborhood (see Figure 1). The Hammersley- Clifford theorem states that this Markov property is equivalent to imposing that E, can be written sum of potentials, which only depends upon values of ã. and T, over local cliques (Clifford & Hammersley, 1971). This decomposition substantially alleviates the curse of dimensionality, since one only needs to estimate potentials over neighborhoods of fixed size which does not grow with
3

---

Published as a conference paper at ICLR 2023
the image size. To model ergodic stationary physical fields, the local potentials of the Gibbs energy Ej have been parameterized linearly using physical models Marchand et al. (2022).
We generalize Markov wavelet conditional models by parameterizing the conditional score with conditional CNN (cCNN) having small receptive fields (RFs):
Væ, log p(z z,) = Væ,
Computing the score (3) is equivalent to specifying the Gibbs model (2) without calculating the normalization constants 2(x,), since these not needed for noise removal, super-resolution or image synthesis applications.
2 SCORE-BASED MARKOV WAVELET CONDITIONAL MODELS
Score-based diffusion models have produced impressive image generation results (c.g., Song et al. (2021); Ho et al. (2022); Rombach et al. (2022); Saharia et al. (2022); Ramesh et al. (2022)). To capture large-scale properties, however, these networks require RFs that encompass the entire image. Our score-based wavelet conditional model leverages the Markov assumption to compute the score using cCNNs with small receptive fields, offering a low-dimensional parameterization of the image distribution while retaining long-range geometric structures.
Let y x be noisy observation of clean image æ N drawn from p(æ)). with (0, oddd) a sample of Gaussian white noise. The minimum mean squared error (MMSE) estimate of the true image is well-known be the conditional mean of the posterior:
+ (v)
app(# (y)
This integral can be re-expressed in terms of the score:
(1y) y + o² o'V logr(y).
(5)
This remarkable result, published in Miyasawa (1961), exposes direct and explicit relationship between the score probability distributions and denoising (we reproduce the proof in Appendix for completeness). Note that the relevant density is not the image distribution, p(x), but the noisy observation density p(y). This density converges to p(x) as the noise variance o² goes to zero.
Given this relationship, the score can be approximated with parametric mapping optimized to estimate the denoised image, fzy Re æ(y). Specifically, we implement this mapping with CNN, and optimize its parameters by minimizing the denoising squared error (] æ|1 over large training set of images and their corrutcd counterparts. Given eq. (5), the denoising residual, f(y) - y. provides an approximation of the variance-weighted score, o²V, log p(y). Also known as denoising score matching (Vincent, 201 I), such denoiser-estimated score functions have been used in iterative algorithms for drawing samples from the density (Song et al., 2021: Ho et al., 2020; Dhariwal & Nichol, 2021; Ho et al., 2022), or solving inverse problems (Kadkhodaie & Simoncelli, 2020; Cohen et al., 2021: Kawar et al., 2021: Laumont et al., 2022).
To model the conditional wavelet distribution p( xz)) Guth et al. (2022) parameterize the score Vg, log y (z,) of noisy wavelet coefficients y conditioned on clean low-pass image x, with a cCNN (eq. (3)). Specifically, the cCNN takes as input three noisy wavelet detail channels, along with the corresponding low-pass channel, and generates three estimated detail channels. The network is trained to minimize mean square distance between 2; and fz m) Thanks to a conditional extension of eq. (5), an optimal network computes fy æ.,) = Vg, 9, log p(z) (zz)) Additionally, at the coarsest scale J, CNN denoiser fy is trained to estimate the score of the low-pass band, Vy, log p(x,) by minimizing mean square distance between J, and (y.).
The following theorem proves that the Markov wavelet conditional property is equivalent to imposing that the cCNN RFs are restricted to the conditioning neighborhoods. The RF of a given element of the network response is defined as the set of input image pixels on which this element depends.
Theorem 1.
The wavelet conditional density p( p(ã, { is Markovian over a family of conditioning neighborhoods if and only if the conditional score V log log p(t (x)) can be computed with a network whose RFs included in these conditioning neighborhoods.
4

---

Published as a conference paper at ICLR 2023
CNN
cCNN
WT
W
cONN
W
Figure 2: Wavelet conditional denoiser architecture used to estimate the score (illustrated for a two- scale decomposition). The imput noisy image y (lower left) is decomposed by recursive application of a fast orthogonal wavelet transform W into successive low-pass images yj (blue) and three wavelet detail images Yj (red). The coarsest low-pass image y. denoised using a CNN with global receptive field to estimate æt.. At all other scales, local conditional CNN (cCNN) estimates ă from y, conditioned on Zy, from which W recovers ø5-1-
The proof of the theorem is provided in Appendix A. Note that even if the conditional distribution of clean wavelet coefficients (falaay)) satisfies local Markov property, the noisy distribution p(y,(æ;) is in general not Markovian. However, we shall parameterize the scores with a cCNN with small RFs and hence show that both the noisy and clean distributions are Markovian. At each scale 1 S j S J, the cCNN has RFs that are localized in both D, and 2j. and have fixed size over all scales, independent of the original image size. From Theorem 1, this defines the Markov conditioning neighborhoods of the learned model. The effect of the RF size is examined in the numerical experiments of Section 3.
Parameterizing the score with convolutional network further implies that the conditional probabilityy p(øy ( is stationary on the wavelet sampling lattice at scale j. Despite these strong simplifications, we shall see that these models models are able to capture complex long-range image dependencies in highly non-stationary image ensembles such centered faces. This relies the low-pass CNN, whose RF is designed to cover the entire image æ,, and thus does not enforce local Markov conditioning nor stationarity. The product density of eq. (1) is therefore not stationary.
3 MARKOV WAVELET CONDITIONAL DENOISING
We now evaluate our Markov wavelet conditional model on a denoising task. We use the trained CNNs to define multi-scale denoising architecture, illustrated in Figure 2. The wavelet transform of the input noisy image y is computed up to coarse-scale J. The coarsest scale image is denoised by applying the denoising CNN learned previously: ě = fz(y). Then for 2 2 1, we compute the denoised wavelet coefficients conditioned the previously estimated coarse image: ã; [ūj ..y)- We then recover a denoised image at the next finer scale by applying an inverse wavelet transform: ② WT ( ).)) At the finest scale obtain the denoised image à 20
Because of the orthogonality of W, the global MSE can be decomposed into a sum of wavelet MSEs
و 1|2 The global MSE thus summarizes the precision of the score models computed over all scales. We evaluate the peak signal-to-noise ratio (PSNR) of the denoised image as function of the noise level, expressed as the PSNR of the noisy image. We use the CelebA dataset (Liu et al., 2015) at 160 X 160 resolution. We use the simplest of all orthogonal wavelet decompositions, the Haar wavelet, constructed from separable filters that compute averages and differences of adjacent pairs of pixel values (Haar, 1910). All denoisers are "universal" (they can operate on images contaminated with noise of any standard deviation), and "blind" (they are not informed of the noise level). They all have the same depth and layer widths, and their receptive field size is controlled by changing the convolutional kernel size of each layer. Appendix C provides architecture and training details.
at each scale, plus the coarsest scale error: || 2 = Σ || - 1|
Figure shows that the multi-scale denoiser based on a conditional wavelet Markov model outper- forms conventional denoiser that implements Markov probability model in the pixel domain. More precisely, we observe that when the Markov structure is defined over image pixels, the per- formance degrades considerably with smaller RFs (Figure 3, left panel), especially at large noise
5

---

Published as a conference paper at ICLR 2023
4bo1 12623
37- 23x23
13613
AF 13033
RF fxd
17 5x3
1zars by
orly low pods
psnr out
Idaniiy
22
psnr in
psnr in
Figure 3: Comparison denoiser performance on 160 160 images from the CelebA dataset. Fach panel shows the error of the denoised image function of the noise level, both expressed as the peak signal-to-noise ratio (PSNR). Left: Conventional CNN denoisers with different RF sizes. The blue curve shows performance of BF-CNN (Mohan et al., 2020). The rest are BF-CNN variants with smaller RF obtained from setting some intermediate filter sizes to 1 x 1. Right: Multi-scale denoisers, depicted in Figure 2, with different cCNN RF sizes. Note that the low-pass denoiser RF 40 x 40 in all cases, and thus covers the entire low-pass band.
levels (low PSNR). Images thus contain long- longrrange global dependencies that cannot be captured by Markov models that are localized within the pixel lattice. On the other hand, multi-scale denoising performance remains nearly the same for RF sizes down to x 9, and degrades for x 5 RFs only at small noise levels (high PSNR) (Figure 3, right panel). This is remarkable considering that, for the finest scale, the 9 x RF implies conditioning on one percent the coefficients. The wavelet conditional score model thus successfully captures long-range image dependencies, even with small Markov neighborhoods.
It is also worth noting that in the large noise regime (i.e., low PSNR), all multi-scale denoisers (even with RF as small as 5 x 5) significantly outperforms the conventional denoiser with the largest tested RF size (43 43). The dependency on RF size in this regime demonstrates the inadequacy of local modeling in the pixel domain. On the contrary, the effective neighborhoods of the multi-scale denoiser are spatially global, but operate with spatially-varying resolution. Specifically, neighborhoods are of fixed size at each scale, but due to the subsampling, cover larger proportions of the image at coarser scales. The CNN applied to the coarsest low-pass band (scale JJ) spatially global, and the denoising of this band alone explains the performance at the highest noise levels (magenta curve, Figure 3).
To further illustrate this point, consider the denoising examples shown in Figure 4. Since all denoisers are bias-free, they are piecewise linear (as opposed to piecewise affine), providing some interpretability (Mohan et al., 2020). Specifically, each denoised pixel is computed as an adaptively weighted sum over the noise input pixels. The last panels show the equivalent adaptive linear filter that was used to estimate the pixel marked by the green square, which can be estimated from the associated row of the Jacobian. The top shows denoising results of conventional CNN denoiser for small images that are the size of the network RF. Despite very heavy noise levels, the denoiser exploits the global structure of the image, and produces result approximating the clean image. The second row shows the results after training the same denoiser architecture on much larger images. Now the adaptive filter is much smaller than the image, and the denoiser solution fails capture the global structure of the face. Finally, the last row shows that the multi-scale wavelet conditional denoiser can successfully approximate the global structure of a face despite the extreme levels of noise. Removing high levels of noise requires knowledge of global structure. In our multi-scale conditional denoiser, this achieved by setting the RF size of the low-pass denoiser equal to the entire low-pass image size, similarly to the denoiser shown on the top row. Then, each successive conditioning stage provides information at a finer resolution, over ever smaller RFs relative to the coefficient lattice. The adaptive filter shown in the last column has a foveated structure: the estimate of the marked pixel depends on all pixels in the noisy image, but those that are farther away are only included within averages over larger blocks. Thus, imposing locality in the wavelet domain lifts the curse dimensionality without loss of performance, as opposed to a locality (Markov) assumption in the pixel domain.

---

Published as a conference paper at ICLR 2023
Figure 4: Denoising examples. Each row shows clean image, noisy image, denoised image, and the adaptive filter (one row of the Jacobian of the end-to-end denoising transformation) used by the denoiser to estimate a specific pixel, indicated in green. The heat-map ranges from red for most negative to black for most positive values. In the last two rows, the last column shows an enlargement of this adaptive filter, for enhanced visibility. Images are displayed proportional to their sizes. Top row: 40 x 40 images estimated with CNN denoiser with RF 40 x 40. Second row: 160 x 160 images estimated with CNN denoiser with RF 43 x 43. Third row: 160 x 160 images, estimated with the proposed conditional multi-scale denoiser of Figure 2. The denoiser uses 40 x 40 RF for the coarsest scale, and 13 x 13 RFs for conditional denoising subsequent finer scales.
MARKOV WAVELET CONDITIONAL SUPER-RESOLUTION AND SYNTHESIS
We generate samples from the learned wavelet conditional distributions in order to visually assess the quality of the model in a super-resolution task. We compare this approach with solving the super-resolution inverse problem directly using CNN denoiser operating in the pixel domain. We also compare the models on image synthesis.
We first give high-level description of our conditional generation algorithm. The low-resolution image x, is used to conditionally generate wavelet coefficients E from the conditional distribution ( An inverse wavelet transform next recovers higher-resolution image I- from both æ, and øJ. The conditional generation and wavelet reconstruction are repeated times, increasing the resolution of the sample each step. In the end, we obtain sample æ from the full-resolution image distribution conditioned on the starting low-resolution image p(x[ æ is thus a stochastic super-resolution estimate of z.j-
To draw samples from the distributions ( |22)) implicitly embedded the wavelet conditional denoisers, we the algorithm of which performs stochastic gradient ascent the log-probability obtained from the cCNN using eq. (5). This is similar to score-based diffusion algorithms (Song Ermon, 2019; Ho et al., 2020; Song et al., 2021). but the timestep syper-parameter require essentially no tuning, since stepsizes are automatically obtained from the magnitude of the estimated score. Extension the conditional case is straightforward. The sampling algorithm detailed in Appendix D. All the cCCN denoisers have RF size of 13 x 13. Train and test images are from the CelebA HQ dataset (Karras et al., 2018) and of size 320 320. Samples drawn using the conditional denoiser correspond to a Markov conditional distribution with neighborhoods restricted to the RFs of denoiser. We compare these with samples from model with local Markov neighbhorhood the pixel domain. This is done using CNN with 40 x 40 RF trained to denoise full-resolution images, which approximates the score log)( Given the same low-pass image æ,,, we can generate samples from p(æ: æ by viewing this sampling from the image distribution of constrained by linear measurements Æ. This is done with the same sampling algorithm, with small modification, again described in Appendix D.
Figure shows super- repouutonol samples from these two learned image models. The local Markov model in the pixel domain generates details that are sharp but artifactual and incoherent over long
7

---

Published as a conference paper at ICLR 2023
Figure 5: Super-resolution examples. Column 1: original images (320 x 320). Column 2: Low-pass
image of a 3-stage wavelet decomposition (downsampled to 40 40) expanded to full size for viewing. Column 3: Conditional generation of full-resolution images using CNN denoiser with RF of size 43 43. Column 4: Coarse-to-fine conditional generation using the multi-scale cCNN denoisers, each with RFs of size 13 x 13. Additional examples are shown in Appendix E.
spatial distances. On the other hand, the Markov wavelet conditional model produces much more lnourag face images. This demonstrates the validity our model: although these face images are not stationary (they have global structures shared across the dataset), and are not Markov in the pixel domain (there are clearly long-range dependencies that operate across the image), the details can be captured with local stationary Markov wavelet conditional distributions.
We also evaluated the Markov wavelet conditional model on image synthesis. We first synthesize a 40 x 40 terminal low-pass image using the score, Væ, log p(æ,), obtained from the low-pass CNN denoiser with a global RF. Again, unlike the conditional wavelet stages, this architectural choice does not enforce any local Markov structure nor stationarity. This global RF allows capturing global non-stationary structures, such as the overall face shape. The synthesis then proceeds using the same coarse-to-fins steps used for super-resolution resolution: wavelet coefficients each successive scale generated by drawing a sample using the cCNN conditioned on the previous scale.
The first (smallest) image in Figure is generated from the low-pass CNN (see Appendix D for algorithm). We can see that it successfully captures the coarse structure face. This image is then refined by application of successive stages of the multi-scale super-resolution synthesis algorithm described above. The next three images in Figure 6 show successively higher resolution images generated in the procedure. For comparison, the last image in Figure shows a sample generated using conventional CNN with equivalent RFs trained on large face images. Once again, this illustrates that assuming spatially localized Markov property on the pixel lattice and estimating the score with CNN with RF smaller than the image fails to capture the non- non-stationary distribution of faces. Specifically, the model is unable to generate structures larger than the RF size, and the samples are texture-like like and composed of local regions resembling face parts.
We note that the quality of the generated images not on par with the most recent basee diffusion methods (e.g.. Ramesh al. (2022); Saharia al. (2022); Rombach et al. (2022)), which have also been used for iterative super-resolution of an initial coarse-scale sample. These methods use much larger networks (more than billion parameters, compared to ours which uses 600k parameters for the low-pass CNN and 200k for the cCNNs), and each scale-wise denoiser is itself a U-Net, with associated RF covering the entire image. Thus, the implicit probability models in these networks
8

---

Published as a conference paper at ICLR 2023
Figure 6: Image synthesis. Left four images: Coarse-to-fine synthesis, achieved by sampling the score learned for each successive conditional distribution. Synthesized images are shown at four resolutions, from coarse-scale only (leftmost, 40 40) to the finest scale (rightmost, 320 x 320). Conditional RFs are all 13 x 13. Right image: Synthesis using a pixel pixel-domain CNN with receptive field (40 x 40) smaller than the synthesized image 320 x 320.
global, and it is an open question whether and how these architectures are able to escape the curse of dimensionality. Our local conditional Markov assumptions provide step towards the goal of making explicit the probability model and its factorization into low-dimensional components.
5 DISCUSSION
We have generalized Markov wavelet conditional probability model of image distributions, and developed an explicit implementation using cCNNs to estimate the conditional model scores. The resulting conditional wavelet distributions stationary and Markov sponding to the cCNN receptive fields. The coarse-scale low-pass band is modeled using the score estimated with CNN with global receptive fields. We trained this model dataset of face images, which are non-stationary with large-scale geometric features. We find that the model, even with relatively small eCNN RFs, succeeds in capturing these features, producing high-quality results denoising and super-resolution tasks. We contrast this with local Markov models in the pixel domain, which are not able to capture these features, and are instead limited to stationary ergodic textures.
neighborhoods corre-
The Markov wavelet conditional model demonstrates that probability distributions of images can be factorized as products of conditional distributions that are local. This model provides a mathematical justification which can partly explain the success of coarse-to-fime fine diffusion synthesis (Ho et al., 2020; Guth al., 2022), which also compute conditional scores at each seale. Although we set out to understand how factorization of density models could allow them to avoid the curse of dimensionality in training,
worth noting that the dimensionality of the conditioning neighborhoods network is still uncomfortably high (4 9 x 9 324). Although this is reduced by factor of roughly 300 relative to the image size (320 x 320 102, 400), and this dimensionality remains constant even if this image size is increased, it is still not sufficient to explain how the conditional score can be trained with realistic amounts of data. In addition, the terminal low-pass CNN operates globally (dimensionality 40 40 1600). Thus, the question of how to further reduce the overall dimensionality of the model remains open.
our
Our experiments were performed cropped and centered face images, which present particular challenge given their obvious non-stationarity. The conditional models are approximately stationary due the fully convolutional structure of the cCNN operations (although this is partially violated by zero-padded boundary handling). As such, the non-stationary capabilities of the full model arise primarily from the terminal low-pass CNN, which uses spatially global RFs. We speculate that for more diverse image datasets (e.g., a large set of natural images), a much larger capacity low-pass CNN will be needed to capture global structures. This is consistent with current deep networks that generate high-quality synthetic images using extremely large networks (Ramesh et al., 2022; Saharia et al., 2022: Rombach et al., 2022). On the other hand, the cCNNs in our model all share the same architecture and local RFs, and may (empirically) be capturing similar local conditional structure at each scale. Forcing these conditional densities to be the same at each scale (through weight sharing of the corresponding cCNNs) would impose scale invariance assumption the overall model. This would further reduce model complexity, and enable synthesis and inference on images of size well beyond that of the training set.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We gratefully acknowledge the support and computing resources of the Flatiron Institute (a research division of the Simons Foundation), NSF NRT HDR Award 1922658 to the Center for Data Science NYU, and grant from the PRAIRIE 3IA Institute of the French ANR 19-P31A-0001 program.
REFERENCES
W Buccigrossi and E P Simoncelli. Image compression via joint statistical characterization in the wavelet domain. IEEE Trans Image Processing, 8(12):168--170) Dec 1999. doi: 10. 109/83. 806616.
P Burt and E H Adelson. The Laplacian pyramid compact image code. IEEE Trans Comm, C0M-31(4):533-5444 Apr 1983.
A Chambolle, R A DeVore, N Lee, and B J Lucier. Nonlinear wavelet image processing: Varia- tional problems. compression. and noise removal through wavelet shrinkage. IEEE Trans Image Processing. 7:319-335, Mar 1998.
T Chen, L Lin, W Zuo, X Luo, and L Zhang. Learning a wavelet-like auto-encoder to accelerate deep neural networks. Proc AAAI Conf Artificial Intelligence, volume 32, 2018.
P Clifford and M Hammersley. Markov fields on finite graphs and lattices. Unpublished Manuscript, 1971.
R Cohen, Y Blau, D Freedman, and E Rivlin. It has potential: Gradient-driven denoisers for convergent solutions to inverse problems, Adv Neural Information Processing Systems (NeurIPS), 34, 2021.
M S Crouse, R D Nowak, and RG Baraniuk. Wavelet-based statistical signal processing using hidden markov models. IEEE Trans Signal Processing, 46(4))886440) 902, 1998. doi: 10. 1011100666854444
L Şendur and W Selesnick. Bivariate shrinkage functions for wavelet-based denoising exploiting interscale dependency. IEEE Trans Signal Processing, 50(11):2744-2756, Nov 2002.
Y Cui and K Wang. Markov random field modeling in the wavelet domain for image denoising. In IEEE Int'l Conf Machine Learning and Cybernetics, volume pp. 5382-5387, 2005.
Daras, M Delbracio, H Talebi, A G Dimakis, and P Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint 2tti::2209005442. Sep 2022.
Dhariwal Nichol. Diffusion models beat GAN on image synthesis. arXiv preprint arXiv:2/05.05233. 2021.
Dobrushin. The description of the random field by its conditional distributions and its regularity conditions. Teoriya Veroyatnostei ee Primeneniya, 13(2):201-229, 1968.
R Gal, D C Hochberg, A Bermano, and D Cohen-Or. Swagan: A style-based wavelet-driven generative model. ACM Transactions on Graphics (TOG), (0)4):1011))) 2021.
S Geman and D Geman. Stochastic relaxation, gibbs distibutions, and bayesian restoration of images. IEEE Trans Pattern Analysis and Machine Intelligence, 6:721-741, Nov 1984.
F Guth, S Coste, V De Bortoli, and S Mallat. Wavelet score-based generative modeling. arXiv preprint urXiv:2208.0500) 2022.
Haar. Zur theorie der orthogonalen funktionensysteme. Math Annal., 69:331-371, 1910.
J Ho. A Jain, and P Abbeel. Denoising diffusion probabilistic models. Adv Neural Information Processing Systems (NeurIPS), 33, 2020.
J Ho, Saharia, Chan, D J Fleet. Norouzi, and T Salimans, Cascaded diffusion models for high fidelity image generation. J Machine Learning Research, 23:47-1, 2022.
10

---

Published as a conference paper at ICLR 2023
Z Kadkhodaie and E P Simoncelli. Solving linear inverse problems using the prior implicit in denoiser. arXiv preprint arXiv:2007.13640, Jul 2020.
Z Kadkhodaie and E P Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. In Adv Neural Information Processing Systems (NeurIPS*2I), volume 34, Dec 2021.
T Karras, T Aila, Laine, and J Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 1eXiv:1710.101100 2018.
Kawar, G Vaksman, and M Elad. Snips: Solving noisy inverse problems stochastically. Adv Neural Information Processing Systems (NeurIPS), 34:21757-21769, 2021.
R Laumont, V De Bortoli, A Almansa, J Delon, A Durmus, and M Pereyra. Bayesian imaging using plug & play priors: when Langevin meets Tweedie. SIAM Journal on Imaging Sciences, 15(2): 701-737, 2022.
S-H Li. Learning non-linear wavelet transformation via normalizing flow. arXiv preprint arXiv:2101.11300 2021.
Z. Liu, Luoo, X Wang, and X Tang. Deep learning face attributes in the wild. In Proc Int'l Conference on Computer Vision (ICCV), Dec 2015.
S Lyu and E P Simoncelli. Modeling multiscale subbands of photographic images with fields of Gaussian scale mixtures. IEEE Trans Pattern Analysis and Machine Intelligence, 31(4):993-706, Apr 2009. doi: 10. 109/1000M.....00..II.
M Malfait and D Roose. Wavelet-based image denoising using a Markov random field a priori model.
IEEE Trans Image Processing, 6(4):549-565, 1997.
Mallat, A wavelet tour of signal processing: The sparse way. Academic Press, 2008
T Marchand, M Ozawa, G Biroli, and S Mallat. Wavelet conditional renormalization group. arXiv preprint NXXi:22207.0994// 2022.
M K Mihçak, I Kozintsev, K Ramchandran, and P Moulin. Low-complexity image denoising based on statistical modeling wavelet coefficients. IEEE Trans Signal Processing, 6(12):00-303, Dec 1999.
K Miyasawa. An empirical Bayes estimator of the mean of normal population. Bull. Inst. Internat. Statist., 38:18 18:181-188, 1961.
S Mohan*, Z Kadkhodaie*, E P Simoncelli, and C Cermandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In Int'l Conf on Learning Representations (ICLR), Addis Ababa, Ethiopia, Apr 2020. URL https: httppp::////xxi/o//// org.g..o.r......o.oooooo.oo abs/ 1906. 05478.
Rupert Paget and Dennis Longstaff. Texture synthesis via noncausal nonparametric multiscale markov random field. IEEE transactions on image processing, 7(6):925-933) 1998.
J Portilla, V Strela, MJ Wainwright, and E P Simoncelli. Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans Image Processing, 12(1 12(11):115 11:18-1-1335, Nov 2003. doi: 10. 10.1109/P...000.811. 818640..
A Ramesh, Dhariwal, A Nichol, Chu, and M Chen. Hierarchical Cext-Conditional Image Generation with CLIP Latents. arXiv e-prints arXiv:2204.06/22 apr 2022.
M Raphan and E P Simoncelli. Learning be Bayesian without supervision. In Adv Neural Information Processing Systems (NIPS*06), volume 19, May 2007. URL http: http::/papers nips. . cc/ / ppaper/3000 learning-to--eer -ba.te----ebbaeiaa---aaaa--a withtttitttttttttio wtt.-attttttttsstsss........
H Robbins. An empirical bayes approach to statistics. In Proc Third Berkeley Symposium on Mathematical Statistics and Probability, volume pp. 157-163. University of CA Press, 1956.

---

Published as a conference paper at ICLR 2023
Rombach, A Blattmann, D Lorenz, P Esser, and B Ommer. High- reshuresolution image synthesis with latent diffusion models. In Proc IEEE/CVF Conf Computer Vision and Pattern Recognition, pp- 10684-10695, 2022.
C Saharia, W Chan, Saxena, L Li, Whang. E Denton, S K S Ghasemipour, B K Ayan, S Mahdavi, R G Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprini arXiv:2205. 1487, 2022.
Sherrington and S Kirkpatrick. Solvable Model of Spin- 35(26):1792-1779)) dec 1975. doi: 10. 1...00/Phy.eeeL.........7
J Sohl-Dickstein, E Weiss, N Maheswaranathan, and S Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc 32nd Int't Conf on Machine Learning (ICML), pp. 2256-2265, 2015.
Y Song and S Ermon. Generative modeling by estimating gradients of the data distribution. Adv Neural Information Processing Systems (NeurIPS), 32, 2019.
Song, Sohl-Dickstein, Kingma, Kumar, S Ermon, and B Poole. Score-based generative modeling through stochastic differential equations. In Int'l Conf on Learning Representutions (ICLR), 2021.
Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-177, 2011. 2011.
MJ Wainwright, E P Simoncelli, and A S Willsky. Random cascades on wavelet trees and their in modeling and analyzing natural imagery. Applied and Computational Harmonic Analysis, I1()): 89-123, Jul 2001. doi: 10.1 00.1117/12.408999
K G Wilson. Renormalization group and critical phenomena. II. Phase-space cell analysis of critical behavior. Physical Review B, 4(9):3184, 1971.
JJ Yu, K G Derpanis, and M A Brubaker. Wavelet flow: Fast training of high resolution normalizing flows. Adv Neural Information Processing Systems (NerulPS), 33:6184-6196 2020.
A PROOF OF THEOREM
To simplify notation, we drop the j subscript. Let I (resp. JJ) denote the set of indices of pixel values of æ (resp. æ). If S is set of indices, we denote x(S) (f(i))essnI- Let G be graph whose nodes are 7 TUJ. For each E T, let N(1) C TUJ be the neighborhood of node i, with g N()), and N (i) (()) (i).
To prove Theorem 1, we need to show that the local Markov property:
Vi I, p()(1) @( (i)), p()(i)
(6)
equivalent to the conditional score heing computable with RFs restricted to neighhorhoods:
Vi I, 8 Olog ) ((E(N.
for some functions ft-
We first prove that eq. (6) implies eq. (7). Let i E T. We have the following factorization of the probability distribution:
p @
w) p
( () ((i) m) p()( x(++)) p(Σ(I ((t)
where we have used eq. (6) in the last step. Then, taking the logarithm and differentiating, only the first term remains:
i log
a log (a(i) ((N())) (N (N)))))
( (())))
0z
Oz
12

---

Published as a conference paper at ICLR 2023
which proves eq. (7).
Reciprocally, we now prove that eq. (7) implies eq. (6). Let E I, and 6(()) = big where dig is the Kronecker delta. We have, by integrating the partial derivative:
{ llog (a tz(()), x))d
logp(# [ x) = log =(1)ō; |
dz(1)
!. (((((N(()) tz(
logp(a: o(1)6; x²)
( (NN+
where we have used eq. (7) in the last step. Note that the first term does not depend on *(t), while the second term only depends on (i)+ and x (t)). This implies that when condition on x( (i)) and ar( N4 (1)), the density factorizes as a term which only involves x(i) and a term which does not involve azi). This further implies conditional independence and thus eq. (6).
B PROOF OF EQ. (5)
Miyasawa's remarkable result (Miyasawa, 1961), sometimes attributed to Tweedie (as communicated by (Robbins, 1956)), is simple to prove (Raphan& & Simoncelli, 2007). The observation distribution, p(y) is obtained by marginalizing æ)y,
ply) = Sllpplyaaee plyle))(ar)d = / g p(#) dx
where the noise distribution g(z) is Gaussian. The gradient of the observation density is then:
1 1 / (x y)o(w x)p(x)dx = दे / (a= ( y)p(y, x dd. v p(v) σ²
Multiplying both sides by o2/p(y) and separating the right side into terms gives:
o2 Vy p(y) xp(xly)ddx: f yp(xly)dx =(v) - y-
p(y)
C TRAINING AND ARCHITECTURE DETAILS
The terminal low- CNN and all cCNNs are "bias-free": we remove all additive constants from convolution and natch-normalizztion operations (i.e., the batch normalization does not subtract the mean) (Mohan" et al., 2020). All networks contain 21 convolutional layers with no subsampling, each consisting of 64 channels. Each layer, except for the first and the last, is followed by a ReLU non-linearity and bias-free batch-normalization. Thus, the transformation is both homogeneous (of order 1) and translation-invariint (apart from handling of boundaries), at each scale. All convolutional kernels in the low- low-pass CNN of size x 3. resulting in 43 x 43 RF size and 665, 856 parameters in total. Convolutional kernels in the cCNNs are adjusted to achieve different RF sizes. For example, 13 13 RF arises from choosing 3 x 3 kernels in every 4th layer and x (i.e., pointwise linear combinations across all channels) for the rest, resulting in a total of 214, 144 parameters. For comparison, we also trained conventional mnon-multi-scale CNNs for denoising. For RF 43 x 43, we used the same architecture as for the coarsest scale band of the multi-scale denoiser: 21 bias-free convolutional layers with no subsampling. To create smaller RFs, followed the same strategy of setting some filter sizes in the intermediate layer to 1 X 1.
Architecture.
Training. For experiments shown in Figure and Figure we use 202, 499 training and 100 test images of resolution 160 x 160 from the CclebA dataset (Liu et al., 2015). For experiments shown in Figure 5, Figure and Figure 6, we use 29, 900 train and 100 test images, drawn from the CelebA HQ dataset (Karras et al., 2018) at 320 x 320 resolution. We follow the training procedure described in (Mohan* et al., 2020), minimizing the mean squared error in denoisingd images corrupted by i..dd Gaussian noise with standard deviations drawn from the range [0. I] (relative to image intensity range [O, 1J). Training is carried out on batches of size 512. Note that all denoisers are universal and blind: they trained handle range of noise, and the noise level is not provided input to the denoiser. These properties are exploited by the sampling algorithm, which can operate without manual specification of the step size schedule Kadkhodaie & Simoncelli (2021).
13

---

Published as a conference paper at ICLR 2023
D WAVELET CONDITIONAL SYNTHESIS ALGORITHM
Sampling from both the CNN and cCNN denoisers achieved using a slightly modified version of the algorithm of ?, as defined in Algorithm This method uses only two hyperparameters, aside from initial and final noise levels, and their settings are more forgiving than those of backward SDE discretization parameters score- based diffusions. A step size parameter, h E [0, 1], controls the trade-off between computational efficiency and visual quality. A stochasticity parameter, 3 E (0, 1], controls the amount of noise injected during the gradient ascent. For the examples in Figure Figure 7 and Figure 6, we chose h = 0.01, do 1, 3 = 0.1 and dx 0.01.
Image synthesis initialized with a terminal low-pass image (either sampled from the associated CNN, or computed from a test image), and successively sampling from the wavelet conditional distributions each scale, as defined in Algorithm
Algorithm 1 Sampling via ascent of the log-likelihood gradient from denoiser residual
Require: denoiser f, step size h, initial noise level on, final noise level doo
1: t =
2: Draw 2o ~N ogld)
3: while 2 Joc do
4:
+
5:
d + f((1-1)- - It-1
D Compute the score from the denoiser residual Compute the current noise level for stopping criterion
6:
o + [dd111//
7:
2²= 22=((1- Bh)2 (1-
8:
Draw 24 I)
9:
Dg < It-1 + hd, + & Perform a partial denoiser step to remove a fraction of the noise
end while
11: return 2t
Algorithm 2 Wavelet Conditional Synthesis
Require: number of scales J, low-pass image æ.,, conditional denoisers (fz) step size h, initial noise level dg. final noise level 0
1: for (J....,
, 1} do
2: 3 - Drw.Sample(fa x,,) 10,00,0000 0x)
Wavelet conditional sampling b Wavelet reconstruction
3: æt-1 < W 2;))
4: end for
5: return 20
E ADDITIONAL SUPER-RESOLUTION EXAMPLES
14

---

Published as a conference paper at ICLR 2023
NG
NGC
NC
PA
MEMA
UND
INDI
Figure 7: Additional super-resolution examples. Columns, in groups of three, from left to right: original images, 320 x 320 pixels. Low-pass images of corresponding 3-stage wavelet decomposition (downsampled to 40 40, expanded to full size for viewing). Coarse-to-fine conditional generation using the multi-scale conditional denoisers, each with RF size 13 x 13.
15

---

Published as a conference paper at ICLR 2023
BEV
PS
PS
Figure Additional super-resolution examples. See caption of Fig.
16


Reviews:
[{"review_id": "dsudOTEU-ue", "reviewer": "Reviewer_CgEx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper generalizes a Markov wavelet conditional probability model developed based on the renormalization group theory. The model is parameterized using conditional CNNs with local receptive fields, which enables stationary Markov properties. The method is extended to handle several face image reconstruction tasks, e.g. image denoising and synthesis. The results show that the model can capture image features with a small receptive field,", "summary_of_the_review": "This is a well-written paper with solid theoretical content. The experiments are insufficient. Given the theoretical nature of the paper, this can be somehow tolerated.\n\nThe authors' responses have well addressed by concerns. I have raised the empirical novelty and significant score and also raised the final recommendation to 8.\n\n", "strength_and_weaknesses": "Strength:\n\n- This work is well-motivated.\n- This paper addresses an important problem in learning prior probability models of images. \n- The paper seems to be solid.\n- Two application scenarios are provided with technical details.\n\nCons:\n- The image reconstruction results can be further improved.\n- There is no comparison with the state-of-the-art methods on the two mentioned tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PRLc3fqlkBe", "reviewer": "Reviewer_Pfsy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a multi-scale conditional probability model to reconstruct and denoise images. Specifically, the scales come from wavelet transform, and they generalize Markov conditional models by parametrizing the conditional gradients with a CNN with a local receptive field. They show how their method performs better than Markov methods conditioned and performed in the pixel domain. They study the effect of the receptive field. They use a previously proposed method to generate new images, i.e., to perform gradient ascent on the log probability of the learned CNNs to draw samples from the learned image distributions. They show applications in denoising, synthesis, and super-resolution.\n", "summary_of_the_review": "The paper proposes a method to generalize multiscale wavelets and markov models to learn conditional distributions of wavelets to recover images. The method needs a thorough experimental study to show its performance against other learning frameworks and to clearly explain how its method differs from prior works and its advantages. Hence, the paper in its current form is not recommended for acceptance.\n\n\n-----------\n\n Given the authors' responses, I have raised my score to 8.", "strength_and_weaknesses": "The paper shows the application of their method in several image tasks (generation, denoising, super-resolution).\n\nThe paper can improve on clarity. It is not clear what their contribution is from a general perspective on the performed tasks (e.g., denoising and super-resolution, generation). Please explain the advantages of your method compared to other general frameworks used for super-resolution, denoising, etc. It is not shown how their framework performed against multiple baselines. There seems to be only one method that they compared with which is the vanilla version of their Markov method. Here are detailed comments.\n\nLack of sufficient baselines:\n- By using local convolutional filters and learning locally, their method alleviates the curse of dimensionality; However, how is their method compared to other popular methods in the literature in terms of performance, computational complexity, runtime, etc? (Perhaps, a quantitative comparison between their method and the cited methods on the last paragraph of page 8 is suggested)\n- It is nice to see (in Figure 3) that their method outperforms a similar approach of Markov probability in the pixel domain. However, more experiments with non-Markov baselines should be included. How is the performance of this method for example compared to generic deep learning (e.g., DnCNN bias-free) for denoising, super-resolution, etc?\n\nThe literature review needs proper citations. Here are some examples in the introduction. Please provide citations to\n- the second line in the intro talks about dimensionality.\n- the fourth line in the intro refers to traditional methods.\n- the last line of the first paragraph in the intro refers to global models.\n\nIt is hard to see the relationship between the conditional CNN and (3). Can the authors explain?\n\nA thorough experimental analysis would be nice to add to the characterization of their method: how the performance changes as the number of wavelets, and the depth of scale change.\n\nIt is not clear if the models are trained on the same noise level compared to the test noise or a range. Please clarify and add such information.\n\nHow is the performance on natural images?\n\nIt is not clear how the right image in Figure 6 is generated. Please elaborate on \"CNN denoiser with TF smaller than the image\".\n\nMinor comments\n\n- Moving Figure 2 to the second page will help to understand the - model.\n\n- Why the first row of Figure 4 is smaller? Please fix.\n\n- Elaborate on the statement after Figure 4: \"... less than three percent of the coefficients\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8-m235esy5", "reviewer": "Reviewer_KhB3", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a local conditional probability model for images in multiscale representations.\nIn this model, the probability of the high-frequency (detail) components is conditioned to the low-frequency (low-resolution) component, and these conditional relations are spatially local. The probability of the final low-resolution residual is modeled globally. \n\nBoth the conditional probabilities and the probability of the low-resolution residual are learnt by CNN denoisers using the relation between the denoiser errors and the gradient of the probability of the signal.\n \nThe local (Markov) structure of the conditional probabilities means that the parameters (receptive fields) of the CNNs are restricted to the local neighborhood and hence the conditional part of the model is relatively low dimensional. This assumes that the conditional relations at each scale are essentially stationary (or translation invariant). \n\nHowever, the receptive fields of the CNN to model the probability of the low-pass residual have to be the same size as the residual in order to capture its non-stationary nature. The requirement of dense connection for the low-pass residual is the main limitation of the proposed model in terms of dimensionality.", "summary_of_the_review": "The paper should be accepted because it builds an explicit image probability model as opposed to implicit (non-accessible) models of autoencoders, GANs, normalizing flows, or diffusion methods. These results are an interesting connection between the efforts of (1) linking the gradient of the PDFs with optimal denoising [Miyasawa61, Raphan&Simoncelli11, Vincent11], and (2) establishing conditional relations between low-frequency and high-frequency aspects in spatial neighborhoods of natural images [Buccigrossi&Simoncelli99, Marchand et al.22]. Experimental results successfully show the good performance of the proposed model. \n\nHowever, some elaboration would be appreciated on: (1) advantages over alternative conditional models (such as PixelCNN) in experiments, (2) acknowledgement of the central role of the low-pass component (not a core of the proposal), (3) possible extensions of the locality concept in space to orientation and scale.\n\nMy current score is 7, but I will be happy to raise to 8 or 9 it if the authors address the suggested points.", "strength_and_weaknesses": "STRENGTHS:\n\nThe paper should be accepted because it builds an explicit image probability model as opposed to implicit (non-accessible) models of autoencoders, GANs, normalizing flows, or diffusion methods. These results are an interesting connection between the efforts of (1) linking the gradient of the PDFs with optimal denoising [Miyasawa61, Raphan&Simoncelli11, Vincent11], and (2) establishing conditional relations between low-frequency and high-frequency aspects in spatial neighborhoods of natural images [Buccigrossi&Simoncelli99, Marchand et al.22].\n \nWEAKNESSES:\n\n* The model reminds me of other recent conditional models such as the PixelCNN [van den Oord et al.16, Salimans et al.17].\nI guess the key difference is the formulation in a multiscale representation instead of in the input spatial domain.\nThe authors already compare with CNN models in the spatial domain. I think it would be interesting for the reader to see some example with more sophisticated conditional models in the spatial domain such as the PixelCNN. Of course, in this comparison the size of the images should be big enough for a couple of scales but small enough for proper training of pixelCNN.\n\n* The authors acknowledge the main limitation of their proposal in the discussion \"the non-stationary capabilities of the full model arise primarily from the terminal low-pass CNN, which uses global receptivie fields\". I think this should be acknowledged from the very beginning (in the abstract and/or intro) because the interesting conditional relations (the core of the proposal) always need \"a seed to grow\" which is modeled in a conventional way.\n\n* The dimensionality of the model is small because the Markov structure is spatially local (the interesting theorem 1).\nCould we get extra reductions in the parameters using locality in the orientation and in the scale in redundant wavelets?\nNote that nonorthonormal wavelets could also be applied since it would only imply the consideration of the determinant |W| in Eq. 1, right?. This would mean the extension to orientation and scale of the locality concept applied in space.\n\nREFERENCES:\n\n* van den Oord et al. 16a, Salimans et al.17 Conditional Image Generation with PixelCNN Decoders NeurIPS 2016 arxiv 1606.05328\n\n* van den Oord et al. 16b Pixel Recurrent Neural Networks ICML 2016 arxiv 1601.06759\n\n* Salimans et al.17 PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications ICLR arxiv 1701.05517\n\nMINOR COMMENTS:\n\n* You could cite Marchand et al. 22 the first time you mention that the conditional can be modeled with conditional Gibbs energies (Eq. 2)\n\n* The first paragraph of section 5 mentiones the super-resolution problem and it does not mention the synthesis problem, but then, the next paragraphs alternate between one problem and the other. The two problems should be mentioned at the begining to avoid confusion.\n\n* The proof of Eq. 5 (appendix B) already was done in Raphan&Simoncelli 11, right?. Why not just cite that? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%