Explicitly Minimizing the Blur Error of Variational Autoencoders
Authors: Gustav Bredell
Kyriakos Flouris
Krishna Chaitanya
Ertunc Erdil
Ender Konukoglu
Venue: ICLR 2023 poster (2023)
Keywords: Variational Autoencoders
Generative Modelling
Blur

Abstract: Variational autoencoders (VAEs) are powerful generative modelling methods, however they suffer from blurry generated samples and reconstructions compared to the images they have been trained on. Significant research effort has been spent to increase the generative capabilities by creating more flexible models but often flexibility comes at the cost of higher complexity and computational cost. Several works have focused on altering the reconstruction term of the evidence lower bound (ELBO), however, often at the expense of losing the mathematical link to maximizing the likelihood of the samples under the modeled distribution. Here we propose a new formulation of the reconstruction term for the VAE that specifically penalizes the generation of blurry images while at the same time still maximizing the ELBO under the modeled distribution.  
We show the potential of the proposed loss on three different data sets, where it outperforms several recently proposed reconstruction losses for VAEs.

Published as a conference paper at ICLR 2023
EXPLICITLY MINIMIZING THE BLUR ERROR OF VARI- ATIONAL AUTOENCODERS
Gustav Bredell, Kyriakos Flouris, Krishna Chaitanya, Ertunc Erdil & Ender Konukoglu Department of Information Technology and Electrical Engineering ETH Zurich
gustav. brede 11 www.svieeeeeeeeeee.ccco
ABSTRACT
Variational autoencoders (VAEs) are powerful generative modelling methods, however they suffer from blurry generated samples and reconstructions compared to the images they have been trained on. Significant research effort has been spent to increase the generative capabilities by creating more flexible models but often flexibility comes at the cost of higher complexity and computational cost. Sev- eral works have focused altering the reconstruction term of the evidence lower bound (ELBO), however, often at the expense of losing the mathematical link to maximizing the likelihood of the samples under the modeled distribution. Here we propose a new formulation of the reconstruction term for the VAE that specifically penalizes the generation of blurry images while at the same time still maximizing the ELBO under the modeled distribution. We show the potential of the proposed loss on three different data sets, where it outperforms several recently proposed reconstruction losses for VAEs.
INTRODUCTION
Generative modelling aims to learn a data distribution Pp from samples, such that new samples can be generated from the learned distribution, ety of tasks ranging from out-o--distributtin detection (Asim et al. (022h) to serving as a prior for reconstruction tasks (Tezcan et al. (2019)). One generative modelling approach of particular interest is Variational Autoencoder (VAE) as introduced by Kingma & Welling (2013). This approach is particularly interesting because yields lower dimensional latent model, which allows generating lower dimensional representations of samples, and for doing that the model parameters are deter- mined through directly maximizing the ELBO of the training samples. The combination of these two points makes VAEs unique compared to other generative modeling approaches (Goodfellow et al. 2014). Rezende & Mohamed (2015), Ho al. (2020), Brehmer Cranmer (2020). Caterini al. 2021)).
Pp. The learned distribution can be used for a vari-
One major drawback of VAEs is that they often produce blurry generative samples even though the images in the training distribution were sharp. This is due to the formulation of the optimiza- tion. Variational autoencoders are optimized by maximizing the evidence lower bound (ELBO) with respect to the network parameters O and 6. The
Ez~qu(a(a)) log log(po (2:22) Dx first term is often referred to as the reconstruction loss and effectively ensures that an observed sample can be mapped to the latent space through the posterior model and reconstructed back from its possible latent representation. The second term is matching the learned approximate posterior distribution, (2(2), with that of a prior distribution p(z). The ELBO can also be formulated as divergence in the augmented space, where the data distribution is
minimizing the Kullback-Leine expanded with the auxiliary variable z, DxL TRLLqD, a (a., 2) (o =)] as described by Kingma et al. 2019). is clear that the model is penalized heavily if samples are likely under QD (ær, 2) but not po(x,2), however not the other way around due to the asymmetry of the Bellbackk-eiblle diver- gence. Therefore, po =) and thus (z will have larger variance than the original data distribu- tion @D (æ, 2) leading to generated samples being more diverse than original data, practically also including blurry images. To alleviate this issue both qa(2) (2|z) and pe(ala (z12) should be flexible enough as has been the objective of other works (Berg et al. 20188) Kingma et al. (2016), Vahdat Kautz
2020

---

Published as a conference paper at ICLR 2023
In addition to the line of work on increasing the flexibility of the posterior (0(2[z), which often come at the cost of being more difficult to optimize and computationally expensive, there is a improving the reconstruction loss formulation. To keep the mathemati- cal link to maximizing po(z) for the observed samples through the ELBO formulation, a distribu- tion po (x]22) has to be assumed for the reconstruction loss. Popular choices for these distributions are Gaussian or Bernoulli as introduced by Kingma & Welling (2013). The Gaussian distribution, po(a))) = (u(2)), 2:) has seen widespread adaption (Castrejon al. (2019) Lee al. (0200) and leads to simplification of the reconstruction loss to the mean squared error (MSE) if Σ is assumed to be identity. Since the latent space 2 has lower dimensionality than z. a perfect re- construction will not be possible and the reconstruction loss will determine which features of x to weigh the most. The MSE places no weight on specific features, all features will be reconstructed with a similar weighting. Since most power in natural images is located in the lower frequency in their spectrum Van der Schaaf & van Hateren [1996), low frequency features, in other words blurry features, will dominate the reconstruction loss. Consequently, reconstruction fidelity in higher fre- quency features will be less important.
line of work focusing
Several previous works have recognized the potential of augmenting the reconstruction loss to ad- dress the blur, but they approached the problem implicitly. They aimed to retrieve more visually pleasing generations. One initial approach by Hou et al. (201)) was replace the pixel-wise loss with a feature based loss that is calculated with a pre-trained convolutional neural network (CNN). However, this loss is limited to the domain of images on which the CNN was pre-trained. It has also been suggested in literature to combine generative adverserial networks with VAEs by replac- ing the reconstruction loss of the VAE with an adversarial loss (Larsen et al. (216)). This approach comes at the expense of architecture changes and with optimization challenges. Another approach by |arron (2019) optimizes the shape of the loss function during training to increase the robust- ness. While their loss formulation has shown improved results, it was not specifically focusing on sharpening VAE generations. By learning the parameters of a Watson perceptual model and using it reconstruction loss, Czolbe et al. (2020) have shown that generative examples of VAEs can be improved. Since the human perception also values sharp examples, the proposed loss improved sharpness of samples, but optimization with the loss is less stable than other methods. Recently, Jiang et al. (202T) tackled the blur problem directly in the frequency domain. They have introduced the focal frequency loss, which applies per frequency weighting for the error term in the frequency domain, where the weighting is dependent on the magnitude of the error at that frequency. The above mentioned methods do not explicitly focus on reducing blur errors, and, except for the work by |Barron (2019), the proposed reconstruction terms lose their mathematical link to maximizing the ELBO, since the distribution pe (=)2)) is not defined.
In this paper we aim to explicitly minimize the blur error of VAEs through the reconstruction term, while the same time still perform ELBO maximization for the observed samples. We derive a loss function that explicitly weights errors that are due to blur more than other errors. In our experiments, we show that the new loss function produces sharper images than other state-of-the art reconstruction loss functions for VAEs three different datasets.
2 BACKGROUND ON BLURRING AND SHARPENING
In this section we will review some background on blurring and image sharpening. The blurring degradation of sharp image, x can be modeled by convolving x, with a blur kernel, k, x k where å denotes the degraded image. Assuming no additive noise and the blurring kernel does not suppress any frequencies, using the convolution theorem, the blurring operation can be inverted in the frequency domain as F F())/((k) = F(x), where F is the Fourier transform. The sharp & can then be obtained by applying the inverse Fourier transform to the ratio.
The assumptions of knowing the blur kernel k that noise was present during the blur generation, and convolving with k not suppressing any frequencies are strong. In the case that is not well known and F(k) has several terms close to zero, division by F(k) could cause extreme values. To provide a more stable inverse operation, one can use the Wiener deconvolution (Wiener (1949))
F*( +( F(t) v F(x),
(1)
2

---

Published as a conference paper at ICLR 2023
where F* is referring to the complex conjugate and C' is constant to offset F(k) values that are close to zero and low signal-noise-rati (SNR). In the presence of noise, C is chosen as 1/ SNR Gonzalez & Woods 2001); Wiener (949)). constant.
3 METHOD
The motivation of our work is to reduce the blur in the generated outputs of variational autoencoders (VAEs). We propose to do this by weighting the reconstruction term focus on errors induced by the blur. In VAEs the evidence lower bound (ELBO) is maximized for the observed samples, that are coming from the real data distribution p(x). Assuming the constraint on the latent space is zero-centered unit Gaussian, this leads to the following optimization problem, |Rezende & Mohamed
2015
max 8,6
log[po(<]=)] (xz(gs(z)x>))
T)),
where x and = correspond to the input image and latent variable, respectively, and and & denote the network parameters. By assuming that likelihood, po(x]2) is a multi-variate Gaussian, the ELBO can be written as:
log log((Zo(z)) + =((0 =o())) (0(=)) + DKL ( 1)),
min 0.0 (=la)
E
(3)
where (2) is the mean of the multi-varte variate Gaussian and Eo(2) 22 is its covariance. Generally, the co- variance matrix Ex(2) is assumed to be the identity times a constant scalar, hence no dependence on 0 nor which has the advantage that the calculation of determinant of Σe(z) is not necessary.
The covariance-matee matrix can also seen as a weighting term of the reconstruction loss compared to the KL-divergence. In the next sections we derive the proposed form for the covariance matrix.
3.1 ERROR INDUCED BY BLUR
To design a weighting map that targets errors that are caused by the blur, those errors need to be formalized. For the sake of notational simplicity, will drop dependencce of igg and Σg for now. Let us assume that the output of the VAE is a blurred version of the input image, i.e., 3g = x * k. Therefore, the squared reconstruction and the blurred version of the input. Using Parseval's 's identity Parseval (1866), in Fourier space this corresponds to
be viewed as the squared distance between the input
[[F(e)]1 (IF(x) F(t) F(A] -()) - 11 F(A)II? (4)
where is the error term We see that the magnitude of the blur error at any frequency is proportional to [Fx))]|1 - F())]. The implication of this can best be understood if we imagine A as a centered Gaussian blur with unit sum. Then, the Fourier transform of & will also be Gaussian with its largest value at the center of the Fourier space, i.c., (u, v) = (0, 0). Moreover, because sums up to 1, the largest value of F(k) will be smaller than Therefore, for low frequencies (u, u) R (0, 0) the value of F(k) would be positive and high, yielding low 1 F(k)]. J. For higher frequencies the value of F(k) would be positive and low, yielding high 1 F(k)]. This looks like the major contribution of blur error is due to higher frequencies. So. minimizing for the squared error, should directly focus on removing the blur. However, images showed that the power of natural images drops with increasing frequency with the power law 1/w" with a z 2 Van der Schaaf & van Hateren (1996). where w denotes an arbitrary convex combination of u and v. Thus, even though the multiplicative factor F(k)] is larger for higher frequencies, because image's power at higher frequencies lower than lower frequencies, their contributions [2 will still be smaller compared to lower frequencies. Hence, reconstruction term will be dominated by the lower frequency errors. As a result, models obtained by minimizing for [e))] will likely tolerate blurry reconstructions and generations.
crucial to note that analyses on the power spectra of natural
3.2 WEIGHING THE ERROR THROUGH WIENER DECONVOLUTION
Given the distribution of the blur induced errors, the next question to answer how to best weigh the error term emphasize blur errors due the convolution with a k. If k is known, one way of
3

---

Published as a conference paper at ICLR 2023
weighting the errors would be with the inverse value of F(k). For small blur errors with F(kk)
the weighting would be 1 and does not change the error term. However, as [F(k) gets smaller the error weighting will increase. With any weight matrix in the Frequency domain, one can compute weighted reconstruction error as
JW. (F(x)- - F(a)))
(5)
where W is a chosen weight matrix. like to chose W F(R). such that
To specifically weigh the blur induced errors higher, we would
F(k)
As mentioned before, this naive deconvolution of the reconstruction term, would lead to instabilities for frequencies where F(k) O. To stabilize the chosen weighting method, we follow the Wiener deconvolution and add a constant C' as
(k) F(A)+ (F(x) F(Eo))
Linking this to the VAE, a reconstruction loss in the Fourier transform corresponds to a Gaussian likelihood p=(2))) with E-1 WTW, where W corresponds the matrix that encodes the convo- lution operation defined by the inverse Fourier transform of W, i.e., (W)). When W is defined in terms of the kernel & in Equation 7 then we denote this dependence with a subscript, i.e.,
b W W& with WE being the matrix corresponding to convolution with F- ()(5))))) As
result, using the Parseval's 's identity the log po (ar) term needed for the computation and optimiza- tion of the BL can written as
D log po (x12) log(2T) log |Ex|
1 1
F (k) [(())² + C (F(x) (F(x) F(EO))
(8)
2
2
3.3 DETERMINING THE BLUR KERNEL
To determine the weight matrix at given iteration during optimization, we need blur kernel k. To determine & we utilize the assumption of image degradation via a convolution with a blur kernel as in Section 2 Specifically, we optimize to find the kernel & that minimizes the means squared error when the input & is blurred with kernel l with the output of the VAE, do.
min | * k - 20
is reasonable to assume that some images might blurred more than others by the VAE, and we confirm this in Section 4.3) and it changes with the parameters of the networks. Therefore, k would have to be determined through the optimization after every iteration and for every image. To this end, utilize neural network, G, that takes input sample from ((2[z) and outputs kernel k. The optimization can consequently be written as
min x G.(()) 20
(10)
where 1: = (2) and NO, T)-
3.4 DETERMINANT COVARIANCE MATRIX
The non-trivial choice of Σ for this method, implies that the determinant [Σ] needs to be accounted for in the likelihood calculation, Equation g Full computations of determinants are costly, never- theless, matrices defined by convolution operations provide special opportunities. The convolution operation can be implemented as a matrix multiplication with an appropriate circulant matrix K as vec(k: z) K vec(2:), where vec(-) is the vectorization operation. If the kernel k is two dimen- sional this will result in a block-circulant matrix with each block being a circulant matrix as well. matrices have the property that their determinant can be calculated analytically as shown by Davis (1979). In addition, the circulant matrices will be sparse if the size of l is much smaller than that of x, which enables efficient computation of the determinant.
Circulant and block-cireulas
4

---

Published as a conference paper at ICLR 2023
In our case Σ-1 is equal to W Wk. where Wa represents the block- matrix that corresponds to the Wiener deconvolution constructed with the kernel k. we assume the constant C' is zero and & does not suppress any frequencies for any 2, then the deconvolution would be the inverse operation of the convolution, i.e., We = =1/ (k), and we can write FF (A))F = 2; Vr. This implies We vec(k .) vec( Vx and W K vec(#) vee(#), Vo with K being the block circulant matrix that corresponds to the convolution with k. Therefore, We = K- and 2x - Ww)) - K K Hence, the determinant of the covariance can be computed in terms of the determinant of K as E& 2| Using the properties of block-cireulant matrices, this determinant can be easily computed. However, any blurring & will likely suppress some frequencies and therefore for optimization purposes C' has to be set to small positive constant to avoid division
by zero. The inverse of W6 then the block-circulant matrix that corresponds to [FRR)
[R)
F(k) F-(K) C/F* can be explicitly computed. Instead, we approximate the inverse of W as K(()= = + ell, with K still being the block-cireulant matrix corresponding to k. For that suppresses large portion of the Fourier space, this approximation will behave similar to Wiener deconvolution in the sense that it will not allow suppression of frequencies completely since the Fourier transform of the kernel that corresponds ((c) will never be zero.
FIR
FF(M
W KK'
XK
F[BI +G
F(c)
D6,
(slols)
We Me po(de)
F(8.
10 t) (M/F =(Ao)]l Dxslsa(4e)][
unin |a (.(()) 2o1²
Figure I: Sketch of the proposed approach for minimizing the blur error in VAEs. We illustrate the two losses that we minimize in the alternating optimization and the corresponding network parame- ters that are updated different colors.
3.5 VAE OPTIMIZATION
In the previous section the dependency of Σ on 2 has been omitted for notational simplicity. We showed the inverse of W₁ can be approximated with K + that would allow an estimate of the determinant of Σ. From Equation TO we also saw that the kernel A that constructs K is dependent on z and is modeled with a neural network with the parameters (z). Thus, the determinant will be both influenced by z and c. If we assume to be small, then the determinant will be mainly dependent on z. However, we assume to be large, then the identity will dominate the determinant calculation and the influence of on the determinant will negligible. Applying the reconstruction term derived in section 3.2 and estimating the determinant of through K(G, (2), e) leads to the following optimization equations:
[=) [ -C (F(x) =(to)) Dxrloo(=)<)) T)),
max logllll/(/KC (K), 8,0
(11)
min * Gz(z) 20
(12)
where alternating optimization is used. We note that after each iteration of Equation 12 the model for which we optimize the ELBO in Equation IT changes. The update of the model is incremental though due to the alternating optimization and practice no convergence issues were observed. We illustrate sketch of the proposed approach in Figure
4 EXPERIMENTS
To validate the contribution of the proposed approach we perform experiments on two natural image datasets and one medical image dataset. We first provide dataset and experimental implemen-
5

---

Published as a conference paper at ICLR 2023
tation description followed by ablation study to validate the assumptions made in the method section. Finally, we provide quantative and qualitative results for our approach compared to other reconstruction terms used in literature.
4.1 DATASETS
To evaluate the potential of the proposed approach on natural images we make use of the popular CelebA dataset as provided by [iu et al. (2015) and Lee al.) (2020) for low and high-resolution images, respectively. For the low-resolution images we center-crop the images to 64 x 64 and for the high-resolution we use 256 x 256. The low-resolution and high-resolution datasets consists of 200'000 and 30'000 images, respectively. Furthermore, to investigate the performance of the proposed method beyond natural images we utilize the Human Connectome Project (HCP) dataset Van Essen al. )2013) and use the same preprocessing as Volokitin et al. (2020). Of the 960 volumes with a 256 x 256 x 256 we utilize 400 volumes, from which we randomly crop patches of size 64 64. For all the datasets use 80% for the training set and 20% for the test set.
4.2 EXPERIMENTAL SETTINGS
The code is written in Python and PyTorch (Paszke et al. (2019) is used as library for the deep learning models. The architecture of the VAE consisted of N convolutional encoding layers and decoding layers, that is connected through a multilayer perceptron which does the final compression and initial of scaling up of the latent variable with the size of 256. Details on the architecture can be found in the code that will be publicly available upon acceptance. Furthermore, the Adam optimizer used with a learning rate of 11-1, For the low- and high-resolution CelebA the number of training epochs were, 100 and 200, respectively. For the HCP dataset the models were trained for 400 epochs.
As metrics we utilized the peak-signal to noise ratio (PSNR), structural similarity meassure index (SSIM), learned perceptual image patch similarity (LPIPS, Zhang et al. (2018) and inception dis- tance (FID, Heusel et al. (2017) for comparing the reconstruction of the VAES with the respective input. To quantitatively evaluate the generative examples examples.
the FID score and provide visual
In section state that assume the current reconstruction of the VAE to be blurred version of the output, dig = x k. This assumption does not hold in the first few epochs of training. Thus, we set I-1 to II for the first ten and twenty epochs for the CelebA and HCP datasets, respectively. After this initial period E-1 becomes a function of (z). To avoid G. (z) initially producing inaccurate estimates of the blur kernel we already start training G. (2) even when Σ- is still set to I. This allows accurate blur kernel prediction when --1 becomes a function of G. (2).
4.3 ABLATION STUDY
To obtain a better understanding of the influence of the different parameters of the proposed approach we ran an evaluation on the low-resolution CelebA dataset as shown in Table [[] We focused on two aspects of the current approach. Firstly, investigated how the choice of Σ influences the performance, by comparing fixed Σ that is not dependent on the latent variable 2 with that of a y that is function of z but with varying size e. We modeled the Σ that not function of as a Gaussian kernel with a with a fixed radius. This corresponds to a fixed weighting term in the reconstruction loss and constant determinant, which can thus be omitted from the optimization. We found that this approach outperformed the configuration where Σ is a function of 2 with a small During optimization the latent space is adjusted minimize the log/E| and even though this normally cause E-1 to increase the error term significantly, the constant C' prevents this term from very large values. The best performance was achieved by assuming e to be large. large e would dominate log) | and the influence of the value of (z) on the determinant would be minimal, allowing us to omit optimizing over the determinant by assuming it is roughly constant. The second parameter we examined was the constant C in the Wiener Filter by changing value for the scenario where is a function of z and e is large. We observe the choice of C to produce robust results for small values 0.025. However, performance degrades with larger C' values.
In addition to the ablation study, we visualized the evolution of the kernel estimated by G, during the course of training for four example images, see Figure As explained in section 4.2
Σ

---

Published as a conference paper at ICLR 2023
Parameter
Value
PSNR
SSIM
LPIPS
FID Recon
FIDGen
((K
(ON 2),c) (O e).c)
22.37
0.6306
0.1217
0.0616
0.0753
22.56
0.646
0.1097
0.0615
0.0823
(2),(2), Camalt)
19.51
0.4463
0.1924
0.171
0.214
K(G,,z) (z). Elarge)
23.21
0.7296
0.1254
0.0364
0,0536
0.005
23.21
0.7294
0.1186
0.0340
0.0556
0.025
23.21
0.7296
0.1254
0.0364
0,0536
0.1
22.87
0.7127
0.1423
0.0454
0.0618
Table I: We investigate the performance differences for different modelling choices the blur kernel estimation, (2) versus constant (0, var). In addition, look at the influence of the optimization. We observe a kernel generator that is dependent on 2, (z),, to perform the best if assumed be large. Under this condition we also examine the influence of the constant C in the Wiener Filter.
Epoch 10
Epoch 11
Epoch 15
Epoch 100
Epoch 10
Epoch
Epoch 15
Epoch 100
The evolution of the estimated kernel, (z). is shown four different different images at different epochs during training. The blur minimizing reconstruction term introduced at epoch 10, after which strong decrease of the estimated blur kernel can be observed along with sharper image reconstruction. In addition, can be seen that different images have different blur kernel estimates, which motivates determining the blur kernel per image and making Σ dependent on 2.
Figure 2:
is initially modeled for ten epochs by an normal multivariate Gaussian until the assumption holds that the output can be seen as a blurred version of the input. can be seen that the estimated blur kernel is large for all four images and in the subsequent epochs (11 and 15) drastically decreases the blur continues end of training at iteration 100, where it be observed that the blur in the image well as the corresponding estimated blur kernel is much smaller than the start. We note that a residual blur remains due the inherent reconstruction limitations of VAE such the information bottleneck by constructing a lower dimensional latent space. Another observation that the estimated blur kernel different for each of the four images. This provides some empirical evidence that it is good idea to model Y as function of z to capture this dependency. This also provides an explanation why (z) outperformed fixed Σ as seen in the ablation study.
as we are specifically penalizing blurry outputs. This minimization
4.4 QUANTITATIVE AND QUALITATIVE RESULTS
We compare our proposed method with several reconstruction terms that are often used in literature on three different datasets. For the learned perceptual losses, namely the Watson perceptual loss (Czolbe et al. (2020)) and VGG-Loss (Zhang et al. (2018)) we use the implementation provided by Czolbe et al. 2020), where the losses were learned for with the Two-Alternative Forced Choice
7

---

Published as a conference paper at ICLR 2023
Ours
L2
Input
Figure 3: Here, we present the qualitative results of reconstructions and generations from the pro- posed method and relevant compared methods CelebA256 dataset. We observe lower blurriness and higher sharpness for the reconstructed images from the proposed method.
Dataset
Method
PSNR
SSIM
LPIPS
FID Recon
FIDGen
CelebA
L2
12.
0.6944+6.99 2
0.1656+6.4e
0.0571
0.0656
L1
22.68:11.9
0.706917.1e
0.176:16.4:
0.0671
0.0725
SSIM
20.7911.88
0.7248116.5e -2
0.169117.22
0.0487
0,0592
CE
22.95:1..7
0.718316.96-2
0.148=16.4e*
0.045
0.0575
FFL
23.02-+166
0.7141+66.5= -2
0.1552+6.1e
0.056
0.0642
Ours
23.21++166
0.7296
0.125415.8.
0.0364
0.0536
VGG
17.64:11.7
0.4439+8.9c
0.0934
0.0282
0.0363
Watson
19.22:12.1
0.580811.1e
0.116415.5e
0.0356
0.0459
CelebA-HQ
L2 19.37
0.563247.9c
0,5372
0.1
0.1138
L1
1..49:11.
0.602717.9c
0.513248.2e
0.1046
0.1038
SSIM
17.99
0.595348.16
0.561#8.0e -2
0.1065
0,0982
CE
1..
0.5958+7.88
0.4922+7.9e
0.0909
0.1053
FFL
10.15.11.4
0.577917.76
0.5179.1..6e
0.1068
0.1124
Ours 20.17:1..5
0.586717.8c
0,487418.00+
0.0858
0.0938
VGG 11.61
0.3462-1..5e
0.4943.1..54
0.1237
0.1231
Watson
12.68:4..6
0.486947.5e
0.5601 #8.0e
0.1295
0.1381
Table 2: We provide quantative results (average++st)) on the low (64 x 64) and high quality (256 256) CelebA dataset. LI, L2 and CE refer the first norm, second norm and cross-entropy, respectively, as losses. Learned reconstruction losses are separated with a line. We observe our method to perform well across all metrics for both datasets.
dataset (Zhang et al. 2018) ) For the focal frequency loss (FFL) we used the code the authors provided (Jiang al. 202T)). )) For all the losses we use the same network architecture and training parameters except for the Watson loss on CelebA, where we used the authors model, which has factor of three more parameters than the model used for the other methods, due instability during training with our model architecture. We tuned the weighting between the two terms in the ELBO for all methods (Higgins al.) (2016)) and reported results where the FID score for the generative samples were minimized to ensure optimal visual performance. In Table 2 we present the results for the CelebA dataset for both low (64 x 64) and high resolution (256 x 256) images. It can be seen that approach performs well across all metrics. For some metrics such the PSNR SSIM of the reconstruction, we get slightly smaller values, but achieve higher scores on the perceptual metrics, including generation. The perceptual losses perform very well the perceptual metrics
8

---

Published as a conference paper at ICLR 2023
Dataset
Method
PSNR
SSIM
Table 3: We provide quantative results
(average:tstd) on the HCP dataset, where we trained the VAE on random crops
HCP
L2
18.0915.7
0.431311.7e"
L1
18.95:1600
0.5005:1..e"
with resolution of 64 64. It can be
SSIM
18,061558
0.5922:1..1
seen that our proposed method can easily
Cross-Entropy
18.69+6.00
0.470
be applied to non-natural image, while
FFI. 18.88+
0.4795+1.8e
outperforming several established recon-
Ours 19.54=6.1
0.520911.96
struction losses.
for the CelebA dataset, however these metrics are often high despite obvious artifacts. In addition, the perceptual losses did not allow for stable optmization for the high resolution CelebA images, where our loss was able to scale up without any modifications, except increasing the kemnel size from 11 to 41 x numerical results, we provide examples of the reconstruction and generated samples for the high quality CelebA in Figure B and a larger Figure 7
account for larger possible blur kernel estimates. addition to the
The advantage of our proposed method is that it is not constrained to the natural image domain and does not need to be re-trained for each domain. We show the potential using our approach the medical image analysis field by training VAE MRI brain slices. Since the perceptual metrics can not be applied here, we only report the MSE and SSIM when compared against other methods in Table 3 It can be seen that approach works well in other domains and outperforms several other reconstruction terms without needing to modify our approach. We also provide some qualitative examples in Figure and more examples in Figure 9 in the Appendix.
L2
FFL
Figure 4: Here, we present the qualitative results of reconstructions and generations from the pro- posed method and relevant compared methods HCP medical dataset. We observe lower blurriness and higher sharpness for the reconstructed images from the proposed method.
CONCLUSIONS
In this paper, we have proposed reconstruction term. which minimizes the blur enror of VAEs, and is simultaneously consistent with the ELBO maximization. Deblurring is achieved by explic- itly shifting the weight importance of the blur errors. The method implies that the likelihood is multi-variate Gaussian, where the covariance is not assumed to be trivially the identity. The re- sulting calculation the determinant efficiency and we also show that it can be omitted under some assumptions leading to strong per- formance. Therefore, despite the additional computational costs, the proposed method retains the desired celerity of VAEs. We perform extensive experimentation three different datasets, that showcase sharper images as produced by the new loss function in comparison to other state-of-the art reconstruction losses.
the specific covariance can be computed with remarkable

---

Published as a conference paper at ICLR 2023
REFERENCES
Muhammad Asim, Fahad Shamshad, and Ali Ahmed. Blind image deconvolution using deep gen- erative priors. IEEE Transactions on Computaaionall Imaging, 1:1493-15000 1506, 2020.
Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4331-4339, 2019.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester nor- malizing flows for variational inference. arXiv preprint arXiv:1803.05649 2018.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density es- timation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 442-453, Curran Asso- ciates, Inc., 2020. URL https::/pproccediggss.. neurips Ctt.ps.cc/papp wttppppppee//////1eeeeeeee (1112223106eeca000ee6600000000-0
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrnns for video pre- diction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 7608-7617, 2019.
Anthony L Caterini, Gabriel Loaiza-Ganem. Geoff Pleiss, and John P Cunningham. Rectangu- lar flows for manifold learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp. neurips. 00228-30241. Curran Associates, Inc., 2021. URL httpp:: Papee800d-Pape.pppoom
Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian Igel. A loss function for generative neural networks based on watson's 's perceptual model, Advances in Neural Information Processing Systems, 33:2051-2066 2020.
Philip Davis. Circulant matrices. John Wiley&Sons, New York, 1979.
Rafacl C Gonzalez and Richard E Woods. Digital image processing, 2001.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680. 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basie visual concepts with constrained variational framework. 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Xianxu Hou, Linlin Shen, Ke Sun, and Guoping Qiu. Deep feature consistent variational autoen- coder. 2017 IEEE winter conference on applications of computer vision (WACV), pp. 1133- 1141. IEEE, 2017.
Liming Jiang, Bo Dai. Wayne Wu, and Chen Change Loy. Focal frequency loss for image recon- struction and synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13919-13929, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 2013.
arXiv preprint
Diederik Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trendy in Machine Learning, 12(4)3377-32,, 2019. 2019.
10

---

Published as a conference paper at ICLR 2023
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther, Autoen- coding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 1558-1566. PMLR, 2016.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances Neural Information Pro- cessing Systems, 33:741-752, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Mare-Antoine Parseval. Mémoire les séries sur l'intégration complète d' l'un équation aux différences partielles linéaires du second ordre, à coefficients constants. Mém. prés. par divers savants, Acad. des Sciences, Paris, 1:638-648, 1806.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Kerem C. Tezcan, Christian F. Baumgartner, Roger Luechinger, Klaas P. Pruessmann, and Ender Konukoglu. Mr image reconstruction using deep density priors. IEEE Transactions on Medical Imaging, 38(7):1633-164) 2019. doi: 10.1 (009/TM..018.28877700
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. Advances Neural Information Processing Systems,
2020.
A Van der Schaaf and JH statistics and information. Vision research, 36(17):275-22770, 1996.
Hateren. Modelling the power spectra of natural images:
David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage, 80:62-79, 2013.
Anna Volokitin, Ertune Erdil, Neerav Karani, Kerem Can Tezcan, Xiaoran Chen, Luc Van Gool, and Ender Konukoglu. Modelling the distribution of 3d brain mri using 2d slice vac. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pp- 657-666. Springer, 2020.
Norbert Wiener. Extrapolation, interpolation, and smoothing of stationary time series: with engi- neering applications, volume 113. MIT press Cambridge. MA. 1949.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
11

---

Published as a conference paper at ICLR 2023
A ARCHITECTURE DETAILS
In our main paper we worked with one single architecture to keep settings consistent across multiple different datasets. The encoder of the VAE, (6 consists of four and six downscaling blocks for 64 x 64 and 256 x 256 input, respectively. The downscaling blocks consists of 2D convolution (kernel size = 3, stride = 2) followed with batch norm (Zhang al. (2018) and Leaky- Relu. The output of the downscaling blocks is then passed through two-layer multipectron to obtain the j and Ø. The sampled latent variable is then used as input for the decoder, (=(2))), which first passes the through a two linear layers and then through four decoding blocks consisting of a 2D transposed convolution (kernel size 4, stride = 2), batch-norm and Leaky-Relu. Afterwards the output is passed through one more 2D convolution (kernel size = 3. stride 1) and tanh activation function to obtain the final output. The number of feature layers are [64, 128, 256, 512] for each encoding block, respectively. For the decoding block, the list of feature layer numbers is reversed.
To generate the kernel & from the latent variable 2 we trained a neural network, G, (2). I The network consists of two linear layers where the number of hidden nodes is 1000, the input is and the output is the size of the desired kernel k.
To ensure that experiments not fine-tuned single model configuration, also evaluated our method with a second architecture on the CelebA64 dataset. The results can be seen in Table We observe that the same trends holds in this evaluation as in Table 2 This second architecture, (=]=2, consists of five encoding blocks (2D convolution (kernel size = 3, stride = 2), batch norm, Leaky-Relun and one linear layer that outputs μ and σ.. The decoder consisted of one linear layer followed by five decoding blocks (bi-linear upsampling, 2D convolution (kernel size = 3, stride = 1). batch norm and Leaky-Relu. Afterwards the output is passed through a 2D convolution (kernel size = 3, stride = 1) and tanh activation function to obtain the final output. The number of feature layers are [32. 64. 128, 256, 512] for each encoding block, respectively. For the decoding block, the list of feature layer numbers is reversed.
Dataset
Method
PSNR
SSIM
LPIPS
FID Recon
FIDGen
CelebA
I.2
21.80+1..7
0.6609
0.1895+7.77
0.0696
0.0774
LI
21.87:119
0.664518.0c
0.1876:17.5e
0.0714
0.0742
SSIM
19.91=119
0.6725-8.1c
0.2026118.1e
0.0747
0,0721
FFL
22.28:1.7
0.6795+47.6
0.1803:47.1e*
0.0638
0.0772
Ours
22.
0.6951
0.1515:16.8e
0.0448
0.0563
VGG
17.10+1.9
0.4837+1.06
0.0971+-4.5
0.0192
0.0330
Table We provide quantative results on the low (64 x 64) CelebA dataset with smaller architec- ture than in the main paper to show that our results can are not sensitive to architecture design. L1 and 1.2 refer to the first and second norm as losses. We observe our method to perform well across all metrics.
B ADDITIONAL RESULTS
In this section we provide additional results that are complementary the results in the main paper. Firstly, we ran out method and several baselines on the CIFARIO dataset (Krizhevsky et al. (2009) and report quantitative results in Table [] It can be seen that our method performs the best across all metrics except for the FID of the generated samples. Although other methods have lower FID score for the generated samples, this is not reflected in the visual examples provided in Figure 5 In fact, our approach seems to be better able to reconstruct details in the generative samples. This observation also holds for the reconstruction results and is supported in the strong perceptual metric results, such LPIPS, compared to other methods. Secondly, we provide comparison of the ELBO value for our method with that of an L2-llss on the CelebA64 dataset. The results can be seen in Table We can see that our ELBO value is lower than that of when assuming Gaussian with a fixed co-variance matrix. To obtain the ELBO results for our method we assumed an of 0.5. Lastly, we also extend the qualitative results of the main paper to all datasets and baselines as can be seen Figures 7 g and 9 for the CelebA256, CelebA64 and HCP datasets, respectively.
varii-e
12

---

Published as a conference paper at ICLR 2023
Dataset
Method
PSNR
SSIM
LPIPS
FIDRocon
FIDGen
CIFARIO
L2
20.43+1.8
0.5757 1.0e 0.
0.2163=6.6e -2
0.0595
0.0833
FFL
1..12.11.8
0.220811.77e"
0.0568
0.0707
CE
20.17:11.8
0.5560.1..1e
0.2275
0.0490
0.0618
Ours
20.7:11.6
0.6382+99.6
0.1298=4.96
0.0465
0.0829
Table 5:
We provide quantative results (average.istd) the CIFARIO dataset Krizhevsky et al. We observe our method to perform well across all metrics.
2009 )
Reconstructions
Generations
Ours
L2
FFL
CE
Input
Figure 5: Here, we present the qualitative results of reconstructions and generations from the pro- posed method and relevant compared methods CIFAR10 dataset (Krizhevsky et al. (2009). ) We observe lower blurriness and higher sharpness for the reconstructed images from the proposed method.
L2
FFL
Ours
Data
1.40
1.23
1,40
1.34
1.27
1.05
Figure 6; Here, we present the absolute value of the Fourier transform in log scale for the reconstruc- tions and samples in the top and bottom row, respectively. To obtain the shown images we average the values over 128 individual images from the CelebA64 dataset. To the right the corresponding image for the data distribution is shown. In addition, report the mean difference between each shown image with respect to the data image, Both qualitatively and quantitatively our approach can better replicate the frequency space of the data distribution.
13

---

Published as a conference paper at ICLR 2023
Dataset
Method
PSNR
SSIM
L.IPS
FeeRe
FIDGen
-log P
CelebA
L2
2..5::1.6
0.69416.9c
0.166=16.4e -
0.057
0.066
<632
Ours
23.2:::.6
0.730
0.125=:.8e
0.036
0.054
<-3062
Table 6: We provide quantitative results on CelebA64 dataset including the ELBO results for single image samples. For our method we assumed to be 0.5
Reconstructions
Generations
Ours
L2
FFL
CE
L1
Watson
SSIM
VGG
Input
Figure 7: Here, we present the qualitative results of reconstructions and generations from the pro- posed method and relevant compared methods on CelebA256 dataset. We observe lower blurriness and higher sharpness for the reconstructed images from the proposed method.
14

---

Published as a conference paper at ICLR 2023
Reconstructions
Generations
Ours
L2
FFL
CE
L1
Watson
SSIM
VGG
Input
Figure 8: Here, we present the qualitative results reconstructions and generations from the pro- posed method and relevant compared methods CelebA64 dataset. We observe lower blurriness and higher sharpness for the reconstructed images from the proposed method.
15

---

Published as a conference paper at ICLR 2023
Reconstructions
Generations
Ours
L2
FFL
L1
SSIM
Input
Figure 9: Here, we present the qualitative results reconstructions and generations from the pro- posed method and relevant compared methods HCP medical dataset. We observe lower blurriness and higher sharpness for the reconstructed images from the proposed method.
16


Reviews:
[{"review_id": "bCxBD054X5", "reviewer": "Reviewer_QwSD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new reconstruction term that minimize the blurriness of the VAE while still maintaining the mathematical connection to the original ELBO objective. The paper provides an ablation study of several hyperparameters and tested on dataset such as CELEBA and HCP medical dataset while comparing to some of the other recently proposed reconstruction term. ", "summary_of_the_review": "I think the paper is interesting and the proposed algorithm is novel but motivation for the new reconstruction term is weak because its improvement is hard to determine. ", "strength_and_weaknesses": "Strength:\n1. The paper provide a detailed ablation study on the choice of Sigma and C.\n2. The proposed method seems quite novel and interesting. \n\nWeakness:\n1. The paper only compares with other algorithms that have alternative reconstruction term, but did not provide comparison with other SOTA VAE variations. A simple example is the beta VAE which weighs the KL divergence. In general, we could either weighs the reconstruction term or the KL term of the VAE. I think for the very least, comparison with other KL divergence weighting algorithm would be meaningful. \n2. in terms of experiment results, it looks like for CELEBA 256 by 256 dataset, the proposed reconstruction term doesn't seem to to be better than the cross entropy reconstruction in a few metrics. Although for the medical dataset, there seems to be improvements. The significance of the improvements are hard to determine because no standard deviations are provided. The reconstruction images of the medical dataset from L2 and proposed method seem quite similar visually. The overall the benefits of the algorithm are not very convincing. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3-uB9vhp_Q", "reviewer": "Reviewer_q8AD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for reducing blur in VAEs by explicitly incorporating a de-blurring convolutional operation, which enters the covariance of the conditional likelihood. This allows the method to upweight errors at higher frequencies. In the proposed approach, the conditional likelihood is evaluated in frequency-space, with the covariance determinant calculated analytically. The de-blurring kernel is calculated per-image as a function of the latent variables. Notably, the entire approach still optimizes the ELBO, unlike other heuristic methods. Experiments are performed on CelebA (high resolution and low resolution versions) and a medical image dataset. The proposed method outperforms similar approaches in terms of various metrics, although the reconstructions and generations still seem somewhat blurry.", "summary_of_the_review": "The paper takes a principled idea, de-blurring, and shows how to incorporate this into VAEs while retaining a proper probabilistic model. The proposed method is evaluated on multiple datasets, comparing against relevant baselines. For these reasons, I feel that this paper warrants publication.", "strength_and_weaknesses": "**Strengths**\n\n- **Principled, well-formulated approach that connects two sub-fields to tackle a known problem.** The authors rigorously demonstrate how to incorporate learned, de-blurring convolutional kernels into the conditional likelihood of VAEs, such that the probability density is still valid. While various previous works have proposed heuristics for improving the blurring in VAEs, even looking at spatial frequencies, to the best of my knowledge, none of these previous works have incorporated these aspects in a rigorous probabilistic modeling framework. The end result is a clear demonstration of how to connect ideas from low-level image processing to VAEs, which then be built upon for further in future work. Further, this approach attempts to directly address a known issue with VAEs (blurring) using a principled approach: adding spatial dependencies to the conditional likelihood.\n\n- **Fairly clear presentation.** For the most part, the authors do a great job of explaining the motivation and mechanics behind their approach. They start by discussing why it is that high-frequency components tend to be neglected in VAEs and other maximum-likelihood generative models. They then present basic concepts from image processing, i.e., de-blurring. They then walk through how to incorporate this operation into the covariance of the conditional likelihood in VAEs, covering the key components and hyperparameters. I especially appreciated the diagram outlining the approach, which was quite clear. While I still have several questions about the formulation, the paper’s presentation is in good shape overall.\n\n- **Reasonable empirical evaluation.** The empirical evaluation appears to be sufficient to demonstrate meaningful benefits over previous similar methods on several representative datasets. The authors evaluate standard metrics to quantify their performance improvements. The provided ablation evaluation is also helpful to assess the dependence on various hyperparameters.\n\n\n**Weaknesses**\n\n- **Could be scaled up further.** The authors demonstrate their method in the setting of single-level convolutional —> fully-connected —> convolutional VAEs. Most competitive VAEs utilize multiple levels of latent variables, e.g., VDVAE (Child, 2021). In principle, this method could be combined with such higher capacity models to possibly provide even further improvement. It would be useful to assess whether this de-blurring technique is still useful in this higher capacity setting. Likewise, in the natural image setting, the authors only demonstrate results on CelebA (although they use two versions of the dataset). I would have expected results on other benchmark datasets, particularly CIFAR10. Generating faces using datasets that have fairly normalized poses feels like perhaps too specific of a setting to make convincing arguments about modeling sharper edges in natural images.\n\n- **Some aspects a bit unclear.** While the method itself appears to be well-formulated, some aspects were a bit unclear to me. For instance, why do we need a second objective to learn the convolutional filter? Shouldn’t this already be rolled into the maximum likelihood objective? Is this primarily for optimization purposes? What dictates the form of this objective — the form of the conditional likelihood? Could we have multiple stages of de-blurring? How does this compare with, for example, using an autoregressive or flow-based conditional likelihood? These alternative forms of densities are also capable of capturing low-level pixel dependencies while still maintaining a valid log-likelihood.\n\n- **Seems somewhat specific to Gaussian conditional likelihoods.** The authors present their approach purely using Gaussian conditional likelihoods. While this is typically a reasonable assumption, this also seems to limit the scope of the approach. For instance, many higher-capacity VAEs utilize discretized mixture of logistic distributions (VDVAE; Child, 2021). While I think it may be possible to apply this technique in these other settings, this is left for future work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0QdJm0Slj9l", "reviewer": "Reviewer_Hcu2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The composition of the encoder and decoder mapping of standard Gaussian variational autoencoders (VAE) is known to \"explain away\" high frequency image components as independent Gaussian noise. The authors propose to address the resulting blur in reconstructed and generated images directly, by adding  a blur kernel to the model parameters, which depends on the latent code. They propose a coordinate block descent approach for solving the optimisation w.r.t. the standard VAE parameters and the parameters of an additional network that maps latent codes to blur kernels. Experiments show that this proposed extension of Gaussian VAEs successfully decreases the blur of the reconstructed/generated images. ", "summary_of_the_review": "The paper proposes an interesting VAE extension that directly penalises the generation of blurry images and shows experimentally that it decreases the blur of the reconstructed/generated images. The main weakness of the manuscript is in my view a missing concise probability model for the extended VAE, which is perhaps the reason why the learning task is formulated in terms of two optimisation tasks with interdependent objectives and solved approximately by coordinate block descent.", "strength_and_weaknesses": "Paper strengths:\nThe authors propose a weighted deblur with more focus on high frequency components in the Fourier space along with a decoder that represents a multivariate Gaussian with non-identity covariance matrix obtained from the blur kernel. This includes a reasonable approximation for computing the determinant of the latter. It is interesting to see that this leads to an VAE objective with parts of it defined in the Fourier space.\n\nThe experiments are well designed and convincing. They clearly show that the proposed VAE extension \nsuccessfully decreases the blur of the reconstructed/generated images. The authors provide an ablation study for the model hyper-parameters. A comparison with other existing methods shows competitiveness of the proposed approach.\n\nPaper weaknesses:\nThe main weakness in my opinion is that the authors give no concise probability model for the decoder in terms of $x$, $\\hat{x}$ and $z$, parametrised by  $\\theta$ and the blur kernel $k$. This is perhaps the reason why the learning of the model splits into two optimisation tasks with inter-connected objectives (11) and (12).\n\nThe practical value of the approach is in my view somewhat restricted because it is well known that more complex VAE models, like hierarchical VAEs or VAEs with normalising flows usually do not show this blur degradation in the reconstructed/generated images. On the other hand,it is known (see [1,2]) that ELBO learning for standard VAEs introduces a bias towards consistent encoder/decoder pairs which correspond to EF-harmoniums as joint models. It would be therefore interesting, to explore whether the \"explaining away\" of high frequency image components could be attributed to the restricted expressive power of EF-harmoniums.\n\n[1] Welling et al, NeurIPS 2005\n\n[2] Shekhovtsov et al., ICLR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster