Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The composition of the encoder and decoder mapping of standard Gaussian variational autoencoders (VAE) is known to "explain away" high frequency image components as independent Gaussian noise. The authors propose to address the resulting blur in reconstructed and generated images directly, by adding  a blur kernel to the model parameters, which depends on the latent code. They propose a coordinate block descent approach for solving the optimisation w.r.t. the standard VAE parameters and the parameters of an additional network that maps latent codes to blur kernels. Experiments show that this proposed extension of Gaussian VAEs successfully decreases the blur of the reconstructed/generated images. 
strength_and_weaknesses: Paper strengths:
The authors propose a weighted deblur with more focus on high frequency components in the Fourier space along with a decoder that represents a multivariate Gaussian with non-identity covariance matrix obtained from the blur kernel. This includes a reasonable approximation for computing the determinant of the latter. It is interesting to see that this leads to an VAE objective with parts of it defined in the Fourier space.

The experiments are well designed and convincing. They clearly show that the proposed VAE extension 
successfully decreases the blur of the reconstructed/generated images. The authors provide an ablation study for the model hyper-parameters. A comparison with other existing methods shows competitiveness of the proposed approach.

Paper weaknesses:
The main weakness in my opinion is that the authors give no concise probability model for the decoder in terms of $x$, $\hat{x}$ and $z$, parametrised by  $\theta$ and the blur kernel $k$. This is perhaps the reason why the learning of the model splits into two optimisation tasks with inter-connected objectives (11) and (12).

The practical value of the approach is in my view somewhat restricted because it is well known that more complex VAE models, like hierarchical VAEs or VAEs with normalising flows usually do not show this blur degradation in the reconstructed/generated images. On the other hand,it is known (see [1,2]) that ELBO learning for standard VAEs introduces a bias towards consistent encoder/decoder pairs which correspond to EF-harmoniums as joint models. It would be therefore interesting, to explore whether the "explaining away" of high frequency image components could be attributed to the restricted expressive power of EF-harmoniums.

[1] Welling et al, NeurIPS 2005

[2] Shekhovtsov et al., ICLR 2022
clarity,_quality,_novelty_and_reproducibility: The paper is well organised and clearly written. I believe that the missing architecture details of the the decoder/encoder pairs do not hinder its reproducibility. However, I would have appreciated to see a short explanation for the gradient computation of the Fourier domain terms in (12).
summary_of_the_review: The paper proposes an interesting VAE extension that directly penalises the generation of blurry images and shows experimentally that it decreases the blur of the reconstructed/generated images. The main weakness of the manuscript is in my view a missing concise probability model for the extended VAE, which is perhaps the reason why the learning task is formulated in terms of two optimisation tasks with interdependent objectives and solved approximately by coordinate block descent.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new reconstruction term that minimize the blurriness of the VAE while still maintaining the mathematical connection to the original ELBO objective. The paper provides an ablation study of several hyperparameters and tested on dataset such as CELEBA and HCP medical dataset while comparing to some of the other recently proposed reconstruction term. 
strength_and_weaknesses: Strength:
1. The paper provide a detailed ablation study on the choice of Sigma and C.
2. The proposed method seems quite novel and interesting. 

Weakness:
1. The paper only compares with other algorithms that have alternative reconstruction term, but did not provide comparison with other SOTA VAE variations. A simple example is the beta VAE which weighs the KL divergence. In general, we could either weighs the reconstruction term or the KL term of the VAE. I think for the very least, comparison with other KL divergence weighting algorithm would be meaningful. 
2. in terms of experiment results, it looks like for CELEBA 256 by 256 dataset, the proposed reconstruction term doesn't seem to to be better than the cross entropy reconstruction in a few metrics. Although for the medical dataset, there seems to be improvements. The significance of the improvements are hard to determine because no standard deviations are provided. The reconstruction images of the medical dataset from L2 and proposed method seem quite similar visually. The overall the benefits of the algorithm are not very convincing. 

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the proposed reconstruction term seems novel. The results in the table don't have variance, thus their significance is hard to determine. 
summary_of_the_review: I think the paper is interesting and the proposed algorithm is novel but motivation for the new reconstruction term is weak because its improvement is hard to determine.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method for reducing blur in VAEs by explicitly incorporating a de-blurring convolutional operation, which enters the covariance of the conditional likelihood. This allows the method to upweight errors at higher frequencies. In the proposed approach, the conditional likelihood is evaluated in frequency-space, with the covariance determinant calculated analytically. The de-blurring kernel is calculated per-image as a function of the latent variables. Notably, the entire approach still optimizes the ELBO, unlike other heuristic methods. Experiments are performed on CelebA (high resolution and low resolution versions) and a medical image dataset. The proposed method outperforms similar approaches in terms of various metrics, although the reconstructions and generations still seem somewhat blurry.
strength_and_weaknesses: **Strengths**

- **Principled, well-formulated approach that connects two sub-fields to tackle a known problem.** The authors rigorously demonstrate how to incorporate learned, de-blurring convolutional kernels into the conditional likelihood of VAEs, such that the probability density is still valid. While various previous works have proposed heuristics for improving the blurring in VAEs, even looking at spatial frequencies, to the best of my knowledge, none of these previous works have incorporated these aspects in a rigorous probabilistic modeling framework. The end result is a clear demonstration of how to connect ideas from low-level image processing to VAEs, which then be built upon for further in future work. Further, this approach attempts to directly address a known issue with VAEs (blurring) using a principled approach: adding spatial dependencies to the conditional likelihood.

- **Fairly clear presentation.** For the most part, the authors do a great job of explaining the motivation and mechanics behind their approach. They start by discussing why it is that high-frequency components tend to be neglected in VAEs and other maximum-likelihood generative models. They then present basic concepts from image processing, i.e., de-blurring. They then walk through how to incorporate this operation into the covariance of the conditional likelihood in VAEs, covering the key components and hyperparameters. I especially appreciated the diagram outlining the approach, which was quite clear. While I still have several questions about the formulation, the paper’s presentation is in good shape overall.

- **Reasonable empirical evaluation.** The empirical evaluation appears to be sufficient to demonstrate meaningful benefits over previous similar methods on several representative datasets. The authors evaluate standard metrics to quantify their performance improvements. The provided ablation evaluation is also helpful to assess the dependence on various hyperparameters.


**Weaknesses**

- **Could be scaled up further.** The authors demonstrate their method in the setting of single-level convolutional —> fully-connected —> convolutional VAEs. Most competitive VAEs utilize multiple levels of latent variables, e.g., VDVAE (Child, 2021). In principle, this method could be combined with such higher capacity models to possibly provide even further improvement. It would be useful to assess whether this de-blurring technique is still useful in this higher capacity setting. Likewise, in the natural image setting, the authors only demonstrate results on CelebA (although they use two versions of the dataset). I would have expected results on other benchmark datasets, particularly CIFAR10. Generating faces using datasets that have fairly normalized poses feels like perhaps too specific of a setting to make convincing arguments about modeling sharper edges in natural images.

- **Some aspects a bit unclear.** While the method itself appears to be well-formulated, some aspects were a bit unclear to me. For instance, why do we need a second objective to learn the convolutional filter? Shouldn’t this already be rolled into the maximum likelihood objective? Is this primarily for optimization purposes? What dictates the form of this objective — the form of the conditional likelihood? Could we have multiple stages of de-blurring? How does this compare with, for example, using an autoregressive or flow-based conditional likelihood? These alternative forms of densities are also capable of capturing low-level pixel dependencies while still maintaining a valid log-likelihood.

- **Seems somewhat specific to Gaussian conditional likelihoods.** The authors present their approach purely using Gaussian conditional likelihoods. While this is typically a reasonable assumption, this also seems to limit the scope of the approach. For instance, many higher-capacity VAEs utilize discretized mixture of logistic distributions (VDVAE; Child, 2021). While I think it may be possible to apply this technique in these other settings, this is left for future work.
clarity,_quality,_novelty_and_reproducibility: **Clarity** - For the most part, the paper is very clear in describing the motivation, proposed approach, and empirical results. The authors clearly define all mathematical terms, first walking through the simple cases, then discussing more nuanced points. The diagram of the model/method clearly lays out the basic ideas, showing both the model, a diagram for calculating the conditional likelihood covariance, and the objective for each component. This could perhaps be further improved by visualizing the kernel, Fourier transform, covariance matrix, etc. as tensors in more of a proper network diagram. The empirical results are generally quite clear, although I would consider several changes. Table 1 should be converted into plots, or at least the best results should be bolded. Similarly, the results in Table 2 could be bolded. It may also be helpful to visualize the results in frequency space, showing the spectrum of the datasets and comparing with that of the reconstructed and generated results. One would expect to see that the proposed result better matches the power of the higher frequencies as compared with baselines.

**Quality** - The quality of the paper is high. The authors take concepts from the image processing literature and show how to incorporate these into VAEs in a principled way. The mathematics are clear throughout. The end result is a valid probabilistic model with added de-blurring capabilities. The authors demonstrate the benefits of this approach on relevant datasets, comparing against multiple relevant baselines, while holding other factors (i.e., model architecture) constant. Results are reported using a variety of metrics, where the proposed method compares favorably. Thus, this paper takes a well-formulated idea and demonstrates that it improves performance. The authors may consider adding additional results analyzing precisely how the proposed method improves performance. Likewise, results on additional datasets would expand the appeal of the paper beyond generating faces and MRI images.

**Novelty** - The paper is somewhat novel. Many previous works have considered ways to better capture the dependencies between input dimensions in VAEs. Some of these works fall into other model classes, like including autoregressive models at the output, while others offer up heuristic approaches, e.g., perceptual losses. This paper takes the idea of de-blurring/sharpening from the image processing literature and demonstrates how to incorporate these into VAEs. The mathematical ideas, though not overly complex, are not generally found within the VAE literature. Further, there are a significant number of choices to be made in properly formulating the approach, i.e., this isn’t simply a matter of tweaking some previous proposed model/method.

**Reproducibility** - I suspect that the results are reproducible, as the authors present comparisons with multiple baselines on multiple datasets. And the results of the proposed method do, qualitatively, look better than the baselines. However, the authors may wish to improve the analysis of their results by running multiple seeds and reporting confidence intervals. Likewise, especially with modular techniques like this, it may help to use a standard, previously-proposed architecture when running experiments, allowing for readers to more easily compare results across papers. Finally, one of the benefits of the proposed approach is that one can still retain the proper probabilistic model formulation, so it may be useful to also report log-likelihood results.
summary_of_the_review: The paper takes a principled idea, de-blurring, and shows how to incorporate this into VAEs while retaining a proper probabilistic model. The proposed method is evaluated on multiple datasets, comparing against relevant baselines. For these reasons, I feel that this paper warrants publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Ratings: 6/6/8
Confidence: 4/2/4

The paper proposes a method for reducing blur in the samples of variational autoencoders (VAEs) by incorporating a de-blurring convolutional operation in the output of the decoder p(x|z), through the covariance of the conditional likelihood. In the approach, the conditional probability p(x|z) is evaluated in frequency space, with the covariance determinant computed analytically. Unlike other heuristic methods, the method still optimizes the ELBO. Experiments show that the method outperforms alternative approaches, although the method was not able to remove all blur.

The reviewers commend the principled approach, and the experiments against reasonable baselines. There were some complaints about the lack of concise presentation of the probabilistic model, that the method seems somewhat specific to Gaussian likelihoods, and that the model on which the method is tested is small by contemporary standards.
justification_for_why_not_higher_score: There were some complaints about the lack of concise presentation of the probabilistic model, that the method seems somewhat specific to Gaussian likelihoods, and that the model on which the method is tested is small by contemporary standards.
justification_for_why_not_lower_score: This seems like a decent paper. The reviewers commend the principled approach, and the experiments against reasonable baselines.
Decision: Accept: poster