Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to solve the low-efficiency problem of diffusion models and presents a pyramidal diffusion model that can generate high-resolution images starting from much coarser resolution images using a single score function.
The key idea is to use a positional embedding, which enables a neural network to be much lighter and also enables time-efficient image generation without compromising its performance. The paper also shows that the proposed approach can be used for super-resolution using a single score function.
strength_and_weaknesses: Strength:

- The core idea of incorporating positional encoding into the diffusion model is a valid contribution. It allows multi-scale sampling with a single score function model. The effectiveness of this strategy is verified with ablation study.

- The paper shows good sampling speed for diffusion-based image generation.

- The paper makes a good literature review about denoising diffusion models in the introduction, especially the works on applications of diffusion models.


Weakness:

- The results of the proposed algorithm do not look very good. While I understand this might be due to the small model size used in this paper, it will be important to show that the algorithm has the potential to achieve better performance by slightly enlarging model sizes.

- For the speed comparison, it is important to compare with existing methods that are also based on multi-scale sampling, for example, SR3 and Cascaded Diffusion Model.

- The paper is not well written, and some contents are hard to understand. For example:

(1) What is 2x2 in positional encoding in Figure 2? 

(2) On Line 6 of Algorithm 1, I think it should be "while $x_0$ is 'not' HR do". Please clarify if I misunderstand this.

(3) On P6, the paper says "1000 diffusion steps for all training". It is not clear how these steps are split between different scales. Also, I don't know why the number of steps is different in inference: $T_f$=1000, $T_s$=100, which is more than 1000 in total.

(4) On P8, the paper mentioned Table 2(b) and Table 1(a) which however are never presented in the paper.

(5) On P9, the paper presents an experiment where the score model is trained with patches of 256×256 and 512×512 images for the generation of 1024×1024 images. What is this experiment for? How do we interpret this result?

(6) Typo: on P6, $T^f$ and $T^s$ should be $T_f$ and $T_s$; in Figure 3, diffsion --> diffusion
clarity,_quality,_novelty_and_reproducibility: The clarity should be improved. The novelty and quality are okay. See more details in Strength And Weaknesses.
summary_of_the_review: The usage of positional encoding in the diffusion model is interesting. However, the results are not quite convincing. Also, the paper has writing issues. See more details in Strength And Weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper develops a variant (positional encoding) of conditional denoising diffusion probabilistic model (DDPM) with the goal of efficient sampling and multi-scale image generation while still maintaining the performance. By enforcing the gradient of the data-consistency during multi-stage sampling, the proposed diffusion model can also be used for image super-solution (SR). Both quantitative and qualitative results of this work are provided, comparing against several existing methods. The authors compared their method with several existing methods in terms of  sampling speed, image generation quality and SR performance.
strength_and_weaknesses: Strengths:

I think the overall idea is interesting: designing a diffusion model that assembles the positional encoding as condition for fast sampling and multi-scale image generation. I believe this is an overall well-written paper with some new results that could spark further research in the interesting topic of image continuous scale generation. But I fear that the justification and benefit for the usage of positional encoding as condition input is not well justified through the experiments. Please find my technical comments below.

Weaknesses:

1, As cooperated by the authors, using positional information for training image generator networks is not new.

2, More importantly, for both image generation quality and SR restoration, the comparison with existing methods seems to be inaccurate. For image generation comparison, it is unclear why only the unofficial implementation results are reported. For example, in the original DDIM [Song.etal2021], the authors reported FID score of 10.84 on the LSUN-Church dataset when sampling 50 iterations.  Besides, the SR3 results show serious mean shift even for SRX4. The baseline methods seem to be re-implemented inappropriately. The pretrained DDPMs are public-available and reported similar results to the official implementation, such as https://github.com/openai/improved-diffusion. 

3, Table 1. reports improved sampling speed of this work, comparing to baseline methods.  However, the reasons lead to such speedup is not clearly presented through ablation studies. As mentioned by the authors the CCDF (Chung et al., 2021) acceleration scheme is used as an acceleration scheme. Since CCDF is a general acceleration scheme, for fair comparison, all methods should use the same acceleration settings.

4, No continuous magnification results are showed in this paper. Since this work takes coordinate information as a condition and uses fully CNN architecture, it would be very informative to show the image generation or SR  results for continuous scale results (e.g., X3.2, X6.5).

5, Some implementation details are confusing. “We trained our model using FFHQ 256×256 (Choi et al., 2020), LSUN-Church 256×256 (Yu et al.,2015) datasets for 1.2M iterations”. Is this means that a single model is trained on multiple dataset ?

clarity,_quality,_novelty_and_reproducibility: Clarity/ Quality:  The paper is overall well-structured and easy to follow. However, some technical and experimental setting is not clearly stated. For example, the settings of hyperparameters $T_s/T_f$ and the use of CCDF  acceleration  are not clearly stated. Besides, the implementation details of Eq(10) are unclear. Finally, the introduction of  the network architecture is missing.

Novelty:  Using positional encoding for conditionally training diffusion models is an interesting direction, although based on existing work, it is overall a novel attempt for efficient and multi-scale image generation. However, I worry about the use of positional encoding is not well studied in this work with somehow insufficient numerical validations, which may diminish this proposal’s significance. 

Reproducibility: Since this work is the integrations of existing works, the reproducibility seems not to be the issue.
summary_of_the_review: Consequently, given the pros and cons on balance, I feel this is a very borderline paper, and I vote for borderline accept tentatively
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is about training a multi-scale diffusion model in a unified framework. Specifically, it uses a casacaded mutli-scale generation process (from lower-resolution to higher-resolution). Unlike prior work which uses several score networks at each scale level. This paper uses a single network and proposes to use spatial positional encoding to make the network scale-aware and location aware. This spatial positional encoding also enables patch-wise training, which makes it feasible for large-scale image generation. Results demonstrated its ability to generate high-fidelity images at various scales.
strength_and_weaknesses: Strength.
1. The idea of using a single score function to progressively generate higher-resolution images is interesting. And the use of spatial positional encoding seems an elegant solution for this task.

2. Introducing spatial positional encoding also enables patch-wise training of the diffusion model, which makes the training process much more flexible.

Weakness.
1. I do not fully agree with the claim of the super-resolution capability of the proposed method, although I do agree with the cascaded generation method. As seen from the red box of Fig.1, the 2x super-resolution breaks the consistency between the input lower-resolution image and the output image. (the mouth part). I think this is because every time a generated image is super-resolved, Gaussian noise is added during the forward diffusion process.  Although this is arguable because super-resolution can be ill-posed, however 2x super-resolution should not produce such a mismatch. I believe that's also why only on 4x~ super-resolution task, can this method outperforms existing works.

2. The comparison with SR3 seems weird. As seen in Figure.4, the generated images of SR3 have severe chromatic distortion. I think the unofficial implementation of SR3 may be buggy.



Other comments
Typos: 1. missing closing brackets in the reverse diffusion process in Algorithm.1 and Algorithm.2 
clarity,_quality,_novelty_and_reproducibility: This paper is clear and reproducible.

The introduction of spatial positional encoding for multi-scale cascaded generation with a single score function is novel and interesting.


summary_of_the_review: I'm not fully aware of the field of diffusion models, so maybe I missed some prior works. From my point of view, this paper is novel and interesting. The contribution of this paper is clear. It uses spatial positional encoding to solve multi-scale cascaded generation with a single score function. Besides, this also enables patch-wise training.
The results also seem good.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new method to progressively generate or super-resolve to high-resolution images. It uses a diffusion approach for every up-sampling step, an idea it shares with Saharia et al. (SR3) and Ho et al. Its main innovation is in using a single model for all up-sampling steps (while SR3 used a separate model for each step).
strength_and_weaknesses: # Strengths

- The idea of using positional embeddings to learn a single model to be applied across scales is interesting.

# Weaknesses

The main problem with the paper is in its evaluation.

- For the image generation task, the paper trains its own models for the competing methods, and the results are extremely surprising. While the competing methods are clustered together in their score, the proposed method does significantly better while also being faster and having an order of magnitude fewer parameters. For example, on the LSUN church dataset, the four other methods have scores ranging from 25.30 to 25.78, while the proposed method has a score of 14.07. The paper also doesn't include any visual examples of the images generated by the other methods, only its own.

Training requires setting and choosing many hyper-parameters, and given the significant gap in scores, it is possible that this difference is because of some hyper-parameter setting. Note that the other papers report their own numbers on different benchmarks. Why not train the proposed model under those settings, and report numbers comparing to those methods that way?

- For the super-resolution experiments, the closest method to the proposed one is SR3 --- which also does iterative scale diffusion, except with separate networks for each scale. The authors report results with their own trained version of SR3, and it appears to perform significantly worse. Note that the reported results show SR3 with a 7-9db worse PSRN than SRGAN while in SR3's own evaluation, it performed equivalently / slightly better than SRGAN (also on face images). Indeed, SR3 has 6-8db worse PSNR than bicubic interpolation! Moreover, visually inspecting the results, there seems to be a very visible change in the color temperature of the SR3 outputs from those of all other methods. This again suggests that there might be some issue with the implementation of SR3 used for testing.

In this case again, the SR3 paper reports results on a benchmark with a clear description of the training and evaluation sets. Why not replicate those settings and compare to SR3 on those reported numbers?

- In addition to the problematic comparisons to the baseline, the ablation studies could be improved. In particular, the ablation study should evaluate its specific novel contributions over previous methods. For example, it could show results from training a separate network for each scale, and demonstrate that with the proposed positional encoding, a single network comes close to matching results from such separate networks. It could show the benefit of CCDF and the projection steps, showing results when those were not applied.

- Given that the paper focuses on image generation, and in particular evaluates on face images, it should discuss the potential for bias in generated images.
clarity,_quality,_novelty_and_reproducibility: The description of the method is clear.
summary_of_the_review: Given the weaknesses in the evaluation as mentioned above, I do not think the paper is ready for publication. A new revision containing evaluations comparing to baselines in settings reported in those papers needs to undergo a fresh round of reviews. Moreover, the paper should include a discussion of bias and fairness concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces pyramidal denoising diffusion models that share one score function for different image resolutions. In addition to noise-injected images, the score function is given positional embeddings per pixel, allowing the model to adapt itself across different scales. 

Pros:
- The core idea of incorporating positional encoding into the diffusion model for multi-resolution training is novel and interesting. 

Cons:
- Most re-implemented baselines perform poorly compared to their official published results. Several reviewers have raised questions on whether experimental comparisons are sound.
- Sampling time comparisons are not performed in a controlled apple-to-apple fashion. 
- Missing details and confusing descriptions at times
justification_for_why_not_higher_score: While the proposed idea is interesting and novel, almost all the reviewers have raised serious concerns regarding the experimental comparisons and reported results for prior works. Given these concerns, this submission does not appear to be ready for publication at ICLR.
Decision: Reject