Venue: ICLR 2022 Workshop DGM4HSD

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Summary**

This paper proposes an approach for semantic image synthesis. Instead of training the transformer on conditioning latents and image latents separately, the paper proposes to jointly train the model. The proposed approach is able to improve semantic image synthesis on commonly seen semantic image datasets.


**Comments**
1. The paper is well written and easy to follow.
2. The proposed approach achieves strong performance in the experiments. The experiments are detailed.
3. It would be good to provide some theoretical analysis on the performance gain.
4. It would be good to add human evaluations for comparison with the baselines.
5. It would be good to show some failure examples and discuss the limitations.


review_title: The paper is easy to follow with good results

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work addresses the problem of semantically conditional image synthesis -- that is generating images conditioned on a segmentation map which specifies various objects and background types. This is an important problem in computer graphics.

This work builds on previous vector-quantized image generation methods by training 2 vector quantized models; one for the natural images, and another for the semantic segmentation maps. This creates 2 discrete codebooks -- one for each modality. The work then replaces the standard autoregressvive transformer model for the latent codes with a conditional autoregressive model. This model generates the image-tokens conditioned on the tokens from the segmentation map.

The authors find that this additional conditioning step improves image generation performance scored by the FID, LPIPS, and SSIM (metrics which qualitatively score perceptual image quality). 

While I do not find this work to be very theoretically novel, the results are nice and demonstrate the flexibility of the vector-quantized approach to image generation. Therefore, I would advocate for the paper's acceptance. 
review_title: Semantic conditioning for VQ generative models with very nice results

Decision: Accept (Poster)