Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper is dedicated to the sign language production problem. In the paper, based on VQ-Diffusion, the authors propose the PoseVQ-Diffusion model to generate the pose sequence with iterative refinement. The proposed method achieves encouraging performance on PHOENIX dataset.
strengths_and_weaknesses: Strength:
The paper leverages the VQ-diffusion model to sign language production, which is novel and interesting. The diffusion model is considered to be potential on generative problems. The attempt made in the paper is very meaningful. Meanwhile, the authors propose several additional improvements for VQ-diffusion on SLP, which is also good.

Weakness:
Despite the novel framework, there are some non-trivial concerns on the effectiveness evaluation of the proposed framework.

First, the authors only compare the proposed method with two previous methods, and the latest method compared is NAT-AT/EA from ACM MM 2021. Several important baseline methods, i.e., [1] and [2], are missing.
[1] Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives. Ben Saunders, et. al. ICCV 2021.
[2] Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production. Ben Saunders, et. al. CVPR 2022.

Secondly, the authors claim that the proposed method has a faster sampling speed compared with autoregressive models. However, there is no experimental results to support this argument.

questions: 1. In Line 16 – 17, the authors claim that the proposed method has a faster sampling speed compared with autoregressive models. However, there is no experimental results to support this argument. As a matter of fact, diffusion models usually have more model parameters and longer training & sampling times. 
2. In Line 164 – 165, the summation of the probability to be unchanged, to be uniformly resampled and to be replaced with [MASK] token is not equal to 1.
3. The authors do not present the qualitative comparison with NAT-AT and NAT-EA.

limitations: I include several constructive suggestions for improvement.
1. The authors should provide the comparison with the missing methods mentioned above.
2. The authors should compare the model parameter and training & sampling time with previous methods.
3. Other types of evaluation, such as user study, will further verify the effectiveness of the proposed method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackles the problem of Sign Language Production (SLP), which aims to translate spoken languages into sign pose sequences. The proposed pipeline is based on a Vector Quantized Diffusion model.  The key contribution consists of separating a sign skeleton into three local point patches representing pose, right hand, and left hand separately. An extension of standard VQ diffusion method by considering mask token and a new architecture CodeUnet as well as lengths prediction of sign glosses. 

The proposed approach demonstrates its effectiveness on RWTH-PHOENIX-WEATHER-2014T.

strengths_and_weaknesses: **Strength**
 
* [S1]: The paper is well organized, and the approach is well presented and self-contained.

* [S2]: The improvement on RWTH-PHOENIX-WEATHER-2014T is consistent over different metrics.

**Weakness**

* [W1]: The main weakness is the novelty. The core idea of this paper, _i.e._ VQ Diffusion, has been demonstrated to be successful in many generation tasks. Thus it is not surprising that it works on SLP. Learning different codebooks for different groups of joints is not novel as well, which has been used in music to dance generation [a]. 

* [W2]: The experimental part is a little bit weak, as only one dataset is considered. Although the paper mentioned there is only one dataset publicly available.

* [W3]: In the related work, the paper mentions [4, 26] as works to generate pose sequences from glosses. I am wondering whether there is a particular reason to make a fair comparison to these approaches. 

* [W4]: In the case of using 3 codebooks (pose, right hand, and left hand), the reconstruction becomes simpler however the generation might be harder as well. I think it would be more convincing to validate the idea of separating codebooks on generation rather than reconstruction. 

**Minor Questions towards Clarification**
 
* [C1]: Training details are missing, such as optimizer, weight decay, learning rate strategy etc. Also, it is not clear whether the paper leverages exponential moving average to train the codebook, which is a common practice to train VQ-VAE.

* [C2]: Figure 2: Leangth Predictor → Length Predictor

_[a]: Siyao, Li, et al. "Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022._
questions: Please refer to the weaknesses. 
limitations: No potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a novel text-to-sign pose sequences generation method. The proposed method is based on the powerful discrete denoising diffusion probabilistic model. It solves the long-range autoregressive generation problem in a non-autoregressive manner. A vector
quantized conditional diffusion model is proposed for the pose generation. The proposed method achieves state-of-the-art results on the SLP evaluation benchmark.
strengths_and_weaknesses: # Strengths
1. The proposed method uses a novel diffusion-based model and achieves great performance.
2. The authors argue they solve the problem in an auto-regressive way, which may be meaningful in the relative fields.
3. A Pose-VQVAE is used to learn the meaningful codebook and the discrete diffusion model is devised in a fancy way.
4. The CodeUnet is proposed in the diffusion model for better leveraging the spatial and temporal information of the quantized pose sequences.

# Weaknesses
## Main Issue
1. This work is based on the works of some diffusion-based models, especially the CVPR 2022 paper Vector quantized diffusion model for text-to-image synthesis [1]. My biggest concern is that we can not tell how big the contribution of this work is. On the one hand, this work
introduces the VQ-Diffusion method proposed in a text-to-image generation to the G2P task and achieves great performance, which is impressive. On the other hand, compared with the former CVPR work, the method in this paper uses the CodeUnet, which is not a big step, from my perspective, and only proposes some small points, like dividing the joints into 3 subsets. Therefore, it is really hard to judge how big the contribution is.

## Some Minor Issues
1. A typo in line 107.
2. For the discrete diffusion model part, why there is a PAD state? I think the meaning of the existence of PAD is not illustrated clearly in the section.
3. The experiments part may not be sufficient enough. For example, the method tries to divide the joints into 3 subsets. What will the results be like if we use a simpler implementation?
4. In table 1 in the results, how to evaluate the numerical results? How to define better results should be discussed in the paper. Higher or smaller? Or closer to the GT?
5. In the checklist, the authors say that the code is provided. However, this is missing in the supplementary materials.

[1] Vector quantized diffusion model for text-to-image synthesis, Gu et al., CVPR 2022.
questions: 1. Have the authors tried to use some other models to replace the CodeUnet? Leveraging transformer-based blocks and Unet is interesting. But how well does the network work compared with other widely used networks?
2. Why is the PAD state used and what is the meaning? It seems using MASK state is enough. And this point is not illustrated in the paper. From the transition matrix, no other state will turn to be PAD and PAD is an isolated state. Then why do we still need to talk about whether
$x_t$ equals V+2?
3. Do you have some other experimental results and ablation studies? It seems experiments should be done to verify the effectiveness of many parts of the work, including the architecture, section 3.3, etc.
limitations: For the checklist, the 6th and 7th "YES" may be misleading.
review_title: null

Meta-review: metareview: The paper is interested to the sign language production (SLP) problem. A vector quantized conditional diffusion model is proposed for the pose generation. The proposed method achieves state-of-the-art results on the SLP evaluation benchmark (PHOENIX dataset).

Reviewers all agree that the key contribution is very interesting --  making VQ-diffusion work on SLP.  However, the technical novelty is low for NeurIPS. In that respect, reviewers agree that given VQ-diffusion has been shown (in past work) to perform quite well on text-to-image generation, the technical novelty is low here.

Several reviewers felt also the experimental section is a bit weak, mostly because PHOENIX is the only available benchmark for SLP.

Most other concerns have been fixed during the rebuttal phase. Following reviewers, the rejection is based on the limited technical novelty for NeurIPS.
confidence: Certain
Decision: Reject