Vector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation
Authors: Pan Xie
Qipeng zhang
Zexian Li
Hao Tang
Yao Du
Xiaohui Hu
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Sign Language Production
Discrete Diffusion Model
Pose Sequence Generation
Vector Quantized

Abstract: Sign Language Production (SLP) aims to translate spoken languages into sign sequences automatically. The core process of SLP is to transform sign gloss sequences into their corresponding sign pose sequences (G2P). Most existing G2P models usually perform this conditional long-range generation in an autoregressive manner, which inevitably leads to an accumulation of errors. To address this issue, we propose a vector quantized diffusion method for conditional pose sequences generation, called PoseVQ-Diffusion, which is an iterative non-autoregressive method. Specifically, we first introduce a vector quantized variational autoencoder (Pose-VQVAE) model to represent a pose sequence as a sequence of latent codes. Then we model the latent discrete space by an extension of the recently developed diffusion architecture. To better leverage the spatial-temporal information, we introduce a novel architecture, namely CodeUnet, to generate higher quality pose sequence in the discrete space. Moreover, taking advantage of the learned codes, we develop a novel sequential k-nearest-neighbours method to predict the variable lengths of pose sequences for corresponding gloss sequences. Consequently, compared with the autoregressive G2P models, our 
model has a faster sampling speed and produces significantly better results. Compared with previous non-autoregressive G2P methods, PoseVQ-Diffusion improves the predicted results with iterative refinements, thus achieving state-of-the-art results on the SLP evaluation benchmark.

Vector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation
Anonymous Author(s)
Affiliation
Address
email
Abstract
Sign Language Production (SLP) aims to translate spoken languages into sign sequences automatically. The core process of SLP is to transform sign gloss sequences into their corresponding sign pose sequences (G2P). Most existing G2P models usually perform this conditional long-range generation in autoregressive manner, which inevitably leads to an accumulation of errors. To address this issue, we propose vector quantized diffusion method for conditional pose sequences generation, called Pose PoseVQ-Diffusion which is an iterative non-autoregressive method. Specifically,
3
5
first introduce vector quantized variational autoencoder (Pose-VQVAle model to represent pose sequence as sequence of latent codes. Then we model the latent discrete space by an extension of the recently developed diffusion architecture. To better leverage the apatial-temport information, we introduce novel architecture, namely CodeUnet, to generate higher quality pose sequence in the discrete space. Moreover, taking advantage of the learned codes, we develop a novel sequential k- hhearest-niighhoon lengths of pose sequences for corresponding gloss sequences. Consequently, compared with the autoregressive G2P models, our model has a faster sampling speed and produces significantly better results. Compared with previous non- autoregressive G2P methods, VQ-Diffuunn improves the predicted results with iterative refinements, thus achieving state-of- the-art results on the SLP evaluation benchmark.
10
11
12
13
14
method to predict the variable
15
16
18
20
Introduction
21
22
Sign Language Production (SLP), as an essential task for the Deaf community, aims to provide continuously sign videos for spoken language sentences. Since sign languages are distinct linguistic systems [ which differ from natural languages, sign languages have different word orders from their corresponding natural languages. Therefore, directly learning the alignment mapping between them challenging. tackle this issue, previous works first translate spoken languages into glosses then generate the sign pose sequences based on the gloss sequences (G2P) 2. and finally optionally use the sign pose sequence to generate the photo-realistic sign video Accordingly, G2P the heart procedure of this task, and is the focus of this paper.
23
24
25
26
27
28
30 Existing approaches for G2P can be categorized into autoregressive
and non-autoregressive methods depending on their decoding strategies. Autoregressive models 2 generate the next pose frame depending on previous frames relying on the teacher forcing strategy [6]. In inference, the recurrent decoding is likely to lead to prediction еттог propagation over time due to the exposure To break the bottleneck of autoregression, -otoeertrreeeeivvve methods are proposed to
31
32
33
34
bias 77
Siign glosses are spoken language words that match the meaning of signs and, linguistically, manifest as minimal lexical items.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

35 induce the decoder to generate all target predictions simultaneously 8 99. Huang at al. [5] proposed
36 a non- autaerrrrgreesvve G2P model to generate sign pose sequence parallelly in a one-shot decoding scheme, and used an External Aligner (EA) for sequence alignment learning.
37
38
Motivated by the recent developed Discrete Denoising Diffusion Probabilistic Model (D3PMs) 10
39
I [2] which achieved impressive results for language generation and vector quantized image generation. We propose a Pose Vector Quantized Diffusion (Pose VQ-Diffusion) model to learn the conditional pose sequence generation in the latent discrete space instead of the continuous coordinate
40
41
42
space. It is also -on-eetorereessvve method that performs parallel refinement on the generated results with iterations and therefore shows expressive generative capacity.
43
44
We will elaborate our approach in three steps. Firstly, utilize vector quantized variational autoencoder ( (VQ-VAE) to represent the pose sequence as sequential latent codes. Different from we devise specific architecture, Pose- VOVA,, to learn the meaningful
45
image VQ-VAE 13 14, codebook by reconstructing the pose sequence. Specifically, we divide a sign skeleton into three local point patches representing pose, right hand and left hand separately, Furthermore, Tokenizer with a vector quantized variational autoencoder is designed to learn discrete point tokens containing local semantic information.
47
48
49
50
51 Next, we extend the standard vector quantized diffusion methods aligmments between sign glosses and quantized codes. The discrete diffusion model samples the data distribution by reversing a forward diffusion process that gradually corrupts the input via a fixed Markov chain. Its corruption process by adding noise data (e.g., [MASK] token) draws our attention to the mask-based generative model, Mask-Predict (9] which is proved to be a variant of diffusion model [1]. In this paper, explore two variants of the diffusion model for our quantized pose sequence generation. Expanding further, to better leverage the spatial and temporal information of the quantized pose sequences, we introduce new architecture CodeUnet. In contrast Unet (15) which is a "fully convolution network" for image data, CodeUnet is a "fully transformer network" designed for discrete tokens. As a result of iterative refinements and better spatial-temporal modelling, our model achieves a higher quality of the conditional pose sequence generation.
to model the sequential
53
54
55
57
58
59
60
61
Lastly, the length prediction of the non-autoregressive G2P models is challenging since the corre- sponding lengths of different sign glosses are different and variable. In this paper, we propose a novel clustering method for this typical sequential data that local adjacent frames should belong to cluster. Specifically, taking advantage of the meaningful learned codes in the first stage, we firstly apply the k-nearest-neighhor based density peaks clustering algorithm [16] 17 to locate the peaks with higher local density. Secondly, we design a heuristic algorithm to find the boundary between two peaks according to their semantic distance with the two peak codes. Finally, we leverage the length of each gloss as the additional supervised information and predict the length of the gloss sequence in the inference.
69
64
65
87
68
69
70
71
Our model significantly improves the generation quality on the challenging RWTH-PHOENIX WEATHER-200 2014T [18] dataset. The evaluation of conditional sequential generation is evaluated using a brck-translated model. Extensive experiments show that our model increases the WER score from 82.01 % (5) to 78.21% on generated pose sequence to gloss sequence, and BLEU score from
73
75
6.66 5 to 7.42 on generated pose sequence to spoken language.
2
Related Works
76
77
Sign Language Production. Most sign language works focus on sign language recognition (SLR)
22 23 aiming to translate the video-based sign language 79 into text-based sequences. And few attempts have been made for the more challenging task of sign
78 and translation (SLT) [18] 19
20
21
80 language production (SLP) [24 25 Stoll et al. proposed the first deep SLP model, which adopts the
81 threc-step pipeline. In the core process for G2P, they learn the mapping between the sign glosses and
82 the skeleton poses via a look-up table. After that, B. Saunders et al. [3 proposed the progressive
83 transformer to learn the mapping with an encoder-decoder architecture and generate the sign pose
84 in an autoregressive manner in the inference. Further, B. Saunders et al. [ proposed a Mixture
85 Density Network (MDN) to generate the pose sequences condition on the sign glosses and utilize a
86 GAN-based method 266 to produce the photo-realistic sign language video. B. Saunders et al. [27
87 translates the spoken language to sign language representation with an autoregressive Transformer network, and uses the gloss information to provide the additional supervision. Then they propose
2

---

Points Embs,
Transformer
Trensformel
Linear
800
134
Encoder
Decoder
Figure 1: The architecture of the first stage model Pose-VQVAE for learning the discrete latent codes.
89
Mixture of Motion (MoM)) architecture to combine distinct motion primitives to produce a continuous sign language sequence.
90
91
Different from these autoregressive methods, Huang et al. [5] proposed a non-autoregressive model to parallelly generate the sign pose sequence avoiding the error accumulation problem. They apply the monotonic alignment search 28 to generate the alignment lengths of each gloss. Our model also explores autoregressive method with diffusion strategy, and the adopted diffusion model architecture provides us with a chance to refine the results with multiple iterations.
90
95
96
Denoising Diffusion Probabilistic Models. Diffusion generative models have achieved outstanding results on continuous data, such image generation [29 30 32 331 and speech synthesis 34 35
97
98
36. However, most previous works focus on Gaussian diffusion processes that operate in continuous state spaces. The discrete diffusion model first introduced in [37]. and it is applied to text generation in Argmax Flow [0]. To improve and extend the discrete diffusion model, D3PM [1]] use structured categorical corruption process to shape data generation and embed structure the forward process. VQ- Difffuusionn [12] apply the discrete diffusion model to conditional vector quantized image synthesis with a baosk--ad-reppaaa diffusion strategy.
99
101
103
3
The Proposed Method
104
The overall objective of this work is to extend the discrete diffusion model for conditional sign pose sequence generation. The proposed VQ-Diffuuno model consists of three key components, Pose- VQVAE learn the latent codes, discrete diffusion model with CodeUnet to model the discrete codes generation, and a aequential-KNN algorithm on the length prediction for this non-autoregressive method.
106
108
110
3.1
Pose VQ-VAE
111
In this section, we introduce how to tokenize the points of sign pose skeleton into a of discrete tokens. A naive approach is to treat per point as one token. However, such points-wise reconstruction model tends to tremendous computational cost due to the quadratic complexity of self-attention 114 in Transformers. On the other hand, since the details of hand points are essential for sign pose understanding, treating all the points into one token leads to remarkable inferior reconstruction performance. To achieve better trade-off between quality and speed, propose simple yet efficient implementation that groups the points of a sign skeleton into three local patches, representing pose, right hand and left hand separately. Figure F iillustrates the framework of our proposed Pose- VQVAE model with the following submodules.
115
115
117
119
Encoder. Given sign pose sequence of N frames s = (Si, .2..... Sn,..., SN) E NxJKK where (x/ presents a single sign skeleton containing joints and denotes the feature dimension
120
121
for human joint data. We separate these points into three local pathes, Sp E ( K) s, R. x(), 8) E R ((jxx) for pose, right hand and left hand respectively, where J = Jp +.+
122
124 In the encoder module E(ess), we first transform these three points sequences into feature sequences by simple three linear layers and concatenate them together. Then we apply spatial-temporal Transformer network to learn the long-range interactions within the sequential point features. Finally,
126
arrive at the encoded features {en E 3x3x B3
In=1"
128
Point Tokenizer. Similar image VQ- 14, we take the encoded features as inputs and convert them into discrete tokens. Specifically, we perform the nearest neighbors method Q(2le) to quantize
129
the point feature to the quantized features (2n N - The quantized features are maintained
130
by codebook whose size is V.

---

16
18
25
18
90
and add P6 (zL (2,,c
CodeUnot
Longth Prodictor
Transfomme Encoder
Transtormer
Transfomar Encoder Decodar
besed Encoder
and sattmax
Sequento: MAL
Figure 2: Our approach uses a discrete diffusion model to represent the Vector-Quantiann sign pose sequence allowing son-autoregressive pose sequence generation. Specifically, after compressing the sign pose sequences to meaningful discrete codes, each code randomly masked or replaced and a Unet model is trained to restore the original data.
Decoder. The decoder (( receives the quantized features as inputs and also applies spatial- temporal Transformer to get the output features for, E R h N
133
I==1 Finally, we separate the output feature for three sub-skeleton and utilize structured prediction layer (SPL) [88 P(š ) to reconstruct the corresponding sub-skeleton 5p Nx,, xk), š IR (X,,X s,- E (J..XK))) We adopt the SPL. to rebuild the skeleton from feature because it explicitly model the spatial structure of the human skeleton and the spatial dependencies between joints. The hierarchy chains of the pose, right hand and left hand skeleton are given in supplemental material.
136
138
Training. The encoder E(els), tokenizer @==le) and decoder D(5 can be trained end-to-end via following loss function:
140
CPose-VOVAE ||sp اpl1 + ||s, 3 + ||s:
s:/l ||sg[e] 211 + |||||[[[] ell.
(1)
141
where sg[:] stands for stop-gradient operation. In practice, we replace the forth term with exponential moving averages (EMA) update the codebook.
142
3.2
Discrete Diffusion Model with CodeUnet
144
To allow conditional sampling, a discrete diffusion model is trained on the latent codes obtained from the Pose- VOVAE model. Figure shows the architecture of our proposed Pose VQ-Diifusiion, which aims to model the latent space in non-autoregressive manner. We will subsequently introduce the diffusion process, reverse denoising process and the parametered model CodeUnet.
146
Diffusion Process. Given a sequence of latent codes xo E Nx3 obtained from the vector quantized
148
model, where (f-j) E {1, 2, ..., }} at location (i,j) represents the index within the codebook. The diffusion process aims to corrupt the original data Zo
149
150
fixed Markov chain p(z, ( 1) by adding small amount noise continuously. After fixed T timesteps, it produces a sequence of increasingly noisy data I,, 2r with the same dimensions as 2o, and IT becomes pure noise sample.
152
For the scalar discrete variables with V categories x (1.5) [1, V]. the forward transition probabilities from -t-1 2, be represented by matrices [Qe]ma q m|#;-1 n) R. VxV Note that we omit the superscripts (f, j) to avoid confusion. Then the forward diffusion process can be written
153
155
=x,
157 where E V is the one-hot version of 2; and Qxx²-1 is the categorical distribution for Ty. A 158 nice property of the above Markov diffusion process is that we can sample x, as any timestep directly from Zp as:
I
g(z1/20) = xx 2ex0 with Q. Qi ....Qu Q1.
(3)

---

D3PM [1]] formulate the transition matrix Qi E RVVV by introducing small number of uniform
160
noises the categorical distribution. As formulated as the first matrix in Eq. with [0. and
161
162
8. (1 -1 oz)) ov./V. V. It can be interpreted as each token having probability of ar + 33 to remain the
163
previous value and a probability of 3, to be the value from the whole V categories. Based on D3PM, diffusion strategy that not only replaces the previous value but also insert [MASK] token to explicitly figure out the tokens that have been replaced. We extend this mask-and-replace strategy
164
VQ- D--Diffusion 12 propose mask-and-replle
165
quantized pose sequence modelling. Since the length of pose sequences may different in a minibatch, we have to add two special tokens, [MASK] and [PA]] tokens, each token has V + states. The mask-and-reyy anplaae diffusion process be defined as follows: each token has a probability of 0g to be unchanged, V V3, to be uniformly resampled and Tt = 02 3, to be replaced with [MASK] token. Note that [MASK] and [PAD] tokens always keep its own state. The difference is that [PAD] is used to represent the padding part in the initial sequence To. and [MASK] is used to replace the original token of the code sequence in the diffusion process. Moreover, in the revised denoising process, the [MASK] positions are required to predict
167
189
171
173
but [PAD] positions need to ignored. The transition matrix Qt E R[(++)x(V+++ is formulated as the second matrix of the following:
174
[ax + 3
Be
0
On + 3t
B
9,
32
a1 +
B1
Be
8g
02
+
Be
3e
Q:-
:Q:
(4)
: Be
81
Be
O1 + 52
8
Ox + 3
7
7/8
7/+
0
0
0
1
178
Finally, the categorical distribution of x, can be derived as following using reparamcterizzttoon trick:
O4 + 3r,
Tt = 3
B1,
It F zg and 24 S V
when vo VV2 Qixo-
7/1
It = V +1
(5)
0,
Ft V + 14 + V +2
00
when To = +2 ¹ix0 =
1,
It = V +2
where de = IT or, 7 7 7)) and 33 (1 - at - -t//V. Therefore, we can directly
177
sample 2, within the computation cost O(V).
179
Reverse Denoising Process. The reverse denoising process aims recreate the real sample from
180
full noise input by gradually sampling from (( However, it is intractable to estimate the conditional probability ((=,)1/2;) since it needs use the whole dataset. Fortunately, the conditional
181
182
probability is tractable when conditioned on using Bayes' rule:
(x] x.TQux._ ()(x 1101 x 21x00
((11-1/2,,00)
q(z; xo)
thus we train denoising model 1 c) to approximate the tractable distribution 184 O(Iz-1/<0, zo), where is the conditional feature of gloss sequence. And the model is trained
183
185 minimize the variational lower bound (37]
Lub =F E. DKL. (g(#T(20) | Po (x(T))
Dra
ll po (are-1l2ee 1 c))
Lr
-log pa (2)
Lo
186 Reparameterization Trick on Model Learning. Compared with directly predicting po(x,-1 - l c). recent works [30] [0.12 [2] find that predict the data zo gives better quality at every reverse step. Thus,
5

---

188 we let our denoising model to predict the distribution pe (o c). With reparameterization trick, the conditional reverse distribution can be formulated as:
189
V
pe(1e-1/04 c)
Σ ((#1-11I4, To To 5o=1
(8)
190
Under this oo-paaaamtteizztioonn trick, we introduce an auxiliary denoising objective to encourage good predictions of the data To at each time step [II]. The final loss function is combined with the negative variational lower bound and the auxiliary loss:
192
Cutm Lub logpo c),
(9)
193
where À is coefficient for the auxiliary loss term.
194
CodeUnet for Model Learning. Most image diffusion models [29 30 39 adopt the Unet [3] as
their architectures since it is effective for data with spatial structure. However, directly applying the Unet in discrete sequence generation, e.g., text generation [1]] and quantized image synthesis [2] will bring information leakage problem, since the convolution layer over adjacent tokens may provide
196
198
shortcuts for the mask-based prediction [40]. Therefore, Austin et al. [1]] and Gu el al [12 use the wise Transformer framework learn the distribution pel (āo (T, c). In this work, to incorporate the advantages of Unet and Transformer networks, we propose a novel architecture CodeUnet to
199
200
201
learn the spatial-temporal interaction for our quantized pose sequence generation.
202 As shown in Figure the CodeUnet consists of a contracting path (left side), an expansive path (right side) and middle module. The middle module is encoder-decoder Transformer framework. The encoder consists of 6 Transformer blocks. It takes the gloss sentence as input and obtains a conditional feature sequence. The decoder has two blocks. Each block has self-attention, cross-attention, forward network and an Adaptive Layer Normalization [41 12 (AdaIN). The AdaLN operator is devised to incorporate timestep information AdaL t) LyLayerNorm(hh (h) + b,, where h is the intermediate activations, and 3r are obtained from a linear projection of the timestep embedding.
204
206
208
209
210
Both contracting path and the expansive path are hierarchical structures and each level has two Transformer encoder blocks. For downsampling in contracting path, given the feature of quantized pose sequence, e.g., h E R x3xxdaa where dmodel is the feature dimension, we first sample uniformly with stride 2 in the temporal dimension and remain constant in the spatial dimension. Then set the downsampled feature query Q E R N/2x3x dasde, and keep key K and value V unchanged for the following attention network. In the upsampling of expansive path, we directly repeat the feature times query, but the key and value remains for the following attention network:
212
215
Vn 1, ...,, N, Q. = hw//2, KuP
Vup = h,
(10)
217
where ///- denotes floor division. Finally, linear layer and softmax layer are applied make the prediction.
218
219
3.3
Length Prediction with Sequential-Ke
220
In this section, inspired by 17 which merges tokens with similar semantic meanings from different locations, propose novel clustering algorithm to get the lengths for corresponding glosses.
222
Specifically, given a token sequence which is obtained from the Pose- VQVAE model, we compute the local density p of each token according its k-nearest-neighbors:
223
P. =eppp- -
Σ 2/ EKNN(=.)
- 22,1),12) where /i - <=
<=1
(11)
where i is the position in the sequence, I is predefined hyperparameter indicating that we only consider the local region since the adjacent tokens are more likely to belong to a gloss. 2; and 2j are
225
226 the latent feature for 4th and jth tokens. KNN(x;) represents the k-nearest neighbors for 2th token.
We assign (PI,...PMs) positions with higher local density as the peaks, where M is the length of the gloss sequence. Then between two adjacent peaks, for example P1 and p2, we sequentially iterate from Pi to p2. and find the first position that is farther from 2p1 and closer to 2p2. which is
228

---

230
the boundary we determined. After finding these boundaries, we get the lengths of the contiguous pose sequence for its corresponding glosses. As shown in Figure we define the obtained lengths as [I,,.., T.ar) }. and the Transformer encoder for gloss sequence is trained under the supervised
231
information of lengths. For each gloss word, we predict number from [1, Pl, where P is the
233
maximum length of the target pose sequence. Mathematically, we formulate the classification loss of
235
length prediction as:
M P M ()
Len
(12)
236
In the training of the discrete diffusion mode, Lien is trained together with a coefficient 6. In the inference, we predict the length of glosses, and their summation is the length of target pose sequence.
237
238 In summary, we arrive our proposed two-stage approach, PoseVO-Diffusion, with the first-stage Pose-Vor VQVAE model and the second-stage stage discrete diffusion model with length predictor. The whole training and inference algorithm is shown in supplementary material.
240
Experiments
241
4.1
Experiment Setups and Implementation Details
Dataset. We evaluate G2P model on SWTH-PHOONIX-Weate -Wather 2014T (RPWT) dataset [18]. It is the only publicly available SLP dataset with parallel sign language videos, gloss annotations, and spoken language translations. This corpus contains 7096 training samples (with 1066 different sign glosses in gloss annotations and 2887 words in German spoken language translations), 519 validation samples, and 642 test samples.
244
245
246
247
248
Evaluation Criteria. Following the widely-used setting in SLP 3. we adopt the back trac-stranlttiin method for evaluation. Specifically, we utilize the state-of-the-an art SLT [19] model to translate the generated sign pose sequence back to gloss sequence and spoken language, where its input is modified as pose sequence. Specifically, compute BLEU [22 and Word Error Rate (WER) between the back translated spoken language translations and gloss recognition results with ground truth spoken language and gloss sequence.
250
252
253
254
Data Processing. Since the RWTH-PHOENIX- Weather 2014T (RPWT) dataset doesn't contain pose information, generate the pose sequence the ground truth. Following B. Saunders et al. (3), we extract 2D joint points from sign video using OpenPose (43) and lift the 2D joints to 3D with skeletal model estimation improvement method [44]. Finally, similar to [24), we apply skeleton normalization to remove the skeleton size difference between different signers.
256
258
259
Model Setting. The Pose VQVAE consists of an Encoder, Tokenizer, and a Decoder. The Encoder contains linear layer to transform pose points to hidden feature with dimension set as 256, a 3-layer Transformer module with divided space-time attention 455. The Tokenizer maintains a codebook with size set as 2048. The Decoder contains the same 3-layer Transformer module as encoder and an SPL. layer to predict the structural sign skeleton. For the discrete diffusion model, we set the timestep as 100. All Transformer blocks of CodeUnet have 5m2 and Nocph= =2. The size of local region / in Equation]] II] is set as 16 which is the average length of a gloss. And the number of nearest neighbors & is set as 16. We optimize our network using AdamW [46) with 0.9and Bg = 0.96. The learning rate is set to 0.0004 after 5000 iterations of warmup. We train the model on NVIDIA Tesla V100 GPUs. We include all hyperparameters setting and the details of implementation in the supplementary material.
260
252
264
286
31
268
270
4.2
Comparisons with State-of-the-Art Methods
271
Competing Methods. We compare our ToseVQ-Diffusion with previous state-of-the-ar G2P models. Progressive Transformer (PTR) 3 the first SLP model to tackle the G2P problem in an 272
autoregressive manner. Since they use the ground-truth first sign pose frame and timing information, their reported results are not comparable to ours. Thus we adopt the results reported by Huang et al. [5). NAT-EA [5] propose non-autoregressive method to directly predict the target pose sequence
274
7

---

Table 1: Quantitative results for G2P task on PHONNI- Weather 2014T test dataset. indicates the results is provided by Huang et al. [5]. Note that smaller WER is better, higher BLEU is better, and lower DTW-MJE is better. The closer all the results are to the GT, the better.
Method
WER
BLEU-1
BLEU-2
BLEU-3
BLEU-4
DTW-MJE
PTR|
3
94.65
11,45
7.08
5.08
4.04
0.191
MoMP
27
92.41
13.17
8.24
6.25
4.75
0.188
NAT-AT 5
88.15
14.26
9.93
7.11
5.53
0.177
NAT-EA 5
82,01
13.12
10,45
7.99
6,66
0.146
Pose VQ-AR (Ours)
85,27
14.26
10.02
7.57
5,94
0.172
PQ-M (Ours)
79.38
15,43
10.69
8.26
6.98
0,146
Pose Do-iff--iiinsion (Ours)
78.21
16.03
11.32
9.17
7.42
0.122
GT
50.23
23.47
15.86
12.03
10.47
0.0
278
with the External Aligner (EA) learn alignments between glosses and pose sequence. NAT-AT is the NAT model without EA that uses the decoder-to-encoder attention to learn the alignments.
277
Quantitative Comparison. The comparison between Pose PoseVO-Diffusion and the competing The row of Pose VQ-AR refs to the vector quantized model with an autoregressive decoder. The row of PoseVQ-ME refs to the vector quantized model with the Mlas--Predict [9] strategy, which is also a variant of discrete diffusion model [1]. PoseVQ-Ile VQ-DDifusioon refs to the vector quantized model with mask-and-reelace diffusion strategy. As indicated in Table [ both diffusion-
279
methods is shown in Tabel [
281
283
models outperform the state-of-the-ee G2P models with relative improvements
284
on WER score by 4.6% (82.01 - 78.21) and on BLEU-4 by 11.4% (6.66 7.42). This shows
285
the effectiveness of the iterative mask-based non-autoregressive method on the vector quantized strategy is a mask only strategy that similar to Pose VQ-Diffusion with 2/2 = 1. Therefore, PoseVQ-Diffun achieves better performance than Pose PoseVQ-Mi VQ-MP. This reflects the mask-and renllce strategy is superior to the mask only strategy.
286
pose sequence. In addition, the
287
288
289
4.3
Model Analysis and Discussions
290
We also investigate the effects of different components and design choices of our proposed model.
Analysis of The Design of Pose-VQVARE VQVAE. As shown in the first three rows of Table 2a study the design of our Pose-VQV VQVAE model. Pose- Pose-VQVAE-jee VQVAE-joing means we compress all points into one token, Pose-Vos Veee--VVV-EE-eeeeeeee
292
means the points separated into three local patches according the structure of a sign skeleton. Empirically, Pose
294
achieves much better reconstruction (MSE) performance. This indicates that compressing all skeleton points into one token embedding is not advisable, leading to information loss. The second row of Figure 3 shows the sample of sign pose sequences reconstructed by Pose Dose-VQVAAE-sppaa
285
296
287
296
CodeUnet vs. Transformer. For fair comparison, we replace our CodeUnet with Transformer network with keeping other settings the same. As shown in the last three rows of Table 2a the diffusion-basee model with our CodeUnet achieves better performance on the back-translate evalua-
300
tion. This phenomenon suggests that the hierarchical structure of CodeUnet makes it particularly effective for data with spatial structure. Moreover, in our experiments with the same batch size, Unet coverages faster than Transformer. Having said that, due sign pose sequences being temporally redundant, the compression of CodeUnet in the time dimension makes it more efficient in training.
302
304
305
306
Number of Timesteps. We compare the performance of the model with different numbers of training steps. As shown in the left two columns of Table 2b we find that the results get better when the training step size is increased from 10 to 100. As it increased further, the results seemed to saturate. Therefore, we set the training step to 100 to trade off performance and speed.
308
310
Length Candidates. Length prediction is essential for non-autoregressive generation. Our approach proposes Sequential-KNN KNN algorithm to learn the lengths for corresponding glosses and then treat the length prediction as a classification problem. As shown in the right two columns in Table 2b
312
we study the performance with different length candidates and compare it with the reference (gold) target sequence length. The results show that multiple candidates can increase performance, too many candidates can even degrade performance.
314
8

---

Sign BESONDERS
GT
Pase-VQVAE
A
PaseVQ-
Diffusion
PTR
A
Sign Gloss: DONNERSTAG SUEDOST WEITER WECHSELHAFT NORDWEST MEHR FREUNDLICH SONNE
Pose-VQVAE
PoseVQ-
Diffusion
PTR
Figure 3: G2P qualitative results. We show some examples of predicted sign pose sequences compared with our reconstruction model and previous G2P model [3]. For readability, we sampled every frames for total of 16 frames. See our supplementary material for more results.
Table 2: Analysis into the effects of different designs and hyperparameters for our propose model.
(a) Ablation on design of reconstruction and pre- diction model.
Ablation hyperparameters training steps and length candidates.
Reconstruction Model
MSE (L)
Training Steps
Length Candidates
WER (L)
WER (1)
Pose-VQU Dnse-VQVAE-joinl
0.0242
10
81.06
79.45
Pose- Posc-VQVABB Leoo-VVVAE--ppraaa
0.0139
50
79.31
2
78.69
Prediction Model
WER (1)
100
78.21
78.21
Transformer
80.36
150
78.17
4
78.74
CodeUnet
78.21
200
78.15
Gold
77.26
5
Conclusion
317
In this paper, we presented novel paradigm for conditional sign pose sequence generation through an iterative non-autoregressivee method. Specifically, we first devise specific architecture Pose-VQVAR learn discrete codes by reconstruction. Then we extend the discrete diffusion model model the sequential alignments between sign glosses and quantized codes. And "fully transformer" network CodeUnet proposed for the spatial-temporal information in discrete space. Finally, we propose a sequential-KNN KNN algorithm to learn the length of corresponding glosses and then predict the length as a classification task. Compared with previous state-of-the autoregressive and non-autoregressive methods, extensive experiments demonstrate the effectiveness of our proposed PoseVQ-Diffune
319
321
323
325
framework.
Broader Impact and Limitations
We develop a general paradigm for conditional pose generation in this paper. We do not foresee any negative ethical/societal impacts at this moment. Although the proposed Pose PoseVO-Diffusio proves effective conditional sign pose sequence generation, we notice several limitations of our approach.
328
329
(i) Although our discrete diffusion model-bases model has faster sampling speed in longer sequence generation than traditional autoregressive models, it is much slower than one-shot non-autoregressive
331
models. (ii) Our proposed two-stage based models are not end-to-end and thus more difficult to train
333
than previous methods. We, therefore, plan to resolve the aforementioned issues and further mitigate the training and inference speed between the one-shot non-autoregressive method.

---

References
335
336
[1] Roland Pfau, Martin Salzmann, and Markus Steinbach. The syntax of sign language agreement: Common ingredients, but unusual recipe. Glossa: journal of general linguisties, 2018.
337
338
[2] Ben Saunders, Richard Bowden, and Necati Cihan Camgöz. Adversarial training for multi- channel sign language production. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.
340
341
[3] Ben Saunders, Necati Cihan Camgöz, and R. Bowden. Progressive transformers for end-to-end sign language production. ArXiv preprint, 000/2004.187744 2020.
342
343
[4] Ben Saunders, Necati Cihan Camgoz, and R. Bowden. Everybody sign now: Translating spoken language to photo realistic sign language video. ArXiv preprint, abs/201 1.09846, 2020.
344
345
[5] Wencan Huang, Wenwen Pan, Zhou Zhao, and Qi Tian. Towards fast and high-quality sign language production. Proceedings of the 29th ACM International Conference Multimedia, 2021.
347
348
[6] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation. 1:270-280, 1989.
349
350
77]
Florian Schmidt, Generalization in generation: A closer look at exposure bias. In Proceedings of the 3rd Workshop Neural Generation and Translation, pages 157-167, Hong Kong, 2019. Association for Computational Linguistics.
352
[8] Jiatao Gu. James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. In 6th International Conference on Learning Rep- resentations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3. 2018, Conference Track Proceedings. OpenReview.net, 2018.
354
356
[9] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zeutlemyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, Hong Kong, China, 2019. Association for Computational Linguistics.
358
380
382
[10] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forr'e, e, and Max Welling, Argmax flows and multinomial diffusion: Towards -on-autoregressive language models. ArXiv preprint, abs/2102.05379, 2021.
383
364
365
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurlPS, 2021.
366
367
[12] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. ArXiv preprint, 2021.
389
370
[13] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 12868-12878. 2021.
371
372
373
[14] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6306-6315. 2017.
374
375
378
377
378
[15] Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
379
380
[16] Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowl. Based Syst., 99:135-1453 2016.
10

---

382
[17] Wang Zeng. Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Ouyang Wanli, and Xiaogang Wang. Not all tokens are equal: Human-centri visual analysis via token clustering transformer. ArXiv preprint, (es/22044086880 2022.
383
[18] Necati Cihan Camgöz. Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7784-7793. IEEE Computer Society, 2018
386
388
389
[19] Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10020-10030. IEEE, 2020.
390
391
382
393
[20] Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and R. Bowden. Multi-channel transform- ers for multi-articulatoon sign language translation. ArXiv preprint, abs/2009.00299 2020.
395
[21]
Hao Zhou, Wen gang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedia, 24:768-779, 2022.
396
397
398
[22] Pan Xie, Mengyi Zhao, and Xiaohui Hu. Pisltre: Position-informed sign language transformer with content-aware convolution. ArXiv preprint, abs/2107. 2021.
[23] Hezhen Hu, Weichao Zhao, Wen gang Zhou, Yuechen Wang, and Houqiang Li. Signbert: Pre- training of -and-ee representation for sign language recognition. 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pages 1067-11076, 2021.
401
402
403
[24] Stephanic Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden. Sign language production using neural machine translation and generative adversarial networks. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 304.
404
405
406
BMVA Press, 2018
[25]
Qinkun Xiao, Minying Qin, and Yuting Yin. Skeleton- based chinese sign language recognition and generation for bidirectional communication between deaf and hearing people. Neural networks: the official journal of the International Neural Nenwork Sociery, 125:41-55, 2020.
408
409
410
[26] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A. Efros. Everybody dance now. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- - November 2, 2019, pages 5932-5941 5941. IEEE, 2019.
411
412
413
[27] Ben Saunders, Necati Cihan Camgöz, and Richard Bowden. Mixed signals: Sign language production via mixture of motion primitives. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 1899-1909. IEEE, 2021.
415
417
[28]
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-Its: A generative flow for text-to-speech via monotonic alignment search. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Fiorina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020
418
419
420
421
422
[29] Prafulla Dhariwal and Alex Nichol. Diffuusion models beat gans on image synthesis. In NeurIPS, 2021.
424
[30]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo I Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information P Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
425
426
427
428
[31] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23:47:1-47:33, 2022.
429
11

---

431
[32] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMIR, 2021.
432
434
(33] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. ArXiv preprint, abs/2112.10752,
436
2021.
438
[34] Myconghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. In Interspeech, 2021.
440
[35] Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: diffusion model for text-to- speech via classifier guidance. 2021.
442
[36] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
443
444
445
[37] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 2256 2265. JMLR.org, 2015.
446
447
448
449
[38] Emre Aksan, Manuel Kaufmann, and Otmar Hilliges. Structured prediction helps 3d human motion modelling. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 71 7143-7152. IEEE, 2019.
451
453
[39] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May
455
3-7, 2021. OpenReview.net, 2021.
457
[40] Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. ArXiv preprint, abs/2110.13311,, 2021.
459
[41] Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv preprint, abs/ 011166670645500 2016.
461
462
[42] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics.
463
464
485
466
[43] Zhe Cao, Gines Hidalgo. Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: Realtime multi- 2d pose estimation using part affinity fields. JEEE Transactions on Pattern Analysis and Machine Intelligence, 43: 13:172-886. 2021.
467
468
[44] Jan Zelinka and Jakub Kanis. Neural sign language synthesis: Words are our glosses. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 3384-3392, 2020.
470
[45] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 813- -224. PMLR, 2021.
472
474
475
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OvenReview.net, 2019.
476
12

---

Checklist
478
479
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
480
481
Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
483
Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6
484
Did you describe the limitations your work? [Yes] See Section
485
Did you state the full set of assumptions of all theoretical results? [Yes]
486
Did you include complete proofs of all theoretical results?
[Yes] More proofs are in
supplemental material.
487
488
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material as URL)? [Yes] The code and instructions are in the the supplemental material
490
491
Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] More training details are in the supplemental material
493
Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [No]
494
Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
496
497
If your work uses existing assets, did you cite the creators? [Yes]
498
Did you mention the license of the assets? [No]
Did you include any assets either in the supplemental material URL? [No]
Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
501
502
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
503
504
Did you include the full text of instructions given to participants and screenshots, if applica- ble? [No]
505
Did you describe any potential participant risks, with links to Institutional Review Board
507
(IRB) approvals, if applicable? [No]
508
Did you describe any potential participant risks, with links to Institutional Review Board
509
(IRB) approvals, if applicable? [Yes]
13


Reviews:
[{"review_id": "naqm7MCeKED", "reviewer": "Reviewer_Ljyz", "summary": "This paper tackles the problem of Sign Language Production (SLP), which aims to translate spoken languages into sign pose sequences. The proposed pipeline is based on a Vector Quantized Diffusion model.  The key contribution consists of separating a sign skeleton into three local point patches representing pose, right hand, and left hand separately. An extension of standard VQ diffusion method by considering mask token and a new architecture CodeUnet as well as lengths prediction of sign glosses. \n\nThe proposed approach demonstrates its effectiveness on RWTH-PHOENIX-WEATHER-2014T.\n", "questions": "Please refer to the weaknesses. ", "limitations": "No potential negative societal impact. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n \n* [S1]: The paper is well organized, and the approach is well presented and self-contained.\n\n* [S2]: The improvement on RWTH-PHOENIX-WEATHER-2014T is consistent over different metrics.\n\n**Weakness**\n\n* [W1]: The main weakness is the novelty. The core idea of this paper, _i.e._ VQ Diffusion, has been demonstrated to be successful in many generation tasks. Thus it is not surprising that it works on SLP. Learning different codebooks for different groups of joints is not novel as well, which has been used in music to dance generation [a]. \n\n* [W2]: The experimental part is a little bit weak, as only one dataset is considered. Although the paper mentioned there is only one dataset publicly available.\n\n* [W3]: In the related work, the paper mentions [4, 26] as works to generate pose sequences from glosses. I am wondering whether there is a particular reason to make a fair comparison to these approaches. \n\n* [W4]: In the case of using 3 codebooks (pose, right hand, and left hand), the reconstruction becomes simpler however the generation might be harder as well. I think it would be more convincing to validate the idea of separating codebooks on generation rather than reconstruction. \n\n**Minor Questions towards Clarification**\n \n* [C1]: Training details are missing, such as optimizer, weight decay, learning rate strategy etc. Also, it is not clear whether the paper leverages exponential moving average to train the codebook, which is a common practice to train VQ-VAE.\n\n* [C2]: Figure 2: Leangth Predictor → Length Predictor\n\n_[a]: Siyao, Li, et al. \"Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022._", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cDyM7jT5tCK", "reviewer": "Reviewer_ndEY", "summary": "The authors propose a novel text-to-sign pose sequences generation method. The proposed method is based on the powerful discrete denoising diffusion probabilistic model. It solves the long-range autoregressive generation problem in a non-autoregressive manner. A vector\nquantized conditional diffusion model is proposed for the pose generation. The proposed method achieves state-of-the-art results on the SLP evaluation benchmark.", "questions": "1. Have the authors tried to use some other models to replace the CodeUnet? Leveraging transformer-based blocks and Unet is interesting. But how well does the network work compared with other widely used networks?\n2. Why is the PAD state used and what is the meaning? It seems using MASK state is enough. And this point is not illustrated in the paper. From the transition matrix, no other state will turn to be PAD and PAD is an isolated state. Then why do we still need to talk about whether\n$x_t$ equals V+2?\n3. Do you have some other experimental results and ablation studies? It seems experiments should be done to verify the effectiveness of many parts of the work, including the architecture, section 3.3, etc.", "limitations": "For the checklist, the 6th and 7th \"YES\" may be misleading.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The proposed method uses a novel diffusion-based model and achieves great performance.\n2. The authors argue they solve the problem in an auto-regressive way, which may be meaningful in the relative fields.\n3. A Pose-VQVAE is used to learn the meaningful codebook and the discrete diffusion model is devised in a fancy way.\n4. The CodeUnet is proposed in the diffusion model for better leveraging the spatial and temporal information of the quantized pose sequences.\n\n# Weaknesses\n## Main Issue\n1. This work is based on the works of some diffusion-based models, especially the CVPR 2022 paper Vector quantized diffusion model for text-to-image synthesis [1]. My biggest concern is that we can not tell how big the contribution of this work is. On the one hand, this work\nintroduces the VQ-Diffusion method proposed in a text-to-image generation to the G2P task and achieves great performance, which is impressive. On the other hand, compared with the former CVPR work, the method in this paper uses the CodeUnet, which is not a big step, from my perspective, and only proposes some small points, like dividing the joints into 3 subsets. Therefore, it is really hard to judge how big the contribution is.\n\n## Some Minor Issues\n1. A typo in line 107.\n2. For the discrete diffusion model part, why there is a PAD state? I think the meaning of the existence of PAD is not illustrated clearly in the section.\n3. The experiments part may not be sufficient enough. For example, the method tries to divide the joints into 3 subsets. What will the results be like if we use a simpler implementation?\n4. In table 1 in the results, how to evaluate the numerical results? How to define better results should be discussed in the paper. Higher or smaller? Or closer to the GT?\n5. In the checklist, the authors say that the code is provided. However, this is missing in the supplementary materials.\n\n[1] Vector quantized diffusion model for text-to-image synthesis, Gu et al., CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JmwYq49RVt4", "reviewer": "Reviewer_vAR2", "summary": "The paper is dedicated to the sign language production problem. In the paper, based on VQ-Diffusion, the authors propose the PoseVQ-Diffusion model to generate the pose sequence with iterative refinement. The proposed method achieves encouraging performance on PHOENIX dataset.", "questions": "1. In Line 16 – 17, the authors claim that the proposed method has a faster sampling speed compared with autoregressive models. However, there is no experimental results to support this argument. As a matter of fact, diffusion models usually have more model parameters and longer training & sampling times. \n2. In Line 164 – 165, the summation of the probability to be unchanged, to be uniformly resampled and to be replaced with [MASK] token is not equal to 1.\n3. The authors do not present the qualitative comparison with NAT-AT and NAT-EA.\n", "limitations": "I include several constructive suggestions for improvement.\n1. The authors should provide the comparison with the missing methods mentioned above.\n2. The authors should compare the model parameter and training & sampling time with previous methods.\n3. Other types of evaluation, such as user study, will further verify the effectiveness of the proposed method.\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper leverages the VQ-diffusion model to sign language production, which is novel and interesting. The diffusion model is considered to be potential on generative problems. The attempt made in the paper is very meaningful. Meanwhile, the authors propose several additional improvements for VQ-diffusion on SLP, which is also good.\n\nWeakness:\nDespite the novel framework, there are some non-trivial concerns on the effectiveness evaluation of the proposed framework.\n\nFirst, the authors only compare the proposed method with two previous methods, and the latest method compared is NAT-AT/EA from ACM MM 2021. Several important baseline methods, i.e., [1] and [2], are missing.\n[1] Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives. Ben Saunders, et. al. ICCV 2021.\n[2] Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production. Ben Saunders, et. al. CVPR 2022.\n\nSecondly, the authors claim that the proposed method has a faster sampling speed compared with autoregressive models. However, there is no experimental results to support this argument.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject