T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text (2024)

Abstract: In this work, we propose a two-stage sign language production (SLP) paradigm that first encodes sign language sequences into discrete codes and then autoregressively generates sign language from text based on the learned codebook. However, existing vector quantization (VQ) methods are fixed-length encodings, overlooking the uneven information density in sign language, which leads to under-encoding of important regions and over-encoding of unimportant regions. To address this issue, we propose a novel dynamic vector quantization (DVA-VAE) model that can dynamically adjust the encoding length based on the information density in sign language to achieve accurate and compact encoding. Then, a GPTlike model learns to generate code sequences and their corresponding durations from spoken language text. Extensive experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. To promote sign language research, we propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts. Experimental analysis on PHOENIXNews shows that the performance of our model can be further improved by increasing the size of the training data. Our project homepage is https://t2sgpt-demo.yinaoxiong.cn.

heading: 1 Introduction
text: Sign language is a visual language with complex grammatical structures and is the primary means of communication for nearly 70 million deaf people worldwide 1. Research on sign language production (Baltatzis et al., 2023a; Fang et al., 2023; Huang et al., 2021; Hwang et al., 2021, 2022; Saunders et al., 2020a) and sign language translation (Camgoz et al., 2018a; Zhang et al., 2023a;
*Corresponding author. 1According to World Federation of the Deaf https://
wfdeaf.org/our-work/
Zhou et al., 2021; Yin et al., 2021, 2023) has attracted widespread attention. Sign language production (SLP) is a challenging problem that aims to automatically translate spoken language descriptions into corresponding continuous sign sequences. SLP can help deaf people better access information and communicate with others, thereby facilitating their lives, which has important social significance.
SLP models are expected to learn precise mapping from the spoken language space to the sign language space. Early work used 2D or 3D skeleton poses to represent sign language (Huang et al., 2021; Saunders et al., 2021b, 2020b), while recent work has suggested using 3D human models, such as SMPL-x(Pavlakos et al., 2019), to represent sign language, as it introduces human priors and can better animate (Baltatzis et al., 2023b). To learn the mapping between these two different modal spaces, some work uses autoregressive models (Saunders et al., 2021a,b, 2020b), non-autoregressive models (Huang et al., 2021; Hwang et al., 2021, 2022), or diffusion models (Baltatzis et al., 2023b) to learn the direct mapping from spoken language text to sign language skeleton poses. (Xie et al., 2023) proposed to learn the discrete representation of sign language through VQ-VAE (van den Oord et al., 2017) and then learn the mapping from text to discrete representation through a discrete diffusion model. However, we found that existing sign language discrete representation methods are fixed-
3345
length encodings, as shown in Figure 1, which overlooks the uneven information density in sign language. In addition, many existing works rely on expert-annotated intermediate representations, i.e. glosses, which limit the scalability of the model.
In this work, we are inspired by recent advances from learning the discrete representation for generation (Huang et al., 2023; Zhang et al., 2023b; Williams et al., 2020; van den Oord et al., 2017; Ao et al., 2022). Specifically, we investigate a twostage framework based on Dynamic Vector Quantized Variational Autoencoders (DVQ-VAE) and Generative Pre-trained Transformer (GPT) (Radford et al., 2018) for text-to-sign language production. In the first stage, as shown in Figure 1, DVQ-VAE will learn the weights of each frame and the boundaries of the basic semantic units. Then, the weighted latent vectors are mapped to discrete code indices. Further quantitative analysis of the uneven information density in sign language is provided in section 3. To encourage models to perform variable-length encoding and compress sequence lengths, we propose a novel budget loss. Additionally, to preserve the semantic information of the reconstructed sign language sequences, we also introduce a translation auxiliary loss. In the second stage, a GPT-like model is learned to to generate code index sequences from spoken language text. Furthermore, since the duration of quantized code in a sequence can also vary dynamically, we further propose a duration transformer to predict the duration of the next code based on the previous code’s duration and the current code.
The experimental results on the widely used SLP dataset PHOENIX14T (Camgoz et al., 2018b) demonstrate that our proposed method achieves superior back translation performance compared to previous approaches. Furthermore, throughout the entire development process of image generation and text generation, the scale of the dataset has played a crucial role. A large amount of highquality corpus is also very important for SLP tasks. In this paper, we present the largest known German Sign Language dataset, PHOENIX-News, which consists of 486 hours of sign language videos, audio, and transcription texts. The native expression, clear hand details, and extensive coverage of our large-scale dataset make it suitable for a variety of sign language research tasks, such as sign language translation and sign language production. Based on this dataset, we further explore the impact of training data size on SLP tasks. Empirical analy-
sis shows that the performance of our model can be further improved by increasing the size of the training data.
Our main contributions are summarized as follows:
• We analyse the uneven information density in sign language. Additionally,we propose for the first time an information density based variable length coding method suitable for sign language.
• We propose a two-stage SLP framework consisting of two components: 1) DVQ-VAE to dynamically assign variable-length codes to sequences based on their different information densities through a novel adaptive downsampling module and budget loss. 2) A novel T2M-GPT model to predict variable-length codes and their corresponding durations.
• Extensive experiments on the challenging PHOENIX14T dataset show the effectiveness of our proposed method.
• We propose the largest known German sign language dataset, PHOENIX-News, which can be used for a variety of sign language research tasks.

heading: 2 Related Work

heading: 2.1 Sign Language Production
text: Sign language production (SLP) has been an active area of research for nearly two decades(Cox et al., 2002; McDonald et al., 2016). Early approaches focused on mapping text to glosses using neural models. (Stoll et al., 2020) proposed a seq2seq architecture for SLP, which mapped text input to glosses. To generate 2D joint locations, they utilized an empirical lookup table paradigm. Then, (Saunders et al., 2020b) proposed a progressive transformer to directly learn the mapping between annotations and skeleton pose sequences. (Saunders et al., 2020a) proposed to improve the quality of skeleton pose generation through adversarial training. In addition, several approaches have been proposed to enhance the generation quality through the utilization of mixture density networks(Saunders et al., 2021a), Mixture-of-Experts(Saunders et al., 2021b), dictionary representations(Saunders et al., 2022), and diffusion models(Baltatzis et al., 2023a). Several studies have proposed the use of non-autoregressive
models to generate sign language, thereby improving generation speed (Huang et al., 2021; Hwang et al., 2021, 2022). Additionally, researchers have explored the generation of photo-realistic sign language videos using Generative Adversarial Networks (GANs) (Saunders et al., 2022) or diffusion models (Fang et al., 2023; Xie et al., 2024). Recent studies have shown that using 3D human models, such as SPML-x(Pavlakos et al., 2019), is a better choice for sign language understanding (Lee et al., 2023) and production tasks (Stoll et al., 2022). (Inan et al., 2022) found that representing the intensification level of glosses connected with the duration of a sign.

heading: 2.2 Vector Quantization for SLP
text: Vector Quantized Variational Autoencoders (VQVAE) proposed by (van den Oord et al., 2017) is an autoencoder structure that aims to learn a discrete representation of data. Recently, VQ-VAE has been used for the SLP task, such as (Saunders et al., 2021a) using a modified VQ-GAN for isolated word sign language video generation. Recently, VQ-VAE has been applied to the SLP task. For instance, (Xie et al., 2024) utilized a modified VQ-GAN (Esser et al., 2021) to generate isolated sign language videos. (Xie et al., 2023) employed VQ-VAE to generate sign pose sequences from gloss sequences. However, existing methods rely on fixed-length encodings and overlook the unequal distribution of information in sign language. To address this issue, we propose a pioneering approach: a variable-length dynamic vector quantization method specifically designed for sign language.

heading: 2.3 Sign Language Dataset
text: High-quality sign language datasets are crucial for the SLP task. Table 1 summarizes the publicly available datasets used for sign language research. The PHOENIX14T (Camgoz et al., 2018b) dataset is the most commonly used dataset for SLP tasks, but it has limited data. As an important supplement, we propose PHOENIX-News, which contains 486 hours of sign language data. To the best of our knowledge, this is the largest German sign language dataset to date.
0 25 50 75 100 125 150 175 200 Gloss Duration
0.00
0.01
0.02
0.03
0.04
0.05
0.06
De ns
ity
(a) Length distribution of different glosses.
0 20 40 60 80 100 120 Gloss Duration
0.00
0.02
0.04
0.06
0.08
De ns
ity
(b) Length distribution of the same gloss.

heading: 3 Analyzing Information Density in Sign Language
text: The most commonly used discrete representation for sign language is glosses, which are the basic semantic units in sign language and are annotated by sign language experts. We first counted the length distribution of glosses in the PHOENIX14T dataset, as shown in Figure 2a. It can be seen that the length distribution of glosses is uneven, with most glosses having a length between 0 and 50, but some glosses have a length of more than 50. This indicates that uneven information density does exist in sign language. We then counted the length distribution of the most frequently occurring glosses (REGEN) in different contexts, as shown in Figure 2b. It can be seen that even the same gloss has differ-
ent lengths in different contexts. These analysis results inspire us to design a dynamic vector quantization method as described in subsection 4.2 and a duration transformer to predict duration based on context as described in subsection 4.3.

heading: 4 Method
text: Our overall two-stage framework is depicted in Figure 3, which consists of two stages: DVQ-VAE and T2M-GPT. In the following, we will first briefly revisit the formulation of VQ and then describe our proposed method in detail.

heading: 4.1 Preliminary
text: Vector quantization (VQ) (van den Oord et al., 2017) represents a technique for learning a codebook to encode sign language sequences into discrete code representations. Given a sign language sequence X = [x1, x2, . . . , xT ] with xt ∈ Rd, where T is the number of frames and d is the dimension of the sign language, we aim to recover the sign language sequence through an autoencoder and a learnable codebook containing K codes C = {ck}Kk=1 with ck ∈ Rdc , where dc is the dimension of codes. The sign language sequence X is first encoded by the encoder E into a sequence of latent vectors Z = [ z1, z2, . . . , zT/l ] , and zt ∈ Rdc , where l represents the temporal downsampling rate of the encoder E. For fixedlength encoding, l is fixed, while for variablelength encoding, l is dynamically changing. For i-th latent feature zi, the quantization through C is to find the most similar element in C, which can be properly written as:
ẑi = arg ck∈C
min ∥zi − ck∥2 (1)

heading: 4.2 Stage 1: Dynamic Vector Quantization VAE (DVQ-VAE)
text: Existing methods employ a fixed downsampling rate l for fixed-length encoding, neglecting the uneven information density in sign language. This oversight introduces redundancy in the learned codebook, leading to a decrease in both generation quality and speed. To address this issue, we propose DVQ-VAE, which consists of a dynamic encoder and a dynamic decoder.
Dynamic Encoder. As shown in Figure 3, the sign language sequence X first passes through a sign language embedding layer. Then, after adding
positional encoding information, it is input into a Transformer Encoder to obtain a sequence of latent vectors H = [h1, h2, . . . , hT ], with ht ∈ Rdh , where dh is the dimension of the latent vectors. We formulate these operations as:
X ′t =relu(LN(W1Xt +B1)) + fpos(t)
H =TransformerEncoder(X ′) (2)
where LN denotes Layer Normalization (Ba et al., 2016), fpos denotes the positional encoding function, and W1 ∈ Rdh×d and B1 ∈ Rdh are learnable parameters.
The dynamic encoder then contains an information-based adaptive downsampling module, which adaptively adjusts the downsampling rate by considering the information weight of each frame. Specifically, we input the latent vector sequence H into a multi-layer perceptron (MLP) to obtain the information weight of each frame I = [i1, i2, . . . , iT ], where it ∈ [0, 1]. We then segment the latent vector sequence H according to the information weight threshold O (we set it to 1.0) for semantic unit, and then perform weighted averaging within the segment to obtain the downsampled latent vector sequence Z = [ z1, z2, . . . , zT/l ] . The downsampling process of the entire module can be formulated as:
I = σ(W3(relu(W2H +B2) +H) +B3) (3)
S = cumsum(I)//O (4)
Zt =
T∑
j=1
Hj · Ij · Fj , Dt = T∑
j=1
·Fj
where Fj = { 1, if Sj = t− 1 0, otherwise
(5)
Equation 3 represents the operation of the MLP, where σ denotes the sigmoid activation function. Equation 4 represents the process of segmenting the latent vector sequence H according to the information weight threshold O, where S = [s1, s2, . . . , sT ] and st ∈ [0, sum(I)//O], representing the position markers of the segments. cumsum denotes the cumulative sum function, and // denotes the integer division. Equation 5 represents the process of weighted downsampling, where Zt denotes the downsampled latent vector and Dt denotes the duration of the current latent vector.
Dynamic Decoder. The goal of the dynamic decoder is to reconstruct the original sign language sequence X based on the quantized latent vector sequence Ẑ and the duration information D = [ d1, d2, . . . , dT/l ] . We use a length regulator module to address the issue of mismatched lengths between the vector sequence Ẑ and the original sign language sequence X during dynamic decoding.
X̂ = LR(Ẑ,D) (6)
where X̂ denotes the extended sequence, and LR denotes the length regulator module. For example, if Ẑ = [ẑ1, ẑ2, ẑ3] and D = [1, 2, 3], then X̂ = [ẑ1, ẑ2, ẑ2, ẑ3, ẑ3, ẑ3]. We then input the extended sequence X̂ into a Transformer-based decoder to obtain the reconstructed sign language sequence Xre.
Training of DVQ-VAE. The optimization goal of the original VQ-VAE (van den Oord et al., 2017) Lvq contains three components: a reconstruction loss Lre , an embedding loss Lembed , and a commitment loss Lcommit .
Lvq = Lre + ∥Z − sg[Ẑ]∥2︸ ︷︷ ︸ Lembed +λ1 ∥sg[Z]− Ẑ∥2︸ ︷︷ ︸ Lcommit
(7)
where λ1 is a hyper-parameter for the commitment loss and sg is the stop-gradient operator. In our work, the calculation formula for the reconstruction loss is as follows:
Lre = Lsmooth1 (X,Xre )+Lsmooth1 (V (X), V (Xre )) (8) where V (·) denotes the calculation of velocity, for example, V (X) = [v1, v2, . . . , vT−1], where vi = xi+1 − xi. In addition to the original optimization goal, we introduce two new loss functions: budget loss Lbudget and sign language translation auxiliary loss Lslt. Without using the budget loss, the model tends to use more codes to represent the sign language sequence, resulting in longer sequence lengths. To encourage the model to use a higher downsampling rate l, we define the budget loss as:
Lbudget = E[max(0, (sum(I)− T/R))] (9)
Since the length of the downsampled sequence is sum(I)//O, the budget loss can be interpreted as the expectation of the length of the downsampled sequence. Where T denotes the length of the original sign language sequence, and R denotes the expected downsampling rate. The goal of the sign
language translation auxiliary loss is to preserve the semantic information of the reconstructed sign language sequence, and its calculation formula is as follows:
Lslt = E[−logP (Y |Xre)] (10)
where Y denotes the spoken language text corresponding to the sign language sequence. The final loss for DVQ-VAE is defined as:
L = Lvq + λ2Lbudget + λ3Lslt (11)
We also use two common training recipes (Razavi et al., 2019), exponential moving average (EMA) and code book restart, to improve the utilization of the codebook.

heading: 4.3 Stage 2: Text-to-Sign GPT (T2S-GPT)
text: Code-Transformer. With a learned DVQ-VAE, a sign language sequence X can be mapped to a sequence of indices S = [ s1, s2, . . . , sT/l, End ] , which are indices from the learned codebook. Note that a special End token is added to indicate the stop of the sign language code sequence. By projecting S back to their corresponding codebook entries, we obtain Ẑ = [ ẑ1, ẑ2, . . . , ẑT/l ] , where ẑi = csi . The generation of the sign language code sequence S can be formalized as an autoregressive next index prediction problem: given the previous i − 1 indices, i.e., S<i, and the text condition Y , our goal is to predict the distribution of the possible next index p (Si | Y, S<i), which can be solved by a transformer, as shown in Figure 3. The negative log-likelihood (NLL) loss for code autoregressive training is:
Lcode = E[− log p (Si | Y, S<i, D<i)] (12)
We introduce a duration embedding layer to embed the duration information D into the transformer.
Duration-Transformer. As mentioned in subsection 4.2 and Equation 6, to decode Xre, we need not only Ẑ, but also the duration information D. Therefore, we design a duration-transformer to predict the duration of the next code based on the previous code’s duration and the current codes. As shown in Figure 3, the duration-transformer takes the sum of Code-Transformer’s output hidden vector Hcode and an extra code embedding as input:
Hdur =Hcode[Ny : Ny + l − 1] + fcode(S[≤ l])
(13)
where Hdur denotes the input of the durationtransformer, and Ny denotes the length of the condition text. The design idea behind this is that when predicting the next code’s duration, the model should not only be aware of previous steps’codes and their duration information but also should be aware of current code information. The optimization goal of the duration-transformer is to minimize the difference between the predicted duration and the real duration. The calculation formula is as follows: Ldur = E[∥Di − D̂i∥2] (14) In inference, we round the output of the durationtransformer to obtain the duration. The final optimization goal is:
L = Lcode + Ldur (15)

heading: 5 The Proposed PHOENIX-News Dataset
text: As shown in Table 1, PHOENIX-News aims to provide the community with a new large-scale
document-level sign language dataset, which contains 486 hours of sign language videos, audio, and transcription texts. We collected daily news programs in German Sign Language from the German public television station PHOENIX from 2013 to 2023. We then used whisper (Radford et al., 2023) to transcribe the program’s speech into text. Finally, we performed preprocessing steps such as domain cropping, sign language pose estimation, and sign language text alignment to obtain the final dataset. Since there are new sign language news programs every day, PHOENIX-News will be a continuously updated project. Therefore, we did not divide the dataset into training and test sets, but used the existing dataset for testing. Each video in the dataset has an average duration of 4.7 seconds, and the average text length is 11 words. We show the distribution of video duration and text length in Figure 4a and Figure 4b.

heading: 6 Experiments

heading: 6.1 Experimental Setup
text: We will introduce our experimental setup in this section, including the dataset and evaluation metrics. We will provide the implementation details of the model in the Appendix C.
Dataset and Evaluation Metrics. We evaluate our proposed T2S-GPT model on the PHOENIX14T (Camgoz et al., 2018b) dataset, which is the most commonly used dataset for SLP tasks and has been used as a benchmark in many previous SLP works (Huang et al., 2021; Saunders et al., 2021a, 2020b; Xie et al., 2023). The PHOENIX14T dataset contains 7,096 training samples (with 2,887 words in German spoken language translations), 519 validation samples, and 642 test samples. We use the pose parameter θ in the SMPLX model to represent the sign language pose, and the rotation 6D representation (Zhou et al., 2019) is used to represent the rotation in the pose. Follow-
ing the most widely used setting in SLP (Saunders et al., 2020b), we use the back translation metric to evaluate the generation quality. Since previous works did not publicly release the weights of their SLT models used to calculate the back translation metric, following the previous setting, we train an SLT model using the code from (Camgoz et al., 2020). We provide the details of the sign language representation in the Appendix B. We provide the training details of the SLT model in the Appendix D.

heading: 6.2 Comparisons with State-of-the-Art Methods
text: We compare our T2S-GPT model with several other models, including the state-of-the-art text-tosign model and the state-of-the-art text-to-motion model.
Comparison methods. 1) Progression Transformer (PT) (Saunders et al., 2020b) directly predicts the sign language pose sequence in an autoregressive manner. 2) NAT-EA (Huang et al., 2021) generates the sign language pose sequence in a nonautoregressive manner. 3) T2M-GPT (Zhang et al., 2023b) is a state-of-the-art autoregressive text-tomotion model, and its prediction target is the discrete representation of sign language processed by VQ-VAE. 4) MDM (Tevet et al., 2022) uses a diffusion model to generate motion sequences based on text in a non-autoregressive manner. Both T2MGPT and MDM use CLIP (Radford et al., 2021) to extract text features as condition signals, but the original CLIP does not support German well. To make a fair comparison, we use a multilingual CLIP model(Reimers and Gurevych, 2019)2.
Quantitative Comparison. We report the back translation metrics (including BLEU (Papineni et al., 2002) scores and ROUGE-L (Lin and
2https://huggingface.co/sentence-transformers/ clip-ViT-B-32-multilingual-v1
Och, 2004) scores) obtained by all models on the PHOENIX14T dataset in Table 2. We only use the PHOENIX14T dataset to train all models, and the training settings are consistent with those in the original papers. As shown in Table 2, our T2SGPT model achieves the best results on all metrics. Specifically, our T2S-GPT model achieves a score of 11.87 on BLEU-4, which is 3.86 points higher than the state-of-the-art T2M-GPT model. On ROUGE-L, our T2S-GPT model achieves a score of 34.65, which is 4.28 points higher than the state-of-the-art MDM model. These results indicate that our T2S-GPT model can generate higherquality sign language.
Qualitative Results. We qualitatively compare our T2S-GPT method with other methods and the ground truth sign language pose sequence on the PHOENIX14T test set, as shown in Figure 5. As shown in Figure 5, compared with other methods, the sign language generated by our T2S-GPT method is closer to the ground truth. Note that we provide a video demonstration of our method on the anonymous project homepage https://t2sgpt-demo.yinaoxiong.cn/, which can better convey the temporal information.

heading: 6.3 Ablation and Analysis
text: Analysis on DVQ-VAE. As shown in the second row of Table 3, when we replace DVQ-VAE with the VQ-VAE proposed by (Zhang et al., 2023b) with a downsampling rate of 4, we find that the back translation metrics of the SLP model have decreased significantly. This indicates that our DVQ-VAE model can obtain more compact and higher-quality discrete representations of sign language.
Analysis on Duration-Transformer. As shown in the third row of Table 3, when we replace the duration-transformer with a simple fully connected layer, we find that the back translation metrics of the SLP model have decreased significantly.
Impact of dataset size. To study whether our proposed T2S-GPT model is scalable, we train the T2S-GPT model by adding different proportions of the PHOENIX-News dataset. The experimental results are shown in , where we find that as
the dataset size increases, the performance of the T2S-GPT model also continues to improve. This indicates that our T2S-GPT model is scalable.

heading: 7 Conclusion
text: In this work, we propose a two-stage text-to-sign model T2S-GPT, which consists of a dynamic vector quantization VAE (DVQ-VAE) and a GPTlike autoregressive generation model. Our method achieves better performance than the previous stateof-the-art text-to-sign model. In addition, we have collected a new large-scale document-level sign language dataset PHOENIX-News, and the experimental results show that a larger dataset can still bring additional improvements to our method.

heading: 8 Limitations and Potential Risks
text: Although using a 3D human body model as the sign language representation introduces prior information about human body shape, it does not constrain the rotation motion of the joints themselves. The model’s predictions occasionally produce some abnormal cases that do not conform to the human joint structure, which may make users feel uncomfortable. At the same time, this is also a manifestation of the model’s generation errors. To address this issue, we plan to introduce more prior information in future work, such as human motion priors and physical constraints on human joint rotation angles. SLP technology itself does not have any obvious potential risks, but since the current SLP technology is still in a relatively early stage, if it is directly applied to practical scenarios, it may mislead users. For example, in weather forecasts, if the model generates sign language with incorrect place names, it may mislead users.

heading: Acknowledgements
text: This work was supported by the Key Research and Development Projects in Zhejiang Province (No. 2024C01106), the NSFC (No. 62272411), the National Key Research and Development Project of China (2018AAA0101900), and Research funding from FinVolution Group.

heading: A Example Appendix

heading: B Sign Language Representation
text: Inspired by the latest advances in sign language processing (Lee et al., 2023; Stoll et al., 2020), to better represent the complex body movements in sign language, we propose to use the pose parameter θ⃗ = [ ω⃗T0 , . . . , ω⃗ T K
]T of the SMPL-X human body model (Pavlakos et al., 2019) as the sign language representation, instead of the 3D joint coordinates in Euclidean space used in previous works. Where ω⃗k ∈ R3 denotes the axis-angle representation of the relative rotation of part k with respect to its parent in the kinematic tree. However, since the axis-angle form is not a continuous rotation representation, which is not conducive to network learning, we further convert it to the rotation 6D representation (Zhou et al., 2019) o⃗ = [ r⃗T0 , . . . , r⃗ T K
]T . We ignore the lower body joints outside the visible range. There are three advantages of using this representation: 1) it has rotation and translation invariance; 2) it separates the modeling of human body shape and pose, and the semantics of sign language should only be related to the pose and independent of the shape; 3) the introduction of human body prior avoids generating abnormal results, such as fingers longer than arms.
C Implementation Details
For DVQ-VAE, we set the dimension of the latent vectors dh to 512, the dimension of the codebook dc to 512, and the number of codes K to 1024. The number of transformer layers in the encoder and decoder is set to 6, and the hidden size, number of heads, and feed-forward dimension for each layer are set to 512, 8, and 2048, respectively. The dropout rate is set to 0.1. We use AdamW (Loshchilov and Hutter, 2018) optimizer with [β1, β2] = [0.9, 0.99], batch size of 256 , and exponential moving constant λ = 0.99. We train for a total of 100K iterations, with an initial learning rate of 2e-4, and then use the cosine learning rate decay strategy during training. λ1, λ2, and λ3 in the final loss are set to 1, 0.5, and 1.0, respectively. The R in Lbudget is set to 12.
For T2S-GPT, the hidden size, number of heads, and feed-forward dimension for each transformer layer are set to 1024, 16, and 4096, respectively. The dropout rate is set to 0.1. The number of transformer layers in the code-Transformer and durationTransformer is set to 18 and 6, respectively. We
use a batch size of 256 and train for 300K iterations. We optimize the models with the AdamW optimizer, warm up the learning rate for the first 4k updates to a peak of 1e-4, and then linearly decay it to 0. We use a 32GB NVIDIA V100 GPU to train our model.

heading: D Back Translation Model
text: To calculate the back translation metric, we train a sign language translation (SLT) model that takes sign language pose sequences as input and outputs the corresponding spoken language text. The SLT model adopts the architecture introduced by (Camgoz et al., 2020). Both the encoder and decoder components of the model are built using transformers. In particular, the hidden size, number of heads, and feed-forward dimension for each layer are configured as 512, 8, and 2048, respectively. Additionally, a dropout rate of 0.4 is applied within the model. The number of transformer layers in the encoder and decoder is set to 3. The training settings are consistent with those in the original paper.

heading: Future Work
text: To address this issue, we plan to introduce more prior information in future work, such as human motion priors and physical constraints on human joint rotation angles. SLP technology itself does not have any obvious potential risks, but since the current SLP technology is still in a relatively early stage, if it is directly applied to practical scenarios, it may mislead users. For example, in weather forecasts, if the model generates sign language with incorrect place names, it may mislead users