Semi-Supervised Spoken Language Glossification (2024)

Abstract: Spoken language glossification (SLG) aims to translate the spoken language text into the sign language gloss, i.e., a written record of sign language. In this work, we present a framework named Semi-Supervised Spoken Language Glossification (SLG) for SLG. To tackle the bottleneck of limited parallel data in SLG, our SLG incorporates large-scale monolingual spoken language text into SLG training. The proposed framework follows the selftraining structure that iteratively annotates and learns from pseudo labels. Considering the lexical similarity and syntactic difference between sign language and spoken language, our SLG adopts both the rule-based heuristic and modelbased approach for auto-annotation. During training, we randomly mix these complementary synthetic datasets and mark their differences with a special token. As the synthetic data may be less quality, the SLG further leverages consistency regularization to reduce the negative impact of noise in the synthetic data. Extensive experiments are conducted on public benchmarks to demonstrate the effectiveness of the SLG. Our code is available at https://github.com/yaohj11/S3LG.

heading: 1 Introduction
text: Sign Language is the most primary means of communication for the deaf. Translating between sign and spoken language is an important research topic, which facilitates the communication between the deaf and the hearing (Bragg et al., 2019; Yin et al., 2021). To support the development of applications, sign language gloss has been widely used as an intermediate step for generating sign language video from spoken language text (Saunders et al., 2020, 2022) or the inverse direction (Pu et al., 2020; Chen et al., 2022). The sign language gloss is the written representation of the signs. As a generally adopted way for sign language transcription, gloss is sufficient to convey most of the key information in sign
∗Corresponding authors: Wengang Zhou and Houqiang Li
language. In this work, we focus on the first step of the former task named spoken language glossification (SLG), which aims to translate the spoken language text into the sign language gloss.
SLG is typically viewed as a low-resource sequence-to-sequence mapping problem. The previous methods (Zhu et al., 2023; Walsh et al., 2022; Egea Gómez et al., 2021, 2022) rely on the encoder-decoder architectures (Luong et al., 2015; Sutskever et al., 2014) to jointly align the embedding space of both languages in a data-driven manner. Since the data collection and annotation of sign language requires specialized knowledge, obtaining a large-scale text-gloss dataset is timeconsuming and expensive (De Coster et al., 2023). As a result, the performance of SLG models is limited by the quantity of parallel data (Camgoz et al., 2018; Zhou et al., 2021). Witnessing the success of introducing monolingual data to enhance lowresource translation quality (Cheng et al., 2016; Sennrich et al., 2016; Pan and Yang, 2009; Zoph et al., 2016) in neural machine translation (NMT), we are motivated to explore the accessible unlabeled spoken language texts to improve SLG.
In this work, we present a framework named Semi-Supervised Spoken Language Glossification (S3LG) to boost SLG, which iteratively annotates and learns from pseudo labels. To implement the above idea, the proposed S3LG adopts both the rule-based heuristic and model-based approach to generate pseudo glosses for unlabeled texts. The rule-based synthetic data has high semantic accuracy, however, the fixed rules make it difficult to cover complex expression scenarios. The model-based approach on the other hand is more flexible for learning the correspondence between sign language and spoken language and generates pseudo gloss with higher synthetic diversity. These complementary synthetic datasets are randomly mixed as a strong supplement for the training of the SLG
9300
model. Besides, the model-based synthetic data is generated by the SLG model, which sets a good stage for iteratively re-training the SLG model.
In addition, S3LG introduces a simple yet efficient design from three aspects. Firstly, in each iteration, the training process is separated into two stages, i.e., pre-training and fine-tuning for domain adaptation. Secondly, to encourage the model to learn from the noisy pseudo labels, we apply the consistency regularization term to the training optimization and gradually increase the weight of the consistency regularization in the training curriculum. It enforces the consistency of the predictions with network perturbations (Gao et al., 2021) based on the manifold assumption (Oliver et al., 2018). Thirdly, to encourage the SLG model to learn complementary knowledge from different types of synthetic data, a special token is added at the beginning of input sentences to inform the SLG model which data is generated by the rule-based or model-based approach (Caswell et al., 2019). Through end-toend optimization, our S3LG achieves significant performance improvement over the baseline. Surprisingly, the experiments show that the translation accuracy on low-frequency glosses is promisingly improved. We conjecture that the SLG model acts differently in annotating the high-frequency and low-frequency glosses, and such bias is mitigated by the rule-based synthetic data.
In summary, our contributions are three-fold:
• We propose a novel framework S3LG for SLG (namely, text-to-gloss translation), which iteratively annotates and learns from the synthetic data. It adopts two complementary methods, i.e., the rule-based heuristic and model-based approach for auto-annotation.
• We further leverage consistency regularization to reduce the negative impact of noise in the synthetic data. The biases of the SLG model on low-frequency glosses are mitigated by incorporating the rule-based synthetic data.
• We conduct extensive experiments to validate our approach, which shows encouraging performance improvement on the two public benchmarks, i.e., CSL-Daily (Zhou et al., 2021) and PHOENIX14T (Camgoz et al., 2018).

heading: 2 Related Work
text: In this section, we briefly review the related works on spoken language glossification and semi-
supervised learning.
Spoken language glossification. Camgoz et al. (2018) publish the first sign language dataset PHOENIX14T and pioneer the linguistic research for sign language (De Coster et al., 2021; Cao et al., 2022). With the advance of NMT, the previous methods (Stoll et al., 2020; Saunders et al., 2020) adopt the encoder-decoder paradigm, which can be specialized using different types of neural networks, i.e., RNNs (Yu et al., 2019), CNNs (Gehring et al., 2017). Considering gloss as a text simplification, Li et al. (2022) propose a novel editing agent. Instead of directly generating the sign language gloss, the agent predicts and executes the editing program for the input sentence to obtain the output gloss. By leveraging the linguistic feature embedding, Egea Gómez et al. (2021) achieve remarkable performance improvement. Egea Gómez et al. (2022) further apply the transfer learning strategy result in continues performance increasing. Recently, Zhu et al. (2023) first introduce effective neural machine translation techniques to SLG with outstanding performance improvements, which lays a good foundation for further research.
Semi-supervised learning. Generating pseudo labels for the unlabeled data is a widely adopted semi-supervised learning algorithm in low-resource NMT, known as back-translation (Gulcehre et al., 2015) and self-training (Zhang and Zong, 2016), respectively. With the target-side monolingual data, back-translation obtains pseudo parallel data by translating the target-side sentences into the sourceside sequences. As an effective data augmentation method, it is widely adopted in the inverse task of SLG, i.e., gloss-to-text translation (Moryossef et al., 2021; Zhang and Duh, 2021; Angelova et al., 2022; Chiruzzo et al., 2022). Due to the lack of the sign language gloss corpus, it is hard to incorporate large-scale monolingual data in the training process of SLG with back-translation (Rasooli and Tetreault, 2015). In contrast, the self-training requires source-side monolingual data to generate pseudo parallel data based on a functional sourceto-target translation system. Since it is hard to optimize a neural translation system with an extremely limited amount of parallel data (Moryossef et al., 2021; Zhang and Duh, 2021; Xie et al., 2020), this motivates us to go along this direction and design more effective algorithms.
Different from the aforementioned methods, we focus on iteratively annotating and learning from
pseudo labels. Considering the lexical similarity and syntactic difference between sign language and spoken language, we adopt two complementary approaches (i.e., rule-based heuristic and modelbased approach) to generate synthetic data. Moreover, we put forward the consistency regularization and tagging strategy to reduce the negative impact of noisy synthetic data.

heading: 3 Methodology
text: In this section, we first introduce the overview of our S3LG in Sec. 3.1; then we elaborate on the annotating methods for monolingual data in Sec. 3.2; and finally, we detail the training strategy in Sec. 3.3.

heading: 3.1 Overview
text: The primary objective of the SLG model is to acquire knowledge about the mapping f(θ) : X 7→ Y , where X and Y denote the collection of spoken language text and sign language gloss associated
with the vocabulary V , respectively. θ is the parameters of the SLG model. Most SLG model adopts the encoder-decoder architecture, where the input x ∈ X is first encoded to devise a high-level context representation. It is then passed to the decoder to generate the output y ∈ Y . The encoder and decoder can be specialized using different types of neural networks. Given a set DL = {(xiL,yiL)}Mi=1 of M labeled samples and a set XU = {xiU}Ni=1 of N unlabeled data, we aim to design a semisupervised framework for SLG to improve text-togloss translation by exploring both the labeled and unlabeled data. To this end, we propose S3LG, which iteratively annotates and learns from two complementary synthetic data generated by the rule-based heuristic and model-based approach, respectively.
Fig. 1 provides an overview of the S3LG approach, which consists of three main steps, namely rule-based annotating, model-based annotating, and two-stage training. At the k-th iteration, the synthetic data Dk−1U = {(xiU ,y i,k−1 U )}Ni=1 is composed of two parts, i.e., rule-based DU,r = {(xiU ,yiU,r)}Ni=1 and model-based synthetic data Dk−1U,m = {(xiU ,y i,k−1 U,m )}Ni=1. Based on the monolingual data XU = {xiU}Ni=1, the rule-based and model-based synthetic data are generated by the fixed rules and the functional SLG model f(θ̂k−1) obtained from the previous iterations, respectively. We randomly mix the two complementary synthetic data and add a special token at the beginning of the input sentences. The synthetic data Dk−1U = {(xiU ,y i,k−1 U )}Ni=1 is then concatenated with the original golden data DL = {(xiL,yiL)}Mi=1 as a strong supplement for training an SLG model f(θk), where N ≫ M . After repeating K times, we obtain the final SLG f(θ̂K) model. Notably, in the first iteration, only the rule-based heuristic is available to generate the pseudo gloss sequences for the monolingual data, as in, D0U = DU,r.

heading: 3.2 Annotating Monolingual Data
text: Compared with the limited size of text-gloss pairs, the unlabeled spoken language sentences are easy to reach. To leverage both the labeled and unlabeled data to enhance the SLG performance, we employ the rule-based heuristic and model-based approach to generate the pseudo parallel data and use it to enrich the original golden data for training. Rule-based annotating. Given that sign language gloss is annotated based on the lexical elements
from the corresponding spoken language, a naive rule is to copy the unlabeled texts as gloss (Zhu et al., 2023; Moryossef et al., 2021). Then, we further apply language-specific rules to different sign languages, respectively. A Chinese spoken language text is first separated at the word level and then one-on-one mapped into the closet glosses based on the lexical similarity. As German sign language texts often include affixes and markers, thus, we perform lemmatization on each word in the text. We leverage the open-source spaCy1 (Honnibal and Montani, 2017) to obtain the linguistics information. Using the above rule-based annotating system, the monolingual data XU = {xiU}Ni=1 is mapped as rule-based synthetic data DU,r = {(xiU ,yiU,r)}Ni=1. We provide a detailed list of rules in Appendix A. Model-based annotating. While the rule-based heuristic allows high lexical similarity between text and gloss, it cannot capture complicated syntactic divergence between two languages. Therefore, following the self-training structure, we further employ a functional SLG model to predict the pseudo glosses for monolingual data, based on more flexible correspondence learned from training data. Because there is a mutually reinforcing relationship between the translation model and the data it generates. As the model-based synthetic data is generated by the SLG model, it is possible to improve performance by iteratively re-training the SLG model. At the k-th iteration (k > 1), based on the best SLG model f(θ̂k−1) in k − 1 iterations, the monolingual data XU = {xiU}Ni=1 is annotated as model-based synthetic data Dk−1U,m = {(xiU ,y i,k−1 U,m )}Ni=1.

heading: 3.3 Two-Stage Training
text: The proposed S3LG iteratively annotates and learns from the synthetic data. As S3LG is a datacentric framework, we keep the SLG model simple but competitive, which is the vanilla Transformer model (Vaswani et al., 2017). Without loss of generality, we take the k-th iteration as an example to introduce the two-stage training strategy, as shown in Fig. 2.
At the beginning of the k-th iteration, we reinitialize a new SLG model f(θk), where the input text x = {xt}Txt=1 with Tx words is first encoded into a context representation. The decoder generates the target sequence y = {yt}Tyt=1 with Ty glosses based on the conditional probability
1https://spacy.io/
Two-Stage Training 𝒇𝒇(Ɵ𝒌𝒌)
p(y|x; θk). Specifically, the conditional probability is formulated as:
p(y|x; θk) = Ty∏
t=1
p(yt|x,y0:t−1; θk), (1)
where y0:t−1 = {y0, . . . , yt−1} denotes the previous output sub-sequence at the t-th step. The initial token y0 represents the beginning of a sequence.

heading: 3.3.1 Pseudo-Glosses Sampling
text: Once the rule-based DU,r = {(xiU ,yiU,r)}Ni=1 and model-based synthetic data Dk−1U,m = {(xiU ,y i,k−1 U,m )}Ni=1 are obtained, we integrate the annotations at the data level to leverage the two complementary synthetic data. Aiming at informing the SLG model that the two auto-annotation methods complement each other, a method-specific token is added at the beginning of the spoken language text. For each unlabeled text xiU , we randomly select a pseudo gloss yi,k−1U from the two pseudo glosses yiU,r and y i,k−1 U,m with equal probability. Based on the previous k − 1 iterations, we obtain the synthetic data Dk−1U = {(xiU ,y i,k−1 U )}Ni=1 for enlarging the training samples to re-train the SLG model. At the initial iteration, the synthetic data is formulated as D0U = DU,r = {(xiU ,yiU,r)}Ni=1.

heading: 3.3.2 Training Objective
text: For optimizing the SLG model f(θk) with both the synthetic data Dk−1U and golden data DL, we introduce two kinds of training objective, i.e., crossentropy loss, and consistency regularization. Cross-entropy loss. As shown in Equ. 1, the SLG generates the target translation based on the conditional probability provided by the decoder. The
cross-entropy loss computed between the annotation and the output of the decoder, which is formulated as:
LCE(x,y, θ k) = −log p(y|x; θk), (2)
where the y denotes the gloss annotation. Consistency regularization. The data distribution should be under the manifold assumption, which reflects the local smoothness of the decision boundary (Belkin and Niyogi, 2001). The consistency regularization is computed between two predictions with various perturbations to conform to the manifold assumption. We apply the network dropout as the perturbation. With the dropout strategy, the activated parts of the same model are different during training. The consistency regularization is formulated as:
LCR(x, θ k) =KL(f(x; θ1), f(x; θ2))
+KL(f(x; θ2), f(x; θ1)), (3)
where θ1 and θ2 denotes the different sub-models of the SLG f(θk) with dropout during training. f(x; θ) denotes the predictions given by the SLG model f(θ). KL(teacher, student) denotes the KL (Kullback-Leibler) divergence loss that aligns the student’s network to the teacher’s network.
Overall, the loss function of the proposed S3LG is formulated as:
L(x,y, θk) = LCE(x,y, θ k) + w · LCR(x, θk),
(4) where the weight w balances the effect of two parts of restraints.

heading: 3.3.3 Stage-One and Stage-Two
text: To alleviate the domain mismatching between the monolingual data XU and golden data DL, the training process is separated into two stages, i.e., pre-training and fine-tuning, which is a conventional way for domain adaption. The SLG model is first trained on the concatenation of large-scale synthetic data Dk−1U and golden data DL with the pre-training epochs T . To amplify the impact of synthetic data, the pre-training epochs gradually increase as the iteration grows. Thus the training objective of stage one is formulated as:
min θk
∑
(x,y)∈DL∪Dk−1U
L(x,y, θk). (5)
Subsequently, this model is fine-tuned only on the low-resource in-domain golden data DL until convergence. The training objective of stage two is
formulated as:
min θk
∑
(x,y)∈DL L(x,y, θk). (6)

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets. We evaluate our approach on two public sign language translation datasets, i.e., PHOENIX14T (Camgoz et al., 2018) and CSLDaily (Zhou et al., 2021). Both datasets provide the original sign language video, sign language gloss, and spoken language text annotated by human sign language translators. We focus on annotated text-gloss parallel data in this work. The PHOENIX14T dataset is collected from the German weather forecasting news on a TV station. The CSL-Daily dataset is a Chinese sign language dataset and covers a wide range of topics in daily conversation. Monolingual data. Following the previous work (Zhou et al., 2021), we obtain the monolingual spoken language sentences close to the topic of golden data. For the PHOENIX14T and CSLDaily dataset, we collect 566, 682 and 212, 247 sentences. The statistics of the data mentioned above are shown in Appendix B and C. Evaluation metrics. Referring to the previous works (Zhou et al., 2021; Li et al., 2022; Zhu et al., 2023), we quantify the performance of the generated gloss in terms of accuracy and consistency based on the BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), respectively. The BLEU-N (N ranges from 1 to 4) is widely used in NMT to show the matching degree of N units between two sequences. Besides, the ROUGE cares more about the fluency degree of generated sequences. Both the evaluation metrics indicate attributes of generated gloss, noting that the higher values demonstrate better translation performance. Training settings. We implement the proposed approach on Pytorch (Paszke et al., 2017). For PHOENIX14T and CSL-Daily, the SLG model consists of 5 and 3 layers, respectively. To mitigate the overfitting problem, we apply the common strategy such as dropout and label smoothing. For the network setting, the dimensions of the embedding layer and the feed-forward network are 512 and 2048, respectively. The number of the attention head is 8. For the optimization setting, we leverage
the Adam (Kingma and Ba, 2014). During training, the learning rate and batch size are fixed to 5 × 10−5 and 32, respectively. As at the beginning, the predictions of the SLG model might be unreliable, the consistency regularization weight w ramps up, starting from zero, along a linear curve until reaching the w (Laine and Aila, 2016). Following the previous setting (Li et al., 2019), we randomly shuffle and drop words of the spoken language sentences as data augmentation. Inference details. In inference, we use the beam search strategy (Wu et al., 2016) to increase the decoding accuracy. For both the CSL-Daily and the PHOENIX14T dataset, the search width and length penalty are set to 3 and 1.0, respectively. In the process of generating the pseudo glosses for monolingual data, we simply set the search width to 1 for efficiency. The experiment is run on an NVIDIA GeForce RTX 3090 with approximately 40 hours of computational time.

heading: 4.2 Comparison with State-of-the-Art Methods
text: We compare the proposed S3LG with the previous text-to-gloss systems on two public benchmarks, i.e., PHOENIX14T (Camgoz et al., 2018) and CSLDaily (Zhou et al., 2021). The performances are shown in Tab. 1 and Tab. 2, respectively.
As our goal is to explore how to incorporate
monolingual data for SLG, our baseline adopts the vallia Transformer as the SLG model which only learns from the original golden data. By combining all proposed components, our S3LG achieves substantial improvements against the baseline across all evaluation metrics. The S3LG achieves 28.24 and 27.95 BLEU-4 on the DEV set of the PHOENIX14T and CSL-Daily dataset, which surpasses the baseline by 5.95 and 13.9, respectively. The quantitative results demonstrate the effectiveness of utilizing the complementary synthetic data and designs in our S3LG. For the PHOENIX14T and CSL-Daily datasets, we evaluate the performance using the gloss-level tokenizer as the original annotations, respectively.
Zhu et al. (2023) provides translation performance in different settings, including semisupervised, transfer learning, and multilingual. For a fair comparison, we cite their best performance in the monolingual and bilingual settings. The results prove the advantage of our novel designs, which distinguishes our approach from previous SLG systems. The previous works mainly tested on the PHOENIX14T datasets, while CSL-Daily is also an important benchmark for different sign language tasks. To attract more research attention on Chinese Sign Language, we report our performance on this dataset.

heading: 4.3 Ablation Study
text: To validate the effectiveness of each component proposed in our S3LG framework, unless otherwise specified, we put forward several ablation studies on the DEV set of the PHOENIX14T dataset. Impact of proposed components. The main difference between our proposed method and the existing works is to leverage the complementary synthetic as a supplement for the training of the SLG model. To evaluate the effectiveness of each proposed component, we gradually add them to the baseline SLG system. Directly applying the iterative self-training process to the baseline delivers a performance gain of 0.68 BLEU-4, which motivates us to design a more effective algorithm. We further apply consistency regularization to enforce the predictions of synthetic data under the manifold assumption, which achieves a gain of 2.28 BLEU-4. Subsequently, combining the rule-based and model-based synthetic data can be helpful with 1.87 BLEU-4 improvements. Besides, the result shows that tagging and applying data augmentation to different types of synthetic data is also a useful strategy, which provides a further gain of 1.12 BLEU-4. The results are shown in Tab. 3. Impact of w. In our experiments, the consistency regularization weight w is set to 20. This hyperparameter determines the importance of the con-

heading: K ROUGE BLEU-1 BLEU-2 BLEU-3 BLEU-4
text: sistency regularization compared with the crossentropy loss. In Tab. 4, we examine the effect of consistency regularization weight with a set of different values. As the w is set to be 20, S3LG achieves its best performance. Impact of iteration number K. The iteration number K is an important hyper-parameter and fixed to 4 in the previous experiments. To explore the effect of K, we conduct experiments with different iteration numbers. Tab. 5 shows the best performance of the K iterations, where K = 1 (i.e., the initial iteration), the training data is composed of the rule-based synthetic data and golden data. By combining both the rule-based and modelbased synthetic data, the performance of the SLG model is converged at the 4-th iteration. Scale of synthetic samples. As mentioned above, the collected monolingual data outnumbers the annotated data over 30 times. In the previous experiments, all the monolingual data is incorporated with the golden data to enhance the training pro-
cess of the SLG model. In this ablation study, we investigate the scale of synthetic data. As shown in Tab. 6, the performance improves approximately along a log function curve according to the synthetic data volume. We speculate that pre-training on too much noisy out-of-domain synthetic data may drown the impact of golden data, which finally causes limited performance improvements. Quantity of annotated golden data. As the purpose of our proposed approach is to learn from the monolingual with pseudo glosses, we provide experiments regarding various quantities of synthetic data, in Tab. 7. Under all settings, S3LG outperforms the baseline. However, the experiment suggests that our proposed approach is more suitable for the scene when the baseline SLG performance is in the range of 7− 20 BLEU-4. Effect of pre-training epochs T . In Tab. 8, we evaluate the impact of pre-training with synthetic data with different epochs. To simplify the hyperparameter search process, we first conduct the experiments with fixed pre-training epochs T between different iterations and further apply the pretraining epochs growing strategy to it. The S3LG achieves the best performance under the setting of that the pre-training epoch is 15 for the first iteration and then gradually increases by 10 between iterations.
Appearance ≤ 3 ≤ 6 ≤ 8 ≤ 10 ≤ 15
Impact of leveraging synthetic data. To verify whether our performance improvement mainly comes from the synthetic data rather than simply enhancing the encoder with the monolingual data, we conduct experiments by enhancing the original baseline with the pre-trained language model, BERT (Devlin et al., 2019). As the results presented in Tab. 9, we observe that leveraging the pre-trained language model improves the translation quality, while our proposed approach achieves larger performance gains. We further combine our approach with the pre-trained language model. The experimental results demonstrate that the performance improvement of our approach stems from two aspects: better comprehension of the encoder and better generation capability of the decoder. Results of CHRF metric. To provide more information, following the previous work (Müller et al., 2023), we evaluate the performance of our proposed approach based on CHRF (Popović, 2016) metric. The S3LP achieves 56.02 and 54.84 CHRF on the DEV and TEST set of PHOENIX14T, which surpasses the baseline (52.00 and 51.32) by 4.02 and 3.52, respectively. Translation accuracy of low-frequency glosses. As the SLG model tends to predict the glosses with high frequency in training data, we believe that utilizing the model-free annotating approach can mitigate the model-based annotating bias. To verify this, we put forward the experiments for different synthetic data settings with the translation accuracy metric under different low-frequency standards, namely, a gloss appears less than how many times are considered as low frequency, as shown in Tab. 10. The translation accuracy of lowfrequency glosses is formulated as accuracy = Npred/Nall, where Npred and Nall denote the number of samples that are predicted with the correct low-frequency glosses and samples that contain the low-frequency glosses, respectively. We can
see that the translation accuracy of the SLG model leveraging both the model-based and rule-based synthetic data achieves promising improvements against the model-based one.

heading: 5 Conclusion
text: In this work, we present a semi-supervised framework named S3LG for translating the spoken language text to sign language gloss. With the goal of incorporating large-scale monolingual spoken language texts into SLG training, we propose the S3LG approach to iteratively annotate and learn from pseudo glosses. Through a total of K iterations, the final SLG model achieves significant performance improvement against the baseline. During each iteration, the two complementary synthetic data generated from the rule-based and modelbased approaches are randomly mixed and marked with a special token. We introduce a consistency regularization to enforce the consistency of the predictions with network perturbations. Extensive ablation studies demonstrate the effectiveness of the above designs. Besides, the translation accuracy on low-frequency glosses is improved.
Limitations
In the hope of attracting more research attention for SLG in the future, we provide the detailed limitations in the next. On the one hand, the results (see Tab. 6 and Tab. 7) of the experiments suggest that in the extremely low-resource scenarios, our performance improvements might be less significant as a large amount of monolingual data is available. We conclude that with the very limited golden data as anchors, it is hard to learn from the large-scale synthetic data. Although utilizing synthetic data as a strong supplement for training data can achieve promising performance improvements, it is hard to achieve the equal impact of enlarging the training data with annotated data from human interpreters. To promote the development of sign language research, the fundamental way might be continuously collecting large amounts of annotated data.
On the other hand, we realize that sign language glosses do not properly represent sign languages (Müller et al., 2023). However, as explained in the introduction section, sign language glosses are sufficient to convey most of the key information in sign language. Given the resource limitations and the current technological capabilities, we believe the two-stage way (i.e., text-to-gloss
and gloss-to-gesture) is more achievable and practical to support the development of the application for converting spoken language into sign language. There are many solutions for animating a 3D avatar and making gloss-indexed gestures smoothly and naturally. Correspondingly, research on the former stage lacks enough attention. We think improving the SLG model’s performance can be a promising way to implement better sign language production systems.
Ethical Considerations
Even with extensive advances in the development of neural machine translation methods in the spoken language area, the study of sign language is still in its infancy. At the same time, the development of different sign languages is very uneven. According to the existing approaches, the current researches mainly focus on DGS, leaving other sign languages unexplored. As studied in (Bragg et al., 2019; Yin et al., 2021), there is limited research to reveal the sign language linguistics character, which also limits the utility of prior knowledge.
As most sign language researchers are hearing people, the provided sign language system might not meet the actual needs of the deaf. Therefore, to bridge the commutation between the two communities, cooperation could be mutual. By consulting with native signers, we will proactively seek to design the translation system to be inclusive and user-centered in the future. We also encourage people of both communities to try out the existing systems and point out their disadvantages to guide the promising direction and accelerate the study of sign language.

heading: Acknowledgements
text: This work is supported by National Natural Science Foundation of China under Contract U20A20183 & 62021001, and the Youth Innovation Promotion Association CAS. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.

heading: A Rules Used in the Rule-based Heuristic
text: for Creating Synthetic Data DU,r
A.1 Chinese Rules
For a spoken text y, 1. Build the vocabulary for the gloss V and spoken word S from the original golden data DL. 2. Tokenize the spoken text at the word-level as y = {y1, y2, . . . , yT } with T words. 3. Replace the words yt ∈ y not in the word vocabulary S by a special token <UNK>. 4. Replace each spoken word yt ∈ y by the most similar gloss in the gloss vocabulary V = {v1, v2, . . . , v|V|} based on the lexical similarity, which is formulated as:
Sim(yt, vi) = E(yt) · E(vi) (7)
where E(·) denotes L2 normalized word embedding processes. The above information is obtained from the Chinese model (zh_core_web_lg) of spaCy.
A.2 German Rules
For a spoken text y = {y1, y2, . . . , yT } with T words,
1. Build the vocabulary for the gloss V and spoken word S from the original golden data DL.
2. Replace the words yt ∈ y not in the word vocabulary S by a special token <UNK>.
3. Lemmatize all the spoken words. 4. Replace the token yt ∈ y that only matches parts of compounds glosses vi ∈ V by it vi. The above information is obtained from the German model (de_core_news_lg) of spaCy.

heading: B Statistics of Sign Language Datasets
text: As shown in Tab. 11 and Tab. 12, we present the key statistics of the PHOENIX14T and CSL-Daily dataset, respectively.
The PHOENIX14T dataset is about weather forecasting and does not contain any information that names or uniquely identifies individual people or offensive content. The CSL-Daily dataset is screened by its publishing team and is about daily life (shopping, school, travel, etc.). Does not contain any information that names or uniquely identifies individual people or offensive content.

heading: C Statistics of the Training Data
text: The statistics of training data is shown in shown in Tab. 13. To collect more spoken language texts, we extract a subset of CLUE corpus (Xu et al., 2020) based on the topic of daily lives.
In the process of selecting this part of the data, the words in the PHOENIX14T and CSL-Daily datasets are used to select content on related topics, so most of the data is related to weather and daily life, which does not contain any information that names or uniquely identifies individual people or offensive content.

heading: Future Work
text: Limitations
In the hope of attracting more research attention for SLG in the future, we provide the detailed limitations in the next. On the one hand, the results (see Tab. 6 and Tab. 7) of the experiments suggest that in the extremely low-resource scenarios, our performance improvements might be less significant as a large amount of monolingual data is available. We conclude that with the very limited golden data as anchors, it is hard to learn from the large-scale synthetic data. Although utilizing synthetic data as a strong supplement for training data can achieve promising performance improvements, it is hard to achieve the equal impact of enlarging the training data with annotated data from human interpreters. To promote the development of sign language research, the fundamental way might be continuously collecting large amounts of annotated data. On the other hand, we realize that sign language glosses do not properly represent sign languages (Müller et al. , 2023). However, as explained in the introduction section, sign language glosses are sufficient to convey most of the key information in sign language. Given the resource limitations and the current technological capabilities, we believe the two-stage way (i. e. , text-to-gloss
and gloss-to-gesture) is more achievable and practical to support the development of the application for converting spoken language into sign language. There are many solutions for animating a 3D avatar and making gloss-indexed gestures smoothly and naturally. Correspondingly, research on the former stage lacks enough attention. We think improving the SLG model’s performance can be a promising way to implement better sign language production systems. Ethical Considerations
Even with extensive advances in the development of neural machine translation methods in the spoken language area, the study of sign language is still in its infancy. At the same time, the development of different sign languages is very uneven. According to the existing approaches, the current researches mainly focus on DGS, leaving other sign languages unexplored. As studied in (Bragg et al. , 2019; Yin et al. , 2021), there is limited research to reveal the sign language linguistics character, which also limits the utility of prior knowledge. As most sign language researchers are hearing people, the provided sign language system might not meet the actual needs of the deaf. Therefore, to bridge the commutation between the two communities, cooperation could be mutual. By consulting with native signers, we will proactively seek to design the translation system to be inclusive and user-centered in the future. We also encourage people of both communities to try out the existing systems and point out their disadvantages to guide the promising direction and accelerate the study of sign language