MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production (2024)

Abstract: Sign language understanding has made significant strides; however, there is still no viable solution for generating sign sequences directly from entire spoken content, e.g., text or speech. In this paper, we propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. In particular, a sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Moreover, by creating a joint embedding space for text, audio, and sign, we bind these modalities and leverage the semantic consistency among them to provide informative feedback for the model training. This embedding-consistency learning strategy minimizes the reliance on sign triplets and ensures continuous model refinement, even with a missing audio modality. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production.

heading: 1 Introduction
text: Sign language, a visual language, combines both manual (hand gestures) and non-manual cues for communication. It is specifically designed for the deaf and hearing-impaired community (Hickok et al., 1996; Armstrong and Wilcox, 2003; Campbell et al., 2008; Zhou et al., 2020). According to the World Federation of the Deaf, there are 70 million deaf people and more than 200 kinds of sign languages in the world (Fenlon and Wilkinson, 2015; Núñez-Marcos et al., 2023). Improvements in sign language production (SLP) can bridge the communication gap between the deaf and hearing (Mehdi and Khan, 2002; Harris et al., 2009; Taskiran et al., 2018; Rastgoo et al., 2021; Kahlon and Singh, 2023; Luo and Yang, 2024).
The challenges primarily arise from phonological difference and data scarcity. Phonological
†Corresponding author.
difference: signs are composed of various manual and non-manual features (Mann et al., 2010), such as hand gestures, facial expressions and limb movements (Liddell and Johnson, 1989; Johnson and Liddell, 2011; Sandler, 2012). The differences in phonological structure and means of expression create challenges in modeling the two languages. Data scarcity: multimodal high-quality sign language datasets are relatively scarce, and some datasets tend to be specific to a particular language or domain, e.g., American sign (Duarte et al., 2021), German weather (Forster et al., 2014; Camgöz et al., 2018). Furthermore, hearing impairments hinder pronunciation (Moeller, 2000; Yoshinaga-Itano, 2003), making it strenuous to collect sign video with aligned audio and usually resulting in the lack of auditory information. Previous researches (Zhang et al.; Camgöz et al., 2017; Hu et al., 2021b,a; Yin et al., 2022) primarily focused on sign language recognition, which identifies sign fragments as the corresponding sign language lexicons (e.g., gloss). Several work (Saunders et al., 2020, 2021a, 2022; Hwang et al., 2021; Walsh et al., 2022) manage the transition from gloss to sign sequences, yet the grammar of gloss can be perplexing for those without sign language training. Saunders et al. (2020, 2021b) can transcribe discrete words or phrases into continuous sign language se-
7241
quences. However, directly producing continuous signs from entire spoken sentences still remains more exploration and efforts.
To promote barrier-free communication between signers and speakers, we introduce a Multimodal Spoken Data-Driven Continuous Sign Language Production (MS2SL) framework (Fig. 1). MS2SL can animate sign keypoint sequences from either speech audio or text. In addition, to alleviate data demands, we adopt an embedding-consistency learning (ECL) strategy, which is inherently based on the reciprocity among modalities, to bolster the model training. Specifically, MS2SL initially employs pre-training models like CLIP (text) (Radford et al., 2021) and HuBERT (audio) (Hsu et al., 2021) to extract features from input. Subsequently, we utilize these features, serving as control conditions for the diffusion, to generate sign sequences. The attention mechanism (Vaswani et al., 2017) is employed to model the relationships among conditions, denoising steps, and sign movements. Besides that, ECL does not require the three modalities to coexist in the dataset. By learning a joint embedding space, inspired by ImageBind (Girdhar et al., 2023), ECL tightly binds the properties of different modalities and generates feedback signals to boost the training process. First, we utilize contrastive learning to bind audio and text in the embedding space. Then, we leverage the semantic consistency between co-occurring data to infer and reconstruct the embedding of missing modalitiy. The reconstruction error between the generated signs and groundtruth can be used to iteratively update MS2SL until convergence. ECL can foster cross-learning between different generation streams, allowing training even in the absence of certain modality. Furthermore, the inclusion of audio data not only enriches sample diversity and enhances multimodal comprehension but also assists in accurately capturing the expression and semantic content of sign language. We validate the effectiveness of our method across two prevalent datasets How2Sign (Duarte et al., 2021) and (Camgöz et al., 2018). Experimental results demonstrate that MS2SL achieves SOTA performance, both in terms of semantic consistency and sign accuracy. In conclusion, our primary contributions are outlined as follows: • We propose MS2SL, a unified diffusion frame-
work for efficient multimodal spoken to sign language production. MS2SL is able to directly convert entire speech or text sentences into corre-
sponding sign keypoints sequences. • We present an ECL strategy that leverages the
intrinsic relations to enhance data utilization. • We show that joint embedding is suitable for gen-
erative tasks that are prone to modality missing.

heading: 2 Related Work
text: Sign Language Understanding. Similar to spoken language, sign language follows specific linguistic rules (Sandler and Lillo-Martin, 2006; Brentari, 2011; Petitto et al., 2016; Sandler, 2017). Existing researches are primarily dedicated to sign language translation (SLT) and recognition. SLT typically involves translating sign language into spoken language (Camgöz et al., 2018; Coster et al., 2022; Camgöz et al., 2020; Lin et al., 2023). Sign language recognition (Adaloglou et al., 2022; Selvaraj et al., 2022) means interpreting and classifying of body movements in videos, covering isolated (Imashev et al., 2020) and continuous signs (Cui et al., 2017; Camgöz et al., 2018, 2020). SLP (Arkushin et al., 2023) is the process of creating sign sequences from spoken text, and can be seen as the reverse process of SLT. These existing studies on SLT and SLP primarily focus on converting between sign videos and discrete glosses, either directly or indirectly. A few of Text2Sign works (Saunders et al., 2020, 2021a,b) are grounded in datasets with relatively homogeneous scenario (Camgöz et al., 2018) and discrete spoken transcriptions. Diffusion Model. The diffusion model demonstrates exceptional proficiency in various generative tasks (Ho et al., 2020; Choi et al., 2021; Lugmayr et al., 2022; Avrahami et al., 2022). Beyond image generation, diffusion models also perform well in generating sequence data (Yuan et al., 2022; Wu et al., 2023). In recent years, some work has begun to apply diffusion models to SLP. By iteratively updating information, diffusion models can gradually infer the distribution of subsequent data, thereby providing more accurate and coherent results. Ham2Pose (Arkushin et al., 2023) leverages diffusion to animate HamNoSys, a lexicon of sign symbols, into sign keypoint sequences. Though impressive, Ham2Pose can only produce videos with a single sign symbol, falling short in conveying sentences with complete semantics. Cross-modal Consistency Learning. Deep learning often requires ample labeled data to work properly. However, the cost of collecting sign data is prohibitive and audio data is often lacking. Recent methods enhance model training by applying con-
sistency training to massive unlabeled data (Bachman et al., 2014; Sajjadi et al., 2016; Clark et al., 2018; Miyato et al., 2019). The principle of consistency learning, employing the cyclical duality between different tasks or data as feedback signals to regularize training (He et al., 2016), has its roots in the domain of language translation (Yi et al., 2017; Lu et al., 2017; Zhao et al., 2020). It primarily encompasses inter-task (dual-learning) and intra-task (cycle-consistency learning) varieties. Dual-learning simultaneously trains bidirectional mapping functions between tasks, creating a primal-dual pair where one function’s output approximates the input of the inverse function (Yi et al., 2017; Wang et al., 2022; Zhang et al., 2018; Shah et al., 2019; Wang et al., 2019; Zhao et al., 2020; Xie et al., 2020). Cycle-consistency learning is designed to enhance the self-reconstruction capabilities of samples produced intrinsically by the same model (Zhu et al., 2017; Almahairi et al., 2018; Rao et al., 2020; Mathew et al., 2020). However, these methods frequently emphasize the duality between two tasks or modalities, overlooking the interplay and mutual influence among multimodal data within the same task.
Limited studies focus on directly generating sign language sequences from entire spoken sentences. To our best knowledge, we are the pioneers in effecting this conversion. This study harnesses sequential diffusion models to incrementally generate noise predictions, enabling cross-modal sign language generation. With the help of ECL, MS2SL can generate various feedback signals even in the absence of co-occurring ternary data: assessing the reconstruction loss with the signs generated from the reconstructed audio embeddings.

heading: 3 Method
text: Assuming the triplets (A, T ,S) represent the audio, text, and sign space respectively, our goal is to learn the mapping from text or audio to sign within a unified framework (Fig. 2). Given a training dataset D={(a, t, s) ∈ A×T ×S}, MS2SL can realize text-to-sign T 7→S : s = G(t) and audio-to-sign A7→S : s = G(a), where G is the sign sequence diffusion generator. We initially employ pretrained models CLIP (Radford et al., 2021) and HuBERT (Hsu et al., 2021) to extract features from text t and audio a. Next, we employ three encoders Ea, Et, Es to encode these features, acquiring their embeddings ea, et, and es. Subsequently, drawing on the operating mechanism of diffusion
models, we employ a diffusion step encoder Eh and a sign noise encoder En to encode step h and noise n to eh and en, respectively. Finally, we utilize the generator G to produce the sign sequences: ŝt = G(et, eh, en) and ŝa = G(ea, eh, en).
The paucity of co-occurring triplet data renders the direct training of MS2SL a formidable task. To overcome this challenge, we develop a joint embedding space that facilitates the natural alignment of multimodal data. Furthermore, we employ ECL strategy to exploit the reciprocity among modalities within the embedding space, effectively furnishing feedback signals to boost the training.

heading: 3.1 Sign Predictor
text: Cross-linguistic Modeling. MS2SL aims to solve the problem of generating variable-length sequences across modalities. It necessitates phonological modeling between spoken and sign language, associating text and audio to the same target sign sequence. The causal attention mechanism can serve as a potent remedy for this challenging issue. Taking textto-sign as an example, we first concatenate the embeddings of text et, denoising step eh and noise en. Next, we apply the causal self-attention (Radford et al., 2018) to model the relationship among them. The mask in causal attention ensures that the model only processes past and present information, maintaining temporal and logical coherence in the output. As such, the output is computed as: CausalAtt[et; eh; en]. During inference, we initiate from the text embedding and produce indices autoregressively, ceasing generation when the model predicts the sequences. Likewise, the concatenated entity of the audio ea, step eh, and noise en can also undergo the causal attention to capture the relationship between audio and sign. In causal attention, we adopt the common practice of positional encoding, which can model keypoints and inter-frame context while capturing cross-modal relations. Thus, to simplify the model structure, we does not explicitly design a temporal module. Finally, we employ two fully connected layers to output the sign prediction ŝh for step h. Sign Language Production. We apply a diffusion model as the sign generator. Similarly, taking textto-sign as an example, the diffusion generator G is responsible for the gradually producing a continuous sign sequence ŝ. Diffusion generator G simulates data distribution through a gradual forward and reversible process (Ho et al., 2020), training by maximizing the evidence lower bound to approxi-
mate target distributions. Diffusion model aims to reconstruct the input from a latent variable. The forward process gradually transforms the input into noise by adding Gaussian noise. The reverse process starts from random noise and progressively removes the noise to recover the original data.
Common training for diffusion models involves independent noise prediction at each forward step h, potentially reducing sequence coherence and consistency. Following (Arkushin et al., 2023), we adopt the holistic training method. We apply a schedule function δh = 1/log(h+ 1) (δ ∈ [0, 1]) and a step size αh = δh − δh+1. The predicted signs ŝh at step h, as: ŝh = αhph + (1− αh)ŝh−1, (1) where the predicted signs ph at step h are given as G(t). This method utilizes the output from the previous iteration as the input for the subsequent step, gradually reducing the step size as the process continues. Each step combines previous outcomes with current predictions, reducing reliance on the initial noise. We also enhance training robustness by introducing a random noise to ŝh at each step. Finally, the predicted initial sign ŝ0 is outputted. The loss of the diffusion is defined as:
Ld = αhs0 + (1− αh)sh+1. (2)

heading: 3.2 Modality Binding
text: MS2SL operates in an aligned embedding space, typically dependent on audio, text, and sign data for tri-modal alignment. However, the difficulty for people with hearing impairments to perceive sound variations poses a challenge in recording these co-occurring triplets. Fortunately, ImageBind (Girdhar et al., 2023) reveals that a model can learn to align modalities in a joint embedding
space by employing contrastive learning (Hadsell et al., 2006). Training with (Image, Modality1) and (Image, Modality2) pairs can lead to a spontaneous alignment of Modality1 and Modality2 in embedding space. This alignment allows the model to excel in various tasks without requiring direct training on specific pairs of (Modality1, Modality2).
We extend the findings of ImageBind and construct a joint embedding space for the triplet dataset (A, T ,S), where MS2SL employs (text, sign) pairs as anchors to establish a cohesive space linking audio, text, and sign. Let’s explore a pair of modalities (T ,S) with aligned observations. Given a sign sequence s and its corresponding caption t. We first employ pretrained models CLIP (Radford et al., 2021) to extract textual features and encode them into normalized embeddings: et and es. Then, we leverages the paired modalities (T ,S) to align the text with sign. The corresponding encoders are optimized by InfoNCE (Oord et al., 2018) loss LT , S :
LT , S = −log exp(sim(et, es)/τ)∑M m=1 exp(sim(et, esm)/τ) . (3)
Within the mini-batch, we consider each instance, whose index is not equal tom, as a negative example. This approach aims to draw different embedding pairs closer within their joint embedding space. Similarly, we can also obtain LA, S and LT , A for the pairs (A,S) and (T ,A). Interestingly, we also observe the emergent alignment between modal pairs (T ,A) in our embedding space. This phenomenon can occur when the training is solely based on pairs (T ,S) and (A,S), a trend that mirrors the findings reported in (Girdhar et al., 2023). Accordingly, MS2SL is designed to mainly leverage modal pairs (T ,S) and (T ,A), circumventing
the need for triplet data. In practice, this is achieved by employing a triadic loss:
Lnce = LT , S + LT , A + LA, S . (4)
As such, the embedding space can not only spontaneously align unseen triples but also be used in reconstructing unobserved modalities in ECL.

heading: 3.3 Embedding-consistency Learning
text: Given a tuple (A, T ,S), we employ a cyclic approach with the bound joint embedding to generate feedback signals for bidirectional cross-learning, fostering model training. When triplet data is available, the encoders first extract features from their respective modalities. Then, audio and text independently generate predicted sign language sequences ŝa and ŝt. To fully utilize real data, we calculate ECL loss after 500 epochs of model training. The vanilla model, built on authentic data, guarantees minimal distribution differences between generated pseudo-embeddings and the original dataset. Semantic consistency is calculated using the embeddings êt and êa from encoder Es, which encodes the two predicted sequences. We can obtain the text-to-sign error ∆(êt, es) and the audio-tosign loss ∆(êa, es):
∆(êt, es)=∥êt, es∥2, ∆(êa, es)=∥êa, es∥2.
(5)
Evaluation scores are derived from comparing the two embeddings êt and êa. Both audio and text can receive feedback signals from the generative streams of each other. To compensate for the missing audio modality and ensure smooth processing, we use a mapping network M and text embeddings to generate pseudo audio features. The operation is conducted in the embedding space, thus minimally affecting inference speed. For unpaired natural audios U , we can get the formula:
L(T , A, S)=∥Es(G(ea))−Es(G(et))∥2, L(T ′, S′) =∥Es(M(G(e′t)))−Es(G(e′t))∥2. (6)
Then our ECL loss is defined as:
Lecl=L(T , A, S)∈D+ L(T ′, S′)∈U . (7) MS2SL translates entire spoken sentences into continuous sign language sequences. Overall, our total loss comprises three components, i.e., the diffusion model loss, ECL loss, and joint embedding loss:
L = λ1Ld + λ2Lecl + λ3Lnce, (8) where the cofficients are empirically set as λ1 = λ2 = λ3 = 1.

heading: 3.4 Implementation Details
text: Training. MS2SL takes speech audio or text as inputs. We utilize pre-trained models for encoding both speech and text, HuBert (Hsu et al., 2021) for speech and CLIP (Radford et al., 2021) for text. We first extract embeddings et, ea, es, eh, en through five encoders. We employ keypoints to represent signs, like the 137 human keypoints in How2Sign (Duarte et al., 2021), which are normalized and standardized before being input into the model. et, ea and es participate in learning the joint embedding space. Concurrently, et, ea, eh and en serve as conditions to control the generation of text-to-sign and audio-to-sign, respectively. Here, we adopt the common practice (Saunders et al., 2021a,b; Arkushin et al., 2023) of using the first sign pose as initial noise. The first 500 epochs skip the audio-to-sign generation flow in the absence of audio. After obtaining a vanilla model, we apply the mapping network M to transform et into ea to continue the training until the model converges. Since PHOENIX (Forster et al., 2014) dataset is in German sign language, and our pretrained model is primarily based on English, we utilize the penultimate layer features of CLIP along with MLP to align and transform between German and English. As for ECL, we incorporate cycles among the three modalities, namely audio-to-sign, text-to-sign, and audio-to-text, greatly enhancing the efficiency of data utilization. We adopt the commonly used exponential moving average (Cai et al., 2021) strategy with diffusion parameters (Cai et al., 2021) to ensure smoother, more robust training. For details, please refer to the supplementary. Inference. The model can perform SLP from audio or text independently. Inference for each modality involves executing the sequence sampling of the diffusion model. Taking text-to-sign as an example, the process starts with CLIP encoding the text into features. These text features are then fed into the sign predictor, which sequentially generates a sequence noise prediction. The completion of this sampling process results in the generation of the desired sign sequence. The process for generating signs from speech is similar. We take the average of twenty generations to mitigate deviation. Reproducibility. Our method is implemented using PyTorch on 2RTX 4090 GPUs, with a training time ofabout12hoursandanaverage inference timeof0.3 seconds. Following (Zhang et al., 2023), we remove data with word count exceeding 20.

heading: 4 Experiments
text: We evaluate the effectiveness of MS2SL under textto-sign and audio-to-sign settings.

heading: 4.1 Experimental Setup
text: Datasets. We conduct experiments on two continuous sign language datasets: • How2Sign (Duarte et al., 2021) is a challenging
multimodal American sign language dataset with a 16k-word vocabulary and comprehensive annotations. It includes 1, 176 entries with audio and has train/dev/test splits of 31165/1741/2357. • PHOENIX14T (Camgöz et al., 2018), a widely applied German weather sign language dataset, contains 2, 887 words, 1, 066 sign annotations, with train/dev/test splits of 7096/519/642.
Evaluation Metrics. Following (Saunders et al., 2020), we adopt back-translation approach for evaluating, i.e., we leverage the cutting-edge SLT model (Camgöz et al., 2020) to ingeniously translate back from generated signs to text. Subsequently, we calculate BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) scores, which are commonly used metrics for SLP and machine translation. We apply ROUGE-L F1-Score and report BLEU-1 to BLEU-4 for translation performance at different phrase lengths. Competitors. For text-to-sign generation stream, we consider four SOTA competitors:
• Ham2Pose (Arkushin et al., 2023), which employs transformer and diffusion model, animates HamNoSys (a sign notation) into sign poses.
• T2M-GPT (Zhang et al., 2023) combines VQVAE (van den Oord et al., 2017) and CLIP (Radford et al., 2021) for motion generation.
• PT (Saunders et al., 2020) translates discrete spoken sentences into sign sequences.
• MOMP (Saunders et al., 2021b) divides SLP into two sub-tasks: latent sign representation and animation imitation.
As for the audio-to-sign stream, since there are not specific methods, we extend MS2SL to multiple implementations for a thorough evaluation, including audio-to-sign, audio-to-text-to-sign, and text-to-audio-to-sign. For audio-to-text-sign, we apply WeNet (Yao et al., 2021) to translate audio into text, followed by the generation of signs. Conversely, for text-to-audio-to-sign, we employ DeepVoice (Gibiansky et al., 2017) to convert text into audio for subsequent sign generation.

heading: 4.2 Comparison to State-of-the-art
text: Quantitative Results. We present the comparative analysis results in Table 1 on How2Sign and PHOENIX14T test set. MS2SL demonstrates impressive gains against the four robust methods, establishing a new benchmark for SOTA performance. In the generation of text-to-sign, our approach yields a ROUGE of 14.67, marking a notable increase of 2.39 over its counterpart (T2MGPT, which has a 13.99 ROUGE). Furthermore, MS2SL combined with ECL surpasses the standalone by 1.28. How2Sign (Duarte et al., 2021) and PHOENIX14T (Camgöz et al., 2018) are datasets of different scales, demonstrating the robustness of our method and the burgeoning potential of diffusion models in generating long sign sequences.
Table 2 reports the audio-to-sign results on How2Sign, noting that PHOENIX14T is not included here due to the absence of audio data. Our method significantly enhance performance, achieving notable improvements (i.e., BLEU-1 increase from 9.49 to 11.77, ROUGE from 9.60 to 12.16). The ECL strategy also enhances ROUGE by 1.12. Considering the scarcity of audio modality data, this achievement is particularly noteworthy and shows its real-world applicability. We can also conclude that it is difficult to obtain a well-performing model by training solely with the limited audio data in How2Sign. This also highlights the urgency of utilizing non-co-occurring triplets.
Qualitative Comparison. Fig. 3 presents visual results on How2Sign (Duarte et al., 2021). It demonstrates that our method can produce signs that are more closely aligned with their semantic meaning. After meticulous examination, it is evident that MS2SL surpasses other models in generating actions with smoother transitions, heightens expressiveness, greater diversity, and superior adherence to physical constraints. Some noise and jitter are noted in the audio-to-sign generation stream. The main reason is that our method focuses on translating complete spoken content into sign sequences, whereas previous studies (Saunders et al., 2020, 2021a; Arkushin et al., 2023) target the creation of discrete lexical symbol or phrase. The challenge of training models to convey extended semantic con-
tent and long sequences often leads to incoherent movements during sign generation. User Study. Given the challenge of finding sign language experts, who require extensive training, we conduct a user study with 10 hearing volunteers. We ask the volunteers to compare sign sequences generated by different methods. We slow down sign sequence playback for easier comparison by volunteers. Volunteers select the sequence closer to the ground truth and assign a score. Our scoring range is from 1 to 5, with higher scores indicating closer proximity to the ground truth. Most participants report that the sign sequences generated by MS2SL are smoother and more accurate (Table 3). User feedback highlight the advantages of MS2SL in terms of expression clarity and pose accuracy.

heading: 4.3 Ablation Study
text: We conduct careful profiling of the impact of each module within MS2SL on How2Sign. Data in Different Modalities. We primarily conduct four experiments: audio-to-sign, text-to-sign, text-to-audio-to-sign, and MS2SL, to compare and analyze the role of different modalities. As shown in Table 4a, although direct generation from audioto-sign and text-to-sign can yield appropriate results, MS2SL significantly outperforms them. Removal of text data leads to a 6.29 decrease in BLEU-1, highlighting its crucial role. The mediating role of text leads to an increase 0.76 in ROUGE. Multimodal data yields superior results compared to its unimodal counterpart, enriching the learning process with more diverse information. Embedding Consistency Learning. We investigate the impact of the cyclical consistency training presented in § 3.3, and the results are illustrated in Table 4b. We note that common training method performs comparably to baseline models, while cyclical consistency boosts model performance akin to adding substantial training data. Compared to the alternative only with single modality, MS2SL approach shows a 1.12 increase in BLEU-2 and a 1.28 increase in ROUGE, demonstrating the synergistic effect of integrating data from multiple modalities. We further pay particular attention to the impact of dataset size. We also observe a direct correlation between dataset size and model accuracy. For smaller datasets (under 10k samples), the accuracy plateau around 15.5. Several insights can be drawn: i) Performances improve as more training data is used. ii) Over 10k unpaired data entries, the signs might be of good quality, but the model cannot further improve on a large scale, possibly due to the scarcity of audio. This trend shows that more data notably improves sequence generation, even without clear semantic boundaries. Diffusion Model. As shown in Table 4c, implementing the diffusion model lead to a significant enhancement. The quality metrics, such as BLEU1 and ROUGE, improved by 5.1 and 6.66, respectively, compared to non-diffusion model approach. Our study explores denoising steps ranging from 5 to 20, revealing a discernible trade-off between generation quality and computational efficiency. Compared to a fixed 10-step denoising process, the 20-step process unsteadily improve 0.78 in BLEU1 by approximately 5.3% with a disproportionate increase in computational load. Thus, in this paper, 10 is set as the default number of denoising steps. Pre-trained Models. We select four widely used models, including, CLIP (text), BERT (text) (Devlin et al., 2019), HuBert (audio) and WavLM (audio) (Chen et al., 2022), to assess their impact on performance. As shown in Table 4d, for audio-tosign generation, the impact of HuBert and WavLM on performance is minor, with negligible differences observed between the two pre-trained models. GPT outperforms CLIP models in text-related tasks, with a slight improvement of up to 0.14 in ROUGE. This may be because BERT focuses on natural language processing, leading to enhanced text understanding capabilities.

heading: 5 Conclusion
text: We explore a unified framework that combines diffusion and pretrained models to generate sign language from spoken depictions. We surpass other competitors and solidify this classic framework as a highly competitive method for SLP. MS2SL effectively handles diverse modalities of data for analysis and decoupling. Despite its advancements, our
model struggles with maintaining contextual flow in generation, and MS2SL cannot handle lengthy data, which is a future focus. Our research pioneers direct sign language generation from speech, offering some insights to advance the community.
Limitations
Despite significant advancements, our method still faces key technical limitations. First, the complexity and fluidity of authentic sign language are challenging to fully capture and reproduce, as it involves not just hand movements but also facial expressions, body language, and the speed of gestures. Moreover, converting text or speech into sign language involves complex natural language processing challenges, especially in handling grammar and semantics. Lastly, MS2SL struggles to effectively generate long sequences of key movements, limiting the coherence and completeness of sign language expression. These limitations indicate that, while the potential of sign language generation technology is immense, significant technical barriers still need to be overcome to achieve comprehensive and precise sign language communication. These are also the directions we are committed to addressing in the future.

heading: Acknowledgment
text: This work is supported by the National Natural Science Foundation of China (No. 62122035) and (No. 62372405), as well as by the CCF-Tencent Open Fund.

heading: A More Experimental Results
text: We conducte multiple experiments and report the results on PhoenixT (Camgöz et al., 2018), and we does not conduct comparative experiments for the audio-to-sign generation stream due to the absence of any audio data in PhoenixT. As shown in Table 7 and 8, we report the diagnostic results on PhoenixT. After integrating the diffusion process into MS2SL, we note modest enhancements. In particular, BLEU-1 score shows a notable improvement, rising by 1.88, and ROUGE score experience a increment of 2.11 (Table 7).
In the comparison of pre-trained models (Table 8), the conclusion is similar to that with How2Sign, indicating no significant differences among various text pre-training models. This is due to the relatively small dataset and vocabulary size of PhoenixT, for which the current models are sufficiently.

heading: B Architecture Details
text: The sign predictor, designed for predicting noise at each step h in the diffusion process (Nichol and Dhariwal, 2021; Arkushin et al., 2023), boasts a streamlined network architecture with several specialized modules. Table 9 details the parameter configurations of each module in MS2SL. Encoders. We employ a total of five encoders to process different types of input content. Each
encoder consists of two attention layers and a Multi-Layer Perceptron (MLP). Attention mechanisms (Radford et al., 2018) in each encoder enable the model to focus on the most relevant features of input data, enhancing its ability to extract and learn complex patterns. MLP further processes those focused information to generate embeddings et, ea, es, eh and en, introducing nonlinear transformations to add depth to the analysis and enabling the extraction of higher-level features. Producer. The producer is a central component of the model, responsible for synthesizing and outputting the final sign predictions. MS2SL utilizes the attention mechanism to learn the relationships between different input content, gathered and processed by the encoders. We utilize six multi-head attention blocks. Finally, we also use an MLP to transform the predicted features into coordinates for 137 sign keypoints in How2Sign (Duarte et al., 2021) and PHOENIX14T (Camgöz et al., 2018).
We also designed a length predictor to forecast the length of the generated sign language sequences. By accurately predicting the sequence length, the length predictor helps maintain the coherence and consistency of the model’s outputs, ensuring they are accurate not only in content but also in their temporal unfolding. To reduce the overall parameters of the model, we employed separate predictors for estimating the length of the input text and audio, respectively.
C Impacts Sign language production technology has significant impacts in both social and technological areas. Socially, it greatly enhances accessible communication, improving information access and interaction for deaf and hard of hearing individuals, especially in daily life, education, and work environments. It can foster social inclusiveness, aiding in the dismantling of communication barriers and facilitating the integration of the deaf community into broader society. SLP also serves as an educational tool, aiding deaf students in better understanding and absorbing information and facilitating the learning of sign language for hearing individuals. Technologically, the advancement of SLP drives progress in image recognition, natural language processing, and machine learning. This involves tackling challenges such as multimodal learning, text and audio comprehension, content generation, and data scarcity simultaneously. We conduct cyclic consistency learning on a joint embedding space, pro-
viding effective insights for niche domains. It aslo poses some potential risks, including insufficient accuracy, cultural nuances, and misinterpretations.

heading: Future Work
text: Despite its advancements, our
model struggles with maintaining contextual flow in generation, and MS2SL cannot handle lengthy data, which is a future focus. Our research pioneers direct sign language generation from speech, offering some insights to advance the community. Limitations
Despite significant advancements, our method still faces key technical limitations. First, the complexity and fluidity of authentic sign language are challenging to fully capture and reproduce, as it involves not just hand movements but also facial expressions, body language, and the speed of gestures. Moreover, converting text or speech into sign language involves complex natural language processing challenges, especially in handling grammar and semantics. Lastly, MS2SL struggles to effectively generate long sequences of key movements, limiting the coherence and completeness of sign language expression. These limitations indicate that, while the potential of sign language generation technology is immense, significant technical barriers still need to be overcome to achieve comprehensive and precise sign language communication. These are also the directions we are committed to addressing in the future