Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization
Authors: Cong Wang
Zexuan Deng
Zhiwei Jiang
Yafeng Yin
Fei Shen
Zifeng Cheng
Shiping Ge
Shiwei Gan
Qing Gu
Venue: NeurIPS 2025 spotlight ()
Keywords: Sign Language
Video Generation
Diffusion Models
Primary area: applications
TL;DR: We propose SignViP which incorporate multiple fine-grained conditions with a discrete tokenization paradigm for improved generation fidelity.

Abstract: Sign Language Video Generation (SLVG) seeks to generate identity-preserving sign language videos from spoken language texts. Existing methods primarily rely on the single coarse condition (e.g., skeleton sequences) as the intermediary to bridge the translation model and the video generation model, which limits both the naturalness and expressiveness of the generated videos. To overcome these limitations, we propose SignViP, a novel SLVG framework that incorporate multiple fine-grained conditions for improved generation fidelity. Rather than directly translating error-prone high-dimensional conditions, SignViP adopts a discrete tokenization paradigm to integrate and represent fine-grained conditions (i.e., fine-grained poses and 3D hands). SignViP contains three core components. (1) Sign Video Diffusion Model is jointly trained with a multi-condition encoder to learn continuous embeddings that encapsulate fine-grained motion and appearance. (2) Finite Scalar Quantization (FSQ) Autoencoder is further trained to compress and quantize these embeddings into discrete tokens for compact representation of the conditions. (3) Multi-Condition Token Translator is trained to translate spoken language text to discrete multi-condition tokens. During inference, Multi-Condition Token Translator first translates the spoken language text into discrete multi-condition tokens. These tokens are then decoded to continuous embeddings by FSQ Autoencoder, which are subsequently injected into Sign Video Diffusion Model to guide video generation. Experimental results show that SignViP achieves state-of-the-art performance across metrics, including video quality, temporal coherence, and semantic fidelity. The code is available at https://github.com/umnooob/signvip/.

Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization
Cong Wang
Zexuan Deng
Zhiwei Jiang"! Yafeng Yin
Fei Shen
Zifeng Cheng' Shiping Ge
Shiwei Gan Qing Gu
State Key Laboratory for Novel Software Technology. Nanjing University National University of Singapore
{cw, dewpeddenxxxmeee wee.aal.njuu........eeeeeea edu. cn {jzw Sizw,yafengee ennjnng@nju.. edu. cn .nw..du..uccc.mm..m {chengzf shipingge, (wr Ssmail nju. edu. guq@nju. edu. cn
shenfei296nus edu.sg
Abstract
Sign Language Video Generation (SLVG) seeks to generate identity-preserving sign language videos from spoken language texts. Existing methods primarily rely on the single coarse condition (e.g., skeleton sequences) as the intermediary bridge the translation model and the video generation model, which limits both the naturalness and expressiveness of the generated videos. To overcome these limitations, we propose Sign ViP, novel SLVG framework that incorporates multiple fine-gra une-grained conditions for improved generation fidelity. Rather than directly translating error- prone high-dimensional conditions, Sign adopts a discrete tokenization paradigm to integrate and represent fine-grained conditions (i.e., fine-grained poses and 3D hands). Sign ViP contains three core components. (1) Sign Video Diffusion Model is jointly trained with multi-con multi-condition encoder to learn continuous embeddings that encapsulate fine-grained motion and appearance. (2) Finite Scalar Quantization (FSQ) Autoencoder is further trained to compress and quantize these embeddings into discrete tokens for compact representation of the conditions. (3) Multi-Condition Token Translator is trained to translate spoken language text to discrete multi-condition tokens. During inference, Multi- Condition Token Translator first translates the spoken language text into discrete multi-condition tokens. These tokens are then decoded to continuous embeddings by FSQ Autoencoder, which subsequently injected into Sign Video Diffusion Model to guide video generation. Experimental results show that Sign ViP achieves state-of- offtthh-arttt performance across metries, including video quality, temporal coherence, and semantic fidelity. The code is available at https: / umnooob/ si gnvip/.
//tpp://gitaiii thub. com/
Introduction
Sign language, visual language, serves the primary communication medium for deaf individuals. Early research focused on Sign Language Recognition (SLR) [32, 45, 90], Translation (SLT) [5, 20, 42, 14]. or Production (SLP) [58, 56, 60, 86, 83]. More recently, Sign Language Vidco Generation (SLVG) [57, 61, 47, 69] has gained increasing attention, which aims to generate realistic and expressive sign language videos from spoken language texts, preserving the unique identity of target signer, as
"Equal contribution. Cooresppondig author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Reference Signerlmage
Sign Langoogg Video
Sign Language Video Generation (SLVG)
BediMtedd
Language
Single- Contle-Condttonn Method
(a) Liigle-Condition Method
(2)
Radi
Gooditand
Transiancr
Skiletoon
Gonorator
SignViP (Quna)
E Good
An Intuitive cutti-Condition Method
Fina-Gralned Posns
3D Handa
(4)
Ground-Truue Truth Fine Grained Conditiins
Sig (Ours)
3D Handa
Emoor Transtation
Translancr
Multi Cundiinn Tohens
Genecator
Fine Pre-Grained
Poses
Fine Ornined Poses
Compreus & Quantized
Hands
Directty-Transttee
(Mulli Condil-Cnnniinnn Method)
Figure 1: (1) The illustration of SLVG task. (2) The pipeline comparison between existing SLVG methods (i.e,, single-condition method and multi-condition method) and our Sign ViP. (3) Single- condition methods struggle to accurately capture the naturalness and expressiveness of sign language videos. (4) Multi-condition methods are prone to translation errors for fine-grained conditions.
shown in Figure 1(1) This growing interest driven by its potential applications in accessibility technologies, educational tools, immersive communication systems, etc.
SLVG presents challenging problem due to the lack of explicit spatial or temporal alignment between the input (i.e., the spoken language texts) and output (i.e., the sign language videos) modalities. To address this, current SVLG methods focus on leveraging synchronized auxiliary conditions as an intermediary to align these two modalities. As shown in Figure (2a),, most of the existing SLVG methods [57, 61] leverage skeletal sequences as an intermediary to bridge text-to-skeleton translation model (i.e., an SLP model) and skeleton-to- to-viido generation model. Because the generative model only guided by a single coarse condition (e.g., skeleton), such single-conditional methods struggle with the naturalness and expressiveness of the generated videos, particularly in capturing facial expressions and figure movements, as illustrated Figure 1(3).
Recent advancements in human video generation have shown that incorporating fine-grained con- ditions (e.g.. dense pose [84] or 3D models [9, 82]) or leveraging multiple conditions (e.g., depth combined with optical flow [85]) can substantially improve generative fidelity. Inspired by these, we consider whether multiple fine-gra grained conditions can be introduced enhance the quality and expressiveness of generated sign language videos. As shown in Figure 1(b),, one intuitive solution is extend the dingle-conditionn methods by considering multiple fine-gu ine-grained conditions as interme- diaries, where a multi-condition translation model can directly predict capable of achieving multiple fine-grained conditions. However, as shown in Figure 1(4), we observe that directly translating such attributes is challenging due to their high-dimensional nature and susceptibility to errors. This raises an important question: How can we overcome the challenges in the translation of multiple fine-grained conditions further advance SLVG?
To address the challenges, we propose Sign novel framework designed to advance SLVG by incorporating multiple fine-grained conditions for enhanced generation fidelity. As shown in Figure 1(2c), instead of directly translating -ishddmmnsional conditions from spoken language ViP texts, Sign ViP adopts a discrete tokenization paradigm to effectively integrate and represent these fine-grained conditions. Central this framework is the construction discrete multi-condition token space, which bridges fine-grained conditions (e.g., fine-grained poses and 3D hands) with the dynamics of sign language video frames. The framework consists three key components: (1) Sign Video Diffusion Model is jointly trained with multi-condition encoder using denoising loss to

---

generate continuous embeddings that encapsulate fine- greeggraind motion and appearance details; (2) Finite Scalar Quantization (FSQ) Autvencoder trained with reconstruction loss to compress and quantize the continuous embeddings into discrete tokens, enabling highly dense representation for the conditions; (3) Multi-Condition Token Translator is built upon an autoregressive model to translate spoken language text to discrete multi-condition tokens. During inference, the spoken language text is first translated into discrete multi-condition tokens by Multi-Condition Token Translator. These tokens are then decoded back into continuous embeddings by FSQ Autoencoder, which are subsequently injected into Sign Video Diffusion Model to guide sign language video generation. Our experimental results demonstrate that Sign ViP achieves state-of of-te-o--hheanne art performance across multiple evaluation metrics, including video quality, temporal coherence, and semantic fidelity.
Our main contributions are summarized as follows:
We introduce Sign novel framework for Sign Language Video Generation (SLVG) that incorporates multiple fine-graing reee-graned conditions for improved video quality and expressiveness.
We propose a discrete tokenization paradigm through the construction of discrete multi-condition token space to bridges fine-graine conditions with the dynamics of sign language video frames.
The experiments validate the effectiveness of Sign ViP, demonstrating state-of-the-art performance across diverse metrics.
2
Related Works
Sign Language Video Generation. Sign Language Video Generation (SLVG) aims to generate
sign language videos from spoken language texts. Early methods decompose the
task into two consecutive sub-tasks [57, 611, which are the text-to-skeleton translation (i.e., SLP)
and the Akeleton-to-vider generation. SignGAN [57] first employs transformer [76] with a mixture density formulation translate spoken language text skeletal sequence. Then, GAN-based [17] kkeleton-conditioned human synthesis model is introduced to generate sign language videos. FS-Net [61] extends SignGAN by predicting the temporal alignment to a continuous signing sequence. Because the single condition focuses solely capturing basic pose structures while neglecting fine-grained details, the generated videos tend to appear less natural and expressive. SignGen [47] seeks to address this limitation through a novel end-to-end pipeline that integrates multi-condition guidance, including optical flow, pose, and depth. However, SignGen suffers from training-inferent inconsistency, which leads to suboptimal results. In this paper, we aim to develop framework that leverages multiple fine-grained conditions to enhance the quality and expressiveness of generated sign language videos.
Human Video Generation. Human video generation has advanced significantly with the deep generative models. Early approaches, such as Pix2PixHD [80] and vid2vid [79], leveraged GANs [17] to generate realistic images and videos from the structured inputs. Several works have also explored the human pose generation, conditioning on the whole body [2, 38, 40, 63], face [10, 33], or hands [73, 35J. However, GAN-based methods often suffer from mode collapse and optimization challenges. robust alternative, producing high-quality images or videos with greater stability. Most prior diffusion-based approaches rely ControlNet [88] and OpenPose [7] to process each video frame independently. neglecting the temporal consistency and leading to the inevitable flickering artifacts. Pose-guided diffusion models [62, 29, 77, 91, 65, 66] addresses this issue by generating temporally consistent human videos while preserving appearance fidelity. Furthermore, recent research shows that incorporating fine-grained conditions, such as dense pose [84] or 3D models [9, 82), leveraging multiple complementary conditions, such as depth and optical flow [85], can significantly enhance generative fidelity. Building on these advancements, we aim to harness state-of-the thrtrr- diffusion-basee methods alongside multiple fine-grained conditions to advance SLVG further.
More recently, diffusion models [70, 28, 67, 78, 64, 68] have emerged
Methodology
3.1 Preliminary
Diffusion Models. As class of the generative models, the diffusion models [70, 28] consists of two processes, which are the diffusion process and the denoising process, respectively. In the diffusion
3

---

(2) FSQ Autoencoder
(3) Sign Video Diffusion Model
Murti- Condition
Murti-Condition Embeniiings
Govditio Guider
Takens
Sull- Altn, Layer
Car-1"
Layer
Com,, Layer
Multi-Condtion Encoder
Derniting U-4ert
Genevated
Noite
dg
d, [2T8
DexE
Decader
Decoder
Decodar
Hands Coaneee Conditions
GPT-2
Nut. imagge
CLIP
Hover
Miaer
Spoten Longunge Toxt
dy-1
(1) Mutti-Conditionn Taken Translator
Figure 2: Framework of our SignViP for sign language video generation (SLVG). (1) The spoken language text translated into the multi-condition tokens by Multi-Condition Token Translator. (2) These tokens are decoded by FSQ Autoencoder into multi-conditio embeddings, which are equivalent to the embeddings of multiple fine ane-grained conditions (i.e., fine fine-grained poses and 3D hands) generated by multi-condition encoder. (3) The embeddings are injected into Sign Video Diffusion Model to guide the generation of sign language videos.
process, the Gaussian noise is iteratively added to degrade the input sample over T steps until the sample becomes completely random noise. In the denoising process, a denoising model is used to iteratively generate sample from the sampled Gaussian noise. When training, given an input sample Xy and condition c, the denoising loss is defined as
Liennise = Ex,,,e ||ea c,t -6 -||²
Among them, X, = -Vaxx0 + VT the noisy sample at timestep i E [1,T], where at is a predefined scalar from the noise scheduler. is the added noise. Eg is the denoiser with the learnable parameters 0, which predicts the noise to be removed from the noisy sample. Latent Diffusion Models (LDMs) [55] stands out one of the most popular diffusion models. It performs the two processes in the latent space, which is encoded by a Variational Encoder (VAE) [31, 54].
Finite Scalar Quantization. Finite Scalar Quantization (FSQ) [41] is concise quantization tech- nique to compress continuous values into discrete values. FSQ can be an alternative of Vector Quantization (VQ) [18] in VQ-VAE [75]. Compared with VQ. FSQ does not suffer from code- book collapse and does not need complex machinery to learn expressive discrete representations. Specifically, given scalar E TR: from the encoded latent, the quantized discrete value by FSQ is
[z; (L- FSQ(=) A rnd( L)) E -.-11......111
1]-
Among them, md(-) is round function. a--) is sigmoid function. L is the predefined quantization level. Through this process, each value z is enumerated, leading to a bijection from z to an integer in {0, 1,..., , 1). For d-dimensional latent vector, the total codebook size is the product of T,; across all dimensions, resulting in I[ L; possible discrete representations.
3.2 Overview
Given a reference signer image x, and spoken language text T. SLVG aims to generate a sign language video X with f frames, where the signer performs sign language accurately aligned with the semantics of the spoken language text. Specifically, the SLVG task can be formulated as pa (X (x., T), where 0 parameters of the SLVG model.
Due to the lack of explicit spatial or temporal alignment between T and X, current SVLG methods focus on leveraging synchronized auxiliary conditions (e.g., skeletal sequence) as intermediary to align them. Such pipeline paradigm can be formulated as
po X x., T)
P8gm (X X C): Pen. (CIT).
(3)

---

Among them, C is the synchronized auxiliary conditions which spatially and temporally aligned with the target video frames. Oran parameters of the text-to-condition translation model, while 9gen is parameters of condition-to-videe generation model.
To address the generation quality issues caused by relying on single coarse condition, we introduce multiple fine-grained conditions as intermediaries, Specifically, we utilize fine-graines poses and 3D hands. Fine-grained poses capture the signer's body posture and facial expressions, while 3D hands provide detailed and accurate descriptions of hand movements, even in the presence of occlusions. To avoid directly translating arror-prone high-dimg sigh-dimensional conditions, Sign ViP employs discrete tokenization paradigm with effective integration and representation of these fine- fine-grained conditions.
As shown in Figure 2, the spoken language text T is first translated into the discrete multi-condition tokens do: 1 by the Multi-Condition Token Translator. The multi-condition tokens do:F-1 are then decoded by the FSQ Autoencoder continuous multi-condition embeddings e0:F-1, which are equivalent to the embeddings obtained from a multi-condition encoder that encodes multiple fine-grained conditions (i.e., fine-grained poses and 3D hands). Finally, F0 are injected into Sign Video Diffusion Model to guide the sign language video generation (i.e., animating the signer in reference image
x.). The overall pipeline of Sign ViP can be formulated as
pa xx, T) pm x., e0:F-1) Рrλs. (0.F-1/do.F-1) Pe... (do:F-1)T)
(4)
where 8 gen. AE, and ira denote parameters of Sign Video Diffusion Model, FSQ Autoencoder, and Multi-Condiing Token Translator, respectively.
3.3 Construction of Multi-Condition Token Space
Sign ViP is trained with three steps to construct the multi-condition token space for the discrete tokenization paradigm.
Step I. We train Sign Video Diffusion Model with a multi-condition encoder (i.e., a multi-layer convolution network) using a denoising loss establish a connection between the conditions and the sign language videos. Specifically, multiple fine-grained conditions (e.g., fine-grained poses and 3D hands) are encoded by the multi-condition encoder into the continuous multi-condition embeddings e- These embeddings, along with the reference image x., serve as the guidance signals for Sign Video Diffusion Model perform diffusion process. More details can be found in Section 3.4.
Step II. We train FSQ Autoencoder using a reconstruction loss to learn the compression and quantiza- tion of multi-condition embeddings F The encoder EFSQ of FSQ Autoencoder compresses and quantizes the embeddings F- into discrete multi-condition tokens do:r-1- while the its decoder Diso reconstructs Co.F-1 from do:f-1 -1- More details can be found in Section 3.5.
Step III. We train the
Token Translator to autoregressively translate spoken language text T to the multi-condition tokens d More details can be found in Section 3.6.
3.4 Sign Video Diffusion Model
Sign Video Diffusion Model aims to generate sign language videos in a diffusion- based manner [70, 28] under the guidance of the reference image x, and the continuous multi-condition embeddings Cu:F-1- Inspired by the previous works [29, ]. as shown in Figure 2(3), Sign Video Diffusion Model consists of three modules: Condition Guider, Denoising U-Net, and Reference Net. Condition Guider and Reference Net respectively encode the multi-condition embeddings Fp:F-1 and the reference image x, to guide the Denoising U-Net.
Denoising U-Net the backbone the Sign Video Diffusion Model, which mirrors the architecture of Stable Diffusion (SD) v v1.5 [55]. Each U-Net block includes a ResNet layer [22]. a self self-attention layer, and a temporal-attentie -tteentionn layer [21]. The self-attention layer and the cemporal-attentire layer perform attention operation [76] along the spatial axes and the temporal axis, respectively.
Condition Guider is lightweight guidance network that encodes Cu:F-1, whose each block consists of convolution layers and a temporal attention layer. The output feature of each block is added to the corresponding block's feature in the downsampling part of the Denoising U-Net.
Reference Net [29] shares the same architecture of SDv1.5 and operates parallel with the Denoising U-Net. The reference image x, is first encoded into the latent space by the VAE encoder EVAE.
5

---

z, = (x). The encoded reference latent z, is then fed into the Reference Net. The output feature of the self-attention layer in each block of the Reference Net is spatially concatenated with the input feature of the self-attention layer in the corresponding block of the Denoising U-Net.
During training. the loss function is the extended denoising loss of Equation
denoise
- =
(5)
Among them, Zo = EVAE Xo) is the target latent which is encoded from the target video Xo- Ea is the Denoising U-Net. r,, R(z.) ) is the reference features, which are encoded by the Reference Net R. E = C() is the conditional features, which are encoded by the Condition Guider C.
Considering that subtle pose variations in sign language videos carry important semantic meaning, the model needs to be robust to potential anomalies in the generated condition sequences. To address this, propose condition augmentation. Specifically, each condition frame has probability p of being randomly replaced with frames from other videos, deliberately introducing controlled disruptions in the temporal continuity. By exposing the model these artificial discontinuities during training, effectively enhance its robustness to unexpected conditional transitions.
The inference starts from the sampled Gaussian noise. Then, the diffusion scheduler (e.g., DDIM [71]) is applied to generate images with multiple denoising steps. For each inference step, the noise prediction relies on -lass Guidance (CFG) [27]. Finally, the generated video is achieved from the latent by a VAE decoder DVAE-
3.5 FSQ Autoencoder
The FSQ Autoencoder is designed to establish connection between the multi-condition embeddings 00:F-1 and the corresponding discrete tokens dg.r-1- The pre-trained multi-condition encoder first encodes multiple conditions to the continuous embeddings Co.F-1. These embeddings are subsequently compressed and quantized into do.F-1 by the FSQ Autoencoder encoder EFsQ, which provides compact representation for the Multi-Condition Token Translator. Finally, the FSQ Autoencoder decoder DrsQ dequantizes do and reconstructs -0. The training objective of the FSQ Autoencoder is an L2 reconstruction loss.
LISQLAE = Eeo.r-1 1 [Prss(Ersp(en.r-))) - 10.--112
(6)
The architecture of the FSQ Autoencoder follows that of the VAE. Instead of applying variational Bayesian inference in the latent space, it performs the FSQ operation, as illustrated in Equation 2.
3.6 Multi-Condition Token Translator
Multi-Conditee Token Translator is designed to translate the spoken language text T into the discrete multi-condition tokens dg:r-1- Since sign language videos often exceed 100 frames, and each frame should maintain coherent temporal relationships without strict internal order, we design frame-level autoregressive model.
As shown in Figure 2(1), following previous works [86, 3], the spoken language text T is firstly encoded by CLIP text encoder [49] to obtain semantic embeddings, which serve as the initial input hidden states of the GPT-2 model [48]. Each output hidden state of the GPT model is decoded through multiple parallel prediction heads to generate all tokens of the corresponding frame simultancously. On the input side, tokens belonging the same frame mixed to obtain unified input hidden states. Unlike methods that require dedicated modules for video length prediction [83] or rely on real video lengths [58], Conti-Conndiionn Token Translator naturally determines the video generation endpoint by producing an [[ESS]" token.
During training, given the pre-trained multi-condition encoder and FSQ Autoencoder encoder, the produced tokens are considered the ground-truth. The cross-entropy loss is computed between the predicted tokens bo.F and the ground-true tokens bo.F",
CAR
Σ 1=0
CrossEntropy(bb:bb b.).
F+1
Note that use F' instead of F here due to the inclusion of the additional endpoint token " "[EOS]".

---

To mitigate the exposure bias issue between training and inference, we employ a scheduled sampling strategy, wherein 40% of the imput tokens are randomly replaced with arbitrary indices from the vocabulary during training. This approach improves the model's robustness and generalization performance during inference.
4 Experiments
4.1 Experimental Settings
Datasets. We employ two sign language datasets for experiments. (1) RWTH-2014T [5] is German sign language dataset. It comprises 8,257 sign language videos. The dataset divided into 7,096 training samples, 519 validation samples, and 642 test samples. To align with the 8x downsampling rate of VAE, the frame size was resized from 260x210 to 272x224. (2) How2Sign [11] is an American sign language dataset. It includes 2,456 sign language videos. Using the provided timestamps, we segmented the videos to create a sentence-level dataset. This dataset consists of 31,128 training samples, 2,322 test samples, and 1,741 validation samples. The frame size is set to 512x512.
Evaluation Metries. To evaluate semantic consistency, we utilize the back-transkan metrics following ProTran [58]. Specifically, we train an SLT model [6] to translate sign language videos or poses back into texts. The back-translated text is then compared with the ground-truth text with metrics of BLEU [44] and ROUGE-L [34]. To provide a more comprehensive evaluation of back-translane texts, we further employ the COMET [52, 51] metric, which is specifically designed to predict human judgments of machine translation quality. COMET is widely used for machine translation tasks [19, 1, 24] and is considered more suitable than BLEU and ROUGE. To evaluate the video quality, we employ FID [26), CLIP-FID, FVD [74], and Identity Similarity (IDS). Among them, CLIP-FID is variation of FID that utilizes CLIP [49] embedding as the frame's ' embedding. IDS measures the identity consistency between generated and ground-truth videos. It calculates the cosine similarity of face embeddings extracted using YOLOSFace [46] and Arc2Face [43]. To further investigate the generative capability of video diffusion models in addition to FVD [74], we employ frame-Ievel level metrics including PSNR [13], SSIM [81], and LPIPS [89], leveraging their suitability in
scenarios where ground ground-truth videos are temporally aligned with the generated videos. Additionally, we introduce Hand SSIM, which measures SSIM specifically in the hand region for a more precise evaluation of the hand quality.
Implementation Details. In Multi-Condition Token Translator, we utilize a multilingual ver- sion of the CLIP model* to enable handling of multiple spoken language texts effectively. In FSQ Autoencoder, the encoder and decoder follow the architecture of their counterparts VAE. Specifically, FSQ Autoencoder applies 4 latent channels, with each channel having quantization level of 5. Together, this results in total vocabulary size of 625, computed 5 625 due the combination of levels across all channels. In Sign Video Diffusion Model, both the Denoising U-Net
and the Reference Net are initialized with Stable Diffusion v1.5². The temporal-attention layers in the Denoising U-Net are initialized from DifmateDiff [21]. ]. The condition augmentation rate is set to 0.001. During inference, Sign Video Diffusion Model utilizes guidance scale of 3.5 for CFG. Additionally, the number of inference steps is configured to 50.
Training Details. The training of the three stages are conducted 4 NVIDIA RTX A6000 GPUs using Adam optimizer [30], with each stage consisting of 50,000 training steps. The batch sizes of stage I, II, and III are 2, 16, and 16. Their learning rates are 1e-5, 5,-5, and 1e-6.
4.2 Comparison
TrennTransltton Comparison. To quantitatively evaluate the semantic accuracy of the generated sign language videos, we perform two types of bickttranslation comparisons. Specifically, we respectively train a video-to-tex translation model and a pose-to-text translation model to compare with SLVG methods and SLP methods [6]. For pose back-translation comparison, we extract pose sequences from the generated videos using OpenPose [7] and 2D-to-3D mapping method [87]. As
hnttpe://hugginggaace ctceeee.oooo//eeeeeeeeoeee co/sentence- htt.:atteeeeeeaaaaaaoootar////// 3----mmm111-------mmm1-1--1---1- 32-multilingull-1
7

---

Table 1: Comparison of video back-translation performance.
RWTH-2014T
How2Sign
BLEC-1
BLEU-2
BLEU-3
BLEU-4
ROUGE
COMET
BLEU-1
BLEU-2
BLEU-3
BLEU-4
ROUGE
COMET
Ground-Tree Truth
33.06
20.81
15.00
11.90
34.27
0.6157
20.37
13.11
9.78
7.55
21.43
0.5882
McMP [S9] ControlNet [8S]
19.12
8.95
533
3.61
21.54
0.5033
12,53
5.59
3.48
2,31
13.72
0,5122
McMP [59] AnimiteAnyone [29]
20.05
8.79
0.5091
13.65
2.25
14.15
0,5208
SigAGAN [6]]
17:41
7.93
4.67
3.16
19.64
0.4977
10.66
4.62
2.92
1,97
11.76
0.5104
AnmoteAAyooe 129]
18.29
7:25
19.70
0.4928
1,92
12.12
0.3135
SignGen (47)
13:28
3.05
1.13
0.51
10.13
0.4086
8.21
1.91
0.64
0.41
9.54
0.4127
Vig (Ours)
26.72
15:65
11:14
8.65
28.85
0.5608
16.21
9.36
6.28
5.04
16.99
0.5534
Table 2: Comparison of pose back-franslation
perforaoa performance.
RWTH-2014T
How25ign
DLEU-1
BLEU-2
BLEU-3
BLBU-
ROUGE
COMET
BLEU-I
BLEU-2
BLEU-3
BLEU-
ROUGE
COMET
Ground
30.99
18.36
12.83
31.00
0.5978
10.31
24.85
0.6250
[58]
17.96
8.90
5.64
4.017
20.97
14:57
7.47
4.59
0.5549
Adversarial [56]
17:70
8.96
5.72
21.15
14.76
7.15
4.66
3.48
17.84
0.5618
MIN 160]
18.06
9:30
6.06
4.52
21.44
0.5251
14.94
7.54
5.10
3.67
18.21
0.5085
MoMP (59)
20:55
10:98
7.02
5.14
23.75
0.5466
16.57
8.47
5.38
4.16
19.38
0.5882
SigaGAN [6]]
12.14
6.10
3,88
2,85
14.79
0.5123
10.86
5:27
3,30
2.62
13.19
0,5073
AnimateAmyone
12.36
6.23
14.93
5.36
3.39
0.5315
SignGen
10:42
2.42
12.68
0.4413
Sign ViP
21.94
10.06
22.67
5.41
0.5738
shown in Table 1, video tack-translation comparison, Sign outperforms all competing meth- ods, including SignGAN [6]], its enhanced version using AnimateAnyone [29], and SignGen [47]. These results validate Sign ViP as a more reliable solution for SLVG by effectively preserving se- mantic consistency. As shown in Table 2, in pose buack-trasslttin comparison, Sign consistently outperforms previous SLVG methods and SLP methods (i.e., Tran [58], Adversarial Training [56], and MDN [60]) across most evaluation metries. Although MoMP [59] achieves slightly higher scores than our SignViP ViP on certain metrics, our method remains highly competitive overall. is worth noting that these SLP baselines translate text directly into pose sequences, which aligns with our pose tockktrraslation evaluation pipeline. In contrast, our SLVG method requires detecting poses from the generated videos, potentially introducing additional errors that could impact evaluation results. To enable a more fair comparison under the SLVG setting, we further combine the state-of state-of-thhe-rr the-art SLP method, MoMP, with a tose-to-video generation approach (i.e., ControlNet [88] Animate one [29]). As shown in the first two rows of Table 1, the video back-translation results demonstrate that our method is better suited for the SLVG task compared to MoMP-based methods. These results further underscore Sign ViP's effectiveness in preserving semantic accuracy.
Video Quality Comparison. Table 3 summarizes the video quality comparison of the generated sign language videos. Our proposed Sign ViP method significantly outperforms prior SLVG approaches across all evaluated metrics. Specifically, the lowest FID, CLIP-FID, and FVD achieved by model demonstrate its superior ability to generate sign language videos that are not only visually realistic but also exhibit high temporal coherence and natural motion consistency. Furthermore, the highest IDS scores achieved by our method highlight its effectiveness in accurately preserving the identity of the signer. These results collectively validate the efficacy of Sign ViP in producing high-fidelity, visually coherent, and perceptually realistic sign language videos.
Generative Capability Comparison for Video Diffusion Models. To compare the generative capabilities of different video diffusion models, we evaluate three methods, which are ControlNet [88], AnimateAnyone [29], and our Sign Video Diffusion Model. As detailed in Table 4, our Sign Video Diffusion Model consistently outperforms other methods across all metrics. Specifically, our Hand model's ability to preserve hand details. The results
SSIM outperforms others, highlighting clearly highlight the superiority of our method.
Qualitative Comparison. We present the qualitative results in Figure 3(a) of the previous SLVG methods and our SignViP. Compared the previous methods, Sign generates higher-quality sign language videos while maintaining greater semantic accuracy with the spoken language text.
8

---

Table 3: Comparison of video quality.
KWIH-20141
How2Sign
FID
CLIP-FID
FVD
IDS
FID
CLIP-FID
IVD
ID5
SigsGAN [6]]
547.90
167.70
1431. 58
0.463
667.44
210.11
2766.97
0.538
Anyoe [19]
161.97
1330.54
215.05
0.533
SigaCion (47]
184.66
1715.32
0.515
815.00
3538.49
0.539
SignViP
154.10
575.67
Table 4: Generative capability comparison of video diffusion models.
XWTH-20141
2ow25igu
FVD L
SSIM
PSNR
TPIPS 4
Hand SSIM 1
FVD 4
1
PSNR 1
1.PIPS L
Hand SSIM
ControlNet (88)
0.784
19:50
0.137
0.483
427.22
0.826
21.32
AnimsteeAryome [29)
365 42
0.794
20.06
0.121
0.505
293 18
0.821
21.54
0.103
0.663
Sigs Video Diffosion Model (Ours)
275.22
0.829
22.91
0.089
0.614
210.63
0.A55
23.11
0.074
0,752
4.3 Model Study
Identity Generalization. Figure 3(b) showcases how SiP generalizes signer identities by
adapting appearance guidance from distinct reference images. This demonstrates the robustness of our method in preserving signer-specific appearances while ensuring accurate sign language translation.
Effect of Multiple Conditions. To evaluate whether incorporating multiple fine-grained conditions im- proves video quality, we ablate the fine-grained poses
Table 5: Effect of multiple conditions.
SSIM
BLEU-
Hands
382.66
0.477
5.39
and 3D hands from our pipeline, respectively. As shown in Table 5, removing one of the conditions leads to a substantial performance degradation across all metrics. These results demonstrate that incorporating multiple fine-grained conditions is essential for enhancing both the semantic accuracy and visual quality of the generated videos.
wob Fine- Grained Powes
461.90 98
0,488
3,13
19.41
Muttiple Conditions
275.22
0.614
Effect of Compression. To investigate the necessity of compression for SiP ViP, we conduct experiments to assess the impact of the compression/owwssapping rate in the FSQ Autoencoder. As illustrated in Figure 4(a), the performance of tack-translation improves notably as the compression rate increases. Notably, when the compression rate is set to (i.e., no compression is applied), the model demonstrates significantly poor performance. These results underscore the critical role of compression in enhancing the effectiveness of Sign
Effect of Quantization. To investigate the necessity of quantization for Sign we conduct an experiment where FSQ is not performed during FSQ Autoencoder, while Multi- Multi-Condition Token Translator is trained with continuous embedding prediction. As illustrated in Figure 3(c), we observed that continuous embedding prediction poses significant challenges for the translator, resulting in weak semantic alignment and low video quality. When incorporating FSQ, we achieve substantial ViP. improved performance. These findings highlight the importance of quantization for our ViP.
Effect of Condition Augmentation. To evaluate the impact of condition augmentation (Section 3.4) on generation quality, we conducted experiments by varying the augmentation probability p. Fig- ure 4(b) presents the results of condition augmentation with varying values of p. Specifically, introducing small probability of augmentation (i.e.,) 10-3) slightly improves FVD and ROUGE- scores, suggesting enhanced video quality and linguistic consistency. However, as p increases further, the effectiveness of condition augmentation diminishes. These results indicate that excessive augmentation introduces too much randomness, impacting both video quality and textual coherence.
Effect of Scheduled Sampling Strategy. To evaluate the effect of varying the sampling ratio r of the scheduled sampling strategy (Section 3.6) on generation quality, we conducted experiments by varying r. The experimental results, as shown in Figure 4(c), reveal that the scheduled sampling strategy significantly impacts the quality of generated outputs. When r = 1, meaning input tokens are replaced with random indices, the results indicate that excessive randomness severely hurts the model's 's consistency and coherence. As r decreases, generation quality improves steadily. The best performance is observed at 0.4. However, is further reduced to 0.2, performance begins to

---

SigIVP
SigidAN
SignGAN
SignDen
R.C0)
34.27
28. 45
13,59
R.(P)
22.67
13.23
(a)
(b)
(c)
Figure 3: (a) Qualitative comparison RWTH-2014T dataset. (b) Visual examples illustrating our SignViP' capability for identity generalization. (c) Effect of quantization. "R. (V) and "R. (P)" mean ROUGE metries of video and pose back-translations as shown Table and Table respec- tively. "FVD" evaluates the protocol described in Table 3.
Codetoo Uooge (N0)
Rouie
ELOU
PS0
5--1 (b)
0.
2X
Compr Rata
Rate
(27)
Codabook Size [e]
Figure 4: (a) Effect of compression rate. (b) Effect of condition augmentation probability. Note that "FVD" evaluates the protocol described in Table 4. (c) Effect of sampling rate for the scheduled sampling strategy. (d) Codebook usage comparison between FSQ and VQ. (e) Multi-conditional reconstruction loss comparison between FSQ and VQ.
decline slightly. This suggests that very low replacement ratio is insufficient simulate the diverse distributions encountered during inference, leading to suboptimal performance.
FSQ vs. VQ. To compare the performance of FSQ with traditional Vector Quantization (VQ), evaluate both methods in terms of codebook usage efficiency and multi-conditional reconstruction loss. (1) Codebook Usage. To assess the efficiency of codebook utilization, conducted experiments with varying codebook sizes ranging from 27 212 and the results are summarized in Figure 4(d). The results demonstrate that FSQ consistently achieves high codebook usage rates, remaining above 97% even with larger codebook. In contrast, VQ experiences sharp decrease in usage as the codebook size increases. These findings highlight the stability and scalability of FSQ. (2) Reconstruction Loss. To evaluate the ability of conditional preservation, we measured the reconstruction loss of both methods under different codebook sizes, as detailed in Figure 4(e). The results show that FSQ achieves consistently lower reconstruction loss compared to VQ, demonstrating its superior capability in preserving original conditional structure.
Conclusion
In this work, we propose Sign novel Sign Language Video Generation (SLVG) framework that incorporates multiple fine finennrained conditions to enhance generation fidelity by adopting a discrete tokenization paradigm. Sign ViP consists of three components: (1) Sign Video Diffusion Model, which learns continuous embeddings encapsulating fine- finengrained motion and appearance details, (2) FSQ Autoencoder, which compresses and quantizes these embeddings into discrete tokens for compact representation, and (3) Multi-Conditian Token Translator, which translates spoken language text to diserete multi-condition tokens. Experimental results demonstrate that Sign achieves state-of-the-e performance in video quality, temporal coherence, and semantic fidelity.
10

---

Acknowledgments
We would like to thank the anonymous reviewers for their insightful comments. This work is supported by the JiangSu Natural Science Foundation under Grant No. BK20251 1999; the National Natural Science Foundation of China under Grants Nos. 62172208, 62441225, 61972192; the Fundamental Research Funds for the Central Universities under Grant No. 14380001. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
[1] Duarte M Alves, José Pombal, Nuno M Guerreiro, Pedro Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. Tower: An open multilingual large language model for translation-related tasks. arXiv preprint arXiv:2402. 177333 2024.
[2] Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag. Synthesizing images of humans in unseen poses. In CVPR, 2018.
[3] Vasilcios Baltatzis, Rolandos Alexandros Potamias, Evangelos Ververas, Guanxiong Sun, Jiankang Deng, and Stefanos Zafeiriou. Neural sign actors: diffusion model for 3d sign language production from text. In CVPR, 2024.
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512,, 2023.
[5] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In CVPR, 2018
[6] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In CVPR, 2020.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEFE conference on computer vision and pattern recognation, pages 7291-7299, 2017.
[8] Zifeng Cheng, Zhonghui Wang. Yuchen Fu, Zhiwei Jiang, Yafeng Yin, Cong Wang, and Qing Gu. Contrastive prompting enhances sentence embeddings in ilms through inference-time steering. arXiv preprint 2505: 12831, 2025.
[9] Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cris- tian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. arXiv preprint arXiv:2403.08764 2024.
[10] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In CVPR, 2020.
[11] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DaH Florian Meize, Jordi Torres, and Xavier Giro-i Nieto. How2sign: large-scale multimodal dataset for continuous american sign language. In CVPR, 2021.
[12] Stefan Elfwing. Eiji Uchibe, and Kenji Doya. Sigmoid linear units for neural network function approximation reinforcement learning. Neural networks, 107:3-11, 2018.
[13] Fernando A Fardo, Victor H Conforto, Francisco C de Oliveira, and Paulo S Rodrigues. A formal evaluation of psnr as quality measurement parameter for image segmentation algorithms. arXiv preprint arXiv:1605.0711 2016.
[14] Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Lei Xie, and Sanglu Lu. Skeleton-aware neural sign language translation. In ACM MM, 2021.
[15] Shiping Ge. Qiang Chen, Zhiwei Jiang, Yafeng Yin, Liu Qin, Ziyao Chen, and Qing Gu. Implicit location- caption alignment via complementary masking for weakly-superviae dense video captioning. In AAAI, 2025.
[16] Shiping Ge, Zhiwei Jiang, Yafeng Yin, Cong Wang, Zifeng Cheng, and Qing Gu. Fine- sine-grained alignment network for zero-shot cross cross-modal retrieval, ACM Transactions on Multimedia Computing, Communica- tions and Applications, 2025.

---

[17] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 2014.
[18] Robert Gray. Vector quantization. IEEE Assp Magazine, 1984.
[19] Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. Hallucinations in large multilingual translation modcls. Transactions of the Association for Computational Linguistics, 11:1500-15 1517, 2023.
[20] Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. Hierarchical Istm for sign language translation. In AAAI, 2018.
[21] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Mancesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized lext-to-image diffusion models without specific tuning. arXiv preprint arXiv.230..04755, 2023.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[23] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211. 13221, 2022.
[24) Zhiwei He. Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang. Rui Wang. Zhaopeng Tu, Shuming Shi, and Xing Wang. Exploring human-like translation strategy with large language models. Trunsactions of the Association for Computaaional Linguistics, 12:229-246, 2024.
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 1606.08415, 2016.
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[27] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
[28] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[29] Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In CVPR, 2024.
[30] Diederik Kingma, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[31] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114. 2013.
[32] Oscar Koller, Sepehr Zargaran, and Hermann Ney. Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent enn-hmms. In CVPR, 2017.
[33] Marek Kowalski, Stephan Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, and Jamie Shotton. Config: Controllable neural face image generation. In ECCV, 2020.
[34] Chin- Yew Lin Franz Josef Och. Automatic evaluation of machine translation quality using longest. common subsequence and skip-bigram statistics. In ACL, 2004.
[35] Yahui Liu, Marco De Nadai, Gloria Zen, Nicu Sebe, and Bruno Lepri. Gesture-to toeture-o-gesture translation in the wild via category-inddependent conditional maps. In ACM MM, 2019.
[36] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NewrIPS, 2022.
[37] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver+4: Fast solver for guided sampling of diffuusion probabilistic models, arXiv preprint arXiv:221 1.0009,, 2022.
[38] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Lue Van Gool. Pose guided person image generation. NeurlPS, 2017.
[39] Yue Ma. Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-vidd generation using pose-free nise-free videos. In AAAI, 2024.
12

---

[40] Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, and Zhouhui Lian. Controllable person image synthesis with attrbble-decomposeed gan. In CVPR, 2020.
[41] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint 2rXiv:2309.15500 2023.
[42] Alptekin Orbay and Lale Akarun, Neural sign language translation by learning tokenization. In FG, 2020.
[43] Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Berhhard Kainz, and Stefamos Zafciriou. Arc2face: A foundation model for id-consistent human faces. In ECCV, 2024.
[44] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. ACL, 2002.
[45] Junfu Pu, Wengang Zhou, and Houqiang Li. Iterative alignment network for continuous sign language recognition. In CVPR, 2019.
[46] Delong Qi, Weijun Tan, Qi Yao, and Jingfeng Liu. YoloSface: Why reinventing a face detector. In ECCV, 2022.
[47] Fan Qi, Yu Duan, Huaiwen Zhang, and Changsheng Xu. Signgen: End-to-end sign language video generation with latent diffusion. In ECCV, 2024.
[48] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask leameers. OpenAl blog, 1(8)9, 2019.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In ICML, 2021.
[50] Prajit Ramachandran, Barret Zoph, and Quac V Le. Swish: a self-gated activation function. arXiv preprint arXiv: 170.55171, 7(1):,, 2017.
[51] Ricardo Rei. Craig Stewart, Ana C. Farinha, and Alon Lavie. Comet: A neural framework for evaluation. arXiv preprint arXiv:2009.09025 2020.
[52] Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation ( (WMT), pages 578-585, 2022.
[53] Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, and Abhinav Shrivastava. Customize-a-vider One-shot motion customization of text-to-video diffusion models. In ECCV, 2024.
[54] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In ICML, 2014.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[56] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Adversarial training for multi-channel sign language production. arXiv preprint arXiv:2008. 22405 2020.
[57] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Everybody sign now: Translating spoken language to photo realistic sign language video. arXiv preprint urXiv:2011.098-66 2020.
[58] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Progressive transformers for end-to-end sign language production. In ECCV, 2020.
[59] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Mixed signals: Sign language production via mixture of motion primitives. In CVPR, 2021.
[60] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Continuous 3d multi-channel sign language production via progressive transformers and mixture density networks. IJCV, 2021.
[61] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Signing at scale: Learning to co-articulate signs for large-scale photo-realistic sign language production. In CVPR, 2022.
[62] Fei Shen and Jinhui Tang. Imagpose: A unified conditional framework for pose-guided person generation. NewrIPS, 2024.
13

---

[63] Fei Shen, Hu Ye. Jun Zhang, Cong Wang. Xiao Han, and Wei Yang. Advancing pose-guided image synthesis with progressive conditional diffusion models. arXiv preprint arXiv:2310.06313, 2023.
[64] Fei Shen, Xiaoyu Du, Yutong Gao, Jian Yu, Yushe Cao, Xing Lei, and Jinhui Tang. Imagharmony: Controllable image editing with consistent object quantity and layout. arXiv preprint arXiv:2506.01949, 2025.
[65] Fei Shen, Xin Jiang. Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinhui Tang. Imagdressing-vl: Customizable virtual dressing. AAAI, 2025.
[66] Fei Shen, Cong Wang, Junyao Gao, Qin Guo, Jisheng Dang, Jinhui Tang, and Tat-Seng Chua. Long-term talkingface generation via motion-prior conditional diffusion model. arXiv preprint arXiv25502009333, 2025.
[67] Fei Shen, Hu Ye, Sibo Liu, Jun Zhang, Cong Wang, Xiao Han, and Yang Wei. Boosting consistency in story visualization with rich-contextual conditional diffusion models. In AAAI, 2025.
[68] Fei Shen, Jian Yu, Cong Wang. Xin Jiang, Xiaoyu Du and Jinhui Tang. Imaggarment-Iy I: Fine-grained garment generation for controllable fashion design. arXiv preprint 1xXi:22504.13770 2025.
[69] Tongkai Shi, Lianyu Hu, Fanhua Shang, Jichao Feng. Peidong Liu, and Wei Feng. Pose-guided Line-grained sign language video generation. In ECCV, 2024.
[70] Jascha Dich-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, In JCML, 2015.
[71] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[72] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
[73] Hao Tang, Wei Wang. Dan Xu, Yan Yan, and Nicu Sebe. Gesturegan for hand gesture-to-gesture translation in the wild. In ACM MM, 2018.
[74] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 1rXiv:181..01717. 2018.
[75] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, NewrlPS, 2017.
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
[77] Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fci Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406. 02511, 2024.. 2024.
[78] Cong Wang. Kuan Tian, Yonghang Guan, Fei Shen, Zhiwei Jiang, Qing Gu, and Jun Zhang, Ensembling diffusion models via adaptive feature aggregation. In ICLR, 2025.
[79] Ting- Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-vides synthesis. arXiv preprint arXiv: 1808. 06601, 2018.
[80] Ting- Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
[81] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. JEER TIP, 2004.
[82] Mengting Wei, Yante Li, Tuomas Varanka, Yan Jiang, Licai Sun, and Guoying Zhao. Magicportrait: Temporally consistent face reenactment with 3d geometric guidance. arXiv preprint arXiv:2504.21497, 2025.
[83] Pan Xic, Qipeng Zhang, Peng Taiying, Hao Tang, Yao Du, and Zexian Li. G2p-ddm: Generating sign posc sequence from gloss sequence with discrete diffusion model. In AAAI, 2024.
[84] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia- Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024.
Jia-Wei Wei Liu, Chenxu Zhang, Jiashi Feng, and
14

---

[85] Jingyun Xue, Hongfa Wang, Qi Tian, Yue Ma, Andong Wang. Zhiyuan Zhao, Shaobo Min. Wenzhe Zhao, Kaihao Zhang, Beungg Shum, al. Follow-your-pose v2: Multiple-onnditinn guided character image animation stable pose control. arXiv preprint arXiv:2406.0300 2024.
[86] Aoxiong Yin. Haoyuan Li, Kai Shen, Siliang Tang, and Yueting Zhuang. T2s-gpt: Dynamic vector quantization for autoregressive sign language production from text. arXiv preprint arXiv:2406.07119 2024.
[87] Jan Zelinka and Jakub Kanis. Neural sign language synthesis: Words are our glosses. In WACV, 2020.
[88] Lvmin Zhang, Anyi Rao. and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. CVPR, 2023.
[89] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018.
[90] Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for continuous sign language recognition. AAAI, 2020.
[91] Jingkai Zhou, Benzhi Wang, Weihua Chen, Jingqi Bai, Dongyang Li, Aixi Zhang. Hao Xu, Mingyang Yang, Fan Wang. Realisdance: Equip controllable character animation with realistic hands, arXiv preprint wxXi:2449.06202. 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in supplemental material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Our paper focuses on experimental work; therefore, it does not include theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the information reproducibility can be found in Section "Experiments" and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The supplementary material includes code and data. We will open-source our work after acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
wttpc//guiese/ooeeuuue wommiiisssiiiiiyy.o..oo.i for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppp.ccppbbbccccc gui des/ Codeeubmi ssi asionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setting and details be found in Section "Experiments" and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
The experiments support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We claim the compute resources in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nwurips.sp.. oc cc/public. haabblic/ttiiceee Bee//thhiissuuie cww.iiissuieeenienneen
Answer:
[Yes]
Justification:
We conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We were allowed to use the dataset and cited them in the paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https htneurriis.coo cctpsssccccoooeeecee/2222233 for what should or should be described.
22

---

A Limitations
In this paper, we propose novel SLVG framework, Vign which demonstrates significant im- provements over previous methods. Nevertheless, our framework still has two notable limitations that warrant further investigation.
The first limitation is its inability to support multiple sign languages simultaneously. As different countries have their own unique sign language systems, adapting Vig for specific sign language currently requires training separate model for each. This dependency on independent model training significantly restricts its practicality real-world upplications. Consequently, the development of unified SLVG system capable of supporting multiple sign languages will be critical for enhancing its versatility and applicability. Addressing this challenge will serve as a key direction our future research endeavors.
The second limitation pertains to the computational inefficiency inherent in the iterative denoising process of diffusion models. Existing methods, such as the consistency model [72] and efficient ODE solvers [36, 37], offer promising approaches accelerate the sampling process of diffusion- models. Incorporating these techniques into SignViP represents another avenue for improving its efficiency and scalability, which will also be prioritized our future research.
B Architecture Details
Multi-Condition Encoder.
The multi-condition token space in framework is constructed using multi-condition encoder. This encoder adopts a simple convolutional architecture with 8 convolutional layers with SiLU activation [12, 50], each achieving 4x spatial downsampling (i.e., 2x along both the height and width axes).
Mixer in Multi-Condite
Token Translator. In Multi-Condition Token Translator, we use mixer to aggregate tokens from the same frame into unified input hidden states. The mixer consists of linear layer, followed by LayerNorm and GELU activation [25]. Tokens are first projected into embeddings, which are then flattened and passed through the mixer to produce a single hidden state
for each frame.
Token Translator. In the Multi-Condition Token Translator, we utilize decoder to decode all tokens of the same frame from the output hidden states of the GPT-2 model. The decoder employs multiple parallel heads decode each token within the frame. Specifically, each head is implemented as lightweight Transformer layer [76, 8, 15, 16].
Decoder in Multi-Conditee
C Training Details of Back-Translation Models
To evaluate the semantic consistency of the generated sign language videos, we follow ProTTan [58] train two SLP models [6] to translate sign language videos (i.e., the video and poses (i.e,, the pose back-translationn model) into texts, respectively.
model)
Video Back- Translation Model. The video
model is trained using single NVIDIA RTX A6000 GPU. An Adam optimizer [30] is utilized with a learning rate of 1e-3, and the batch size set to 32. Validation performance is logged every steps, providing checkpoints throughout the training process. The best-performing model checkpoint is observed at step 10,500.
Pose Back-Translation Model. The pose back-translation model is also trained single NVIDIA RTX A6000 GPU. The training configuration mirrors that of the video back-translation model, using an Adam optimizer [30] with learning rate of 1e-3, batch size of 32, and validation logged every steps. The optimal model checkpoint is reached at step 5,400.
D Human Evaluation
The evaluation protocol based on mack-translation models is highly dependent on the quality of the back-translation system, which may introduce certain biases. Although human evaluation could potentially offer a more reliable solution, recruiting qualified sign language experts presents significant challenges.
23

---

To address this limitation, we employ a compromise human evaluation strategy that does not require expert knowledge of sign language. Specifically, we present human evaluators with the ground-trudt sign language video, along with several anonymized candidate videos generated by different SLVG methods. Evaluators are instructed as follows: "Please choose the video where the signer's actions appear most similar to those in the ground-truth video."
For this evaluation, we recruited 10 non-expert participants. Each participant evaluated 25 groups of samples for the RWTH-2014T dataset and 50 groups for the How2Sign dataset, resulting in a total of 250 and 500 votes, respectively. We report the proportion of votes received by each SLVG method in Table 6. Notably, our method, Sign ViP, achieves the highest vote proportion across both datasets, indicating that SignViP generates sign language videos that are more consistent with the ground-true truth references as perceived by human evaluators.
Table 6: Human Evaluation Results on RWTH-2014T and How2Sign datasets.
RWTTI-2014T
How 25imn
VVotes
VotetS
#Votes
Vote56
TOTAL
250
100.0%
300
100%
SignGAN
25
10.0%
6.2%
w/ AnimsteeAnyoee
25
9.3%
24
4.8%
SignCion
27
10.8%
36
7.2%
Sig (Cus)
175
70.0%
409
81.8%
E
More Experiments
E.1
Comparison with Direct Condition Prediction
As stated in Section 1. one of the motivations for introducing the multi-condition token space the inherent difficulty of directly translating fine-grained attributes. In Figure 1(4), we illustrate examples of direct multi-condition predictions, demonstrating significant discrepancies between ground-truth and predicted results. To quantitatively compare the direct condition prediction approach with our proposed method under the pose back-translation paradigm, we present experimental results in Table 7. The results demonstrate that direct condition prediction performs poorly across all metrics. These results clearly demonstrate that direct multi-conditio prediction struggles effectively model the fine-grained attributes and fails to maintain semantic consistency during the translation process.
Table Comparison of pose back-translation performance between direct condition prediction and our proposed ViP.
RWT1E-2011 2014T
2Sngn
Methods
BLEU-1
BLEU-2
BLBU-3
BLEU4
ROUGE
BLEU-1
BLEBC2
BLEU-3
BLEU-
ROUGE
Ground: Truth
30,99
18:36
12.83
31.02
24.56
14.96
10.31
7.91
24.88
Condition Prediction
9.27
2:23
0.80
0.32
11.64
5.39
1.63
2.26
L11
8.12
SignViP (Qurs)
22.67
E.2 Efficiency Comparison
To compare the efficiency of our method with other approaches, we conducted experiments measuring the number of model parameters and inference time per frame. As shown in Table 8, the results demonstrate that our Sign ViP achieves comparable model size and inference time to the diffusion- based baselines. This indicates that our method delivers high-quality video generation without incurring significant additional computational cost, making it an efficient and practical solution for sign language video generation.
F.3 Effectiveness of Pretrained Parameter Initialization
The comparison among methods is fair, as all the diffusion-based SLVG approaches (i.e., Sign- GAN+AnimateAyyom SignGen, and our Sign are initialized with parameters from existing pretrained diffusion models.
24

---

Table 8: Efficiency comparison between diffusion- baselines and our Sign ViP.
Parameters
Inference (siirame)
SigaGAN AnimateAnyune (29]
2575. 85M
1.3404 9777.92M
SignGen
22177.880
1.2225
(Ouns)
1.2398
Training a video generation diffusion model from scratch is extremely challenging due the high computational costs and slow convergence. Therefore, leveraging pretrained diffusion parameters to initialize customized diffusion models has become fundamental strategy to significantly accelerate training [23, 4, 53, 21, 39].
To further validate the effectiveness of pretrained parameter initialization, conducted ablation study, evaluating the quality of generated videos both with and without pretrained initialization under the same number of training steps. The results, presented in Table 9, clearly demonstrate the substantial advantage of initializing with pretrained parameters.
Table 9: Ablation study on the effect of pretrained initialization.
FID J.
FVD
wio Pretrained Initialization w/ Preteained Initialization (Ours)
2278. 23
3277. 20
508.91
1025.45
E.4 Order-Preserving Evaluation of Back- Translation Models
To further validate the reliability and comparability of our back-translation models, we conduct order-preserving experiments. Specifically, we introduce pose sequences with varying levels of errors and evaluate whether the corresponding output metrics display consistent ranking that reflects the severity of the errors. In other words, a comparable trecssranslttion model should exhibit a steady degradation output quality as the degree of input error increases.
To simulate realistic pose errors, we independently apply spatial and temporal perturbations as follows: (1) Spatial Perturbation: Additive bias is applied to pose keypoints. To mimic real-world errors while avoiding excessive deformation, we first compute the variance of cach keypoint's 's coordinates from the dataset. The bias added to each keypoint is sampled from normal distribution A(, o²). where a reflects the perturbation intensity. (2) Temporal Perturbation: We randomly delete, repeat, or duplicate pose frames at ratio of p. where p controls the perturbation intensity.
The results of the pose back-transian translation model under both spatial and temporal perturbations are summarized in Figure 5(a) and (b). As shown in the figures, all metrics decrease monotonically the perturbation intensity increases. This demonstrates that our pose back-translation model is sensitive varying levels of pose errors and can provide reliable evaluation metrics for comparison.
For the video back-translation model, we first synthesize videos from the perturbed poses using AnimateAnyone [29], and then evaluate the corresponding metries. The results of the video back- translation model under both spatial and temporal perturbations are summarized in Figure 5(c) and (d). Consistent with our findings for the pose model, video back-translation model also provides reliable evaluation metrics for comparison. Note that since we use AnimateAnyone to synthesize new videos, the metries without perturbation differ from the ground -trutthtrhhhhh metrics reported in Table 1.
01203
10 1OLEE
RO
5
85 Tomporal Peruuaaaon Intenilty
5p Spa6al Rontrrattinn Inconcity
Temprcal Returaddonn inye
Portu Ronturattio Daton Rizonsity
Figure 5: (a) Effect of spatial perturbation for the pose back-translation model. (b) Effect of temporal perturbation for the pose nocktransslation model. (c) Effect of spatial perturbation for the video tack-translation model. (d) Effect of temporal perturbation for the video fack-translation model.
25

---

F
More Cases
We demonstrate more video cases in Figure 6 and Figure
4
6
A
B
8
A
6
9
Figure 6: More generated cases of RWTH-201 dataset.
3
3
3
8
2
2
3
30
2
3
3
3
3
3
3
Figure 7: More generated cases of How2Sign dataset.
26

---

G Token Translation Accuracy
Directly computing the accuracy of token translation our framework presents significant challenges. Due to temporal shifts and uneven scaling, translated tokens during inference may not be perfectly aligned with ground truth tokens, making traditional accuracy metrics less reliable.
To address this, we employ normalized Dynamic Time Warping (DTW) distance as an alternative evaluation metric. DTW is similarity measure that computes the optimal alignment between two sequences, even if they differ in length or are not aligned one-to-one. By normalizing the DTW distance, accommodate variations in sequence length, enabling fair comparison across different settings. In our evaluation pipeline, we decode the translated tokens into condition embeddings using the decoder the FSQ Autoencoder, and then compare these embeddings ground- trutt condition embeddings via normalized DTW distance.
To validate the effectiveness of normalized DTW distance metric, conduct experiments following the settings described in "Effect of Compression" and "Effect of Scheduled Sampling Strategy" of Section 4.3. As shown in Tables 10 and 11, the trends of normalized DTW distance closely match those of previously reported metrics (BLEU-4 and ROUGE), confirming its reliability for evaluating token translation accuracy.
Table 10: Effect of Compression Rate on Token Translation Metrics
Cumpressium
(1)
BLEU-
1,9020
1,7937
1,7458
5.64
23.68
(Our))
1,5968
8.65
28.85
1,7291
Table 11: Effect of Scheduled Sampling Rate on Token Translation Metries
Sampling Rate
Norm, DTW Distance (1)
BLEU-4
ROUGE
0.2 0.4 (Ours)
1.6133
8.12
27.72
1.5968
8.65
28.85
0.8
1.6191
7,82
27.15
1.0
1.7354
5,89
24.71
27


Reviews:
[{"review_id": "cyRIH2tCJG", "reviewer": "Reviewer_g4KT", "summary": "The paper introduces SignViP, a new framework for Sign Language Video Generation from spoken language texts. Unlike previous approaches that rely on a single coarse condition, SignViP leverages multiple fine-grained conditions, such as fine-grained body poses and 3D hands, to improve the naturalness and expressiveness of generated sign language videos. Experimental results on two different datasets demonstrate that SignViP achieves state-of-the-art performance in video quality, temporal coherence, and semantic fidelity.", "questions": "Is there any other evaluation metric to determine the quality of video generation? As BLEU and Rouge scores have already shown not perfect in language generation tasks.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Clear motivation to incorporate multiple fine-grained conditions.\n2. Achieving new SOTA results on two evaluation datasets when using video back-translation.\n\nWeaknesses:\n1. The method introduction is a bit complicated, without emphasizing the novel part of the paper.\n2. The performance evaluation mainly depends on back-translation techniques, which highly relates to the quality of back-translation model. While the method achieves SOTA on Table 1, several metrics in Table 2 fall behind MoMP method.\n3. Sign Video Diffusion Model is initialized with an existing diffusion model, and it lacks ablation to show the effect of such initialization, making whether the comparison is fair is not sure.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. The additional experimental results further verifiy the effectiveness of the proposed method. I have raised the score accordingly.", "rebuttal": "We sincerely appreciate your recognition of the clear motivation and SOTA performance demonstrated in our work. We will address your concerns below.\n\n# Weaknesses\n\n**`W1:`** The method introduction is a bit complicated, without emphasizing the novel part of the paper.  \n\n**`R1:`** The main novelties of our paper are detailed in Sections 3.2 and 3.3. Our key innovations are as follows: \n\n1. **Discrete Tokenization Paradigm** (lines 144–145): To mitigate errors in directly translating high-dimensional conditions, we propose a discrete tokenization approach to integrate and represent these conditions;\n2. **Novel Module Design** (lines 146–154): We present new modules specifically designed for SignViP, enabling more robust processing of multi-condition inputs;\n3. **Multi-Stage Training Strategy** (Section 3.3): We introduce a multi-stage training strategy to construct a comprehensive multi-condition token space for our discrete tokenization paradigm.\n\nWe will improve the method section to more clearly emphasize these core innovations and improve its clarity and readability.\n\n---\n\n**`W2:`** The performance evaluation mainly depends on back-translation techniques, which highly relate to the quality of the back-translation model.  \n\n**`R2:`** We agree that the evaluation protocol is highly dependent on the quality of the back-translation model, which may introduce bias. While human evaluation maybe provide a more reliable resolution, it is challenging to recruit sign language experts.  \n\nTo address this, we further adopt a compromise human evaluation strategy that does not require the sign language experts. Specifically, we present human evaluators with the ground-truth video alongside several corresponding anonymized videos generated by different SLVG methods. The evaluators are given the following instruction:  \n\n\"*Please choose the video where the signer's actions appear most similar to those in the ground-truth video.*\"  \n\nWe invited 10 evaluators, each of whom received 25 group samples for the RWTH-2014T dataset and 50 for the How2Sign dataset, resulting in totals of 250 and 500 votes, respectively. We calculate the proportion of votes for each SLVG method, with the results shown below. Notably, our SignViP receives the highest proportion of votes, indicating that SignViP generates more accurate sign language videos.\n\n| | #Votes (RWTH) | Vote% (RWTH) | #Votes (H2S) | Vote% (H2S) |\n| :- | :-: | :-: | :-: | :-: |\n| SignGAN | 25 | 10.0% | 31 | 6.2% |\n| SignGAN w/ AnimateAnyone | 23 | 9.2% | 24 | 4.8% |\n| SignGen | 27 | 10.8% | 36 | 7.2% |\n| **SignViP (Ours)** | **175** | **70.0%** | **409** | **81.8%** |\n| *TOTAL* | *250* | *100.0%* | *500* | *100%* |\n\n---\n\n**`W3:`** While the method achieves SOTA on Table 1, several metrics in Table 2 fall behind the MoMP method.  \n\n**`R3:`** In Table 2, we compare our SLVG  method with SLP (Sign Language Production) approaches rather than other SLVG methods. These SLP baselines translate text directly into pose sequences, which aligns with our pose back-translation evaluation pipeline. However, for our SLVG method, we need to detect poses from the generated videos, which may introduce additional errors and impact the evaluation results. Consequently, the state-of-the-art SLP method (i.e., MoMP) outperforms our method on certain metrics in Table 2, although our results remain comparable.  \n\nTo enable a fair comparison in the SLVG setting, we further combine MoMP with a pose-to-video generation approach. The following video back-translation results demonstrate that our method is better suited for the SLVG task compared to MoMP-based methods.\n\n| *[RWTH-2014T Dataset]* | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE |\n| :- | :-: |:-: | :-: | :-: | :-: |\n| MoMP+ControlNet | 19.12 | 8.95 | 5.33 | 3.61 | 21.54 |\n| MoMP+AnimateAnyone | 20.05 | 8.79 | 5.24 | 3.72 | 21.68 |\n| **SignViP (Ours)** | **26.72** | **15.65** | **11.14** | **8.65** | **28.85** |\n\n| *[How2Sign Dataset]* | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE |\n| :- | :-: | :-: | :-: | :-: |:-: |\n| MoMP+ControlNet | 12.53 | 5.59 | 3.48 | 2.31 | 13.72 |\n| MoMP+AnimateAnyone | 13.65 | 5.82 | 3.39 | 2.25 | 14.15 |\n| **SignViP (Ours)** | **16.21** | **9.36** | **6.28** | **5.04** | **16.99** |\n\n---\n\n**`W4:`** Sign Video Diffusion Model is initialized with an existing diffusion model, and it lacks ablation to show the effect of such initialization, making whether the comparison is fair is not sure.  \n\n**`R4:`** The comparison is fair, because all the diffusion-based SLVG methods (i.e., SignGAN+AnimateAnyone, SignGen, and our SignViP) are initialized with parameters from existing pretrained diffusion models.  \n\nTraining a video generation diffusion model from scratch is extremely challenging due to the high computational costs and slow convergence. Therefore, using pretrained diffusion parameters to initialize the customized diffusion model is a fundamental strategy to significantly accelerate training [1, 2, 3, 4, 5].   \n\nTo further validate the effectiveness of pretrained parameter initialization, we conducted an ablation study, evaluating the quality of generated videos both with and without pretrained initialization under the same training steps. The results below clearly demonstrate the substantial advantage of initializing with pretrained parameters.\n\n| | FID (↓) | FVD (↓) |\n| :- | :-: | :-: |\n| w/o Pretrained Initialization | 2278.23 | 3277.20 |\n| **w/ Pretrained Initialization (Ours)** | **508.91** | **1025.45** |\n\n# Questions\n\n**`Q1:`** Is there any other evaluation metric to determine the quality of video generation? As BLEU and ROUGE scores have already shown not perfect in language generation tasks.  \n\n**`A1:`** Thank you for your suggestion. To provide a more comprehensive evaluation of back-translated texts, we further employ the COMET [6, 7] metric, which is specifically designed to predict human judgments of machine translation quality. COMET is widely used for machine translation tasks [8, 9, 10] and is considered more suitable than BLEU and ROUGE.  \n\nAs shown below, for video back-translation, our SignViP still outperforms other SLVG baselines according to the COMET metric.  \n\n| | COMET (RWTH) | COMET (H2S) |\n| :- | :-: | :-: |\n| *Ground-Truth* | *0.6157* | *0.5882* |\n| SignGAN | 0.4977 | 0.5104 |\n| SignGAN w/ AnimateAnyone | 0.4928 | 0.5135 |\n| SignGen | 0.4086 | 0.4127 |\n| *MoMP+ControlNet* | 0.5033 | 0.5122 |\n| *MoMP+AnimateAnyone* | 0.5091 | 0.5208 |\n| **SignViP (Ours)** | **0.5608** | **0.5524** |\n\nAs shown below, for pose back-translation, SignViP achieves performance that is slightly lower but comparable to the state-of-the-art SLP method (i.e., MoMP). However, when MoMP is combined with pose-to-video generative models (i.e., ControlNet or AnimateAnyone) for the SLVG task, the resulting MoMP-based methods yield lower COMET scores than our approach. These results on the COMET metric indicate that SignViP is more suitable for the SLVG task compared to MoMP-based methods.\n\n| | COMET (RWTH) | COMET (H2S) |\n| :- | :-: | :-:|\n| *Ground-Truth* | *0.5978* | *0.6250* |\n| ProTran | 0.5091 | 0.5549 |\n| Adversarial | 0.5127 | 0.5618 |\n| MDN | 0.5251 | 0.5685 |\n| *MoMP* | **0.5466** | **0.5802** |\n| **SignViP (Ours)** | 0.5347 | 0.5738 |\n\n# References\n\n`[1]` Latent Video Diffusion Models for High-Fidelity Long Video Generation, arXiv:2211.13221.  \n`[2]` Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, arXiv:2311.15127.  \n`[3]` Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models, ECCV 2024.  \n`[4]` AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning, ICML 2024.  \n`[5]` Follow Your Pose: Pose-Guided Text-to-Video Generation Using Pose-Free Videos, AAAI 2024.  \n`[6]` COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task, WMT 2022.  \n`[7]` COMET: A Neural Framework for MT Evaluation, EMNLP 2020.  \n`[8]` Hallucinations in Large Multilingual Translation Models, TACL 2023.  \n`[9]` Tower: An Open Multilingual Large Language Model for Translation-Related Tasks, COLM 2024.  \n`[10]` Exploring Human-Like Translation Strategy with Large Language Models, TACL 2024."}, {"review_id": "jU2h0Eddso", "reviewer": "Reviewer_Hho8", "summary": "This paper proposes SignViP, a novel framework for Sign Language Video Generation (SLVG) from spoken language input.\nSignViP introduces a multi-condition tokenization paradigm, leveraging both fine-grained body poses and 3D hand information to enhance generation fidelity.\nSpoken language input is translated into multi-condition tokens, decoded into motion embeddings, and used to guide the video generation.", "questions": "1. The model heavily relies on the accuracy of the discrete multi-condition tokens generated from spoken language input, it is essential to evaluate how well this component performs. Have you measured the accuracy of these tokens?\n2. FSQ is used as a core quantization method, but VQ-VAE is also commonly used in generative tokenization tasks. Could you justify the use of FSQ over more established alternatives like VQ-VAE beyond discussion in 3.1?\n3. While your paper includes popular baselines like SignGAN and AnimateAnyone, would you please include more SLG works? \n\nQ1 and Q3 addressing Significance & Quality\nQ2 for Quality", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Clarity: Clear writing and figure\n2. Quality: Strong results and diverse evaluation\n\nWeaknesses:\n1. Originality & Significance: LImited techinal novelty. Its core innovation is more like combining existing concepts (tokenization, diffusion, multi condition control) in a reasonable way in SLVG tasks.\n2. Quality: Some baseline are not for sign language\n3. Quality: Lack of direct evaluation on token translation accuracy", "was_revised": true, "final_justification": "Thanks for the response.\nI have raised my score as most of my concerns have been reasonably addressed.\nI recognize the contribution of this work in proposing a new representation strategy for sign language, and I acknowledge that the multi-condition setting presents a meaningful alternative to the traditional gloss space. This approach is particularly promising within the proposed multimodal pipeline.\nMy initial concerns primarily revolved around this design choice and its implications. While the authors have responded, I found their rebuttal somewhat underdeveloped in articulating the significance and impact of the multi-condition formulation.\nIn my view, the authors may have underappreciated its potential, and a more thorough discussion would have strengthened the paper.\nNonetheless, I respect differing perspectives on emphasis and framing. \n\nMy remaining concern is that the end-to-end sign language video generation significantly underperforms compared to staged approaches (such as text-to-pose and pose-to-video), and that the overall task itself appears to lag behind related work in adjacent fields, thereby limiting its potential contribution to the broader community.\nSpecifically, the methods generate text-to-pose have much higher back-translation scores than the methods combining end-to-end text-to-pose(including the authors' work: multi-conditions)-to-video. This observation also raises a broader question for the sign language generation domain. Across the entire SL-generation track, many subtracks, such as text-to-pose and pose-to-video, appear capable of achieving relatively strong generation performance. Why, then, does the end-to-end sign language video generation, which follows a similar conceptual pipeline (in the authors' paper) but integrates the stages into a single model, exhibit such a substantial drop in performance? \n\nIn this case, I understand that it is improper to ask authors to be responsible for field-related issues, but it is necessary to at least prove that their very low back-translation setup can reliably reflect model performance or meaningfully support model comparison.\nIn response, the authors further provide tables demonstrating the linear correlation between the error rate in a test set and the result of their evaluation metrics on the test set. However, this can only demonstrate effectiveness under the same distribution and does not ensure that the evaluation metric remains comparable across different distributions. From a deep learning perspective, given such low absolute scores, some parameter adjustments could lead to large fluctuations in the evaluation metrics. This makes it necessary to further examine the absence of multiple runs (of their experiments) and statistical measures such as variance in the authors’ reporting.\n\nOverall, I believe the work is technically solid and introduces novel elements that merit recognition. However, the overall task track appears to lag behind related work in adjacent SL fields (such as text-to-pose), which suggests there may be a limit to the extent to which it contributes to the broader community. Due to this potentially constrained impact, I rate it as Borderline accept.", "rebuttal": "We sincerely appreciate your recognition of the paper's clear writing, as well as the quality and diversity of its evaluation results. We will address your concerns below.\n\n# Weaknesses\n\n**`W1:`** This paper lacks novelty, whose core innovation is more like combining existing concepts (tokenization, diffusion, multi condition control) in a reasonable way in SLVG tasks.\n\n**`R1:`** We respectfully disagree with your opinion and would like to further clarify the novelty and unique contributions of our paper. Our primary innovation is **a discrete tokenization paradigm through the construction of a discrete multi-condition token space** to bridge fine-grained conditions with the sign language video frames. While prior works have explored the use of multiple [1] or fine-grained [2, 3, 4] conditions for diffusion-based human video generation, there are significant distinctions compared to our method:\n\n1. In the SLVG task, conditions are limited to serving as **intermediary** that bridge spoken language text and sign language videos. While incorporating multiple fine-grained conditions can enhance the generative process, it also increases the complexity of the translation process. To address this challenge, we introduce a compression strategy (i.e., FSQ Autoencoder) for conditions, enabling an efficient yet expressive conditional space. This strategy facilitates a balanced trade-off between translation and generation.\n2. As a compression strategy, although quantization-based tokenization has been widely explored [5, 6, 7, 8], existing methods focus primarily on tokenizing **pixel space**. To the best of our knowledge, our work is the first to introduce tokenization on multiple fine-grained conditions for human video generation.  \n\nTherefore, we believe that our core contribution is indeed novel and advances the state-of-the-art in SLVG.\n\n---\n\n**`W2:`** Some baselines are not for sign language.\n\n**`R2:`** The baselines in Tables 1 and 2 correspond to Sign Language Video Generation (SLVG) methods and Sign Language Production (SLP) methods, respectively. In Table 3, \"ControlNet\" and \"AnimateAnyone\" are trained on sign language video datasets to evaluate their condition-to-video generative capabilities.  \n\nHowever, we acknowledge that our paper lacks a comparison of video quality among SLVG methods. To further address the gap, we introduce two novel metrics in addition to FID and FVD, which are CLIP-FID and Identity Similarity (IDS), since temporal alignment cannot be fully preserved.\n\nSpecifically, CLIP-FID is a variant of FID that leverages CLIP embeddings as frame representations. IDS assesses identity consistency between generated and ground-truth videos by calculating the cosine similarity of face embeddings extracted using YOLO5Face and Arc2Face.\n\nThe following experimental results validate the state-of-the-art performance of SignViP. \n\n| *[RWTH-2014T Dataset]* | FID (↓) | CLIP-FID (↓) | FVD (↓) | IDS |\n| :- | :-: | :-: | :-: | :-: |\n| SignGAN | 547.90 | 167.70 | 1431.38 | 0.463 |\n| SignGAN w/ AnimateAnyone | 595.99 | 161.97 | 1330.54 | 0.462  |\n| SignGen | 644.06 | 184.66 | 1715.32 | 0.515 |\n| **SignViP (Ours)** | **508.91** | **154.10** | **1025.45** | **0.571** |\n\n| *[How2Sign Dataset]* | FID (↓) | CLIP-FID (↓) | FVD (↓) | IDS |\n| :- | :-: | :-: | :-: | :-: |\n| SignGAN | 667.44  | 210.11 | 2766.97 | 0.538 |\n| SignGAN w/ AnimateAnyone | 679.41 | 215.05 | 2484.39 | 0.533  |\n| SignGen | 815.69  | 186.32 | 3538.49 | 0.539 |\n| **SignViP (Ours)** | **575.67** | **109.61** | **2207.67** | **0.624** |\n\n---\n\n**`W3:`** Lack of direct evaluation of token translation accuracy.\n\n**`R3:`** It is challenging to directly compute the accuracy of token translation. Because of the temporal shifts and uneven scaling, the translated tokens during inference may not perfectly align with ground-truth tokens. As a replacement, we employ **normalized DTW distance** as the metric.\n\n**Dynamic Time Warping (DTW)** distance is a similarity measure that computes the optimal alignment between two sequences, even if they differ in length or are not aligned one-to-one. The DTW distance is normalized to accommodate variations in sequence length.\n\nFor our method, we decode the translated tokens into condition embeddings using the decoder of the FSQ Autoencoder, and compare these embeddings with the ground-truth condition embeddings using the normalized DTW distance.\n\nTo validate the effectiveness of normalized DTW distance, we follow the experimental settings in **Section 4.3 (Effect of Compression)** and **Appendix D.7 (Effect of Scheduled Sampling Strategy)**. The following experimental results show that the trend of the normalized DTW distance closely matches that of previously reported metrics, confirming its reliability as an evaluation metric for token translation.\n\n| Compression Rate | Norm. DTW Distance (↓) | BLEU-4 | ROUGE |\n| :- | :-: | :-: | :-: |\n| 1 | 1.9020 | 1.32 | 17.38 |\n| 2 | 1.7937 | 4.14 | 21.89 |\n| 4 | 1.7438 | 5.64 | 23.68 |\n| **8 (Ours)** | **1.5968** | **8.65**| **28.85** |\n| 16 | 1.7291 | 7.02 | 24.72 |\n\n| Sampling Rate | Norm. DTW Distance (↓) | BLEU-4 | ROUGE |\n| :- | :-: | :-: | :-: |\n| 0.2 | 1.6123 | 8.12 | 27.72 |\n| **0.4 (Ours)** | **1.5968** | **8.65** | **28.85** |\n| 0.6 | 1.5978 | 8.30 | 28.23 |\n| 0.8 | 1.6191 | 7.82 | 27.15 |\n| 1.0 | 1.7354 | 5.89 | 24.71 |\n\n\n# Questions\n\n**`Q1:`** The model relies on the accuracy of the discrete multi-condition tokens. Have you measured the accuracy of these tokens?\n\n**`A1:`** Please refer to our response to Weakness 3.\n\n---\n\n**`Q2:`** Could you justify the use of FSQ over more established alternatives like VQ-VAE?\n\n**`A2:`** In **Appendix D.8 (FSQ vs. VQ)**, we have compared FSQ and VQ as quantization strategies for the FSQ Autoencoder. Specifically, we evaluated both methods in terms of **codebook usage** efficiency and multi-conditional **reconstruction loss**.\n\nThe results are summarized as follows: the codebook usage experiments demonstrate the stability and scalability of FSQ, while the reconstruction loss experiments show that FSQ more effectively preserves the original conditional structure.\n\n| Codebook Size | Usage% (FSQ) | Usage% (VQ) | Recon. Loss (FSQ) (↓) | Recon. Loss (VQ) (↓) |\n| :- | :-: | :-: | :-: | :-: |\n| $2^7$ | 100% | 90.6% | 0.1545 | 0.1569 |\n| $2^8$ | 100% | 51.6% | 0.1389 | 0.1545 |\n| $2^9$ | 99.4% | 26.6% | 0.1268 | 0.1528 |\n| $2^{10}$ | 99.9% | 14.6% | 0.1270 | 0.1512 |\n| $2^{11}$ | 99.2% | 6.9% | 0.1188 | 0.1520 |\n| $2^{12}$ | 97.81% | 3.6% | 0.1160 | 0.1511 |\n\nAdditionally, in **Section 4.3 (Effect of Quantization)**, we abalated quantization from FSQ Autoencoder, treating this as a special quantization strategy. The qualitative comparison in Figure 7 demonstrates that ablating quantization leads to weak semantic alignment and low video quality. It highlights the importance of quantization for our SignViP. Here, we present the detailed quantitative results on RWTH-2014T.\n\n| *[Video Back-Translation]* | BLEU-1 | BLEU-4 | ROUGE |\n| :- | :-: | :-: |:-:|\n| w/o FSQ | 9.43 | 0.42 | 13.58 |\n| **w/ FSQ (Ours)** | **26.72** | **8.65** | **28.85** |\n\n| *[Pose Back-Translation]* | BLEU-1 | BLEU-4 | ROUGE |\n| :- | :-: | :-: | :-: |\n| w/o FSQ | 8.11 | 0.30 | 11.29 |\n| **w/ FSQ (Ours)** | **21.94** | **4.61** | **22.67** |\n\n| *[Video Quality]* | FVD (↓) | SSIM   | PSNR  | LPIPS (↓) | Hand SSIM |\n| :- | :-: | :-: | :-: | :-: | :-: |\n| w/o FSQ | 1932.26 | 0.473 | 10.02 | 0.434 | 0.258 |\n| **w/ FSQ (Ours)** | **1025.45** | **0.609** | **13.43** | **0.331** | **0.294** |\n\n---\n\n**`Q3:`** Would you please include more SLVG works as baselines?\n\n**`A3:`** Thank you for your suggestion.\n\nWe add new baselines that combine SLP methods (i.e., MDN and MoMP) with pose-to-video methods (i.e., ContorlNet and AnimateAnyone). Specifically, we first use an SLP method to convert spoken language text into skeleton poses, and then employ a pose-to-video method to synthesize sign language videos from these poses. \n\nThe video back-translation results are presented below, which demonstrate that our SignViP still outperforms the newly added baselines.\n\n| *[RWTH-2014T Dataset]* | BLEU-1 | BLEU-4 | ROUGE |\n| - | :-: | :-: | :-: |\n| MDN+ControlNet | 18.52 | 3.25 | 20.88 |\n| MDN+AnimateAnyone | 19.45 | 3.33 | 21.02 |\n| MoMP+ControlNet | 19.12 | 3.61 | 21.54 |\n| MoMP+AnimateAnyone | 20.05 | 3.72 | 21.68 |\n| **SignViP (Ours)** | **26.72** | **8.65** | **28.85** |\n\n\n| *[How2Sign Dataset]* | BLEU-1 | BLEU-4 | ROUGE |\n| - | :-: | :-: | :-: |\n| MDN+ControlNet | 11.95 | 2.09 | 13.05 |\n| MDN+AnimateAnyone | 13.08 | 2.03 | 13.48 |\n| MoMP+ControlNet | 12.53 | 2.31 | 13.72 |\n| MoMP+AnimateAnyone | 13.65 | 2.25 | 14.15 |\n| **SignViP (Ours)** | **16.21** | **5.04** | **16.99** |\n\n# References\n\n`[1]` Follow-Your-Pose V2: Multiple-Condition Guided Character Image Animation for Stable Pose Control, arXiv:2406.03035.  \n`[2]` Magicanimate: Temporally Consistent Human Image Animation Using Diffusion Model, CVPR 2024.  \n`[3]` Magicportrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance, arXiv:2504.21497.  \n`[4]` Vlogger: Multimodal Diffusion for Embodied Avatar Synthesis, arXiv:2403.08764.  \n`[5]` Finite Scalar Quantization: VQ-VAE Made Simple, ICLR 2024.  \n`[6]` Language Model Beats Diffusion–Tokenizer is Key to Visual Generation, arXiv:2310.05737.  \n`[7]` Image and Video Tokenization with Binary Spherical Quantization, arXiv:2406.07548.  \n`[8]` Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis, arXiv:2412.04431."}, {"review_id": "4r137bww8o", "reviewer": "Reviewer_8mrn", "summary": "This paper introduces a new approach to generating sign language videos given a spoken language text, more specifically for American & German Sign Language. This is done through a multi-step process, leveraging multiple conditions to help improve the generation. Results showed strong performance in comparison to other recent approaches in this domain.", "questions": "You introduce Hand SSIM -- are you planning on releasing the details for this so others can leverage this?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality - the paper is technically sound for the most part. My biggest concern is the experiments, specifically the back translation. Sign language translation models still perform not too well in this domain, thus it makes it difficult to judge if the videos are generating correct sign language. Human evals would have strengthened the paper, though understandably it is difficult to find German/American Sign Language experts that can evaluate the results.\n\nClarity - the paper is well written and can be understood fairly well. Some areas of improvement though could be in the figures. The first one is a bit difficult to follow while reading the text, since you almost have 4 different figures lumped into one. And some of the figures can be very difficult to see (the ones with example video frames of signers) as they have been made so small.\n\nSignificance - These results are impactful, even with the concerns mentioned about about back translation in this domain. Sign language generation is a very difficult task that has seen more interest lately, and this paper does a good job at pushing the domain forward.\n\nOriginality -  the approach seems rather novel, incorporating new ideas to help it achieve better results than past approaches.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the recognition of the quality, clarity, significance, and originality of our work. We will address your concerns below.\n\n# Weaknesses\n\n**`W1:`** Sign language translation models still do not perform too well in this domain, thus it makes it difficult to judge if the videos are generating correct sign language. Human evaluations would have strengthened the paper, though understandably it is difficult to find sign language experts that can evaluate the results. \n\n**`R1:`** Thank you for your suggestion. To facilitate human evaluation for the SLVG task, we further adopt a compromise human evaluation strategy that does not require the sign language experts. Specifically, we present human evaluators with the ground-truth video alongside several corresponding anonymized videos generated by different methods. The evaluators are given the following instruction:  \n\n\"*Please choose the video where the signer's actions appear most similar to those in the ground-truth video.*\"  \n\nWe invited 10 evaluators, each of whom received 25 group samples for the RWTH-2014T dataset and 50 for the How2Sign dataset, resulting in totals of 250 and 500 votes, respectively. We calculate the proportion of votes for each method, with the results shown below. Notably, our SignViP receives the highest proportion of votes, indicating that SignViP generates more accurate sign language videos.\n\n| | #Votes (RWTH) | Vote% (RWTH) | #Votes (H2S) | Vote% (H2S) |\n| :- | :-: | :-: | :-: | :-: |\n| SignGAN | 25 | 10.0% | 31 | 6.2% |\n| SignGAN w/ AnimateAnyone | 23 | 9.2% | 24 | 4.8% |\n| SignGen | 27 | 10.8% | 36 | 7.2% |\n| **SignViP (Ours)** | **175** | **70.0%** | **409** | **81.8%** |\n| *TOTAL* | *250* | *100.0%* | *500* | *100%* |\n\n---\n\n**`W2:`** Some figures could be improved. (1) Figure 1 is a bit difficult to follow while reading the text, since you almost have 4 different figures lumped into one. (2) Some figures are very difficult to see (the ones with example video frames of signers) as they have been made so small.  \n\n**`R2:`** Thank you for your suggestions. We will enhance the presentation of the figures. Specifically, Figure 1 will be reorganized to improve readability. In addition, all other figures containing video frames will be enlarged to make them easier for readers to view.\n\n# Questions\n\n**`Q1:`** Will you share the details of Hand SSIM to enable others to utilize it?  \n\n**`A1:`** Yes, we are committed to sharing the implementation details of Hand SSIM by publicly releasing our code repository."}, {"review_id": "QBlRx0mvxv", "reviewer": "Reviewer_uzjH", "summary": "The paper proposes SignViP, a novel Sign Language Video Generation method that incorporates Multiple Fine-Grained Conditions (poses and 3D hands) to enhance the quality of generated sign language videos. Their method includes three main components: 1) a sign video diffusion model, 2) a Finite Scalar Quantization autoencoder, and 3) a Multi-Condition Token Translator.", "questions": "**References**\n\nCould use more references for SLVG? e.g., Pose-Guided Fine-Grained Sign Language Video Generation, ECCV 2024\n\n**Questions**\n- For video back-translation, you used sign language transformers [1] and [2] for pose back-translation, is that right?\n- What is the resolution of the output video? Is it the same as input?\n- In Figure 2, what is a Mixer? Please provide a short explanation for this figure, too. It is difficult to understand at first glance.\n- In Section 3.6, the term mixed is vague. The authors didn't introduce what the mixer is, and it is found in the appendix. It is important to explain it in the paper and not the appendix. \n\n[1] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In CVPR, 2020.\n\n[2] Jan Zelinka and Jakub Kanis. Neural sign language synthesis: Words are our glosses. In WACV, 2020.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper presents strong comparative evaluations and ablation studies.\n- Integrating the three main components demonstrates thoughtful engineering.\n- The authors plan to release the code.\n\n**Weaknesses**\n- Since the training is not end-to-end and involves multiple separately trained stages, extending the pipeline to more datasets—especially in a multilingual setting—can significantly complicate the task.", "was_revised": true, "final_justification": "**final score**: The paper presents a clear engineering and robust method for sign language video generation, an area that needs more attention in the sign language domain. The method is presented clearly and is supported by thorough experiments. There were some minor issues from my side, and they have been addressed. Based on the rebuttal, I keep my score as **5: Accept**.", "rebuttal": "Thank you for recognizing the evaluations, engineering, and openness of our work. We will address your concerns below. \n\n# Weaknesses \n\n**`W1:`** Since the training involves multiple separately stages, extending the pipeline to more datasets—especially in a multilingual setting—can significantly complicate the task.  \n\n**`R1:`** Although our SignViP framework consists of three training stages, only the final stage (Step III: Multi-Condition Token Translator) is directly language-dependent. The first and second stages are entirely language-agnostic, as they operate solely on videos and associated conditions without considering language categories.  \n\nConsequently, when extending to multilingual datasets, the training of these two stages can be shared across languages. Specifically, the Sign Video Diffusion Model and FSQ Autoencoder can be pre-trained on large-scale human video datasets, even if these videos are not for sign language. Subsequently, for each target language, only the Multi-Condition Token Translator needs to be trained separately.  \n\nThus, our pipeline remains efficient and does not introduce significant complexity when scaling to multiple languages.\n\n# Questions\n\n**`Q1:`** Could you use more references for SLVG (e.g., PGMM [1])?  \n\n**`A1:`** Thank you for pointing out the omission of one relevant work on SLVG. As this work has not released its code and pre-trained weights, we are unable to reproduce it from scratch and include it as a baseline in our experiments within the limited timeframe. However, we will reference and introduce this work in Section 2 (Related Works).\n\n---\n\n**`Q2:`** Do you use Sign Language Transformers (SLT) [2] for video back-translation and the method from [3] for pose back-translation?  \n\n**`A2:`** No. For both pose and video back-translation evaluations, we use SLT as the back-translation model. Specifically, for video back-translation, video frames are input into the SLT model. For pose back-translation, we follow the evaluation protocol of ProTran [4] and modify the original SLT architecture to accept pose points as input.\n\n---\n\n**`Q3:`** What is the resolution of the output video? Is it the same as input?  \n\n**`A3:`** For RWTH-2014T and How2Sign, the output resolutions are 272×224 and 512×512, respectively. The resolution of output video is the same as that of input. Although we have noted the resolution in Appendix B.1, we will include them in Section 4.1 to ensure clarity.\n\n---\n\n**`Q4:`** In Figure 2, what is a Mixer?  \n\n**`A4:`** The Mixer module is a lightweight MLP that aggregates tokens from the same frame into unified hidden states, which is described in Appendix C.\n\n---\n\n**`Q5:`** Please provide a short explanation for Figure 2. It is difficult to understand at first glance.  \n\n**`A5:`** Thanks for your suggestion. We will provide a detailed caption for Figure 2, as follows:  \n\n\"*Figure 2. Framework of our SignViP for sign language video generation (SLVG). (1) The spoken language text is translated into the multi-condition tokens by Multi-Condition Token Translator. (2) These tokens are decoded by FSQ Autoencoder into multi-condition embeddings, which are equivalent to the embeddings of multiple fine-grained conditions (i.e., fine-grained poses and 3D hands) generated by a multi-condition encoder. (3) The embeddings are injected into Sign Video Diffusion Model to guide the generation of sign language videos.*\"\n\n---\n\n**`Q6:`** In Section 3.6, the term \"mixed\" is vague. It is important to explain what the mixer is in the paper and not the appendix.  \n\n**`A6:`** Thanks for your suggestion. We realize that not describing the structure and function of Mixer in the main text may cause confusion for readers, and we will include a detailed description of Mixer in the main part of the paper.\n\n# Paper Formatting Concerns\n\n**`C1:`** In line 257, \"SPL\" is not defined before.  \n\n**`R1:`** Thank you for your reminder. This is a typographical error. It should be \"SLP (Sign Language Production)\" rather than \"SPL\".  We will correct this mistake.\n\n# References\n\n`[1]` Pose-Guided Fine-Grained Sign Language Video Generation, ECCV 2024.  \n`[2]` Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation, CVPR 2020.  \n`[3]` Neural Sign Language Synthesis: Words are Our Glosses. WACV 2020.  \n`[4]` Progressive Transformers for End-to-End Sign Language Production, ECCV 2020."}]

Author rebuttal:

Meta-review: The submission proposed a method on text to Sign Language Video. To generate the video details in multiple scale and key components, multiple fine-grained conditions are applied. It provides details from finger, face to body pose. However, the generated video is not smooth enough from frame to frame. A stabilizer will be helpful to generate the video. Before rebuttal, this submission received two borderline rejects and two accepts. Both reviewers and authors engaged in discussion, the rebuttal persuaded the reviewers and the two reviewers upgraded their rating to borderline accepts. Most of the concerns are well addressed. The AC recommends accepting this submission.
Decision: Accept (spotlight)