SuhaoYu1020/Uni-Sign-ASL718k-stage1-t5large

Uni-Sign ASL-718k · Stage-1 · T5-Large

Two pose-only sign language translation (SLT) models pre-trained from scratch on a 718k-clip American Sign Language corpus, each paired with a frozen-architecture t5-large decoder.

These are Stage-1 pre-training checkpoints at the final epoch (epoch 19 of 20, no checkpoint selection). No downstream fine-tuning has been applied.

They come from an ablation of the pose encoder in Uni-Sign (ICLR 2025). The language model, the data and the optimisation recipe are held fixed across the two runs; only the pose encoder differs.

Checkpoints

Path	Variant	Pose encoder	Total params
A_baseline_pose_nt/checkpoint_19.pth	A — baseline	pose_nt: per-frame spatial Transformer (dim 256, depth 6, 8 heads) → per-part JointPool → fusion FFN → linear projection to T5	759.78 M
E_posept3_n0k4d3p2_lpool/checkpoint_19.pth	E	pose_pt3 configured as A plus a zero-initialised depthwise temporal conv (k=3, the "door conv") right before the T5 projection, and a 2-layer residual FFN after pooling in place of the single fusion FFN. Joint-level cross-part attention is disabled (cross_depth=0).	761.89 M

The 2.1 M parameter difference is exactly the extra post-pooling FFN plus the door conv.

Why E is configured this way

The pose_pt3 family was introduced to add whole-body attention across the 69 joints after the per-frame spatial stage. That turned out to be harmful at every scale tested (−2.5 BLEU-4 with t5-base, −2.1 with t5-large on in-domain dev). Setting cross_depth=0 removes it and keeps only the two components that were positive on their own — the deeper post-pooling FFN and the zero-init door conv. E is therefore best read as "the baseline plus the two changes that survived ablation", not as a different architecture family.

Pose input

69 keypoints per frame, split into four streams: body 9 / left hand 21 / right hand 21 / face 18. Clips are capped at 256 frames. Keypoints were extracted with RTMW-x (rtmw-dw-x-l_simcc-cocktail14_270e) via rtmlib; see the corpus notes below for the two resolutions involved.

Corpus

718,099 clips, ASL → English:

Source	Clips	Pose extraction
YouTube-ASL (Uthus et al., 2023)	566,928	RTMW-x at 256×192 crop input, frames downscaled to width 720
asl_dataset_clean (in-house ASL collection)	151,171	RTMW-x at 384×288 crop input

Split: the loader takes the first 99 % as train (710,918 clips) and the last 1 % as dev (7,181 clips). The dev score is computed on this corpus's own 1 % slice, so it is comparable between A and E but not against numbers reported on any other corpus.

Training recipe

Identical for both runs, and matched to the Uni-Sign Stage-1 recipe (paper Table 2):

Hardware	8 × NVIDIA GH200 (1 GPU per node)
Per-GPU micro-batch	16
Gradient accumulation	4
Effective batch	512
Epochs	20
Optimizer	AdamW, β = (0.9, 0.999), weight decay 1e-4
Peak LR	3e-4
Schedule	cosine decay over 20 epochs, 1 epoch linear warmup
Updates	1,388 per epoch, 27,760 total
Max frames	256

Uni-Sign's own Stage-1 recipe uses the same effective batch of 512 (16 per GPU × 8 accumulation × 4 GPUs) and the same 20 epochs / AdamW / 3e-4 / cosine; the one deliberate difference here is the 1-epoch warmup, which upstream leaves at 0.

Results

BLEU-4 (sacrebleu, 13a tokenizer) and ROUGE-L.

Read this before using the table. Only the dev column belongs to the checkpoint in this repo. The dev scores are logged per epoch during pre-training, so "final" is checkpoint_19.pth. The How2Sign and OpenASL numbers were produced by initialising from best_checkpoint.pth — the best-dev epoch, which is epoch 17 for A and epoch 18 for E, not epoch 19. Those files are not currently in this repo. The two epochs are close (dev 12.35 vs 12.25 for A, 12.50 vs 12.43 for E), but the downstream numbers are strictly not this file's numbers.

"final" is the last epoch; "max" is the best epoch, reported only for transparency — the final-epoch number is the one to compare, since no checkpoint selection was used.

	Stage-1 dev † (this file)	How2Sign zero-shot ‡	How2Sign fine-tuned ‡ (final / max)	OpenASL-clean zero-shot ‡
A	12.25 final / 12.35 max	5.31	13.38 / 14.00	14.57 (ROUGE 32.50)
E	12.43 final / 12.50 max	5.02	13.51 / 13.64	14.77 (ROUGE 32.66)

† 718k corpus's own 1 % slice — not comparable across corpora. ‡ Initialised from best_checkpoint.pth (A epoch 17, E epoch 18), not from the checkpoint_19.pth in this repo.

OpenASL-clean is the official OpenASL test split with every clip whose YouTube video also appears in the 718k pre-training corpus removed (61 % of the official test clips overlap at video level), leaving 337 clips from 206 videos. Without this filtering, OpenASL "zero-shot" numbers on any model pre-trained on YouTube-ASL are inflated by memorisation.

What the two runs say

At this scale the encoder change is close to a wash: E is ahead on in-domain dev (+0.18), on fine-tuned How2Sign (+0.13 final) and on OpenASL-clean (+0.20), but behind on How2Sign zero-shot (−0.29). Single runs, no seed replicates — differences of this size are within noise.

The same encoder change was worth +1.01 dev BLEU-4 with t5-base on a 567k corpus. The gain shrinks monotonically as the language model and the corpus grow, which is the main finding these two checkpoints support: encoder improvements compensate for a small LM and a small corpus, and the lever that keeps paying is data.

Usage

The checkpoints are plain torch.save payloads with the weights under the model key:

import torch

ckpt = torch.load("A_baseline_pose_nt/checkpoint_19.pth", map_location="cpu")
state_dict = ckpt["model"]
missing, unexpected = model.load_state_dict(state_dict, strict=False)

To build a matching model, use the Uni-Sign codebase with --lm_path <path to t5-large> and:

# A
--pose_arch pose_nt \
--pose_nt_dim 256 --pose_nt_depth 6 --pose_nt_heads 8 --pose_nt_mlp_ratio 4 \
--pose_nt_pool_dim 256 --pose_nt_pool_ffn_depth 1 \
--pose_nt_fusion_ffn_depth 1 --pose_nt_fusion_mlp_ratio 1 \
--pose_nt_dropout 0 --pose_nt_drop_path 0 --pose_nt_body_anchor --pose_nt_out_scale 16

# E
--pose_arch pose_pt3 \
--pose_nt_dim 256 --pose_nt_depth 6 --pose_nt_heads 8 --pose_nt_mlp_ratio 4 \
--pose_nt_pool_dim 256 --pose_nt_pool_ffn_depth 1 \
--pose_nt_fusion_ffn_depth 1 --pose_nt_fusion_mlp_ratio 1 \
--pose_nt_dropout 0 --pose_nt_drop_path 0 --pose_nt_body_anchor --pose_nt_out_scale 16 \
--pose_pt3_cross_depth 0 --pose_pt3_rooms whole --pose_pt3_collapse late_pool \
--pose_pt3_post_ffn_depth 2 --pose_pt3_door_k 3 --pose_pt3_door_pos late \
--pose_pt3_readout_k 4 --pose_pt3_part_fuse_depth 0

Caveat: transformers version affects the scores

All numbers above were produced with transformers==4.53.3. Uni-Sign upstream pins 4.40.0, and beam search behaves differently between the two: under 4.53.3 a fraction of generations degenerate into repetition loops and outputs run longer. The direction of the bias is dataset-dependent — on one in-domain dev set 4.40.0 scored +2.9 BLEU-4 higher, while on How2Sign it scored lower because its shorter outputs trigger a harsher brevity penalty. If you compare against these numbers, match the transformers version, or re-evaluate both sides under the same one.

Related

• SuhaoYu1020/Uni-Sign-YTASL-stage1-t5large — the same A and E variants plus a third (Pose-T0), trained on YouTube-ASL only (566,928 clips).
• SuhaoYu1020/Uni-Sign-YTASL-stage1-v2 — the unmodified Uni-Sign architecture (ST-GCN pose encoder + mT5-base) trained on YouTube-ASL.
• Upstream: Uni-Sign · YouTube-ASL · OpenASL

Licensing

These weights are released as research artifacts. Use of them is subject to the licence of the upstream Uni-Sign codebase and to the terms attached to the source corpora (YouTube-ASL videos remain subject to YouTube's terms and their creators' rights). No licence is asserted here beyond that; check those sources before redistribution or commercial use.