0xRafie/uni-sign-sibi

Uni-Sign (pose) fine-tuned for SIBI isolated sign recognition

A Uni-Sign pose-only model fine-tuned to recognize 114 isolated signs of the Indonesian Sign Language System (SIBI): 18 affixes, 41 numbers, 26 letters of the alphabet, and 29 words. It was trained on all clips of the 22 signers of the SIBI dataset of Darmawan et al., with no test split, so that it can be used for new signers.

Code: https://github.com/0xRafie/sibi-islr-benchmark

Dataset: SIBI dataset of Darmawan et al., available from its owner under a data use agreement (not included here)

Base model: ZechengLi19/Uni-Sign, from Uni-Sign: Toward Unified Sign Language Understanding at Scale

Expected accuracy on new signers

The model has no test set of its own. Its expected accuracy on a signer it has never seen is the SI result of the same recipe in our benchmark, the Uni-Sign (pose) row below. SI (signer-independent) is leave-one-signer-out cross-validation over 22 folds; SD (signer-dependent) is stratified 5-fold cross-validation over clips, the protocol of earlier SIBI work.

Accuracy (%) under SI: per instance (P-I), and per class (P-C) for Overall. The BiLSTM under SD is shown for reference.

Input	Model	Protocol	Affix	Number	Alphabet	Word	Overall P-I	Overall P-C
MediaPipe	BiLSTM (Ho and Santoso)	SD	95.92	99.44	98.73	98.89	97.29	97.34
MediaPipe	BiLSTM (Ho and Santoso)	SI	83.70	95.13	90.59	95.92	85.72	86.09
MediaPipe	SignBart	SI	86.16	91.64	95.71	97.16	86.68	87.17
RTMW	DSTA-SLR	SI	83.48	97.32	90.44	99.07	88.57	89.29
RTMW	Uni-Sign (pose)	SI	93.56	96.98	96.36	99.25	95.16	95.40
DINOv2 + MediaPipe	SHuBERT	SI	84.52	80.37	86.42	99.42	83.10	84.27

Category columns use the within-category score. Overall uses the 114-class score.

Files

• sibi_pose_only_islr.pth: the fine-tuned weights (Uni-Sign pose encoder, mT5 encoder, and a 114-class linear classifier), named after the original wlasl_pose_only_islr.pth.
• classes.json: the output index, category, and class name of each of the 114 classes.

Training

• Initialization: wlasl_pose_only_islr.pth from ZechengLi19/Uni-Sign (pretrained on YouTube-ASL and fine-tuned on WLASL).
• Input: 69 of the 133 RTMW whole-body keypoints per frame, with their confidence scores, extracted with rtmlib.
• Recipe: the official Uni-Sign isolated sign recognition settings (AdamW, cosine schedule, label smoothing), 20 epochs, batch size 32, fp32, on one NVIDIA Tesla T4 (about 4.3 hours).
• Data: 22,671 clips from 22 signers; the word clips are cut from sentence videos by forced alignment.

Usage

Load the weights with the Uni-Sign code (commit eed438b) and the fine-tuning script of our benchmark, scripts/unisign_finetune.py, at github.com/0xRafie/sibi-islr-benchmark, which builds the same model and keypoint input.

Data and license

The SIBI dataset is available from its owner under a data use agreement for non-commercial research and is not included here. The weights are released under CC BY-NC 4.0, the license of the Uni-Sign weights they start from.

The data use agreement asks users to cite the dataset and its related articles:

• I. D. M. B. A. Darmawan, Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, "Indonesian Sign Language System (SIBI) Dataset," Mendeley Data, V3, 2024. https://doi.org/10.17632/44pbrbsnkh.3
• I. D. M. B. A. Darmawan, Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, "Indonesian sign language system (SIBI) dataset: Sentences enhanced by diverse facial expressions for total communication," Data in Brief, vol. 60, 111642, 2025. https://doi.org/10.1016/j.dib.2025.111642
• I. D. M. B. A. Darmawan, Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, R. Pulungan, Mulyanto, and N. K. D. Hariyanti, "Advancing total communication in SIBI: A proposed conceptual framework for sign language translation," in Proc. ICSGTEIS, 2023, pp. 23–28. https://doi.org/10.1109/ICSGTEIS60500.2023.10424020
• I. D. M. B. A. Darmawan, Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, "Temporal action segmentation in Sign Language System for Bahasa Indonesia (SIBI) videos using optical flow-based approach," Jurnal Ilmu Komputer dan Informasi, vol. 17, no. 2, pp. 195–202, 2024. https://doi.org/10.21609/jiki.v17i2.1284

Limitations

All signers come from one dataset recorded in one setting, so accuracy for signers from other schools, novice signers, or other cameras may be lower than the figures above.

Citation

The paper is under review. A citation will be added here once it is published.