FatimahEmadEldin/ytasl-mt5-base

YouTube-ASL (YT-ASL) Fine-Tuned mt5-base for Sign Language Translation

Multilingual T5 base (mT5) fine-tuned on YouTube-ASL keypoint features for multilingual Sign Language Translation.

Architecture

• Base Architecture: google/mt5-base
• Input Dimension: 208 (MediaPipe Holistic keypoints: pose + hands)
• Visual Projection: Linear projection (208 -> 768) with GELU activation
• Training Strategy: Pretrained / Fine-tuned on YouTube-ASL with Adafactor optimizer

Usage

import torch
from transformers import AutoModel, AutoTokenizer

# Load model and tokenizer
repo_id = "FatimahEmadEldin/ytasl-mt5-base"
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo_id)

# Example input pose sequence [batch_size, sequence_length, 208]
batch_size, seq_len, feature_dim = 1, 100, 208
dummy_features = torch.randn(batch_size, seq_len, feature_dim)
attention_mask = torch.ones(batch_size, seq_len)

# Generate translation
model.eval()
with torch.no_grad():
    output_ids = model.generate(features=dummy_features, attention_mask=attention_mask)
    translation = tokenizer.decode(output_ids[0], skip_special_tokens=True)
    print("Translation:", translation)

Citation

If you use this model, please cite the YouTube-ASL research and the Saudi Sign Language translation benchmark.