Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Paper • 1910.10683 • Published • 20
Request access to Soma v1 Preview
Soma v1 Preview is an early research model for supervised Kenyan Sign Language testing. Ishara reviews requests individually. Please explain who you are and how you plan to use the model.
By requesting access, you acknowledge that Soma v1 Preview is experimental, can produce incorrect text, is not an interpreter, and must not be relied on for emergency communication or independent medical or legal decisions.
Log in or Sign Up to review the conditions and access this model content.
Ishara Soma v1 Preview
An early Kenyan Sign Language-to-English text model for supervised research and community testing.
Soma v1 Preview is experimental. It can return incorrect text with high confidence. It is an assistive recognition and captioning model, not a replacement for a qualified interpreter. Do not rely on it for emergency communication or use it on its own for medical or legal decisions.
What is Soma?
Soma v1 Preview converts a short video clip of Kenyan Sign Language (KSL) into written English. It is the sign-to-text engine behind the Ishara Translate demonstration.
The model does not read raw pixels directly during recognition. A preprocessing step extracts body and hand landmarks from the video. Soma then represents the movement across frames and uses two possible recognition paths:
- Phrase retrieval matches the clip against phrases already represented in its index.
- Text generation uses a T5-small decoder to attempt an English output.
This is a gloss-free approach: the system goes from landmarks to text without requiring an intermediate sign-by-sign gloss sequence.
Soma v1 Preview is being shared so that researchers, Deaf community members, universities and accessibility partners can test it in controlled settings, identify failures and help shape a stronger future release.
Preview status
This release demonstrates progress on short KSL clips, especially for phrases represented during training. It does not solve general KSL translation.
The most important limitations are:
- conversational training data comes primarily from one signer, so multi-signer robustness has not been established;
- unfamiliar phrases are not recognised reliably;
- performance can change with signer, framing, lighting, recording conditions and clip length;
- the retrieval path can only return phrases already present in its index;
- the system can confuse similar phrases and may produce a confident but incorrect result;
- continuous, unsegmented signing and arbitrary fingerspelling are outside the scope of this release.
Any evaluation should let the signer review and correct the returned text before it is treated as meaning.
Evaluation
Conversational KSL benchmark
The conversational expert was trained on 567 clips covering 314 phrases. It was evaluated on 140 held-out clips: 70 development clips and 70 test clips. The evaluation contains 123 clips of phrases represented during training and 17 clips of unfamiliar phrases.
| Recognition path / subset | Result | How to read it |
|---|---|---|
| Phrase retrieval, top-1, all 140 clips | 85.7% (120/140) | Correct first choice across the full held-out pool |
| Phrase retrieval, top-3, all 140 clips | 89.3% | Correct phrase among the first three choices |
| Phrase retrieval, familiar-phrase subset | 95.9% | 123 held-out clips of phrases represented during training |
| Phrase retrieval, unfamiliar-phrase subset | 11.8% (2/17) | The two successes used manual mappings and do not show open generalisation |
| Generative path, all 140 clips | 62.1% exact match; 0.34 WER | Separate result from the retrieval accuracy |
| Generative path, unfamiliar-phrase subset | 0.0% (0/17) | Unfamiliar-phrase generation is not solved |
These figures measure this specific held-out development/test pool. They are not an accuracy estimate for KSL generally or for broad signer coverage.
Signer-disjoint word benchmark
On a separate public 30-class KSL word dataset, with test signers excluded from training, the word-class expert achieved 21.60% top-1 accuracy (chance: 3.33%). This is a different task and should not be compared directly with the conversational retrieval result.
Intended uses
Soma v1 Preview is intended for:
- supervised research and evaluation;
- controlled demonstrations with Deaf participants;
- testing correction and feedback experiences;
- studying KSL landmark-to-text methods;
- accessibility research conducted with appropriate community participation and consent.
Uses outside this release
Do not use Soma v1 Preview:
- as a certified or qualified interpreter;
- for emergency communication;
- to make medical, legal, eligibility, employment or other high-impact decisions;
- as an unattended transcription system;
- to claim broad KSL fluency or support for other sign languages;
- for surveillance, identity inference or attempts to identify a signer;
- in a public-facing service without visible human review and a correction path.
Accessing the files
This repository uses manual gated access. Access is granted to individual Hugging Face accounts after review by Ishara AI.
After your request is approved, authenticate and download the repository:
pip install -U huggingface_hub
hf auth login
from huggingface_hub import snapshot_download
local_path = snapshot_download(
repo_id="IsharaAI/ishara-soma-v1-preview-weights",
)
print(local_path)
This repository is a weights release, not a standard one-line transformers.pipeline() package. Running end-to-end inference also requires the compatible Ishara preprocessing and serving code, configuration, phrase index and menu files. Approved collaborators should contact Ishara for the integration package and current setup guidance.
Model details
| Field | Value |
|---|---|
| Owner | Ishara AI, Nairobi, Kenya |
| Version | Soma v1 Preview; model-card evidence version 2.1, 27 August 2026 |
| Input | Short KSL video converted to per-frame body and hand landmarks |
| Output | English text, a model score and ranked alternative phrases |
| Architecture | Routed mixture of four landmark-encoder/T5-decoder experts, plus phrase retrieval |
| Total parameters | Approximately 299M |
| Active parameters per clip | Approximately 74.8M plus the router |
| Recognition landmarks | 33 body points and 21 points per hand, represented across frames |
| Release stage | Research preview for supervised testing |
The model uses MediaPipe Holistic for landmark extraction. Each routed expert contains a four-layer Transformer landmark encoder and a T5-small text decoder. Only one expert is active for a clip.
Training and evaluation data
The KSL data pool used in the research includes:
| Collection | Size | Role/source |
|---|---|---|
| Conversational phrase collection | 707 clips, 314 phrases | Assembled by Ishara from publicly available KSL dictionary material with human phrase labels |
| General gloss collection | 2,643 clips | Ishara internal collection |
| Medical gloss collection | 1,002 clips | Collected by Ishara with Deaf signers |
| Maseno University KSL pose collection | 1,388 clips, 40 glosses, 40 signers | Provided by the Maseno University research team |
| Public KSL 30-class word dataset | 2,237 clips, multiple signers | Public research dataset used for signer-disjoint evaluation |
| KSL dictionary corpus | 2,646 clips, 2,229 words | Used for research benchmarking |
The Maseno contribution is separate from the 140-clip conversational benchmark. The 85.7% retrieval result should not be read as performance across the 40 Maseno signers.
Raw training datasets are not included in this weights repository. Access to model weights does not grant access to participant recordings or partner datasets.
Responsible testing and reporting
If you test Soma, please:
- obtain appropriate consent before recording or sharing a participant's video;
- explain what will be recorded, retained and used for further training;
- keep a human correction step visible;
- record the signed phrase, predicted phrase, alternatives and relevant recording conditions;
- report both successful and failed examples;
- avoid presenting model scores as calibrated certainty;
- stop testing in situations where an error could cause harm.
Corrections do not improve the deployed model automatically. They must be reviewed, permissioned where necessary, incorporated into development, and followed by a fresh evaluation.
Known failure modes
Observed failure modes include:
- replacing a phrase with a semantically similar known phrase;
- returning sensitive or stigmatising text for an unrelated sign;
- returning harmless text for an emergency phrase;
- routing a clip to the wrong domain;
- degradation when signer or recording conditions differ from training;
- low performance for phrases outside the indexed vocabulary.
Confidence thresholds can flag some errors, but they do not catch every confident mistake. This is why supervised testing and signer confirmation are required.
Access review
Ishara reviews access requests individually. In your request, describe:
- your organisation or affiliation;
- your country;
- the purpose of the work;
- how Deaf people or KSL expertise will be involved, where relevant;
- how you will evaluate outputs and handle recordings;
- whether you need only the weights or also integration support.
Approval provides repository access to the requesting Hugging Face account. It is not approval for access to Ishara or partner datasets, and it does not remove your responsibility to comply with applicable law, consent requirements and the repository's licence terms.
Licence
Soma v1 Preview is provided under the Ishara Soma Research Preview Licence v1.0 for approved research, education and supervised evaluation.
Commercial use, production deployment, model hosting, sublicensing, and redistribution of the original or modified weights require separate written permission from Ishara AI.
Access approval does not grant access to training videos, participant data or partner datasets. Users must comply with applicable consent, privacy and data-protection requirements.
Acknowledgements
Soma v1 Preview was made possible through the work of many contributors:
- AISCA provided compute support over a two-month period;
- Qubit Hub provided compute and technical support;
- Maseno University contributed a KSL pose dataset for research;
- members of the Deaf community participated in data collection and validation;
- the Ishara team developed, evaluated and reviewed the system.
We are grateful for the time, resources and expertise that made this preview possible.
Research context
Google DeepMind introduced its multilingual sign-language-to-text system, SL2T, in August 2026. Its release supports the broader direction of landmark-based, gloss-free sign-language translation, but it does not imply equivalent performance, endorsement or a partnership with Ishara. Soma v1 Preview currently supports KSL-to-English; support for other African sign languages is a longer-term ambition.
- Google DeepMind: Putting sign language AI into users' hands
- MediaPipe Holistic
- T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Contact and collaboration
We welcome responsible collaboration from Deaf organisations, KSL researchers, universities, accessibility teams, funders and technology partners. Areas of support include community-led testing, responsibly sourced data, compute grants, technical expertise and research funding.
Use the repository's Community tab for questions that can be discussed publicly. For private collaboration or data-governance questions, contact Ishara through its approved organisational contact channel.
Citation
If you use the model in research, cite the repository and record the exact commit or revision used:
@software{ishara_soma_v1_preview_2026,
author = {{Ishara AI}},
title = {Ishara Soma v1 Preview},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/IsharaAI/ishara-soma-v1-preview-weights},
note = {Research preview; cite the accessed repository revision}
}
Version history
- Soma v1 Preview / external evidence version 2.1 - 27 August 2026: initial gated research-preview documentation.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support