Package: sign-language-translator
Sign Language Translator ⠎⠇⠞Build Custom Translators and Translate between Sign Language & Text with AI.OverviewSolutionMajor ComponentsGoalsInstallation🛠️UsagePython🐍Command Line>_LanguagesModelsHow to Build a Translator for your Sign LanguageDirectory TreeHow to ContributeResearch Papers & CitationCredits and GratitudeOverviewSign language consists of gestures and expressions used mainly by the hearing-impaired to talk. This project is an effort to bridge the communication gap between the hearing and the hearing-impaired community using Artificial Intelligence.The goal is to provide a user-friendly translation API and a framework for building sign language translators that can easily adapt to any regional sign language. Unlike most other projects, this python library can translate full sentences and not just the alphabet.A big hurdle is the lack of datasets (global & regional) and frameworks that deep learning engineers and software developers can use to build useful products for the target community. This project aims to empower sign language translation by providing robust components, tools, datasets and models for both sign language to text and text to sign language conversion. It aims to facilitate the creation of sign language translators for any region, while building the way towards sign language standardization.SolutionI've have built anextensible rule-basedtext-to-sign translation system that can be used to generate training data forDeep Learningmodels for both sign to text & text to sign translation.To create a rule-based translation system for your regional language, you can inherit the TextLanguage and SignLanguage classes and pass them as arguments to the ConcatenativeSynthesis class. To write sample texts of supported words, you can use our language models. Then, you can use that system to fine-tune our AI models. See thedocumentationand ourdatasetsfor more.Major ComponentsSign language to TextExtract features from sign language videosSee theslt.models.video_embeddingsub-package and the$ slt embedcommand.Currently Mediapipe 3D landmarks are being used for deep learning.Transcribe and translate signs into multiple text languages to generalize the model.To train for word-for-word gloss writing task, also use a synthetic dataset made by concatenating signs for each word in a text. (Seeslt.models.ConcatenativeSynthesis)Fine-tune a neural network, such as one fromslt.models.sign_to_textor the encoder of any multilingual seq2seq model, on your dataset.Text to Sign LanguageThere are two approaches to this problem:Rule Based ConcatenationLabel a Sign Language Dictionary with all word tokens that can be mapped to those signs. See our mapping formathere.Parse the input text and play appropriate video clips for each token.Build a text processor by inheritingslt.languages.TextLanguage(seeslt.languages.textsub-package for details)Map the text grammar & words to sign language by inheritingslt.languages.SignLanguage(seeslt.languages.signsub-package for details)Use our rule-based modelslt.models.ConcatenativeSynthesisfor translation.It is faster but theword sense has to be disambiguatedin the input. See the deep learning approach to automatically handle ambiguous words &words not in dictionary.Deep learning (seq2seq)Either generate the sequence of filenames that should be concatenatedyou will need aparallel corpusof normal text sentences against sign language gloss (sign sequence written word-for-word)Or synthesize the signs directly by using a pre-trained multilingual text encoder anda GAN or diffusion model or decoder to synthesize a sequence of pose vectors (shape = (time, num_landmarks * num_coordinates))Move an Avatar with those pose vectors (Easy)Use motion transfer to generate a video (Medium)Synthesize a video frame for each vector (Difficult)a video synthesis model (Very Difficult)Language ProcessingSign Processing3D world landmarks extraction with Mediapipe.Pose Visualization with matplotlib.Pose transformations (data augmentation) with scipy.Text ProcessingNormalize text input by substituting unknown characters/spellings with supported words.Disambiguate context-dependent words to ensure accurate translation.
"spring" -> ["spring(water-spring)", "spring(metal-coil)"]Tokenize text (word & sentence level).Classify tokens and mark them with Tags.Data Collection and CreationSee our dataset conventionshere.Utilities available in this package:Clip extraction from long videos using timestamps (notebook)Multithreaded Web scrapingLanguage Models to generate sentences composed of supported wordsTry to capture variations in signs in a scalable and diversity accommodating way and enable advancing sign language standardization efforts.DatasetsFor our datasets, see thesign-language-datasetsrepoand itsreleases.
See thesedocsfor more on building a dataset of Sign Language videos (or motion capture gloves' output features).Your data should include:A word level Dictionary (Videos of individual signs & corresponding Text tokens (words & phrases))Parallel sentencesNormal text language sentences against sign language videos.Normal text language sentences against thetext glossof the corresponding sign language sentence.Sign language sentences against their text glossSign language sentences against translations in multiple text languagesGrammatical rules of the sign languageWord order (e.g. SUBJECT OBJECT VERB TIME)Meaningless words (e.g. "the", "am", "are")Ambiguous words (e.g. spring(coil) & spring(water-fountain))Try to incorporate:Multiple camera anglesDiverse performers to capture allaccentsof the signsUniqueness in labeling of word tokensVariations in signs for the same conceptGoalsEnable integration of sign language into existing applications.Improve education quality for the deaf and increase litracy rates.Promote communication inclusivity of the deaf.How to install the packagepipinstallsign-language-translatorEditable mode (git clone):gitclonehttps://github.com/sign-language-translator/sign-language-translator.gitcdsign-language-translator
pipinstall-e.pipinstall-egit+https://github.com/sign-language-translator/sign-language-translator.git#egg=sign_language_translatorUsageHead over tosign-language-translator.readthedocs.ioto see the detailed usage in Python & CLI.See thetest casesor thenotebooksrepoto see the internal code in action.Also see theHow to build a custom sign language translatorsection.Python# Documentation: https://sign-language-translator.readthedocs.ioimportsign_language_translatorasslt# print(slt.TextLanguageCodes, slt.SignLanguageCodes)# The core model of the project (rule-based text-to-sign translator)# which enables us to generate synthetic training datasetsmodel=slt.models.ConcatenativeSynthesis(text_language="urdu",sign_language="pk-sl",sign_format="video")text="یہ بہت اچھا ہے۔"# "This very good is."sign=model.translate(text)# tokenize, map, download & concatenatesign.show()# sign.save(f"{text}.mp4")# model.text_language = "hindi"# sign = model.translate("पाँच घंटे।") # "5 hours."importsign_language_translatorasslt# # Load sign-to-text model (pytorch) (COMING SOON!)# translation_model = slt.get_model(slt.ModelCodes.Gesture)embedding_model=slt.models.MediaPipeLandmarksModel()sign=slt.Video("video.mp4")embedding=embedding_model.embed(sign.iter_frames())# text = translation_model.translate(embedding)# print(text)sign.show()# slt.Landmarks(embedding, connections="mediapipe-world").show()# custom translator (https://sign-language-translator.readthedocs.io/en/latest/#building-custom-translators)help(slt.languages.SignLanguage)help(slt.languages.text.Urdu)help(slt.models.ConcatenativeSynthesis)# help(slt.models.TransformerLanguageModel)Command Line$slt

Usage:slt[OPTIONS]COMMAND[ARGS]...SignLanguageTranslator(SLT)commandlineinterface.Documentation:https://sign-language-translator.readthedocs.io
Options:--versionShowtheversionandexit.--helpShowthismessageandexit.
Commands:assetsAssetsmanagertodownload&displayDatasets&Models.completeCompleteasequenceusingLanguageModels.embedEmbedVideosUsingSelectedModel.translateTranslatetextintosignlanguageorviceversa.Generate training examples: write a sentence with a language model and synthesize a sign language video from it with a single command:slttranslate--model-coderule-based--text-langurdu--sign-langpk-sl--sign-formatvideo\"$(sltcomplete'<'--model-codeurdu-mixed-ngram--join'')"LanguagesText LanguagesAvailable Functions:Text NormalizationTokenization (word, phrase & sentence)Token Classification (Tagging)Word Sense DisambiguationNameVocabularyAmbiguous tokensSignsUrdu2090 words+phrases227790Hindi34 words+phrases516Sign LanguagesAvailable Functions:Word & phrase mapping to signsSentence restructuring according to grammarSentence simplification (drop stopwords)NameVocabularyDatasetParallel CorpusPakistan Sign Language78923 hoursn gloss sentences with translations in m text languagesModelsTranslation: Text to sign LanguageNameArchitectureDescriptionInputOutputConcatenative SynthesisRules + Hash TablesThe Core Rule-Based translator mainly used to synthesize translation dataset.Initialize it using TextLanguage, SignLanguage & SignFormat objects.stringslt.SignSign Embedding/Feature extraction:NameArchitectureDescriptionInput formatOutput formatMediaPipe Landmarks(Pose + Hands)CNN based pipelines. See Here:Pose,HandsEncodes videos into pose vectors (3D world or 2D image) depicting the movements of the performer.List of numpy images(n_frames, height, width, channels)torch.Tensor(n_frames, n_landmarks * 5)Data generation: Language ModelsAvailable Trained modelsNameArchitectureDescriptionInput formatOutput formatN-Gram Langauge ModelHash TablesPredicts the next token based on learned statistics about previous N tokens.List of tokens(token, probability)Transformer Language ModelDecoder-only Transformers (GPT)Predicts next token using query-key-value attention, linear transformations and soft probabilities.torch.Tensor(batch, token_ids)List of tokenstorch.Tensor(batch, token_ids, vocab_size)(token, probability)How to Build a Translator for Sign LanguageTo create your own sign language translator, you'll need these essential components:Data CollectionGather a collection ofvideosfeaturing individuals performing sign language gestures.Prepare aJSON filethat maps video file names to corresponding text language words, phrases, or sentences that represent the gestures.Prepare aparallel corpuscontaining text language sentences and sequences of sign language video filenames.Language ProcessingImplement a subclass ofslt.languages.TextLanguage:Tokenize your text language and assign appropriate tags to the tokens for streamlined processing.Create a subclass ofslt.languages.SignLanguage:Map text tokens to video filenames using the provided JSON data.Rearrange the sequence of video filenames to align with the grammar and structure of sign language.Rule-Based TranslationPass instances of your classes from the previous step toslt.models.ConcatenativeSynthesisclass to obtain a rule-based translator object.Construct sentences in your text language and use the rule-based translator to generate sign language translations. (You can use our language models to generate such texts.)Model Fine-TuningUtilize the sign language videos and corresponding text sentences from the previous step.Apply our training pipeline to fine-tune a chosen model for improved accuracy and translation quality.Remember to contribute back to the community:Share your data, code, and models by creating a pull request (PR), allowing others to benefit from your efforts.Create your own sign language translator (e.g. as your university thesis) and contribute to a more inclusive and accessible world.See thecodeatBuild Custom Translator section in ReadTheDocsor in thisnotebook.Directory Treesign-language-translator├──.readthedocs.yaml├──MANIFEST.in├──README.md├──poetry.lock├──pyproject.toml├──requirements.txt├──docs│   └──*├──tests│   └──*│
└──sign_language_translator├──cli.py────────────────────────────────────────────────────── `> slt` command line interface├──assets(auto-downloaded)│   └──*│
    ├──config│   ├──assets.py───────────────────────────────────────── download, extract and remove models & datasets│   ├──enums.py─────────────────────────────────────────── string short codes to identify models & classes│   ├──settings.py────────────────────────────────────── global variables in repository design-pattern│   ├──urls.json│   └──utils.py│
    ├──languages│   ├──utils.py│   ├──vocab.py─────────────────────────────────────────── reads word mapping datasets│   ├── sign
    │   │   ├──mapping_rules.py────────────────────── strategy design-pattern for word mapping│   │   ├──pakistan_sign_language.py│   │   └──sign_language.py────────────────────── Base class for sign mapping and sentence restructuring│   │
    │   └── text
    │       ├──english.py│       ├──hindi.py│       ├──text_language.py────────────────────── Base class for text normalization, tokenization & tagging│       └──urdu.py│
    ├──models│   ├──_utils.py│   ├──utils.py│   ├── language_models
    │   │   ├──abstract_language_model.py│   │   ├──beam_sampling.py│   │   ├──mixer.py───────────────────────────────────── wrap multiple models into a single object│   │   ├──ngram_language_model.py────────── uses hash-tables & frequency to predict next token│   │   └── transformer_language_model
    │   │       ├──layers.py│   │       ├──model.py───────────────────────────── decoder-only transformer with controllable vocabulary│   │       └──train.py│   │
    │   ├── sign_to_text
    │   ├── text_to_sign
    │   │   ├──concatenative_synthesis.py──── join sign clip of each word in text using rules│   │   └──t2s_model.py───────────────────────────── Base class│   │
    │   ├── text_embedding
    │   └── video_embedding
    │       ├──mediapipe_landmarks_model.py─ 3D coordinates of points on body│       └──video_embedding_model.py──────── Base class│
    ├──text│   ├──metrics.py──────────────────────────────────────── numeric score techniques│   ├──preprocess.py│   ├──subtitles.py───────────────────────────────────── WebVTT│   ├──synonyms.py│   ├──tagger.py────────────────────────────────────────── classify tokens to assist in mapping│   ├──tokenizer.py───────────────────────────────────── break text into words, phrases, sentences etc│   └──utils.py│
    ├──utils│   ├──archive.py──────────────────────────────────────── zip datasets│   ├──arrays.py────────────────────────────────────────── common interface for numpy.ndarray and torch.Tensor│   ├──download.py│   ├──parallel.py────────────────────────────────────── multi-threading│   ├──tree.py───────────────────────────────────────────── print file hierarchy│   └──utils.py│
    └──vision├──_utils.py├──utils.py├── landmarks
        ├── sign
        │   └──sign.py────────────────────────────────────── Base class to wrap around sign clips│
        └── video
            ├──display.py───────────────────────────────── jupyter notebooks inline video & pop-up in CLI├──transformations.py─────────────────── strategy design-pattern for image augmentation├──video_iterators.py─────────────────── adapter design-pattern for video reading└──video.pyHow to ContributeDatasets:See our datasets & conventionshere.Contribute by scraping, compiling, and centralizing video datasets.Help with labelingword mapping datasets.Establish connections with Academies for the Deaf to collaboratively develop standardizedsign language grammarand integrate it into the rule-based translators.New Code:Create dedicated sign language classes catering to various regions.Develop text language processing classes for diverse languages.Experiment with training models using diverse hyper-parameters.Don't forget to integratestring short codesof your classes and models intoenums.py, and ensure to update functions likeget_model()andget_.*_language().Enhance the codebase with comprehensive docstrings, exemplary usage cases, and thorough test cases.Existing Code:Optimize the codebase by implementing techniques like parallel processing and batching.Strengthen the project's documentation with clear docstrings, illustrative usage scenarios, and robust test coverage.Contribute to the documentation forsign-language-translator ReadTheDocsto empower users with comprehensive insights. Currently it needs a template for auto-generated pages.Product Development:Engage in the development efforts acrossMLOps,backend,web, andmobiledomains, depending on your expertise and interests.Research Papers & CitationStay Tuned!Upcoming/RoadmapTEXT_EMBEDDING: v0.7# text embedding models# synonyms by similarityLANDMARKS_WRAPPER: v0.8# landmarks wrapper class# landmark augmentation# subtitles# trim signs before concatenation# stabilize video batch using landmarksLANGUAGES: v0.9# improve Urdu code# implement NLP class for English# expand dictionary video data by scraping everythingMISCELLANEOUS# bugfix: inaccurate num_frames in video file metadata# improvement: video wrapper class uses list of sources instead of linked list of videos# video transformations# clean demonstration notebooks# * host video dataset online, descriptive filenames, zip extraction# dataset info table# sequence diagram for creating a translator# make deep_translator optional.# GUI with gradioDEEP_TRANSLATION: v1.X# parallel text corpus# LanguageModel: Dropping space tokens, bidirectional prediction & train on max vocab but mask with supported only when inferring.# sign to text with custom seq2seq transformer# sign to text with fine-tuned whisper# pose vector generation with fine-tuned mBERT# custom 3DLandmark model (training data = mediapipe's output on activity recognition or any dataset)# motion transfer# pose2video: stable diffusion or GAN?# speech to text# text to speechRESEARCH PAPERs# datasets: clips, text, sentences, disambiguation# rule based translation: describe entire repo# deep sign-to-text: pipeline + experiments# deep text-to-sign: pipeline + experimentsPRODUCT DEVELOPMENT# ML inference server# Django backend server# React Frontend# React Native mobile appCredits and GratitudeThis project started in October 2021 as a BS Computer Science final year project with 3 students and 1 supervisor. After 9 months at university, it became a hobby project for Mudassar who has continued it till at least 2024-01-17.Immense gratitude towards: (click to expand)Mudassar Iqbalfor coding the project so far.Rabbia Arshadfor help in initial R&D and web development.Waqas Bin Abbas for assistance in initial video data collection process.Kamran Malik for setting the initial project scope, idea of motion transfer and connecting us with Hamza Foundation.Hamza Foundation(especially Ms Benish, Ms Rashda & Mr Zeeshan) for agreeing to collaborate and providing their sign dictionary, hearing-impaired performers for data creation, and creating the text2gloss dataset.UrduHackfor their work on Urdu character normalization.Telha Bilalfor help in designing the architecture of some modules.BonusCount total number oflines of code(Package:9,960+ Tests:1,622):gitls-files|grep'\.py'|xargswc-lJust forfunQ: What was the deaf student's favorite course?
A: Communication skills