Hugging Face model: 5dimension/sentinel-universal-tokenizer
Author: 5dimension
Created: 2026-04-30 03:03:31
Last modified: 2026-04-30 16:06:54
Pipeline: text-generation
Library: transformers
Likes: 0 | Downloads: 0
Tags: transformers, tokenizer, multimodal, sentinel-manifold, universal-tokenizer, bpe, byte-level, multilingual, image-tokens, audio-tokens, video-tokens, text-tokens, mathematics, gradient-axiom, text-generation, en, fr, de, es, zh, ja, ar, ru, ko, hi, pt, it, nl, pl, vi, th, tr, uk, sv, license:mit, endpoints_compatible, region:us
Card metadata:
language: en, fr, de, es, zh, ja, ar, ru, ko, hi, pt, it, nl, pl, vi, th, tr, uk, sv, multilingual
license: mit
tags: tokenizer, multimodal, sentinel-manifold, universal-tokenizer, bpe, byte-level, multilingual, image-tokens, audio-tokens, video-tokens, text-tokens, mathematics, gradient-axiom
library_name: transformers
pipeline_tag: text-generation