--- base_model: Unbabel/TowerInstruct-7B-v0.1 license: cc-by-nc-4.0 language: - tt - en - de - fr - zh - pt - nl - ru - ko - it - es tags: - tweety datasets: - oscar-corpus/OSCAR-2301 --- Tweety-Tatar-7B: A Tatar Large Language Model # Tweety Tatar / Hydra-Base 7b / 2024-v1 ## Model description This model is our Hydra LLM for the [Tatar language](https://en.wikipedia.org/wiki/Tatar_language), converted from the [TowerInstruct-7b-v0.1](https://huggingface.co/Unbabel/TowerInstruct-7B-v0.1) model trained by Unbabel. Hydra LLMs are trans-tokenized language models finetuned to produce output in a particular language, while accepting input encoded using either their own tokenizer, the one of their base model, or a mix of both. This enables them to receive code-switched input in both their native language and other languages, which is an ideal setup for translation tasks, or retrieval-augmented generation (RAG) in cross-lingual scenarios. - **Developed by:** [François Remy](https://huggingface.co/FremyCompany) (UGent), [Alfiya Khabibullina](https://huggingface.co/justalphie) (BeCode), [et al.](#citation) - **Funded by:** IDLab / GPULab - **Model type:** Foundation model using the mistral architecture - **Language(s) (NLP):** Tatar - **License:** Creative Commons Attribution Non Commercial 4.0 ## In-scope usage This model can be used as-is to answer questions in Tatar based on a cross-lingual context, or finetuned into a machine translation system from one of the 10 languages supported by TowerInstruct into the Tatar language. This list of languages nobably includes English and Russian. The model performs best when translating sentences or small paragraphs, and is not suited for document translation tasks. This model should not be used in the reverse direction, to translate Tatar into English. When the system isn't finetuned, enabling beam search is recommended for best results. We also provide a model [finetuned for translation](https://huggingface.co/Tweeties/tweety-tatar-hydra-trans-7b-2024-v1), but take note of the non-commercial license imposed by Unbabel on the base model. ## Usage instructions Using this model usually requires building the prompts by mixing tokens from two tokenizers, the original TowerInstruct tokenizer for input in the source language, and the new Tatar tokenizer for the prompt and output, as described in the examples below: ```py import re import torch import torch.nn as nn import transformers MODEL_NAME = "Tweeties/tweety-tatar-hydra-base-7b-2024-v1" MAIN_TOKENIZER_NAME = "Tweeties/tweety-tatar-hydra-base-7b-2024-v1" UTIL_TOKENIZER_NAME = "Unbabel/TowerInstruct-7B-v0.1" model = transformers.AutoModelForCausalLM.from_pretrained(MODEL_NAME, trust_remote_code=True) main_tokenizer = transformers.LlamaTokenizerFast.from_pretrained(MAIN_TOKENIZER_NAME) util_tokenizer = transformers.LlamaTokenizerFast.from_pretrained(UTIL_TOKENIZER_NAME) main_tokenizer_len = len(main_tokenizer) ``` ### Cross-lingual question answering ```py def answer_english_question(english_text: str) -> str: # craft the input input_ids = torch.concat([ main_tokenizer.encode(f"Татар телендә түбәндәге сорауга җавап бирегез:\n", return_tensors='pt'), util_tokenizer.encode(f"{english_text}", add_special_tokens=False, return_tensors='pt') + torch.tensor([main_tokenizer_len]), main_tokenizer.encode(f"\n\nҗавап:\n", add_special_tokens=False, return_tensors='pt') ], axis=1) # prevent the model from repeating the prompt prompt_starts = [ main_tokenizer.encode("Түбәндәге"), main_tokenizer.encode("\nТүбәндәге")[2:], main_tokenizer.encode("Текстны"), main_tokenizer.encode("\nТекстны")[2:] ] # prevent the model from repeating the English text english_starts = [ main_tokenizer.encode(re.sub(r'[ ].*', '', english_text)), main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text))[2:], main_tokenizer.encode(re.sub(r'[ ].*', '', english_text.upper())), main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text.upper()))[2:], ] # genereate the output model_inputs = {'input_ids':input_ids.to(model.device)} model_outputs = model.generate( **model_inputs, max_new_tokens=5, num_beams=8, no_repeat_ngram_size=6, early_stopping=False, pad_token_id=main_tokenizer.eos_token_id, eos_token_id=main_tokenizer.convert_tokens_to_ids(['<0x0A>','']), bad_words_ids=english_starts+prompt_starts ) # decode the output return (main_tokenizer.decode(model_outputs[0][input_ids.shape[1]:])) answer_english_question("Is Paris located in France?\n") # Әйе, Парижда ``` ### Machine Translation (see [finetuned model](https://huggingface.co/Tweeties/tweety-tatar-hydra-trans-7b-2024-v1)) ```py def translate_english_text(english_text: str) -> str: # craft the input input_ids = torch.concat([ main_tokenizer.encode(f"Түбәндәге текстны инглиз теленнән татар теленә тәрҗемә итегез:\n", return_tensors='pt'), util_tokenizer.encode(f"{english_text}", add_special_tokens=False, return_tensors='pt') + torch.tensor([main_tokenizer_len]), main_tokenizer.encode(f"\nТекстны татар теленә тәрҗемә итү:\n", add_special_tokens=False, return_tensors='pt') ], axis=1) # prevent the model from repeating the prompt prompt_starts = [ main_tokenizer.encode("Түбәндәге"), main_tokenizer.encode("\nТүбәндәге")[2:], main_tokenizer.encode("Текстны"), main_tokenizer.encode("\nТекстны")[2:] ] # prevent the model from repeating the English text english_starts = [ main_tokenizer.encode(re.sub(r'[ ].*', '', english_text)), main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text))[2:], main_tokenizer.encode(re.sub(r'[ ].*', '', english_text.upper())), main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text.upper()))[2:], ] # genereate the output model_inputs = {'input_ids':input_ids.to(model.device)} model_outputs = model.generate( **model_inputs, max_new_tokens=128, num_beams=8, no_repeat_ngram_size=6, early_stopping=False, pad_token_id=main_tokenizer.eos_token_id, eos_token_id=main_tokenizer.convert_tokens_to_ids(['<0x0A>','']), bad_words_ids=english_starts+prompt_starts ) # decode the output return (main_tokenizer.decode(model_outputs[0][input_ids.shape[1]:])) translate_english_text("The city of Paris is very pretty.") # Париж шәһәре бик матур. ``` ## Citation If you use this model, please cite our work as: ``` @article{tweeties2024, title = {Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP}, author = {François Remy and Pieter Delobelle and Hayastan Avetisyan and Alfiya Khabibullina and Miryam de Lhoneux and Thomas Demeester}, url = {https://raw.githubusercontent.com/LAGoM-NLP/transtokenizer/paper/Trans-Tokenization.pdf}, year = {2024}, note = {Under review at COLM 2024} } ```