Tweeties/tweety-tatar-hydra-base-7b-2024-v1

Tweety Tatar / Hydra-Base 7b / 2024-v1

Model description

This model is our Hydra LLM for the Tatar language, converted from the TowerInstruct-7b-v0.1 model trained by Unbabel. Hydra LLMs are trans-tokenized language models finetuned to produce output in a particular language, while accepting input encoded using either their own tokenizer, the one of their base model, or a mix of both. This enables them to receive code-switched input in both their native language and other languages, which is an ideal setup for translation tasks, or retrieval-augmented generation (RAG) in cross-lingual scenarios.

• Developed by: François Remy (UGent), Alfiya Khabibullina (BeCode), et al.
• Funded by: IDLab / GPULab
• Model type: Foundation model using the mistral architecture
• Language(s) (NLP): Tatar
• License: Creative Commons Attribution Non Commercial 4.0

In-scope usage

This model can be used as-is to answer questions in Tatar based on a cross-lingual context, or finetuned into a machine translation system from one of the 10 languages supported by TowerInstruct into the Tatar language. This list of languages nobably includes English and Russian. The model performs best when translating sentences or small paragraphs, and is not suited for document translation tasks. This model should not be used in the reverse direction, to translate Tatar into English. When the system isn't finetuned, enabling beam search is recommended for best results. We also provide a model finetuned for translation, but take note of the non-commercial license imposed by Unbabel on the base model.

Usage instructions

Using this model usually requires building the prompts by mixing tokens from two tokenizers, the original TowerInstruct tokenizer for input in the source language, and the new Tatar tokenizer for the prompt and output, as described in the examples below:

import re
import torch
import torch.nn as nn
import transformers

MODEL_NAME = "Tweeties/tweety-tatar-hydra-base-7b-2024-v1"
MAIN_TOKENIZER_NAME = "Tweeties/tweety-tatar-hydra-base-7b-2024-v1"
UTIL_TOKENIZER_NAME = "Unbabel/TowerInstruct-7B-v0.1"

model = transformers.AutoModelForCausalLM.from_pretrained(MODEL_NAME, trust_remote_code=True)
main_tokenizer = transformers.LlamaTokenizerFast.from_pretrained(MAIN_TOKENIZER_NAME)
util_tokenizer = transformers.LlamaTokenizerFast.from_pretrained(UTIL_TOKENIZER_NAME)

main_tokenizer_len = len(main_tokenizer)

Cross-lingual question answering

def answer_english_question(english_text: str) -> str:
    
    # craft the input
    input_ids = torch.concat([
        main_tokenizer.encode(f"Татар телендә түбәндәге сорауга җавап бирегез:\n", return_tensors='pt'),
        util_tokenizer.encode(f"{english_text}", add_special_tokens=False, return_tensors='pt') + torch.tensor([main_tokenizer_len]),
        main_tokenizer.encode(f"\n\nҗавап:\n", add_special_tokens=False, return_tensors='pt')
    ], axis=1)

    # prevent the model from repeating the prompt
    prompt_starts = [
        main_tokenizer.encode("Түбәндәге"), 
        main_tokenizer.encode("\nТүбәндәге")[2:], 
        main_tokenizer.encode("Текстны"), 
        main_tokenizer.encode("\nТекстны")[2:]
    ]

    # prevent the model from repeating the English text
    english_starts = [
        main_tokenizer.encode(re.sub(r'[ ].*', '', english_text)), 
        main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text))[2:], 
        main_tokenizer.encode(re.sub(r'[ ].*', '', english_text.upper())),
        main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text.upper()))[2:],
    ]
    
    # genereate the output
    model_inputs = {'input_ids':input_ids.to(model.device)}
    model_outputs = model.generate(
        **model_inputs,
        max_new_tokens=5,
        num_beams=8,
        no_repeat_ngram_size=6,
        early_stopping=False,
        pad_token_id=main_tokenizer.eos_token_id,
        eos_token_id=main_tokenizer.convert_tokens_to_ids(['<0x0A>','</s>']),
        bad_words_ids=english_starts+prompt_starts
    )

    # decode the output
    return (main_tokenizer.decode(model_outputs[0][input_ids.shape[1]:]))

answer_english_question("Is Paris located in France?\n") # Әйе, Парижда

Machine Translation (see finetuned model)

def translate_english_text(english_text: str) -> str:
    
    # craft the input
    input_ids = torch.concat([
        main_tokenizer.encode(f"Түбәндәге текстны инглиз теленнән татар теленә тәрҗемә итегез:\n", return_tensors='pt'),
        util_tokenizer.encode(f"{english_text}", add_special_tokens=False, return_tensors='pt') + torch.tensor([main_tokenizer_len]),
        main_tokenizer.encode(f"\nТекстны татар теленә тәрҗемә итү:\n", add_special_tokens=False, return_tensors='pt')
    ], axis=1)

    # prevent the model from repeating the prompt
    prompt_starts = [
        main_tokenizer.encode("Түбәндәге"), 
        main_tokenizer.encode("\nТүбәндәге")[2:], 
        main_tokenizer.encode("Текстны"), 
        main_tokenizer.encode("\nТекстны")[2:]
    ]

    # prevent the model from repeating the English text
    english_starts = [
        main_tokenizer.encode(re.sub(r'[ ].*', '', english_text)), 
        main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text))[2:], 
        main_tokenizer.encode(re.sub(r'[ ].*', '', english_text.upper())),
        main_tokenizer.encode('\n'+re.sub(r'[ ].*', '', english_text.upper()))[2:],
    ]
    
    # genereate the output
    model_inputs = {'input_ids':input_ids.to(model.device)}
    model_outputs = model.generate(
        **model_inputs,
        max_new_tokens=128,
        num_beams=8,
        no_repeat_ngram_size=6,
        early_stopping=False,
        pad_token_id=main_tokenizer.eos_token_id,
        eos_token_id=main_tokenizer.convert_tokens_to_ids(['<0x0A>','</s>']),
        bad_words_ids=english_starts+prompt_starts
    )

    # decode the output
    return (main_tokenizer.decode(model_outputs[0][input_ids.shape[1]:]))

translate_english_text("The city of Paris is very pretty.") # Париж шәһәре бик матур.

Citation

If you use this model, please cite our work as:

@article{tweeties2024,
    title = {Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP},
    author = {François Remy and Pieter Delobelle and Hayastan Avetisyan and Alfiya Khabibullina and Miryam de Lhoneux and Thomas Demeester},
    url = {https://raw.githubusercontent.com/LAGoM-NLP/transtokenizer/paper/Trans-Tokenization.pdf},
    year = {2024},
    note = {Under review at COLM 2024}
}