Package: ouroboros-hf-text-generation
Unofficial Hugging Face Text Generation Utility Package.[OUROBOROS VERSION 0.0.1]Install ouroboros-hf-text-gen-utilsSimple installation from PyPIpipinstallouroboros_hf_text_gen_utilsOther installation optionsInstall from github directlypipinstallgit+https://github.com/VINUK0/Ouroboros-HF-TXT-GEN-UTILS.git[Supported Features]✅ONNX model inferece support via hugging face.✅OPENVINO model inferece support via hugging face.✅NF8 (8bit) and NF4 (4bit) inferece support via hugging face.✅Better Transformers inferece support via hugging face.⚠️Hugging face is slowly dropping better transformers support since(scaled_dot_product_attention)is being added to pytorch nativly.✅Flash Attention 2 inferece support via hugging face⚠️Your GPU architecture need to be Ampere or higher for flash attention 2 to work.(First Ampere based GPU is RTX 3080 as said by google bard.)[Coming In The Future]🚧Save and Load Conversations.🚧Export the conversation's to human readable multiple prompt formats to create datasets.🚧Print total params in the model (I am busy to do it right now.)[Supported Prompting Format's]✅Alpaca Version 1Backend StyleYou are an helpful ai assistant.

### Instruction:
What is a computer.

### Response:
A computer is a programmable machine that manipulates information: it takes data, processes it, and stores it. Think of it as a powerful calculator that can follow instructions to do almost anything!✅Alpaca Version 2(Self Proclaimed)Backend Style### Instruction:
You are an helpful ai assistant.

### Input:
What is a computer.

### Response:
A computer is a programmable machine that manipulates information: it takes data, processes it, and stores it. Think of it as a powerful calculator that can follow instructions to do almost anything!✅ChatMLBackend Style
system:
You are an helpful ai assistant.



user:
What is a computer.



assistant:
A computer is a programmable machine that manipulates information: it takes data, processes it, and stores it. Think of it as a powerful calculator that can follow instructions to do almost anything!
✅Ouroboros(Same as TinyLlama 1B format)Backend Style

You are an helpful ai assistant.

<|user>
What is a computer.



A computer is a programmable machine that manipulates information: it takes data, processes it, and stores it. Think of it as a powerful calculator that can follow instructions to do almost anything!✅Mixtral(This format is confusing to me.)Backend Style
 What is a computer.


 A computer is a programmable machine that manipulates information: it takes data, processes it, and stores it. Think of it as a powerful calculator that can follow instructions to do almost anything![How to basic]⚠️Available dtype's areF32,F16,BF16.⚠️Use can run these using this tooaccelerate launch your_file_name.py⚠️max_sys_prompt_length,max_prompt_lengthandmax_hist_lengthis mesured by letters not tokens.🛠️Simple InfereceCode examplefromouroboros_text_gen_utilsimporttext_generationapi=text_generation(tokenizer_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",model_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",dtype="BF16")history=[]system_prompt="""### Instruction:You are an helpful ai assistant."""history,output=api.inplace_alpaca_v2_style(history=history,system_prompt=system_prompt,prompt="What is a computer.",user_name="user",character_name="assistant",max_sys_prompt_length=2048,max_prompt_length=1024,max_hist_length=8096,max_new_tokens=256,min_new_tokens=10,top_p=0.8,top_k=50,temperature=0.5,repetition_penalty=1.1)print(f"Model Generated Output:{output}")🛠️8 bit and 4 bit inferece.⚠️When loading models in 8 bit or 4 bit make sure the dtype is eitherF16orBF16.Code examplefromouroboros_text_gen_utilsimporttext_generationapi=text_generation(tokenizer_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",model_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",dtype="BF16",load_in_4bit=True)# load_in_8bit=True can be also used.history=[]system_prompt="""### Instruction:You are an helpful ai assistant."""history,output=api.inplace_alpaca_v2_style(history=history,system_prompt=system_prompt,prompt="What is a computer.",user_name="user",character_name="assistant",max_sys_prompt_length=2048,max_prompt_length=1024,max_hist_length=8096,max_new_tokens=256,min_new_tokens=10,top_p=0.8,top_k=50,temperature=0.5,repetition_penalty=1.1)print(f"Model Generated Output:{output}")🛠️Flash Attention 1.⚠️Use can use this withload_in_4bitorload_in_8bittoo.⚠️I have found a error when trying to run the model with flash attention 1 on a T4 GPU inBF16. If that happens useF16instead.Code examplefromouroboros_text_gen_utilsimporttext_generationapi=text_generation(tokenizer_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",model_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",dtype="BF16",flash_attention="flash_attention_1")history=[]system_prompt="""### Instruction:You are an helpful ai assistant."""history,output=api.inplace_alpaca_v2_style(history=history,system_prompt=system_prompt,prompt="What is a computer.",user_name="user",character_name="assistant",max_sys_prompt_length=2048,max_prompt_length=1024,max_hist_length=8096,max_new_tokens=256,min_new_tokens=10,top_p=0.8,top_k=50,temperature=0.5,repetition_penalty=1.1)print(f"Model Generated Output:{output}")🛠️Flash Attention 2.⚠️Use can use this withload_in_4bitorload_in_8bittoo.⚠️You need this package to run FA2pip install flash-attn --no-build-isolationCode examplefromouroboros_text_gen_utilsimporttext_generationapi=text_generation(tokenizer_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",model_name="Doctor-Shotgun/TinyLlama-1.1B-32k-Instruct",dtype="BF16",flash_attention="flash_attention_2")history=[]system_prompt="""### Instruction:You are an helpful ai assistant."""history,output=api.inplace_alpaca_v2_style(history=history,system_prompt=system_prompt,prompt="What is a computer.",user_name="user",character_name="assistant",max_sys_prompt_length=2048,max_prompt_length=1024,max_hist_length=8096,max_new_tokens=256,min_new_tokens=10,top_p=0.8,top_k=50,temperature=0.5,repetition_penalty=1.1)print(f"Model Generated Output:{output}")🛠️Onnx model.⚠️If you have a GPU change the onnx_execution_provider toonnx_execution_provider="CUDAExecutionProvider".Code examplefromouroboros_text_gen_utilsimporttext_generationapi=text_generation(tokenizer_name="example/onnx_model_1B",model_name="example/onnx_model_1B",onnx_model=True,onnx_execution_provider="CPUExecutionProvider")history=[]system_prompt="""### Instruction:You are an helpful ai assistant."""history,output=api.inplace_alpaca_v2_style(history=history,system_prompt=system_prompt,prompt="What is a computer.",user_name="user",character_name="assistant",max_sys_prompt_length=2048,max_prompt_length=1024,max_hist_length=8096,max_new_tokens=256,min_new_tokens=10,top_p=0.8,top_k=50,temperature=0.5,repetition_penalty=1.1)print(f"Model Generated Output:{output}")🛠️OpenVino model.⚠️If you have a GPU avaliable it will try to run from it.Code examplefromouroboros_text_gen_utilsimporttext_generationapi=text_generation(tokenizer_name="example/openvino_model_1B",model_name="example/openvino_model_1B",openvino_model=True)history=[]system_prompt="""### Instruction:You are an helpful ai assistant."""history,output=api.inplace_alpaca_v2_style(history=history,system_prompt=system_prompt,prompt="What is a computer.",user_name="user",character_name="assistant",max_sys_prompt_length=2048,max_prompt_length=1024,max_hist_length=8096,max_new_tokens=256,min_new_tokens=10,top_p=0.8,top_k=50,temperature=0.5,repetition_penalty=1.1)print(f"Model Generated Output:{output}")🛠️Better Transformer.Code examplefromouroboros_text_gen_utilsimporttext_generationapi=text_generation(tokenizer_name="example/openvino_model_1B",model_name="example/openvino_model_1B",better_transformers=True,dtype="BF16")history=[]system_prompt="""### Instruction:You are an helpful ai assistant."""history,output=api.inplace_alpaca_v2_style(history=history,system_prompt=system_prompt,prompt="What is a computer.",user_name="user",character_name="assistant",max_sys_prompt_length=2048,max_prompt_length=1024,max_hist_length=8096,max_new_tokens=256,min_new_tokens=10,top_p=0.8,top_k=50,temperature=0.5,repetition_penalty=1.1)print(f"Model Generated Output:{output}")[All the supported prompt functions]inplace_alpaca_style()inplace_alpaca_v2_style()inplace_chatml_style()inplace_ouroboros_style()inplace_mixtral_style()