Package: simple-generation
Simple GenerationSimple Generation offers a minimal interface to run text generation with HuggingFace checkpoint.
The core idea is to ship many neat features out of the box and avoid boilerplate.This is mainly for personal use and simple hardware setups (ideally, single-node single- or multi-gpu).Moreover, please note thatthe library will apply some (sensible) defaults (on where to place models, generation configuration, and so on) which might not suit your use caseand should be edited accordingly. Please head todefaultsto see a list of things you should be aware of.Install:pipinstallsimple-generationOr, install it from source with:pipinstallgit+https://github.com/MilaNLProc/simple-generation.gitFeaturesgenerate with any model that can be loaded withAutoModelForCausalLMorAutoModelForSeq2SeqLMbatched inference for speed (batch_size=256)auto find the largest batch size fitting in your accelerator (batch_size="auto",starting_batch_size=512)torch.compile the model for speed (compile_model=True)load and attach LoRA weights (lora_weights=...)chat templates for modern chat models (apply_chat_template=Truein__call__)carbon emission estimates usingcodecarbonsparsity and fused kernels for speed withoptimum(use_bettertransformer=True)DDP for single-node, multi-gpu setups usingaccelerate. SeeDistributed InferenceGUI for quick interaction with models using Gradio'sChatInterface. SeeChat InterfaceLoading a Modelfromsimple_generationimportSimpleGeneratormodel_name="meta-llama/Llama-2-7b-chat-hf"generator=SimpleGenerator(model_name,load_in_8bit=True)Any named argument toSimpleGeneratorwill be passed thefrom_pretrainedHF method. For example, you canload models with 8bit or 4bit quantization (load_in_[4|8]bit=True)set torch dtypes (torch_dtype=torch.bfloat16)use auto GPUs placement and inference (device_map="auto")Running Inferencetexts=["Write a poem in the style of Leonardo Da Vinci","Tell me what's 2 + 2.","Translate the following sentence to Spanish: I went to the supermarket to buy a lighter."]responses=generator(texts,apply_chat_template=True,add_generation_prompt=True)The__call__function accepts several named arguments (see examples below). For example:return only the generated text (skip_prompt=True)periodic logging of decoded samples (log_batch_sample=)WIPauto find the best device placement for speedefficient duplicate and quasi-duplicate removalsupport system prompt chat formatting following standard templates (e.g., Vicuna, LLaMA 2)support auto gptq quantized models and tentatively GGMLspawn web app to quickly local test conversation with gradioeven faster inference engine withvllmdistributed inference for single-node, multi-gpu setupsWhat Is Not Supportedframeworks other thantorchmodels not in the Huggingface Hubexample-specific decoding parameters (i.e., given a batch of samples passed to the__call__, we will apply the same set of parameters for every sample)ExamplesGetting Startedfromsimple_generationimportSimpleGeneratormodel_name="google/flan-t5-xxl"gen=SimpleGenerator(model_name,load_in_8bit=True)texts=["Today is a great day to run a bit. Translate this to Spanish?","Today is a great day to run a bit. Translate this to German?",]responses=gen(texts,max_new_tokens=128,do_sample=False,num_beams=4,skip_prompt=False)The script will generate aemissions.csvfile with estimated emissions.LoRA examplegen=SimpleGenerator(model_name_or_path="yahma/llama-7b-hf",lora_weights="tloen/alpaca-lora-7b",load_in_8bit=True,)texts=["Write a recipe where the first step is to preheat the oven to 350 degrees.","What is 2 + 2?","What is the capital of France?","There are 5 apples and 3 oranges in a basket. How many pieces of fruit are in the basket?","There are 5 apples and 3 oranges in a basket. How many pieces of fruit are in the basket? Let's think step by step.",]responses=gen(texts,max_new_tokens=256,do_sample=True,num_beams=1,batch_size="auto")This code will, in sequence:load a base llama model in 8bitattach Alpaca LoRA weights to itrun inference on the given texts finding the largest batch size fitting the available resourcesChat TemplatesStarting from v0.2.0, we leverage Hugging Face'schat templating system. You can activate it by usingapply_chat_template=Truewhen invoking__call__.
You can also enable the generation prompt by settingadd_generation_prompt=True. Seeherewhy that might be a good idea.Note: chat templates are not enabled by default!Chat InterfaceThe main SimpleGenerator class exposes the methodgui(). If invoked, it fires up a local chat interface to interact with the model.
Note that, since everything will run locally, you can fill up your VRAM quite easily with long chats. Keep an eye on your VRAM usage and clean the chat frequently -- you might notice that memory does not get freed up immediately, but cleaning the chat will reuse the already allocated memory for new chats.model_name="mistralai/Mistral-7B-Instruct-v0.2"generator=SimpleGenerator(model_name,torch_dtype=torch.bfloat16,device="cuda:0")generator.gui(do_sample=True,max_new_tokens=256,temperature=0.8,top_p=0.85,top_k=50,)Simple TranslationThis is the minimum code to translate the En->It EuroParl split using a Opus MT neural model.generator=SimpleGenerator("Helsinki-NLP/opus-mt-en-it")dataset=load_dataset("europarl_bilingual",lang1="en",lang2="it")["train"]texts=[sample["translation"]["en"]forsampleindataset]references=[sample["translation"]["it"]forsampleindataset]output=generator(texts,skip_prompt=False,num_beams=5,max_new_tokens=256,starting_batch_size=128,)Multiple-Request ConversationThe library supports creating a conversation by prompting models with multiple requests. I.e., it is possible to build a multi-turn conversation with fixed user requests. You can use theconversation_from_user_prompts()method, that accepts the same arguments of__call__.For example:fromsimple_generationimportSimpleGeneratorimporttorchtexts=["What kind of noises did dinosaurs make?","What is the most popular programming language?","What is 2 + 2?","Tell me how to make a cake.",]generator=SimpleGenerator("lmsys/vicuna-7b-v1.3",load_in_8bit=True,torch_dtype=torch.bfloat16,)conversation=generator.conversation_from_user_prompts(texts,do_sample=True,top_p=0.95,temperature=0.1,max_new_tokens=512,)print("Conversation:")print(conversation)will print:Conversation:
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: What kind of noises did dinosaurs make? ASSISTANT: It is difficult to know for sure what noises dinosaurs made, as they lived millions of years ago and did not leave behind any recorded sounds. However, scientists can make educated guesses about what some dinosaurs might have sounded like based on their physical characteristics and behavior.

Some scientists have suggested that certain dinosaurs, such as birds, may have made sounds similar to those of modern birds. For example, some scientists believe that the ancestors of modern birds may have made a variety of sounds, including calls, chirps, and trills.

Other scientists have suggested that some dinosaurs, such as theropod dinosaurs, may have made sounds similar to those of modern reptiles, such as hissing or grunting sounds.

It is also possible that some dinosaurs may have made sounds that are not similar to any sounds made by modern animals. For example, some scientists have suggested that certain dinosaurs, such as some of the large herbivorous dinosaurs, may have made sounds similar to those of modern elephants or rhinoceroses.

Overall, it is difficult to know for sure what noises dinosaurs made, but scientists can make educated guesses based on their physical characteristics and behavior.
USER: What is the most popular programming language? ASSISTANT: It is difficult to determine the most popular programming language, as different programming languages are popular for different purposes and in different contexts. Some programming languages are more popular for certain types of applications, while others are more popular for others.

For example, Python is a popular programming language for data science and machine learning, while Java is a popular programming language for developing enterprise-level applications. C++ is a popular programming language for developing operating systems and other low-level systems software, while JavaScript is a popular programming language for developing web applications.

Overall, the popularity of a programming language can depend on a variety of factors, including the needs of the project, the skills and preferences of the developer, and the availability of resources and support for the language.
USER: What is 2 + 2? ASSISTANT: 2 + 2 is equal to 4. This is a basic arithmetic calculation, and the answer is not dependent on any specific context or situation.
USER: Tell me how to make a cake. ASSISTANT: Here is a basic recipe for making a cake:

Ingredients:

* 1 1/2 cups all-purpose flour
* 1 cup granulated sugar
* 1/2 cup unsalted butter, softened
* 2 large eggs
* 1 teaspoon baking powder
* 1/2 teaspoon salt
* 1 cup milk
* 1/2 cup vegetable oil

Instructions:

1. Preheat the oven to 350°F (175°C). Grease and flour a 9-inch (23 cm) round cake pan.
2. In a medium bowl, whisk together the flour, baking powder, and salt.
3. In a large mixing bowl, beat the softened butter and sugar together until light and fluffy.
4. Beat in the eggs one at a time, then stir in the flour mixture until just combined.
5. Stir in the milk and vegetable oil until smooth.
6. Pour the batter into the prepared cake pan and smooth the top.
7. Bake for 30-35 minutes, or until a toothpick inserted into the center of the cake comes out clean.
8. Remove the cake from the oven and let it cool in the pan for 10 minutes. Then, remove the cake from the pan and let it cool completely on a wire rack.

This is a basic recipe for a cake, and there are many variations and modifications that can be made to suit your preferences and the occasion for which you are making the cake. For example, you can add flavorings such as vanilla extract or chocolate chips, or use a different type of flour or sugar if you have a specific dietary need or preference.

Distributed InferenceSimple Generation supports DDP to run distributed inference in Single-Node, Multi-GPUs setups. Note that a copy of the model will be instantiated in each GPU (instead of smart weights placements across multiple GPUs withdevice_map="auto"), soeach of your GPU will need to have enough space to fit a copy of the model.The only change you'll need to take is launching your script withaccelerate. E.g.,:acceleratelaunch--num_processes2examples/inference.py# uses 2 GPUsNote: if you do not specify--num_processesall local GPUs will be used.Some timing tests on 2xA5000 with Llama-2-7b-chat and 384 input prompts:# single GPUCUDA_VISIBLE_DEVICES=0pythonexamples/inference.py
>>217s# two GPUs, smart placementCUDA_VISIBLE_DEVICES=0,1pythonexamples/inference.py
>>219s# two GPUs, DDPCUDA_VISIBLE_DEVICES=0,1acceleratelaunch--num_processes2examples/inference.py
>>105sDefaultsIf not specified we set some sensible defaults.Please note that they might not fit your use case.Default Generation ConfigurationIf not specified otherwise, the library will use the generation configs listed below,overriding the default config loaded from the specified model.@dataclasses.dataclassclassDefaultGenerationConfig(GenerationConfig):"""Default generation config.We apply this parameters to any .generate() call, unless they are not overridden."""max_new_tokens:int=512do_sample:bool=True# set to False for greedy decodingtemperature:float=0.7top_p:float=1.0top_k:int=50num_return_sequences:int=1We set the tokenizer to use left padding. This is required for batched inference withAutoModelForCausalLMbut should also be fine with any otherAutoModelForSeq2SeqLMsince we use attention masks.WarningThere seem to be instabilities while using 4bit quantization (not related to this library). Use it only if strictly necessary.AcknowledgmentsThanks to Paul Röttger for the many inputs and priceless bug hunting.Reference@misc{milanlp-2023-simple-generation,author={Giuseppe Attanasio},title={{S}imple {G}eneration},year={2023},publisher={GitHub},journal={GitHub Repository},howpublished={\url{https://github.com/MilaNLProc/simple-generation}}}