Package: transformers-cfg
🤗 Transformers CFGLatest NewsIntegration with Text-Generation-WebUI(2023-12-17)We are thrilled to announce thattransformers_cfghas been used in theText-Generation-WebUIproject.
This integration enables users to utilize our CFG capabilities within the popular, 30.5K-starred web interface for text generation.
For more details, seeRelevent Pull RequestIntroductiontransformers_cfgis an extension library for the popular Transformers library by Hugging Face, tailored for working with context-free grammars (CFG).
This package provides additional tools and functionalities to enhance your experience with natural language processing tasks involving CFGs.It was initially developed as a pull request to theHugging Face Transformerslibrary.
See relevant discussionhere.Installationpipinstalltransformers-cfgQuickStart: Force LLM to generate a valid json objectThe below example can be found inexamples/generate_json.pyimporttorchfromtransformersimportAutoModelForCausalLM,AutoTokenizerfromtransformers_cfg.grammar_utilsimportIncrementalGrammarConstraintfromtransformers_cfg.generation.logits_processimportGrammarConstrainedLogitsProcessorif__name__=="__main__":# Detect if GPU is available, otherwise use CPUdevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")print(f"Using device:{device}")model_id="mistralai/Mistral-7B-v0.1"# Load model and tokenizertokenizer=AutoTokenizer.from_pretrained(model_id)tokenizer.pad_token=tokenizer.eos_tokenmodel=AutoModelForCausalLM.from_pretrained(model_id).to(device)# Load model to defined devicemodel.generation_config.pad_token_id=model.generation_config.eos_token_id# Load json grammarwithopen("examples/grammars/json.ebnf","r")asfile:grammar_str=file.read()grammar=IncrementalGrammarConstraint(grammar_str,"root",tokenizer)grammar_processor=GrammarConstrainedLogitsProcessor(grammar)# Generateprefix1="This is a valid json string for http request:"prefix2="This is a valid json string for shopping cart:"input_ids=tokenizer([prefix1,prefix2],add_special_tokens=False,return_tensors="pt",padding=True)["input_ids"]output=model.generate(input_ids,max_length=50,logits_processor=[grammar_processor],repetition_penalty=1.1,num_return_sequences=1,)# decode outputgenerations=tokenizer.batch_decode(output,skip_special_tokens=True)print(generations)"""'This is a valid json string for http request:{ "request": { "method": "GET", "headers": [], "content": "Content","type": "application" }}'This is a valid json string for shopping cart:{ "name": "MyCart", "price": 0, "value": 1 }"""Why should I use transformers-CFG?We support EBNF grammar description formatWe offer the same grammar interface as llama-cpp project, allowing you to drop-in replace llama-cpp with transformers-CFG.We allow you to use any of the models in the 🤗 Transformers library, including the ones that are not supported by llama-cpp.What is grammar ?TL;DR: Think of it as an enhanced version of regular expressions.Here is an example of a simplified JSON grammar:# A JSON object is the root of the grammar
root::=object

# An object starts with "{" and ends with "}" and contains pairs separated by ","
object::="{" pair ("," pair)* "}"

# A pair is a string followed by a ":" and a value
pair::=string ":" value

# A string is a sequence of alphanumeric characters enclosed in double quotes
string::='"' [a-zA-Z0-9]* '"'

# A value can be a string, another object, or a boolean value
value::=string | object | "true" | "false" | "null"This grammar describes the structure of a JSON object. It specifies that a JSON object is a pair of key-value pairs, where the key is a string and the value can be a string, another object, or a boolean value.Grammar doesn't need to be complicated.
You can use it to describe very simple but useful things, like a valid email address, a valid URL, or phone number.phone_number ::= "+" [0-9]+More details can be found in thisdoc from llama-cppAdvanced grammar debugging guide can be foundhereAutomatic Grammar GenerationHere is an awesome tool to generate grammars for you:Grammar BuilderGrammar CollectionWe provide a collection of grammars in theexamples/grammarsfolder, which are mostly identical to the grammars in llama-cpp project.
We try to keep the grammars up-to-date with the original grammars from llama-cpp project.
But up to now, we can not yet guarantee that all grammars from llama-cpp project can be directly used in transformers-CFG.The list of grammars contains:json.ebnf: A grammar for generating valid json objects.json_arr.ebnf: A grammar for generating valid json arrays.c.ebnf: A grammar for generating valid C programs.chess.ebnf: A grammar for generating valid chess moves.arithmetic.ebnf: A grammar for generating valid arithmetic expressions.Supported ModelsLLaMa family modelsGPT family modelsBloom family modelsMistral family modelsFalcon family models...Seesupported_models.yamlfor the full list of supported models.As a rule of thumb, all models with the same tokenizer should naturally be supported.
If you find any model that is not supported, please open an issue or submit a pull request.CitationPlease consider citing our work, if you found the provided resources useful.@inproceedings{geng-etal-2023-grammar,
	title        = {Grammar-Constrained Decoding for Structured {NLP} Tasks without Finetuning},
	author       = {Geng, Saibo  and Josifoski, Martin  and Peyrard, Maxime  and West, Robert},
	year         = 2023,
	month        = dec,
	booktitle    = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
	publisher    = {Association for Computational Linguistics},
	address      = {Singapore},
	url          = {https://aclanthology.org/2023.emnlp-main.674},
	editor       = {Bouamor, Houda  and Pino, Juan  and Bali, Kalika}
}LicenseThis project is licensed under theMIT License.AcknowledgementThis project is derived from thetorch-grammarsproject, which was derived from thellama-cppproject.