Package: cappr
CAPPr: Completion After Prompt ProbabilityCAPPr performs text classification. No training. No post-processing.Just have your LLM pick from a list of choices.Or compute the probability of a completion given a prompt.Squeezemoreout
of open source LLMs.UsageUse a GGUF modelThis model must be able to be loaded usingllama_cpp.Llama.fromllama_cppimportLlamafromcappr.llama_cpp.classifyimportpredict# Load modelmodel=Llama("./TinyLLama-v0.Q8_0.gguf",verbose=False)prompt="""Gary told Spongebob a story:There once was a man from Peru; who dreamed he was eating his shoe. Hewoke with a fright, in the middle of the night, to find that his dreamhad come true.The moral of the story is to"""completions=("look at the bright side","use your imagination","eat shoes",)pred=predict(prompt,completions,model)print(pred)# use your imaginationNotice that a completion can contain many tokens. CAPPr is100% guaranteedto return
an output from the list of possible answers.Seethis page of the
documentationfor more info on using GGUF models.Use a HuggingFace AutoModelForCausalLMThis model must be able to be loaded usingtransformers.AutoModelForCausalLM.from_pretrained.fromtransformersimportAutoModelForCausalLM,AutoTokenizerfromcappr.huggingface.classifyimportpredict# Load a model and its tokenizermodel_name="gpt2"model=AutoModelForCausalLM.from_pretrained(model_name)tokenizer=AutoTokenizer.from_pretrained(model_name)prompt="Which planet is closer to the Sun: Mercury or Earth?"completions=("Mercury","Earth")pred=predict(prompt,completions,model_and_tokenizer=(model,tokenizer))print(pred)# MercurySeethis page of the
documentationfor more info on usingtransformersmodels.Use an AutoGPTQ modelcappr.huggingfaceis
compatible with models loaded viaauto_gptq.AutoGPTQForCausalLM.from_quantized.
Seethis
notebookfor a minimal demo.Note that fortransformers>=4.32.0, you can load GPTQ models usingtransformers.AutoModelForCausalLM.Seethis page of the
documentationfor more info on using these models.Use an AutoAWQ modelcappr.huggingface.classify_no_cacheis compatible with models loaded viaawq.AutoAWQForCausalLM.from_quantized. Seethis
notebookfor a minimal demo.Note that fortransformers>=4.35.0, you can load AWQ models usingtransformers.AutoModelForCausalLM. AWQ models loaded this way are compatible withcappr.huggingface.classify,
which is usually faster.Seethis page of the
documentationfor more info on using these models.Use a model from the OpenAI APIThis model must be compatible with the/v1/completionsendpoint
(excludinggpt-3.5-turbo-instruct).fromcappr.openai.classifyimportpredictprompt="""Tweet about a movie: "Oppenheimer was pretty good. But 3 hrs...cmon Nolan."This tweet contains the following criticism:""".strip("\n")completions=("bad message","too long","unfunny")pred=predict(prompt,completions,model="text-ada-001")print(pred)# too longSeethis page of the
documentationfor more info on using OpenAI models.Extract the final answer from a step-by-step completionStep-by-step and chain-of-thought prompts are highly effective ways to get an LLM to
"reason" about more complex tasks. But if you need a structured output, a step-by-step
completion is unwieldy. Use CAPPr to extract the final answer from these types of
completions, given a list of possible answers.See this idea in actionhere in the
documentation.Run in batchesLet's use a PyTorchtransformersmodel. Also, let's predict probabilities instead of
the class.fromtransformersimportAutoModelForCausalLM,AutoTokenizerfromcappr.huggingface.classifyimportpredict_proba# Load a model and its tokenizermodel_name="gpt2"model=AutoModelForCausalLM.from_pretrained(model_name)tokenizer=AutoTokenizer.from_pretrained(model_name)prompts=["Stephen Curry is a","Martina Navratilova was a","Dexter, from the TV Series Dexter's Laboratory, is a","LeBron James is a",]# Each of the prompts could be completed with one of these:class_names=("basketball player","tennis player","scientist")prior=(1/6,1/6,2/3)# Say I expect most of my data to have scientists# Run CAPPrpred_probs=predict_proba(prompts=prompts,completions=class_names,model_and_tokenizer=(model,tokenizer),batch_size=32,# whatever fits on your CPU/GPUprior=prior,)# pred_probs[i,j] = probability that prompts[i] is classified as class_names[j]print(pred_probs.round(1))# [[0.5 0.3 0.2]#  [0.3 0.6 0.2]#  [0.1 0.1 0.8]#  [0.8 0.2 0. ]]# For each prompt, which completion is most likely?pred_class_idxs=pred_probs.argmax(axis=-1)preds=[class_names[pred_class_idx]forpred_class_idxinpred_class_idxs]print(preds)# ['basketball player',#  'tennis player',#  'scientist',#  'basketball player']Run in batches, where each prompt has a different set of possible completionsAgain, let's use a PyTorchtransformersmodel to predict probabilities.fromtransformersimportAutoModelForCausalLM,AutoTokenizerfromcappr.huggingface.classifyimportpredict_proba_examplesfromcapprimportExample# Load a model and its tokenizermodel_name="gpt2"model=AutoModelForCausalLM.from_pretrained(model_name)tokenizer=AutoTokenizer.from_pretrained(model_name)# Create a sequence of Example objects representing your classification tasksexamples=[Example(prompt="Jodie Foster played",completions=("Clarice Starling","Trinity in The Matrix"),),Example(prompt="Batman, from Batman: The Animated Series, was played by",completions=("Pete Holmes","Kevin Conroy","Spongebob!"),prior=(1/3,2/3,0),),]# Run CAPPrpred_probs=predict_proba_examples(examples,model_and_tokenizer=(model,tokenizer))# pred_probs[i][j] = probability that examples[i].prompt is classified as# examples[i].completions[j]print([example_pred_probs.round(2)forexample_pred_probsinpred_probs])# [array([0.7, 0.3]),#  array([0.03, 0.97, 0.  ])]# For each example, which completion is most likely?pred_class_idxs=[example_pred_probs.argmax()forexample_pred_probsinpred_probs]preds=[example.completions[pred_class_idx]forexample,pred_class_idxinzip(examples,pred_class_idxs)]print(preds)# ['Clarice Starling',#  'Kevin Conroy']See thedemosfor demonstrations
of slightly harder classification tasks.For CAPPr, GPTQ models are the most computationally performant. These models are
compatible withcappr.huggingface.classify. Seethis page of the
documentationfor more info on using these models.Documentationhttps://cappr.readthedocs.ioInstallationSeethis page of the
documentation.MotivationReduce engineering complexity.Seethis page of the
documentationfor more info.PerformanceStatistical performanceFor open source models, seethe 4-bit 4 GB Llama 2COPA
demothe 4-bit 4 GB Llama 2AG News
demothe 4 GB MistralCraigslist Bargains
demothe 4 GB MistralBanking 77
demo,
with 77 multi-token choicesthe 4 GB MistralSciQ
demo, with a
multiple choice promptthe 4 GB MistralTweet Emotions
demo,
which is a multilabel task—each tweet can be tagged with 1-12 emotions.In general, you should expect similar or identical performance to text generation when
every completion is 1 token long.Seethis page of the
documentationfor some discussion.For OpenAI models (some deprecated), see2 SuperGLUE
datasetsRAFT zero-shot training
setsComputational performanceSeethis page of the
documentation.How it worksYou input apromptstring, aend_of_promptstring (a whitespace or empty) and a set
of candidatecompletionstrings such that the string—{prompt}{end_of_prompt}{completion}—is a naturally flowing thought. CAPPr picks thecompletionwhich is mostly likely to
followpromptby computing the:CompletionAfterPromptProbabilityThe method is fleshed out in myquestion on Cross
Validated.Related workSeethis page of the
documentation.Local developmentSeethis page of the documentation.TodoI'm dumping todos here:Code changesReseach experimentsFeel free to raise issues ofc