Package: nutcracker-py
Nutcracker - Large Model Evaluations and Experimentslike LM-Eval-Harness but without PyTorch madness. Bring-your-own-API + straightforward data managementhttps://github.com/brucewlee/nutcracker/assets/54278520/151403fc-217c-486c-8de6-489af25789ceInstallationRoute 1. PyPI (Recommended)Install Nutcrackerpipinstallnutcracker-pyDownload Nutcracker DBgitclonehttps://github.com/brucewlee/nutcracker-dbRoute 2. GitHubInstall Nutcrackergitclonehttps://github.com/brucewlee/nutcracker
pipinstall-enutcrackerDownload Nutcracker DBgitclonehttps://github.com/brucewlee/nutcracker-dbCheck all tasks implemented inNutcracker DB's readme page.QuickStartCase Study: Evaluate (Any) LLM API on TruthfulQA (Script)STEP 1: Define ModelDefine a simple model class with a "respond(self, user_prompt)" function.We will use OpenAI here. But really, any api can be evaluated if the "respond(self, user_prompt)" function that returns LLM response in string exists. Get creative (Hugginface API, Anthropic API, Replicate API, OLLaMA, and etc.)fromopenaiimportOpenAIimportos,logging,syslogging.basicConfig(level=logging.INFO)logging.getLogger('httpx').setLevel(logging.CRITICAL)os.environ["OPENAI_API_KEY"]=""client=OpenAI()classChatGPT:def__init__(self):self.model="gpt-3.5-turbo"defrespond(self,user_prompt):response_data=Nonewhileresponse_dataisNone:try:completion=client.chat.completions.create(model=self.model,messages=[{"role":"user","content":f"{user_prompt}"}],timeout=15,)response_data=completion.choices[0].message.contentbreakexceptKeyboardInterrupt:sys.exit()except:print("Request timed out, retrying...")returnresponse_dataSTEP 2: Run Evaluationfromnutcracker.dataimportTask,Pilefromnutcracker.runsimportSchemafromnutcracker.evaluatorimportMCQEvaluator,generate_report# this db_directory value should work off-the-shelf if you cloned both repositories in the same directorytruthfulqa=Task.load_from_db(task_name='truthfulqa-mc1',db_directory='nutcracker-db/db')# sample 20 for demotruthfulqa.sample(20,in_place=True)# running this experiment updates each instance's model_response property in truthfulqa data object with ChatGPT responsesexperiment=Schema(model=ChatGPT(),data=truthfulqa)experiment.run()# running this evaluation updates each instance's response_correct property in truthfulqa data object with evaluationsevaluation=MCQEvaluator(data=truthfulqa)evaluation.run()foriinrange(0,len(truthfulqa)):print(truthfulqa[i].user_prompt)print(truthfulqa[i].model_response)print(truthfulqa[i].correct_options)print(truthfulqa[i].response_correct)print()print(generate_report(truthfulqa,save_path='accuracy_report.txt'))Case Study: Task vs. Pile? Evaluating LLaMA on MMLU (Script)STEP 1: Understand the basis of NutcrackerDespite our lengthy history of model evaluation, my understanding of the field is that we have not reached a clear consensus on what a "benchmark" is (Is MMLU a "benchmark"? Is Huggingface Open LLM leaderboard a "benchmark"?).Instead of using the word benchmark, Nutcracker divides the data structure into Instance, Task, and Pile (See blog post:HERE)Nutcracker DB is constructed on the Task-level but you can call multiple Tasks together on the Pile-level.STEP 2: Define ModelSince we've tried OpenAI API above, let's now try Hugginface Inference Endpoint. Most open-source models are accessible through this option. (See blog post:HERE)classLLaMA:def__init__(self):self.API_URL="https://xxxx.us-east-1.aws.endpoints.huggingface.cloud"defquery(self,payload):headers={"Accept":"application/json","Authorization":"Bearer hf_XXXXX","Content-Type":"application/json"}response=requests.post(self.API_URL,headers=headers,json=payload)returnresponse.json()defrespond(self,user_prompt):output=self.query({"inputs":f"

 
 You are a helpful assistant. You keep your answers short. 
{user_prompt}",})returnoutput[0]['generated_text']STEP 3: Load Datafromnutcracker.dataimportPileimportlogginglogging.basicConfig(level=logging.INFO)mmlu=Pile.load_from_db('mmlu','nutcracker-db/db')STEP 4: Run Experiment (Retrieve Model Responses)Running evaluation updates each instance'smodel_responseattribute within the data object, which is mmlu Pile in this case.You can save data object at any step of the evaluation. Let's try saving this time to prevent API requesting again in case anything happens.fromnutcracker.runsimportSchemammlu.sample(n=1000,in_place=True)experiment=Schema(model=LLaMA(),data=mmlu)experiment.run()mmlu.save_to_file('mmlu-llama.pkl')You can load and check how the model responded.loaded_mmlu=Pile.load_from_file('mmlu-llama.pkl')foriinrange(0,len(loaded_mmlu)):print("\n\n\n---\n")print("Prompt:")print(loaded_mmlu[i].user_prompt)print("\nResponses:")print(loaded_mmlu[i].model_response)STEP 5: Run EvaluationLLMs often don’t respond in immediately recognizable letters like A, B, C, or D.Therefore, Nutcracker supports an intent-matching feature (requires OpenAI API Key) that parses model response to match discrete labels, but let’s disable that for now and proceed with our evaluation.We recommend using intent-matching for almost all use cases. We will publish a detailed research later.fromnutcracker.evaluatorimportMCQEvaluator,generate_reportevaluation=MCQEvaluator(data=loaded_mmlu,disable_intent_matching=True)evaluation.run()print(generate_report(loaded_mmlu,save_path='accuracy_report.txt'))https://github.com/brucewlee/nutcracker/assets/54278520/6deb5362-fd48-470e-9964-c794425811d9TutorialsEvaluating on HuggingFace Inference Endpoints ->HERE / MediumUnderstanding Instance-Task-Pile ->HERE / Medium