Package: promptbench
PromptBench: A Unified Library for Evaluating and Understanding Large Language Models.Paper·Documentation·Leaderboard·More papersTable of ContentsNews and UpdatesIntroductionInstallationUsageDatasets and ModelsBenchmark ResultsAcknowledgmentsNews and Updates[15/12/2023] Add detailed instructions for users to add new modules (models, datasets, etc.)examples/add_new_modules.md.[05/12/2023] Published promptbench 0.0.1.IntroductionPromptBenchis a Pytorch-based Python package for Evaluation of Large Language Models (LLMs). It provides user-friendly APIs for researchers to conduct evaluation on LLMs. Check the technical report:https://arxiv.org/abs/2312.07910.What does promptbench currently provide?Quick model performance assessment:We offer a user-friendly interface that allows for quick model building, dataset loading, and evaluation of model performance.Prompt Engineering:We implemented several prompt engineering methods. For example:Few-shot Chain-of-Thought[1],Emotion Prompt[2],Expert Prompting[3] and so on.Evaluating adversarial prompts:promptbench integratedprompt attacks[4], enabling researchers to simulate black-box adversarial prompt attacks on models and evaluate their robustness.Dynamic evaluation to mitigate potential test data contamination:we integrated the dynamic evaluation frameworkDyVal[5], which generates evaluation samples on-the-fly with controlled complexity.InstallationInstall viapipWe provide a Python packagepromptbenchfor users who want to start evaluation quickly. Simply runpipinstallpromptbenchInstall via GitHubFirst, clone the repo:gitclonegit@github.com:microsoft/promptbench.gitThen,cdpromptbenchTo install the required packages, you can create a conda environment:condacreate--namepromptbenchpython=3.9then use pip to install required packages:pipinstall-rrequirements.txtNote that this only installed basic python packages. For Prompt Attacks, it requires to install textattacks.Usagepromptbench is easy to use and extend. Going through the bellowing examples will help you familiar with promptbench for quick use, evaluate an existing datasets and LLMs, or creating your own datasets and models.Please seeInstallationto install promptbench first. We provide ipynb tutorials for:evaluate models on existing benchmarks:please refer to theexamples/basic.ipynbfor constructing your evaluation pipeline.test the effects of different prompting techniques:examine the robustness for prompt attacks, please refer toexamples/prompt_attack.ipynbto construct the attacks.use DyVal for evaluation:please refer toexamples/dyval.ipynbto construct DyVal datasets.Supported Datasets and ModelsDatasetsWe support a range of datasets to facilitate comprehensive analysis, including:GLUE: SST-2, CoLA, QQP, MRPC, MNLI, QNLI, RTE, WNLIMMLUSQuAD V2IWSLT 2017UN MultiMathBool Logic (BigBench)Valid Parentheses (BigBench)Object Tracking (BigBench)Date (BigBench)GSM8KCSQA (CommonSense QA)NumersenseQASCLast Letter ConcatenateModelsgoogle/flan-t5-largedatabricks/dolly-v1-6bllama2 (7b, 13b, 7b-chat, 13b-chat)vicuna-13b, vicuna-13b-v1.3cerebras/Cerebras-GPT-13BEleutherAI/gpt-neox-20bgoogle/flan-ul2palmchatgpt, gpt4Benchmark ResultsPlease refer to ourbenchmark websitefor benchmark results on Prompt Attacks, Prompt Engineering and Dynamic Evaluation DyVal.TODOAdd prompt attacks and prompt engineering documents.AcknowledgementstextattacksREADME TemplateWe thank the volunteers: Hanyuan Zhang, Lingrui Li, Yating Zhou for conducting the semantic preserving experiment in Prompt Attack benchmark.Reference[1] Jason Wei, et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." arXiv preprint arXiv:2201.11903 (2022).[2] Cheng Li, et al. "Emotionprompt: Leveraging psychology for large language models enhancement via emotional stimulus." arXiv preprint arXiv:2307.11760 (2023).[3] BenFeng Xu, et al. "ExpertPrompting: Instructing Large Language Models to be Distinguished Experts" arXiv preprint arXiv:2305.14688 (2023).[4] Zhu, Kaijie, et al. "PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts." arXiv preprint arXiv:2306.04528 (2023).[5] Zhu, Kaijie, et al. "DyVal: Graph-informed Dynamic Evaluation of Large Language Models." arXiv preprint arXiv:2309.17167 (2023).Citing promptbench and other research papersPlease cite us if you fine this project helpful for your project/paper:@article{zhu2023promptbench2,
  title={PromptBench: A Unified Library for Evaluation of Large Language Models},
  author={Zhu, Kaijie and Zhao, Qinlin and Chen, Hao and Wang, Jindong and Xie, Xing},
  journal={arXiv preprint arXiv:2312.07910},
  year={2023}
}

@article{zhu2023promptbench,
  title={PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts},
  author={Zhu, Kaijie and Wang, Jindong and Zhou, Jiaheng and Wang, Zichen and Chen, Hao and Wang, Yidong and Yang, Linyi and Ye, Wei and Gong, Neil Zhenqiang and Zhang, Yue and others},
  journal={arXiv preprint arXiv:2306.04528},
  year={2023}
}

@article{zhu2023dyval,
  title={DyVal: Graph-informed Dynamic Evaluation of Large Language Models},
  author={Zhu, Kaijie and Chen, Jiaao and Wang, Jindong and Gong, Neil Zhenqiang and Yang, Diyi and Xie, Xing},
  journal={arXiv preprint arXiv:2309.17167},
  year={2023}
}

@article{chang2023survey,
  title={A survey on evaluation of large language models},
  author={Chang, Yupeng and Wang, Xu and Wang, Jindong and Wu, Yuan and Zhu, Kaijie and Chen, Hao and Yang, Linyi and Yi, Xiaoyuan and Wang, Cunxiang and Wang, Yidong and others},
  journal={arXiv preprint arXiv:2307.03109},
  year={2023}
}ContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.If you have a suggestion that would make promptbench better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the projectCreate your branch (git checkout -b your_name/your_branch)Commit your changes (git commit -m 'Add some features')Push to the branch (git push origin your_name/your_branch)Open a Pull RequestTrademarksThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must followMicrosoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.