Package: bocoel
☂️ BoCoELBayesian Optimization as a Coverage Tool for Evaluating Large Language Models🤔 Why BoCoEL?Large language models are expensive and slow behemoths, and evaluating them on gigantic modern datasets only makes it worse.If only there is a way to just select a meaningful (and small) subset of the corpus and obtain a highly accurate evaluation.....Wait, sounds likeBayesian Optmization!Bocoel works in the following steps:Encode individual entry into embeddings (way cheaper / faster than LLM and reusable).Use Bayesian optimization to select queries to evaluate.Use the queries to retrieve from our corpus (with the encoded embeddings).Profit.The evaluations generated are easily managed by the provided manager utility.🚀 Features🎯 Accurately evaluate large language models with just tens of samples from your selected corpus.💂‍♂️ Uses the power of Bayesian optimization to select an optimal set of samples for language model to evaluate.💯 Evalutes the corpus on the model in addition to evaluating the model on corpus.🤗 Support forGPT2,Pythia,LLAMAand more through integration with huggingfacetransformersanddatasets🧩 Modular design.⭐ Give us a star!Like what you see? Please consider giving this a star (★)!♾️ Bayesian OptimizationSimply put, Bayesian optimization aims to optimize either the exploration objective (the purple area in the image) or the exploitation object (the height of the black dots). It uses Gaussian processes as a backbone for inference, and uses anacquisition functionto decide where to sample next. Seeherefor an a more in-depth introduction.SinceBayesian optimization works well with expensive-to-evaluate black-box model (paraphrase: LLM), it is perfect for this particular use case. Bocoel uses Bayesian optimization as a backbone for exploring the embedding space given by our corpus, which allows it to select a good subset acting as a mini snapshot of the corpus.🏎️ Performance ImplicationsLLMs are painfully slow, especially generative ones (which is what is usually referred to as LLM), since sequence generation is sequential by nature.Despitebocoel's requirement to use an embedder to encode the entire corpus, embedders are faster than LLMs by orders of magnitude and the time is gained back by practically any savings in evaluating LLMs.⬇️ InstallationI don't want optional dependencies:pip install bocoelGive me the full experience (all optional dependencies):pip install "bocoel[all]"🔬 UsageSee the folderexamples/getting_startedfor a simplistic usage of the library to get started with just a few lines of code.✍️ Develop with BoCoELUsage examples are under the folderexamples. API reference can be foundhere.🥰 ContributingContributors wanted! Don't be shy. Feel free to file issues and PRs. For PRs, please follow the guide oncontributingand thecode of conduct. Openness and inclusiveness are taken very seriously.🗺️ Roadmap: work in progress🪑 Simpler usage. I should provide a high level wrapper for the entire library s.t. evaluations can be run in one line.📊 Visualization module of the evaluation.🎲 Integration of alternative methods (random, kmedoids...) with Gaussian process.🥨 Integration with more backends such asVLLMandOpenAI's API.🆕 Support for Python 3.11+🏷️ License and CitationThe code is available underApache License.If you find this project helpful in your research, please cite this work at@misc{bocoel2024,
    title = {BoCoEL: Bayesian Optimization as a Coverage Tool for Evaluating Large Language Models},
    url = {https://rentruewang.github.io/bocoel/research/},
    author = {Wang, RenChu and Chuang, Yung-Sung},
    month = {January},
    year = {2024}
}