Package: leaf-playground
Introductionleaf-playgroundis a "definition driven development" framework to build scenario simulation projects that human and LLM-based agents can participant in together to compete to or co-operate with each other. It is primarily designed to efficiently evaluate the performance of LLM-based agents at the action level in specific scenarios or tasks, but it also possesses enormous potential for LLM native applications, such as developinga language-based game.Apart from the framework itself, a bunch of CLI commands are provided to help developers speedup the process of building a scenario simulation project, and easily deploy a server with aWEB UIwhere users can create simulation tasks, manually and(or) automatically evaluate agents' performance, visualize the simulation process and evaluation results.Below are sister projects ofleaf-playground:leaf-playground-webui: the implementation of the leaf-playground's WEB UI.leaf-playground-hub: hosts our officially implemented scenario simulation projects.Features"Definition Driven Development": advanced syntax for structured scenario definitions and programming conventions.Human + Multiple Agents: facilitates human and AI Agents interaction in designated scenarios.Auto Evaluation: automated action-level evaluation and report visualization for AI Agents.Local server support: one-click local service deployment for scenario simulation tasks management and execution.Containerization: containerization support for running scenario simulation tasks.Auto generate projects: auto-generate and auto-complete code for scenario simulation projects.Debug Friendly: support remote debugger across processes in Pycharm professional IDE.InstallationEnvironment SetupMake sure you havePythonandNode.jsinstalled on your computer, if not, you can set up the environment by following instructions:installPython: we recommend to useminicondato configure Python virtual environment.installNode.js: you can download and install Node.js fromNode.js official site.Quick Installleaf-playgroundhas already been upload to pypi, thus you can usepipto quickly install:pipinstallleaf-playgroundIf you want to save data in PostgreSQL instead of SQLite, you need to include thepostgresqlextra dependency:pipinstallleaf-playground[postgresql]If you are a framework or scenario simulation project developer who want to debug the code, you need to include thedebugextra dependency:pipinstallleaf-playground[debug]Install from sourceTo installleaf-playgroundfrom the source, you need to clone the project by usinggit clone, then in your localleaf-playgrounddirectory, run:pipinstall.UsageStart Server and Create a TaskTo start the server that contains projects hosted inleaf-playground-hub, you need to first clone this project, then in the directory of your localleaf-playground-hub, using CLI command to start server with webui:leaf-outstart-server[--portPORT][--ui_portUI_PORT]By default, the backend service will run on port 8000, the UI service will run on port 3000, you can use--portand--ui_portoptions to use different ports respectively.Below is a video demonstrates how to create and run a task that using MMLU dataset to evaluate LLM-based agents.https://github.com/LLM-Evaluation-s-Always-Fatiguing/leaf-playground/assets/754493/0c980a97-1b7f-4884-bd85-fbdc60121ac8Maintainers@PanQiWei;@Pandazki.RoadmapThe Frameworksupport human participant in the scenario simulation as a dynamic agentrunning each scenario simulation task in a docker containersupport manage task status(pause, restart, interrupt, etc.)support full task data persistencesave task info, logs and message in databasesave task results in database or remote file systemsupport for resuming runtime state and information from checkpoint and continuing executionsupport complete project automaticallycomplete scene definition automaticallycomplete agents automaticallycomplete agent base classes automaticallycomplete specific agent class automaticallycomplete evaluator automaticallycomplete scene automaticallyrefactorai_backendtollm_backend_toolsto remove some heavy dependenciessupport streaming agents' responsesThe Huboptimize scene flow ofwho_is_the_spyproject and add metrics and evaluatorscreate a new project to support using OpenAIevalscreate a new project to support using Microsoftpromptbench