Package: bananalyzer
🍌 Open source AI Agent evaluations for web tasks 🍌🔗 Main site•🐦 Twitter•📢 DiscordBanana-lyzerIntroductionBanana-lyzer is an open source AI Agent evaluation framework and dataset forweb taskswith Playwright (And has a
banana theme because why not).
We've created our own evals repo because:Websites change overtime, are affected by latency, and may have anti bot protections.We need a system that can reliably save and deploy historic/static snapshots of websites.Standard web practices are loose and there is an abundance of different underlying ways to represent a single
individual website. For an agent to best generalize, we require building a diverse dataset of websites across
industries and use-cases.We have specific evaluation criteria and agent use cases focusing on structured and direct information retrieval
across websites.There exists valuable web task datasets and evaluations that we'd like to unify in a single
repo (Mind2Web,WebArena, etc).https://github.com/reworkd/bananalyzer/assets/50181239/4587615c-a5b4-472d-bca9-334594130af1How does it work?⚠️ Note that this repo is a work in progress. ⚠️Banana-lyzer is a CLI tool that runs a set of evaluations against a set of example websites.
The examples are defined
inexamples.jsonusing a schema
similar toMind2WebandWebArena. The examples
store metadata like the agent goal and the expected agent output in addition to snapshots of urls via mhtml to ensure
the page is not changed over time. Note all examples today expect structured JSON output using data directly extracted
from the page.The CLI tool will sequentially run examples against a user defined agent by dynamically constructing a pytest test suite
and executing it.
As a user, you simply create a file that implements theAgentRunnerinterface and defines an instance of your
AgentRunner in a variable called "agent".
AgentRunner exposes the example, and a playwright browser context to use.In the future we will support more complex evaluation methods and examples that require multiple steps to complete. The
plan is to translate existing datasets like Mind2Web and WebArena into this format.Test intentsWe have defined a set of test intents that an agent can be evaluated on. These intents are defined in theGoalTypeenum inexamples.json.fetch: The agent must retrieve specific JSON information from the page. This is the most common test type.links: The agent must scrape all detail page links from a pagelinks_fetch: The agent must scrape all detail page links from a page and additionally extract JSON information
for each linkpagination: Must fetch data across pages. Either links or fetch for now.Getting StartedLocal testing installationpip install --dev bananalyzerImplement theagent_runner.pyinterface and make a banalyzer.py test file (The name doesn't matter). Below is an
example fileimportasynciofromplaywright.async_apiimportBrowserContextfrombananalyzer.data.schemasimportExamplefrombananalyzer.runner.agent_runnerimportAgentResult,AgentRunnerclassNullAgentRunner(AgentRunner):"""A test agent class that just returns an empty string"""asyncdefrun(self,context:BrowserContext,example:Example,)->AgentResult:page=awaitcontext.new_page()awaitpage.goto(example.get_static_url())# example.url has the real url, example.get_static_url() returns the local mhtml file urlawaitasyncio.sleep(0.5)returnexample.evals[0].expected# Just return expected output directly so that tests passRunbananalyze ./tests/banalyzer.pyto run the test suiteYou can also runbananalyze .to run all tests in the current directoryArguments-hor--help: Show help--headless: Run with Playwright headless mode-idor--id: Run a specific test by id-ior--intent: Only run tests of a particular intent (fetch, links, etc)-cor--category: Only run tests of a particular category (healthcare, manufacturing, software, etc)-nor--n: Number of test workers to use. The default is 1-skipor--skip: A list of ids to skip tests on, separated by commas-tor--type: Only run tests of a particular type (links, fetch, etc)ContributingRunning the serverThe project has a basic FastAPI server to expose example data. You can run it with the following command:cd server
poetry run uvicorn server:app --reloadThen travel tohttp://127.0.0.1:8000/api/docsin your browser to see the API docs.Adding examplesAll current examples have been manually added through running thefetch.ipynbnotebook at the root of this project.
This notebook will load a site with Playwright and use the chrome developer API to save the page as an MHTML file.RoadmapLaunchFunctions to serve local MHTML sitesAgent interface required for running the toolPytest wrapper to enable CLI testing with additional argumentsDocument a majority of the repoFeaturesCLI param to filter tests by intentAdditional CLI params to select for specific tests or test categoriesAbility to add multiple site pages to examplesAbility to add in-page actions to examplesTranslate WebArena evalsTranslate Mind2Web evalsLag and bot detection emulationUpdated test visualization with separation of categories and outputsDataset updates15 additional data retrieval examples15 additional link examples15 click examples15 navigation examplesTests requiring multi-step navigationTests requiring both navigation and data retrievalTests requiring pop-up closingTests requiring sign-inTests requiring captcha solvingCitationsbibtex
@misc{reworkd2023bananalyzer,
  title        = {Bananalyzer},
  author       = {Asim Shrestha and Adam Watkins and Rohan Pandey and Srijan Subedi},
  year         = {2023},
  howpublished = {GitHub},
  url          = {https://github.com/reworkd/bananalyzer}
}