Package: tarsier
🙈 Vision utilities for web interaction agents 🙈🔗 Main site•🐦 Twitter•📢 DiscordTarsierIf you've tried using GPT-4(V) to automate web interactions, you've probably run into questions like:How do you map LLM responses back into web elements?How can you mark up a page for an LLM better understand its action space?How do you feed a "screenshot" to a text-only LLM?At Reworkd, we found ourselves reusing the same utility libraries to solve these problems across multiple projects.
Because of this we're now open-sourcing this simple utility library for multimodal web agents... Tarsier!
The video below demonstrates Tarsier usage by feeding a page snapshot into a langchain agent and letting it take actions.https://github.com/reworkd/tarsier/assets/50181239/af12beda-89b5-4add-b888-d780b353304bHow does it work?Tarsier works by visually "tagging" interactable elements on a page via brackets + an id such as[1].
In doing this, we provide a mapping between elements and ids for GPT-4(V) to take actions upon.
We define interactable elements as buttons, links, or input fields that are visible on the page.Can provide a textual representation of the page. This means that Tarsier enables deeper interaction for even non multi-modal LLMs.
This is important to note given performance issues with existing vision language models.
Tarsier also provides OCR utils to convert a page screenshot into a whitespace-structured string that an LLM without vision can understand.InstallationpipinstalltarsierUsageVisit ourcookbookfor agent examples using Tarsier:An autonomous LangChain web agent🦜⛓️An autonomous LlamaIndex web agent🦙Otherwise, basic Tarsier usage might look like the following:importasynciofromplaywright.async_apiimportasync_playwrightfromtarsierimportTarsier,GoogleVisionOCRServiceimportjsondefload_google_cloud_credentials(json_file_path):withopen(json_file_path)asf:credentials=json.load(f)returncredentialsasyncdefmain():# To create the service account key, follow the instructions on this SO answer https://stackoverflow.com/a/46290808/1780891google_cloud_credentials=load_google_cloud_credentials('./google_service_acc_key.json')ocr_service=GoogleVisionOCRService(google_cloud_credentials)tarsier=Tarsier(ocr_service)asyncwithasync_playwright()asp:browser=awaitp.chromium.launch(headless=False)page=awaitbrowser.new_page()awaitpage.goto("https://news.ycombinator.com")page_text,tag_to_xpath=awaittarsier.page_to_text(page)print(tag_to_xpath)# Mapping of tags to x_pathsprint(page_text)# My Text representation of the pageif__name__=='__main__':asyncio.run(main())Keep in mind that Tarsier tags different types of elements differently to help your LLM identify what actions are performable on each element. Specifically:[#ID]: text-insertable fields (e.g.textarea,inputwith textual type)[@ID]: hyperlinks (<a>tags)[$ID]: other interactable elements (e.g.button,select)[ID]: plain text (if you passtag_text_elements=True)Local DevelopmentSetupWe have provided a handy setup script to get you up and running with Tarsier development../script/setup.shIf you modify any TypeScript files used by Tarsier, you'll need to execute the following command.
This compiles the TypeScript into JavaScript, which can then be utilized in the Python package.npmrunbuildTestingWe usepytestfor testing. To run the tests, simply run:poetryrunpytest.LintingPrior to submitting a potential PR, please run the following to format your code:./script/format.shSupported OCR ServicesGoogle Cloud VisionAmazon Textract(Coming Soon)Microsoft Azure Computer Vision(Coming Soon)RoadmapAdd documentation and examplesClean up interfaces and add unit testsLaunchImprove OCR text performanceAdd options to customize tagging stylingAdd support for other browsers drivers as necessaryAdd support for other OCR services as necessaryCitationsbibtex
@misc{reworkd2023tarsier,
  title        = {Tarsier},
  author       = {Rohan Pandey and Adam Watkins and Asim Shrestha and Srijan Subedi},
  year         = {2023},
  howpublished = {GitHub},
  url          = {https://github.com/reworkd/tarsier}
}