Package: browserpilot
🛫 BrowserPilotAn intelligent web browsing agent controlled by natural language.Language is the most natural interface through which humans give and receive instructions. Instead of writing bespoke automation or scraping code which is brittle to changes, creating and adding agents should be as simple as writing plain English.🏗️ Installationpip install browserpilotDownload Chromedriver (latest stable release) fromhereand place it in the same folder as this file. Unzip. In Finder, right click the unpacked chromedriver and click "Open". This will remove the restrictive default permissions and allow Python to access it.Create an environment variable in your favorite manner setting OPENAI_API_KEY to your API key.🦭 Usage🗺️ APIThe form factor is fairly simple (see below).frombrowserpilot.agents.gpt_selenium_agentimportGPTSeleniumAgentinstructions="""Go to Google.comFind all textareas.Find the first visible textarea.Click on the first visible textarea.Type in "buffalo buffalo buffalo buffalo buffalo" and press enter.Wait 2 seconds.Find all anchor elements that link to Wikipedia.Click on the first one.Wait for 10 seconds."""agent=GPTSeleniumAgent(instructions,"/path/to/chromedriver")agent.run()The harder (but funner) part is writing the natural language prompts.📑 Writing PromptsIt helps if you are familiar with how Selenium works and programming in general. This is because this project uses GPT-3 to translate natural language into code, so you should be as precise as you can. In this way, it is more like writing code with Copilot than it is talking to a friend; for instance, it helps to refer to things asinputs ortextareas(vs. "text box" "search box") or "button which says 'Log in'" rather than "the login button". Sometimes, it will also not pick up on specific words that are important, so it helps to break them out into separate lines. Instead of "find all the visible textareas", you do "find all the textareas" and then "find the first visible textarea".You can look at some examples inprompts/examplesto get started.Create "functions" by enclosing instructions inBEGIN_FUNCTION func_nameandEND_FUNCTION, and then call them by starting a line withRUN_FUNCTIONorINJECT_FUNCTION. Below is an example:BEGIN_FUNCTION search_buffalo
Go to Google.com
Find all textareas.
Find the first visible textarea.
Click on the first visible textarea.
Type in "buffalo buffalo buffalo buffalo buffalo" and press enter.
Wait 2 seconds.
Get all anchors on the page that contain the word "buffalo".
Click on the first link.
END_FUNCTION

RUN_FUNCTION search_buffalo
Wait for 10 seconds.You may also choose to create a yaml or json file with a list of instructions. In general, it needs to have aninstructionsfield, and optionally acompiledfield which has the processed code.Seebuffalo wikipedia example.You may pass ainstruction_output_fileto the constructor of GPTSeleniumAgent which will output a yaml file with the compiled instructions from GPT-3, to avoid having to pay API costs.✋🏼 ContributingThere are two ways I envision folks contributing.Adding to the Prompt Library: Read "Writing Prompts" above and simply make a pull request to add something toprompts/! At some point, I will figure out a protocol for folder naming conventions and the evaluation of submitted code (for security, accuracy, etc). This would be a particularly attractive option for those who aren't as familiar with coding.Contributing code: I am happy to take suggestions! The main way to add to the repository is to extend the capabilities of the agent, or to create new agents entirely. The best way to do this is to familiarize yourself with "Architecture and Prompt Patterns" above, and to (a) expand the list of capabilities in the base prompt inInstructionCompilerand (b) write the corresponding method inGPTSeleniumAgent.⛩️ Architecture and Prompt PatternsThis repo was inspired by the work ofYihui He,Adept.ai, andNat Friedman. In particular, the basic abstractions and prompts used were built off of Yihui's hackathon code. The idea to preprocess HTML and use GPT-3 to intelligently pick elements out is from Nat.The prompts used can be found ininstruction compiler. The base prompt describes in plain English a set of actions that the browsing agent can take, some general conventions on how to write code, and some constraints on its behavior.These actions correspond one-for-one with methods inGPTSeleniumAgent. Those actions, to-date, include:env.driver, the Selenium webdriver.env.find_elements(by='id', value=None)finds and returns list of elements.env.find_element(by='id', value=None)is similar toenv.find_elements()except it only returns the first element.env.find_nearest(e, xpath)can be used to locate an element near another one.env.send_keys(element, text)sendstextto element.env.get(url)goes to url.env.click(element)clicks the element.env.wait(seconds)waits forsecondsseconds.env.scroll(direction, iframe=None)scrolls the page. Will switch toiframeif given.directioncan be "up", "down", "left", or "right".env.get_llm_response(text)asks AI about a stringtext.env.retrieve_information(prompt)returns a string, information from a page given a prompt. Use prompt="Summarize:" for summaries. Invoked with commands like "retrieve", "find in the page", or similar.env.ask_llm_to_find_element(description)asks AI to find an element that matches the description.env.query_memory(prompt)asks AI with a prompt to query its memory (an embeddings index) of the web pages it has browsed. Invoked with "Query memory".env.save(text, filename)saves the stringtextto a filefilename.env.get_text_from_page()returns the free text from the page.The rest of the code is basically middleware which exposes a Selenium object to GPT-3.For each action mentioned in the base prompt, there is a corresponding method in GPTSeleniumAgent.AnInstructionCompileris used to parse user input into semantically cogent blocks of actions.The agent has aMemorywhich enables it to synthesize what it sees.🎉 Finished0.2.42 - 0.2.44Small changes inexamples.pyand dependencies.Refactor for the big Llama Index upgrade.0.2.38 - 0.2.41Changeenable_memorytomemory_fileto enable more control over what the memory is called. Allow users to load memory as well.Makeget_text_from_pagesimpler.0.2.26 - 0.2.37Bit the bullet and switched the default model to gpt-3.5-turbo. Will be much cheaper!Also fixed retries. I wasn't actually getting the retry action!Fiddle with the prompt a bit for GPT-3.5.Concerningly, gpt-3.5-turbo keep trying to import modules. I manually remove lines that try to import modules.Compatibility with new Llama Index updates.0.2.14 - 0.2.25Add options to avoid website detection of bots.Add more OpenAI API error handling.Improve stack trace prompt and a few other prompts.Add "displayed in viewport" capability.Makefrom browserpilot.agents import <agent>possible.Makefind_elementandfind_elementssearch only for displayed elements.Save memory once finished running.Add scroll option to top and bottom.0.2.10 - 0.2.13Add more error handling for OpenAI exceptions.Change all the embedding querying to use ChatGPT.Get rid of the nltk dependency! Good riddance.Expand the max token window for asking the LLM a question on a web page.Fix an issue with the Memory module which tried to access OpenAI API key before it's initialized. Change the prompt slightly.❗️ Enable ChatGPT use with GPT Index, so that we can use GPT3.5-turbo to query embeddings.0.2.7 -  0.2.9Vacillating on the default model. ChatGPT does not work well for writing code, as it takes too many freedoms with what it returns.Also, I tried condensing the prompt a bit, which is growing a bit long.0.2.4 - 0.2.6Give the agent a memory (still very experimental and not very good). Add capability to screenshot elements.Bug fixes around versioning and prompt injection.0.2.3Movechrome_optionsto somewhere more sensible. Just keep the yaml clean, you know?0.2.2ChatGPT support.0.2.1Dict support for loading instructions.0.2.0🎬 aStudioCLI which helps iteratively test prompts!JSON loading.Basic iframe support.<0.2.0GPTSeleniumAgent should be able to load prompts and cached successful runs in the form of yaml files. InstructionCompiler should be able to save instructions to yaml.💭 Add a summarization capability to the agent.Demo/test something where it has to ask the LLM to synthesize something it reads online.🚨 Figured out how to feed the content of the HTML page into the GPT-3 context window and have it reliably pick out specific elements from it, that would be great!🚨 Disclaimer 🚨This package runs code output from the OpenAI API in Python usingexec. 🚨This is not considered a safe convention🚨. Accordingly, you should be extra careful when using this package. The standard disclaimer follows.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.