Package: self-operating-computer
Self-Operating Computer FrameworkA framework to enable multimodal models to operate a computer.Using the same inputs and outputs as a human operator, the model views the screen and decides on a series of mouse and keyboard actions to reach an objective.Key FeaturesCompatibility: Designed for various multimodal models.Integration: Currently integrated withGPT-4v, Gemini Pro Vision, and LLaVa.Future Plans: Support for additional models.Ongoing DevelopmentAtHyperwriteAI, we are developing Agent-1-Vision a multimodal model with more accurate click location predictions.Agent-1-Vision Model API AccessWe will soon be offering API access to our Agent-1-Vision model.If you're interested in gaining access to this API, sign uphere.Demohttps://github.com/OthersideAI/self-operating-computer/assets/42594239/9e8abc96-c76a-46fb-9b13-03678b3c67e0RunSelf-Operating ComputerInstall the projectpip install self-operating-computerRun the projectoperateEnter your OpenAI Key: If you don't have one, you can obtain an OpenAI keyhereGive Terminal app the required permissions: As a last step, the Terminal app will ask for permission for "Screen Recording" and "Accessibility" in the "Security & Privacy" page of Mac's "System Preferences".Alternatively installation with.shClone the repoto a directory on your computer:git clone https://github.com/OthersideAI/self-operating-computer.gitCd into directory:cd self-operating-computerRun the installation script:./run.shUsingoperateModesMultimodal Models-mAn additional model is now compatible with the Self Operating Computer Framework. Try Google'sgemini-pro-visionby following the instructions below.Startoperatewith the Gemini modeloperate -m gemini-pro-visionEnter your Google AI Studio API key when terminal prompts you for itIf you don't have one, you can obtain a keyhereafter setting up your Google AI Studio account. You may also needauthorize credentials for a desktop application. It took me a bit of time to get it working, if anyone knows a simpler way, please make a PR.Locally Hosted LLaVA Through OllamaIf you wish to experiment with the Self-Operating Computer Framework using LLaVA on your own machine, you can with Ollama!Note: Ollama currently only supports MacOS and LinuxFirst, install Ollama on your machine fromhttps://ollama.ai/download.Once Ollama is installed, pull the LLaVA model:ollama pull llavaThis will download the model on your machine which takes approximately 5 GB of storage.When Ollama has finished pulling LLaVA, start the server:ollama serveThat's it! Now startoperateand select the LLaVA model:operate -m llavaImportant:Error rates when using LLaVA are very high. This is simply intended to be a base to build off of as local multimodal models improve over time.Learn more about Ollama at itsGitHub RepositoryVoice Mode--voiceThe framework supports voice inputs for the objective. Try voice by following the instructions below.Clone the repoto a directory on your computer:git clone https://github.com/OthersideAI/self-operating-computer.gitCd into directory:cd self-operating-computerInstall the additionalrequirements-audio.txtpip install -r requirements-audio.txtInstall device requirementsFor mac users:brew install portaudioFor Linux users:sudo apt install portaudio19-dev python3-pyaudioRun with voice modeoperate --voiceOptical Character Recognition Mode-m gpt-4-with-ocrThe Self-Operating Computer Framework now integrates Optical Character Recognition (OCR) capabilities with thegpt-4-with-ocrmode. This mode gives GPT-4 a hash map of clickable elements by coordinates. GPT-4 can decide toclickelements by text and then the code references the hash map to get the coordinates for that element GPT-4 wanted to click.Based on recent tests, OCR performs better thansomand vanilla GPT-4 so we made it the default for the project. To use the OCR mode you can simply write:operateoroperate -m gpt-4-with-ocrwill also work.Set-of-Mark Prompting-m gpt-4-with-somThe Self-Operating Computer Framework now supports Set-of-Mark (SoM) Prompting with thegpt-4-with-somcommand. This new visual prompting method enhances the visual grounding capabilities of large multimodal models.Learn more about SoM Prompting in the detailed arXiv paper:here.For this initial version, a simple YOLOv8 model is trained for button detection, and thebest.ptfile is included undermodel/weights/. Users are encouraged to swap in theirbest.ptfile to evaluate performance improvements. If your model outperforms the existing one, please contribute by creating a pull request (PR).Startoperatewith the SoM modeloperate -m gpt-4-with-somContributions are Welcomed!:If you want to contribute yourself, seeCONTRIBUTING.md.FeedbackFor any input on improving this project, feel free to reach out toJoshon Twitter.Join Our Discord CommunityFor real-time discussions and community support, join our Discord server.If you're already a member, join the discussion in#self-operating-computer.If you're new, firstjoin our Discord Serverand then navigate to the#self-operating-computer.Follow HyperWriteAI for More UpdatesStay updated with the latest developments:Follow HyperWriteAI onTwitter.Follow HyperWriteAI onLinkedIn.CompatibilityThis project is compatible with Mac OS, Windows, and Linux (with X server installed).OpenAI Rate Limiting NoteThegpt-4-vision-previewmodel is required. To unlock access to this model, your account needs to spend at least $5 in API credits. Pre-paying for these credits will unlock access if you haven't already spent the minimum $5.Learn morehere