Package: openadapt
Read our latest slidesJoin us on DiscordRead our Architecture documentJoin the Discussion on the Request for CommentsSee also:https://github.com/OpenAdaptAI/SoMhttps://github.com/OpenAdaptAI/pynputhttps://github.com/OpenAdaptAI/atomacosOpenAdapt: AI-First Process Automation with Large Multimodal Models (LMMs).OpenAdaptis theopensource softwareadapter between Large Multimodal Models (LMMs) and traditional desktop and web Graphical User Interfaces (GUIs).Enormous volumes of mental labor are wasted on repetitive GUI workflows.Foundation Models (e.g.GPT-4,ACT-1) are powerful automation tools.OpenAdapt connects Foundation Models to GUIs:System Overview:Early demo:https://www.loom.com/share/9d77eb7028f34f7f87c6661fb758d1c0(more coming soon!)Welcome to OpenAdapt! This Python library implements AI-First Process Automation
with the power of Large Multimodal Modals (LMMs) by:Recording screenshots and associated user inputAggregating and visualizing user input and recordings for developmentConverting screenshots and user input into tokenized formatGenerating synthetic input via transformer model completionsGenerating task trees by analyzing recordings (work-in-progress)Replaying synthetic input to complete tasks (work-in-progress)The goal is similar to that ofRobotic Process Automation,
except that we use Large Multimodal Models instead of conventional RPA tools.The direction is adjacent toAdept.ai, with some key differences:OpenAdapt is model agnostic.OpenAdapt generates prompts automatically bylearning from human demonstration(auto-prompted, not user-prompted). This means that agents aregroundedinexisting processes, which mitigates hallucinations and ensures successful task completion.OpenAdapt works with all types of desktop GUIs, including virtualized (e.g. Citrix) and web.OpenAdapt is open source (MIT license).InstallInstallation MethodRecommended forEase of UseScriptedNon-technical usersStreamlines the installation process for users unfamiliar with setup stepsManualTechnical UsersAllows for more control and customization during the installation processInstallation ScriptsWindowsPress Windows Key, type "powershell", and press EnterCopy and paste the following command into the terminal, and press Enter (If Prompted forUser Account Control, click 'Yes'):Start-Process powershell -Verb RunAs -ArgumentList '-NoExit', '-ExecutionPolicy', 'Bypass', '-Command', "iwr -UseBasicParsing -Uri 'https://raw.githubusercontent.com/OpenAdaptAI/OpenAdapt/main/install/install_openadapt.ps1' | Invoke-Expression"MacOSDownload and install Git and Python 3.10Press Command+Space, type "terminal", and press EnterCopy and paste the following command into the terminal, and press Enter:/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/OpenAdaptAI/OpenAdapt/HEAD/install/install_openadapt.sh)"Manual SetupPrerequisite:Python 3.10GitTesseract (for OCR)For the setup of any/all of the above dependencies, follow the stepsSETUP.md.Install withPoetry:git clone https://github.com/OpenAdaptAI/OpenAdapt.git
cd OpenAdapt
pip3 install poetry
poetry install
poetry shell
alembic upgrade head
pytestPermissionsSee how to set up system permissions on macOShere.UsageRecordCreate a new recording by running the following command:python -m openadapt.record "testing out openadapt"Wait until all three event writers have started:| INFO     | __mp_main__:write_events:230 - event_type='screen' starting
| INFO     | __mp_main__:write_events:230 - event_type='action' starting
| INFO     | __mp_main__:write_events:230 - event_type='window' startingType a few words into the terminal and move your mouse around the screen
to generate some events, then stop the recording by pressing CTRL+C.Current limitations:recording should be short (i.e. under a minute), as they are
somewhat memory intensive, and there is currently anopen issuedescribing a
possible memory leakthe only touchpad and trackpad gestures currently supported are
pointing the cursor and left or right clicking, as described in thisopen issueVisualizeVisualize the latest recording you created by running the following command:python -m openadapt.visualizeThis will open a scrollable window that looks something like this:For a browser-based visualization, run:python -m openadapt.deprecated.visualizeThis will open up a tab in your browser that looks something like this:PlaybackYou can play back the recording using the following command:python -m openadapt.replay NaiveReplayStrategyOther replay strategies include:StatefulReplayStrategy: Proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data.Seehttps://github.com/OpenAdaptAI/OpenAdapt/tree/main/openadapt/strategiesfor a complete list. More ReplayStrategies coming soon! (seeContributing).FeaturesState-of-the-art GUI understanding viaSegment Anything in High Quality:Industry leading privacy (PII/PHI scrubbing) viaAWS Comprehend,Microsoft PresidioandPrivate AI:Decentralized and secure data distribution viaMagic Wormhole:Detailed performance monitoring viapymplerandtracemalloc:And much more!🚀 Open Contract Positions at OpenAdapt.AIWe are thrilled to open new contract positions for developers passionate about pushing boundaries in technology. If you're ready to make a significant impact, consider the following roles:Frontend DeveloperResponsibilities: Develop and test key features such as process visualization, demo booking, app store, and blog integration.Skills: Proficiency in modern frontend technologies and a knack for UI/UX design.Machine Learning EngineerRole: Implement and refine process replay strategies using state-of-the-art LLMs/LMMs. Extract dynamic process descriptions from extensive process recordings.Skills: Strong background in machine learning, experience with LLMs/LMMs, and problem-solving aptitude.Software EngineerFocus: Enhance memory optimization techniques during process recording and replay. Develop sophisticated tools for process observation and productivity measurement.Skills: Expertise in software optimization, memory management, and analytics.Technical WriterFocus: MaintainingOpenAdaptrepositoriesSkills: Passion for writing and/or documentation🔍 How to ApplyStep 1: Submit an empty Pull Request toOpenAdaptorOpenAdapt.web. Format your PR title as[Proposal] <your title here>Step 2: Include a brief, informal outline of your approach in the PR description. Feel free to add any questions you might have.Need Clarifications?Reach out to us onDiscord.We're looking forward to your contributions. Let's build the future 🚀ContributingProblem StatementOur goal is to automate the task described and demonstrated in aRecording.
That is, given a newScreenshot, we want to generate the appropriateActionEvent(s) based on the previously recordedActionEvents in order to
accomplish the task specified in theRecording.task_descriptionand narrated by the user inAudioInfo.words_with_timestamps,
while accounting for differences in screen resolution, window size, application
behavior, etc.If it's not clear whatActionEventis appropriate for the givenScreenshot,
(e.g. if the GUI application is behaving in a way we haven't seen before),
we can ask the user to take over temporarily to demonstrate the appropriate
course of action.Data ModelThe data model consists of the following entities:Recording: Contains information about the screen dimensions, platform, and
other metadata.ActionEvent: Represents a user action event such as a mouse click or key
press. EachActionEventhas an associatedScreenshottaken immediately
before the event occurred.ActionEvents are aggregated to remove
unnecessary events (seevisualize.)Screenshot: Contains the PNG data of a screenshot taken during the
recording.WindowEvent: Represents a window event such as a change in window title,
position, or size.APIYou can assume that you have access to the following functions:create_recording("doing taxes"): Creates a recording.get_latest_recording(): Gets the latest recording.get_events(recording): Returns a list ofActionEventobjects for the given
recording.SeeGitBook Documentationfor more.InstructionsJoin us on Discord. Then:Fork this repository and clone it to your local machine.Get OpenAdapt up and running by following the instructions underSetup.Look through the list of open issues athttps://github.com/OpenAdaptAI/OpenAdapt/issuesand once you find one you would like to address, indicate your interest with a comment.Implement a solution to the issue you selected. Write unit tests for your
implementation.Submit a Pull Request (PR) to this repository. Note: submitting a PR before your
implementation is complete (e.g. with high level documentation and/or implementation
stubs) is encouraged, as it provides us with the opportunity to provide early
feedback and iterate on the approach.Evaluation CriteriaYour submission will be evaluated based on the following criteria:Functionality: Your implementation should correctly generate the newActionEventobjects that can be replayed in order to accomplish the task in
the original recording.Code Quality: Your code should be well-structured, clean, and easy to
understand.Scalability: Your solution should be efficient and scale well with
large datasets.Testing: Your tests should cover various edge cases and scenarios to
ensure the correctness of your implementation.SubmissionCommit your changes to your forked repository.Create a pull request to the original repository with your changes.In your pull request, include a brief summary of your approach, any
assumptions you made, and how you integrated external libraries.Bonus: interacting with ChatGPT and/or other language transformer models
in order to generate code and/or evaluate design decisions is encouraged. If
you choose to do so, please include the full transcript.TroubleshootingMacOS: if you encounter system alert messages or find issues when making and replaying recordings, make sure toset up permissions accordingly.In summary (fromhttps://stackoverflow.com/a/69673312):Settings -> Security & PrivacyClick on the Privacy tabScroll and click on the Accessibility RowClick +Navigate to /System/Applications/Utilities/ (or wherever Terminal.app is installed)Click okay.DevelopingGenerate migration (after editing a model)alembic revision --autogenerate -m "<msg>"Pre-commit HooksTo ensure code quality and consistency, OpenAdapt uses pre-commit hooks. These hooks
will be executed automatically before each commit to perform various checks and
validations on your codebase.The following pre-commit hooks are used in OpenAdapt:check-yaml: Validates the syntax and structure of YAML files.end-of-file-fixer: Ensures that files end with a newline character.trailing-whitespace: Detects and removes trailing whitespace at the end of lines.black: Formats Python code to adhere to the Black code style. Notably, the--previewfeature is used.isort: Sorts Python import statements in a consistent and standardized manner.To set up the pre-commit hooks, follow these steps:Navigate to the root directory of your OpenAdapt repository.Run the following command to install the hooks:pre-commit installNow, the pre-commit hooks are installed and will run automatically before each commit. They will enforce code quality standards and prevent committing code that doesn't pass the defined checks.Status ChecksWhen you submit a PR, the "Python CI" workflow is triggered for code consistency. It follows organized steps to review your code:Python Black Check: This step verifies code formatting using Python Black style, with the--previewflag for style.Flake8 Review: Next, Flake8 tool thoroughly checks code structure, including flake8-annotations and flake8-docstrings. Though GitHub Actions automates checks, it's wise to locally runflake8 .before finalizing changes for quicker issue spotting and resolution.Submitting an IssuePlease submit any issues tohttps://github.com/OpenAdaptAI/OpenAdapt/issueswith the
following information:Problem description (please include any relevant console output and/or screenshots)Steps to reproduce (please help others to help you!)