Package: mobile-env-rl
NEWS!!(2023-12-18 v3.5)Owing to the long time delay of VH check and screenshot check, we updated
the mechanism of managing the check time. By this way, the requirement of
sufficient check for the episode events and the resulted long delay can be
balanced.Added multiple rating methods toResponseEvent: regex matching, fuzzy
matching, and vector encoding matching.ImprovedVhIoWrapperandTapActionWrapper. Added support toSCROLLandTYPEtoTapActionWrapper.OptimizedRemoteSimulator. In order to reduce the delay of network
transfering, enabled action batch to send and execute a group of actions
and enabled resizing the image before and after transferring to shrink the
transferred data.Merged annotation tool to the main branch. The original annotation-tool
branch is deprecated.Added support toResponseEventto annotation tool.Supplemented several commandline options to annotation-tool.(2023-10-31 v3.0) Migrated VH node specification from the original VH path to
Mobile-Env-customized CSS selector (me-selector) and added repeatability
control to EventSlots. Repeatability control for EventSlots may be useful to
prevent repetitive triggering of anOR-type virtual event combining
multiple types of event sources.Please see ourChange LogandDocument.(2023-09-21 v2.1) AddedREMOTE SIMULATORto solve the problem that
hardware-based acceleration for virtualization is not enabled on many GPU
clustersPlease see ourChange LogandDocument.(2023-06-30 v2.0) New type of event "response to human user" (RHU,ResponseEvent). Now enables the agent to generate response to human user
and parses episode signales from it. This will enable interaction tasks like
question-answering, retrieval,etc.Please see ourChange Log,Usage Document,
andTask Definition Document.Mobile-Env: An Evaluation Platform and Benchmark for Interactive Agents in LLM EraMobile-Env is a interaction platform for evaluation and training of the
interaction of information user interface (InfoUI). Our paper is available atarXiv.Mobile-Env is developed based onAndroidEnv. The agent can take the
screenshot and the view hierarchy (disabled defaultly for the long latency) as
the observation and take a touch or type a token as the action to interact with
the Android apps. Several episode signals like step instructions, rewards, or
the episode end will be informed during interaction at some crucial steps. A
so-called crucial step may be opening a target page, srolling to a correct
area,etc. and is depending on the specifictask
definition.The proposed WikiHow task set is available at theHugging Face
Platform.Documents in Chinese (中文文档)IndexEvaluating and Training Agents on Mobile-EnvExtending a New Environment (App) or a New Task Based on
Mobile-EnvCertificate Pinning Problem & SolutionsMiscellaneous Auxiliary ToolsPlatform FeaturesMobile-Env is a flexible, adaptable, and easily-extendable platform for InfoUI
interaction with the following features:Both screenshot and view hierarchy are provided as the observation. The touch
and token typing are provided as the action. Wrappers are also supported to
customize the observation and action spaces. Thus, both visual-based and
text-based agents, both agents with continuous action space and discrete
action space, can be evaluated on Mobile-Env.New tasks can be easily extended through task definition files.Multiple sources are enabled to parse the task events from the operating
system: screen text, screen icon, view hierarchy, and the system log, which
makes Mobile-Env capable of adapting to most real-world apps without
dedicated development. (Screen text and screen icon will be enabled with an
external OCR tool and icon recognition tool. Currently, a wrapper ofEasyOCRis integrated in the platform
and can be enabled directly. An intergrated icon model will be embedded soon
as well.)Getting StartedInstallationInstall from PyPI:pipinsallmobile-env-rlor clone the repository and build locally.gitclonehttps://github.com/X-LANCE/Mobile-EnvcdMobile-Env
pipinstall.SeveralDocker imageswith
well-configured Android AVD are also available.Load and Run Mobile-Env for Evaluation or TrainingBefore loading the Mobile-Env environment, you will need to set up anAndroid
Emulatordevice. Then you can load the
environment with some existing task definitions and start your experiments. A
detailed guidance is provided inEvaluating and Traning Agents on
Mobile-Env. Several examples with a random agent or a
human agent is also provided underexamples.Extend a New Environment or a New TaskTo extend a new environment for Mobile-Env, the environment designer needs to
prepare the app package and ensure that the package manages to launch and run
on some versions of Android Emulator. If the app requires varying online data,
the necessary data should be crawled and dumped and then be replayed for a
consistent evaluation. In such case, the designer is supposed to validate the
certain effectiveness ofcertificate unpinning planfor the package.  As regards to extend new tasks, task definition files are
just required. Detailed instructions can be found inExtending a New
Environment (App) or a New Task Based on
Mobile-Env.Several demo task definitions are provided underdemos. Three of them are
migrated from  AndroidEnv:classic_2048.m.textproto- Classic 2048 game.accessibility_forwarder_clock_set_timer.m.textproto- A simple task
requiring the agent to reset a running timer.systemui_egg_land_default.m.textproto- Flappy Droid. An open-sourced
implementation of classic game, Flappy Bird.Another one,openmoneybox.add_billings.textprotois defined upon an
open-sourced billing app,OpenMoneyBox.
Details are referred to in the task definition files.Miscellaneous Auxiliary ToolsWe also developed an annotation tool for the human demonstrations, and a suite
of template tool to auto-generate task definitions according to templates and
to combine multiple task definitions to form a multi-step task. The details are
referred to inMiscellaneous Auxiliary Tools.Reference Time-Consuming and Memory Usage o Mobile-EnvThe data are measured under the configuration below:OS and hardware:Operating System: Manjaro 23.1.0 VulcanKernel Version: x86_64 Linux 6.1.64-1-MANJAROCPU: Intel Core i7-10700 @ 16x 4.8GHzGPU: NVIDIA GeForce RTX 3090RAM: 64 GBKVM acceleration enabledAndroid development toolsAndroid emulator version 32.1.14.0Android platform tools 34.0.4libvert 1:9.9.0Python & packagesPython 3.8.16EasyOCR 1.7.2sentence-transformers 2.2.2Android Virtual DeviceDevice type: Pixel 2API version: API 30OS Variant: Google APIsCPU cores: 4Memory: 8 GBScreen size: 1080×1920ItemAvg TimeTime Std DevTOUCHaction410.50µs64.71µsLIFTaction412.30µs84.18µsTEXTaction1.30s0.28sscreenshot capturing19.94ms21.47msinvocation of Sentence Transformer（all-MiniLM-L12-v2）8.51ms0.17msVH capturing2.53s1.90sinvocation of EasyOCR0.44s0.08sWhen only anapp of WikiHow
2.9.6is running, the Android emulator occupies 6,031 MiB of virtual memory and 3,444
MiB of residual memory.AboutThis library is developed and maintained bySJTU
X-Lance. The corresponding paper is available
athttps://arxiv.org/abs/2305.08144.If you find Mobile-Env useful in your research, you can cite the project using
the following BibTeX:@article{DanyangZhang2023_MobileEnv,title={{Mobile-Env}: An Evaluation Platform and Benchmark for Interactive Agents in LLM Era},author={Danyang Zhang andLu Chen andZihan Zhao andRuisheng Cao andKai Yu},journal={CoRR},volume={abs/2305.08144},year={2023},url={https://arxiv.org/abs/2305.08144},eprinttype={arXiv},eprint={2305.08144},}