Skill: xjx0524/mobizen-gui (version 1.0.0)

--- name: mobizen-gui description: Helps users set up and run MobiZen-GUI to perform mobile-use tasks — automating Android phone operations via natural language. Use when the user wants to control a phone, execute mobile-use tasks, install/configure MobiZen-GUI, deploy MobiZen-GUI-4B model, or customize the agent's input/output. Triggers on keywords like phone task, mobile-use, mobile automation, Android control, MobiZen, ADB, deploy model, run agent, configure model. --- # MobiZen-GUI VLM-based mobile automation framework — control Android devices via natural language. Repo: https://github.com/alibaba/MobiZen-GUI --- ## 1. Environment Setup ### 1.1 Install ADB ```bash # macOS brew install android-platform-tools # Linux sudo apt-get install android-tools-adb # Windows: download from https://developer.android.com/studio/releases/platform-tools adb version # verify ``` ### 1.2 Connect Device & Install ADBKeyboard ```bash adb devices # USB; or: adb tcpip 5555 && adb connect <ip>:5555 adb install ADBKeyboard.apk # download from https://github.com/senzhk/ADBKeyBoard ``` Then on device: Settings → System → Languages & Input → Virtual Keyboard → Enable ADBKeyboard. ### 1.3 Install Project ```bash git clone https://github.com/alibaba/MobiZen-GUI.git && cd MobiZen-GUI pip install -r requirements.txt # openai, pillow, pyyaml ``` --- ## 2. Quick Start (Config Only, No Code Changes) Copy example config: ```bash cp config_example.yaml my_config.yaml ``` Only **3 fields** need to be configured — `api_key`, `base_url`, `model_name`: ```yaml [REDACTED_SECRET]" base_url: "https://api.openai.com/v1" # your model endpoint model_name: "gpt-4o" # model identifier ``` **How to set these 3 fields**: When the user asks to run a phone task but hasn't configured yet, the AI should **ask the user to provide** `api_key`, `base_url`, and `model_name`, then write them into `my_config.yaml`. The user can also manually edit the file. Any **OpenAI-compatible** API works. Provider examples: ```yaml # OpenAI base_url: "https://api.openai.com/v1" [REDACTED_SECRET]" model_name: "gpt-4o" # DeepSeek / Moonshot / Zhipu AI etc. base_url: "https://api.deepseek.com/v1" [REDACTED_SECRET]" model_name: "deepseek-chat" # Ollama (local) base_url: "http://localhost:11434/v1" api_key: "dummy" model_name: "llava" ``` Run: ```bash python main.py --config my_config.yaml --instruction "打开微信并发送消息" ``` --- ## 3. Configuration Reference | Field | Default | Description | |-------|---------|-------------| | `device_id` | `null` (auto) | ADB device; null = first available | | `api_key` | `""` | Model API key | | `base_url` | `null` | Model API endpoint | | `model_name` | `"gpt-4o"` | Model identifier | | `model_type` | `"qwen3vl"` | Coordinate system (999x999 virtual space) | | `max_steps` | `25` | Max execution steps | | `step_delay` | `2.0` | Delay between steps (seconds) | | `first_step_delay` | `4.0` | Delay after first step | | `temperature` | `0.1` | Sampling temperature | | `top_p` | `0.001` | Top-p sampling | | `max_tokens` | `1024` | Max output tokens | | `timeout` | `60` | Request timeout (seconds) | | `use_adbkeyboard` | `true` | Chinese text input via ADBKeyboard | | `screenshot_dir` | `"./screenshots"` | Screenshot save directory | --- ## 4. Advanced: Deploy MobiZen-GUI-4B Locally For best results on Chinese mobile tasks, deploy the dedicated 4B model. ### 4.1 Download Model ```bash pip install -U huggingface_hub # China mirror (optional) export HF_ENDPOINT=https://hf-mirror.com hf download alibabagroup/MobiZen-GUI-4B --local-dir ./MobiZen-GUI-4B ``` Alternatively from ModelScope: https://modelscope.cn/models/GUIAgent/MobiZen-GUI-4B ### 4.2 Serve with vLLM ```bash pip install vllm==0.11.0 vllm serve ./MobiZen-GUI-4B --host 0.0.0.0 --port 8000 --trust-remote-code ``` ### 4.3 Point Config to Local Model ```yaml api_key: "dummy" base_url: "http://localhost:8000/v1" model_name: "MobiZen-GUI-4B" model_type: "qwen3vl" ``` Then run as usual: `python main.py --config my_config.yaml --instruction "..."` --- ## 5. Customization (Requires Code Changes) The framework uses a plugin architecture — three components can be swapped via config class paths: | Component | Role | Base Class | Default Implementation | |-----------|------|-----------|------------------------| | **MessageBuilder** | Builds prompt + screenshot for model | `core.message_builders.base.BaseMessageBuilder` | `core.message_builders.qwen.QwenMessageBuilder` | | **ModelClient** | Calls the model API | `core.model_clients.base.BaseModelClient` | `core.model_clients.openai.OpenAIClient` | | **ResponseParser** | Parses model output → action | `core.response_parsers.base.BaseResponseParser` | `core.response_parsers.qwen.QwenResponseParser` | ### 5.1 Custom Model Client For non-OpenAI-compatible APIs: ```python # core/model_clients/my_client.py from .base import BaseModelClient class MyClient(BaseModelClient): def __init__(self, api_key: str, base_url: str = None, model: str = "", timeout: int = 60): pass # init your client def chat(self, messages, **kwargs): pass # must return obj with .choices[0].message.content ``` Config: ```yaml model_client_class: "core.model_clients.my_client.MyClient" model_client_kwargs: {} # extra kwargs passed to __init__ ``` ### 5.2 Custom Message Builder To change the system prompt or how screenshots/history are formatted: ```python # core/message_builders/my_builder.py from .base import BaseMessageBuilder from utils.image import image_to_data_url class MyBuilder(BaseMessageBuilder): def build_system_prompt(self, **kwargs) -> str: return "your system prompt" def build_messages(self, instruction, current_screenshot, history, **kwargs): return [{"role": "system", "content": [...]}, {"role": "user", "content": [...]}] ``` Config: ```yaml message_builder_class: "core.message_builders.my_builder.MyBuilder" ``` ### 5.3 Custom Response Parser To parse a different model output format: ```python # core/response_parsers/my_parser.py from .base import BaseResponseParser, ParsedResponse class MyParser(BaseResponseParser): def parse(self, response) -> ParsedResponse: content = response.choices[0].message.content # parse content into structured fields return ParsedResponse( thought="...", summary="...", action={"arguments": {"action": "click", "coordinate": [x, y]}}, subtask="..." ) ``` **Action dict format**: `{"arguments": {"action": "<type>", ...}}` — supported types: `click`, `long_press`, `swipe`, `type`, `system_button`, `wait`, `terminate`. Config: ```yaml response_parser_class: "core.response_parsers.my_parser.MyParser" ``` ### 5.4 Add New Action Type 1. Add `_execute_<action>(self, args)` method in `core/executor/action_executor.py` 2. Add dispatch branch in `ActionExecutor.execute()` 3. Update system prompt in `QwenMessageBuilder.build_system_prompt()` --- ## 6. Troubleshooting - **Device not found**: Run `adb devices` — check USB/wireless connection - **ADBKeyboard not working**: Ensure enabled in device settings; test: `adb shell am broadcast -a ADB_INPUT_TEXT --es msg "test"` - **Model connection error**: Verify `base_url` + `api_key`; check network - **Coordinate mismatch**: Ensure `model_type` matches your model; check screen size: `adb shell wm size` - **Duplicate action loop**: Agent auto-stops after 5 identical actions; may indicate model confusion


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)