Instructions to use afx-team/UI-UG2-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use afx-team/UI-UG2-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="afx-team/UI-UG2-4B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("afx-team/UI-UG2-4B")
model = AutoModelForMultimodalLM.from_pretrained("afx-team/UI-UG2-4B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use afx-team/UI-UG2-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "afx-team/UI-UG2-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "afx-team/UI-UG2-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/afx-team/UI-UG2-4B

SGLang

How to use afx-team/UI-UG2-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "afx-team/UI-UG2-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "afx-team/UI-UG2-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "afx-team/UI-UG2-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "afx-team/UI-UG2-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use afx-team/UI-UG2-4B with Docker Model Runner:
```
docker model run hf.co/afx-team/UI-UG2-4B
```

🎯 UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning

🌟 Overview

UI-UG 2 is a compact 4B-parameter multimodal model that unifies fine-grained mobile UI understanding and structured UI generation. It combines a Qwen3-VL backbone, a front-end-aligned component taxonomy, expanded grounding capabilities, and online render-aware reinforcement learning.

🆕 What changed from UI-UG?

	UI-UG1	UI-UG 2
Backbone	Qwen2.5-VL, 7B	Qwen3-VL, 4B
Understanding	Referring and dense grounding	Referring and Grounding (dense grounding and instruction-conditioned GUI grounding)
Taxonomy	20 categories	71 fine output labels, grouped into 9 parent classes using the paper's hierarchy
Generation post-training	Offline preference optimization with DPO	Online render-aware GRPO with absolute judge scores and contrastive preference ranking
Representation	Structured JSON DSL	Retains the renderable JSON DSL and progressive rendering

This repository provides inference, serving, and JSON DSL rendering. The training recipe is described in the paper; training code is not included.

🧩 Fine-grained UI taxonomy

The taxonomy covers 71 fine-grained categories across nine parent groups, distinguishing component states, functional roles, and partially visible elements.

📊 Benchmark highlights

The following charts summarize the results reported in the paper. Each panel shows the top 12 supported entries for its metric under the paper's evaluation protocol.

🔍 UI Understanding

UI Understanding consists of Referring and Grounding. Grounding includes dense UI element detection and action-conditioned GUI target localization.

🖼️ Dense Grounding: qualitative comparison

Two dense UI cases from the paper, with the same model order in each row. UI-UG 2-4B is shown in the rightmost column.

🎨 UI Generation

🖼️ Generation: qualitative comparison

Poster card and login interface comparisons from the paper. The reference image is on the left, and UI-UG 2-4B is shown in the rightmost column.

🤗 Model and earlier releases

Version	Model	Code
UI-UG 2-4B	afx-team/UI-UG2-4B	https://github.com/afx-team/UI-UG
UI-UG-7B	afx-team/UI-UG-7B	UI-UG1 archive
UI-UG-7B-2601	afx-team/UI-UG-7B-2601	UI-UG1 archive

Weights are downloaded from the model repositories or supplied through a local checkpoint path; model binaries are not stored in this Git repository.

🛠️ Installation

Use Python 3.10 or newer with a compatible PyTorch/CUDA environment for GPU inference.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For vLLM serving:

pip install -r requirements-vllm.txt

For optional screenshot rendering:

pip install -r requirements-render.txt
playwright install chromium

The dependency minimums follow the upstream Qwen3-VL installation and deployment guide: Transformers 4.57+ and vLLM 0.11+.

🚀 Quick start: Transformers

export MODEL_PATH=afx-team/UI-UG2-4B
# A local checkpoint directory can also be used for MODEL_PATH.

Use your screenshot path in the commands below.

🔍 UI Understanding

🔍 Referring

python -m demo.demo_simple \
  --image path/to/screenshot.png \
  --prompt 'Describe the region [344, 358, 524, 390].'

📍 Grounding

📍 Dense grounding

All UI items:

python -m demo.demo_simple \
  --image path/to/screenshot.png \
  --prompt 'List all ui items in the image.'

Items from one of the nine parent categories, for example Button:

python -m demo.demo_simple \
  --image path/to/screenshot.png \
  --prompt 'List all Button items in the image.'

Items from one of the 71 subcategories, for example CommonButton:

python -m demo.demo_simple \
  --image path/to/screenshot.png \
  --prompt 'List all CommonButton items in the image.'

Replace Button or CommonButton with another parent category or subcategory.

🎯 GUI grounding

python -m demo.demo_simple \
  --image path/to/screenshot.png \
  --prompt 'Detect the bounding box of the UI element corresponding to the action: Tap the search button.'

🎨 UI Generation

python -m demo.demo_simple \
  --prompt 'Generate a login form with email field, password field, and submit button' \
  --output generated_ui.json

Add --image path/to/reference.png to provide a reference image.

⚡ Serving and API inference

MODEL_PATH=afx-team/UI-UG2-4B bash scripts/vllm_deploy.sh

🎯 UI Understanding / Grounding / GUI grounding

python -m demo.run_demo \
  --image path/to/screenshot.png \
  --prompt 'Detect the bounding box of the UI element corresponding to the action: Tap the search button.'

🎨 UI Generation

python -m demo.run_demo \
  --prompt 'Generate a login form with email field, password field, and submit button' \
  --stream

Use the same prompts above with python -m demo.run_demo for API inference.

The server defaults to 127.0.0.1:8000 with served model name uiug2. Configure MODEL_PATH, HOST, PORT, TENSOR_PARALLEL_SIZE, or VLLM_API_BASE as needed. See .env.example. Authentication, when required by a model repository, uses your normal Hugging Face credentials.

🎨 JSON DSL rendering

python -m utils.render_utils \
  --input examples/ui_card.json \
  --html rendered_card.html

Add --tailwind-css /path/to/compiled.css for Tailwind styles and --png rendered_card.png to launch Chromium for a screenshot. Use --font /path/to/font.ttf for a custom font. The included Python renderer supports Tag nodes and data bindings; Component nodes require the corresponding front-end component runtime, which is not bundled here.

🤝 Acknowledgments

Qwen3-VL - Multimodal foundation model
vLLM - High-performance inference framework
Ant Group & AFX Team - Technical support and scenario applications

📖 Citation

Please cite the version relevant to your work.

@misc{yang2026uiug2,
  title={UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning},
  author={Hao Yang and Shuoyang Liu and Xiaoyu Lin and Zhou Fang and Maji Huang and Zhaosong Huang and Ruichao Mao and Teng Guo and Hai Rao},
  year={2026},
  note={Preprint}
}

@misc{yang2025uiugunifiedmllmui,
  title={UI-UG: A Unified MLLM for UI Understanding and Generation},
  author={Hao Yang and Weijie Qiu and Ru Zhang and Zhou Fang and Ruichao Mao and Xiaoyu Lin and Maji Huang and Zhaosong Huang and Teng Guo and Shuoyang Liu and Hai Rao},
  year={2025},
  eprint={2509.24361},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2509.24361}
}

📄 License

Repository code retains the Apache 2.0 license and the existing legal notice. Refer to the corresponding model cards for checkpoint terms and to the paper's distribution terms for the manuscript.

Built with ❤️ by Alipay AFX

Downloads last month: -

Safetensors

Model size

570k params

Tensor type

BF16

Paper for afx-team/UI-UG2-4B

UI-UG: A Unified MLLM for UI Understanding and Generation

Paper • 2509.24361 • Published Sep 29, 2025 • 1