LiangYan3612/guage

🤖 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning

Experiment Update

tmux new -s train_grpo_web_liang
tmux attach -t train_grpo_web_liang
conda activate agentrl_web
bash /root/userdata/liangyan/ARL-Arena/train_grpo_web_liang.sh > train_grpo_web_liang.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_grpo_web_liang
tmux attach -t train_grpo_web_liang
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_grpo_web_liang_v2.sh > train_grpo_web_liang_v2.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_grpo_w_full_clb_web
tmux attach -t train_grpo_w_full_clb_web
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_grpo_w_full_clb.sh > train_grpo_w_full_clb_web_v1.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_grpo_w_naive_dense
tmux attach -t train_grpo_w_naive_dense
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_grpo_w_naive_dense.sh > train_grpo_w_naive_dense.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_grpo_w_lyapunov_only
tmux attach -t train_grpo_w_lyapunov_only
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_grpo_w_lyapunov_only.sh > train_grpo_w_lyapunov_only.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_sampo_w_lyapunov_only
tmux attach -t train_sampo_w_lyapunov_only
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_sampo_w_lyapunov_only.sh > train_sampo_w_lyapunov_only.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_sampo_w_full_clb
tmux attach -t train_sampo_w_full_clb
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_sampo_w_full_clb.sh > train_sampo_w_full_clb.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_sampo_naive_dense
tmux attach -t train_sampo_naive_dense
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_sampo_naive_dense.sh > train_sampo_naive_dense.log 2>&1 &

✅✅✅✅✅✅

tmux new -s train_sampo_outcome_only
tmux attach -t train_sampo_outcome_only
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_sampo_outcome_only.sh > train_sampo_outcome_only.log 2>&1 &

⛔️⛔️⛔️⛔️⛔️⛔️

tmux new -s train_sampo_web
tmux attach -t train_sampo_web
conda activate agentrl_web
cd /root/userdata/liangyan/ARL-Arena
bash /root/userdata/liangyan/ARL-Arena/train_sampo_web.sh > train_sampo_web.log 2>&1 &

⛔️⛔️⛔️⛔️⛔️⛔️

---

Our Framework Design

• ARLArena profoundly analyzes the existing sufferings of Agentic RL from the perspective of policy gradient.
• ARLArena comprehensively compares existing Agentic RL algorithms and provides a systematic discussion and analysis across multiple dimensions.
• ARLArena universally provides experimental results and findings on multiple agentic tasks.

🔥 Key Features

• ✅ Support Training Multi-turn Math+Code Interpreter Agents
• ✅ Support Training Multi-turn Embody Agents
• ✅ Support Training Multi-turn Multi-modal Game Agents
• ✅ Support Training Multi-turn Web Agents
• ✅ Support Training Multi-turn Search Agents

🔧 Upcoming Features and Changes

• ➡️ Support Software Enginnering Agents

📅 TODO

• Cross-domain agentic reasoning
• Multiple tool integration reasoning

💡 Getting Started

ARLArena is based on the following main dependencies:

Python=3.11, VeRL=0.4.0, PyTorch=2.6.0, and vLLM=0.8.5

🚀 Existing Support

🛒 Web Agents

# 1. Build the webshop environments
bash prepare_all_web.sh

# 2. Run the demo code with:
conda activate agentrl_web
bash examples/shop_agent_trainer/train_xxxx.sh

🤖 Embodied Agents

# 1. Build the environments
bash prepare_all_embody.sh

# 2. Run the demo code with:
bash examples/world_agent_trainer/train_xxx.sh

🧮 Math+CI

1. We use Sandbox Fusion as an asynchronous code interpreter. You can follow the Guidance to run the CI.

2. The training datasets are Math3-5 from SimpleRL in datasets.

# 3. Install the requirements
bash prepare_all_science.sh

# 4. Run the demo code with:
bash examples/simpletir_trainer/train_xxx.sh

🎮 OpenAI Game Agents

# 1. Install the requirements
bash prepare_all_game.sh

# 2. Run the demo code with:
bash examples/game_agent_trainer/train_xxx.sh

🕸️ Search Agents

#! 1. Build the RAG server environments
bash prepare_all_search.sh


# 2. Run the demo code with:
bash examples/search_agent_trainer/train_xxx.sh

🌊 Easy Extension

🔹 All of the methods utilized is in recipe, you can warp the verl worker for your code to join our codebase. The folder under recipe can represent either a method for different tasks or a series methods for one task. You can refer to Easy Extension for examples.

🔹 All of the environments utilized is in agent_system, you can warp the env for your code to join our codebase.

🔹 Add specific dependencies to requirements_xxx.txt

🔹 Feel free to add the folder of the third-party tools, e.g., sandbox for code implementation.

📊 Further Details

Figure 2: A summary of policy optimization methods studied in ARLArena.

Figure 3: Performance comparison of policy optimization methods across four agentic tasks, evaluated on the SFT version of Qwen3-4B.

Figure 4: Training curves on ALFWorld (left) and Sokoban (right).

✍️ Citation

@article{wang2026arlarena,
  title={ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning},
  author={Wang, Xiaoxuan and Zhang, Han and Wang, Haixin and Shi, Yidan and Li, Ruoyan and Han, Kaiqiao and Tong, Chenyi and Deng, Haoran and Sun, Renliang and Taylor, Alexander and others},
  journal={arXiv preprint arXiv:2602.21534},
  year={2026}
}

🌟 Star History

🎆 Awesome work for reference

• TinyZero: a reproduction of DeepSeek R1 Zero recipe for reasoning tasks
• SkyThought: RL training for Sky-T1-7B by NovaSky AI team.
• simpleRL-reason: SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• Easy-R1: Multi-modal RL training framework
• OpenManus-RL: LLM Agents RL tunning framework for multiple agent environments.
• rllm: async RL training with verl-pipeline
• PRIME: Process reinforcement through implicit rewards
• RAGEN: a general-purpose reasoning agent training framework
• Logic-RL: a reproduction of DeepSeek R1 Zero on 2K Tiny Logic Puzzle Dataset.
• Search-R1: RL with reasoning and searching (tool-call) interleaved LLMs
• DeepRetrieval: RL Training of Search Agent with Search/Retrieval Outcome
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
• Code-R1: Reproducing R1 for Code with Reliable Rewards
• Skywork-OR1: Skywork open reaonser series
• ToRL: Scaling tool-integrated RL
• verl-agent: A scalable training framework for long-horizon LLM/VLM agents, along with a new algorithm GiGPO
• PF-PPO: Policy Filtration for PPO based on the reliability of reward signals for more efficient and robust RLHF.
• GUI-R1: GUI-R1: A Generalist R1-style Vision-Language Action Model For GUI Agents
• DeepResearcher: Scaling deep research via reinforcement learning in real-world environments
• VAGEN: Training VLM agents with multi-turn reinforcement learning
• ReTool: ReTool: reinforcement learning for strategic tool use in LLMs
• Seed-Coder: RL training of Seed-Coder boosts performance on competitive programming
• all-hands/openhands-lm-32b-v0.1: A strong, open coding agent model, trained with multi-turn fine-tuning
• RM-R1: RL training of reasoning reward models
• Absolute Zero Reasoner: A no human curated data self-play framework for reasoning
• LUFFY: Learning to Reason under Off-Policy Guidance
• verl-tool: An unified and easy-to-extend tool-agent training framework based on verl
• DeepMath: DeepMath-103K data and series models for math reasoning