Kati8/ICLR_2023_2025 huggingface dataset:
authors: Peng Chen
authors: Pi Bu
authors: Jun Song
authors: Yuan Gao
authors: Bo Zheng
reviews: 
Comments: This paper introduces a VLM-based agent that can play a AAA action role-playing (ARPG) game “Black Myth: Wukong” (BMW). Notably, The system features a human-guided trajectory system that generates an action library based on human gameplay data. The contribution includes defining 13 game-playing tasks in BMW, releasing a BMW gameplay dataset, and introducing a new framework for playing ARPG games based on VLM. The experiment shows the introduced VLM-based agent outperforms Cradle, a general-purpose computer control agent, and is competitive against human players.
Strengths: This paper not only introduces an agent framework but also introduces a benchmark and a dataset. Notably, the dataset is collected by recruiting 200 human players to play the game. The proposed agent generates an action library from human gameplay data, does not rely on text-based guiding information. The agent directly takes screenshots as input and output mouse and keyboard commands, and achieves good performance in the 13 introduced tasks.
Weaknesses: The significance of this work is not so clear. According to the presented results, Cradle, a general computer control agent based on VLMs, can also play BMW. Though the results show the proposed VARP outperforms Cradle, to my understanding, the Cradle framework does not use human gameplay data and is designed for general purposes. It is not surprising that the proposed agent can outperform Cradle. 
Meanwhile, there are some concerns about the experiment part: 
1. The proposed agent is not compared to the RL-based agent for BMW, i.e., “Other project” in Tab. 1 (I guess it’s AI-Wukong), which is also an agent playing BMW
2. How many trials do the authors repeat for each task? It seems that all success rates are divisible by 10%, so perhaps all tasks are tested for 10 trials. If possible, I would recommend the authors test agents for more trials, or explain why 10 trials are enough/why not test more trials.

Minor issues:

Lots of opening brackets are not separated from the texts with a space. For example, “action role-playing games(ARPG),”; “VARP(Vision Action Role-Playing)”.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper introduces a new eval for Multimodal LLMs (those which accept images). The authors take the Action RPG “Black Myth: Wukong” and produce a benchmark of 13 tasks from the game. The game produces screenshots. The environment asks agents to produce high level actions which are transformed by python functions into specific actions of the model.

The authors collect human performance on the task and produce a dataset of expert trajectories. This is used as a retrieval system during the agents playthrough 

The authors also simultaneously propose a method for composing LLMs to approach these problems (VARP agent). The VARG agent, using Gemini / Claude / GPT-4-Turbo is able to solve 7/13 tasks completely.
Strengths: The paper is well presented and the task is original 

The writing is clear

The 5 final tasks are clearly difficult 

Good ablations to identify what works well
Weaknesses: Given the baseline (Cradle) is also able to perform well on the 5 easy tasks this reduces the usefulness of those tasks.



My biggest concern is that i’m not sure how performance on this evaluation relates to real world capabilities. For example montezuma’s revenge was directly related to hard exploration or hinabi related to identify cooperative policies.  I think some framing to explain what models failing on the final 5 tasks demonstrates. 

I think a strong evaluation protocol could be suggested - e.g. have held out tasks or tasks that composed of easier tasks. This would help interpret what models fail at. In particular currently this seems to be a task that measures in-distribution performance, with accessing to expert trajectories. 

The dataset collected is largely skewed towards similar task - given the first 5 are not as useful - how big is the actual dataset?


The Decomposable Task-Specific Auxiliary (DTSA) is very tailored to the game (as mentioned in the limitations).

I think most of the performance is coming from the human-guided trajectory system. Why was removing this subcomponent not part of the ablation?
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 4
reviews: 
Comments: This work builds an AI agent framework, aiming to deal with Action Role-Playing Games. It successfully demonstrates the huge potential of LMM in decision-making. In addition, it builds a benchmark that may be useful for the future research in this filed.
Strengths: 1. This work needs lots of engineering efforts and the proposed benchmark will be handy for future research.
2. This work explores some of the potential of existing large models, offering the audience plenty of room for imagination.
Weaknesses: 1. The novelty of this work seems very low. The framework is similar to Cradle and many other AI agent works, as well as some tailored modules, mainly in section 3.2.3, for BMW. 
2. The game is not an open-ended world and the skill library can be enumerated. The level of difficulty is still somewhat limited.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2