Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks
Authors: Yun Qu
Boyuan Wang
Jianzhun Shao
Yuhang Jiang
Chen Chen
Zhenbin Ye
Lin Liu
Yang Jun Feng
Lin Lai
Hongyang Qin
Minwen Deng
Juchao Zhuo
Deheng Ye
QIANG FU
YANG GUANG
Yang Wei
Lanxiao Huang
Xiangyang Ji
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: offline reinforcement learning
offline multi-agent reinforcement learning
datasets and benchmarks
TL;DR: We propose Hokoff, a comprehensive set of pre-collected datasets that covers both offline RL and offline MARL, accompanied by a robust framework, to facilitate further research.

Abstract: The advancement of Offline Reinforcement Learning (RL) and Offline Multi-Agent Reinforcement Learning (MARL) critically depends on the availability of high-quality, pre-collected offline datasets that represent real-world complexities and practical applications. However, existing datasets often fall short in their simplicity and lack of realism. To address this gap, we propose Hokoff, a comprehensive set of pre-collected datasets that covers both offline RL and offline MARL, accompanied by a robust framework, to facilitate further research. This data is derived from Honor of Kings, a recognized Multiplayer Online Battle Arena (MOBA) game known for its intricate nature, closely resembling real-life situations. Utilizing this framework, we benchmark a variety of offline RL and offline MARL algorithms. We also introduce a novel baseline algorithm tailored for the inherent hierarchical action space of the game. We reveal the incompetency of current offline RL approaches in handling task complexity, generalization and multi-task learning.

Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks
Yun Qu"t, Boyuan Wang"1, Jianzhun Shao"t, Yuhang Jiang', Chen Chen', Zhenbin Ye', Lin Liu?, Junfeng Yang", Lin Lai², Hongyang Qin Minwen Deng", Juchao Zhuo Deheng Ye², Qiang Fu!, Wei Yang', Guang Yang', Lanxiao Huang', Xiangyang Ji
TTinggua University. *Tencent Timi Studio, Tencent AI Lab (qy22, 2.ag... @ wwtngghua.educccccheeeee. peach @ gmail.ccom (zhenbinye, www.:iiiu.eggguunye.....ocn hongyaggnng,,g manngii,, danierdenggjojozzuuoo jerreeo deofucy,, mikoyagn.ooyaaoakaiaiiee @eencent.com.xyji @singhua.edu..n
Abstract
The advancement of Offline Reinforcement Learning (RL) and Offline Multi-Agent Reinforcement Learning (MARL) critically depends the availability of high- quality, pre-collected offline datasets that represent real-world complexities and practical applications. However, existing datasets often fall short in their simplicity and lack of realism. To address this gap, we propose Hokoff, a comprehensive set of pre-collected datasets that covers both offline RL and offline MARL, accompa- nied by a robust framework, to facilitate further research. This data is derived from Honor of Kings, recognized Multiplayer Online Battle Arena (MOBA) game known for its intricate nature, closely resembling real-life situations. Utilizing this framework, we benchmark variety of offline RL and offline MARL algorithms. We also introduce a novel baseline algorithm tailored for the inherent hierarchi- cal action space of the game. We reveal the incompetency of current offline RL approaches in handling task complexity, generalization and multi-task learning.
Introduction
Online Reinforcement Learning (Online RL) relies on the interaction between the training policy and the environment for data collection and policy optimization [33 18 77. However, this paradigm makes online RL unsuitable for certain real-world scenarios, such as robotics and autonomous driving [23 18) as deploying untested policies to the environment can be costly and dangerous [19].
In contrast, Offline Reinforcement Learning (Offline RL) can learn satisfactory policies using fixed dataset without the need for further interaction with the environment [23] 18] 7 19]. This
characteristic alleviates the aforementioned issue, making offline RL potentially more suitable for certain real-world scenarios compared to online RL 23
The research on offline RI. has attracted significant attention in recent years and has made substantial progress both theoretical analysis and practical performance. The core challenge of offline RI. the value overestimation issue induced by distributional shift [9] 21] Existing studies mitigate this problem by constraining the learning policy to closely resemble the behavior policy induced by the
dataset [39 18 7,, or adopting conservative value iteration [19 16). The success of offline RL can be largely attributed to the availability widely-adopted open-access datasets, such as D4RL (6] and RL Unplugged [12]. These datasets offer standardized and diverse pre-collected data for the development of new algorithms, while also offering proper evaluation protocols that facilitate fair comparisons between different algorithms. However, despite their benefits, tasks contained in these datasets (such
Authors contributed equally
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

as Atari 2600 [ or Mujoco (35) are often overly simplistic or purely academic, failing to simulate the complexity of real-world scenarios and lacking practical applications. Thus, there is a significant gap between offline RL. research and its practical application in real-world settings. This disparity hinders the usefulness of offline RI. addressing real world problems, and thus, is indispensable to create datasets that reflect a realistic level of complexity and practicality in real-world applications.
Offline Multi-Agent Reinforcement Learning (Offline MARL) [28
43 gaining increasing attention due to its close relationship to real scenarios, such as games [1 38, sensor networks (49 and
autonomous vehicle control 42 However, the lack of standardized datasets restricts the development 40 of offline MARL. Existing works only rely on self-mad datasets, which hampers fairness and reproducibility. Moreover, the settings they focus on are typically limited to toy examples (e.g.. Multi-agent Particle Environment (26)) or simplified versions of classic games (c.g., StarCraft Multi- Agent Challenge 31), facing the same impractical issues encountered in offline RI. Thus, there is an urgent need for open-access datasets to further the progress of offine MARL..
The connections between offline RL and offline MARL are closely linked due to similar challenges pertaining to offline learning. While, offline MARL also introduces distinct algorithm development requirements, it involves unique characteristics like multiple agents and intra-team cooperation. The current offline MARL algorithms (28] 43 are mainly adaptations of offline RL algorithms, necessitating the availability of standardized offline datasets that cater to both single-agent and multi-agent settings. Thus, to enhance versatility and practicality, it is crucial to propose datasets that encompass both single-agent settings and multi-agent settings.
In this paper, we present Hokoff, a suite of pre-collected datasets for both offline RI. and offline MARL, along with a comprehensive framework for conducting corresponding research. Our paper makes several novel contributions, which are shown below:
The tasks we adopt are based on one of the world's most popular Multiplayer Online Battle Arena (MOBA) games, Honor of Kings (HoK), which has over 100 million daily active players (88). ensuring the practicality of our datasets. The complexity of this environment dramatically surpasses those of its counterparts, demonstrating the potential for simulating real-world scenarios.
We present an open-source, easy-to-use framework Jnder Apache License V2.0. This framework includes comprehensive processes for offline RI (sampling, training, and evaluation), and some useful tools. Based on the framework, We release a rich and diverse set of datasets which are generated using a series of pre-trained models featuring distinct design factors. These datasets cater not only to offline RL but also offline MARL.
?
Building on the framework, we reproduce various offinn RL and offline MARL algorithms and propose a novel baseline algorithm tailored for the inherent hierarchical structured action space of Honor of Kings. We fully validate and compare these baselines on our datasets. The results indicate that current offline RL and offline MARL approaches are unable to effectively address complex tasks with discrete action space. Additionally, these methods exhibit shortcomings in terms of their generalization capabilities and their ability to facilitate multi-task learning.
2
Related Works
2.1 Offline RL and Offfine MARL
Offline RL gains significant attention in recent years, primarily due to the inherent difficulties of directly applying online RL algorithms to offline environments. The main hurdles encountered is the issue of erroneous value overestimation, which arises from the distributional shift between the dataset and the learning policy [9]. Theoretical studies have demonstrated that the overestimation
issue can be alleviated by pessimism, which results in satisfactory performance even with imperfect
data coverage [4] 5 14 20 25
30 41 48). In practice. certain studies [[ 2] 40
I 2 40 employ
uncertainty-brut based methods to estimate Q-values pessimistically or to perform learning on pessimistic dynamic models by estimating the epistemic uncertainty of Q-values or dynamics. Some studies [[8]
9
16
7 adopt behavior regularization- based approaches by imposing constraints on the
http:://gttaab.ccc///cceesaaaaas

---

learned policy to align closely with the behavior policy, either explicitly or implicitly, which offers better computational efficiency and memory consumption compared to imcertainty-base methods.
Offline MARL,, combining offline RL and MARI,, has emerged in recent years to address safety and training efficiency concerns in practical multi-agent scenarios. Most studies in this domain adopt multi-agent paradigm, such as independent learning [34] or centralized training with decentralized exccution (CTDE) [26]. These investigations also incorporat offline methods, similar to those employed in single-agent settings, to mitigate distributional shift. Moreover, innovative treatments are introduced for cooperation, such as zeroth- order optimization in OMAR [28] or decomposing the joint-policy in MAICQ [4].. In addition, Jiang & Lu [13] specifically focuses on decentralized learning using BCQ [9) while Tseng et al. [36] regards offline MARL as a sequence modeling
problem, utilizing supervised learning and knowledge distillation to tackle the challenges it presents.
2.2 Offiine Datasets
The availability of large-scale pre-collected datasets has greatly facilitated the progress of deep supervised learning [1I]. Offline RL, which is regarded as bridge between RL and supervised learning, also requires learning policies from pre-collected datasets [6]. Therefore, high-quality pre-collected offline datasets play a significant role in the development of offline RL. To meet this demand, some datasets have been published and widely adopted. D4RI. [6] is designed to address key challenges often faced in practical applications where datasets may have limited and biased distributions, incomplete observations, and suboptimal data. To tackle these issues, D4RL offers range of datasets that enjoy these characteristics. Similarly, RL Unplugged (12) introduces benchmark to evaluate and compare offline RL methods with various settings, such as partially or fully observable and continuous or discrete actions. These offiin datasets play a significant role in offline RL research, and many previous works train and evaluate their methods based on these
datasets [6
2
15
28 43).
However, both D4RL and RL Unplugged primarily focus on relatively simple tasks and lack high- dimensional, practical and multi-agent tasks that closely resemble real-world scenarios. StarCraft Unplugged (27) introduces a benchmark for StarCraft II, a complex simulated environment with several practical properties. However, they only utilize a dataset derived from human replays, which lacks diversity in design for offline RL, and they did not evaluate existing offline RL methods. To address this research gap, we propose Hokoff, benchmark based on HoK, which aims provide diverse offlin datasets for high-dimensional, practical tasks, and present a comprehensive evaluation of previous offline RL and offline MARL methods with general, easy-to-use framework.
Background
Honor of Kings (HoK) one of the most popular MOBA games worldwide, boasting over 100 million daily active players [38]. The game involves two teams, cach consisting of several players who have the option select from wide range of heroes with diverse roles and abilities. In the game, heroes are expected to eliminate enemy units, such as heroes, creeps, and turrets, to gain gold and experience. The primary objective is destroy the enemies' turrets and crystal while defending their own. To succeed in MOBA games, players must learn how to choose the appropriate hero combination, master complex information processing and action control, plan for long-term decision maaiiin... cooperate with allies, and balance multiple interests. The complex rules and properties of HoK make it be more in line with the complex decision-making behavior of human society. Thus, HoK has attracted
numerous researchers interest 45 44 38
10]
The underlying system dynamic of HoK can be characterized by Partially Observable Markov Decision Process (POMDP (33), denoted by M = (S, private features, each agent has access to only local observations rather than the global state s. Specifically, the agents are limited to perceiving information about game units within their field of view, as well as certain global features. Due to the intricate nature of control, the action space A is organized in hierarchically structured manner, rather than being flattened, which avoids the representation of millions of discretized actions. Randomness added into the transition distribution P in the form of critical hit rate. The reward r is decomposed into multi-head form and each hero's 's reward is a weighted sum of different reward items and is designed to be zero-sum. Details of observation space, action space and reward are presented in Appendix D
7, d). Due to the fog of war and
3

---

Global CONUU
Mint-Map
Wildrnees
Indiible Grag
Creep
TURTEE
Satis
Gold
(a) HoKlvI
(b) HoK3v3
Figure 1: (a) The Game replay user interface (UI) in HoKIvl. (b) The UI in HoK3v3. Important information and units of the game are highlighted using orange boxes.
Hokoff
This study is based on the HoK gaming environment, which encompasses both Ivl and 3v3 maps. Our research proposes a comprehensive offline RL framework applicable to this gaming environment and utilizes it to generate diverse datasets. This section provides an introduction to the framework, game modes, datasets, and evaluation protocol employed in this study.
4.1 Framework
To enhance the usability of our Hokoff, we propose a reliable and comprehensive Offline RL framework that consists of three modules: sampling, training, and evaluation. This framework streamlines the process of sampling new datasets, developing and training baselines, and evaluating their performance. The sampling module provides simple and unified program for sampling diverse datasets using any pre-trained checkpoints. There are several reasons why our framework excels in sampling. Firstly, diverse datasets different levels of expertise Multi-Leeve Models as described in Sec. 4.1.1 Secondly, our framework employs parallel sampling techniques, ensuring efficient sampling of large and diverse datasets. Based on the training module, we have implemented various offline RL and offline MARL algorithms as baselines. Additionally, we consolidate crucial components and provide user-friendly APIs, facilitating researchers effortlessly develop novel algorithms or innovative network architectures. The evaluation module enables the assessment of trained models from different algorithms, ensuring fair comparisons. Fig. 2 demonstrates the architecture of our framework and Appendix E provides an example of the APls.
be sampled by leveraging
4.1.1 Multi-Level Models
To ensure a valid and unbiased comparison of the performance of distinct algorithms, it is crucial establish appropriate evaluation protocols 12). One such effective evaluation protocol the normalized score [6]. However, HoK is zero-sum adjustable rewards MOBA game. The episode return in the game is heavily influenced by the opponents and game settings, and the objective is to win, which renders the use of return a performance metric biased. Therefore, normalized score may not fully capture our requirements. Furthermore, similar our situation, the evaluation protocol for SMAC (31) competitive game, is based on win rate against pre-programmed AI. Nonetheless, it exceedingly challenging to create a built-in with human-like performance due to the complexity of MOBA games.
Inspired by prior works HoK 388 we present Multi-Level Models for sampling and evaluating which contains multiple checkpoints with different level. Specifically, we have extracted several checkpoints from pre-trained dual-clip PPO 45 44 models with varying levels determined by the outcome of the battle separately for 1oK1v1 and HoK3v3. We adopt the win rate against different checkpoints as our evaluation protocols to assess the ability of models. Additionally, these models, with varying levels, can be utilized on both sides to sample diverse battle data. The capabilities of these models surpass those of rule-based Al and match the levels of different human players, thus making these evaluation protocols more suitable for comparing algorithmic performance with human- level performance and facilitating diverse and effortless sampling. The details of these Multi-Level Models are provided in the Appendix F

---

ONLINE INTERACTION
OFFLINE TRAIN
Datasets
Sampling
HoK Eav
Training
Multi-Level Models
Exaluation
Trained Model
Figure 2: The architecture of the framework. The sampling and evaluation modules should interact with the environment. Multi-Level Models are the foundation baseline models of these two modules, serving as opponents in the evaluation module and being on both sides the sampling module, as described in See 4.1.1 The training module is responsible for training offline RI. algorithms using fixed datasets and producing trained models for evaluation.
4.2
Game Modes
We have incorporated two game modes from HoK into our study, namely HoKIvI [38 and HoK3v3.
The environment code of HoK3v3 is integrated into the open-source HoKlvl code following Apache License V2.0. These game modes differ in the number of agents involved and the underlying map used. Detailed information on each game mode is presented below.
B
4.2.1 Honor of Kings Arena
Honor of Kings Arena (HoK Arena or HoK1vl) is Ivl game mode where each player attempts to beat other and destroy its opponent's crystal. Specifically, each player chooses hero before game starts and controls it during the whole game. There are total of 20 heroes available for players to select, each possessing distinct skills that exert diverse effects on game environment. The observation space is continuous space consisting of 725 dimensions that contain partial observable information about the hero, opponent, and other game units. The action space is hierarchically structured and discretized, covering all possible actions of the hero in hierarchical triplet form: (I) which action button to take; (2) who to target; and (3) how to act. Furthermore, the reward is weighted sum of five categories: farming, kill-death-assist (KDA). damage, pushing, and win-lose. For a full description of this game mode, please refer to the Appendix[D.I D..
4.2.2 Honor of Kings 3v3 Arena
To further cater to the demand for Offline MARL, we adopt Honor of Kings 3v3 Arena (HoK3v3) as our experimental platform. HoK3v3 is a MOBA game, where each team comprises three heroes who collaborate to defeat their opponents. The basic rules and win conditions of HoK3v3 are similar to HoKlv1. However, the HoK3v3 map contains additional turrets on the middle road and features new area called the "wilderness" inhabited by diverse monsters, Besides, collaboration is essential in HoK3v3, where players must select different heroes and fulfill distinct roles to work together more efficiently. For instance, one hero might focus on slaying monsters in the wilderness to earn gold and experience, while the other heroes engage offensive tacties against the enemy heroes and game units. The design philosophies for observation space, action space, and reward are comparable to those used HoK However, the level of complexity in HoK3v3 significantly elevated. We provide detailed description of the game mode in the Appendix D.2 for reference.

---

4.2.3 Subtasks
Both HoKlvl and HoK3v3 are full MOBA games, featuring multi-camp competitions, which inherently pose challenges and limitations. Consequently, training on these game modes demands extensive training time and computational resources. However, HoK game comprises various sub- objectives, allowing us to decompose the overall game into manageable subtasks. These subtasks can represent diverse scenarios and are suitable for evaluating various algorithms. In this study, propose two specific noncompetitive subtasks as outlined below. It is worth noting that researchers can readily expand upon our framework to develop additional subtasks.
Destroy Turret:
One of the key sub-objectives in HoK is to destroy the enemy's turrets as quickly as possible, to gain access to the enemy crystal. To train this specific skill, we have devised subtask called Destroy Turret, which is based on Io In this subtask, the focus is solely on destroying the enemy's 's turret and crystal as quickly possible, and the enemy hero is removed.
Gain Gold: Gold is critical resource in HoK that can be used to purchase equipment, which enhances the abilities of the heroes. Inspired by resource collection tasks from previous studies [2]. we have designed a subtask called Gain Gold, which is based on HoK3v3, where the new objective to collect golds in restricted time steps, and the enemy heroes are removed. As multi-agent setting, focuses on the cooperation or intra-team competition while avoiding inter-team competition.
4.3 Datasets
To enhance the practical implications of our datasets, we have incorporated design factors that align with the real-world applications of both HoK and other relevant scenarios.
Multi-Difficltty
Intuitively, the level of difficulty in the environment significantly impacts the performance of al- gorithms. However, previous researches only utilized one set of datasets with uniform level of difficulty in the environment, which is not appropriate for HoK, where the difficulty of the task can be substantially affected by the level of opponents. Therefore, to examine the effects of varying levels of difficulty in the environment, we propose several multi-difficulty datasets with different difficulty levels. Specifically, we develop two sets of datasets: norm and hard, which are categorized based on the opponent's ' level. Within each level, we propose four datasets according to diverse win rates against the opponent: poor, medium, expert and mixed. To elaborate, the poor/medium/evy expert dataset is generated by recording the battle trajectories of a relative lower/equal/higher level model compared to the opponent, and the mixed dataset is an equal mixture of the three datasets mentioned above.
Multi-Task
As a MOBA game, HoK features a diverse cast of heroes with distinct roles and skillsets. While the overall objective remains consistent throughout matches, the selection of heroes can significantly alter the nature of the task at hand. Consequently, HoK presents multi-task challenge which requires single model to handle multiple tasks [47] 24 46). However, none of the current works provide uniform datasets for multi-task offline RL. To address this research gap, we propose a series of multi-task datasets based on the multi-task nature of HoK and evaluate the multi-task learning ability of current offline RL and offline MARL algorithms. Specifically, we define hero pool with several heroes and randomly select heroes from it to sample data. Depending on whether the selected heroes are on the controlled side or the opponent side, we sample either the multi_her or the mulli ooppo dataset. cases where both sides choose random heroes, we sample the mulli palli_hero_oppe dataset.
Furthermore, as mentioned in the previous section, different levels of opponents naturally form multiple tasks with varying environmental difficulties. Thus, we propose several level-based multi- task datasets by sampling data with randomly selected opponent levels. According to different difficulty levels, we have proposed two datasets, named norm_mule multi_levee and Laltilallllleeeee
Generalization
The unique gameplay mechanics of HoK, characterized by diverse cast of heroes with distinct roles and skillsets, lend themselves well to multi-task and serve as an ideal testbed for evaluating the generality of models across a range of tasks. Building on the previous work (38 and taking into account the realities of human combat in HoK, we have identified three key challenges for generalization: hero generalization, opponent generalization, and level generalization.

---

We have developed six experiments: "norm "norm_geneere" general" and "hard generdal""rl"""r for level generalization. "norm hero peroo_geeeall""" and "hard hero general" for hero generalization, and "norm oppo "epooggeeaal"" and "hard_opar loppo_generaae Among them, the first two experiments, "norm Lainnngeerall""" and "hard generale have their corresponding datasets, and we train models on these datasets. The latter four experiments do not require extra datasets because we directly use the existing models that have already been trained using other datasets. For more details on the design of generalization, please refer to the Appendix C
hero_geeeral""
for opponent generalization, for HoKlvl and HoK3v3, respectively.
Heterogeneous Teammate
Heterogeneous teammate a crucial research direction in MARL [2, In the practical scenarios of HoK, the capacity of each player generally different, making it naturally suitable for investigating the challenges associated with heterogeneous teammate. In order to mimic real-world scenarios and facilitate research heterogeneous teammate challenges, design two datasets in HoK3v3: These datasets were collected in standard manner,
wetsstiddpprtteee and caim__exer_____e_ with the exception that one random hero in each team controlled by a model with relatively low/high level of expertise, while the remaining heroes are controlled by the regular model.
Sub-Task
As introduced in Sec 4.2.3 we designed several practical and meaningful sub-tasks to provide diverse scenarios based on HoK. Based on these sub-tasks, we proposed diverse datasets to support Offline
RL research similar to the design of previous studies 12
Table 1: Details of datasets HoKIvl Ivl game mode
Factors
Datasets/Experiments
Capacity
Heroes
Oppo_herve heroes
Win_rate
Levels
norm_poor
1000
default
default
12%
norm medium
1000
default
default
50%
1
norm expert
1000
default
default
88%
1
norm mixed
1000
default
default
50%
1
Multi-Difficulty
hard poor
1000
default
default
6%
5
hard medium
1000
default
default
50%
5
hard eard_expert hard_mixe
1000
default
default
84%
5
1000
default
default
45%
5
hard_general norm_gemert
1000
default
default
90%
5
1000
default
default
46%
norm norm_hero heroo_genera
multi_hery
default
Generalization
hard vero_general
multi hero
default
norm geperal
default
multi hero
hard hard_oppo hoppo_generra
default
multi hero
5
norm multi level hard multi level
1000
default
default
50%
1
1000
default
default
50%
5
Multi-Task
norm norm_multi hero
1000
multi hero
default
23%
norm norm_multh patl_i_ttppppoooooo oppo
1000
default
multi_ hero
77%
multi hero oppo
1000
multi hero
multi hero
50%
1
4.3.1
Datasets Details
Table Table and Table Fpresents the details of our proposed datasets. All the datasets sampled using checkpoints with different levels as introduced in Sec. [.1.T Typically, each dataset consists of 1000 trajectories, except for the sub-task datasets, which contain 100 trajectories. The default heroes chosen for both camps are luban with Summoner Spells set to frenzy in HoKlvl and / //zhaoyun), (diaochan). fliyuanfan/// with Summoner Spells assigned on their respective roles in HoK3v3. However, in specific scenarios such as Generalization or Multi-u Task settings, we employ a random selection of heroes from a predefined set, multi hero. For the HoKlvl mode, the set comprises five heroes, (luban, direnjie, houyi, makeboluo, gongsunli]. In HoK3v3, the set consists six heroes, with two heroes assigned to each role, namely ([zhaoyun, shongwuyan), [diaochun, zhugeliang), /liymanfang, sunshangsiang/).. The win rate of the behavior policy is recorded the column labeled Win_ rate for reference. The column labeled Levels denotes the levels of opponents used for evaluation. More details of the datasets presented in Appendix
[/smite), [purify). [purify]/ based
7

---

Table 2: Details of datasets in HoK3v3 game mode
Factors
Datasets/Experiments
Capacity
Heroes
Oppo heroes
rate
Levels
norm_poor
1000
default
default
16%
worm_mediun
1000
default
default
50%
1
norm_expert
1000
default
default
82%
1
norm norm_mixed
1000
default
default
49%
1
Multi-Difficulty
hard_ _porr
1000
default
default
18%
7
rrd__medium
1000
default
default
50%
7
1000
default
default
83%
7
hard_mixe
1000
default
default
51%
7
hard gendral
1000
default
default
94%
8
norm general gemmea
1000
default
default
57%
S
norm hero Lherheerrreeeeeeee
general
multi hero
default
Generalization
hard hero gero_geneaal
multi hero
default
7
norm Saann_opo___ue geperal
default
multi hero
hard hard_oppo__tt _opoo_penerrl
default
multi heer
7
norm_ norm_multi level
1000
default
default
50%
hard_m Lid__ulti__eee
1000
default
default
50%
Multi-Task
norm_ permmuutti__mm hero
1000
multi_hen hero
default
74%
norm multi oppo
1000
default
multi_hen
26%
norm multi hero_oppo
1000
multi hero
multi hero
50%
norm stupid partner
1000
default
default
50%
Heterogencous
norm experi pariner
1000
default
default
50%
norm mixed partner
1000
default
default
50%
Table 3: Details of datasets in
Sub-Task
Datasets/Experiments
Capacity
Heroes
Oppo_ Oppo_seroes
Average Score
Levels
destroy_ turret_medium
medium
100
default
0.55
medium
Destroy Turret
cestroy_turret _eppeet
100
default
no
1.00
expert
Restroy_turret__ee mixed
100
default
no 0.73
gain gold medium gain gold expert gain gold mixed
100
default
no
0.13
medium
Gain Gold
100
default no
1.04
expert
100
default no
0.58
5
Benchmarking
Based on our framework, we reproduce various Offline RL and Offline MARL algorithms. Besides, we fully validate and compare these baselines on our datasets. The results are presented in the form of test winning rate. Each algorithm is run for three random seeds, and we report the mean performance with standard deviation. The performance behaviour policies is presented in Appendix Details of the implementations and experimental results can be referenced in Appendix G
5.1 Baselines
5.1.1 HoKlvl
The Offline RL baseline algorithms we implement are briefly introduced below: BC: Behavior cloning. TD3+BC (7): One of the state-of-the-art single agent offline algorithm, simply adding the BC term to TD3 [8] CQL [19] Conservative Q-Learning conducts conservative value iteration by adding regularizer to the critic loss. IQL [T] Implicit Q-Learning leverages upper expectile value function to learn Q-function and extracts policy via advantage-weiighted weighted behavioral cloning.
The structured action space in HoK is similar to the joint action space in multi-agent settings, which inspires us to resort to the design MARL methods. We propose novel baseline algorithm, named QMIX+CQL. Specifically, we import QMIX algorithm from the MARL literature (2) to tackle the structured action space by regarding each head of the action space as single agent and incorporate CQL regularizer term into local Q Q-funtion in QMIX for offline learning.

---

5.1.2 HoK3v3
The Offline MARL baseline algorithms are briefly introduced below: IND+BC: Behavior cloning with independent learning paradigm. IND+CQL: Adopts an independent learning paradigm for multi-agent settings, using conservative Q-learning [19]. COMM+CQL: Incorporate inter-agent communication based on IND+CQL. IND+ICQ [3]: Implicit Constraint Q-learning with inde- pendent learning paradigm, which only uses insample data for value estimation to alleviate the extrapolation error. MAICQ [43] Multi-agent version of implicit constraint Q-learning by decom- posed multi-agent joint-policy under implicit constraint with CTDE paradigm. OMAR [28]: Using zeroth-order optimization for better coordination among agents' policies, based on independent CQL.
Table 4: Averaged test winning rate normalized score (Sub-Task) of baselines in HoKlvl game mode.
Factors
Datasets
CQL
QMIX+CQL
IQL
TD3+BC
norm __poor
0.08=0.02
0.06+0.01
0.08=0.02
0.07=0.01
0.0=0.0
norm_ sem_mddiumm
0.33+0.01
0.32+0.01
0.31#0.03
0.32+0.01
0.01=0.01
norm_ norm_expert expert
0.64=0.01
0.5840.03
0.67+0.01
0.62+0.02
0.03=0.01
norm mixed
0.17=0.01
0.23+0.04
0.20=0.01
0.25+0.01
0.01=0.01
Multi-Diffielty
hard _ooo
0.01+0.01
0.01+0.01
0.01
0.01
0.00+0.00
hard hard_medium medium
0.13+0.01
0.11+0.01
0.20=0.01
0.12+0.02
0.00
hard expeert
0.33+0.01
0.30#0.01
0.44+0.05
0.34+0.04
0.00=0.00
hard mixed
0.05=0.3
0.02+0.01
0.08+0.01
0.06#0.01
0.01=0.01
norm_general
0.1000.01
0.20#0.04
0.32+0.03
0.18
0.02+0.02
hard lend__eeeral
0.04=0.01
0.03+0.01
0.08+0.02
0.02=0.01
0.00=0.00
norm_hes
0.06+0.01
0.0640.01
0.08+0.01
0.07=0.01
0.00=0.000
Generalization
hard heer_ general
0.03+0.01
0.03+0.01
0.04+0.01
0.06#0.01
0.00=0.00
norm oppo general
0.58+0.03
0.52+0.04
0.42+0.22
0.51=0.07
0.12+0.01
hard _edd__opp_____ general
0.15+0.02
0.1240.03
0.23+0.04
0.14+0.03
0.01+0.01
norm multi level
0.32+0.03
0.25+0.03
0.41=0.02
0.02=0.01 0.30#0.02
hard multi level
0.08=0.02
0,06#0.01
0.16#0.03
0.08=0.02
0.00=0.00
Multi-Task
norm_mee meuti_hero hero
0.08+0.01
0.07+0.02
0.11+0.01
0.06#0.01
0.00=0.00
norm pat_mmutt_oppp oppo
0.59+0.02
0.55+0.03
0,65+0.02
0.60=0.05
00
norm multi_maaaaa hero_oprpppp _oppo
0.26+0.01
0.21
0.32+0.03
0.28+0.05
0.03+0.01
destroy turret medium
0.61#0.06
0.63+0.01
0.61=0.03
0.60=0.02
0.67=0.03
Sub-Task
destroy_turee cestroy_turrt_ttte tttpttuurrr_ppppe expert
0.94
0.94+1002
0.92+0.05
0.95#0.01
0.57+0.13
destroy_turee turretu mixed
0.88+0.04
0.87=0003
0.8920.02
0.89:0.04
0.82=20.03
5.2 Benchmark Results
We have validated the offline RL and offline MARL baselines on our datasets and aggregated the results in TableIaan and Table 5
As indicated in Table QMIX+CQL exhibits superior performance in comparison to other approaches, implying that the integration of MARL methods may be a suitable choice for environments with a structured action space. Moreover, in HoK3v3, IND+ICQ exhibits the highest performance across most datasets, except for the Heterogeneous datasets. Conversely, algorithms based on TD3, namely TD3+BC and OMAR, yield poor results.
Baselines Comparison:
Multi- :ulti-Difficulty: The baseline performance exhibits significant decrease on the hard-level datasets compared with norm-level datasets, highlighting the limitations of current offline methods in addressing challenging tasks with discrete action space.
The disparities between training and evaluation in Generalization settings impede the achievement of desirable performance, indicating the inadequacy of current methods' generalization ability.
Generalization:
Multi-Task: Training models on Multi-Task datasets results substantial performance
enhancement compared to generalization settings. However, none of these models have been able to exceed the performance achieved by the behavior policy, underscoring the need for further research into the direct application of offline methods to multiple tasks.

---

Table 5: Averaged test winning rate or normalized score Tasbk of baselines in HoK3v3 game mode.
Factors
Datasets
IND+BC
COMM+CQL
IND+CQL.
IND+ICQ
MAICQ
OMAR
norm_ poor
0.1+0.01
0.0920.02
0.03+0.01
0.12#00.2
0.1240.04
0.02+0.01
medium
0,4820.01
0.47a0.04
0.480.03
0.45+0.01
0.23:0.03
norm_ce expeer
0.52+0.03
0.76=0.13 13
0.84:0.06
0.65+0.12
0.61+0.09
0.39+0.16
norm mixed
0.35=0.25
0.48=0.12
0,46:0.12
0.44#0.19
0.24#0.16
0.17=0.2
Multi-ifficllty
hard poor
0.1640.03
0.1140.04
0.12+0.03
0. 0.1740.00
0.1240.03
0.08+0.04
hard medium
0.38+0.05
0.35=0.03
0.31#0.02
0.420.08
0,2+0.06
0.23+0.07
hard expert
0.6520.01
0.6620.05
0.67=0.04
0.67+0.02
0.52=0.1
0.35+0.17
hard_mae mixed
0.32+0.14 14
0,34
0,3+0.1
0,34+00.88
0.23+0.08
0.16+0.17
norm general
0.3420.05
0,35+0.04
0.29
0.37=0.04
0.2940.06
0.09
hard general
0.28s0.03
0.310.05
0,2840.04
0.31#0.09
0.1310.04
norm her general
O. 0.17+0.03
0.13=0.02
0.14=0.04
0.2
0.2g40.06
0.13+0.004
Generalization
hard hero general
0.1610.05
0.1920.05
0.17.0.02
0.1740.02
0.0740.05
0.0840.03
papmm_pp_p_ppp general
0.21
0.14=0.03
0.14
0.18+0.06
0.13+0.07
0.12+0.03
hard oeperral
0.09
0.08=0.02
0.09#0.02
0.08+0.04
0.04+0.02
0.04+0.01
norm multi Jevel
0.43=0.09
0.36=0.02
0.34
0.44+0002
0.3840.11
0,22+0.07
hard_multi multi level
0.38=0.08
0.33=0.08
0.29=0.07
0.37+0.05
0.27+0005
0.2+0.01
Multi-Task
norm multi hero
0.57=0.07
0.31
0.3+0.07
0.59#0.05
0.51
0.39
norm multi oppo 0.09
0,08&0.05
0.07
0.1240.04
0.07
0.02+0.01
norm multi hero_oppo _ppo
0.3+0.04
0.23+0.1
0.26=0.07
0,31=0,07
0.26+0003
0.07+0.02
norm stupid partner
0.1120.15
0,33+0.06
0.24
0.2240.14
0.16110.99
0.0810.05
Heterogeneous
norm expert, partner
0.36e00.0
norm mixed partner
0.49+0.2
0,59=0.19
0.32+0.03
0.42+0.38
0.17+0.27
0.15+0.04
gain gold medium
0.1320.01
0.12#0.01
0.12+0.01
0.15+0.01
0.1340.03
0.1420.01
Sub-Task
gain gold expert
1.01=0.03
0.98=0.02
1.00=001
L03+0.01
0.98+0.00 08
0.79#0.06
gain gixd_mixedd mixed
0.6420.29
0.41=0.21
0.41+0.1
0.4620.16
0.25+0.23
0.13+0.04
As expected, the presence a low-ability partner can disrupt cooperation and hinder offline learning the stupid_ _aciidd_ateerrrree datasets, whereas expert partner has the opposite effect, highlighting the limitations of existing research on heterogeneous offline MARL.
Heterogeneous:
The offline baselines exhibit robust performance in the Sub-Task at low training cost, Additionally, BC demonstrates a competitive capability as well.
Sub-Task:
Ablations of learning paradigms: We conduct ablation experiments to investigate the impact of communication and the CTDE paradigm. Specifically, from the comparison of COMM-CQL and IND-CQL, we can reveal that incorporating communication generally results in better performance due to promotion of cooperation. Surprisingly, we found that the independent paradigm (IND- ICQ) outperformed the CTDE paradigm (MAICQ), which may be attributed to the challenges in the CTDE paradigm associated with credit assignment of agents with distinct rewards and roles.
6 Conclusion
In this paper, taking into account the limitations of existing offline RI. datasets about practical applications, we introduce Hokoff, based on Honor of Kings, well-known MOBA game that offers high level of complexity for simulating real-world scenarios. We present a comprehensive framework for conducting research in offline RL and release diverse and extensive collection of datasets, incorporating various levels of difficulty and a range research factors. Moreover, the chosen tasks for dataset collection not only cater Offline RL but also serve the purpose of offline MARL. We replicate multiple offline RL and offline MARL algorithms and thoroughly validate these baselines our datasets. The obtained results highlight the shortcomings of existing Offline RL methods, underscoring the necessity for further research in areas such as challenging task settings, generalization capabilities, and multi-task learning. All components, including the framework, datasets, and baseline implementations, discussed this paper are fully open-source.
References
[1] Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective offline rein- forcement learning. International Conference on Machine Learning, pp. 104-114, PMIR, 2020.
10

---

[2] An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- breertainty-based based offline reinforcement learning with diversified g-ensemble. Advances in Neural Information Processing Systems, 34, 2021.
[3] Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
[4] Buckman, J., Gelada, C., and Bellemare, M. G. The importance of pessimism in fixed-dataset policy optimization. arXiv preprini arXiv:2009.06799 2020.
[5] Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.022444 2022.
[6] Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[7] Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning, Advances neural information processing systems, 34:20132-20145, 2021.
[8] Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. International conference on machine learming, pp. 1587-1596. PMLR, 2018.
[9] Fujimoto, S., Meger, D., and Precup, D. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019.
[10] Gao, Y., Shi, B., Du, X., Wang. L., Chen, G., Lian, Z., Qiu, F., Han, G., Wang, W., Ye, D., et al. Learning diverse policies in moba games via macro-goals. Advances Neural Information Processing Systems, 34: 14:16171-1611
2021.
[1]] Goodfellow, I., Bengio, Y., and Courville, Deep learning. MIT press, 2016.
[12] Gulcehre, C., Wang, Z., Novikov, A., Paine, T., Gómez, S., Zolna, K., Agarwal, R., Merel, J. S., Mankowitz, D. Paduraru, C., al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:7248-7253 2020.
[13] Jiang. J. and Lu, Z. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2108.0188 2021.
[14] Jin, Y., Yang, Z., and Wang, Z. pessimism provably efficient for offline rI? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
[15] Kidambi, R., Rajeswaran, A., Netrapalli, P., Joachims, T. Morel: Model-based offline reinforcement learning. arXiv preprint (=xxi:200050555// 2020.
[16] Kostrikov, I., Nair, A., and Levine, Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:21 2021.
[17] Kuba, J. G., Chen, R., Wen, M., Wen. Y., Sun, F., Wang. J., and Yang. Y. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109. 11251, 2021.
[18] Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-po qelearning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019.
[19] Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative g-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1 179-199919, 2020.
[20] Kumar, A., Hong, J., Singh, A., and Levine. S. Should run offline reinforcement lcarning behavioral cloning? Deep RL Workshop NeurlPS 2021, 2021.
[21] Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline-to-online reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference on Robot Learning, pp. 1702-1712 PMLR, 2022.
11

---

[22] Leibo, J. Z., Duefiez-Gzzman, E. A., Vezhnevets, A., Agapiou, J. P., Sunehag, P., Koster, R.. Matyas, J., Beattie, C., Mordatch, I., and Graepel, Scalable evaluation of aultii-aeent reinforcement learning with melting pot. In International Conference Machine Learning (ICMI.), 2021.
[23] Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[24] Li, Vuong, Q., Liu, S., Liu, M., Ciosek, K., Ross, K., Christensen, H. I., and Su, H. Multi-task batch reinforcement learning with metric learning, 2020.
[25] Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. Provably good batch reinforcement learning without great exploration. arXiv preprint arXiv:2007.08205 2020.
[26] Lowe, R., Wu, Y. L., Tamar, A., Harb, J., Abbeel, O. P., and Mordatch, I. Multi-agent actor-critic for mixed competiiivee environments, In Advances in neural information processing systems (NeurIPS), 2017.
[27] Mathieu, M., Ozair, S., Srinivasan, S., Gulcehre, C., Zhang, S., Jiang, R., Le Paine, T., Zolna, K., Powell, R., Schrittwieser, J., et al. Starcraft ii unplugged: Large scale offline reinforcement learning. In Deep RL Workshop NeurlPS 2021, 2021.
[28] Pan, L., Huang. L., Ma, T., and Xu, H. Plan better amid conservatism: Offline multi-agent rein- forcement learning with actor rectification. In International Conference on Machine Learning, pp. 17221-17237. PMLR, 2022.
[29] Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1803. 14855, 2018
[30] Rashidinejad, P., Zhu, B., Ma, C.,, Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34, 2021.
[31] Samvelyan, M., Rashid, T., de Witt, C. C. Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint arXiv: 7902.04043, 2019.
[32] Seraj, E., Wang, Z., Paleja, R., Martin, D., Sklar, M., Patel, A., and Gombolay, M. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 2/st international conference on autonomous agents and multiagent systems, pp. 1172-1182, 2022.
[33] Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
[34] Tan, M. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Interna- tional Conference on Machine Learning (ICML), 1993.
[35] Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 JEFF/RS international conference on intelligent robots and systems, pp. 5026-5033. IEFE, 2012.
[36] Tseng, W.-C., Wang. T.-H., Lin, Y.-C., and Isola, P. Offline multi-agent reinforcement learning with knowledge distillation, In Advances in Neural Information Processing Systems, 2022.
[37] Wei, H., Ye, D., Liu, Z., Wu, H., Yuan, B., Fu, Q., Yang, W., and Li, Z. Boosting offline reinforcement learning with residual generative modeling. arXiv preprint arXiv:2106.0041 2021.
[38] Wei, H., Chen, J., Ji, X., Qin, H., Deng, M., Li, S., Wang, L., Zhang, W., Yu, Y., Linc, L., et al. Honor of kings arena: environment for generalization in competitive reinforcement learning. Advances in Neural Information Processing Systems, 35: 1881-1111, 2022.
[39] Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint 191.11111. 2019.
12

---

[40] Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., and Goh. H. Uncer- tainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08144 2021.
[41] Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[42] Xu, Z.,, Lyu, Y., Pan, Q., Hu, J., Zhao, C., and Liu, S. Multi-vehicle flocking control with deep deterministic policy gradient method. In International Conference Control and Automation (ICCA), 2018.
[43] Yang. Y., Ma, X., Li, C., Zheng, Z., Zhang, Q.. Huang, G. Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning, Advances in Neural Information Processing Systems, 34:10299-100 -0309, 2021.
[44] Ye, D., Chen, G., Zhang, W., Chen, S., Yuan, B., Liu, B., Chen, J., Liu, Z., Qiu, F., Yu, H., al. Towards playing full moba games with deep reinforcement learning. arXiv preprint arXiv:2011. Sttv:2011.12622 12692, 2020.
[45] Ye, D., Liu. Z., Sun, M., Shi, B., Zhao. P., Wu, H., Yu, H., Yang, S., Wu, X.., Guo, Q.. et al. Mastering complex control moba games with deep reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 6672 -679,, 2020.
[46] Yoo, M., Cho, S., and Woo, H. Skills regularized task decomposition for multi-task offline reinforcement learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 37432-37444, Curran Associates, Inc., 2022. URL http:ttts////rooeeeeiii neurips cc/ wwtppps.ccpppp..ppeee fallsr Conference, pdf
[47] Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Levine, S., and Finn, Conservative data sharing for multi-task offline reinforcement learning, 2021.
[48] Zanette, A., Wainwright, M. J., and Brunskill, E. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[49] Zhang, C. and Lesser, V. R. Coordinating multi-agent reinforcement learning with limited communication. In International Conference Autonomous Agents and Multiagent Systems (AAMAS), 2013.
13

---

A Author Statement
The authors of this work would like to state that we bear full responsibility for any potential violation of rights, including copyright infringement or unauthorized use of data. We affirm our commitment to conducting this research in accordance with ethical guidelines and legal requirements.
We further guarantee that we will ensure access to the data and the framework codo used in this study, making them available to interested researchers for verification and replication purposes. Additionally, we are committed to providing the necessary maintenance and support to ensure the longevity and accessibility of the data. For datasets, we have plans to consistently offer more datasets in the future. These datasets will include larger sizes for larger models, higher levels for expert agents, and novel design factors for other research directions. Updating our datasets is an ongoing and long-term effort, and we welcome contributions from the community. Regarding benchmarks, we will actively monitor the latest state-of-the-arr (SOTA) algorithms in the offline RL domain and integrate them into our benchmarks. Additionally, we will develop new algorithms within the benchmarks based on existing datasets and baselines. This ensures that our benchmarks remain up-to-date and reflect the advancements in offline RL research.
Should any concerns or inquíries arise regarding the contents of this work or the associated data, we encourage readers and fellow researchers to contact us directly. We are dedicated to addressing any issues promptly and transparently to uphold the integrity of our research.
B Limitations and Future Works
In our future endeavors, we plan to integrate our framework with a large-scale deep reinforcement learning platform namely KaiwuDRL, specifically designed support Honor of Kings. By doing so, we will gain access to greater computational resources, enabling us to delve deeper into our current research endeavors and expand our investigations.
C Additional Datasets Details
C.1 HoK1vl
In the Generalization category, "norm general" and "hard soard_general, have their corresponding datasets. For example, to sample the "norm general" dataset, we let the level-1 model fight with level-O, level- 2, and level-4 models. However, during the test stage, we assess the generalization capabilities of the trained model by letting it fight against the level-1 model. Details about how we sample the generalization datasets can be referred to Table. [ The latter four experiments do not require extra datasets. For example, in the "norm_ hero _erereal experiment, we directly use the model trained on the moorm_mediun dataset and let the model control different heroes. This is possible because the "norm_ "norm_mediiu" mediumm dataset only contains the fixed default hero "luban."TTherr Therefore, we use the model trained on this dataset to test its generalization ability at controlling different heroes.
In the Sub-Task: Destroy Turret category presented in Table B there are three datasets sampled, cach consisting of 100 trajectories. Notably, these datasets lack an opponent hero, making them simpler in nature. This design choice allows for broad applicability, diversity, and cost-effictiveness in research endeavors.
The primary objective in the Sub-Task: Destroy Turret scenarios is to efficiently dismantle the enemy's turret and crystal, with the enemy hero removed. Consequently, we adopt the number of game frames elapsed from the start of the game until the crystal's destruction as our evaluation protocol. Equation T outlines the scoring methodology employed, following similar approach as presented in (6) The score is normalized by two factors: random frame_lenger set to 2880, and expert_ prame_ength,, set to 1812 A higher score is achieved by minimizing the time required to destroy the crystal.
In addition, we have generated violin charts represent the distribution of episode returns in each dataset as shown Fig. We calculate episode returns using the formula R= Σ 7'r,, where
14

---

gamma is set to 1.0 to showcase the overall rewards obtained throughout an entire episode. For the Sub-Tase Task: Destroy Turrer datasets HoKIvl, we have normalized the scores based Equation The violin charts demonstrate the diverse distribution of episode returns within our datasets.
Trame_eengll random_ expert paneert_rameeleneee
normalized_sub Laak score
(1)
40
20
0510000 raturn
20
40
40
-50
norm poor
norm, medium
norm, export
niom, mixed
hard_poc
hand_medium
hand_exper
hard, mard_mixed
(a) level
(b) hard_level
60
40
optcode rszum
-20
40
-00
600
-80
norm geniraal
hard_genera
nom,, mattimmillleee Jevel
hard, multi level
Level_general
(d) multi_leve
60
1.2
cptodd retum
-00
-80
popp. (e) multi hero_oppo
destroy medium
derremy_turee_ Burret expert
destmy burret, misd
(f) sub_task: destroy_turree
Figure 3: Violin diagrams of all datasets HoKIvl.
C.2 HoK3v3
The design datasets and experiments pertaining to the concept of Generalization closely resembles that of the Hollvl.
15

---

We introduce sub-task called "Gain Gold" that builds upon the HoK3v3 game. In this modified version, we remove opponents and redefine the primary objective focus on collecting gold within limited number of time steps. This transforms the original competitive task into resource collection task. Specifically, we set a maximum episode length of 8000 frames, and the controlled heroes are required to efficiently gather gold by killing monsters or creeps. As demonstrated in Table 3 we generate three datasets based on this sub-task, each consisting of 100 trajectories. godd__ddddddium
The gain_gold__ag medium dataset is collected by a model with moderate performance, averaging 5904 gold collected. While, the gain cain_gold__a gold_expert dataset is obtained from a model with expert performance, averaging 12271 gold collected. Lastly, the gain gold mixed dataset combines the data from the previous two datasets equally. The scores are normalized based on Equation 2 where the random Landom_gaiin_ata ggoldloll is 5000 and expert_gaaie laigganinngllla is 12000.
We have also generated violin charts in HoK3v3 to represent the distribution of episode returns in each dataset as shown in Fig. The plot method used is similar to that in HoKlv with the exception of not using normalized scores in the Sub-Task: Gain Gold.
gain gold_gold- random gain loiin_gold
normalized_sule bttttiedd_____tss____e__ toabb_tas_scorrr score
experl_gain_ - random_gain
Table 6: Details of sampling generalization datasets.
Sampling and Training
Testing
Environments
Datasets
Controlled side model
Opponent side model
Opponent side model
HoKIvI
norm_general
level-1
level-0,2,4
level-1
hard general
level-5
level-0,2,4
level-5
norm general
level-5
level-1,4.7 level-1,4,7
level-5
HoK3v3
hard_general
level-8
level-8
D Environment Details
D.1 Honor Kings Arena
For a more detailed account of the game settings, please refer to the original paper [38 and its documentatior of Honor of Kings Arena. this context, we will only summarize the critical information that is relevant to the RL research.
Observation Space
We have utilized the fundamental set observations presented in the aforementioned paper [38]. Specifically, the observation space of Honor of Kings Arena consists of normalized vec- tor with 725 dimensions, which includes five main components: hero_state_commonn_ hero perro__iivate__ pritvateeeeatureeeeeeeeteeeeaee servation vector are demonstrated in Table [7] In the table, Main _ammpaaapppp and Enemy _aampppcapppp refer to the information of the controlled side and enemy side, respectively. Moreover, the information of invisible units is set to the default value.
feature, creep gree_feature turret feature, and flobal_feature The details of the ob-
Action Space To tackle the complicated control, the Honor of Kings adopt a structured action space. Specifically, illustrated Fig. 5 the action space is dimensions, consisting of a triplet form, i.e. the action button, the movement skill offset and the target game unit, which covers all the possible actions of the hero hierarchically: 1) what action button to take, e.g. skill or move.; 2) who to target, e.g., turret, an enemy hero, or creep in the troop; 3) how to act, e.g., the discretized direction to move and release skills 38) Please refer to Table 8 for details of action space in HoKIVI.
Action Mask There are two action masks designed to reduce the complexity of the action space, namely the
maan and the sub bub_action___ maation_mask. The former is constructed based on the rules of the game in order to exclude illegal actions, while the latter is determined by the selected button to eliminate actions that cannot be executed simultaneously with the chosen button, such as 'skill offset" and 'target unit' are not needed for 'move'.
https://aiarena tencent. www.hoo/ooooc///ccoccocoococccce
16

---

Table 7: Details of observation vector in Honor of Kings Arena
feature name
dimensions
description
102
hero's status, including whether it's alive, its and its health points (HP)
Main_camp camp_hero parro__rivttt__eee private_feature feature
133
hero's specific kill information enemy hero's status, including whether it's alive, ID, its health points (HP)
Enemy_ _amp hero ctate_comonn feature
102
haany__cmp_hero_prrvattteeee
133
enemy hero's specific kill informa- tion
Public_feature
14
visible information because of the turret
Main_camp__olddierfeauuur
18
the status of the creeps in a troop, including location and HP
Enemy_ Sacpeyy__ampaaee haary_aamm_ssooo_ soldier feature
18*4
the status of the enemy's 's creeps in troop, including location and HP the status of turret and crystal the status of enemy's turret and crys- tal
Main_ cari__aap__org organ feature
18
themy_camp-orran_feer
18*2
flatur_
25
the period of the match
Table 8: Description of action space in HoK1vl
Action Class
Numbers
Description
Button
12
what action button to take, e.g. skill move. move direction along X-axis.
Move
16
Move Y
16
move direction along Y-axis.
Skill
16
skill offset along X-axis. skill offset along Y-axis. who to target, e.g., a turret or an enemy hero
Skill X
16
Target
Reward Design The basic hero reward weighted average of several reward items, which is demonstrated in Equation Subsequently, the hero's 's reward is transformed into zero_sum value by subtracting the enemy's ' reward from it, as shown in Equation 4 Here, team_ ream_reward represents the average reward of the heroes within the team. Details of the reward items demonstrated in Table
hero_reward =1 farming_related + w2 KDA + wy * domage_related
+ WA Beachinnggreeted-ee + ws * win/lose hate:llooeeeeateeee related
hero_rewardzera.
hero_reward - learm_rewardennms
(4)
D.2 Honor Kings 3v3 Arena
For more detailed account of the game settings, please refer to the documentation of Honor of Kings 3v3 Arena [HoK3v3) | The environment code HoK3v3 integrated into the open-source 1v1 code, both with official authorization from Honor of Kings 8
Observation Space Specifically, the observation space of HoK3v3 consists of a normalized vector with 4586 dimensions. The details of observation vector are presented in Table I0
Action Space The form of action space HoK3v3 similar that in Ivl while the number of actions is larger. Description of action space in HoK3v3 is presented in Table II
kinga/

---

Table 9: Description of reward items in HoKIvl
Items
Type
Description
hp_point
dense
the rate of health point of hero the rate of health point tower the increment of gold the rate of mana point being killed
tower_hp point
dense
money
dense
ep_rate
dense
death
sparse
kill
sparse dense
killing an enemy hero the increment of experience the Ist for soldier
exp
jast_hit
sparse
Table 10: Details of observation vector HoK3v3.
feature name
dimensions
description
FeaturelmgLikeMg
6*17*17
image-like feature, comprising six channels, which include barriers, grass, and other elements.
VecFeatureHero
6*251
the status of six heroes from the re- spective of controlled hero.
MainHeroFeature
44
private information of controlled hero.
VecSoldier
20
the status of all creeps. the status of turrets and erystals in both side.
VecOrgan 6*29
VecMonster
20*28
the status of all monsters.
VecCampsWholelnfo
68
the status feature of the whole game.
Reward Design The basic hero reward is a weighted average of several reward items. Then the reward of each hero is processed to be zero sum in minus the team reward of enemy which is the average of the hero rewards of enemy heroes. The details of reward items are demonstrated in Table
E
Framework APIs
We provide an example the APIs in our framework, Listing A comprehensive account of our framework can be found in our readily accessible open-access code repository.
F
Evalution Protocols: Multi-Level Models
Based on the parallel training system named SAIL proposed by previous work 45 we have extracted and published several checkpoints from pre-trained dual-clip PPO [45 44) models with varying levels determined by the outcome of the battle separately for HoKlvl and HoK3v3.
Here, present tables displaying the win rate of each level model against the model listed di- rectly below it. The win rate is calculated with fixed hero selection, i.c. luban for HoKlv1 and
Table 11: Description of action space in HoK3v3
Action Class
Numbers
Description
Button
13
what action button to take, e.g. skill move.
Move
25
move direction.
Skill X
42
skill offset along X-axis.
Skill X
42
skill offset along Y-axis.
Target
39
who to target, e,g., turret enemy hero
18

---

Table 12: Description of reward items in HoK3v3
Items
Type
Description
hp_ hp_rate_ up__rae__qrt__t___ sqrtsqrt
dense
the fourth root of the rate of health point of hero the increment of gold
money
dense
exp
dense
the increment of experience the rate of health point turrets
tower
dense
killCnt
sparse
kill enemy
assistCnt
sparse
assisting in the termination of an adversary being killed
dcadCnt
sparse
Inta_hurr__e
dense
damage dealt to the enemies
_ik_mmoster
dense
attack monster
atk atk_crystal
dense
attack the crystal of enemy
win_crystal
sparse
destroy the crystal of enemy
diaoccan, liyuanfng// for HoK3v3, which may not right for other hero selection. Table I3 presents the win rate in the HoK1v1, where the win rate column represents the win rate of modell against model2. Table 14 displays the win rate in HoK3v3. Additionally, we have included an API in our framework that allows researchers to conveniently test the win rate between any levels.
Table 13: Win rate of multi-level models in HoKIvl
modell
model2
win_rate
Ivl_level]
1v] [vl_levelo 0
88%
Ivl leveel_22
lvl_level_
79%
1vl level 3
Ivl level
59%
Ivl level 4
Ivl level 3
97%
Ivl level 5
Ivl level 70%
Ivl level 6
Ivl level 5
73%
_1__evel__
1v1 [1__evel_6 70%
Table 14: Win rate of multi-level models in HoK3v3
modell
model2
win_rate
3v3 [+3_levell
3v3_level_
97%
Sv3_level_ 2
3v3 [v3_level_
83%
3v3_level_3
3v3_level_2
50%
3v3 [e3_le
3v3 level
65%
level_
3v3 [33_evee__ 4
63%
3v3 level_ 6
3v3 level_
59%
3v3 _eveleee" 7
6
80%
3v3 level_ 8
3v3 level_
82%
G Addtional Experimental Details
G.1 Additional Algorithm Details
Due to the complexity of the observation space, it is necessary to utilize a well-designed encoder for effective feature extraction. Taking inspiration from the "divide and conquer" approach employed in previous works [4 44,, in each algorithm, we implement a shared encoder network to process features, instead of directly feeding observations into the policy critic network. For further details on the design of the encoder network, please refer to the mentioned papers [5] as well as code.
Encoder:
BC: Behavior clone with maximum likelihood estimation loss. While, in multi-agent setting, HoK3v3, we adopt shared parameter and independent learning paradigm (34].
19

---

The implementation of Conservative Q-Learning is based on the original version 19 designed for discrete action space:
CQL [9]
CMI Due to the decoupling of control dependencies 45 the action space of HoK is structured with multi-head, which is similar to the joint action space in multi-agent settings. Inspired by this, we propose QMIX-CQL by incorporating mixer in QMIX (29] with CQL and use global Q calculate td error term and use local Q to calculate CQL-Ioss term.
Our implementation of TD3-BC based on the open-source code To In addition, the policy network and critic network share an encoder, which is updated simultaneously by both losses. Besides, We utilize Gumbel-Softma reparameterization method to generate discrete actions
TD3+BC 7
for TD3 88.
IQL [6] We implement IQI. based on the open-source pytorch versior The network design similar to TD3-BC except for an additional value network.
IND+CQL and COMM+CQL: To accommodate multi- aggett setting, based on the implemen- tation of CQL, we adopt the independent learning paradigm and shared parameters, referred to as IND-CQL. Additionally, we introduce COMM-CQL which adds communication between agents by means of shared information that is constructed using max pooling.
We implement IND+ICQ and MAICQ based the original published code [2] IND+ICQ adopts independent learning paradigm, while MAICQ adopts CTDE paradigm by decomposing the joint-policy under the implicit constraint. The actor and critic networks update the shared encoder simultaneously as TD3-BC.
IND+ICQ and MAICQ 431:
The open-access code of OMAR is not suitable for discrete action space. Consequently, based on the core idea of it, we have undertaken the task of re-implementing OMAR to accommodate discrete version.
OMAR 128
G.2 Hyperparameters
We have compiled the hyperparameters of HoKIvI and HoK3v3 in Tables 15 and 16 respectively, These tables encompass the parameters of the training process, algorithm and optimizer settings.
Regarding the computing resources employed in HoKIvl, we utilize the Tesla T4 GPU and the AMD EPYC 7K62 48-Core Processor CPU. For the sampling process. 50 CPU cores are utilized, and each dataset required approximately 30 to 40 minutes for sampling. During the training process, each training experiment conducted with one Tesla T4 GPU and two CPU cores, with an average training time of 9 hours per seed for 500000 training steps.
Regarding the computing resources employed in HoK3v3, we utilize the Tesla T4 GPU and the AMD EPYC 7K62 48-Core Processor CPU. For the sampling process, 50 CPU cores are utilized, and each dataset required approximately 80-90 minutes for sampling. During the training process, each training experiment is conducted with one Tesla T4 GPU and four CPU cores, with an average training time of 20 hours per seed for 500000 training steps.
Consequently, a total 14 GPUs and 552 CPU cores are used to accommodate the overall computa- tion requirements.
G.3 Additional Results Discussion
Why is the performance of baseline models in the HoKIvl comparatively inferior to those in the HoK3y3 setting?
The experimental results conducted in the HoKlv] reveal that the performance of baseline models is comparatively inferior to those in the HoK3v3 setting. This disparity can be attributed to the higher level of adversarial conditions present in the HoKlvl environment. Furthermore, within the context
hatpe://ggtaab..ooo///tttijnn/1p
httpe://githuu.cco//ingg
20

---

Table 15: Hyperparameters for HoKlv1. The values of hyperparameters for algorithms are derived from their original implementation.
Hyperparameters
Value
Batch Size
128
0.99
Max Steps (Exclude Sub-Task Datasets)
500000
Max Steps (Sub-Task Datasets)
100000
LSTM Time Steps
16
(Soft-Targe-Updaall
0.005
num num_thresd num_threads
2
final anal_evatuationeppistee
150
CQL
10.0
TD3+BC a
2.5
IQL
0.7
IQL 3
3.0
Optimizer
Adam
betal
0.9
beta2
0.999
eps
1. 1.00E-08
Learning Rate
3.00E-04
Table 16: Hyperparameters for HoK3v3. The values of hyperparameters for algorithms are derived from their original implementation.
Hyperparameters
Value
Batch Size
512
7
0.99
Hard Update Frequency
2000
Max Steps
500000
Max Steps (Sub-Task)
100000
Iteration Steps
1000
Buffer Workers
2
num num_threade threads
4
final Shal_eealuation_eegg eppisodes
150
CQL
10.0
ICQ critic 8
1000
ICQ policy 3
0.1
OMAR coe
0.5
Optimizer
Adam
betal
0.9
heta2
0.999
eps
1.00E-08
Learning Rate
1.00E-04
21

---

of HoK3v3, if one teammate makes sacrifice during a battle, the remaining two teammates are able maintain their collaboration and continue to fight. This aspect ensures greater level of robustness in the HoK3v3 when compared to the HoK1vI.
What are the reasons behind the underperformance of TD3+BC and OMAR?
TD3+BC and OMAR demonstrated subpar performance IoKlvl and HoK3v3, respectively. The main cause of their lackluster outcomes stems from the fact that TD3 [ upon which TD3+BC and OMAR are built, is incompatible with discrete action spaces. To enhance OMAR's performance, we replaced TD3 with advantage-weighted BC. This modification resulted in performance improvements.
QMIX+CQL HoK3v3.
We also implemented CMIX+CQL in the HoK3v game mode by adopting an independent learning paradigm. We thoroughly validated the performance of QMIX+CQL. and compared with IND+BC and IND+CQL, aggregating the results in Table 17 It demonstrated that QMIX+CQL, exhibits superior performance compared IND+CQL, indicating that our novel method is also suitable for multi-agent settings with a structured action space.
Table 17: Validation of QMIX+CQL in HoK3v3 game mode.
Factors
Datasets
IND+BC
IND+CQL
QMIX+CQL
norm poor norm medium
0.1+0.01
0.03+0.01
0.11=0.03
norm_medium
0,48+0.01
0,420.03
0.52+0.04
norm_exper norm mixed hard _poo hard _eddimmeimmm
0.52+0.03
0.84+0.06
0.85+0.04
0.35+0.25
0.46+0.12
0.47#0.29
Multi-Level
0.1620.03
0.12+0.03
0.13+0.02
0.38+0.05
0.31+0.02
0.3740.08
hard_expert
0.65+0.01
0.67+0.04
0.7*0.03
hard_mixed
0.32+0.14
0.3+0.1
0.43+0.04
gorm_genneal
0.34+0.05
0.29+0.09
0.34+0.02
hard_general
0.28+0.03
0.28+0.04
0.32+0.01
multi hero ghero_general
0.17+0.03
0.14
0.18+0.06
Generalization
hard_ _urli_ Seulti_errrr_ee
0.16+0.005
0.17+0.02
0.1920.02
norm_muee Becill_ppp__eeeee
0.21#0.01
0.14+0.04
0.14+0.03
hard pord_mmlti_ove htt__mmtttpppopeeee oppoo__eprrraalll
0.09+0.06
0.09+0.02
0.1+0.02
norm_ ciin_mmlti_ler level
0.43+0.09
0.34+0.0 044
0.45+0.02
hard_multh level
level multi hero
0.38+0.08
0.29+0.07
0.35+0.04
Multi-Task
0.57=0.07
0.3
0.56+0.13
norm _ullti_uliiii oppo norm multi hero oppo
0.0920.04
0.07+0.03
0.09#0.02
0.3+0.04
0.26+0.07
0.3+0.03
norm stupid partuer
0.11+0.15
0.24+0.17
0.55+0.05
Heterogeneous
norm norm_expert norm partner
0,36+0.09
0.57+0.04
0.72+0.07
0.49+0.2
0.32+0.03
0.66+0.05
gain gold medium gain gold_expedrrrlle eeppert gain gold mixed
0.13+0.01
0.12+0.01
0.13#0.02
Task
1.01=0003
1+0.01
1.02+0.01
0.64#0.29
0.41+0.1
0.58+0.17
H
Additional Discussion
H.1
The significance of our design factors in the context of offline reinforcement learning
Task difficulty: Intuitively, the level of difficulty in the environment significantly impacts the performance algorithms. However, previous researches only utilized one set of datasets with uniform level of difficulty in the environment. Providing datasets with diverse difficulty can not only more comprehensively evaluate the ability of offline algorithms but also be more suited for real- world tasks like HoK, which are characterized by diverse levels of difficulty.
22

---

Combining offline reinforcement learning with multi-task learning enables efficient use of limited data. Sharing knowledge[[] and representations across tasks enhances data efficiency, leading to more general and robust feature learning. Besides, multi-task learning facilitates knowledge transfer between tasks. Leveraging shared parameters and representations accelerates learning for the target task in offline reinforcement learning, benefiting from related tasks' knowledge.
Multi-task:
Firstly, in offline RL, learning is based fixed dataset collected from previous experiences. This dataset might not cover all possible scenarios, so the learned policy needs to generalize well to new, unseen situations to perform effectively. Secondly, real-world environments are often complex and diverse. A policy only limited to the dataset without generalizing would likely fail when facing even slightly different conditions. Generalization ensures the policy's adaptability to various situations in the real-world scenarios.
Generalization:
Heterogeneous Teammate: Heterogeneous teammates crucial research direction in Multi-Agem Reinforcement Learning (MARL). In practical scenarios such as HoK or other multi-agent systems, players typically possess varying capacities. Consequently, the datasets collected from real-world scenarios consist of heterogeneous teammate data, necessitating the need for corresponding research in the offline MARL domain.
H.2 From the perspective of the Honor of Kings game, why offline reinforcement learning is necessary and what potential limitations exist when compared to online reinforcement learning?
Training agents for the Honor of Kings game using offline reinforcement learning (RL) offers several advantages, including reduced training time, lower computation resource requirements, and better utilization of existing data resources. We compare the computational and time costs of online RL and offline RL in Table 18 It evident that training an online agent from scratch reach specific levels (level_5 for HoK I and level_7 for HoK3v3) requires thousands of CPU cores and dozens of hours. On the other hand, training offline agents to reach same levels only requires few CPUs and shorter training time using pre-collected datasets. Additionally, there is a wealth of previously collected battle data that can be used for training offline RL agents. However, compared online RL, it is important to note that offline RL in the HoK game heavily relies on large amounts of high-level battle data to train expert-level agents, which may be a potential limitation.
Table 18: The comparison of the computational and time costs between online RL and offline RL.
Online/Offline CPU cores
GPU cores
Performance
Training time (hours)
HoKIvl (online)
4000
2
Ivl_level_ Ivl level 5
60h
HoKlvl (offline)
2
I
9h
HoK3v3 (online)
1000
3v3 level 7
97h
HoK3v3 (offline)
4
1
3v3 [evee_ee_
20h
23

---

40
20
Epicodo Ruturn
0
Egicods
20
20
40
40
.60
-60
nomm
nar,, medium
norm, expert
mixedddd
madium
(a) norm_level
(b) hard_level
40
20
Episodo Rusurn
40
-50
-60
norm, general
hand general
nomm commmulli_Mae level
hard mulii Jevel
(c) level general
(d) multi level
100
Eptoods fetum
Epicode hetur
20
500
.100
nom Com_mult_ain aly
norm commmultt.opp oppo
norm, www.muttLslly.oppo
(c) multi multi_hero__a_ _epo__pppoppo
(f) heterogeneous
40
Sodod Racurn
10
0
.00
gain_ medium
gain_gold,_o expert
gain hwwiigold__mmii mixed
gg) sub lask: gain gold
Figure 4: Violin diagrams of all datasets in HvK3v3.
24

---

Move Attack Skills Return Home
What
Move Offset Skill Offset (Ax, Ay)
(Ax, Ay)
How
Target Unit
Who
Figure 5: Action space in HoKlv
38
cd <root_ proot_path>>>>
sample example sh off.ine..mmmleocccpp/sssstttpppp sh <args>
train example
python offline hraineetrain/traih..p. <-<agg>>>>>
evaluate example
python ofaline_eval / py --<args>
Listing 1: APIs example
25


Reviews:
[{"review_id": "hRwBPP6W2T", "reviewer": "Reviewer_D3sf", "summary": "", "questions": "", "limitations": "I am not opposed to encouraging sample-efficient learning, but 1000 trajectories seems like a relatively small amount of data for such complex tasks. By comparison, it seems like the StarCraft II dataset you mention in the paper has 2.8 million total episodes. I understand that this is an area you seem to be working on, but I think it should at least be stated up-front in the main body of the paper. The results from your baselines look reasonable for a dataset in general, as there is reasonable success but also room for future work using your dataset to improve, but I wonder partially how much of that is due to the dataset size. (The problem could alternatively be framed and justified as encouraging sample-efficiency, but that is not currently being done.)", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The idea seems sound; MOBA games seem like a reasonable source of complex, interesting behaviors for demonstrating new methods on.\n\n1. The datasets presented span a variety of potential research directions (generalization, multi-agent work, etc), making their benchmark widely useful. Additionally, it seems like their method could be readily extended to even more settings, and harder tasks.\n\n1. They provide initial results for a number of baseline methods, validating the design and utility of their framework, as well as providing a base on which others can build.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3p2Xjeo6xo", "reviewer": "Reviewer_KDwt", "summary": "", "questions": "", "limitations": "Have not been discussed", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "Really nice benchmark for someone developing algorithms for honor of kings. Therefore, this is a valuable benchmark for people developing for that specific game", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "rEQtKCZNxS", "reviewer": "Reviewer_DxzA", "summary": "", "questions": "", "limitations": "The authors should consider discussing the potential limitations of their work. They could explore how and why the proposed benchmark may be useful for researchers in the field. This discussion could be divided into two parts:\n\n1. From the perspective of offline reinforcement learning, the authors claim that the proposed benchmark is more complex than existing benchmarks like D4RL and RL Unplugged. However, to save computation, only simplified sub-tasks such as Honor of Kings Arena, Destroy Turret, and Gain Gold are implemented in this benchmark. The authors may provide a rationale for why these simplified tasks are sufficient and what potential limitations may arise from not including more complex scenarios.\n\n2. From the perspective of the Honor of Kings game, the authors may explain why offline reinforcement learning is necessary and what potential limitations exist when compared to online reinforcement learning. This discussion would help readers understand the context in which the proposed benchmark is relevant and how it can contribute to advancing the field.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The introduction of a benchmark for offline reinforcement learning and multi-agent reinforcement learning (MARL) based on the popular game Honor of Kings is a critical contribution for researchers in this community. This benchmark has the potential to accelerate the development of this field and aid in the evaluation of algorithms.\n\n2. The data, code, and documentation provided in this work are comprehensive, making it easy for interested readers to get started with these materials. This level of detail is beneficial for researchers looking to build upon the proposed benchmark and apply it to their own work.\n\n3. The paper is generally well-written and easy to follow, presenting the problem and proposed solution in a clear and concise manner. The organization of the paper and the use of appropriate technical language make it accessible to a wide range of readers, including both experts and newcomers to the field.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work presents a novel set of pre-collected datasets for use in Offline RL and Multi-Agent Offline RL, taken from demonstrations for Honour of Kings, the world's most widely played Mobile Online Battle Arena (MOBA).  Along with presenting the datasets, which are available for certain tasks, the other contributions of the work include the benchmarking of various offline RL and offline MARL algorithms. 

**Perceived Strengths:**
* The reviewers generally agreed on the utility of such a dataset in advancing research into Offline RL and MultiAgent Offline RL.  The complexity of the game and the MOBA aspect of the game were highlighted as reasonable source of complex, interesting behaviours. 
* The code, data and documentation of the submission were commended for being ‘comprehensive’ and appear to be high quality.  

**Perceived Weaknesses:**

* One reviewer raised concerns around a lack of diversity in the benchmark, given that the environment always revolves around one game.  Authors responded arguing that, despite being only one game, HoK poses offers significant of complexity and diversity to be a challenge in multiple areas of research including generalization, multitask learning.
* Some criticisms were made of the paper around clarity, and explaining motivation, which have been addressed in comments or in the paper.

**AC View**

Following on from the reviews and discussions, I recommend acceptance of this work on account of the accepted utility of this work to researchers in offline RL and offline MARL.  This decision was made easier thanks to the reviews and substantial responses made over the course of the review period, and the changes to the paper which I think greatly improved the clarity. 

While I share concerns that 1000 trajectories for each task may be too few for such complex tasks, I believe taken as a whole, the dataset is convincing in offering opportunities to research learning complex (multi-agent) behaviours from data in a widely-popular 'real world' environment.  I do not share the opinion of Reviewer KDwt, that such a single environment does not contain enough diversity for relevant RL research and agree with Reviewer D3sf that the different datasets span a variety of compelling research directions, with ample scope for expansion.
Decision: Accept (Poster)