SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yun Qu\*, Boyuan Wang\*, Jianzhun Shao\*, Yuhang Jiang†, Chen Chen†, Zhenbin Ye‡, Lin Liu‡, Junfeng Yang‡, Lin Lai‡, Hongyang Qin§, Minwen Deng§, Juchao Zhuo§, Deheng Ye§, Qiang Fu§, Wei Yang§, Guang Yang‡, Lanxiao Huang‡, Xiangyang Ji†
†Tsinghua University, ‡Tencent Timi Studio, §Tencent AI Lab {qy22, wangby22, sjz18, jiangyh19} @ mails.tsinghua.edu.cn, chenchen.peach@gmail.com, {zhenbinye, lincliu, fengjunyang, linlai, hongyangqin, danierdeng, jojozhuo, dericye, leonfu, willyang, mikoyang, jackiehuang} @ Tencent.com, xyji@tsinghua.edu.cn
The advancement of Offline Reinforcement Learning (RL) and Offline Multi-Agent Reinforcement Learning (MARL) critically depends on the availability of high-quality, pre-collected offline datasets that represent real-world complexities and practical applications. However, existing datasets often fall short in their simplicity and lack of realism. To address this gap, we propose Hokoff, a comprehensive set of pre-collected datasets that covers both offline RL and offline MARL, accompanied by a robust framework, to facilitate further research. This data is derived from Honor of Kings, a recognized Multiplayer Online Battle Arena (MOBA) game known for its intricate nature, closely resembling real-life situations. Utilizing this framework, we benchmark a variety of offline RL and offline MARL algorithms. We also introduce a novel baseline algorithm tailored for the inherent hierarchical action space of the game. We reveal the incompetence of current offline RL approaches in handling task complexity, generalization and multi-task learning.
Online Reinforcement Learning (Online RL) relies on the interaction between the training policy and the environment for data collection and policy optimization [33, 18, 7]. However, this paradigm makes online RL unsuitable for certain real-world scenarios, such as robotics and autonomous driving [23, 18], as deploying untested policies to the environment can be costly and dangerous [19]. In contrast, Offline Reinforcement Learning (Offline RL) can learn satisfactory policies using a fixed dataset without the need for further interaction with the environment [23, 18, 7, 19]. This characteristic alleviates the aforementioned issue, making offline RL potentially more suitable for certain real-world scenarios compared to online RL [23].
The research on offline RL has attracted significant attention in recent years and has made substantial progress in both theoretical analysis and practical performance. The core challenge of offline RL is the value overestimation issue induced by distributional shift [9, 21]. Existing studies mitigate this problem by constraining the learning policy to closely resemble the behavior policy induced by the dataset [39, 18, 7], or adopting conservative value iteration [19, 16]. The success of offline RL can be largely attributed to the availability of widely-adopted open-access datasets, such as D4RL [6] and RL Unplugged [12]. These datasets offer standardized and diverse pre-collected data for the development of new algorithms, while also offering proper evaluation protocols that facilitate fair comparisons between different algorithms. However, despite their benefits, tasks contained in these datasets (such
* Authors contributed equally
as Atari 2600 [3] or Mujoco [35]) are often overly simplistic or purely academic, failing to simulate the complexity of real-world scenarios and lacking practical applications. Thus, there is a significant gap between offline RL research and its practical application in real-world settings. This disparity hinders the usefulness of offline RL in addressing real-world problems, and thus, it is indispensable to create datasets that reflect a realistic level of complexity and practicality in real-world applications.
Offline Multi-Agent Reinforcement Learning (Offline MARL) [28, 43] is gaining increasing attention due to its close relationship to real scenarios, such as games [31, 38], sensor networks [49] and autonomous vehicle control [42]. However, the lack of standardized datasets restricts the development 40 of offline MARL. Existing works only rely on self-made datasets, which hampers fairness and reproducibility. Moreover, the settings they focus on are typically limited to toy examples (e.g., Multi-agent Particle Environment [26]) or simplified versions of classic games (e.g., StarCraft Multi-Agent Challenge [31]), facing the same impractical issues encountered in offline RL. Thus, there is an urgent need for open-access datasets to further the progress of offline MARL.
The connections between offline RL and offline MARL are closely linked due to similar challenges pertaining to offline learning. While, offline MARL also introduces distinct algorithm development requirements, as it involves unique characteristics like multiple agents and intra-team cooperation. The current offline MARL algorithms [28, 43] are mainly adaptations of offline RL algorithms, necessitating the availability of standardized offline datasets that cater to both single-agent and multi-agent settings. Thus, to enhance versatility and practicality, it is crucial to propose datasets that encompass both single-agent settings and multi-agent settings.
In this paper, we present Hokoff, a suite of pre-collected datasets for both offline RL and offline MARL, along with a comprehensive framework for conducting corresponding research. Our paper makes several novel contributions, which are shown below:
- The tasks we adopt are based on one of the world's most popular Multiplayer Online Battle Arena (MOBA) games, Honor of Kings (HoK), which has over 100 million daily active players[38], ensuring the practicality of our datasets. The complexity of this environment dramatically surpasses those of its counterparts, demonstrating the potential for simulating real-world scenarios.
- We present an open-source, easy-to-use framework  ${}^{1}$  under Apache License V2.0. This framework includes comprehensive processes for offline RL (sampling, training, and evaluation), and some useful tools. Based on the framework, We release a rich and diverse set of datasets  ${}^{2}$  which are generated using a series of pre-trained models featuring distinct design factors. These datasets cater not only to offline RL but also offline MARL.
- Building on the framework, we reproduce various offline RL and offline MARL algorithms and propose a novel baseline algorithm tailored for the inherent hierarchical structured action space of Honor of Kings. We fully validate and compare these baselines on our datasets. The results indicate that current offline RL and offline MARL approaches are unable to effectively address complex tasks with discrete action space. Additionally, these methods exhibit shortcomings in terms of their generalization capabilities and their ability to facilitate multi-task learning.
2.1 Offline RL and Offline MARL
Offline RL gains significant attention in recent years, primarily due to the inherent difficulties of directly applying online RL algorithms to offline environments. The main hurdles encountered is the issue of erroneous value overestimation, which arises from the distributional shift between the dataset and the learning policy [9]. Theoretical studies have demonstrated that the overestimation issue can be alleviated by pessimism, which results in satisfactory performance even with imperfect data coverage [4, 5, 14, 20, 25, 30, 41, 48]. In practice, certain studies [1, 2, 40, 1, 2, 40] employ uncertainty-based methods to estimate Q-values pessimistically or to perform learning on pessimistic dynamic models by estimating the epistemic uncertainty of Q-values or dynamics. Some studies [18, 9, 39, 16, 19, 7] adopt behavior regularization-based approaches by imposing constraints on the
<sup>1</sup>https://github.com/tencent-ailab/hokoff
$^{2}$ https://sites.google.com/view/hok-offline
learned policy to align closely with the behavior policy, either explicitly or implicitly, which offers better computational efficiency and memory consumption compared to uncertainty-based methods.
Offline MARL, combining offline RL and MARL, has emerged in recent years to address safety and training efficiency concerns in practical multi-agent scenarios. Most studies in this domain adopt a multi-agent paradigm, such as independent learning [34] or centralized training with decentralized execution (CTDE) [26]. These investigations also incorporate offline methods, similar to those employed in single-agent settings, to mitigate distributional shift. Moreover, innovative treatments are introduced for cooperation, such as zeroth-order optimization in OMAR [28] or decomposing the joint-policy in MAICQ [43]. In addition, Jiang & Lu [13] specifically focuses on decentralized learning using BCQ [9], while Tseng et al. [36] regards offline MARL as a sequence modeling problem, utilizing supervised learning and knowledge distillation to tackle the challenges it presents.
2.2 Offline Datasets
The availability of large-scale pre-collected datasets has greatly facilitated the progress of deep supervised learning [11]. Offline RL, which is regarded as a bridge between RL and supervised learning, also requires learning policies from pre-collected datasets [6]. Therefore, high-quality pre-collected offline datasets play a significant role in the development of offline RL. To meet this demand, some datasets have been published and widely adopted. D4RL [6] is designed to address key challenges often faced in practical applications where datasets may have limited and biased distributions, incomplete observations, and suboptimal data. To tackle these issues, D4RL offers a range of datasets that enjoy these characteristics. Similarly, RL Unplugged [12] introduces a benchmark to evaluate and compare offline RL methods with various settings, such as partially or fully observable and continuous or discrete actions. These offline datasets play a significant role in offline RL research, and many previous works train and evaluate their methods based on these datasets [16, 2, 15, 28, 43].
However, both D4RL and RL Unplugged primarily focus on relatively simple tasks and lack high-dimensional, practical and multi-agent tasks that closely resemble real-world scenarios. StarCraft II Unplugged [27] introduces a benchmark for StarCraft II, a complex simulated environment with several practical properties. However, they only utilize a dataset derived from human replays, which lacks diversity in design for offline RL, and they did not evaluate existing offline RL methods. To address this research gap, we propose Hokoff, a benchmark based on HoK, which aims to provide diverse offline datasets for high-dimensional, practical tasks, and present a comprehensive evaluation of previous offline RL and offline MARL methods with a general, easy-to-use framework.
Honor of Kings (HoK) is one of the most popular MOBA games worldwide, boasting over 100 million daily active players [38]. The game involves two teams, each consisting of several players who have the option to select from a wide range of heroes with diverse roles and abilities. In the game, heroes are expected to eliminate enemy units, such as heroes, creeps, and turrets, to gain gold and experience. The primary objective is to destroy the enemies' turrets and crystal while defending their own. To succeed in MOBA games, players must learn how to choose the appropriate hero combination, master complex information processing and action control, plan for long-term decision-making, cooperate with allies, and balance multiple interests. The complex rules and properties of HoK make it be more in line with the complex decision-making behavior of human society. Thus, HoK has attracted numerous researchers interest [45, 44, 38, 37, 10].
The underlying system dynamic of HoK can be characterized by a Partially Observable Markov Decision Process (POMDP [33]), denoted by  $\mathcal{M} = (\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{P},r,\gamma ,d)$ . Due to the fog of war and private features, each agent has access to only local observations  $o$  rather than the global state  $s$ . Specifically, the agents are limited to perceiving information about game units within their field of view, as well as certain global features. Due to the intricate nature of control, the action space  $\mathcal{A}$  is organized in a hierarchically structured manner, rather than being flattened, which avoids the representation of millions of discretized actions. Randomness is added into the transition distribution  $\mathcal{P}$  in the form of critical hit rate. The reward  $r$  is decomposed into multi-head form and each hero's reward is a weighted sum of different reward items and is designed to be zero-sum. Details of observation space, action space and reward are presented in Appendix D.
(a) HoK1v1
(b) HoK3v3
Figure 1: (a) The Game replay user interface (UI) in HoK1v1. (b) The UI in HoK3v3. Important information and units of the game are highlighted using orange boxes.
4 Hokoff
This study is based on the HoK gaming environment, which encompasses both 1v1 and 3v3 maps. Our research proposes a comprehensive offline RL framework applicable to this gaming environment and utilizes it to generate diverse datasets. This section provides an introduction to the framework, game modes, datasets, and evaluation protocol employed in this study.
4.1 Framework
To enhance the usability of our Hokoff, we propose a reliable and comprehensive Offline RL framework that consists of three modules: sampling, training, and evaluation. This framework streamlines the process of sampling new datasets, developing and training baselines, and evaluating their performance. The sampling module provides a simple and unified program for sampling diverse datasets using any pre-trained checkpoints. There are several reasons why our framework excels in sampling. Firstly, diverse datasets at different levels of expertise can be sampled by leveraging Multi-Level Models as described in Sec. 4.1.1. Secondly, our framework employs parallel sampling techniques, ensuring efficient sampling of large and diverse datasets. Based on the training module, we have implemented various offline RL and offline MARL algorithms as baselines. Additionally, we consolidate crucial components and provide user-friendly APIs, facilitating researchers to effortlessly develop novel algorithms or innovative network architectures. The evaluation module enables the assessment of trained models from different algorithms, ensuring fair comparisons. Fig. 2 demonstrates the architecture of our framework and Appendix E provides an example of the APIs.
4.1.1 Multi-Level Models
To ensure a valid and unbiased comparison of the performance of distinct algorithms, it is crucial to establish appropriate evaluation protocols [6, 12]. One such effective evaluation protocol is the normalized score [6]. However, HoK is a zero-sum adjustable rewards MOBA game. The episode return in the game is heavily influenced by the opponents and game settings, and the objective is to win, which renders the use of return as a performance metric biased. Therefore, normalized score may not fully capture our requirements. Furthermore, similar to our situation, the evaluation protocol for SMAC [31], a competitive game, is based on win rate against a pre-programmed AI. Nonetheless, it is exceedingly challenging to create a built-in AI with human-like performance due to the complexity of MOBA games.
Inspired by prior works of HoK [38], we present Multi-Level Models for sampling and evaluating which contains multiple checkpoints with different level. Specifically, we have extracted several checkpoints from pre-trained dual-clip PPO [45, 44] models with varying levels determined by the outcome of the battle separately for HoK1v1 and HoK3v3. We adopt the win rate against different checkpoints as our evaluation protocols to assess the ability of models. Additionally, these models, with varying levels, can be utilized on both sides to sample diverse battle data. The capabilities of these models surpass those of rule-based AI and match the levels of different human players, thus making these evaluation protocols more suitable for comparing algorithmic performance with human-level performance and facilitating diverse and effortless sampling. The details of these Multi-Level Models are provided in the Appendix F.
Figure 2: The architecture of the framework. The sampling and evaluation modules should interact with the environment. Multi-Level Models are the foundation baseline models of these two modules, serving as opponents in the evaluation module and being on both sides in the sampling module, as described in Sec 4.1.1. The training module is responsible for training offline RL algorithms using fixed datasets and producing trained models for evaluation.
4.2 Game Modes
We have incorporated two game modes from HoK into our study, namely HoK1v1 [38] and HoK3v3. The environment code of HoK3v3 is integrated into the open-source HoK1v1 code<sup>3</sup>, following Apache License V2.0. These game modes differ in the number of agents involved and the underlying map used. Detailed information on each game mode is presented below.
4.2.1 Honor of Kings Arena
Honor of Kings Arena (HoK Arena or HoK1v1) is a 1v1 game mode where each player attempts to beat the other and destroy its opponent's crystal. Specifically, each player chooses a hero before the game starts and controls it during the whole game. There are a total of 20 heroes available for players to select, each possessing distinct skills that exert diverse effects on the game environment. The observation space is a continuous space consisting of 725 dimensions that contain partial observable information about the hero, opponent, and other game units. The action space is hierarchically structured and discretized, covering all possible actions of the hero in a hierarchical triplet form: (1) which action button to take; (2) who to target; and (3) how to act. Furthermore, the reward is a weighted sum of five categories: farming, kill-death-assist (KDA), damage, pushing, and win-lose. For a full description of this game mode, please refer to the Appendix D.1.
4.2.2 Honor of Kings 3v3 Arena
To further cater to the demand for Offline MARL, we adopt Honor of Kings 3v3 Arena (HoK3v3) as our experimental platform. HoK3v3 is a MOBA game, where each team comprises three heroes who collaborate to defeat their opponents. The basic rules and win conditions of HoK3v3 are similar to HoK1v1. However, the HoK3v3 map contains additional turrets on the middle road and features a new area called the "wilderness", inhabited by diverse monsters. Besides, collaboration is essential in HoK3v3, where players must select different heroes and fulfill distinct roles to work together more efficiently. For instance, one hero might focus on slaying monsters in the wilderness to earn gold and experience, while the other heroes engage in offensive tactics against the enemy heroes and game units. The design philosophies for observation space, action space, and reward are comparable to those used in HoK1v1. However, the level of complexity in HoK3v3 is significantly elevated. We provide a detailed description of the game mode in the Appendix D.2 for reference.
<sup>3</sup>https://github.com/tencent-ailab/hok_env
4.2.3 Subtasks
Both HoK1v1 and HoK3v3 are full MOBA games, featuring multi-camp competitions, which inherently pose challenges and limitations. Consequently, training on these game modes demands extensive training time and computational resources. However, HoK game comprises various subobjectives, allowing us to decompose the overall game into manageable subtasks. These subtasks can represent diverse scenarios and are suitable for evaluating various algorithms. In this study, we propose two specific noncompetitive subtasks as outlined below. It is worth noting that researchers can readily expand upon our framework to develop additional subtasks.
Destroy Turret: One of the key sub-objectives in HoK is to destroy the enemy's turrets as quickly as possible, to gain access to the enemy crystal. To train this specific skill, we have devised a subtask called Destroy Turret, which is based on HoK1v1. In this subtask, the focus is solely on destroying the enemy's turret and crystal as quickly as possible, and the enemy hero is removed.
Gain Gold: Gold is a critical resource in HoK that can be used to purchase equipment, which enhances the abilities of the heroes. Inspired by resource collection tasks from previous studies [22], we have designed a subtask called Gain Gold, which is based on HoK3v3, where the new objective is to collect golds in restricted time steps, and the enemy heroes are removed. As a multi-agent setting, it focuses on the cooperation or intra-team competition while avoiding inter-team competition.
To enhance the practical implications of our datasets, we have incorporated design factors that align with the real-world applications of both HoK and other relevant scenarios.
Multi-Difficulty
Intuitively, the level of difficulty in the environment significantly impacts the performance of algorithms. However, previous researches only utilized one set of datasets with a uniform level of difficulty in the environment, which is not appropriate for HoK, where the difficulty of the task can be substantially affected by the level of opponents. Therefore, to examine the effects of varying levels of difficulty in the environment, we propose several multi-difficulty datasets with different difficulty levels. Specifically, we develop two sets of datasets: norm and hard, which are categorized based on the opponent's level. Within each level, we propose four datasets according to diverse win rates against the opponent: poor, medium, expert and mixed. To elaborate, the poor/medium/expert dataset is generated by recording the battle trajectories of a relative lower/equal/higher level model compared to the opponent, and the mixed dataset is an equal mixture of the three datasets mentioned above.
Multi-Task
As a MOBA game, HoK features a diverse cast of heroes with distinct roles and skillsets. While the overall objective remains consistent throughout matches, the selection of heroes can significantly alter the nature of the task at hand. Consequently, HoK presents multi-task challenge which requires a single model to handle multiple tasks [47, 24, 46]. However, none of the current works provide uniform datasets for multi-task offline RL. To address this research gap, we propose a series of multi-task datasets based on the multi-task nature of HoK and evaluate the multi-task learning ability of current offline RL and offline MARL algorithms. Specifically, we define a hero pool with several heroes and randomly select heroes from it to sample data. Depending on whether the selected heroes are on the controlled side or the opponent side, we sample either the multihero or the multi OPPO dataset. In cases where both sides choose random heroes, we sample the multihero OPPO dataset.
Furthermore, as mentioned in the previous section, different levels of opponents naturally form multiple tasks with varying environmental difficulties. Thus, we propose several level-based multitask datasets by sampling data with randomly selected opponent levels. According to different difficulty levels, we have proposed two datasets, named norm-multi_level and hard-multi_level.
The unique gameplay mechanics of HoK, characterized by a diverse cast of heroes with distinct roles and skillsets, lend themselves well to multi-task and serve as an ideal testbed for evaluating the generality of models across a range of tasks. Building on the previous work [38] and taking into account the realities of human combat in HoK, we have identified three key challenges for generalization: hero generalization, opponent generalization, and level generalization.