SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We have developed six experiments: "norm_general" and "hard_general" for level generalization, "norm_hero_general" and "hard_hero_general" for hero generalization, and "norm_oppo_general" and "hard_oppo_general" for opponent generalization, for HoK1v1 and HoK3v3, respectively. Among them, the first two experiments, "norm_general" and "hard_general," have their corresponding datasets, and we train models on these datasets. The latter four experiments do not require extra datasets because we directly use the existing models that have already been trained using other datasets. For more details on the design of generalization, please refer to the Appendix C.
Heterogeneous Teammate
Heterogeneous teammate is a crucial research direction in MARL [32, 17]. In the practical scenarios of HoK, the capacity of each player is generally different, making it naturally suitable for investigating the challenges associated with heterogeneous teammate. In order to mimic real-world scenarios and facilitate research on heterogeneous teammate challenges, we design two datasets in HoK3v3: norm_stupidpartner and norm_expertpartner. These datasets were collected in a standard manner, with the exception that one random hero in each team is controlled by a model with a relatively low/high level of expertise, while the remaining heroes are controlled by the regular model.
Sub-Task
As introduced in Sec 4.2.3, we designed several practical and meaningful sub-tasks to provide diverse scenarios based on HoK. Based on these sub-tasks, we proposed diverse datasets to support Offline RL research similar to the design of previous studies [6, 12].
Table 1: Details of datasets in HoK1v1 game mode
Factors	Datasets/Experiments	Capacity	Heroes	Oppoheroes	Win_rate	Levels
Multi-Difficulty	norm_poor	1000	default	default	12%	1
norm_medium	1000	default	default	50%	1
norm_expert	1000	default	default	88%	1
norm MIXED	1000	default	default	50%	1
hard_poor	1000	default	default	6%	5
hard_medium	1000	default	default	50%	5
hard_expert	1000	default	default	84%	5
hard MIXED	1000	default	default	45%	5
Generalization	hard_general	1000	default	default	90%	5
norm_general	1000	default	default	46%	1
norm_hero_general	-	multi_hero	default	-	1
hard_hero_general	-	multi_hero	default	-	5
norm_oppo_general	-	default	multi_hero	-	1
hard_oppo_general	-	default	multi_hero	-	5
Multi-Task	norm-multi_level	1000	default	default	50%	1
hard-multi_level	1000	default	default	50%	5
norm-multi_hero	1000	multi_hero	default	23%	1
norm-multi_oppo	1000	default	multi_hero	77%	1
norm-multi_hero_oppo	1000	multi_hero	multi_hero	50%	1
4.3.1 Datasets Details
Table 1, Table 2 and Table 3 presents the details of our proposed datasets. All the datasets are sampled using checkpoints with different levels as introduced in Sec. 4.1.1. Typically, each dataset consists of 1000 trajectories, except for the sub-task datasets, which contain 100 trajectories. The default heroes chosen for both camps are luban with Summoner Spells set to frenzy in HoK1v1 and  $\{\{zhaoyun\}$ ,  $\{diaochan\}, \{liyuanfang\}\}$  with Summoner Spells assigned as  $\{\{smite\}, \{purify\}, \{purify\}\}$  based on their respective roles in HoK3v3. However, in specific scenarios such as Generalization or Multi-Task settings, we employ a random selection of heroes from a predefined set, multihero. For the HoK1v1 mode, the set comprises five heroes,  $\{luban, direnjie, houyi, makeboluo, gongsunli\}$ . In HoK3v3, the set consists six heroes, with two heroes assigned to each role, namely  $\{\{zhaoyun, zhongwuyan\}, \{diaochan, zhugeliang\}, \{liyuanfang, sunshangxiang\}\}$ . The win rate of the behavior policy is recorded in the column labeled Win_rate for reference. The column labeled Levels denotes the levels of opponents used for evaluation. More details of the datasets are presented in Appendix C.
Table 2: Details of datasets in HoK3v3 game mode
Factors	Datasets/Experiments	Capacity	Heroes	Oppoheroes	Win_rate	Levels
Multi-Difficulty	norm_poor	1000	default	default	16%	1
norm_medium	1000	default	default	50%	1
norm_expert	1000	default	default	82%	1
norm MIXED	1000	default	default	49%	1
hard_poor	1000	default	default	18%	7
hard_medium	1000	default	default	50%	7
hard_expert	1000	default	default	83%	7
hard MIXED	1000	default	default	51%	7
Generalization	hard_general	1000	default	default	94%	8
norm_general	1000	default	default	57%	5
norm_hero_general	-	multi_hero	default	-	1
hard_hero_general	-	multi_hero	default	-	7
norm_oppo_general	-	default	multi_hero	-	1
hard_oppo_general	-	default	multi_hero	-	7
Multi-Task	norm-multi_level	1000	default	default	50%	1
hard-multi_level	1000	default	default	50%	7
norm-multi_hero	1000	multi_hero	default	74%	1
norm-multi_oppo	1000	default	multi_hero	26%	1
norm-multi_hero_oppo	1000	multi_hero	multi_hero	50%	1
Heterogeneous	norm_stupidpartner	1000	default	default	50%	1
norm_expertpartner	1000	default	default	50%	1
norm MIXEDpartner	1000	default	default	50%	1
Table 3: Details of datasets in Sub-Tasks
Sub-Task	Datasets/Experiments	Capacity	Heroes	Oppoheroes	Average Score	Levels
Destroy Turret	destroy_turret_medium	100	default	no	0.55	medium
destroy_turret_expert	100	default	no	1.00	expert
destroy_turret MIXED	100	default	no	0.73	-
Gain Gold	gain_gold_medium	100	default	no	0.13	medium
gain GOLD_expert	100	default	no	1.04	expert
gain GOLD MIXED	100	default	no	0.58	-
Based on our framework, we reproduce various Offline RL and Offline MARL algorithms. Besides, we fully validate and compare these baselines on our datasets. The results are presented in the form of test winning rate. Each algorithm is run for three random seeds, and we report the mean performance with standard deviation. The performance of behaviour policies is presented in Appendix C. Details of the implementations and experimental results can be referenced in Appendix G.
5.1Baselines
5.1.1 HoK1v1
The Offline RL baseline algorithms we implement are briefly introduced below: BC: Behavior cloning. TD3+BC [7]: One of the state-of-the-art single agent offline algorithm, simply adding the BC term to TD3 [8]. CQL [19]: Conservative Q-Learning conducts conservative value iteration by adding a regularizer to the critic loss. IQL [16]: Implicit Q-Learning leverages upper expectile value function to learn Q-function and extracts policy via advantage-weighted behavioral cloning.
The structured action space in HoK is similar to the joint action space in multi-agent settings, which inspires us to resort to the design in MARL methods. We propose a novel baseline algorithm, named QMIX+CQL. Specifically, we import QMIX algorithm from the MARL literature [29] to tackle the structured action space by regarding each head of the action space as a single agent and incorporate CQL regularizer term into local Q-function in QMIX for offline learning.
5.1.2 HoK3v3
The Offline MARL baseline algorithms are briefly introduced below: IND+BC: Behavior cloning with independent learning paradigm. IND+CQL: Adopt an independent learning paradigm for multi-agent settings, using conservative Q-learning [19]. COMM+CQL: Incorporate inter-agent communication based on IND+CQL. IND+ICQ [43]: Implicit Constraint Q-learning with independent learning paradigm, which only uses insample data for value estimation to alleviate the extrapolation error. MAICQ [43]: Multi-agent version of implicit constraint Q-learning by decomposed multi-agent joint-policy under implicit constraint with CTDE paradigm. OMAR [28]: Using zeroth-order optimization for better coordination among agents' policies, based on independent CQL.
Table 4: Averaged test winning rate or normalized score (Sub-Task) of baselines in HoK1v1 game mode.
Factors	Datasets	BC	CQL	QMIX+CQL	IQL	TD3+BC
Multi-Difficulty	norm_poor	0.08±0.02	0.06±0.01	0.08±0.02	0.07±0.01	0.0±0.0
norm_medium	0.33±0.01	0.32±0.01	0.31±0.03	0.32±0.01	0.01±0.01
norm_expert	0.64±0.01	0.58±0.03	0.67±0.01	0.62±0.02	0.03±0.01
norm MIXED	0.17±0.01	0.23±0.04	0.20±0.01	0.25±0.01	0.01±0.01
hard_poor	0.01±0.01	0.01±0.01	0.01±0.01	0.01±0.00	0.00±0.00
hard_medium	0.13±0.01	0.11±0.01	0.20±0.01	0.12±0.02	0.00±0.00
hard_expert	0.33±0.01	0.30±0.01	0.44±0.05	0.34±0.04	0.00±0.00
hard_mixED	0.05±0.3	0.02±0.01	0.08±0.01	0.06±0.01	0.01±0.01
Generalization	norm_general	0.19±0.01	0.20±0.04	0.32±0.03	0.18±0.01	0.02±0.02
hard_general	0.04±0.01	0.03±0.01	0.08±0.02	0.02±0.01	0.00±0.00
norm_hero_general	0.06±0.01	0.06±0.01	0.08±0.01	0.07±0.01	0.00±0.00
hard_hero_general	0.03±0.01	0.03±0.01	0.04±0.01	0.06±0.01	0.00±0.00
norm_oppo_general	0.58±0.03	0.52±0.04	0.42±0.22	0.51±0.07	0.12±0.01
hard_oppo_general	0.15±0.02	0.12±0.03	0.23±0.04	0.14±0.03	0.01±0.01
Multi-Task	normmulti_level	0.32±0.03	0.25±0.03	0.41±0.02	0.30±0.02	0.02±0.01
hard-multi_level	0.08±0.02	0.06±0.01	0.16±0.03	0.08±0.02	0.00±0.00
norm-multi_hero	0.08±0.01	0.07±0.02	0.11±0.01	0.06±0.01	0.00±0.00
norm-multi_oppo	0.59±0.02	0.55±0.03	0.65±0.02	0.60±0.05	0.10±0.02
norm-multi_hero_oppo	0.26±0.01	0.21±0.02	0.32±0.03	0.28±0.05	0.03±0.01
Sub-Task	destroy_turret_medium	0.61±0.06	0.63±0.01	0.61±0.03	0.60±0.02	0.67±0.03
destroy_turret_expert	0.94±0.02	0.94±0.02	0.92±0.05	0.95±0.01	0.57±0.13
destroy_turret_mixED	0.88±0.04	0.87±0.03	0.89±0.02	0.89±0.04	0.82±0.03
We have validated the offline RL and offline MARL baselines on our datasets and aggregated the results in Table 4 and Table 5.
- Baselines Comparison: As indicated in Table 4, QMIX+CQL exhibits superior performance in comparison to other approaches, implying that the integration of MARL methods may be a suitable choice for environments with a structured action space. Moreover, in HoK3v3, IND+ICQ exhibits the highest performance across most datasets, except for the Heterogeneous datasets. Conversely, algorithms based on TD3, namely TD3+BC and OMAR, yield poor results.
- Multi-Difficulty: The baseline performance exhibits a significant decrease on the hard-level datasets compared with norm-level datasets, highlighting the limitations of current offline methods in addressing challenging tasks with discrete action space.
- Generalization: The disparities between training and evaluation in Generalization settings impede the achievement of desirable performance, indicating the inadequacy of current methods' generalization ability.
- Multi-Task: Training models on Multi-Task datasets results in a substantial performance enhancement compared to generalization settings. However, none of these models have been able to exceed the performance achieved by the behavior policy, underscoring the need for further research into the direct application of offline methods to multiple tasks.
Table 5: Averaged test winning rate or normalized score (Sub-Task) of baselines in HoK3v3 game mode.
Factors	Datasets	IND+BC	COMM+CQL	IND+CQL	IND+ICQ	MAICQ	OMAR
Multi-Difficulty	norm_poor	0.1±0.01	0.09±0.02	0.03±0.01	0.12±0.02	0.12±0.04	0.02±0.01
norm_medium	0.48±0.01	0.47±0.04	0.4±0.03	0.45±0.01	0.38±0.16	0.23±0.03
norm_expert	0.52±0.03	0.76±0.13	0.84±0.06	0.65±0.12	0.61±0.09	0.39±0.16
norm_mix	0.35±0.25	0.48±0.12	0.46±0.12	0.44±0.19	0.24±0.16	0.17±0.2
hard_poor	0.16±0.03	0.11±0.04	0.12±0.03	0.17±0.02	0.12±0.03	0.08±0.04
hard_medium	0.38±0.05	0.35±0.03	0.31±0.02	0.4±0.08	0.2±0.06	0.23±0.07
hard_expert	0.65±0.01	0.66±0.05	0.67±0.04	0.67±0.02	0.52±0.1	0.35±0.17
hard_mix	0.32±0.14	0.34±0.11	0.3±0.1	0.34±0.08	0.23±0.08	0.16±0.17
Generalization	norm_general	0.34±0.05	0.35±0.04	0.29±0.09	0.37±0.04	0.29±0.06	0.09±0.1
hard_general	0.28±0.03	0.3±0.05	0.28±0.04	0.31±0.09	0.14±0.06	0.13±0.04
norm HERO_general	0.17±0.03	0.13±0.02	0.14±0.04	0.2±0.09	0.2±0.06	0.13±0.04
hard HERO_general	0.16±0.05	0.19±0.05	0.17±0.02	0.17±0.02	0.07±0.05	0.08±0.03
norm_oppo_general	0.21±0.01	0.14±0.03	0.14±0.04	0.18±0.06	0.13±0.07	0.12±0.03
hard_oppo_general	0.09±0.06	0.08±0.02	0.09±0.02	0.08±0.04	0.04±0.02	0.04±0.01
Multi-Task	normmulti_level	0.43±0.09	0.36±0.02	0.34±0.04	0.44±0.02	0.38±0.11	0.22±0.07
hardmulti_level	0.38±0.08	0.33±0.08	0.29±0.07	0.37±0.05	0.27±0.05	0.2±0.01
normmulti HERO	0.57±0.07	0.31±0.2	0.3±0.07	0.59±0.05	0.51±0.17	0.39±0.06
normmulti_oppo	0.09±0.04	0.08±0.05	0.07±0.03	0.12±0.04	0.07±0.03	0.02±0.01
normmulti HERO_oppo	0.3±0.04	0.23±0.1	0.26±0.07	0.31±0.07	0.26±0.03	0.07±0.02
Heterogeneous	norm_stupidpartner	0.11±0.15	0.33±0.06	0.24±0.17	0.22±0.14	0.16±0.09	0.08±0.05
norm_expertpartner	0.36±0.09	0.52±0.1	0.57±0.04	0.55±0.22	0.31±0.15	0.07±0.06
norm_mix Partner	0.49±0.2	0.59±0.19	0.32±0.03	0.42±0.38	0.17±0.27	0.15±0.04
Sub-Task	gain GOLD_medium	0.13±0.01	0.12±0.01	0.12±0.01	0.15±0.01	0.13±0.03	0.14±0.01
gain GOLD_expert	1.01±0.03	0.98±0.02	1.00±0.01	1.03±0.01	0.98±0.08	0.79±0.06
gain GOLD MIXED	0.64±0.29	0.41±0.21	0.41±0.1	0.46±0.16	0.25±0.23	0.13±0.04
- Heterogeneous: As expected, the presence of a low-ability partner can disrupt cooperation and hinder offline learning on the stupidpartner datasets, whereas an expert partner has the opposite effect, highlighting the limitations of existing research on heterogeneous offline MARL.
- Sub-Task: The offline baselines exhibit robust performance in the Sub-Task at a low training cost. Additionally, BC demonstrates a competitive capability as well.
- Ablations of learning paradigms: We conduct ablation experiments to investigate the impact of communication and the CTDE paradigm. Specifically, from the comparison of COMM-CQL and IND-CQL, we can reveal that incorporating communication generally results in better performance due to the promotion of cooperation. Surprisingly, we found that the independent paradigm (IND-ICQ) outperformed the CTDE paradigm (MAICQ), which may be attributed to the challenges in the CTDE paradigm associated with credit assignment of agents with distinct rewards and roles.
In this paper, taking into account the limitations of existing offline RL datasets about practical applications, we introduce Hokoff, based on Honor of Kings, a well-known MOBA game that offers a high level of complexity for simulating real-world scenarios. We present a comprehensive framework for conducting research in offline RL and release a diverse and extensive collection of datasets, incorporating various levels of difficulty and a range of research factors. Moreover, the chosen tasks for dataset collection not only cater to Offline RL but also serve the purpose of offline MARL. We replicate multiple offline RL and offline MARL algorithms and thoroughly validate these baselines on our datasets. The obtained results highlight the shortcomings of existing Offline RL methods, underscoring the necessity for further research in areas such as challenging task settings, generalization capabilities, and multi-task learning. All components, including the framework, datasets, and baseline implementations, discussed in this paper are fully open-source.
[1] Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
[40] Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., and Goh, H. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021.
[41] Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[42] Xu, Z., Lyu, Y., Pan, Q., Hu, J., Zhao, C., and Liu, S. Multi-vehicle flocking control with deep deterministic policy gradient method. In International Conference on Control and Automation (ICCA), 2018.
[43] Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021.
[44] Ye, D., Chen, G., Zhang, W., Chen, S., Yuan, B., Liu, B., Chen, J., Liu, Z., Qiu, F., Yu, H., et al. Towards playing full moba games with deep reinforcement learning. arXiv preprint arXiv:2011.12692, 2020.
[45] Ye, D., Liu, Z., Sun, M., Shi, B., Zhao, P., Wu, H., Yu, H., Yang, S., Wu, X., Guo, Q., et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6672-6679, 2020.
[46] Yoo, M., Cho, S., and Woo, H. Skills regularized task decomposition for multi-task offline reinforcement learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 37432-37444. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/f376f5cff6f6ec6364aea7a46ab49574-Paper-Conference.pdf.
[47] Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Levine, S., and Finn, C. Conservative data sharing for multi-task offline reinforcement learning, 2021.
[48] Zanette, A., Wainwright, M. J., and Brunskill, E. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[49] Zhang, C. and Lesser, V. R. Coordinating multi-agent reinforcement learning with limited communication. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2013.
A Author Statement
The authors of this work would like to state that we bear full responsibility for any potential violation of rights, including copyright infringement or unauthorized use of data. We affirm our commitment to conducting this research in accordance with ethical guidelines and legal requirements.
We further guarantee that we will ensure access to the data<sup>4</sup> and the framework code<sup>5</sup> used in this study, making them available to interested researchers for verification and replication purposes. Additionally, we are committed to providing the necessary maintenance and support to ensure the longevity and accessibility of the data. For datasets, we have plans to consistently offer more datasets in the future. These datasets will include larger sizes for larger models, higher levels for expert agents, and novel design factors for other research directions. Updating our datasets is an ongoing and long-term effort, and we welcome contributions from the community. Regarding benchmarks, we will actively monitor the latest state-of-the-art (SOTA) algorithms in the offline RL domain and integrate them into our benchmarks. Additionally, we will develop new algorithms within the benchmarks based on existing datasets and baselines. This ensures that our benchmarks remain up-to-date and reflect the advancements in offline RL research.
Should any concerns or inquiries arise regarding the contents of this work or the associated data, we encourage readers and fellow researchers to contact us directly. We are dedicated to addressing any issues promptly and transparently to uphold the integrity of our research.
B Limitations and Future Works
In our future endeavors, we plan to integrate our framework with a large-scale deep reinforcement learning platform namely KaiwuDRL, specifically designed to support Honor of Kings. By doing so, we will gain access to greater computational resources, enabling us to delve deeper into our current research endeavors and expand our investigations.
C Additional Datasets Details
C.1 HoK1v1
In the Generalization category, "norm_general" and "hard_general," have their corresponding datasets. For example, to sample the "norm_general" dataset, we let the level-1 model fight with level-0, level-2, and level-4 models. However, during the test stage, we assess the generalization capabilities of the trained model by letting it fight against the level-1 model. Details about how we sample the generalization datasets can be referred to Table. 6. The latter four experiments do not require extra datasets. For example, in the "norm_hero_general" experiment, we directly use the model trained on the "norm_medium" dataset and let the model control different heroes. This is possible because the "norm_medium" dataset only contains the fixed default hero "luban." Therefore, we use the model trained on this dataset to test its generalization ability at controlling different heroes.
In the Sub-Task: Destroy Turret category presented in Table 3, there are three datasets sampled, each consisting of 100 trajectories. Notably, these datasets lack an opponent hero, making them simpler in nature. This design choice allows for broad applicability, diversity, and cost-effectiveness in research endeavors.
The primary objective in the Sub-Task: Destroy Turret scenarios is to efficiently dismantle the enemy's turret and crystal, with the enemy hero removed. Consequently, we adopt the number of game frames elapsed from the start of the game until the crystal's destruction as our evaluation protocol. Equation 1 outlines the scoring methodology employed, following a similar approach as presented in [6]. The score is normalized by two factors: random_frame_length, set to 2880, and expert_frame_length, set to 1812. A higher score is achieved by minimizing the time required to destroy the crystal.
In addition, we have generated violin charts to represent the distribution of episode returns in each dataset as shown in Fig. 3. We calculate episode returns using the formula  $R = \sum_{t=0}^{T} \gamma^t r^t$ , where
4https://sites.google.com/view/hok-offline
<sup>5</sup>https://github.com/tencent-ailab/hokoff
gamma is set to 1.0 to showcase the overall rewards obtained throughout an entire episode. For the Sub-Task: Destroy Turret datasets of HoK1v1, we have normalized the scores based on Equation 1. The violin charts demonstrate the diverse distribution of episode returns within our datasets.
$$
n o r m a l i z e d \_ s u b \_ t a s k \_ s c o r e = \frac {r a n d o m \_ f r a m e \_ l e n g t h - f r a m e \_ l e n g t h}{r a n d o m \_ f r a m e \_ l e n g t h - e x p e r t \_ f r a m e \_ l e n g t h} \tag {1}
$$
(a) norm_level
(b) hard_level
(c) level_general
(d) multi_level
(e) multi_hero_oppo
Figure 3: Violin diagrams of all datasets in HoK1v1.
(f) sub_task: destroy_turret
C.2 HoK3v3
The design of datasets and experiments pertaining to the concept of Generalization closely resembles that of the HoK1v1.
We introduce a sub-task called "Gain Gold" that builds upon the HoK3v3 game. In this modified version, we remove opponents and redefine the primary objective to focus on collecting gold within a limited number of time steps. This transforms the original competitive task into a resource collection task. Specifically, we set a maximum episode length of 8000 frames, and the controlled heroes are required to efficiently gather gold by killing monsters or creeps. As demonstrated in Table 3, we generate three datasets based on this sub-task, each consisting of 100 trajectories. The gain_gold_medium dataset is collected by a model with moderate performance, averaging 5904 gold collected. While, the gain_gold_expert dataset is obtained from a model with expert performance, averaging 12271 gold collected. Lastly, the gain_gold MIXed dataset combines the data from the previous two datasets equally. The scores are normalized based on Equation 2, where the random_gain_gold is 5000 and expert_gain_gold is 12000.
We have also generated violin charts in HoK3v3 to represent the distribution of episode returns in each dataset as shown in Fig. 4. The plot method used is similar to that in HoK1v1, with the exception of not using normalized scores in the Sub-Task: Gain Gold.
$$
n o r m a l i z e d \_ s u b \_ t a k \_ s c o r e = \frac {\text {g a i n} _ {\text {g o l d}} - \text {r a n d o m} }{\text {e x p e r t}  - \text {r a n d o m} } \tag {2}
$$
Table 6: Details of sampling generalization datasets.
Sampling and Training	Testing
Environments	Datasets	Controlled side model	Opponent side model	Opponent side model
HoK1v1	norm_general	level-1	level-0,2,4	level-1
hard_general	level-5	level-0,2,4	level-5
HoK3v3	norm_general	level-5	level-1,4,7	level-5
hard_general	level-8	level-1,4,7	level-8
D Environment Details
D.1 Honor of Kings Arena
For a more detailed account of the game settings, please refer to the original paper [38] and its documentation<sup>6</sup> of Honor of Kings Arena. In this context, we will only summarize the critical information that is relevant to the RL research.
- Observation Space