Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain
Authors: Yiming Gao
Feiyu Liu
Liang Wang
Dehua Zheng
Zhenjie Lian
Weixuan Wang
Wenjin Yang
Siqin Li
Xianliang Wang
Wenhui Chen
Jing Dai
QIANG FU
Yang Wei
Lanxiao Huang
Wei Liu
Venue: ICLR 2024 poster (2024)
Keywords: human enhancement
human-agent collaboration
game playing
deep reinforcement learning
Primary area: reinforcement learning
TL;DR: We propose a human-centered modeling approach for collaborative agents that aims to effectively enhance the experience of humans.

Abstract: Existing game AI research mainly focuses on enhancing agents' abilities to win games, but this does not inherently make humans have a better experience when collaborating with these agents. For example, agents may dominate the collaboration and exhibit unintended or detrimental behaviors, leading to poor experiences for their human partners. In other words, most game AI agents are modeled in a "self-centered" manner. In this paper, we propose a "human-centered" modeling scheme for collaborative agents that aims to enhance the experience of humans. Specifically, we model the experience of humans as the goals they expect to achieve during the task. We expect that agents should learn to enhance the extent to which humans achieve these goals while maintaining agents' original abilities (e.g., winning games). To achieve this, we propose the Reinforcement Learning from Human Gain (RLHG) approach. The RLHG approach introduces a "baseline", which corresponds to the extent to which humans primitively achieve their goals, and encourages agents to learn behaviors that can effectively enhance humans in achieving their goals better. We evaluate the RLHG agent in the popular Multi-player Online Battle Arena (MOBA) game, Honor of Kings, by conducting real-world human-agent tests. Both objective performance and subjective preference results show that the RLHG agent provides participants better gaming experience.

Published as a conference paper at ICLR 2024
ENHANCING HUMAN EXPERIENCE IN HUMAN- AGENT COLLABORATION: A HUMAN-CENTERED MODELING APPROACH BASED ON POSITIVE HUMAN GAIN
Yiming Gao-* Feiyu Liu Liang Wang! Dehua Zheng'
Zhenjie Lian'
Weixuan Wang! Wenjin Yang* Siqin Lif
Xianliang Wang' Wenhui Chen
Jing Dai Qiang Fu' Wei Yang'
Lanxiao Huang² Wei Liu'
Teeneett AI Lab, Shenzhen, China "Tencent TiMi LI Studio, Chengdu, China
vatminggao@teencent com; ww2223tcolllmeeee www.coolumii...mmm edu
ABSTRACT
Existing game AI research mainly focuses on enhancing agents' abilities to win games, but this does not inherently make humans have a better experience when collaborating with these agents. For example, agents may dominate the collabora- tion and exhibit unintended or detrimental behaviors, leading to poor experiences for their human partners. In other words, most game Al agents are modeled in "self-centered" manner. In this paper, we propose a "human-centered" modeling scheme for collaborative agents that aims to enhance the experience of humans. Specifically, we model the experience of humans as the goals they expect to achieve during the task. We expect that agents should learn to enhance the extent to which humans achieve these goals while maintaining agents' original abilities (e.g.. win- ning games). To achieve this, we propose the Reinforcement Learning from Human Gain (RLHG) approach. The RLHG approach introduces "baseline", which corresponds to the extent to which humans primitively achieve their goals, and encourages agents to learn behaviors that effectively enhance humans in achiev- ing their goals better. We evaluate the RLHG agent in the popular Multi-pli platerrplyyrr Online Battle Arena (MOBA) game, Honor of Kings, by conducting real-world numan-agent tests. Both objective performance and subjective preference results show that the RLHG agent provides participants better gaming experience.
INTRODUCTION
Recently, Reinforcement Learning (RL) has been widely used in developing Artificial Intelligence (AI) systems for games, developing various agents that perform at human-level performance, such as AlphaGo (Silver et al., 2016; 2017) in Go, AlphaStar (Vinyals et al., 2019) in StarCraftll, OpenAI Five (OpenAl et al., 2019) in Dota2, and Wukong AI (Ye al., 2020a) in Honor of Kings. To further expand the applications of these agents, researchers are exploring ways to improve their generalization human partners (Carroll et al., 2019; Hu al., 2020; Strouse al., 2021; Lupu et al., 2021; Knott et al., 2021; McKee et al., 2022; Yu et al., 2022), as well as enabling effective explicit communication between agents and humans (FAIR et al., 2022: Gao al., 2022) in collaborative tasks. However, these agents primarily focus on maximizing their own rewards to complete the task, less considering the role of their human partners. This potentially leads to behaviors that are inconsistent with human values and preferences, resulting in a poor experience for human partners (Fisac et al., 2020; Alizadeh Alamdari al., 2022). Thus, we say that the optimization objective of these agents self-centered. In a qualitative study (Cerny, 2015) on companion behavior, it was found that humans reported greater enjoyment the game when the AI assisted them more like sidekick.
Consider the scenario depicted in Figure 1(=), the self-centered agent may push the obstacle to the human side and pass through to get the coin itself, which can complete the task, but its human partner has no experience. However, illustrated in Figure 1(=), the human may prefer the agent to play more assisting role by pulling the obstacle to its side, thereby facilitating the human to get the coin. Consequently, not only is the task completed, but the experience of the human, hereafter referred to as human experience, is also enhanced. In many real-world collaborative scenarios, such as robotic
These authors contributed equally this work.

---

Published as a conference paper at ICLR 2024
assistants and autonomous driving, humans not only want to complete the task but also pursue better experience (Wilson & Daugherty, 2018a;b; Crandall et al., 2018). Therefore, it is important for collaborative agent to learn haman-centered objective, that is, to enhance the human experience.
Agent gets the coin
Human gets the coin
Task Reward +1, Human Experience
+0
Task Reward +1, Human Experience +1
Toy scenario, where an agent and its human partner are on either side of an obstacle. Only the agent is capable of pushing pulling the obstacle. Their task goal is to obtain the coin. <=: The agent gets the coin by itself. The task is completed, but the human has experience. =: The agent assists the human to get the coin. Both the task is completed and the experience the human is enhanced.
Figure 1:
In this paper, we conceptualize the human experience as the human goals they expect to achieve during the task. Note that the human experience is saskddependent, and the corresponding goals may be explicit, such as obtaining coins, improving safety, and enhancing empowerment (Du et al., 2020). Additionally, these goals may be implicit and require inference through goal reward models learned with human feedback (Ng et al., 2000; Ziebart et al., 2008; Ho & Ermon, 2016; 6; Christiano et al., 2017; Ouyang al., 2022). These human goals can then be quantified as human goal rewards, hereafter referred to as human rewards, which measure the extent to which humans achieve these goals. Our work does not aim to define or infer human goals accurately, but rather focuses on training agents to enhance the extent to which humans achieve these goals. One intuitive way is to directly combine agents' original (task-related) rewards with human rewards. However, we find that this approach may encounter the human-agen credit assignment challenge, where the human rewards are assigned to the agent without any assistive behavior, potentially leading to the agent learning incorrect behavior and even losing its autonomy, i.e., the original abilities to complete the task, as shown in Section 4.2, A potential solution to solve this is to carefully reshape the human reward function. However, this approach heavily relies on domain knowledge and expertise.
We propose a novel approach that enables agents to learn to enhance the extent to which humans achieve their goals while maintaining agents' autonomy as much as possible. Our key insight is that the contribution made by humans themselves needs to be separated from the human rewards, and the remaining benefits can be considered as the real contribution of the agents to enhancing the humans, which we refer to as human gains. To realize this insight, we propose the Reinforcement Learning from Human Gain (RLHG) approach, which involves two stages. Firstly, we evaluate the primitive human performance in achieving human goals and consider this as a "baseline". We train value network to estimate the primitive expected human return in achieving human goals with episodes collected by directly teaming the human and the self-centered agent execute. Secondly, we train the agent to learn effective human enhancement behaviors. We train a gain network to estimate the expected positive gain of human return when subjected effective enhancement, compared to the "baseline". The agent is fine-tuned using the combination of its original advantage and the human-centered advantage calculated by the positive human gains.
We conducted experiments to evaluate the effectiveness of the RLHG approach in Honor of Kings ( (Wei al., 2022), a typical Multi-player Online Battle Arena (MOBA) game (Silva & Chaimowicz, 2017), which has received much attention from researchers lately (Ye et al., 2020a;b;e; Gao et al.. 2021; 2022). We first evaluated the RLHG approach in simulated environments, i.c., human model-agent tests. Our experimental results demonstrate that the RLHG agent outperforms bascline agents in enhancing the performance of the human model in achieving human goals. We further conducted real-world human-agen tests, with the RLHG agent teaming up with participants of varying skill levels. Our experimental results show that the RLHG agent effectively enhanced the performance of general-level participants in achieving their goals, bringing them closer to the performance of highl participants. And we find that this enhancement is generalizable across different participant skill levels. Additionally, subjective preference results reveal that most participants were satisfied with the RLHG agent, believing it provided a better gaming experience. In general, our contributions follows:
We proposed human-cenneee centered modeling scheme for guiding human-leved agents to enhance the experience of their human partners in collaborative tasks.
2

---

Published as a conference paper at ICLR 2024
We gained insights into the challenge of human-agent credit assignment and addressed this challenge by presenting the RLHG algorithm, along with a detailed implementation framework.
We conducted human-agent tests in Honor of Kings, and both objective performance and subjective preference results show that the RLHG agent provides participants better gaming experience.
2
BACKGROUND
2.1 GAME INTRODUCTION
Honor of Kings is a typical MOBA game, characterized by multi-agent cooperation and competition mechanisms, often used as testbed for game AI research (Wu, 2019; Ye et al., 2020a;b;e; Gao et al., 2021; 2022). The game is played by two opposing teams on a symmetrical map, each comprising five players. The game environment, depicted in Figure 2(a), consists of the main hero with peculiar skill mechanisms and attributes, controlled by each player. Players can maneuver the hero's 's movement using the wheel (C.1) and release the hero's skills through the buttons (C.2. C.3). They can view the local environment on the screen, the global environment on the mini- map (A), and access game states on the dashboard (B). The agent and the human player share the same game information and the action mechanism. Agents team up with human players to compete against the enemy camp through collaboration. The gaming experience is crucial for a player's ' engagement and satisfaction. Along with the task goal, players also pursue multiple individual goals (Figure 2(b)), such as achieving higher MVP score, experiencing more highlight moments, and obtaining more in-game resources. The pursuit of these goals contribute to more enjoyable and rewarding gaming experience. Agents can enhance the gaming experience of their human partners through interactive behaviors, such timely support, sharing resources, and active protection.
Goal
Human Goals
Mar
Partal
Game Victory
MVP Soore
In-game Resource
Highight Moment
C.1
Low
Few
Erjoyublle
Win
High
More
(b)
Figure 2: (a) The UI of Honor of Kings. (b) In in-game goals, based on our participant survey (see Figure 4(c)).
Human players pursue multiple goals for enjoyable gaming experience.
2.2 HUMAN-AGENT COLLABORATION
We formalize a human-agent collaboration task as an extension of the Decentralized Partially Observ- able Markov Decision Processes (Dec-POMDP) (Bernstein et al., 2002), which can be represented as tuple < N, S. A, O, P. R, 7, H. 9 R >, where N denotes the number of agents. denotes the space of global states. A = {AA,, A li=1.... denotes the space of actions of N agents and human to be enhanced, respectively. O = 1O' " denotes the space of observations of N agents and the human, respectively. P S x A
+ and R: : S x A + E denote the shared state transition probability function and the task reward function of N agents, respectively. y E [0. 1) denotes the discount factor. We define an agent policy, σ², to be a mapping from histories of observations of = (of ..., }f} E O' to actions a' E A'. A joint policy, no =< π n. >, parameterized by 0, defined to be a tuple of N agent policies. We define human policy म. to be a mapping from observations " E " to actions " A", which remains fixed during the agent's optimization process and cannot be accessible to the agent. g" (=.........MM denotes the human goals, where gi is designated goal and M is the total number human goals. R S G + IR denotes the human reward function.
Most previous research work focuses on learning self-cented centered agents. In agent-only team settings, the agent is optimized to complete the task, with the optimization objective being to maximize the value function for a given state 8, i.c., (s) = En, [G,|s; = s]. where G, = Ei x* R represents the discounted cumulative task rewards. In human-agent team settings, the agent is
3

---

Published as a conference paper at ICLR 2024
optimized to adapt to its human partners to complete the task, and the corresponding optimization objective is VT (s) = Ex [2 = s]. This optimization objective paradigm is referred to as the self-centered objective. While optimizing the self-centered objective may enhance the agents' abilities to complete tasks (i.e., task rewards Rt in collaboration with humans, it does not necessarily enhance the human experience (i.e., human rewards R
METHODS
In this section, we introduce the human-centered modeling scheme. We start with formalizing the human-centered objective (Section 3. Then we propose novel insight that enables agents learn to enhance the extent to which humans achieve their goals while maintaining the agents' autonomy (Section 3.2). Finally, we implement our insights by providing the RLHG algorithm and practical implementation (Section 3.3).
3.1 HUMAN-CENTERED ORJECTIVE
We define the human-centered objective VH+ (s) BT [G s]. where G Et= n R R++++1 is the discounted cumulative human rewards. Intuitively, the self-centered ob- jective optimizes the agents' abilities to complete the task, while the human-centered objective optimizes the agents' abilities to enhance human performance in achieving human goals G" Thus, the overall optimization objective can be formulated as:
((0) TT. (s) V.T0.T (s) Emg.TH [G. + a C =
(1)
where a a balancing parameter. The approximation to the policy gradient defined as follows:
VJ(0) = BTg.
Vo logo Vologno(a)(o)) (Go t (GL+a
(2)
x BTg
Σ L-o
Vo (oloo (A(s,, a Au(s)
(3)
where A(s,a) a) ET [Gr| 8, d: al VaoTH (s) is the self-centered advantage and Au(s,a) E. [G 8 s, a4 a)
-H VT..TN (s) is the centaneeenteed advantage.
However, incorporating human rewards directly into the optimization objective may lead to negative consequences, such as human-agent credit assignment issues. Intrinsically, humans possess the primitive abilities to achieve certain goals independently. Therefore, it is unnecessary to reward the agents for assisting when the human can easily achieve human goals or to reward the agents who do not provide any assistance, as it potentially leads to the agents learning incorrect behavior and even losing their autonomy. To this end, we propose a novel insight below to achieve effective human enhancement without compromising the agents' original abilities to complete the task.
3.2 EFFECTIVE HUMAN ENHANCEMENT
Our key insight is that the human contribution, termed human primitive value, should be distinguished from human rewards. The residual benefits, representing the agent's actual contribution in enhancing human to achieve goals, are referred to as human gains.
We define the human primitive value as V To (s), which represents the expected human return for given state under the setting of collaboration between the human ти and the pre-trained agent To- We define the human gain as (ss, (s, a) = TF, Fr,,, [G [s; 8, a = a] V VTo. (s), which represents the benefits brought by taking a specific action for a given state compared to the human primitive value. In the process of learning to enhance the human, the agents explore two types behaviors in state effective enhancement behaviors, i.e., A+(s)
{aja иeo A(s, a) > 0}. and invalid enhancement behaviors, i.e., A (s) = A(s) \ (+)s) fala ~ ne. A(s, a) I 0}. Intuitively, A
can help the human achieve human goals better than the primitive and A provides no benefits or even hinders the human from achieving human goals. Therefore, the agent is only encouraged to
learn effective enhancement behaviors, and the Eq. can be reformulated
4

---

Published as a conference paper at ICLR 2024
Ve log To(arlo)) (A(s;, + As(s)
VJ(0) Exo.n
(4)
where (s(s,a) a) as A(s) is the advantage of the human gain over the expected positive
human gain [(s) = Ba-A-(a) (A(s, a)]. We use (s) to denote an estimate (ss), which can be trained by minimizing the following loss function:
1.
x 0
L(w)= = EsE ([A(s,a)) [A(s, - (s))ll [(x)
(5)
0,
x 20
where I in Eq. 5 is an indicator function to filter invalid enhancement samples.
3.3 THE ALGORITHM & PRACTICAL IMPLEMENTATION
Algorithm Reinforcement Learning from Human Gain (RLHG)
1:
while not converged do
2: Freeze No and collect human-agent team < To. कe trajectories;
Compute human return C and update human primitive value V4 <- C
end while
Stage I: Human Primitive Value Estimation
5:
while not converged do
6: Freeze Ve and collect human-agent team <, To, FH > trajectories;
7:
Compute original return G. self-centered advantage A;
8:
Compute human return G human gain A, and human-centered advantage Ag;
9:
Update agent policy network using Eq. 4, and value network V₂ + G;
10: Update human gain network A., using Eq. 5; // Stage
11:
end while
Human Enhancement: Training
We introduce the RLHG algorithm 1 to actualize our insights, comprising two stages: the Human Primitive Value Estimation (Stage I) and the Human Enhancement Training (Stage II). In Stage the RLHG algorithm freezes the pre-trained agent policy To and collects trajectory samples to compute the human return G Subsequently, the RLHG algorithm updates the human primitive value network V. by minimizing the Temporal Difference (TD) errors (Sutton & Barto, 2018). In Stage II, the RLHG algorithm freezes Va and collects trajectory samples to compute the original return and human return GH Both G and G contribute to determining the self-centered advantage and 's policy network Ti is fine-tuned according to Eq. 4 and the value network Ve is fine-tuned by minimizing the TD errors. A is trained by minimizing the loss function in Eq. 5.
human-centered advantage Ag, respectively. The agent's
Figures 3(a) and (b) show the training framework of Stage I and Stage II, respectively. The human policy тH can be trained via Behavior Cloning (BC) (Bain & Sammut, 1995) or any Supervised Learning (SL) techniques (Ye et al., 2020b), but this is not the focus of our concern. Since in many practical scenarios, agents cannot access the human policy, we instead learn a human policy embed- ding, enabling agents to adapt and infer human intentions. Similar to the Theory-of -f-Mind (Rabinowitz et al., 10) we input the observed human historical information h; = (s .... s into an LSTM module (Hochreiter & Schmidhuber, 1997) to extract the human policy embedding. The human policy embedding is subsequently fed into two extra value networks, i.e., the human primitive value network V, and the gain network A,, and fused into the agent's original network Tg. We use surgery techniques (Chen al., 2015; OpenAl et al., 2019) to fuse the human policy embedding into the agent's s original network, i.e. adding more randomly initialized units to an internal fully-connected layer. We apply the absolute activation function ensure that the predicted gains non non-negative. In practical training, we find that only conducting human enhancement training has certain nega- tive impact the agent's original abilities complete the task. Therefore, introduce 1 -3% agent-only team settings to maintain the agent's 's original abilities and reserve 3% human-agent team settings learn effective enhancement behaviors, These two environments can be easily controlled through the task gate, i.c., the task gate is set to 1 in the human-agent team settings and 0 otherwise.
5

---

Published as a conference paper at ICLR 2024
Observation Oz
Human- Agent
Architocture
Encoder
Environment
LSTM
Frozen
History
(5) 5t
+1
(g(h))
Human-Agent
Agent-Only
(b)
Observation
ky(o,A))
St
Encodar
Ky(AAA))
Vg(h.)
Human History Info 16
abs
(A.)
8%
Figure 3: The RLHG training framework. (a) Human Primitive Value Estimation stage. The human primitive value network V₂ trained the human-agent team settings Enhancement Training stage. Vg frozen and added to a downstream network Aw to learn to estimate the expected positive human gain. 5% human-agent team settings are used to learn human enhancement behaviors, and 1 5% agent-only team settings are used to maintain the agent's original ability.
the agent's policy л frozen. (b) Human
4 EXPERIMENTS
In this section, we evaluate the proposed RLHG approach by conducting both simulated human model-agent tests and real-world human-agent tests Honor of Kings (HoK). All experiments' were conducted in the 5v5 mode with full hero pool (over 100 heroes, Appendix A.2). Our demo vidcos and code are present at attps://sitt google. wwttwwww watgeeewwwhgggeeee demo.
4.1 EXPERIMENTAL SETUP
To evaluate the performance of the RLHG agent, we conducted experiments in both the simulated environment, i.e., human model-agent game tests, and the real-world environment, iee. human-agent game tests, as shown in Figure 4(a) and 4(b), respectively. All game tests were played in 5v5 mode, that is, 4 agents plus 1 human human model team up against fixed opponent team. To conduct our experiments, we communicated with the game provider and obtained testing authorization. The game provider assisted in recruiting 30 experienced participants with anonymized personal information, which comprised 15 high-level (top 1%) and 15 general- (top 30%) participants. We first did an aap-ropd participant survey on what top 5 goals participants want to achieve in-game, and the result is shown in Figure 4(c). We can see that the top 5 goals voted the most by the 30 participants including the task goal, i.e., game victory, and 4 human goals, i.e., high MVP score, high participation, more highlights, and more resources. We find that participants consistently rated the high MVP score goal most, even more than the task goal.
Environment Setup:
Humen
Goals
Votes
High MVP Score
Victory
High Participation
More Highlights
More Resources
Fixed Opponent
Fixed Opponent
(4)
(b)
(4)
Figure 4: Environment Setup. (a) Simulated environment: human model-agent game tests. (b) Real-world environment: the human-agent game tests. (c) Top 5 goals based on the stats of our participant survey. denotes the task goal. The participant survey contains 8 initial goals, each participant can vote to non-repeating goals, and can also add additional goals. 30 participants voluntarily participated in the voting.
All experiments conducted subject oversight by an Institutional Review Board (IRB).

---

Published as a conference paper at ICLR 2024
Training Setup: We were authorized by Ye et al. use the Wukong agent (Ye al., 2020a) the pre-trained agent and use the JueWu-SL agent (Ye et al., 2020b) as the fixed human model. Note that both the Wukong agent and the Jue agent were developed at the same level the high-level (top 1%) players. We adopted the top 4 human goals as G for the pre-trained agent to enhance the human model. The corresponding goal reward function the human primitive value network and fine-tune the agent until they converge for 12 and 40 hours, respectively, using physical computer cluster with 49600 CPU cores and 288 NVIDIA VI00 GPU cards. The batch size of each GPU is set to 256. The parammerrsesters a and / are set to 2 and 50, respectively. The step size and unit size of the LSTM module are set 16 and 4096, respectively.
be found in Appendix B.3. We trained
Baseline Setup: We compared the RLHG agent with two baseline agents: the Wukong agent (Ye al., 2020a) and the Human Reward Enhancement (HRE) agent.
Wukong: A state-of-the-ard agent in HoK, trained using the PPO algorithm to optimize the Game Victory goal in an agent-only team setting.
HRE: An agent that is fine-tuned from the pre-trained Wukong agent by directly incorporating the human-centered objective into the optimization objective (see Eq. 3).
RLHG: An agent that is fine-tuned from the pre-trained Wukong agent by incorporating the human-centered objective based on positive human gains into the optimization objective.
The human model-agent team (4 Wukong agents plus 1 human model) was adopted as the fixed opponent for all tests. For fair comparisons, both the HRE and RLHG agents were trained using the same human reward function, and all common parameters, network structures, and training resources were kept consistent. Results were reported over five random seeds.
4.2 HUMAN MODEL-AGENT TEST
Directly evaluating agents with humans is expensive, which is not conducive to model selection and iteration. Instead, we build a simulated environment, i.e., human model-agent game tests, to evaluate agents, in which the human model, i.e., the Ju-Wu agent, teams up with different agents. The results of the human model on different goal after teaming up with different agents are shown in Figure 5, including the top 4 human goals and the task goal.
380
(8) Win Rate
(b.1) MVP Score
(b.2) Highlight Times (b.3) Pariiptino Rate (b.4) Resource Quantity
(c) Follow Rate
Figure 5: The performance of the human model in achieving human goals after teaming up with different agents. human goals 1, b.2, b.3, and b.4). (c) The follow metrie: the frequency
(a) The task goal. (b) The top with which an agent follows a human in the entire game. Each agent played 10,000 games. Error bars represent 95% confidence intervals, calculated over games.
From Figure 5 (b), we can observe that both the RLHG agent and the HRE agent significantly enhance the performance of the human model in achieving the top 4 human goals, and the RLHG agent has achieved the best enhancement effect on most of the human goals. These results validate that the objective can encourage agents to learn behaviors that better enhance humans in achieving their goals.
However, as shown in Figure 5 (a), the HRE agent drops significantly on the task goal. We observed the actual performance of the HRE agent and found that the HRE agent did many unreasonable behaviors. For example, to assist the human model in achieving the Participation Rate and Highlight Times goals, the HRE agent had been following the human model throughout the entire game. Such excessive following behaviors greatly affected its original abilities to complete the task and led to a decreased Win Rate. This can also be reflected in Figure 5(c), in which the HRE agent has the highest Follow Rate metric. Although the Follow Rate of the RLHG agent has also increased, observed that most of the following behaviors of the RLHG agent can effectively assist the human model. We also observed that the Win Rate of the RL.H agent slightly decreased, which is consistent
7

---

Published as a conference paper at ICLR 2024
with expectations, enhancing the abilities to achieve human goals inevitably sacrifices the abilities to achieve the task goal. We implement an adaptive adjustment mechanism to balance the two optimization objectives. We simply utilize the agent's original value network measure the degree of completing the task goal and set the task gate to (enhancing the human) when the original value above the specified threshold ç, and to 0 (completing the task) otherwise. The threshold depends on the human preference, i.c. the relative importance of the task goal and the human goals. We verify the effectiveness of the adaptive adjustment mechanism in Appendix C.3.
4.3 HUMAN-AGENT TEST
In this section, we conduct online experiments examine whether the RLHG agent can effectively enhance the experience of human participants. Note that, We did not compare the HRE agent, since the Win Rate of the HRE agent extremely low. We used a within-pant aathin-participant design for the experiment: each participant teams up with four agents. This design allowed us evaluate both objective performance as well as subjective preference. All participants read detailed guidelines and provided informed consent before the testing. Each participant tested 20 matches. After each test, participants reported their preference over their agent tcammates. For fair comparisons, participants told the type of their agent teammates.
Table I; The results of high-level participants achieving goals after teaming up with different agents. Results for the task goal are expressed as percentages. and results for human goals are expressed as mean (std.).
Task Goal
Top human Goals
Agent Goals
Win Rate
MVP Score
Highlight Times
Participation Rate
Resource Quantity
Wakong
52%
8.86 (0.79)
0.53 (0.21)
0.46 (0.11)
5.3 (2.87)
RL.H
46.7%
10.28 (0.75)
0.87 (0.29)
0.58 (0.09)
6.28 (2.71)
Table 2: The results of general-level participants achieving goals after teaming up with different agents. Results for the task goal expressed percentages, and results for human goals expressed mean (std.).
Task Goal
Top Human Goals
Agent 1
Goals
Win Rate
MVP Score
Highlight Times
Participation Rate
Resouree Quantity
Wukong
34%
(0.71)
(0.349)
(0.11)
(2.73)
30%
(0.61)
(0.253)
(0.05)
5.8 (2.78)
We first compare the objective performance of the participants on different human goal metrics after teaming up with different agents. Table and Table 2 demonstrate the results of high-level and general-level participants, respectively. We see that both high-level and general-leve level participants had significantly improved their performance all top human goals after teaming up with the RLHG agent. Notably, the RLHG agent effectively improved the performance of general-leve participants achieving human goals even better than the primitive performance of high-level participants. We also notice that the Win Rate of the participants decreased when they teamed up with the RLHG agent, which is consistent with the results the simulated environment. However, we find in the subsequent subjective preference statistics that the improvement of Gaming Experience brought by the enhancement outweighs the negative impact of the decrease in Win Rate.
Wakorg
REME
Proventno
General-level
Highlevel
General-level
High-level
Highleval
General-level
High-level
(a) Behavioral Rationality
(b) Enhancemant Degres
(e) Gaming Experfence
(4) Overall Preferanca
Figure Participants' preference over their agent teammates. (a) Behavioral Rationality: the reasonableness of the agent's behavior. (b) Enhancement Degree: the degree to which the agent enhances your abilities to achieve goals. (c) Gaming Experience: your overall gaming experience. (d) Overall Preference: your overall preference for your agent teammates. Participants scored (1: Terrible, 2: Poor, 3: Normal, 4: Good, 5: Perfect) these metries after each game test. Error bars represent 95% confidence intervals, calculated over games.
8

---

Published as a conference paper at ICLR 2024
We then compare the subjective preference metrics, i... the Behavioral Rationality, the Enhancement Degree, the Gaming Experience, and the Overall Preference, reported by participants over their agent teammates, as shown in Figure 6. We find that most participants showed great interest in the RLHG agent, and they believed that the RLHG agent's ' enhancement behaviors were more reasonable than that of the Wukong agent, and the RL.G agent's enhancement behaviors brought them better gaming experience. A high-level participant commented on the RLHG agent "The agent frequently helps me do what I want do, and this feeling amazing." In general, participants were satisfied with the RLHG agent and gave higher scores in the Overall Preference metric (Figure 6(d)).
4.4 CASE STUDY
To better understand how the RLHG agent effectively enhances the experience participants, we visualize the values predicted by the gain network in two test scenarios where participants benefitted from the RLHG agent's assistance, as illustrated in Figure 7. In the first scenario (Figure 7()), the RLHG agent successfully assisted the participant in achieving the highlight goal, whereas the Wukong agent disregarded the participant, leading to a failure in achieving the highlight goal. Figure 7(b) illustrates that the gain of the RLHG agent, when accompanying the participant, is positive, reaching the maximum when the participant achieved the highlight goal. In the second scenario (Figure 7(c)),
the RLHG agent actively relinquishes the acquisition of the monster resource, enabling the participant to successfully achieve the resource goal. Conversely, the Wukong agent competes with the participant for the monster resource, resulting the participant's 's failure to achieve the resource goal. Figure 7(d) also reveals that the gain of the RLHG agent's 's behavior actively forgoing the monster resource, positive, with the gain peaking when the participant achieved the resource goal. These results indicate that the RLHG agent learns effective enhancement behaviors through the guidance of the gain network.
(T2) Agant ignoree Haran.
(Tt) gane to Top Lans. Wakoeg
Gain
Nvn
(TZ) Agure Sallows and asuiess Hamann
Hursan achirves Highkght
Primiive Value
Enhanced Value
BLHO
(b)
(T)) Hamoon wart Monieer
(T3) Agant compatee with Humaan for Monsterr
Agest
Wikkong
Gain
Morittr
T1
T2
Primitive Value Enhanced Value
(c)
RUHG
(d)
Figure 7: The RLHG agent enhances the experience of participants in two scenarios. The Wukong agent ignores the participant; The RL.HG agent accompanies the participant and assists the participant in achieving the highlight goal. (b) The gain value in scenario (a). (c) The Wukong agent competes with the participant for the monster resource; The RLHG agent actively forgoes the monster resource, and the participant successfully achieves the resource goal. (d) The gain value in scenario (c).
5 CONCLUSION
In this work, we proposed "human-centered" modeling scheme for collaborative agents, designed to enhance the human experience. We represent the human experience as the goals they expect to achieve during the task. To enhance the extent to which humans achieve these goals while maintaining agent's original abilities, we introduced the RLHG approach. The RLHG approach initially trains a value network to estimate the primitive expected human return in achieving human goals, utilizing episodes collected by directly partnering the human and the self-centered agent for execution. Subsequently. the RLHG approach trains a gain network to estimate the expected positive gain of human return when subjected to effective enhancement, compared to the "baseline." 'The agent is fine-tuned using combination of its original advantage and the numan-centered advantage calculated by the positive human gains. We conducted real-world numan-agent tests in Honor of Kings, and the results in both objective performance and subjective preference demonstrate that the RLHG agent offers a superior gaming experience for humans.

---

Published as a conference paper at ICLR 2024
REFERENCES
Parand Alizadeh Alamdari, Toryn Q Klassen, Rodrigo Toro Icarte, and Sheila A Mcllraith. Be considerate: Avoiding negative side effects in reinforcement learning. In Proceedings of the 21st International Conference Autonomous Agents and Multiagent Systems, pp. 18-26, 2022.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 219-840,44 2002.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32, 2019.
Martin Cerny. Sarah and sally: Creating a likeable and competent ai sidekick for a videogame. In Pro- ceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 11, pp. 2-8, 2015.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv: I51 1.0641, 2015.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Jacob W Crandall, Mayada Oudah, Fatimah Ishowo-Oloko, Sherief Abdallah, Jean-François Bonne- fon, Manuel Cebrian, Azim Shariff, Michael Goodrich, and lyad Rahwan. Cooperating with machines. Nature Communications, 9(1):233, 2018.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Picter Abbeel, and Anca Dragan. Ave: Assistance via empowerment, Advances in Neural Information Processing Systems, 33:4560-4571, 2020.
FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 37866624) 2022.
Jaime Fisac, Monica Gates, Jessica B Hamrick, Chang Liu, Dylan Hadfield-Menell Malayandi Palaniappan, Dhruv Malik, Shankar Sastry, Thomas L Griffiths, and Anca D Dragan. Pragmatic- pedagogic value alignment. In Robotics Research: The 1 18th International Symposium ISRR, pp. 49-57 57. Springer, 2020.
Yiming Gao, Bei Shi, Xueying Du, Liang Wang, Guangwei Chen, Zhenjie Lian, Fuhao Qiu, Guoan Han, Weixuan Wang. Deheng Ye, al. Learning diverse policies in moba games via macro-goals. Advances in Neural Information Processing Systems, 34:16 34:16171-16182 2021.
Yiming Gao, Feiyu Liu, Liang Wang. Zhenjie Lian, Weixuan Wang. Siqin Li, Xianliang Wang. Xianhan Zeng. Rundong Wang, QIANG FU, et al. Towards effective and interpretable human- agent collaboration moba games: communication perspective. In The Eleventh International Conference Learning Representations, 2022.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9()): 1735-1780, 1997.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference Machine Learning, pp. 4399 4499. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in Neural Information Processing Systems, 31, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 11Xx::1412.6980,, 2014.
Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, and Rohin Shah. Evaluating the robustness of collaborative agents. arXiv preprini arXiv: 100.000.0550 101.05507, 2021.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International conference machine learning, pp. 7204-7213. PMLR, 2021.
Kevin McKee, Joel Z Leibo, Charlie Beattie, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Mueti-Agent Systems, 36(1) (6(1))21, 2022.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Andrew Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, pp. 2. 2000.
OpenAl, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz. Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1912.06682 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory mind. International Conference on Machine Learning, pp. 4218-4221 PMLR, 2018.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.00 024388 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Victor do Nascimento Silva and Luiz Chaimowicz. Moba: A new arena for game ai. arXiv preprint 1.X.::700S.14433 2017.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484
2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-3555 2017.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.

---

Published as a conference paper at ICLR 2024
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using agett reinforcement learning. Nature, 575(7782)3300-354 2019.
Hua Wei, Jingxiao Chen, Xiyang Hongyang Qin, Minwen Deng, Siqin Li, Liang Wang. Weinan Zhang. Yong Yu, Liu Linc, et al. Honor of kings arena: An environment for generalization in competitive reinforcement learning. Advances in Neural Information Prucessing Systems, 35: 11881-11892, 2022.
H James Wilson and Paul R Daugherty. Human+ machine: Reimagining work in the age of Al. Harvard Business Press, 2018a.
H James Wilson and Paul R Daugherty. Collaborative intelligence: Humans and ai are joining forces. Harvard Business Review, 14-112,, 2018b.
Bin Wu. Hierarchical macro strategy model for moba game ai. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 1206-1213, 2019.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621-632, 2020a.
Deheng Ye, Guibin Chen, Peilin Zhao, Fuhao Qiu, Bo Yuan, Wen Zhang, Sheng Chen, Mingfei Sun, Xiaoqian Li, Siqin Li, et al. Supervised learning achieves human-level performance in moba games: case study of honor of kings. IFEE Transactions on Neural Networks and Learning Systems, 2020b.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6672-6679, 2020c.
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. In The Eleventh International Conference on Learning Representations, 2022.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence, volume 8, pp. 1433. Chicago, IL, USA, 2008.
12

---

Published as a conference paper at ICLR 2024
A ENVIRONMENT DETAILS
A.1 GAME INTRODUCTION
MOBA (Multiplayer Online Battle Arena) games (Silva & Chaimowicz, 2017), characterized by multi-agent cooperation and competition mechanisms, long time horizons, enormous state-action spaces (1020000, (Wu, 2019)), and imperfect information (OpenAI et al., 2019; Ye al., 2020a), have attracted much attention from researchers. The general layout of the MOBA games is depicted in the Figure 8(a). The red and blue on the map indicate two teams, and the map can be divided into 3 lanes (i.e., yellow areas separated into the top, middle, and bottom lane), 4 jungle areas (i.e. green areas), and base areas. Circles symbolize turrets.
Top Lane
Base Area
Jungle
Jungle Area
Mid Lane
C.1
C.2
(a)
(b)
Figure 8: (a) A generic map of MOBA games. (b) The UI interface of Honor of Kings.
Honor of Kings (Wei al., 2022) typical MOBA game often used as a testbed for game Al research (Wu, 2019; Ye et al., 2020a;b;e; Gao et al., 2021; 2022). The game is played by two opposing teams on symmetrical map, each comprising five players. The game environment depicted in Figure 8(b) comprises the main hero with peculiar skill mechanisms and attributes, controlled by each player. The player can maneuver the hero's s movement using the bottom-left wheel (C.1) and release the hero's ' skills through the bottom-right buttons (C.2, C.3). The player can view the local environment on the screen, the global environment on the top-left mini-map (A), and access game stats on the top-right dashboard (B), Players of each camp compete for resources through team confrontation and collaboration, etc., with the task goal of winning the game by destroying the crystal in the opposing team's base area.
Bittersweet
Enjoyable
Task Goal
Game Victory
Win
Win
MVP Score
Low
High
Human Goals
In-game Resource
Few
More
Legendary!
Highlight Moment
Few
More
Pentakill!
In-game goals, based on our participant survey (see Figure 24 (c)). Human players pursue multiple goals for more enjoyable gaming experience.
Figure 9:
The gaming experience is crucial for player's engagement and satisfaction. Along with the task goal, players also pursue multiple individual goals (Figure 9, such as achieving a higher MVP score, experiencing more highlight moments, and obtaining more in-game resources. The pursuit of these
13

---

Published as a conference paper at ICLR 2024
goals contribute to more enjoyable and rewarding gaming experience. Agents enhance the gaming experience their human partners through interactive behaviors such as timely support, sharing resources, and active protection.
For fair comparisons, all experiments this paper were carried out using a fixed released game engine version (Version 8.2 series) of Honor of Kings.
A.2 HERO POOt
Table 3 shows the full hero pool used in Experiments, Each match involves two camps playing against each other, and each camp consists of five randomly picked heroes.
Table Hero pool used in Experiments.
Lian Po. Xiao Qiao, Zhao Yun, Mo Zá, Di Ying Zheng, Sun Shangxiang. Lubun Qihuo, Zhuang Zhou. Liu Chan Gao Jianli, Ke, 7hong Wayan, Sun Bin, Bian Que, Bai Mi Yoe. Ba, Zhou Yu, Yuan Gr. Chengji Sihan Xin Houdun, Zhen Ji, Cao Cao, Dian Wei, Gongben Wucang, Li Bai, Make Boluo, Di Renjic, Do Mo. Xiang Yu Chan, Qila, Cheng Yaujin, Han Xin, Wang Zhaojun, Lan Lingwang, Hun Mulan, Ai Lin, Zhang Liang, Buzhi Huown, Nake Lalu, Ju Youjing Sun Wakong. Nio Mo, Hou Yi. Liu Bei, Zhang Fei, Yuanfang. Yu JI, Zhong Kui. Yang Yohuan, Zhu Bajic Yang Jian. Ny Wa, Ne Zha, Gunjiang Moye, Ya Dianna. Cai Wenji. Taiyi Zbeuren, Donghuang Taiyi, Gui Guzi
Zetian, Mayi,
Jiany Ziya, Bang, Chang E
Full Hero pool
Zhu Geliang, Da Qiao, Huang Zhong. Kai, Su Lie. Baili Xuance, Buili Shouyue, Yi Xing, Meng Qi, Gong Sunli Shen Mengxi, Ming Shiyin, Pei Qinhu, Kuang Tie. Mi Laidi, Yso, Yun Zhongjun, Xin, Jin Luo, Dun Shan, Sun Shangguan Waner, Ma Chao, Dong Fangyao, Xi Shi, Mong Yo, Luban Dushi. Pan Gu. Mong Tian, Jing. A Guduo Laote, Lan, Sikong
Frin, ying. Chan, Sang Ya. Yue, Thao Huaizhen,
B FRAMEWORK DETAILS
B.1 INFRASTRUCTURE DESIGN
Inference Server
Ogt Fu1
parameter sync
Natwork prodhction
Sample management
Al Server with Game Env
Learner
self-play
Policy network
Sagents
Em
agents
netwock
Memory Pool
Eav
GPUs AildRuel
Figure The infrastructure of training system.
Figure 10 shows the infrastructure of the training system (Ye et al., 2020a), which consists of four key components: AI Server, Inference Server, RL Learner, and Memory Pool. The AI Server (the actor) covers the interaction logic between the agents and the environment. The Inference Server is used for the centralized batch inference on the GPU side. The RL Learner (the learner) a distributed training environment for RL models. And the Memory Pool is used for storing the experience, implemented as memory-efficient circular queue.
14

---

Published as a conference paper at ICLR 2024
B.2
TASK REWARD DESIGN
Table 4 demonstrates the details of the designed task reward from environment.
Table 4: The details of the environment reward.
Head
Reward Item
Weighr
Type
Deseription
Gold
0.005
Dense
The gold gained.
Experience
0.001
Deuse
The experience gained.
Farming
Dense
rate
fourth power).
No-op
-0.00001
Dense
Stop and do nothing.
Attack monster
Sparse
Attack monster.
Kill
1
Sparsc
Kill enemy hcro.
Death
Spurse
Being killed.
Assist
1
Sparse
Assists.
Tyrant
Sparse
killing tyrant, tyrant, storm tyrant.
Overlord butf
1.5
Sparse
Get buff of killing the overlord.
Expose invisible enemy
Spurse
Get visions of enemy heroes.
Last hit
Sparse
Last hitting enemy minion.
Health point
health point
(In fourth power).
Damage Related
Hurt hero
0.3
Sparse
Attack enemy heroes.
Attack
Sparse Spurse
Attack
Pushing Related
Atlack crystal
Attack enemy home buse.
Win/Lose Related
Destroy home hase
4
Sparse
Destray enemy home hase.
B.3 HUMAN REWARD DESIGN
Table 5 demonstrates the details of the designed human reward.
Table 5: The details of the human reward.
Head
Reward Item
Weight
Type
Description
Sparse
enemy hero.
Death
Sparse
Being killed.
MVP Score Related
Assist
Sparse
Assists.
0.3
Sparse
Attack enemy beroes.
Health point
Dense
The health point of the bero (to the fourth power).
Participation Related
Participation
Dense
Pereentage players participating in the ream fight.
Highlight
Highlight
Sparse
Double kill, triple quadra penta
Bulf
Sparse
Gel red bulf, blue bulT. Get health cake.
Resource Related
Health cake
Sparse
15

---

Published as a conference paper at ICLR 2024
B.4 FEATURE DESIGN
Table 6 shows the designed features of the Wukong agent (Ye et al., 2020a), some of which (observ- able) used human features.
Table 6: The observation space of agents.
used human features.
Feature Class
Field
Description
Dimension
Type
1. Unit feature
Scalar
Includes heroes, minions, monsters, and turrets
8599
Current HIP, speed, level, gold, KDA, buff, bad states, orientation, visihility, etc.
(ane- normalized float)
Position
coordinates
(normalized float)
Is main hero or not, hero TD, camp (team), joh, physical attack and defense, magical attack defense, etc.
Heroes
Atrribute
1330
(one-hot, normalized float)
Skill Skill N's cool down time. usability, level,
Skills
(one-hot, normalized float)
range, bulf elfects, bad effects, etc.
Ttem
Current item lists
(ane- hol)
Status
Current HP, spoed, visibility, killing income, etc.
1160
(one-hot, normalized float) (normalized float)
Position
Current 2D coordinates
80
Minions
Altribute
Camp (teum))
(one- hol)
Type minions (melee creep, ranged creep,
Type
(one-hot)
siege croop, super creep, etc.)
Status
Current HP, speed, visibility, killing income,
868
(onc-hor, nocmalized float)
*onsters
Position
Current 2D coordinates
(normalizzd float)
Type
monsters (normal, blue, red, lyrant, overlard, etc.)
168
(ane hol)
targets, attack speed, etc.
normalized float) (normalized float)
Turrets
Position
Current 2D coordinates
40
Type
Type turrets (tower, high tower, crystal, etc.)
(one-hot)
2. In-game stats feature
Scalar
Real time statistics of the game
68
Time
Current game time
5
(onc-hot)
Gold
Golds of two camps
12
(normalized float)
Static statistics*
Alive heroes
Number alive heroes of camps
(onc-hot)
Kill
Kill number of each camp (Segment representation)
6
(one-hot)
Alive turrets
Number alive
two camps
(ane hot)
Gold diff
Gold difference between two camps (Segment representation)
5
(ane-hot)
Alive heroes diff
Alive heroes difference between camps
(onc-hot)
Comparative statistics*
Kill diff
Kill difference between two camps
5
(one-hot)
Alive turrets diff
turrets difference between camps
(one hol)
3. Invisible opponent information
Scalar
Invisible information used for the value net
560
Opponent heroes
Position
Current 2D coordinates, dislances, etc.
120
(normalized fluat)
Current 2D coordinates of all non-player characters, including minions, monsters, and turrets
Position
(normalized float)
4. Spatial fenture
Spatial
2D image like, extracted in chunnels for convolution
7x17x17
Region
Potential damage regions of ally and enemy skills Bullets of ally and enemy skills
2x17x17
Bullet"
2x17x17
Obstacles*
Region
Forbidden region for heroes to
1x17x17
Bushes
Region
Bush region for heroes hide
1x17x17
Health cake*
Region
Cake for heroes to recover blood
1x17x17
16

---

Published as a conference paper at ICLR 2024
B.5
ACTION DESIGN
Table 7 shows the action space of agents.
Table The action space of agents.
Action
Detail
Description
Oleegal action
Placeholder.
None action
Executing nothing or stopping continuous action.
Move
Moving certain direction determined by move and y.
Normal Attack
Executing normal attack to an enemy unit.
SkillI
Executing first skill.
Skill2
Executing the second skill. Executing
What
SkillS
Skill4
Exocuting the fourth skill (only few heroes have Skill-). An additional skill choosing before the game begins (10 to choose).
Summoner ability
Return home Recall)
Returning speing, should continuuusly executed.
Trem skill
Some items can enable an additional skill to player's hero.
Restore
Blood recovering continueusly 10s, be distarbed. Skill given by special ally heroes.
Collaborative skill
Move X
The axis offset of moving direction.
Move Y
The y-axis offiet of moving direction.
How
Skill X
The x-axis offict of skill.
Skill Y
The y-axis offiet of skill.
Who
Turget
The game unit(s) chosen to allack.
B.6 NETWORK ARCHITECTURE
Figure 11 shows the detailed network architecture of the RLHG agent, which consists of two parts: the pre-trained Wukong model (Ye et al., 2020a), and the human enhancement module.
Bone Stati Feature
Unbi Festure
Apent2
Actioro
Ofhexx Dfset
NGP
Spetial Feature
Cebeda
Deatc
de Opponentiris
Marti Hosd
Wukong Model
Hammn
Mali-hed
Hurra Futture
Humsin Calin
Human Enhancement Module
Figure 11: The network structure.
Human Enhancement Module. Human features are sequentially fed into the Fulle-Connected (FC) layers with LSTT (Hochreiter Schmidhuber, 1997) to extract human policy embedding. The policy embedding is used to predict human primitive values and gains. We apply the absolute activation function to ensure that the gains are nen-tegative. To manage the uncertain value of state-action in the game, introduce the multi-head value estimation (Ye al., 2020a) into the network by grouping the human reward in Table 5.
Human Conditioned Policy Modeling. We use surgery techniques (Chen et al., 2015; OpenAI et al., 2019) to fuse the human policy embedding into the agent's original network, i.c. adding more randomly initialized units internal FC layer. The task gate is used to control the agent's policy
17

---

Published as a conference paper at ICLR 2024
style, i.e., for the non-enhancement mode, the task gate is set to O, and for the enhancement mode, the task gate is set to 1. The agent's 's policy network predicts a sequence of actions for each agent based on its observation and human policy embedding.
of the Wukong model are consistent with the original (Ye et al., 2020a). The unit size and step size of the LSTM module in the human enhancement module are set to 4096 and 16, respectively. The parameters of each FC layer are shown in our code. We Adam (Kingma & Ba, 2014) with an initial learning rate of 0.0001 for fine-turning.
Network Parameter Details. All
B.7 ALGORITHM DETAILS
The more detailed pseudocode of RLHG is shown in Algorithm 2. The RLHG approach aims to fine-tune a pre-trained agent mo to enhance given human model и. The RLHG algorithm consists of two stages: the Human Primitive Value Estimation stage and the Human Enhancement Training stage.
The RLHG algorithm starts with initializing human primitive value network Vg- Trajectories are collected by pairing the pre-trained agent To and the human wH to perform in collaborative environment. Next, these trajectories are utilized compute the human return H for achieving human goals G Lastly, the human primitive value network Vo is updated using the human return by minimizing the Temporal Difference (TD) error. After Va converges, is frozen and used as the baseline for calculating human gains in the next stage.
Human Primitive Value Estimation:
Human Enhancement Training: The RLHG algorithm initializes the agent's policy network me and value network Vp conditioned on the human policy тH. and gain network A.. Trajectories are then collected by pairing the agent The and the human ти to perform in a collaborative environment. Subsequently, these trajectories are employed to compute the original return G and human return GH The RLHG algorithm calculates the human gain from the human return based on the predicted human primitive value (s). This human gain is used for human-centered advantage calculations. The agent's policy network sy is fine-tuned using the Policy Gradient algorithm (Sutton et al., 1999; Mnih et al., 2016), like PPO (Schulman et al., 2017), which combines the original advantage A and the human-centered advantage Åμ The agent's value network Ve is fine-tuned using the agent's original return. Finally, the gain network is updated when the human gain is positive, as per Equation 5.
Algorithm 2 Reinforcement Learning from Human Gain (RLHG)
Require:
Human policy network тH. human goals g agent policy network mo agent value network V.
hyper-parameter a
Process:
1:
Initialize human primitive value network Vo:
// Stage I: Human Primitive Value Estimation
2: while not converged do
3: Freeze To and collect buman-agent team To, uy trajectories;
4: Compute human return G for achieving goals R!.
5: Update human primitive value network Vg + C
6: end while
7: Initialize agent policy network Ty <- no, agent value network V V, human gain network A
// Stage Human Enhancement Training
8: while not converged do
9: Freeze V₂ and collect human- agent team < Te. RH > trajectories;
10:
Compute agent original return G and human return GH;
11:
Compute agent self-centered advantage
Vv:
12:
Compute human gain A = GH Vo
13:
Compute human-centered advantage Ag = A Awi
14:
Update agent policy network To + A + a Ahi
15:
Update agent value network Vi <- G;
16:
if A > then
17:
Update human gain network A_ < A: end if
18:
19:
end while
18

---

Published as a conference paper at ICLR 2024
C SUPPLEMENTARY EXPERIMENT
C.I BASELINE DETAILS
We deseribe the training process of two baseline agents here, including the Wukong agent and the Human Reward Enhancement (HRE) agent.
Wukong (Ye et al., 2020a): A state-of-the-ar art agent in Honor of Kings, which easily beat the high-level human players. As shown in Figure 12, the agent is trained in agent-only team settings, with the optimization objective being maximize Game Victories.
Agent-Only
Ty (op)he
Observation 9t
Encader
Ge
Vy(ssh))
The Wukong training process. The policy network Ty and value network Vi are trained in the agent-only team settings.
Figure 12:
To avoid instability training within large-scale distributed environments, Wukong uses the Dual- clip PPO algorithm to train the policy network Tg. which is practical improvement of the PPO
algorithm (Schulman et al., 2017). When Tz(als) > Tiai (als) and syfals)
0, the radio r(0)
Teata (a|s) huge, which causes the large and unbounded variance since r(0) < Dual-clip
PPO introduces another clipping parameter e that indicates the lower bound when A The policy
loss is the following:
L La (0) ESES,EAA max(cA, min((lip(((())))))) ( - +0)
where is the original clip parameter in PPO. In the experiments, use the same parameters as Wukong, with the two clipping hyperparameters e and c set to 0.2 and respectively. The discount factor is set 0.998. Besides, Wukong uses Generalized Advantage Estimation (GAE) (Schulman et al., 2015) for return calculation, with À = 0.95 to reduce the variance caused by delayed effects.
To decrease the variance of value estimation, Wukong uses full information about the game state, including observations hidden from the policy, as input to the value network Vi in training. To estimate the value of the ever-changing game state more accurately, Wukong introduces multi-head value by decomposing the reward. The multi-head value loss is:
Σ(c -V
T.V (0)
EyES
where Gh and V are the discounted return and value estimation the k-th head, respectively. Then, the total value estimation is the weighted sum of the head value estimates Viotai(s) Σx wkk (s). where Wa is the weight of the k-thh head.
HRE (Human Reward Enhancement): An agent that is fine-tuned from the pre-trained Wukong agent by directly incorporating the human-centered objective into the optimization objective. The optimization objective can be formulated as:
((0) = v (s) + a V VTT E G.+ a G =
where a is balancing parameter. The approximation to the policy gradient is defined as follows:
( = E RH
V lolgoo (A(s;, ar) + AH(s),
19

---

Published as a conference paper at ICLR 2024
where A(s,a) = Emaan [c 8, al ( is the self-centered advantage and As(s,a) a) Eng.N [ 8t 8, ar a)
H V (s) is the human-centered advantage.
Human--Aeet
Agent-Only
Observation Or
malorA))
Encoder
Ge
Task Gate
GF
Human History 2. of)
(h(
10%
Figure The HRE training process. The policy network mo and value network Vy conditioned the human policy TH are trained - 3% agent-only team settings and 8% buman-agent team settings. The human value network V6 is trained to estimate the expected human return in human-agcns team settings.
In comparison Wukong (see Figure 13). HRE introduces a value network (s) estimate the human value V (s) when teaming up with agent policy which is utilized to calculate the human-centered advantage. The difference between HRE and RLHG is that HRE lacks modeling of human gains. Apart from these differences, other settings remain consistent with RLHG. Firstly, the policy network Te and value network Vi are also conditioned on the human policy ли- Secondly, the agent is trained in both 8% human-agent team settings and 1 - -9% agent-only team settings. The human value network V& only trained in human-agent team settings by minimizing the following loss function:
[IGH - Vo(s))l
L(Ø) E.ES,0EA
C.2 ABLATION STUDY
We examine the influence of the balance parameter a, i.e., the relative importance of human goals relative to the task goal. The results of RLHG agents trained with different values of a are shown in Figure 14. We see that with the increase of a, the human model's performance in achieving human goals is significantly improved, but the negative effect is that the agent sacrifices its original ability achieve the task goal (The Win Rate metric is reduced). We also notice that when large, the Win Rate is significantly reduced, which will also have negative impact on the MVP score goal. We find that when set to 2, it not only greatly improves the human model's performance in achieving human goals, but also has little impact the Win Rate. Therefore, in our experiments, set to 2.
too
D"11
D"2
274
(8) Rate
(b.1) MVP Score
(b.2) Highlight Times
(b.3) Participation Rate
(b.4) Resource Quantity
Figure 14: Influence of the balance parameter Note that means training without enhancement.
C.3 ADAPTIVE ADJUSTMENT MECHANISM
We implement adaptive adjustment mechanism by simply utilizing the agent's original value network to measure the degree of completing the task goal. We first normalize the output of the original value network and then the task gate (enhancing the human) when the normalized
20

---

Published as a conference paper at ICLR 2024
value is above the specified threshold Ç. and to (completing the task) otherwise. The threshold is used to control the timing of enhancement. The results RLHG agents with different values of & are that the threshold increases, the Win Rate increases, and the
shown in Figure 15. We human model's performance on human goals decreases. In practical applications, the threshold g can be set according human preference.
0.33
BM
(2) Win Rate
(b.1) MVP Score
(b.2) Highlight Times
(b.3) Participation Rate
(b.4) Resource Quantity
Figure 15: Influence of the threshold ç. Note that { = means never enhancement, and { = means always enhancement.
D DETAILS OF HUMAN-AGENT COLLABORATION TEST
D.1 ETHICAL REVIEW
The ethics committee of a third-party organization conducted an ethical review of our project. They reviewed our experimental procedures and risk avoidance methods (see Appendix D.1.3). They believed that our project complies with the "New Generation of AI Ethics Code" 2 of the country which the participants belonged (China), they approved our study. In addition, all participants consented to the experiment and provided informed consent (see Appendix D.1.1) for the study.
D.1.1 INFORMED CONSENT
All participants were told the following experiment guidelines before testing:
This experiment is to study human-agent collaboration technology MOBA games.
Your identity information will not be disclosed to anyone.
All game statistics are only used for academic research.
You will be invited into matches where your opponents and teammates are agents.
Your goal is to win the game much possible by collaborating with agent teammates.
Your agent teammates will assist you in achieving your individual goals in the game.
After each test, you can report your gaming experience and express your preferences regarding the agent teammates.
After each test, you may also voluntarily fill out debriefing questionnaire to tell us your open-ended feedback about the agent teammates.
Each game lasts 10-20 minutes.
You may voluntarily choose whether to take the test. You can terminate the test at any time if you feel unwell during the test.
At any time, you want to delete your data, you can contact game provider directly to delete
If participants volunteer to take the test, they will first provide written informed consent, then we will provide them with the equipment and game account, and explain the experimental details on the screen.
CChina: MOST issues New Generation of AT Ethics Code, https: //www datagui dance. com/n ews/ /ch most assuss new www.-ggeneratiin onn-ai hithi as cs-cade
21

---

Published as a conference paper at ICLR 2024
D.1.2 SCREENSHOTS
Screenshots of detailed experimental instructions shown below.
1. Read tutorial and instruction on the study and gameplay. (Figure 16)
2. Read the detailed test content and precautions. (Figure 17)
3. Play the game with agents until the game is complete. (Figure 18)
4. Answer questions about perceptions and preferences.(Figure 19, 20, 21, and 22)
5. Volunteer to complete debriefing questionnaire regarding open-ended feedback from your agent teammates. (Figure 23)
6. Repeat steps 3, 4, and 5 for a total of 20 times.
After the participant has read carefully and confirmed complete understanding, the test will begin.
22

---

Published as a conference paper at ICLR 2024
Welcome to the study!
ezu prent toet of you bean topped special accoutt and you use the sey spone nnd
The tes eouipeee do sot prrsoullly idverlly you.
intormat be duickosd
Frommtinn la anduat vocil accen..
(a) Welcome participants to the experiment.
(b) Introduce test equipment.
contell
you Stant button scnun nut.
(c) Introduces mode,
(d) Introduce participant's 's controllable hero.
courvel mechoniss exuntyy the same what pou ssed toin game
Your goal to the ga as must as poodile by doordd with ajen saaniatte
The wddiionn which additoed
( Misik/ap
VICTORY
(c) Introduce the control mechanism.
(f) Explain the task goal the game.
Your individal
Logendary!
Pentakill
MyP
Particiption
Momrect
Reroucce
(g) Explain the enhanced individual goals.
Figure 16: Screenshots of tutorial and instruction screens.
23

---

Published as a conference paper at ICLR 2024
dftweet
In tasts opponeta fled teurs of agents
You movi at the gare twc, the whover additinna
Afar each wit, zan mocnt your pretennce over the agent maunt mannirate
unturily debrieling cuarto male
agen
any frow you durin
(a) Introduce agent teammates and opponents.
(b) Describe testing requirements and compensation.
Figure Screenshot experiment content.
vnluran
Deleat
Vietory
(a) Repeat the following process to test.
(b) Confirm completion of each test.
Figure 18: Screenshots of each test.
nammattrs beharior susomati
taurer buhadkr manemaale
your
dur ge your
5) béc boially untosometis bit..
12 Tumble lotaly uvmenonabll.
2) more bamteios
mow unmems valle behasior.
Perkeco
Ferfect saeaasrable beherkor
(a) Elicit participant's preference.
(b) Confirm participant's preference.
Figure Screenshots of Behavioral Rationality elicitation over test.
24

---

Published as a conference paper at ICLR 2024
high your their, charge your decaion pwn les
number on your bypbourt comeporth to your choi..
Ier as encorceennt youat wilurtion
Resille to cemen, your redurtions
enhanoments,
enfencmmnts redlctions.
Perkect ensrrmments.
Perfect onfencemnt
(a) Elicit participant's preference.
(b) Confirm participant's preference.
Figure 20: Screenshots Enhancement Degree elicitation over each test.
How your Gaming Depariance
Now la your Ganning Expertena"
sate fotl leinn worts accrdding ling suleectiv feelings
Your seliction co-ciri
2//oxe
Paak
Normit tournng humen buerenite
hamnin humen launmaass
(a) Elicit participant's preference.
(b) Confirm participant's preference.
Figure 21: Screenshots Gaming Experience elicitation over each test.
you agent
your your agen
Tleau sute owing worts ding suty festings.
snlection hight grtdd cocasior your choke, came again change pour decaees pres "ESC" bey
Freu the number oe your bepbourd that con conepandi to your cheiee
Tamble.
Turible
Pooe
Poak
Sy duse to touni wenaaaannneeeeeee
Nownal choe to 3oa to humen tounnaass
(a) Elicit participant's preference.
(b) Confirm participant's preference.
Figure 22: Screenshots of Overall Preference elicitation over cach test.
25

---

Published as a conference paper at ICLR 2024
You can ended feedback about teammates the text box. for esample, the aspect the agent that impressses you, or the dissaticfaction with the agent.
After filing the "Confirm' button.
COMM
Figure 23: Screenshot of open-ended feedback about the agent teammates from debrief questionnaire.
D.1.3 POTENTIAL PARTICIPANT RISKS
First, we analyze the risks of this experiment to the participants. The potential participant risks of the experiment mainly include the leakage of identity information and the time cost, And we have taken a series of measures to minimize these risks.
Identity Information. A series of measures have been taken to avoid this risk:
All participants will be recruited with the help of third party (the game provider of Honor of Kings), and we do not have access to participants' identities.
We make a risk statement for participants and sign an identity information confidentiality agreement under the supervision of third party.
We only unidentifiable game statistics in research, which are obtained from third parties.
Special equipment and game accounts are provided the participants prevent equipment and account information leakage.
The identity information of all participants is not disclosed to the public.
Time Cost. will pay participants to compensate for their time costs. Participants receive $5 at the end of each game test, and the winner will receive additional S2. Each game test takes approximately 10 to 20 minutes, and participants can get about an average of $20 an hour.
D.2 EXPERIMENTAL DETAILS
D.2.1 PARTICIPANT DETAILS
To conduct our experiments, we communicated with the game provider and obtained testing authoriza- tion. The game provider assisted in recruiting 30 experienced participants with anonymized personal information, which comprised 15 high-level (top 1%) and 15 general-level (top30%) participants. All participants have more than three years of experience in Honor of Kings and promise be familiar with all mechanics in the game.
And special equipment and game accounts are provided to each participant to prevent equipment and account information leakage. The game statistics we collect are only for experimental purposes and are not disclosed to the public.
D.2.2 EXPERIMENTAL DESIGN
We used a within participant design for the experiment: each participant teams up with four agents. This design allowed us to evaluate both objective performance well subjective preference. All participants read detailed guidelines and provided informed consent before the testing. Each participant tested 20 matches. Each participant is asked to randomly team up with different
26

---

Published as a conference paper at ICLR 2024
types of agents: the Wukong agent and the RLHG agent. After each test, participants reported their preference over their agent teammates. For fair comparisons, participants were not told the type of their agent teammates. The human model-agent team (4 Wukong agents plus 1 human model) was adopted as the fixed opponent for all tests.
In addition, mentioned in Ye al. (2020a); Gao et al. (2021), the response time of agents is usually set to 193ms, including observation delay (133ms) and response delay (60ms). The average APM of agents and top e-sport players are usually comparable (80.5 and 80.3, respectively). To make our test results more accurate, we adjusted the agents' capability to match the performance of high-level humans by increasing the observation delay (from 133ms to 200ms) and response delay (from 60ms to 120 ms).
D.2.3 PARTICIPANT SURVEY DESCRIPTION
We designed an IRB-approved participant survey on what top goals participants want to achieve in-game. The participant survey contains 8 initial goals, including Game Victory, High MVP Score, More Highlights, More Kill Counts, Few Death Counts, High Participation, More Resources, and More Visible Information. Each participant can vote up to non-repeating goals, and also add additional goals. 30 participants voluntarily participated in the voting, and the result is shown in Figure 24.
Human Goals
Votes
High MVP Score
Game Victory
High Participation
More Highlights
More Resources
More Kill Counts
Few Death Counts
More Visible Information
Figure 24: Voting results on human goals in Honor of Kings, based statistics from our participant survey,
D.2.4 PREFERENCE DESCRIPTION
After each test, participants gave scores on several subjective preference metrics to evaluate their agent teammates, including Rehavioral Rationality: the reasonableness of the agent's behavior, the Enhancement Degree: the degree to which the agent enhances your abilities to achieve your goals, the Gaming Experience: your overall gaming experience, and the Overall Preference: your overall preference for your agent teammates.
For each metric, we provide detailed problem description and a description of the reference scale for the score. Participants rated their agent teammates based on how well their subjective feelings matched the descriptions in the test. The different metrics are described as follows:
For the Behavioral Rationality, "Do you think your agent teammate's 's behavior is reasonable? Please evaluate Behavioral Rationality according to the following scales."
1) Terrible: totally unreasonable.
2) Poor: more unreasonable behavior.
3) Normal: some behavior is unreasonable.
Good: less unreasonable behavior.
Perfect: no unreasonable behavior.
For the Enhancement Degree, "To what extent do you think the agent enhances your abilities to achieve your individual goals? Please evaluate the Enhancement Degree according to the following scales."
27

---

Published as a conference paper at ICLR 2024
1) Terrible: no enhancements, great reductions.
Poor: no enhancements, slight reductions.
3) Normal: no enhancements, no reductions.
4) Good: slight enhancements
5) Perfect: great enhancements.
For the Gaming Experience, "How is your gaming experience? Please rate the following words according to your subjective feelings."
1) Terrible.
2) Poor.
3) Normal: close to teaming up with human teammates.
4) Good.
5) Perfect.
For the Overall Preference, "What is your overall preference for your agent teammates? Please rate the following words according your subjective feelings.".
1) Terrible.
2) Poor.
3) Normal: close to teaming up with human teammates.
4) Good.
5) Perfect.
Table 8: The subjective preference results (95% confidence intervals) of all participants in the Human-Agent Game Tests.
Participant Preference Metrics (from terrible to perfect, I~55)
Type of Agent
Participant Level
Wukong
RLHG
General-level
2.03 + 0.31
3.60 = 0.30
Behavioral Rationality
High-level
2.81 + 0.21
4.06
0.30
General-tevel
2.10+t 0.40
3.50 + 0.24
Enhancement Degree
High-level
2.94 = 0.21
4.14 = 0.25
General-level
2.40 - 0.41
4.01 = 0.30
Gaming Experience
High-level
3.06 + 0.27
4.22 + 0.30
General-level
2.63 + 0.35
3.95
0.28
Overall Preference
High-level
3.06 = 0.21
4.19 = 0.29
D.2.5 ADDITIONAL SUBJECTIVE PREFERENCE RESULTS
Detailed subjective preference statistics presented in Table 8. We that both high-level and general-level participants preferred the RLHG agent over the Wukong agent.
Behavioral Rationality, We can see that the Behavioral Rationality of the Wukong agent was lower than normal, indicating that participants believed that most of the behaviors of the Wukong agent lacked rationality. The participants generally believed that the behavior of the RLHG agent more reasonable, therefore they scored the RLHG agent more than normal.
Enhancement Degree. Participants believed that the Wukong agent did not bring them any effective enhancement, while they believed that the RLHG agent effectively enhanced their abilities to achieve their individual goals.
Gaming Experience. Participants agreed that effective enhancement gave them a good gaming experience, while the irrational behavior of the Wukong agent degraded their gaming experience.
28

---

Published as a conference paper at ICLR 2024
Overall Preference. In general, participants were satisfied with the RLHG agent and gave higher scores in the Overall Preference metric. The results of these subjective preference metrics are also consistent with the results of objective performance metrics, further verifying the effectiveness of the RLHG approach.
D.2.6 PARTICIPANT COMMENTS
After each game test, participants provided voluntary feedback on their agent tearmates. Some participants commented on the RLHG agent "Teaming up with the agent (RLHG) as teammates makes me feel good, they helped me achieve a higher MVP score" and "The agent teammates (RLHG) proactively considered my in in-game needs, assisted me in building advantages, and provided the resources required". Other participants provided feedback on the Wukong agent, stating that "The agent (Wukong) brought me less enjoyable experience, as they rarely paid attention my gameplay behavior" and "My agent teammates (Wukong) frequently left me feeling isolated and undervalued". Such voluntary feedback from participants can offer insights into the effectiveness of the RLHG approach.
E BROADER IMPACTS
The main goal of our research is to develop better technologies that enable artificial agents to assist humans more effectively in complex environments. This technology has the potential to benefit the research community and various real-world applications, such as friendly assistive robots.
To the research community. Games, as the microcosm of real-world problems, have been widely used as testbeds to evaluate the performance of Artificial Intelligence (AI) techniques for decades. And MOBA poses a great challenge to the AI community, especially the field of Human-Agent Collaboration (HAC). Even though the existing MOBA-game AI systems have achieved or exceeded human-level performance, they mainly focus on how to compete rather than how to assist humans, leaving HAC in complex environments still be investigated. To this end, this paper introduces a learning methodology to train agents to assist humans and enhance humans' ability to achieve goals in complex human-agent teaming environments. We herewith expect that this work can provide inspiration for the human enhancement and human assistance in various Al research.
To the real-world applications. Firstly, AI has found real-world applications in games and is changing the way MOBA game designers work. For example, for PVE (player vs environment) teaching mode, introducing AI with human enhancement into the game is a low-cost method to increase the interest of novice players. Secondly, our method can be directly applied to any pre- trained agent, and only needs to be fine-tuned with human gain to change it from apathetic to human-enhann cumannenhanced. It could be directly applied to assistive robotics, such as enhancing the safety of humans collaboration with industrial robotic arms.
However, we should take into consideration the possibility of human goals being harmful. Therefore, agents are optimized for harmful goals, this can have negative social impacts, as with all advanced AI techniques, such as AlphaStar (Vinyals et al., 2019), OpenAl Five (OpenAI et al., 2019) and Cicero (FAIR et al., 2022). To avoid these problems, we increase regulation and scrutiny during technological research and development ensure that human goals do not negatively impact society, In addition, we recommend that when releasing the pre-trained agent model, some restrictions need to be added for fine-tuning. such enhancing the safety of humans.
F FUTURE WORK
In some application scenarios, human experience may be difficult to quantify directly but can be implicitly modeled through human feedback, such as in Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ibarz al., 2018; Ouyang et al., 2022). Therefore, combining RLHF with RLHG for more general collaborative tasks is a promising research direction. Besides, our approach and experiments only consider the scenario where multiple agents enhance one human. Another worthy research direction is how to simultancously enhance the experience of multiple humans with diverse behaviors.
29


Reviews:
[{"review_id": "G262cxjdff", "reviewer": "Reviewer_zdWR", "summary": "This paper proposes a human-centered modeling scheme to improve the design of human experiments in cooperative human-AI games. Specifically, the authors train two networks: a value network to estimate the expected human return in achieving human goals and a gain network to estimate the expected positive gain of human return. Empirically, they evaluate their proposed method in the game Honor of Kings, and find that the RLHG agent provides participants with a better gaming experience, both in terms of objective performance and subjective preference.", "questions": "As weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's lies in its focus on human experiments in human-AI collaborative games, rather than the current popular AI-centered training scheme. This is a very important area of research, as human experience is essential in human-AI collaboration. Additionally, the paper conducts a comprehensive experimental evaluation of the proposed algorithms, and shows that they can provide a better human experience in terms of both objective performance and subjective preference.", "weaknesses": "Although the reviewer agrees that human experience is important in human-agent collaboration, it seems the novelty of this paper is limited. The proposed approach seems to simply add a human modeling network to the existing RL framework. \nThe details of the algorithms are also not very clear in the paper, and the reviewers suggest that Algorithm 1 should be moved to the main body of the paper to make it easier for readers to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBde5CUR1A", "reviewer": "Reviewer_uNSX", "summary": "* This paper proposes a very interesting problem in multi-agent RL that designing robot policies with the goal of providing the human a good experience. The paper proposes a new RL training procedure to obtain such policies and deployed them in the game of \"Honor of Kings\", in both simulation (4 robot + 1 simulated human playing against 4 robot + 1 simulated human), and in real game play (4 robot + 1 real human playing against 4 robot + 1 simulated human).\n\n* This paper separate the task reward from the human reward, which captures the human experience in the game. Both rewards are known to the system. The training procedure optimizes the robot policies to maximize the difference between the total human reward obtained compared to the total human reward obtained if the robot is just maximizing the task reward.", "questions": "* I think in multi-agent setting, the value function depends on both the robot policies, denoted by π or πθ, in this paper, and also the human policy πH. However, some notations in Sec.2.2 and Sec.3.1, 3.2 are only explicitly associated with one policy, rather than both the robot and human policies, which makes me confused.\n    * E.g., in the 2nd line of the paragraph before Eq.1, it says V^πθ, where I am confused that what is the human policy inducing this value function.\n    * E.g., in Eq.1's last term - expectation under πH, the total return Gt should depend on the robot policy and the human policy, but the expectation is only taken under πH, not (πH,πθ).\n        * Similar to Eq.2's last term - E_πH[...].\n        * Similar to the line below Eq.3: E_πH[...].\n        * Simlar to Sec.3.2's 2nd paragraph's ∆π(s, a)=E_πH[...].\n* I am also not sure how to get the last inequality in page number 4.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* The formulation and algorithm are sound.\n* The result is rich. In both simulation and real game play, the authors demonstrate that the proposed algorithm achieved higher human experience, without sacrificing a lot of task performance.", "weaknesses": "* I have some questions about the notations. I think the notation in Sec.2.2 and Sec.3.1, 3.2 are a bit confusing and hard to follow. See my questions in the \"questions\" box.\n* I think Dec-POMDP usually assumes that all agents maximize the same reward function. However, in this paper, the human and robot are not just optimizing the same reward function. Instead, it seems that the human is just a fixed policy that is not optimizing anything. The robot is a policy that optimizes the human gains ∆πH in Sec.3.2. Maybe it is worthy to clarify this.\n    * Due to this conflict, it did confuse me in the beginning and took me some time to figure out that in this paper, the human policy is fixed (e.g., learned from data), while the robot policy is learned as the \"best-response\" to the human policy. In other words, the human might not adapt to or might not be the best respond to the robot policy. It might be useful to make it clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PkPCheWpFA", "reviewer": "Reviewer_qZtz", "summary": "This work focusses on fine-tuning high-performing AI agents trained for commercial games to be more enjoyable as teammates for human players.  It has been observed that strong AI policies for team vs. team games can perform well as members of human-AI teams, but nonetheless be unenjoyable for human players due to a lack of perceived cooperation, or opportunities for the human to lead.  They propose to address this issue using \"RL from Human Gain\" or RLHG, a framework in which the AI's objective is to maximize the impact of the human's own actions on their reward (which may differ from the task-level reward).  They present a policy gradient update that maximizes a mixture of the one-step RLHG objective with the task-level reward.\n\nThey evaluate RLHG by fine-tuning the existing Wukong agent developed for the game \"Honor of Kings\".  Through a user-study with expert players they show that these players generally preferred to play on teams with RLHG-tuned agents rather than the high-performing Wukong agent.", "questions": "1. Were an experiments conducted using RLHG with the task reward and human reward being equivalent (solely optimizing human agency)?\n2. Do the HRE experiments correspond to maximizing the $\\alpha$-weighted mixture of human and task reward?\n3. Was the human-centric reward a pre-existing score in the game environment, or was it derived based on the survey of player preferences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "I see two main contributions from this paper:\n1. Highlighting the importance, in the use of RL for game AI, of maximizing human agency and enjoyment, rather than absolute performance\n2. The RL from human gain algorithm for learning policies that maximize human agency itself, rather than\n\nWhile conceptually simple, they are able to demonstrate empirically that the RLHG algorithm can fine-tune high-performing models into ones that achieve high player satisfaction, while retaining much of their original performance.", "weaknesses": "My main concern with this work is that its technical presentation is currently unclear to the point that it is not possible to determine which algorithm is actually being evaluated, yielding two very different possible explanations for the empirical results.\n\nThe way RLHG is currently presented suggests that the algorithm attempts to select AI actions that maximize the difference between the advantage of the human's own actions under the AI's action, and their advantage under some baseline AI policy (this advantage captured by the term $\\hat{A}_H (s, a)$).  Working through the derivation of the RLHG update, however, it appears that RLHG may essentially be equivalent to optimizing the weighted combination of the human and task rewards $R + \\alpha R^H$.  This may simply be a misunderstanding of the notation used, but in either event the presentation needs to be improved for a final version of the paper.\n\nLooking at the RLHG policy gradient in equation 4, it appears that we can express the human gain advantage as: \n\n$$\n     \\hat{A}_H (s, a) = E_{\\pi^H} [ G^{H}_t \\vert s_t = s, s_t = a ] - V^{\\pi^{e}_{\\theta}, \\pi^{H}}_H\n$$\n\nThis is where the notation makes things unclear.  As I understand it, $V^{\\pi^{e}_{\\theta}, \\pi^{H}}_H$ is to be interpreted as a function of the state $s$ alone, in which case it would have no impact on the expectation of the policy gradient (though it would potentially reduce the variance).\n\nThe other interpretation, which is more consistent with the intuition behind the algorithm (and with equation 5), is that $V^{\\pi^{e}_{\\theta}, \\pi^{H}}_H$ is really a function of both $s$ and the AI's action $a$ (or equivalently, $V : S \\mapsto \\Re^A$).\n\nSome improvements to the presentation that could help clear things up:\n1. Specify the domain and range of the value functions in question\n2. Always condition value functions on their inputs ($V(s)$ instead of $V$)\n3. clearly distinguishing in the definitions between the AI actions and those taken by the human\n\nThis also raises another question about the relative importance of the \"human-gain\" objective vs. the distinct human reward function $R^H$ in the empirical results.  My understanding is that the baseline HRE fine-tuning process corresponds to simply maximizing the human's reward function, starting from the pre-trained Wukong agent.  The fact that this performs so poorly would suggest that the human-gain objective is really the critical factor.  Unfortunately the HRE agent is not sufficiently well-defined.  Explicitly writing down the gradient update or objective function for HRE agent would further improve the reader's understanding of RLHG.\n\nIf these issue could be clarified it would significantly impact my recommendation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)