Title: Make You Better: Reinforcement Learning from Human Gain
Venue: NeurIPS 2023
Accepted: no
Keywords: deep reinforcement learning

human enhancement

human-agent collaboration

game playing
Abstract: In human-agent collaboration tasks, it is essential to explore ways for developing assistive agents that can improve humans' performance in achieving their goals. In this paper, we propose the Reinforcement Learning from Human Gain (RLHG) approach, designed to effectively enhance human goal-achievement abilities in collaborative tasks with known human goals. Firstly, the RLHG method trains a value network to estimate primitive human performance in achieving goals. Subsequently, the RLHG method trains a gain network to estimate the positive gain of human performance in achieving goals when subjected to effective enhancement, in comparison to the primitive performance. The positive gains are used for guiding the agent to learn effective enhancement behaviors. Distinct from directly integrating human goal rewards into optimization objectives, the RLHG method largely mitigates the human-agent credit assignment issues encountered by agents in learning to enhance humans. We evaluate the RLHG agent in the widely popular Multi-player Online Battle Arena (MOBA) game, Honor of Kings, by conducting experiments in both simulated environments and real-world human-agent tests. Experimental results demonstrate that the RLHG agent effectively improves the goal-achievement performance of participants across varying levels.

Abstract

In human-agent collaboration tasks, it is essential to explore ways for developing assistive agents that can improve humans' performance in achieving their goals. In this paper, we propose the Reinforcement Learning from Human Gain (RLHG) approach, designed to effectively enhance human goal-achievement abilities in collaborative tasks with known human goals. Firstly, the RLHG method trains a value network to estimate primitive human performance in achieving goals. Subsequently, the RLHG method trains a gain network to estimate the positive gain of human performance in achieving goals when subjected to effective enhancement, in comparison to the primitive performance. The positive gains are used for guiding the agent to learn effective enhancement behaviors. Distinct from directly integrating human goal rewards into optimization objectives, the RLHG method largely mitigates the human-agent credit assignment issues encountered by agents in learning to enhance humans. We evaluate the RLHG agent in the widely popular Multi-player Online Battle Arena (MOBA) game, Honor of Kings , by conducting experiments in both simulated environments and real-world human-agent tests. Experimental results demonstrate that the RLHG agent effectively improves the goal-achievement performance of participants across varying levels.

1 Introduction

An intriguing research direction in the field of Artificial Intelligence (AI), particularly in the humanagent field, is how to effectively enhance human goal-achievement abilities within collaborative tasks. Human-Agent Collaboration (HAC) (Crandall et al. , 2018; Dafoe et al. , 2020) has gained significant attention from researchers, and numerous agents have been successfully developed to collaborate with humans in complex environments (Jaderberg et al. , 2019; Carroll et al. , 2019; Hu et al. , 2020; Strouse et al. , 2021; Bakhtin et al. , 2022; Gao et al. , 2023). However, as Amodei et al. (2016) stated, '[F]or an agent operating in a large, multifaceted environment, an objective function that focuses on only one aspect of the environment may implicitly express indifference over other aspects of the environment'. The current agents focus mainly on maximizing their own rewards to complete the task, less considering the role of their human partners, which potentially leads to behaviors that are inconsistent with human preferences (Fisac et al. , 2020; Alizadeh Alamdari et al. , 2022). For instance, consider the scenario depicted in Figure 1, where there is an agent and a human on either side of an obstacle. Only the agent is capable of pushing or pulling the obstacle once. Both the human and the agent share the same task goal, i.e., obtaining the coin, while the human needs the agent's assistance to get the coin. In this scenario, the HAC agent may push the obstacle to the human side and pass through to get the coin by itself. However, in a qualitative study (Cerny, 2015) on companion behavior, humans reported greater enjoyment of the game when AI assisted them more like a sidekick. Thus, the human may prefer that the agent plays a more assisting role by pulling the obstacle to its side, thereby enabling the human to get the coin. To advance AI techniques for the betterment of humanity, it is crucial to consider ways to assist humans in improving their goal-achievement abilities rather than replacing them outright (Wilson and Daugherty, 2018).

In complex collaborative environments, such as Multi-player Online Battle Arena (MOBA) 40 games (Silva and Chaimowicz, 2017), humans pursue multiple individual goals, such as achieving 41 higher MVP scores and experiencing more highlight moments, beyond simply winning the game to 42 enhance their gaming experience (see Figure 4 (c), our participant survey). When human goals are 43 aware, an intuitive approach to learning assistive agents would be to combine the agents' original 44 rewards with the human's goal rewards (Hadfield-Menell et al. , 2016; Najar and Chetouani, 2021; 45 Alizadeh Alamdari et al. , 2022). Nevertheless, directly incorporating the human's goal rewards may 46 cause negative consequences, such as human-agent credit assignment issues, i.e., human rewards 47 for achieving goals are assigned to non-assisting agents, which potentially leads the agent to learn 48 poor behaviors and forfeits its autonomy. When human goals are unknown, some studies attempt to 49 infer them from prior human behaviors using Bayesian Inference (BI) (Baker et al. , 2005; Foerster et 50 al. , 2019; Puig et al. , 2020; Wu et al. , 2021) and Inverse Reinforcement Learning (IRL) (Ng et al. , 51 2000; Ziebart et al. , 2008; Ho and Ermon, 2016). Other work introduces auxiliary rewards, such as 52 the human empowerment (Du et al. , 2020), i.e., the mutual information of human trajectories and 53 current state, for guiding agents to learn assistive behaviors. However, the diverse and noisy human 54 behaviors(Majumdar et al. , 2017) may be unrelated to actual human goals, leading agents to learn 55 assistance behaviors that are not aligned with human preferences. Moreover, in tasks where human 56 goals are known, these methods may not be as effective as explicitly modeling human goals (Du et 57 al. , 2020; Alizadeh Alamdari et al. , 2022). 58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

This paper focuses on the setting of known human goals in complex collaborative environments. Our key insight is that agents can enhance human goal-achievement abilities without compromising AI autonomy by learning from the human positive gains toward achieving goals under the agent's effective enhancement. We propose the Reinforcement Learning from Human Gain (RLHG) method, which aims to fine-tune a given pre-trained agent to be assistive in enhancing a given human model's performance in achieving specified goals. Specifically, the RLHG method involves two steps. Firstly, we determine the primitive performance of the human model in achieving goals. We train a value network to estimate the primitive human return in achieving goals with episodes collected by directly teaming the agent and the human to execute. Secondly, we train the agent to learn effective human enhancement behaviors. We train a gain network to estimate the positive gain of human return in achieving goals when subjected to effective enhancement, in comparison to the primitive performance. The agent is fine-tuned using the combination of its original advantage and the human-enhanced advantage calculated by the positive gains. The RLHG method can be seen as a plug-in that can be directly utilized to fine-tune any pre-trained agent to be assistive in human enhancement.

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

We conducted experiments in Honor of Kings (Wei et al. , 2022), one of the most popular MOBA games globally, which has received much attention from researchers lately (Ye et al. , 2020a,b,c; Gao et al. , 2021, 2023). We first evaluated the RLHG method in simulated environments, i.e., human model-agent tests. Our experimental results indicate that the RLHG agent is more effective than baseline agents in improving the human model goal-achievement performance. We further conducted real-world human-agent tests to verify the effectiveness of the RLHG agent. We tested the RLHG agent teaming up with different levels of participants. Our experimental results demonstrate that the RLHG agent could effectively improve the performance of general-level participants in achieving their individual goals to be close to those of high-level participants and that this enhancement can be generalized to different levels of participants. In general, our contributions are as follows:

We propose a novel insight to effectively enhance human abilities in achieving goals within collaborative tasks by training an assistive agent to learn from human positive gains.

We achieve our insight by proposing the RLHG algorithm and providing a practical implementation.

We validated the effectiveness of the RLHG method by conducting human-agent tests in the complex MOBA game Honor of Kings .

2 Problem Settings 88

2.1 Game Introduction 89

MOBA games, characterized by multi-agent cooperation and competition mechanisms, long time 90 horizons, enormous state-action spaces ( 10 20000 ), and imperfect information (OpenAI et al. , 2019; 91 Ye et al. , 2020a), have attracted much attention from researchers. Honor of Kings is a renowned 92 MOBA game played by two opposing teams on the same symmetrical map, each comprising five 93 players. The game environment depicted in Figure 2 comprises the main hero with peculiar skill 94 mechanisms and attributes, controlled by each player. The player can maneuver the hero's movement 95 using the bottom-left wheel (C.1) and release the hero's skills through the bottom-right buttons (C.2, 96 C.3). The player can view the local environment on the screen, the global environment on the top-left 97 mini-map (A), and access game states on the top-right dashboard (B). Players of each camp compete 98 for resources through team confrontation and collaboration, etc., with the task goal of winning the 99 game by destroying the opposing team's crystal. 100

Individual Goals

MVP Score

Low

High

Few

More

…

Task Goal

Game Victory

Highlight Moment

In-game Resource

Win

Win

Few

More

Bittersweet

Enjoyable

(a)

(b)

Main Hero

Legendary!

VICTORY

Pentakill!

254

C.1

2.2 Human-Agent Enhancement

We formulate the human enhancement problem in collaborative tasks as an extension of the DecPOMDP, which can be represented as a tuple < N, S , A , O , P, R, γ, π H , G H , R H > , where N denotes the number of agents. S denotes the space of global states. A = { A i , A H } i =1 ,...,N denotes the space of actions of N agents and a human to be enhanced, respectively. O = { O i , O H } i =1 ,...,N denotes the space of observations of N agents and the human, respectively. P : S × A → S and R : S × A → R denote the shared state transition probability function and reward function of N agents, respectively. γ ∈ [0 , 1) denotes the discount factor. π H ( a H | o H ) is the human policy, which cannot be directly accessible to the agent. G H = { g i } i =1 ,...,M denotes the human individual goals, where g i is a designated goal and M is the total number of individual goals. R H : S × A ×G H → R denotes the goal reward function of the human. In agent-only scenarios, the optimization objective is to maximize the expected return V π θ = E π θ [ G ] , where G = ∑ ∞ t =0 γ t R t is the discounted total rewards (OpenAI et al. , 2019; Ye et al. , 2020a). In human non-enhancement scenarios, the optimization objective is V π θ ,π H = E π θ ,π H [ G ] = ∑ a π θ ( a | o, π H ) E π H [ G ] (Carroll et al. , 2019; Strouse et al. , 2021). However, in human enhancement scenarios, the agent learns to enhance the human in achieving their goals G H . Therefore, the optimization objective can be formulated as:

where V π θ ,π H H = E π θ ,π H [ G H ] , G H = ∑ ∞ t =0 γ t R H t is the discounted total human goal rewards, and α is a balancing parameter. The agent's policy gradient can be formulated as:

where A = G -V π θ ,π H and A H = G H -V π θ ,π H H are the agent's original advantage and the human's enhanced advantage, respectively.

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

However, incorporating human rewards directly into the optimization objective may lead to negative 121 consequences, such as human-agent credit assignment issues. Intrinsically, humans possess the 122 primitive ability to achieve certain goals independently. Therefore, it is unnecessary to reward the 123 agent for assisting in goals that the human can easily achieve, as it potentially impacts the agent's 124 original behavior, resulting in losing its autonomy. In the subsequent section, we propose a novel 125 insight to achieve effective human enhancement by instead learning from the positive gains that the 126 human achieves goals better than his/her primitive performance. 127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

3 Reinforcement Learning from Human Gain

In this section, We present the RLHG method in detail. We start with describing the key insight in the RLHG method (Section 3.1). Then we implement our insights and present the RLHG algorithm (Section 3.2). We end by providing a practical implementation of the RLHG algorithm (Section 3.3).

3.1 Effective Human Enhancement

In the process of learning to enhance humans, agents explore three types of behaviors: effective enhancement, invalid enhancement, and negative enhancement. Intuitively, effective enhancement can help humans achieve their goals better than their primitive performance, invalid enhancement provides no benefits for humans in achieving their goals but also causes no negative impact, and negative enhancement hinders humans from achieving their goals. Our key insight is that agents are only encouraged to learn effective enhancement behaviors, which we refer to learn from positive gains . Formally, we denote the effective enhancement policy as π ef θ , the invalid enhancement policy as π in θ , and the negative enhancement policy as π ne θ . The agent's policy can be expressed as follows:

where π is a given pre-trained policy and V π,π H H is the primitive value of the human policy π H teaming with π to achieve goals. We use the ρ -function to represent the probability of exploring each policy, and we have ρ ( π ef θ ) + ρ ( π in θ ) + ρ ( π ne θ ) = 1 . Intuitively, the expected return of human goal-achievement under arbitrary enhancement is a lower bound of the expected return under effective enhancement, that is,

To ensure that the agent only learns effective enhancement behaviors, we replace the lower bound V π θ ,π H H with V π,π H H . Therefore, the agent's policy gradient 1 can be reformulated as:

where ̂ A H = ( G H -V π,π H H ) -Gain π ef θ ,π H and Gain π ef θ ,π H = V π ef θ ,π H H -V π,π H H is the expected of the effective enhancement benefit. We use Gain ω to denote an estimate of Gain π ef θ ,π H , which can be trained by minimizing the following loss function:

where I is an indicator function to filter invalid and negative enhancement samples and V ϕ is an estimate of V π,π H H .

3.2 The Algorithm

We achieve our insights and propose the RLHG algorithm as shown in Algorithm 1, which consists of two steps: the Human Primitive Value Estimation step and the Human Enhancement Training step.

Human Primitive Value Estimation: The RLHG algorithm initializes a value network V ϕ ( s ) , which is used to estimate the expected primitive human return for achieving G H in state s . V ϕ ( s ) is trained by minimizing the Temporal Difference (TD) errors (Sutton and Barto, 2018) with trajectory samples collected by teaming the agent π and the human π H to execute in a collaboration environment. Afterward, V ϕ ( s ) is frozen for subsequent human enhancement training.

Human Enhancement Training: The RLHG algorithm initializes the agent's policy network π θ and value network V ψ by conditioned on the human policy π H , respectively. The RLHG algorithm also initializes a value network Gain ω ( s ) , which is used to estimate the benefit value of the human return G H in state s under effective enhancement over V ϕ ( s ) . Gain ω ( s ) is trained by minimizing the loss function Eq. 4. The trajectory samples are also collected by teaming π θ and π H to execute in

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

the collaboration environment. The agent's policy network π θ is fine-tuned by the PPO (Schulman et al. , 2017) algorithm using the combination of the original advantage A and the human-enhanced advantage ̂ A H . The agent's value network V ψ is fine-tuned using the agent's original return G .

Algorithm 1 Reinforcement Learning from Human Gain (RLHG)

Require : Human policy network π H , human individual goals G H , agent policy network π , agent value network V , hyper-parameter α

Process :

1: Initialize human primitive value network V ϕ ; // Step I: Human Primitive Value Estimation

2: while not converged do

3: Collect human-agent team < π, π H > trajectories;

4: Compute human return G H for achieving goals G H ;

5: Update V ϕ ( s ) ← G H

6: end while

7: Initialize agent policy network π θ ( a | o, π H ) ← π , agent value network V ψ ( s, π H ) ← V , human gain network Gain ω ( s ) ; // Step II: Human Enhancement Training

8: while not converged do

9: Collect human-agent team < π θ , π H > trajectories;

10: Compute agent original return G and human return G H ;

11: Compute agent original advantage A = G -V ψ ( s, π H ) ;

12: Compute human-enhanced advantage ̂ A H = ( G H -V ϕ ( s )) -Gain ω ( s ) ;

13: Update agent policy network π θ ← A + α · ̂ A H ;

14: Update agent value network V ψ ( s, π H ) ← G ;

15: Update human gain network Gain ω ( s ) with Eq. 4

16: end while

3.3 Practical Implementation

We provide the overall training framework of the RLHG algorithm, as shown in Figure 3. We elaborate on the integral components of the RLHG framework, including the human model, the agent model, and the training details.

Human Model: The RLHG algorithm introduces a human model as a partner of the agent during the training process. The human model can be trained via Behavior Cloning (BC) (Bain and Sammut, 1995) or any Supervised Learning (SL) techniques (Ye et al. , 2020b), but this is not the focus of our concern. The RLHG algorithm aims to fine-tune a pre-trained agent to enhance a given human model.

Agent Model: Any pre-trained agent can be used within our framework. Since in many practical scenarios agents cannot directly access human policies, we input the observed human historical info h t = ( s H t -m , ..., s H t ) into an LSTM (Hochreiter and Schmidhuber, 1997) module to extract the human policy embedding, similar to Theory-of-Mind (ToM) (Rabinowitz et al. , 2018). The human policy embedding is fed into two extra value networks, i.e., V ϕ and Gain ω , and fused into the agent's original network. We use surgery techniques (Chen et al. , 2015; OpenAI et al. , 2019) to fuse the human policy embedding into the agent's original network, i.e. adding more randomly initialized units to an internal fully-connected layer. V ϕ ( h t ) and Gain ω ( h t ) output values estimate the human return for achieving goals without enhancement and the benefit under enhancement in state s t , respectively.

Training Details: The overall training framework of the RLHG algorithm is shown in Figure 3. Figure 3 (a) shows the training process of the human primitive value network V ϕ , in which the agent's policy network is frozen. V ϕ is trained by minimizing the TD errors. Figure 3 (b) shows the human enhancement training process, in which V ϕ is frozen. The agent's policy and value networks are trained using the PPO algorithm. Gain ω ( h t ) is trained by minimizing the loss function Eq. 4. we apply the absolute activation function to ensure that the gains are non-negative. In practical training, we found that only conducting human enhancement training has a certain negative impact on the agent's original ability to complete the task. Therefore, we introduce 1 -β % agent-only environment to maintain the agent's original ability and reserve β % human-agent environment to learn effective enhancement behaviors. These two environments can be easily controlled through the task gate, i.e., the task gate is set to 1 in the human-agent environment and 0 otherwise.

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

Environment

+

𝑠

!

𝑎

!

Observation

Human History Info

Human-Agent

𝑜

!

ℎ

!

= (𝑠

!

"

, 𝑠

!#$

"

, …,

𝑠

$

"

)

𝜋

(

𝑠

!

)

𝐺

!

"

𝑉

%

(ℎ

!

)

Observation

Human History Info

𝑜

!

ℎ

!

= (𝑠

!

"

, 𝑠

!#$

"

, …,

𝑠

$

"

)

Encoder

x

Gain

&

(ℎ

!

)

-

𝜋

'

(

𝑠

!

,ℎ

!

)

𝑉

(

(𝑠

!

,ℎ

!

)

Task Gate

abs

+

+

Agent-Only

Human-Agent

𝐺

!

"

𝑠

!

𝑎

!

𝐺

!

LSTM

𝑉

%

(ℎ

!

)

Encoder

LSTM

minus

Frozen

Architecture

(a)

(b)

𝛽%

1 - 𝛽%

00

11

00

4 Experiments

In this section, we evaluate the proposed RLHG method by conducting both simulated human modelagent tests and real-world human-agent tests in Honor of Kings . All experiments 1 were conducted in the 5v5 mode with a full hero pool (over 100 heroes, see Appendix A.2). Our demo videos and code are present at https://sites.google.com/view/rlhg-demo .

4.1 Experimental Setup

Environment Setup: To evaluate the performance of the RLHG agent, we conducted experiments in both the simulated environment, i.e., human model-agent game tests, and the real-world environment, i.e., human-agent game tests, as shown in Figure 4 (a) and (b), respectively. All game tests were played in a 5v5 mode, that is, 4 agents plus 1 human or human model team up against a fixed opponent team. To conduct our experiments, we communicated with the game provider and obtained testing authorization. The game provider assisted in recruiting 30 experienced participants with anonymized personal information, which comprised 15 high-level (top 1%) and 15 general-level (top30%) participants. We first did an IRB-approved participant survey on what top 5 goals participants want to achieve in-game, and the result is shown in Figure 4 (c). We can see that the top 5 goals voted the most by the 30 participants including the task goal, i.e., game victory, and 4 individual goals, i.e., high MVP score, high participation, more highlights, and more resources. We found that participants consistently rated the high MVP score individual goal most, even more than the task goal.

0

10

20

30

More Resources

More Highlights

High Participation

Game Victory *

High MVP Score

Votes

+

+

Goals

VS

VS

Human Model

+

Fixed Opponent

Fixed Opponent

Human

(a)

(b)

(c)

+

Training Setup: We were authorized to use the Wukong agent (Ye et al. , 2020a) as the pre-trained 215 agent and use the JueWu-SL agent (Ye et al. , 2020b) as the fixed human model. Note that both 216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

the Wukong agent and the JueWu-SL agent were developed at the same level as the high-level (top 1%) players. We adopted the top 4 individual goals as G for the pre-trained agent to enhance the human model. The corresponding goal reward function can be found in Appendix B.3. We trained the human primitive value network and fine-tune the agent until they converge for 12 and 40 hours, respectively, using a physical computer cluster with 49600 CPU cores and 288 NVIDIA V100 GPU cards. The batch size of each GPU is set to 256. The hyper-parameters α and β are set to 2 and 50, respectively. The step size and unit size of the LSTM module are set to 16 and 4096, respectively. Due to space constraints, detailed descriptions of the network structure and ablation studies on these hyper-parameters can be found in Appendix B.6 and Appendix C.1, respectively.

Baseline Setup: We compared the RLHG agent with two baseline agents: the Wukong agent (the pre-trained agent) and the Human Reward Enhancement (HRE) agent (the pre-trained agent learns to be assistive by incorporating the human's goal rewards). The human model-agent team (4 Wukong agents plus 1 human model) was adopted as the fixed opponent for all tests. For fair comparisons, both the HRE and RLHG agents are trained using the same goal reward function, and all common parameters and training resources are kept consistent. Results are reported over five random seeds.

4.2 Human Model-Agent Test

Directly evaluating agents with humans is expensive, which is not conducive to model selection and iteration. Instead, we build a simulated environment, i.e., human model-agent game tests, to evaluate agents, in which the human model, i.e., the JueWu-SL agent, teams up with different agents.

Wukong

HRE

RLHG

0.5

H

0.70

0.6

0.9

0.65

0.4

H

0.5

0.60

0.8

0.3

0.7

0.4

0.55

0.6

0.2

0.50

0.3

(a)WinRate

(b.1)MVP Score

(b.2) Highlight Times

(b.3)ParticipationRate(b.4)Resource Quantity

(c)FollowRate

Figure 5 shows the results of the human model on different game goals, including the top 4 individual goals and the task goal, i.e., the Win Rate, after teaming up with different agents. From Figure 5 (b), we can observe that both the RLHG agent and the HRE agent significantly enhance the performance of the human model in achieving the top 4 individual goals, and the RLHG agent has achieved the best enhancement effect on most of the individual goals. However, as shown in Figure 5 (a), the HRE agent drops significantly on the task goal. We observed the actual performance of the HRE agent teamed with the human model and found that the HRE agent did many unreasonable behaviors. For example, to assist the human model in achieving the goals of Participation Rate and Highlight Times, the HRE agent had been following the human model throughout the entire game, such excessive following behaviors greatly affect its original ability to complete the task and lead to a decreased Win Rate. This can also be reflected in Figure 5(c), in which the HRE agent has the highest Follow-Rate metric. Although the Follow-Rate of the RLHG agent has also increased, we observed that most of the following behaviors of the RLHG agent can effectively assist the human model. We also found that the Win Rate of the RLHG agent decreased slightly, which is in line with expectations because the RLHG agent made certain sacrifices to the task goal while enhancing humans in achieving their individual goals. In practical applications, we implemented an adaptive adjustment mechanism by simply utilizing the agent's original value network to measure the degree of completing the task goal and setting the task gate to 1 (enhancing the human) when the original value is above the specified threshold ξ , and to 0 (completing the task) otherwise. The threshold ξ depends on the human preference, i.e. the relative importance of the task goal and the human's individual goals. We verify the effectiveness of the adaptive adjustment mechanism in Appendix C.2.

4.3 Human-Agent Test

In this section, we conduct online experiments to examine whether the RLHG agent can effectively enhance human participants (We did not compare the HRE agent, since the HRE agent learned lots

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

of unreasonable behaviors, resulting in a low Win Rate). We used a within-participant design for the experiment: each participant teams up with four agents. This design allowed us to evaluate both objective performances as well as subjective preferences. All participants read detailed guidelines and provided informed consent before the testing. Each participant tested 20 matches. After each test, participants reported their preference over their agent teammates. For fair comparisons, participants were not told the type of their agent teammates. See Appendix D for additional experimental details, including experimental design, result analysis, and ethical review.

We first compare the objective performance of the participants on different goal-achievement metrics after teaming up with different agents. Table 1 and Table 2 demonstrate the results of high-level and general-level participants, respectively. We see that both high-level and general-level participants had significantly improved their performance on all top 4 individual goals after teaming up with the RLHG agent. Notably, the RLHG agent effectively improves the performance of general-level participants in achieving individual goals even better than the primitive performance of high-level participants. We also notice that the Win Rate of the participants decreased when they teamed up with the RLHG agent, which is consistent with the results in the simulated environment. However, we find in the subsequent subjective preference statistics that the improvement of Gaming Experience brought by the enhancement outweighs the negative impact of the decrease in Win Rate.

Wukong

RLHG

5

Preference

4

3

3

2

2

2

2

General-level

High-level

General-level

High-level

General-level

High-level

General-level

High-level

(a)BehavioralRationality

(b)EnhancementDegree

(c) Gaming Experience

(d)OverallPreference

We then compare the subjective preference metrics, i.e., the Behavioral Rationality, the Enhancement Degree, the Gaming Experience, and the Overall Preference, reported by participants over their agent teammates, as shown in Figure 6. We find that most participants showed great interest in the RLHG agent, and they believed that the RLHG agent's enhancement behaviors were more reasonable than that of the Wukong agent, and the RLHG agent's enhancement behaviors brought them a better gaming experience. A high-level participant commented on the RLHG agent "The agent frequently helps me do what I want to do, and this feeling is amazing." In general, participants were satisfied with the RLHG agent and gave higher scores in the Overall Preference metric (Figure 6 (d)).

285

4.4 Case Study

To better understand how the RLHG agent effectively enhances participants, we visualize the values 286 predicted by the gain network in two test scenarios where participants benefitted from the RLHG 287 agent's assistance, as illustrated in Figure 7. In the first scenario (Figure 7 (a)), the RLHG agent 288 successfully assisted the participant in achieving the highlight goal, whereas the Wukong agent 289 disregards the participant, leading to a failure in achieving the highlight goal. The visualization 290 (Figure 7 (b)) of the gain network illustrates that the gain of the RLHG agent, when accompanying 291 the participant, is positive, reaching the maximum when the participant achieved the highlight goal. 292 In the second scenario (Figure 7 (c)), the RLHG agent actively relinquishes the acquisition of the 293 monster resource, enabling the participant to successfully achieve the resource goal. Conversely, the 294 Wukong agent competes with the participant for the monster resource, resulting in the participant's 295 failure to achieve the resource goal. The visualization (Figure 7 (d)) of the gain network also reveals 296 that the gain of the RLHG agent's behavior - actively forgoing the monster resource, is positive, with 297 the gain peaking when the participant achieved the resource goal. These results indicate that the 298 RLHG agent learns effective enhancement behaviors through the guidance of the gain network. 299

(T1) Human wants Monster.

Human gets Monster.

(T2) Agent leaves Monster to Human.

(T2) Agent competes with Human for Monster.

Agent gets Monster.

First Defeat

Human achieves Highlight.

(T1) Human goes to Top Lane.

Human does nothing.

(T2) Agent ignores Human.

(T2) Agent follows and assists Human.

Wukong

RLHG

Wukong

RLHG

(a)

(b)

(c)

(d)

T1

T2

Primitive Value

Enhanced Value

Gain

T1

T2

a

&

Primitive Value

Enhanced Value

Gain

T1

T2

5 Discussion and Conclusion

Summary. In this work, we introduced the Reinforcement Learning from Human Gain method, dubbed RLHG, designed to effectively enhance human goal-achievement abilities within collaborative tasks. The RLHG method first trains a value network to estimate the primitive performance of humans in achieving goals. Subsequently, the RLHG method trains a gain network to estimate the positive gain of human performance in achieving goals under effective enhancement over that of the primitive. The positive gains are used for guiding the agent to learn effective enhancement behaviors. The RLHG method can be regarded as a continual learning plug-in that can be directly utilized to fine-tune any pre-trained agent to be assistive in human enhancement. The experimental results in Honor of Kings demonstrate that the RLHG agent effectively improves the performance of general-level participants in achieving their individual goals to be close to those of high-level participants and that this enhancement is generalizable across participants at different levels.

Limitations and Future Work. In this work, we only focus on the setting of known human goals. But for many practical complex applications, human goals may be difficult to define and formalize, and the goal reward function needs to be inferred using Inverse Reinforcement Learning (IRL) (Ng et al. , 2000; Ziebart et al. , 2008; Ho and Ermon, 2016) or Reinforcement Learning from Human Feedback (RLHF) (Christiano et al. , 2017; Ibarz et al. , 2018; Ouyang et al. , 2022) techniques. Future work can combine the RLHG method with goal inference methods to solve complex scenarios where human goals are unknown. Besides, our method and experiments only consider the scenario where multiple agents enhance one human. Another worthy research direction is how to simultaneously enhance multiple humans with diverse behaviors.

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

References

Parand Alizadeh Alamdari, Toryn Q Klassen, Rodrigo Toro Icarte, and Sheila A McIlraith. Be considerate: Avoiding negative side effects in reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pages 18-26, 2022.

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 , 2016.

Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15 , pages 103-129, 1995.

Chris Baker, Rebecca Saxe, and Joshua Tenenbaum. Bayesian models of human action understanding. Advances in Neural Information Processing Systems , 18, 2005.

Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, and Noam Brown. Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning. arXiv preprint arXiv:2210.05492 , 2022.

Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems , 32, 2019.

Martin Cerny. Sarah and sally: Creating a likeable and competent ai sidekick for a videogame. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume 11, pages 2-8, 2015.

Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641 , 2015.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems , 30, 2017.

Jacob W Crandall, Mayada Oudah, Fatimah Ishowo-Oloko, Sherief Abdallah, Jean-François Bonnefon, Manuel Cebrian, Azim Shariff, Michael A Goodrich, and Iyad Rahwan. Cooperating with machines. Nature Communications , 9(1):233, 2018.

Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08630 , 2020.

Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave: Assistance via empowerment. Advances in Neural Information Processing Systems , 33:4560-4571, 2020.

Jaime F Fisac, Monica A Gates, Jessica B Hamrick, Chang Liu, Dylan Hadfield-Menell, Malayandi Palaniappan, Dhruv Malik, S Shankar Sastry, Thomas L Griffiths, and Anca D Dragan. Pragmaticpedagogic value alignment. In Robotics Research: The 18th International Symposium ISRR , pages 49-57. Springer, 2020.

Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. In International Conference on Machine Learning , pages 1942-1951. PMLR, 2019.

Yiming Gao, Bei Shi, Xueying Du, Liang Wang, Guangwei Chen, Zhenjie Lian, Fuhao Qiu, Guoan Han, Weixuan Wang, Deheng Ye, et al. Learning diverse policies in moba games via macro-goals. Advances in Neural Information Processing Systems , 34:16171-16182, 2021.

Yiming Gao, Feiyu Liu, Liang Wang, Zhenjie Lian, Weixuan Wang, Siqin Li, Xianliang Wang, Xianhan Zeng, Rundong Wang, Jiawei Wang, et al. Towards effective and interpretable human-agent collaboration in moba games: A communication perspective. arXiv preprint arXiv:2304.11632 , 2023.

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in Neural Information Processing Systems , 29, 2016.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9(8):17351780, 1997.

Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. 'other-play' for zero-shot coordination. In International Conference on Machine Learning , pages 4399-4410. PMLR, 2020.

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in Neural Information Processing Systems , 31, 2018.

Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Humanlevel performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019.

Anirudha Majumdar, Sumeet Singh, Ajay Mandlekar, and Marco Pavone. Risk-sensitive inverse reinforcement learning via coherent risk models. In Robotics: Science and Systems , volume 16, page 117, 2017.

Anis Najar and Mohamed Chetouani. Reinforcement learning with human advice: A survey. Frontiers in Robotics and AI , 8:584075, 2021.

Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning , volume 1, page 2, 2000.

OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:2773027744, 2022.

Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890 , 2020.

Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International Conference on Machine Learning , pages 4218-4227. PMLR, 2018.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Victor do Nascimento Silva and Luiz Chaimowicz. Moba: A new arena for game ai. arXiv preprint arXiv:1705.10443 , 2017.

DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems , 34, 2021.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

Hua Wei, Jingxiao Chen, Xiyang Ji, Hongyang Qin, Minwen Deng, Siqin Li, Liang Wang, Weinan Zhang, Yong Yu, Liu Linc, et al. Honor of kings arena: An environment for generalization in competitive reinforcement learning. Advances in Neural Information Processing Systems , 35:11881-11892, 2022.

H James Wilson and Paul R Daugherty. Collaborative intelligence: Humans and ai are joining forces. Harvard Business Review , 96(4):114-123, 2018.

Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science , 13(2):414-432, 2021.

Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems , 33:621-632, 2020.

Deheng Ye, Guibin Chen, Peilin Zhao, Fuhao Qiu, Bo Yuan, Wen Zhang, Sheng Chen, Mingfei Sun, Xiaoqian Li, Siqin Li, et al. Supervised learning achieves human-level performance in moba games: A case study of honor of kings. IEEE Transactions on Neural Networks and Learning Systems , 2020.

Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6672-6679, 2020.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new reinforcement learning objective, dubbed Reinforcement Learning from Human Gain (RLHG), that explicitly incorporates an understanding of human performance with an intervention into the objective function. They show that training with this added component improves outcomes in a MOBA, both on the overall objective of winning and on subobjective related to satisfaction.\n\nSTRENGTHS:\nThey present an important research problem (ML systems working as collaborators) and make a good attempt at overcoming it.\nThey did experiments with human participants.\nThe domain they trained models in is non-trivial and showing results shows good ability\n\nWEAKNESSES:\nI don't believe the main claim. The authors say that by explicitly adding their more complex models of human wants/behaviours they can get better performance, but they don't compare against a model that attempts to optimize for those things directly. A perfect RL agent that has correct values for the different objectives should be able to learn the optimal policy without using their proposed more complicated methods. They only test against a model that is trained to win, and a model that is trained on short term value optimization. How does the method compare to another model trained with similar data and objectives? The lack of comparison makes this feel like they shows that PPO and deep learning work, not that there new methods are better.\n\nQUESTIONS:\n\"compromising AI autonomy\" (line 61), what does this mean? Isn't the whole point of this method to shape the agent's actions?\nWhere did you get 10^20000 for the state space? (line 91)\nYou don't need to specify that Honor of Kings is a MOBA and that it is (renowned|complex|popular|etc.) each time you reference it\nDec-POMDP is never defined\nLine 215 \"We were authorized\" by who?\n\nLIMITATIONS:\nThe authors do not explain why their decomposition of the loss/training loop is _theoretically_ better than any other, they only provided some _empirical_ evidence to suggest it.\nThe authors only discuss the RL literature in their framing/discussion. In other domains, such as recomender systems, the issue of over optimizing for a single metric to the detriment of other objectives is a major area of research. Consider looking over some data science papers from KDD as a starting point", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method called Reinforcement Learning from Human Gain (RLHG) to effectively enhance human goal-achievement abilities in collaborative tasks with known human goals. The paper evaluates the RLHG agent in the widely popular Multi-player Online Battle Arena (MOBA) game, Honor of Kings, by conducting experiments in both simulated environments and real-world human-agent tests.\n\nSTRENGTHS:\nThe problem setting considered by the paper is tightly connected to some real-world problems (e.g., assistive agents in MOBA games). \nExperiments are performed in a real-world application (Honor of King).\n\nWEAKNESSES:\nIt is difficult to evaluate the major contribution of the paper (the two-step training process) because\n1) the major contribution of the paper, as the authors claimed in the paper, is orthogonal to many of the complications in the paper (e.g., multiple agents, multiple goals, partial observability). These complications are not contributions and they make it hard to understand the contribution of the paper clearly. Maybe the authors added them because their experiments are in Honor of Kings?\n2) probably because the paper focuses too much on these complications, the paper fails to explain why RLHG is a good idea and provides clear evidence. For example, why do the authors propose estimating primitive human performance rather than primitive human+agent performance? What is making estimating primitive human performance helpful? Can this idea be used in environments without humans?\n3) the writing of the paper is unsatisfying. I was completely lost when reading the paper. Please see Questions for my questions about the paper.\n4) in experiments, the new method achieved a worse winning rate compared with the baseline method. I can understand this performance drop given that there are improvements regarding other metrics. What should I learn from this indeterminate result?\n\nQUESTIONS:\nSection 1:\n\"rather than replacing them outright\" why would you talk about this? How is this related to the previous text?\n\"The RLHG method can be seen as a plug-in that can be directly utilized to fine-tune any pre-trained agent to be assistive in human enhancement.\" Do you literally mean ANY pre-trained agent?\n\"human goals\". \"human goals\" sounds too broad and vague. What do you actually mean here? \n\"AI autonomy\". ???\n\"such as human-agent credit assignment issues, i.e., human rewards for achieving goals are assigned to non-assisting agents, which potentially leads the agent to learn poor behaviors and forfeits its autonomy.\" Why is this an issue? In RL you can have environments that emit rewards even if the agent does nothing.\n\nSection 2:\nCould you explain how the agents interact with the human? \nDo you consider one human? \nDo you consider a finite set of global states, actions, and observations?\nHow are global states used?\nDo you assume that the policy only depends on the current observation?\nWhat do ''scenarios'' mean? \npi_\\theta is not defined, theta is not defined.\nWhat do R_t and R^H_t mean?\nWhat do Eπθ [G], and EπH [G] mean?\nIt is important to distinguish between definition and equality. In this paper, \"=\" is used for both purposes.\n\"The agent’s policy gradient can be formulated as\". I know what \"policy gradient\" means in classic RL. But to be honest I am not sure what \"policy gradient\" means here. How do you derive the following equation? \n\n\nSection 3:\nWhat does Equation 2 mean? What does \"the probability of exploring each policy\" mean?\nWhy do you separate human rewards from the agent's rewards, given that you are going to combine them together eventually?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Reinforcement Learning from Human Gain, an RL algorithm that explicitly optimizes for enhancing human abilities in cooperative human-AI settings. Given a predefined set of human goals, the main approach first learns a value network to estimate the primitive human performance at achieving said goals. Then, a secondary Gain network is trained to estimate the enhancement the human return receives under interactions with the cooperative agent. The cooperative policy is trained with a combination of a traditional agent value network and the proposed human gain-based value network. The authors test the RLHG framework in a cooperative game and find that across experiments with real humans, the RLHG agent is preferred over an agent without the human gain objective, despite having a lower overall win rate.\n\nSTRENGTHS:\n- The problem setting is interesting and relevant, and the proposed solution of optimizing for human gain is intuitive — it makes sense that a cooperative agent should account for improvements in human behaviours, rather than having the agent directly optimize for its own reward or what it perceives human rewards to be (which may reduce human enjoyment and overall autonomy in a task, as noted by the authors).\n- The experiments use a complex multiagent task and test both human models and real human participants. The results show a significant improvement in human preference for the RLHG agent across the predefined goals as well as various subjective metrics.\n\nWEAKNESSES:\n- The method seems sensitive to the choice of partner $\\pi$ while collecting the primitive human episodes. If the initial partner is already very good, will its success be attributed to the human? And vice versa -- if the partner is very bad, would that lead to a false representation of the base human skill? The paper would be strengthened with additional studies on how sensitive the method is to different pretrained policies.\n- The method relies on knowing the set of human goals beforehand. It would be interesting to have additional analyses of how the method is affected in the case where some of the goals are missing or misspecified, which is more representative of a real-world scenario.\n\nQUESTIONS:\nWhat about scenarios where we don’t know human goals beforehand? This method relies on being able to accurately measure human rewards/goals which may not always be feasible (ill-defined goal spaces, human preferences changing over time). The authors comment on combining RLHG with goal inference methods, but those methods may still be faulty or difficult to learn. Could this method be applied to more general metrics of 'gain'-- e.g. increasing empowerment in the general assistance case?\n\nLIMITATIONS:\nThe authors adequately discuss the limitations of this work. Additional discussion on societal impacts would be helpful, as these agents are trained to interact with people directly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the fine-tuning of a pre-trained agent to assist and enhance the performance of a given human model in achieving specific goals. The authors assume access to a human model and a pre-trained agent. The authors propose a two-step approach.\n\n1. The human model's initial performance is determined by training a value network to estimate its effectiveness in goal attainment using episodes generated through joint execution with the agent.\n\n2. The is trained agent to learn effective behaviors for enhancing the human model's performance using a gain network that estimates the improvement in human return when compared to the initial performance.\n\nThe algorithm is evaluated on a Multi-player Online Battle Arena (MOBA) game.\n\nSTRENGTHS:\nThe idea of developing algorithms to assist humans to solve tasks is interesting and of great practical interest, and this paper makes headway in this direction. The authors conduct extensive experiments which includes evaluating using real players to test their algorithm in a game.\n\nWEAKNESSES:\n1. The algorithmic contribution in this paper appears to be relatively modest, as it mainly builds upon the existing Proximal Policy Optimization (PPO) approach. The authors introduce a gain function, which essentially computes an advantage by comparing it to another state-dependent baseline ( $V_\\phi(s)$)\n \n2. The assumption of having a human model is justifiable; however, the strong reliance on assuming knowledge of human goals, in my opinion, limits the direct applicability of this research (as acknowledged in the limitations section).\n\n3. There is a lot of notational ambiguity in the paper (Section 2.2), which makes reading a little hard. For example, \n \n3a. Advantage function generally depends on state/observation and action. In this setting, the Advantage is independent of both. \n\n3b. Is V value of a state or the infinite horizon discounted reward? It is unclear as its used in both contexts. \n\n3c. G is used as return-to-go, which should be a state dependent function. \n\n4. I do not understand the exact need for the gain network. What would happen if in line 12 of the algorithm, you drop the - Gain(s) part? This essentially means that you are computing advantage with respect to the human primitive baseline. \n\n5. The authors have invested significant effort and computational resources in conducting their experiments, making it extremely challenging to reproduce or recreate such experiments due to the demanding compute requirements. Although this does not diminish the value of their work, it would be beneficial if the authors could incorporate simpler environments into their experimental setup. This addition would aid in evaluating the algorithm's performance and further validate its quality.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors discuss limitations and future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}