Title: Agents Explore the Environment Beyond Good Actions to Improve Their Model for Better Decisions
Venue: NeurIPS 2023
Accepted: no
Keywords: decision making

artificial intelligence

reinforcement learning

curiosity

Gumbel MuZero

Java
Abstract: Improving the decision-making capabilities of agents is a key challenge on the road to artificial intelligence. To improve the planning skills needed to make good decisions, MuZero's agent combines prediction by a network model and planning by a tree search using the predictions. MuZero's learning process can fail when predictions are poor but planning requires them. We use this as an impetus to get the agent to explore parts of the decision tree in the environment that it otherwise would not explore. The agent achieves this, first by normal planning to come up with an improved policy. Second, it randomly deviates from this policy at the beginning of each training episode. And third, it switches back to the improved policy at a random time step to experience the rewards from the environment associated with the improved policy, which is the basis for learning the correct value expectation. The simple board game Tic-Tac-Toe is used to illustrate how this approach can improve the agent's decision-making ability. The source code, written entirely in Java, is available at <<a github url>>.

Abstract

Improving the decision-making capabilities of agents is a key challenge on the road to artificial intelligence [25]. To improve the planning skills needed to make good decisions, MuZero's agent [17, 10, 1, 18, 13, 7] combines prediction by a network model and planning by a tree search using the predictions. MuZero's learning process can fail when predictions are poor but planning requires them [28]. We use this as an impetus to get the agent to explore parts of the decision tree in the environment that it otherwise would not explore. The agent achieves this, first by normal planning to come up with an improved policy [7]. Second, it randomly deviates from this policy at the beginning of each training episode. And third, it switches back to the improved policy at a random time step to experience the rewards from the environment associated with the improved policy, which is the basis for learning the correct value expectation. The simple board game Tic-TacToe is used to illustrate how this approach can improve the agent's decision-making ability. The source code, written entirely in Java, is available at «a github url».

1 Introduction

Areinforcement learning agent has a simple interface to its environment [26, 25]: It partially observes the environment, acts, and receives rewards (Figure 1).

Despite this simplicity, it is hypothesised [23] that intelligence, and its associated abilities, can be understood as subserving the maximisation of reward .

Following this idea, MuZero [17] achieved a new state-of-the-art, outperforming all previous algorithms on the Atari suite and matching the superhuman performance of its predecessor AlphaZero at Go, Chess and Shogi. The MuZero agent learned acting through self-play, without even knowing the rules of the game - strictly following the agent-environment interface.

The agent's mind combines fast predictions from a neural network model with slow algorithmic 24 planning. This is similar to the way humans use fast intuitive and slow rational thinking [11]. 25

Despite MuZero's successes, its learning procedure can fail if the value prediction is poor where 26 27

planning needs it. It has recently been shown how an amateur-level agent can beat KataGo [29, 30], a state-of-the-art open-source Go implementation based on MuZero's predecessor AlphaZero, by 28

leading KataGo to parts of the decision tree that it would never visit in any self-play training 29 games [28]. 30

We use this as an impetus to make the agent curious about parts of the decision tree for which it 31 otherwise gains little or no experience in the environment. We do not claim to provide a solution that 32

solves all related problems, especially not the motivation example. 33

planning - "rationale"

finding an improved

policy for the next action

action

decision making

deciding the next action

observation

reward

legal actions

environment

agent

experience

memory of episodes

such as games

model - "intuition"

state representation,

next state generation,

reward prediction,

value prediction,

policy prediction

black

box

decision making

uses the

improved policy

planning

is based

on the model

model learns

from

experience

Reanalyse

is an additional

learning option

Since the agent in this approach is actively seeking new experiences to feed into its model, we call 34

planning process, but not necessarily on a one-to-one basis. So when we structure the agent, we 46

add a decision making component that is responsible for deciding the next action, Figure 1. This 47

responsibility includes, in particular, adding curiosity. With this structuring, we hope to contribute to 48

the cross-disciplinary quest for a common model of the intelligent decision maker [25]. 49

50

51

We also investigate two other cases with small contributions from us, arriving at three cases where we contribute - all three about the role of randomness:

52

53

54

55

Additional randomness after planning Use of the hybrid policy introduced here in the training context.

Additional randomness before planning In AlphaZero and MuZero, a Dirichlet noise was added to the prior probabilities in the root node when entering the tree search to aid exploration.

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

It was removed in Gumbel MuZero since it was not necessary to improve the policy with the model fixed. However, we use Dirichlet noise for the following heuristic reason: it adds a force toward choosing actions without unfair preference if the actions would not differ in value under a perfect strategy. If no force is added, one such action may be favoured by the agent, potentially preventing the agent from gaining experience from the parts of the decision tree after the unfavourable actions. This can lead to a worse model, a worse planning result, and therefore worse decisions. Another argument for avoiding unwarranted bias is to be stable against potential future changes in the environment that would favour an action other than the one the agent has learned to choose. We are aware that changing the policy with Dirichlet noise may cost some inference steps from the planning budget.

Less randomness during planning in eager playout Gumbel MuZero enters the planning for training playouts with the model's policy and draws from this policy - technically introducing a Gumbel value to achieve drawing without replacement. For the training context, this ensures that all root actions are considered exactly according to the existing knowledge of the agent. For an eager playout context, the situation is different. When making a decision only once, it can be beneficial for the agent to decide eagerly - like changing the temperature from 1 to 0. With this in mind, we also examine the playout case T=0 by setting the Gumbel value to 0 .

We show for the simple board game Tic-Tac-Toe that these three contributions improve the decisions made by the agent. We use confidence intervals at the 99% confidence level. In addition, we provide experimental examples to support our interpretation of how the improvements through using the hybrid policy and through using the Dirichlet noise occur - in these cases without proving statistical significance.

A limitation of this work is that we do not prove that we can reproduce all the results obtained by applying MuZero, in particular to the board games Go, Chess, Shogi and the Atari game suite.

Another limitation is that we do not show the application to the Reanalyse [17, 18, 31] loop here.

2 Recent Historical Background

AlphaGo [20] was the first AI engine to beat the best human player in a full-sized game of Go in March 2016. It used value networks to evaluate board positions and policy networks to select moves. The networks were trained using a combination of supervised learning from human expert games, and reinforcement learning from self-play games. The reinforcement learning used a tree search, which combines Monte Carlo simulation with value and policy networks.

AlphaGo Zero [21] eliminated the need to train with external input games. Thus, AlphaZero [22] generalised the AlphaGo algorithm and applied it to the games of Chess and Shogi. A major improvement to the algorithm was the continuous updating of the network.

In 2020, MuZero [17] has eliminated the need for a resettable simulator. Instead, MuZero learns a model of the environment to the extent necessary for its in-mind planning. It extends AlphaZero's successful application of the classic board games Go, Chess and Shogi to 57 Atari games. MuZero Unplugged [18] allows the agent to learn by re-analysing previously experienced episodes in mind.

Sampled Muzero [10] extends MuZero to domains with arbitrarily complex action spaces by planning over sampled actions. Stochastic MuZero [1] extends MuZero's deterministic model to a stochastic model that incorporates after states. It is demonstrated in the games 2048 and Backgammon.

EfficientZero [31], based on MuZero Unplugged [18] and SPR [19] achieved above-average human performance on Atari games with only two hours of real-time gaming experience. This experience efficiency was a milestone. Main contributions are (1) Self-Supervised Consistency Loss , (2) End-ToEnd Prediction of the Value Prefix , (3) Model-Based Off-Policy Correction . EfficientZero's source code is available on GitHub.

While MuZero's planning step produces an asymptotic policy improvement when many steps are used to unfold the decision tree, Gumbel MuZero [7, 6] introduced a planning algorithm that could improve the policy for any budget of unfolding steps - using a given model. The source code for the tree search is available on GitHub.

As a commercially relevant use case, MuZero has been applied to video stream compression [13]. 106 And as the first extension of AlphaZero to mathematics, AlphaTensor [8] demonstrates the ability to 107 accelerate the process of algorithmic discovery by finding faster matrix multiplication algorithms. 108 The open-source community has applied the AlphaZero and MuZero algorithms to various projects. 109 Notable examples in the field of board games are Leela Chess Zero [14] and KataGo [29, 30]. 110

111

112

113

114

The existence of open-source implementations encouraged the search for weaknesses in the agents. It was shown how adversarial policies could beat professional-level KataGo agents [28] using a strategy that an amateur player could follow. The main idea of the strategy is to lead the KataGo agent into areas of the decision tree where it has a poor value premise and therefore makes weak decisions.

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

3 Related Work

Finding an appropriate trade-off between exploration and exploitation is a core challenge in reinforcement learning [26]. By building a model that includes dynamics, as in MuZero [17], the magnitude of this challenge has increased because there are two worlds on stage: The environment as the real world and the model as an in-mind world. Gumbel MuZero [7] brought a planning algorithm that monotonically improves the policy with any budget of recurrent inference steps within MuZero's given in-mind world.

AlphaZero [22] and MuZero [17] add a Dirichlet noise to the model's policy predictions before starting the tree search in their planning step to ensure exploration.

The off-policy maximum entropy deep reinforcement learning algorithm SAC [9] uses the entropy of the policy times a temperature factor as an additional reward. Adding such an additional reward falls into the category of curious about known unknowns as this intrinsic reward is derived from the agent's policy.

After planning, MuZero [17] uses a temperature parameter T to vary between T = 1 for exploration and T = 0 for exploitation following AlphaZero's [22] approach for board games. For Atari games, this is done for all moves, not just the first few moves as in board games. The temperature is lowered as a function of the number of training steps of the network, thereby shifting the planning policy from exploration to exploitation.

Go-Exploit [27] based on AlphaZero samples the starting state of its self-play trajectories from an archive of states of interest . This approach can only be used if the environment interaction allows episodes to start from any state.

4 What This Work Builds Upon

This work builds on MuZero [17]. For our examples, we use the case of non-intermediate rewards from the environment. For the planning component and the model base we use Gumbel MuZero [7]. The model is extended for Self-Supervised Consistency Loss from EfficientZero [31]. The resulting model is presented in Appendix A.

5 Illustrating the Need for Improvement of the Agent

The agent's need to explore the decision tree beyond good actions can be illustrated by the simple game of Tic-Tac-Toe [3].

In Tic-Tac-Toe, the optimal outcome for both players is a draw. Figure 2 shows such a game.

time

Suppose an agent takes the role of both players - self-play - and only makes perfect moves in the 145 environment. Then he would never observe from the environment what could happen after a bad 146 move, e.g. after the first bad move shown in Figure 3. 147

time

after a bad action

Suppose such an agent takes the role of player x and plays against another player o. If a player o makes a bad move, the agent may not be able to take advantage of it and win. Instead, the agent might make a bad move as in Figure 3 and lose.

To observe more than the world of perfect actions, the agent must deviate from the perfect game when it acts in the environment during training. This could be achieved by separating the search for an optimised policy for the next action from the decision of what to do next. During training the decision component shown in Figure 1 could deviate from its optimised policy to get to novel parts of the decision tree and finish from there according to its optimised policy.

6 Agent Improvements

Two of the three contributions of the paper mentioned in the introduction are described here in more detail - supported by small proofs in the Appendix C.

6.1 Exploring Using a Hybrid Policy

Suppose we have a normal policy P normal and an exploring policy P explore . Also, suppose the model is to be trained using P normal . In a playout with P explore = P normal there would be an off-policy-issue for the value target [18]. To avoid this problem, the playout could be done with a hybrid policy P hybrid , starting with P explore and switching to P normal at a random time t startNormal before the expected end of the episode t end .

̸

...

...

The value target after t startNormal could then be set up just as without exploration as the value 165 information propagates backwards in time during training and is therefore not influenced by what 166 happened before t startNormal . But the value target before t startNormal would be set to keep its 167 existing value. Therefore, the value function would only learn from the normal policy . 168

We concretise P hybrid in two steps. In the first step, we specify P exploring as a drawing from a 169 probability distribution 170

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

with a temperature of T > 1 . 1 p normal is used for the models policy training target. 171

In the second step, we concretise p normal to be the improved policy of Gumbel MuZero derived from 172 the completed Q-values in the notation of Gumbel MuZero [7]. Using equation 21 in Appendix C.1 173 we get 174

The value target for the non-intermediate reward case is then given by 175

where r measured t end is the reward returned by the environment and v initialInference is the value v 0 t produced by the model version that is used when acting in the environment. This ensures that the value for this model version is not forced but later model versions taken from a buffer are forced towards v initialInference . See Appendix D.5 for why we did not use the improved value from planning as the target value.

6.2 Eager Playout without Gumbel noise

When planning according to Gumbel MuZero [7] in an eager playout, we can enter planning with a temperature 0 ≤ T ≤ 1 and still improve the decision by planning as shown in Appendix C.4. This is especially true for T → 0 which we achieve by setting the Gumbel value to 0 (see Appendix C.3).

7 Experiments - Game Tic-Tac-Toe

176

177

178

179

180

181

182

183

184

185

186

187

The paper's three contributions are tested on the game Tic-Tac-Toe. Appendix D informs about experimental details, Appendix E about the open source implementation used to run the experiments.

7.1 Training With and Without Exploring - All Games 188

0

500

1000

1500

2000

2500

3000

3500

4000

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

0

2

4

6

8

10

12

14

16

18

20

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

In Tic-Tac-Toe, the agent shows a large difference in the quality of decisions depending on whether 189 the exploration introduced in the previous section is turned on or off. While without this exploration 190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

the average of bad decisions for a trained model applied to all possible game decisions is 340 ± 80 after 1000 epochs, with exploration, it is 0 , 8 ± 0 , 3 (see Figure 5). This is an improvement by a factor 435 ± 190 .

7.2 Training With and Without Exploring - One Game

To gain insight into the cause of the effect seen in Figure 5, we now restrict our investigation to the particular game shown in Figure 3. Note that the second move in this particular game is already a bad move, so all the states behind that move would not occur in perfect play.

From the model versions trained without exploration, we look for a model version with which the agent would make the second bad decision in the situation of Figure 3. To find out why it does so, and why the agent using a model trained with exploration would not, we look at the value expectation of the model v τ t , since planning relies heavily on the quality of the value expectation. t denotes the time at which the initial inference starts and the in-mind time τ denotes the number of recurrent inference steps from that point. For a detailed definition of v τ t , see Appendix A.1.

When examining the value expectation of the model, it is not sufficient to consider the value expectation v 0 t immediately after the initial inference. Since the unfolding of the decision tree happens at the in-mind time τ , we need to look for all relevant in-mind times τ .

Therefore we examine

with 0 ≤ t ′ ≤ 18 and 0 ≤ t start ≤ 8 .

In Figure 6 we examine what the expectations of a particular model v τ t look like.

-1

-0,5

0

0,5

1

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

-1

-0,5

0

0,5

1

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

exploration-on

exploration-off

7.3 Playout With and Without Gumbel Noise - All Games 214

The playouts during the test in Figure 5 were done with the same Gumbel value as during training. 215 Playing out eagerly by setting the Gumbel value to 0 during planning reduces the number of bad 216 decisions. Figure 7 plots the difference number of bad decisions with Gumbel minus the number of bad 217 decisions without Gumbel . In the exploration-on case, we see an improvement by a factor 2 . 1 ± 0 . 3 . 218

0

20

40

60

80

100

120

140

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

7.4 Training With and Without Dirichlet Noise - All Games

219

Figure 5 is based on models trained with Dirichlet noise added to the policy entering the tree search. 220 We compare the decision performance of these models with models trained without Dirichlet noise, 221 leaving all other hyperparameters unchanged. In the exploration-on case, we see an improvement by 222 a factor 3 . 6 ± 1 . 2 using Dirichlet noise, see Figure 8 compared to Figure 5. Appendix D.7 speculates 223 why this is the case. 224

0

50

100

150

200

250

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

0

2

4

6

8

10

12

14

16

18

20

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

8 Discussion

We have introduced a new exploration approach to learn more from the environment. The new idea is to use two separate policies in a combined hybrid policy P hybrid , starting episodes with one for exploration P exploring - to take the agent to situations it would otherwise not experience - and randomly switching to the other policy P normal for finishing the episode with normal training. We derived P exploring from P normal by using a softmax temperature to introduce noise, set P normal to be the improved policy from Gumbel MuZero [7] and applied it to the game Tic-Tac-Toe. In these experiments, at a statistical confidence level of 99% , we observe a reduction in bad decisions by a factor of 435 ± 190 . A selective check suggests that the reason for the wrong decisions before introducing the new exploration approach lies in an incorrect value expectation of the agent's model.

In further experiments on the Tic-Tac-Toe game at a statistical confidence level of 99% , we observed that training with Dirichlet noise resulted in a network with better decision ability than training without Dirichlet noise and that playout of the trained network without Gumbel noise showed better decision ability than playout with Gumbel noise.

Having found large improvement factors for Tic-Tac-Toe, we should ask ourselves: have we reached state-of-the-art? For all situations where we could fully unfold the decision tree in a classical manner, we should consider perfect decisions as state-of-the-art. Therefore we have not fully reached the state-of-the-art for Tic-Tac-Toe.

It would be interesting to see how the approaches tested here for Tic-Tac-Toe pay off for Go, Chess, Shogi, and the Atari games on which MuZero was tested.

What could be done to improve the approach presented here?

Exploration Level When using the hybrid policy in the experiments, we used a fixed exploration temperature. In general, a higher temperature will distribute the agent's starting points for normal policy actions more randomly, whereas a lower temperature would keep the starting points closer to the best action path the agent could take. A strategy needs to be found on how to best set the exploration level to improve decisions.

Entropy reward Entropy as an intrinsic reward[9] could be added as a curiosity mechanism for known unknowns . We speculate that this - as one aspect - would remove the need for Dirichlet noise. The measurements from Appendix D.7 seem to point in this direction.

Reanalyse learning cycle The use of the Reanalyse learning cycle is a key feature in reducing the need for interaction with the environment. Extending the use of the techniques presented here to the Reanalyse learning cycle would therefore be useful. It would be of particular interest to Reanalyse the states that lead to rewarded actions, as the reward is a direct input from the environment and the source of the derived value. We speculate that this could improve the model's value predictions and thus the quality of decisions. A theoretically sound solution to the off-policy problem would be helpful in this regard.

Adversarial Exploration If the agent randomly deviates from the optimised strategy during exploration, it is unlikely to get into the situations the adversarial player put it into in Go [28]. Therefore, it may be necessary to devise an exploration strategy using an adversary as a counterpart in such games.

We hope to provide a useful technique for better learning the value function of the model as a basis for better planning-based decisions by the agent. It could serve as a starting point to help the agent become more curious.

References

Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. Planning in stochastic environments with a learned model. 2021.

Junjie Bai, Fang Lu, Ke Zhang, et al. Onnx: Open neural network exchange. https://github.com/ onnx/onnx , 2019.

József Beck. Combinatorial games: tic-tac-toe theory , volume 114. Cambridge University Press Cambridge, 2008.

Tyson R Browning and Ranga V Ramasesh. Reducing unwelcome surprises in project management. MIT 275 Sloan Management Review , 56(3):53-62, 2015. 276

Johannes Czech, Patrick Korus, and Kristian Kersting. Improving alphazero using monte-carlo graph 277 search. In Proceedings of the International Conference on Automated Planning and Scheduling , volume 31, 278 pages 103-111, 2021. 279

Ivo Danihelka. Planning and Policy Improvement . PhD thesis, UCL (University College London), 2023. 280

Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with 281 gumbel. In International Conference on Learning Representations , 2021. 282

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin 283 Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. 284 Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53, 285 2022. 286

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum 287 entropy deep reinforcement learning with a stochastic actor. In International conference on machine 288 learning , pages 1861-1870. PMLR, 2018. 289

Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, 290 and David Silver. Learning and planning in complex action spaces. pages 4476-4486, 2021. 291

Daniel Kahneman. Thinking, fast and slow . Macmillan, 2011. 292

Joseph Luft and Harry Ingham. The johari window. Human relations training news , 5(1):6-7, 1961. 293

Amol Mandhane, Anton Zhernov, Maribeth Rauh, Chenjie Gu, Miaosen Wang, Flora Xue, Wendy Shang, 294 Derek Pang, Rene Claus, Ching-Han Chiang, et al. Muzero with self-competition for rate control in vp9 295 video compression. arXiv preprint arXiv:2202.06626 , 2022. 296

Pascutto, Gian-Carlo and Linscott, Gary. Leela chess zero. URL http://lczero.org/ . 297

Mary Phuong and Marcus Hutter. Formal algorithms for transformers. arXiv preprint arXiv:2207.09238 , 298 2022. 299

Lutz Roeder. Netron, Visualizer for neural network, deep learning, and machine learning models, 12 2017. 300 URL https://github.com/lutzroeder/netron . 301

302

303

304

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems , 34, 2021.

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929 , 2020.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence , 299:103535, 2021.

Student. The probable error of a mean. Biometrika , 6(1):1-25, 1908. 322

Richard S Sutton. The quest for a common model of the intelligent decision maker. arXiv preprint 323 arXiv:2202.13252 , 2022. 324

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018. 325

Alexandre Trudeau and Michael Bowling. Targeted search control in alphazero for effective policy 326 improvement. arXiv preprint arXiv:2302.12359 , 2023. 327

Tony Tong Wang, Adam Gleave, Nora Belrose, Tom Tseng, Joseph Miller, Michael D Dennis, Yawen 328 Duan, Viktor Pogrebniak, Sergey Levine, and Stuart Russell. Adversarial policies beat professional-level 329 go ais. arXiv preprint arXiv:2211.00241 , 2022. 330

David J Wu. Katago. URL https://github.com/lightvector/KataGo/ . 331

David J Wu. Accelerating self-play learning in go. arXiv preprint arXiv:1902.10565 , 2019. 332

Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with 333 limited data. Advances in Neural Information Processing Systems , 34:25476-25488, 2021. 334

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nModel-based planning agents such as MuZero leverages a learned model of the environment dynamics to learn a policy to follow in the actual environment. By planning using the learned model dynamics, the agent may generate a stronger policy than when limited to only experience from the real environment. The drawback to this approach lies in the agent being forced into decision states it has not seen before. The agent's model will have incorrect value predictions for these states and the planning stage will result in weaker or sub-optmal policies in these states. This paper proposes an exploration scheme that ensures the agent explores sub-optimal regions of the search space to build better value estimates of the states within.\n\nSTRENGTHS:\n**Originality:** The ideas introduced in this paper have not been published before as far as I can tell.\n\n**Significance:** The algorithm proposed by the authors is nominally significant. It is a naive implementation of the explore-first-then-exploit type of algorithms.\n\n**Clarity:** The writing itself of the paper was fair but could use considerable improvement.\n\n**Quality:** It is hard to assess the strengths in the quality of this paper.\n\nWEAKNESSES:\nFirst off, I think it needs to be said that this is not a paper that can be improved upon with changes in the writing or presentation. The algorithm introduced is only a minor change to an already existing algorithm and it provides very little new insight.\n\nThat being said, I do want to encourage the authors to continue learning and working in the field. For the authors, I would recommend attending workshop or course on technical writing and building a stronger background in RL and planning.\n\nFor the submission specifically, I have a number of critiques. However, I will touch on only a few because as I've said this is not a paper that I think can be reworked and resubmitted:\n\n1) The introduction of the paper goes well into page 3. A strong problem statement and motivation section can go along way in emphasizing the importance of the work but the paper needs to focus more on the actual contributions.\n\n2) A similar note on the background and related works section. A lot of it deals with the history of the body of the work and not their relation to this particular paper. Furthermore, the background section does not provide the actual background needed for the paper. A good background section typically formalizes the problem state; clarifies any assumptions being made; defines notation and how the problem is being modelled; and provides a primer on the literature specifically needed to understand the rest of the paper. This submisison fails in those regards.\n\n3) It is difficult to understand the algorithm introduced because nothing has been defined.\n\n4) The experiments are limited a single small domain and does not give any indication of how the approach scales to larger ones.\n\n5) The experiments do not provide comparisons to other baselines, that is, to other algorithms that take the explore-first approach which are, in my opinion, a direct competitor to proposed algorithm.\n\nQUESTIONS:\n1) In equation 4, should this be the definition of $P_{normal}$ or are you defining $P_{exploring}$ twice?\n\n2) On line 40, you initially restrict $T> 1$ yet $T$ is set to values outside of this range. What is going on?\n\n3) Re Lines 45 - 49: Do agents not already make decisions about next actions?\n\n4) What is Gumbel MuZero's policy or how is it learned?\n\nLIMITATIONS:\nThe authors provide a list of limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method to improve the exploration of the MuZero agent in games. The authors propose an hybrid policy that mixes an exploratory policy and the optimized policy. The exploratory policy is meant to reduce brittleness of optimal policies.\nThe new method is demonstrated on Tic-Tac-Toe.\n\nSTRENGTHS:\nThe paper addresses maybe the most important issue in decision making. The authors' approach to encouraging exploration in the decision tree addresses an important problem that could have significant implications if successful. Adversarial examples in games and self-play is certainly an interesting domain to investigate this.\nThe motivation is good, and the authors also evaluate their approach on a small game. The idea is simple and makes sense. The appendix is also nice, and the source code is included. Prior work is also well researched.\nI like that the authors analyzed an example gameplay in Section 6.2\n\nWEAKNESSES:\nSection 2 Recent Historical Background\n\n- For relevant work on adversarial policies, I like that you included “Adversarial policies beat professional-level 330 go ais. arXiv preprint arXiv:2211.00241, 2022.”. Another paper that investigates this issue and I think should also be included is then “Timbers, Finbarr, et al. \"Approximate exploitability: learning a best response.\" Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 2022.”\n- When it comes down to AlphaZero extensions, a good addition in this section would be \"Player of games.\" arXiv preprint arXiv:2112.03178 (2021).”, as it extends AlphaZero to imperfect information games\n\n\nPaper sometimes does not read that well and individual sections sometimes feel bit disconnected (i.e. little connection to other sections in terms of semantics and flow). E.g. the word curiosity is only really used at the beginning, and my understanding is that authors really just mean exploration (and indeed exploration is used in the rest of the paper). Also note that \"curiosity\" clashes with prior work on Intrinsic motivation (Oudeyer et al., 2007; Schmidhuber, 1991, ...).\n\n\nThe idea is very simple, adding exploration at the beginning, which AlphaZero already does using different methods. I am not sure if the main evaluation in Section 7, namely Figure 5 is entirely fair (I might be wrong). The figure compares exploration-on vs exploration-off, but I don’t think that exploration-off in this case collapses to the “baseline” case? (e.g. MuZero or AlphaZero, as both do also force exploration at the beginning)\n\n\nThis issue connects to the second one, which is that the experiments are not that well explained. It is fine that the authors moved many details to the appendix (which contains a lot of interesting information), but the main body of the paper still should include enough for people to understand what is presented.\n\n\nBut my biggest issue is that the main challenge in balancing exploration in learning in games is when the approximator can’t really memorize the full game. If it could, than any exploration just works as we get to see all the states and memorize them (i.e. no generalization is needed). The game presented now is too small to see the effect, I suggest the authors run it on e.g. connect four. I believe the current experiments are simply not sufficient.\n\n\nFinally, I think the measure of whether the exploration helps or not should ultimately be exploitability rather than just a uniform measure of bad decision over all the game states.\n\nQUESTIONS:\nCan you elaborate why you chose MuZero rather than AlphaZero to investigate the exploration issue? What’s the point of learning the model in your case? \n\n\nCan you also measure exploitability for exploration on/off? I believe that is the ultimate metric in your case, and I also believe the suggested exploration would indeed help there.\n\n\nHow would the proposed method perform in more complex environments where the representational power of the estimator is limited with respect to the complexity of the full environment?\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
Inspired by the failure of KataGo against an amateur-level agent, the author proposes to use an additional randomization scheme to encourage the agent to explore the less experienced part of the decision tree. Such a randomization scheme allows the agent to randomly deviate from the planned policy, and then switch back to learn the correct value function. Empirically, they evaluate their method on Tic-Tac-Toe and justify the effectiveness of their method.

STRENGTHS:
1. The problem of exploring the less experienced part of the decision tree is interesting. 
2. The paper provides detailed related work.

WEAKNESSES:
The main concern of this paper is the contribution is marginal. 

The proposed method is a minor modification (a randomization scheme) to the existing method. This is not to say the investigated problem itself is not important. But with such a minor modification, the paper needs to provide sufficient evidence to prove its effectiveness, either by theoretical analysis or a large body of experiments. However, I am afraid the paper provides neither. For the evaluation, the paper only evaluates the proposed method in a very simple scenario, Tic-Tac-Toe, which is way too easy. More experiments on more complicated scenarios are definitely needed.  

The writing is unprofessional and should be improved substantially. For example, references should not be included in the abstract.

QUESTIONS:
n/a

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-06

comments: SUMMARY:
To improve prediction accuracy, this paper divides the training phase's policy into exploration and normal policies. The proposed algorithm is tested in the game of Tic-Tac-Toe, and different noise strategies are introduced for experimentation.

STRENGTHS:
The paper introduces a novel method for increasing policy exploration and experiments with different noise strategies for exploration.

WEAKNESSES:
1. The experiments lack persuasiveness as there is no comparison or theoretical analysis with advanced exploration algorithms.
    
2. The discrepancy between the mentioned "poor" predictions and the proposed exploration method is not fully addressed.
    
3. The setting of random time $t_{startNormal}$ is not clear.

QUESTIONS:
See the weakness part.

LIMITATIONS:
The application of the proposed algorithm in more complex tasks and its effectiveness compared to algorithms with other exploration techniques remains to be studied.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07