SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Only  $s_t$  is considered to be constrained with PC loss in real implementation because the sampled batch  $L_b$  is too short to deal with the subsequent states in the same way. The reanalyze technique is used to prepare the learning targets for policy and value without enabling off-policy value correction. All target preparations are summarized in Algorithm 1.
Algorithm 1: Sample Preparation for GW-PCZero
Input: Replay buffer  $\mathcal{R}$ , Unrolled steps  $l$ , Batch size  $N$ .
Output:  $(\pi, z, t_{PC})$ .
1: Sample  $N$  unrolled sequences from  $\mathcal{R}$  and each sequence consists of  $l + 1$  consecutive states.
2: for each sampled sequence  $L_{i} = \{s_{t}^{i}, s_{t+1}^{i}, \dots, s_{t+1}^{i}\}$  do
3: Reanalyze policy target  $\pi$  by performing one iteration MCTS for each state in  $L_{i}$
4: Recalculate value target  $z$  by bootstrapping in Eq. (9) for each state in  $L_{i}$ .
5: Estimate PC target  $t_{PC}(s_t)$  according to Algorithm 2.
6: end for
7: return Tuple  $(\pi, z, t_{PC})$
4.3 Weighting the PC target
The estimation of  $t_{PC}(s_t)$  in Eq. (15) takes the mean of all  $f$  values, treating all states equally. However, the states far away from  $s_t$  are less reliable than the ones near  $s_t$ , which is mainly caused by two operations, as shown in Figure 1. One is the exploration uncertainty during the iteration process with the environment. As  $s_{t + i}$  is sampled from the policy distribution injected with noise, it may not be the optimal choice. The other more important reason is policy reanalyzing. As illustrated in
the right of Figure 1,  $s_{t+1}$  is the following state of  $s_t$  in the real iteration sequence collected with an older policy  $\pi^{old}$ , but the optimal choice of the latest reanalyzed policy  $\pi^{current}$  may be another state  $s_{t+1}^*$ . According to Definition 3.1, PC requires that all states should be on the same optimal path, and it will be violated due to the above two operations. Especially, the probability that  $s_{t+i}$  and  $s_t$  are not in the same optimal path grows as  $i$  increases.
Figure 1: Left: exploration uncertainty when playing the game. Action is sampled from  $\pi$  and  $\varepsilon$  is a Gaussian noise. Right: difference between collected sequence  $\{s_{t + i}\}$  and current reanalyzed policy's optimal choice  $\{s_{t + i}^*\}$ .
$$
\pi \left(s _ {t + 1} ^ {i}\right) = N \left(s _ {t + 1} ^ {i}\right) / N \left(s _ {t}\right)
$$
$$
\varepsilon_ {t + 1} ^ {i} \sim \mathcal {N} (\mu , \sigma^ {2})
$$
State collected with  $\pi^{old}$
Optimal state under  $\pi^{current}$
In order to mitigate the high variance brought by the above uncertainty, a weighting mechanism is devised to give a larger discount to farther states. Specifically,  $t_{PC}(s_t)$  is calculated by a weighted summation and the weight  $w_i$  decreases linearly with index  $i$ , i.e.,
$$
w _ {i} = \max  \left\{c _ {b} - c _ {a} \times i, 0 \right\}, \tag {16}
$$
where  $c_{a}$  and  $c_{b}$  are two constants. While there are numerous algorithms available for weighting, the linear weighting method employed in this paper is simple and highly interpretable, which deserves further investigation in future research. The PC target  $t_{PC}$  is calculated as:
$$
t _ {P C} \left(s _ {t}\right) = \sum_ {i = 0} ^ {l} \widetilde {w} _ {i} \left[ \sum_ {j = 1} ^ {i} r _ {t + j} + v \left(s _ {t + i; \theta}\right) \right], \tag {17}
$$
where  $\widetilde{w}_i = w_i / \sum_{i=0}^{l} w_i$  is the normalized weight. The details are summarized in Algorithm 2.
Algorithm 2: Weighted PC target  $t_{PC}$  estimation
Input:  $S = \{s_t, r_{t+1}, s_{t+1}, \dots, s_{t+l}\}$ , value function  $v(s)$ .
Output: Target  $t_{PC}(s_t)$
1: Initialize the value summation  $T \gets 0$  and weight summation  $W \gets 0$ .
2: for each state  $s_{t + i}$  in S do
3: Calculate weight  $w_{i}$  for state  $s_{t + i}$  according to Eq. (16).
$$
4: \quad T \leftarrow T + w _ {i} \times \left[ \sum_ {j = 1} ^ {i} r _ {t + j} + v \left(s _ {t + i}\right) \right], W \leftarrow W + w _ {i}
$$
6: return target  $t_{PC}(s_t) \gets T / W$ .
4.4 The coefficient for the PC loss
PC is incorporated into the policy-network learning as a regularization term in Eq. (2), and there is a non-negative coefficient  $\lambda$  to be determined. GW-PCZero degenerates back to a variant of EfficientZero (without MCTS value target correction) when  $\lambda = 0$ . In general, an appropriate  $\lambda$  is required to balance between maximizing the accumulated reward and satisfying the PC constraint. According to the empirical experience of PCZero on Hex game [33], it has been found that PC improves the generalization ability for testing while sacrificing the prediction accuracy for training. For games with low complexity, it is easy to generate all possible states for training and  $\lambda$  should be small to improve the training accuracy. For games with high complexity, the generalization ability should be paid more attention by giving a larger  $\lambda$ . See the details in Appendix 3.
Therefore,  $\lambda$  is adjusted with the game complexity, which is evaluated by two criteria. One is the action space size  $|\mathcal{A}|$ , which determines the number of brunch factors for tree search. The other is the difficulty for the computer program to master, which is approximately evaluated by  $D_{m} = \text{MuZero Score}/\text{Human Score}$  in Atari games, where  $\text{MuZero}$  is trained without  $100k$  limitation. In the experiment, if  $|\mathcal{A}| < 18$ ,  $\lambda$  is set to be 0.2. For games with  $|\mathcal{A}| \geq 18$ , if  $D_{m}$  is larger than 100, indicating that this game is easier to master,  $\lambda$  will be set to be 0.35. Otherwise, this game is difficult to master, and  $\lambda = 0.4$ .
Experiments are conducted on Atari  $100k$  benchmark containing 26 games [8]. Only  $100k$  interaction steps are allowed to be collected, i.e.,  $400k$  frames based on a frameskip of 4, which is roughly the number of experiences played by a human in two hours. The mean performance of human players was collected [17], which is used to evaluate the model's performance against the baseline of the random player's score. The human normalized score of a learned agent is defined as  $(Score_{agent} - Score_{random}) / (Score_{human} - Score_{random})$ .
Table 1: Comparison of different algorithms. The computational cost is measured by the number of MCTS executions.
Value correction	Weighting	Adjustable ca	Frames	Train steps	Cost	Score
EfficientZero	✓	N/A	N/A	100k	120k	61.54M	1.943
EfficientZero†	×	N/A	N/A	100k	60k	15.46M	1.218
G-PCZero	×	×	×	100k	60k	15.46M	1.378
GW-PCZero	×	✓	×	100k	60k	15.46M	1.980
GW-PCZero(ca)	×	✓	✓	100k	60k	15.46M	2.066
Table 2: Scores on the Atari  $100k$  benchmark tested with 32 evaluation seeds. Benchmark methods include SimPLE [8], CURL[12], DrQ [31], and SPR+reset [16]. EfficientZero $^{\dagger}$  [32] is retrained under the same computational resources as GW-PCZero.
Game	Human	SimPLe	CURL	DrQ	SPR+retsets	EfficientZero†	G-PCZero	GW-PCZero
Alien	7127.7	616.9	558.2	771.2	911.2	850.6	761.3	699.7
Amidar	1719.5	88.0	142.1	102.8	201.7	60.6	91.6	97.0
Assault	742.0	527.2	600.6	452.4	953.0	994.8	1093.3	1224.1
Asterix	8503.3	1128.3	734.5	603.5	1005.8	17734.4	15712.5	14771.9
BankHeist	753.1	34.2	131.6	168.9	547.0	276.9	280.9	207.2
BattleZone	37187.5	5184.4	14870.0	12954.0	8821.2	15875.0	13875.0	13500.0
Boxing	12.1	9.1	1.2	6.0	32.2	28.2	11.8	41.6
Breakout	30.5	16.4	4.9	16.1	23.4	366.7	402.7	450.0
ChopperCmd	7387.8	1246.9	1058.5	780.3	1380.6	818.8	1221.9	1150.0
CrazyClimber	35829.4	62583.6	12146.5	20516.5	28936.2	8059.4	8031.3	9734.4
DemonAttack	1971.0	208.1	817.6	1113.4	2778.0	7940.8	15163.4	24074.1
Freeway	29.6	20.3	26.7	9.8	18.0	0.0	0.0	0.0
Frostbite	4334.7	254.7	1181.3	331.1	1834.3	229.1	245.6	249.7
Gopher	2412.5	771.0	669.3	636.3	930.4	1325.6	851.3	1286.9
Hero	30826.4	2656.6	6279.3	3736.3	6735.6	7537.2	9958.1	8171.3
Jamesbond	302.8	125.3	471.0	236.0	415.7	300.0	298.4	525.0
Kangaroo	3035.0	323.1	872.5	940.6	2190.6	525.0	1012.5	262.5
Krull	2665.5	4539.9	4229.6	4018.1	4772.4	3818.5	4233.0	7782.0
KungFuMaster	22736.3	17527.2	14307.8	9111.0	14682.1	8956.3	10059.4	20543.8
MsPacman	6951.6	1480.0	1465.5	960.5	1324.6	967.5	784.7	1594.1
Pong	14.6	12.8	-16.5	-8.5	-9.0	15.6	19.8	19.8
PrivateEye	69571.3	58.3	218.4	-13.6	82.2	0.0	100.0	96.9
Qbert	13455.0	1288.8	1042.4	854.4	3955.3	8120.3	8281.3	13651.6
RoadRunner	7845.0	5640.6	5661.0	8895.1	13088.2	3443.7	9262.5	16809.4
Seaquest	42054.7	683.3	384.5	301.2	655.6	478.1	838.1	768.1
UpNDown	11693.2	3350.3	2955.2	3180.8	30185.0	7592.5	6129.4	12344.7
Normed Mean	1.000	0.443	0.381	0.357	0.901	1.212	1.378	1.980
# >Human	0	2	2	2	7	7	7	11
Table 3: Scores on the Atari  $100k$  benchmark with different  $c_{a}$  in Eq. 16.  $c_{a} = 0$  means no weighting.  $c_{a} = 0.1$  has been used in Table 2.
ca	Seaquest	Asterix	Boxing	Alien	BattleZone
0.0	838.1	15712.5	11.8	761.3	13875.0
0.05	1215.6	22131.3	22.0	568.1	14281.3
0.1	768.1	14771.9	41.6	699.7	13500.0
0.2	776.9	14950.0	53.8	982.5	16812.5
ca	Breakout	Qbert	KungFuMaster	Krull	Kangaroo
0.0	402.7	8281.3	10059.4	4233.0	1012.5
0.05	443.9	12568.8	12575.0	6874.9	787.5
0.1	450.0	13651.6	20543.8	7782.0	262.5
0.2	396.7	11269.5	10990.6	2182.5	206.3
In EfficientZero [32], MCTS consumes the majority of computing resources, much more than network updating. The number of MCTS is determined by two factors: data collection, which requires  $100k$  MCTS for  $100k$  steps, and data reanalysis.  $120k$  updating steps are performed with a batch size of 256. A total of 30.72 million samples are reanalyzed, with each sample requiring two MCTS runs: one for policy re-estimation and the other for MCTS value correction. Thus, a total of 61.44 million MCTS runs are used to prepare the training data, and the computational cost for data generation is relatively negligible. Therefore, the limit of  $100k$  interaction steps does not fully constrain the computational resources used, and the total number of MCTS runs should be considered. Due to the reanalysis operation, EfficientZero requires a total of 61.54 million MCTS runs, which accounts for the majority of the computational resources consumed.
In this paper, we evaluate our GW-PCZero's learning efficiency under a more difficult setting than the original EfficientZero [32]. As summarized in Table 1, the updating steps are limited to  $60k$ , and MCTS off-policy value correction is disabled. Models in our experiment only consume  $25\%$  computational resources of the original EfficientZero. G-PCZero is included for comparisons by using the unweighted computation of the PC target in Eq. (15). EfficientZero† is retrained under identical conditions. Experiments are conducted on 4 NVIDIA Tesla A100 GPUs with 16 CPU cores. We set  $c_{b} = 1.0$  and  $c_{a} = 0.1$  in Eq. (16). Totally 32 of different random seeds are used. Other hyperparameter settings are the same as EfficientZero, as summarized in Appendix 5.
The mean testing results are reported in Table 2. GW-PCZero performs better than EfficientZero† in 19 out of all 26 Atari games. The mean of the human normalized score of GW-PCZero achieves  $198.0\%$ , much higher than EfficientZero†'s  $121.2\%$ . Moreover, GW-PCZero slightly outperforms the original EfficientZero [32], which achieved a score of  $194.3\%$ . It should be noted that GW-PCZero only consumes  $25\%$  computational resources of the original EfficientZero, as shown in Table 1. Experiment results demonstrate the effectiveness of our GW-PCZero, and indicate that PC improves the learning efficiency remarkably in Atari games. Notice that GW-PCZero performs poorly in several games requiring great exploratory, such as Freeway, probably due to the quick convergence of PC. Moreover, examples of learning curves of GW-PCZero and EfficientZero* on Atari games are displayed in Appendix 6. Analysis of test variances is included in Appendix 4.
GW-PCZero outperforms G-PCZero in 15 games and achieves a significantly higher mean normalized score. This ablation study indicates that the weighting mechanism is important to provide a more accurate target for PC learning. The performance is greatly improved by considering different uncertainty levels in the states along the path while calculating the mean  $f$  values. It is critical to observe that G-PCZero outperforms EfficientZero† in 16 Atari games, and its mean score of  $137.8\%$  is slightly higher than that of EfficientZero†, indicating that PC improves the model's performance even without the weighting mechanism. We also employ the weighting mechanism in PCZero [33] and conduct experiments on board games. The game performance exhibits improvement as well. Details are included in Appendix 7.
Path consistency requires that  $f$  values on the optimal path should be identical. The selection of the optimal path from the collected interaction experiences is not only necessary but also presents a significant challenge. Due to the difficulty of definitively determining the optimality of a particular path, one alternative approach is to strive for the collection of training paths that are near-optimal, thus
satisfying the requirement of PC. In on-policy reinforcement learning, game experiences collected through interactions between the updated policy and the environment improve gradually and approach a near-optimal level throughout the training process. Alternatively, near-optimal paths obtained from human experts or other high-level computer players can also fulfill the requirement of PC, allowing the model to be trained in an off-policy manner. Both aforementioned training approaches have been demonstrated to be effective in PCZero [33]. The critical factor for the success of PC lies in the selection of the optimal path, and is relatively unaffected by whether the path is collected through on-policy or off-policy methods. However, in many scenarios, the availability of near-optimal paths provided by experts is limited, and on-policy learning methods demonstrate low sample efficiency. Therefore, in addition to the aforementioned methods, it is crucial to investigate the training of PC using suboptimal paths. For example, game frames in Atari  $100k$  are collected using a poorly trained model in the early state, which are far from near-optimal. GW-PCZero implements a weighting mechanism to address the limitation of suboptimal paths by excluding unreliable nodes along the collected path from consideration, and experimental results in Table 2 demonstrate the effectiveness of this node filtering approach.
We further investigate the weighting mechanism in more detail. As shown in Eq. (16), linear decay is adopted from the initial value  $c_{b} = 1.0$ , with the decay rate controlled by  $c_{a}$ . A larger  $c_{a}$  gives less weight to the latter states, while  $c_{a} = 0$  represents the situation without weighting. The effects of  $c_{a}$  are illustrated in Table 3. The optimal  $c_{a}$  varies across different games, indicating that there is still room for improving the weighting strategy. On one hand, more reliable neighboring states should be considered to reduce the variance of the mean  $f$  value. On the other hand, unreliable neighbors increase uncertainty in the estimation of the mean  $f$  value.  $c_{a}$  is to achieve a trade-off between including more neighbors and reducing unreliability. Notice that when considering Asterix, Alien, BattleZone, and Kangaroo, GW-PCZero is inferior to EfficientZero in Table 2, but it becomes better than EfficientZero under appropriate values of  $c_{a}$ . When  $c_{a}$  is adjustable, GW-PCZero( $c_{a}$ ) achieves a mean score of 2.066, with only 2 out of 26 games performing worse than EfficientZero†. These results suggest that there is potential for further improvement in the weighting strategy and that more effort should be devoted to exploring this area in future research.
In this paper, we have proposed GW-PCZero to consider path consistency (PC) in a general setting and provided a theoretical analysis of the convergence property of MCTS with the PC constraint. GW-PCZero extends the previous PCZero from board games to more practical applications, for which the environment emits immediate rewards, such as Atari games. An uncertainty-aware weighting mechanism, which gives a larger discount to the farther states along the searching path, is devised to compute a more reliable learning target for PC. What's more, we have theoretically proved that MCTS is guaranteed to find the optimal solution under the constraint of PC. Experiments demonstrate that PC is beneficial to obtain strong artificial intelligence programs but consumes fewer computational resources. Neural-guided MCTS has been extensively utilized to address optimization problems in various practical applications, such as de novo drug molecular generation [18], organic molecule retrosynthesis [21] and so on. There is potential for the development of PC to contribute to addressing these critical problems by providing more robust value estimators. This work is still in the nascent stages without further applications related to people's daily lives currently and thus there is no immediate ethical or harmful social impacts.
This work was supported by the National Natural Science Foundation of China (62172273), and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102). Shikui Tu and Lei Xu are corresponding authors.
[1] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.
[22] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
[23] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140-1144, 2018.
[24] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
[25] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
[26] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[27] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
[28] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
[29] Lei Xu. Deep bidirectional intelligence: Alphazero, deep ia-search, deep ia-infer, and tpc causal learning. In Applied Informatics, volume 5, pages 1-38. Springer, 2018.
[30] Lei Xu, Pingfan Yan, and Tong Chang. Algorithm cnneim-a and its mean complexity. In Proc. of 2nd international conference on computers and applications. IEEE Press, Beijing, pages 494-499, 1987.
[31] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2020.
[32] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25488, 2021.
[33] Dengwei Zhao, Shikui Tu, and Lei Xu. Efficient learning for AlphaZero via path consistency. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26971-26981. PMLR, 17-23 Jul 2022.