SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dengwei Zhao
zdwccc@sjtu.edu.cn
Shikui Tu
tushikui@sjtu.edu.cn
Lei Xu
Guangdong Institute of Intelligence Science and Technology
leixu@sjtu.edu.cn
Reinforcement learning with the help of neural-guided search consumes huge computational resources to achieve remarkable performance. Path consistency (PC), i.e.,  $f$  values on one optimal path should be identical, was previously imposed on MCTS by PCZero to improve the learning efficiency of AlphaZero. Not only PCZero still lacks a theoretical support but also considers merely board games. In this paper, PCZero is generalized into GW-PCZero for real applications with non-zero immediate reward. A weighting mechanism is introduced to reduce the variance caused by scouting's uncertainty on the  $f$  value estimation. For the first time, it is theoretically proved that neural-guided MCTS is guaranteed to find the optimal solution under the constraint of PC. Experiments are conducted on the Atari 100k benchmark with 26 games and GW-PCZero achieves  $198\%$  mean human performance, higher than the state-of-the-art EfficientZero's  $194\%$ , while consuming only  $25\%$  of the computational resources consumed by EfficientZero.
In recent years, combining neural networks with search algorithms has achieved remarkable success in Reinforcement Learning (RL) [22, 24, 23]. The application scenarios include not only board games but also knowledge-free real-world problems [19]. One critical challenge is that huge computational resources are required to collect interactive experiences in order to obtain reliable policy and value networks with remarkable performance. For example, 1000 third-generation TPUs were used to generate selfplay games in MuZero [19], roughly  $10 \sim 50$  years of experience per Atari game, which is unaffordable in regular situations. It is essential for an algorithm to achieve high performance with limited computational cost. Otherwise, it is difficult to generalize the algorithm to more applications.
For sequential decision problems, an interactive process between an agent and the environment is regarded as a playing path, which is collected to update the parameters of the policy and value network. In turn, the learned policy and value provide guidance for Monte Carlo Tree Search (MCTS) in determining the next action. This iterative process generates a large number of playing paths in order to obtain reliable policy and value estimators. The evaluation function  $f$  at state  $s_t$  is defined as the summation of  $g(s_t)$ , the accumulated reward from the initial state  $s_0$  to  $s_t$ , and  $h(s_t)$ , the expected return received after  $s_t$ , which is equivalent to the state value  $v(s_t)$ . That is, we have
$$
f \left(s _ {t}\right) = g \left(s _ {t}\right) + h \left(s _ {t}\right). \tag {1}
$$
To find the optimal path  $L^{*}$  that leads to the desired termination state, the  $\mathbf{A}^*$  search algorithm [5] establishes an optimality condition for  $f$  values, i.e.,  $f(s_{t}) = f(s_{0})$  for all  $s_t \in L^*$ , where  $s_0$  is
the initial state that definitely belongs to the optimal path. In 1987, CNNEIM-A [30] relied on this optimality to improve  $\mathrm{A}^*$  by estimating the  $f$ -values on  $L^{*}$  with the average of the  $f$ -values on the historical trajectory and on a lookahead scouting segment, plus revealing an implication that the  $f$ -values are regarded as i.i.d. samples from a same distribution. This idea has been adopted to equations.(8)&(9)&(10) in [29] with a regularization term added into the loss function of deep reinforcement learning such as AlphaZero [23]. The term aims at reducing the deviation of  $f(s_{t})$  from ones on  $L^{*}$  towards satisfying path consistency (PC) - "  $f$  values on one optimal path should be identical". Simplifying equation (10) in [29] by considering  $L_{2}$  error and ignoring weighting, recently PCZero [33] realised the PC constraint for the first time by considering
$$
\mathcal {L} (\theta) = \mathcal {L} _ {R L} (\theta) + \lambda \mathcal {L} _ {P C} (\theta), \tag {2}
$$
where  $\mathcal{L}_{RL}(\theta)$  is the loss function of AlphaZero,  $\lambda$  is a hyperparameter,  $\mathcal{L}_{PC}$  is implemented as  $(v(s_{t}) - \bar{v}(s_{t}))^{2}$ , the squared  $L_{2}$  deviation of the state value  $v(s_{t})$  from the learning target  $\bar{v}(s_{t})$ , which is the average state value along the estimated optimal path. Experiments on three board games have demonstrated that PCZero improves the learning efficiency of AlphaZero.
However, ignoring weighting by PCZero is oversimplified because deeper states along the search path have greater uncertainty due to the exploration randomness of self-play and policy reanalysis [19]. Thus, it is essential to take into account of the uncertainty levels in  $\mathcal{L}_{PC}$ . Also, PCZero has only been applied to board games, wherein the immediate reward is always zero until the game terminates. Thus,  $g(s_{t}) = 0$  holds for all states in PCZero, while PC has degenerated into the consistency of state values. It is unclear whether PC is effective generally as suggested by equation (8) in [29] to take  $g(s_{t})$  in consideration for applications with non-zero immediate rewards, such as Atari games. Moreover, there lacks a strong theoretical support for the effectiveness of PC, and the advantage conferred by the PC constraint has not been clearly explained.
This paper tackles the above problems by generalizing PCZero into GW-PCZero (short for Generalized Weighted PCZero), with the main contributions summarized as follows<sup>1</sup>.
- The application of PC is extended to scenarios where the environment emits immediate rewards, such as Atari games. Loss function in EfficientZero [32] is adopted as  $\mathcal{L}_{RL}(\theta)$  in Eq. (2), instead of using the loss function of AlphaZero like PCZero [33].  $\mathcal{L}_{PC}(\theta)$  ensures the consistency of  $f$  values in Eq (1) across the search path, rather than only maintaining the consistency of  $h$  value like PCZero.  $\bar{f}(s_t)$ , the mean of the  $f$  values of states along the path, is adopted as the learning target for  $\mathcal{L}_{PC}$ .
- An uncertainty-aware weighting mechanism is introduced for accurately estimating  $\bar{f}(s_t)$ . States farther away from  $s_t$  are considered less reliable and given larger discounts to reduce noise in  $\bar{f}(s_t)$ . Experiments are conducted on the Atari 100k benchmark with 26 games to evaluate GW-PCZero in diverse environments. Under the same computing resource consumption, our GW-PCZero achieves  $198\%$  mean human normalized performance, significantly outperforms the state-of-the-art EfficientZero's  $121\%$ . Additionally, GW-PCZero surpasses the original version of EfficientZero, which achieved a score of  $194\%$  [32], while utilizing only  $25\%$  of the computational resources required by EfficientZero.
- For the first time, it is theoretically proved that neural-guided MCTS is guaranteed to find the optimal solution under the constraint of PC, and that the optimal solution for minimizing  $\mathcal{L}_{PC}(s_t)$  is achieved by satisfying the multi-step temporal difference (TD) value estimation relationship between state  $s_t$  and its neighboring states in the same path. Instead of estimating state value independently, PC leads to a more reliable value estimator.
The Arcade Learning Environment (ALE) is an evaluation platform that contains plenty of Atari games, which is used to evaluate the general competency of AI agents [2]. Interaction steps are restricted to  $100k$ , which is roughly two hours of experience for human [14], to explore models' sample efficiency. SimPLe [8] further selects 26 games from ALE on the basis of being solvable with existing state-of-the-art reinforcement learning algorithms, which is adopted as the benchmark by many following works [9, 12, 20, 31, 13, 16]. So far, EfficientZero [32] is the only model that
<sup>1</sup>The source code is available at https://github.com/CMACH508/GW_PCZero.
has achieved super-human performance on Atari  $100k$ . Here, our generalized PC is applied to EfficientZero and improves its performance with theoretical support.
In  $A^{*}$  search algorithm [5], the heuristic function  $\hat{f}(s)$  is used to approximately estimate the optimal cost  $f^{*}(s)$  to guide the search process. If  $\hat{f}(s)$  is never over-estimated,  $A^{*}$  search will be guaranteed to find the optimal solution. Otherwise,  $A^{*}$  search may fail. MCTS [10, 3] is a best-first search method, which is generally not guaranteed to converge to the optimal solution. UCT [10] chose Upper Confidence Bounds (UCB) [1] as MCTS's tree policy to do simulations, and it has been proved that UCT definitely finds the optimal solution for the infinite memory case [11]. UCT evaluates states by simulating the real outcomes by fast rollouts in a Monte Carlo manner, which provides reliable information for making decisions. However, feedback value used in neural-guided MCTS in recent RL algorithms [23, 19] is provided by the predictive value network. The decision's quality highly depends on the guidance network. The optimality of MCTS is only established for part of heuristic functions and neural-guided MCTS will hardly find the optimal solution if the value estimation is poor. Though originated from  $A^{*}$  search, PC was also promising on MCTS with deep RL algorithms like AlphaZero [29, 33]. In this paper, we prove that when constraining value estimator with PC, neural-guided MCTS used in those modern RL algorithms is guaranteed to find the optimal solution.
3.1 Path consistency
For a Markov decision process (MDP), the agent observes state  $s_t$  from state space  $S$ , chooses an action  $a_{t+1}$  from action space  $A$ , and receives reward  $r_{t+1}$  according to the mapping  $S \times A \to \mathbb{R}$ . Starting from  $s_0$ , a playing path  $L$  consisting of  $n_L + 1$  states is generated while the agent interacts with the environment for  $n_L$  steps:
$$
L = \left\{s _ {0}, s _ {1}, s _ {2}, \dots , s _ {n _ {L} - 1}, s _ {n _ {L}} \right\}. \tag {3}
$$
If  $s_{n_L}$  is the preferred termination state, i.e., the path  $L$  receives the most accumulated reward, then  $L$  becomes the optimal path  $L^*$ . In Eq. (1),  $g(s_t)$  is the accumulated reward from  $s_0$  to  $s_t$  and the  $f$  value can be reformulated as
$$
f \left(s _ {t}\right) = g \left(s _ {t}\right) + v \left(s _ {t}; \theta\right) = \sum_ {i = 1} ^ {t} r _ {i} + v \left(s _ {t}; \theta\right), \forall s _ {t} \in L. \tag {4}
$$
where  $v(s_{t};\theta)$  is the state value estimated by the policy-value neural network with parameter  $\theta$ . The general PC is defined as the consistency of  $f$  values.
Definition 3.1. Path consistency (PC for short) is that  $f$  values of states along any optimal path in a search graph should be identical, i.e.
$$
f \left(s _ {0}\right) = f \left(s _ {1}\right) = f \left(s _ {2}\right) = \dots = f \left(s _ {n _ {L ^ {*}}}\right), \forall s \in L ^ {*} \tag {5}
$$
3.2 The neural-guided Monte Carlor Tree Search (MCTS)
In MCTS, for each possible action  $a \in \mathcal{A}$  in a given state  $s$ , a set of statistics is used to record the situation, including visit count  $N(s,a)$ , prior policy  $p(a|s)$ , cumulative reward from multiple simulations  $W(s,a)$ , and reward  $r(s,a)$ . The action value  $Q(s,a)$  is calculated as  $W(s,a) / N(s,a)$ . Starting from the current root state  $s_t$ , each simulation first traverses to a leaf node  $s_{t + \ell}$  by maximizing over a probabilistic upper confidence tree (PUCT) bound at each time-step, i.e.,
$$
a = \arg \max  _ {a ^ {\prime} \in \mathcal {A}} \left\{\frac {W (s , a ^ {\prime})}{N (s , a ^ {\prime})} + p \left(a ^ {\prime} \mid s\right) \frac {\sqrt {\sum_ {b \in \mathcal {A}} N (s , b)}}{1 + N (s , a ^ {\prime})} \right\}. \tag {6}
$$
The children states of  $s_{t + \ell}$  are expanded and  $v(s_{t + \ell})$  is backed up to all edges  $(s_i, a_{i+1})$  in the traversing path  $\{(s_t, a_{t+1}), (s_{t+1}, a_{t+2}), \dots, (s_{t+\ell-1}, a_{t+\ell})\}$  via
$$
N \left(s _ {i}, a _ {i + 1}\right) \leftarrow N \left(s _ {i}, a _ {i + 1}\right) + 1,
$$
$$
W \left(s _ {i}, a _ {i + 1}\right) \leftarrow W \left(s _ {i}, a _ {i + 1}\right) + \sum_ {\tau = 0} ^ {\ell - 1 - i} \gamma^ {\tau} r _ {i + 1 + \tau} + \gamma^ {\ell - i} v \left(s _ {t + \ell}\right), \tag {7}
$$
where  $\gamma$  is the discount factor. If  $\gamma$  is set to 1.0, the above equation is rewritten as:
$$
W \left(s _ {i}, a _ {i + 1}\right) \leftarrow W \left(s _ {i}, a _ {i + 1}\right) + f \left(s _ {t + \ell}\right) - g \left(s _ {i}\right). \tag {8}
$$
After  $K$  time simulations, the action with the maximum visit count will be selected as the next move, and the selected child will become the new root of the search tree.
3.3 Reanalyze
To improve the sample efficiency, MuZero [19] proposes a reanalyze technique to revisit its past time steps. MCTS is re-executed with the latest model parameters, potentially resulting in a better-quality policy  $\pi^{current}$  than the original policy  $\pi^{old}$ . The  $l$ -step bootstrapped value target is estimated based on multi-step TD value relationships
$$
z _ {t} = \sum_ {j = 1} ^ {l} \gamma^ {j - 1} r _ {t + j} + \gamma^ {l} v \left(s _ {t + l}; \theta\right). \tag {9}
$$
The reanalyze technique updates  $z_{t}$  with the most recent value network parameters  $\theta^{current}$ . EfficientZero [32] employs an off-policy value correction method, which conducts an additional MCTS to estimate the value as  $v^{MCTS}(s_{t + l};\theta)$  and computes  $z_{t}$  by replacing  $v(s_{t + 1};\theta)$  with  $v^{MCTS}(s_{t + l};\theta)$  in Eq. (9). Enabling off-policy value correction for reanalysis requires running two MCTS for each sample, resulting in a doubling of the computational cost.
The proposed GW-PCZero is built on EfficientZero with a generalized PC constraint. The PC constraint enables our GW-PCZero to not only remove the time-consuming off-policy value correction which has a significant contribution to the performance of EfficientZero, but also improve the performance to surpass EfficientZero with theoretical guarantee on the optimality. The loss function for learning the policy-value network is taken as the same form of Eq. (2). Different from PCZero, here  $\mathcal{L}_{RL}(\theta)$  is set to be the same as EfficientZero's loss function, consisting of policy loss, value loss, value prefix loss, and self-supervised consistency loss [32], while  $\mathcal{L}_{PC}(\theta)$  is evaluated by the deviation of  $f$  value from its mean along the path. Specifically, on the state  $s_t$ , we can rewrite the squared  $L_2$  deviation  $[f(s_t) - \bar{f}(s_t)]^2$  by noticing  $v(s_t; \theta) = h(s_t)$  in Eq. (1) as follows:
$$
\mathcal {L} _ {P C} \left(s _ {t}; \theta\right) = \left\{v \left(s _ {t}; \theta\right) - \left[ \bar {f} \left(s _ {t}\right) - g \left(s _ {t}\right) \right] \right\} ^ {2}. \tag {10}
$$
Built on the basis of MuZero, EfficientZero has achieved superhuman performance on Atari games [32] with limited  $100k$  training data. EfficientZero has made three critical changes to MuZero. First, a self-supervised consistency loss was proposed to help model the environment's state transition dynamics. Second, one-step reward prediction was replaced by the estimation of value prefix  $\sum_{j=1}^{l} \gamma^{j-1} r_{t+j}$ . Third, MCTS value target correction was executed by replacing the policy-value network's output with root values estimated by MCTS. To take advantage of the merits of EfficientZero, we keep the first two changes and disable the third one, because the MCTS's root value target correction doubles the need of computing resources. Experimental results indicate that values estimated by the network trained under the PC constraint is more reliable than this time-consuming value correction method. With the help of the PC constraint, our method is able to improve the efficiency of the policy-value network parameter learning and save computing resources.
4.1 The optimality of the PC loss
It is noted from Eq. (10) that  $\mathcal{L}_{PC}(s_t)$  is the square error loss and  $\mathcal{L}_{PC}(s_t) \geq 0$  holds. We present the following theorem for the optimality of the PC loss.
Theorem 4.1. The PC loss of state  $s_t$  achieves the optimal value when the following equation holds:
$$
\left\{ \begin{array}{l} v \left(s _ {t}\right) = \sum_ {j = t + 1} ^ {i} r _ {j} + v \left(s _ {i}\right), \quad i > t, \\ v \left(s _ {i}\right) = \sum_ {j = i + 1} ^ {t} r _ {j} + v \left(s _ {t}\right), \quad i <   t. \end{array} \right. \tag {11}
$$
The above Eq. (11) actually represents the multi-step TD value estimation relationships between  $s_t$  and all other states  $\{s_i\}_{i \neq t}$  in the same path.
The estimated values given by an effective value function predictor should satisfy the multi-step TD value relationships specified by the Bellman equation. Theorem 4.1 indicates that  $\mathcal{L}_{PC}$  will be minimized if multi-step TD value relationships are established for the estimated values, indicating that path consistency is a nature that a well-learned value predictor should have. The term of PC loss guides the learning process towards it proactively, which gives an explanation about why the adoption of PC improves learning efficiency. State values learned by minimizing  $\mathcal{L}_{PC}$  will establish the relationship among them, rather than learning the value predictions for different states independently. In particular, by setting discount factor  $\gamma = 1$  in Eq. (9), the bootstrapping target for the state  $s_i$ 's value learning in MuZero becomes the right formula of Eq. (11), implying that PC is beneficial to the learning of value estimation. It should be noted that Theorem 4.1 is also established for any  $L_p$  norm deviation loss with  $p \geq 1$ , not limited to  $L_2$  norm used in this paper.
To prove Theorem 4.1, we define  $t_{PC}(s_t) = \bar{f}(s_t) - g(s_t)$  as the learning target for the value network in Eq. (10), and let  $\bar{f}(s_t)$  be the mean over all states in the path  $L$ . Based on Eq. (3) & (4),  $\mathcal{L}_{PC}(s_t; \theta)$  is equivalently derived to be
$$
\begin{array}{l} \mathcal {L} _ {P C} (s _ {t}; \theta) = \frac {1}{(n _ {L} + 1) ^ {2}} \left\{\sum_ {i > t} \left[ v (s _ {t}; \theta) - \left(\sum_ {j = t + 1} ^ {i} r _ {j} + v (s _ {i})\right) \right] \right. \\ \left. - \sum_ {i <   t} \left[ v \left(s _ {i}\right) - \left(\sum_ {j = i + 1} ^ {t} r _ {j} + v \left(s _ {t}; \theta\right)\right) \right] \right\} ^ {2}. \tag {12} \\ \end{array}
$$
According to Eq. (12), the PC loss is a square error function with  $\mathcal{L}_{PC}(s_t;\theta)\geq 0$ . When the multi-step TD value relationships by Eq. (11) are satisfied, each term in the square brackets of Eq. (12) is equal to zero. Then,  $\mathcal{L}_{PC}(s_t;\theta)$  is minimized at zero, and it implies that PC is established, which suggests that Theorem 4.1 holds. The derivation details of Eq. (12) are given in Appendix 1.
Previous research extensively employed TD error for training value function models in various algorithms. Minimizing TD error is a commonly utilized learning objective among model-free RL algorithms, such as Q learning [28] and its variants [15, 27, 6, 26]. For model-based RL algorithms, the TD error is employed as one of the auxiliary training objective during the learning process of the environment model. Dyna [25] updated the estimated values to be consistent with the environment model by minimizing the TD(0) error, while  $\mathrm{TD}(k)$  is adopted by value equivalent models, such as MuZero [19] and Muesli [7]. SCZero [4] utilizes the TD(0) error to update the environment model and value function simultaneously to encourage them to be jointly self-consistent, demonstrating that the TD constraint on value estimations is beneficial for both policy evaluation and environment model construction. Compared with those TD based algorithms, PC is a global constraint for all states on the optimal path. As illustrated in Eq. (12), PC loss of  $s_t$  is minimized if the TD errors for state  $s_t$  with all states on the same path are minimized to be 0, while TD error itself represents a local constraint that pertains to the relationship between two states. The PC constraint on state  $s_t$  encompasses both consistency with subsequent states and consistency with preceding states. In contrast, the TD error only takes into account the consistency with subsequent states. Both SCZero and our GW-PCZero propose a consistency constraint to improve the learning efficiency. SCZero [4] is built on model-based algorithms, encouraging the learned environment model and value function to be consistent by minimizing the TD(0) error. The concept of path consistency, explored in this paper, is conceptually different from the notion of self-consistency discussed in SCZero. PC requires that estimated  $f$  values on the same optimal path are consistent, which is applicable to all value-based RL algorithms, regardless of their reliance on a model-based framework.
We further prove that the PC constraint on the estimated values guarantee the neural-guided MCTS to find the optimal solution, under an assumption below, which is made from the perspective of probability based on Definition 3.1. We use the notation  $\mathcal{N}(\mu, \sigma^2)$  to denote a normal distribution and use  $Pr(\mu, \sigma^2)$  to denote an arbitrary distribution, where  $\mu$  denotes mean and  $\sigma^2$  denotes variance.
Assumption 4.2. For the state  $s$  on the optimal path, suppose  $f(s) = \mu_0^f$ ; For every state  $s'$  not on the optimal path,  $\{f(s')\}$  are assumed to be sampled from an i.i.d.  $Pr(\mu_1^f, \sigma_1^2)$ .
The  $\mu_0^f$  and  $\mu_1^f$  are the expected total rewards for optimal and non-optimal solutions respectively according to the definition of  $f$ , and the inequality  $\mu_0^f > \mu_1^f$  holds.
Theorem 4.3. Under the PC condition,  $P_{g}$ , defined as the probability of finding the optimal path with the neural-guided MCTS, satisfies the following inequality as the number of simulations  $K$  goes sufficiently large,
$$
\lim  _ {K \rightarrow + \infty} P _ {g} \geq \lim  _ {K \rightarrow + \infty} \prod_ {t = 1} ^ {n _ {L}} \left\{1 - \frac {1}{2} \exp \left\{- \frac {\left[ b _ {t} \left(\mu_ {0} ^ {f} - \mu_ {1} ^ {f}\right)\right] ^ {2}}{2 \sigma_ {1} ^ {2} / K ^ {\prime}} \right\}\right\} ^ {m - 1}, \tag {13}
$$
where  $b_{t}$  is a constant, and  $K^{\prime}$  is the minimum simulation times for nodes in the search tree.
The proof of Theorem 4.3 is given in Appendix 2. Intuitively, if  $K$  goes large  $(K\to +\infty)$ , every child in the search tree will be visited enough times because of the exploration term in Eq. (6), leading to  $K^{\prime}\rightarrow +\infty$ . Then, the lower bound of Eq. (13) approaches one, which pushes the probability of finding the optimal path to be one. Formally, we have the following result.
Theorem 4.4. For any Markov decision process, the neural-guided MCTS with path consistency as the constraint of the estimated values is guaranteed to find the optimal solution, when the number of simulations  $K$  is sufficiently large,
$$
\lim  _ {K \rightarrow + \infty} P _ {g} = 1. \tag {14}
$$
4.2 Empirical computation of PC on  $f$  consistency
Considering that interaction sequences in practical applications like Atari games are quite long, the generated samples are used to update parameters before reaching the termination. In practice, the state  $s_t$  and its  $\ell$  following unrolled states are sampled from the replay buffer as a small training batch  $L_b = \{s_t, s_{t+1}, \dots, s_{t+\ell}\}$ . Reanalyzing is adopted to re-estimate the learning target with the latest policy-value network for all states in  $L_b$ . States in  $L_b$  share part of the cumulative reward before  $s_t$ . The PC target  $t_{PC}$  for the value network in Eq. (10) is calculated as
$$
t _ {P C} \left(s _ {t}\right) = \frac {1}{l + 1} \sum_ {i = 0} ^ {l} \left[ \sum_ {j = 1} ^ {i} r _ {t + j} + v \left(s _ {t + i}; \theta\right) \right]. \tag {15}
$$