Title: Generalized Weighted Path Consistency for Mastering Atari Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Monte Carlo Tree Search

Reinforcement learning

Path consistency.
Abstract: Reinforcement learning with the help of neural-guided search consumes huge computational resources to achieve remarkable performance. Path consistency (PC), i.e., $f$ values on one optimal path should be identical, was previously imposed on MCTS by PCZero to improve the learning efficiency of AlphaZero. Not only  PCZero still lacks a theoretical support but also considers  merely board games. In this paper,  PCZero is  generalized into GW-PCZero for  real applications with non-zero immediate reward. A weighting mechanism is introduced to reduce the variance caused by scouting's uncertainty on the $f$ value estimation. For the first time, it is theoretically proved that neural-guided MCTS is guaranteed to find the optimal solution under the constraint of PC. Experiments are conducted on the Atari $100$k benchmark with $26$ games and GW-PCZero achieves $198\%$ mean human performance, higher than the state-of-the-art EfficientZero's $194\\%$, while consuming only $25\\%$ of the computational resources consumed by EfficientZero.

Generalized Weighted Path Consistency for Mastering Atari Games

Abstract

Reinforcement learning with the help of neural-guided search consumes huge computational resources to achieve remarkable performance. Path consistency (PC), i.e., f values on one optimal path should be identical, was previously imposed on MCTS by PCZero to improve the learning efficiency of AlphaZero. Not only PCZero still lacks a theoretical support but also considers merely board games. In this paper, PCZero is generalized into GW-PCZero for real applications with non-zero immediate reward. A weighting mechanism is introduced to reduce the variance caused by scouting's uncertainty on the f value estimation. For the first time, it is theoretically proved that neural-guided MCTS is guaranteed to find the optimal solution under the constraint of PC. Experiments are conducted on the Atari 100 k benchmark with 26 games and GW-PCZero achieves 198% mean human performance, higher than the state-of-the-art EfficientZero's 194% , while consuming only 25% of the computational resources consumed by EfficientZero.

1 Introduction

In recent years, combining neural networks with search algorithms has achieved remarkable success in Reinforcement Learning (RL) [22, 24, 23]. The application scenarios include not only board games but also knowledge-free real-world problems [19]. One critical challenge is that huge computational resources are required to collect interactive experiences in order to obtain reliable policy and value networks with remarkable performance. For example, 1000 third-generation TPUs were used to generate selfplay games in MuZero [19], roughly 10 ∼ 50 years of experience per Atari game, which is unaffordable in regular situations. It is essential for an algorithm to achieve high performance with limited computational cost. Otherwise, it is difficult to generalize the algorithm to more applications.

For sequential decision problems, an interactive process between an agent and the environment is regarded as a playing path, which is collected to update the parameters of the policy and value network. In turn, the learned policy and value provide guidance for Monte Carlo Tree Search (MCTS) in determining the next action. This iterative process generates a large number of playing paths in order to obtain reliable policy and value estimators. The evaluation function f at state s t is defined as the summation of g ( s t ) , the accumulated reward from the initial state s 0 to s t , and h ( s t ) , the expected return received after s t , which is equivalent to the state value v ( s t ) . That is, we have

To find the optimal path L ∗ that leads to the desired termination state, the A ∗ search algorithm [5] establishes an optimality condition for f values, i.e., f ( s t ) = f ( s 0 ) for all s t ∈ L ∗ , where s 0 is

the initial state that definitely belongs to the optimal path. In 1987, CNNEIM-A [30] relied on this optimality to improve A ∗ by estimating the f -values on L ∗ with the average of the f -values on the historical trajectory and on a lookahead scouting segment, plus revealing an implication that the f -values are regarded as i.i.d. samples from a same distribution. This idea has been adopted to equations.(8)&(9)&(10) in [29] with a regularization term added into the loss function of deep reinforcement learning such as AlphaZero [23]. The term aims at reducing the deviation of f ( s t ) from ones on L ∗ towards satisfying path consistency (PC)- " f values on one optimal path should be identical ". Simplifying equation (10) in [29] by considering L 2 error and ignoring weighting, recently PCZero [33] realised the PC constraint for the first time by considering

where L RL ( θ ) is the loss function of AlphaZero, λ is a hyperparameter, L PC is implemented as ( v ( s t ) -¯ v ( s t )) 2 , the squared L 2 deviation of the state value v ( s t ) from the learning target ¯ v ( s t ) , which is the average state value along the estimated optimal path. Experiments on three board games have demonstrated that PCZero improves the learning efficiency of AlphaZero.

However, ignoring weighting by PCZero is oversimplified because deeper states along the search path have greater uncertainty due to the exploration randomness of self-play and policy reanalysis [19]. Thus, it is essential to take into account of the uncertainty levels in L PC . Also, PCZero has only been applied to board games, wherein the immediate reward is always zero until the game terminates. Thus, g ( s t ) = 0 holds for all states in PCZero, while PC has degenerated into the consistency of state values. It is unclear whether PC is effective generally as suggested by equation (8) in [29] to take g ( s t ) in consideration for applications with non-zero immediate rewards, such as Atari games. Moreover, there lacks a strong theoretical support for the effectiveness of PC, and the advantage conferred by the PC constraint has not been clearly explained.

This paper tackles the above problems by generalizing PCZero into GW-PCZero (short for G eneralized W eighted PCZero), with the main contributions summarized as follows 1 .

The application of PC is extended to scenarios where the environment emits immediate rewards, such as Atari games. Loss function in EfficientZero [32] is adopted as L RL ( θ ) in Eq. (2), instead of using the loss function of AlphaZero like PCZero [33]. L PC ( θ ) ensures the consistency of f values in Eq (1) across the search path, rather than only maintaining the consistency of h value like PCZero. ¯ f ( s t ) , the mean of the f values of states along the path, is adopted as the learning target for L PC .

An uncertainty-aware weighting mechanism is introduced for accurately estimating ¯ f ( s t ) . States farther away from s t are considered less reliable and given larger discounts to reduce noise in ¯ f ( s t ) . Experiments are conducted on the Atari 100 k benchmark with 26 games to evaluate GW-PCZero in diverse environments. Under the same computing resource consumption, our GW-PCZero achieves 198% mean human normalized performance, significantly outperforms the state-of-the-art EfficientZero's 121% . Additionally, GW-PCZero surpasses the original version of EfficientZero, which achieved a score of 194% [32], while utilizing only 25% of the computational resources required by EfficientZero.

For the first time, it is theoretically proved that neural-guided MCTS is guaranteed to find the optimal solution under the constraint of PC, and that the optimal solution for minimizing L PC ( s t ) is achieved by satisfying the multi-step temporal difference (TD) value estimation relationship between state s t and its neighboring states in the same path. Instead of estimating state value independently, PC leads to a more reliable value estimator.

2 Related work

The Arcade Learning Environment (ALE) is an evaluation platform that contains plenty of Atari games, which is used to evaluate the general competency of AI agents [2]. Interaction steps are restricted to 100 k , which is roughly two hours of experience for human [14], to explore models' sample efficiency. SimPLe [8] further selects 26 games from ALE on the basis of being solvable with existing state-of-the-art reinforcement learning algorithms, which is adopted as the benchmark by many following works [9, 12, 20, 31, 13, 16]. So far, EfficientZero [32] is the only model that

has achieved super-human performance on Atari 100 k . Here, our generalized PC is applied to EfficientZero and improves its performance with theoretical support.

In A ∗ search algorithm [5], the heuristic function ˆ f ( s ) is used to approximately estimate the optimal cost f ∗ ( s ) to guide the search process. If ˆ f ( s ) is never over-estimated, A ∗ search will be guaranteed to find the optimal solution. Otherwise, A ∗ search may fail. MCTS [10, 3] is a best-first search method, which is generally not guaranteed to converge to the optimal solution. UCT [10] chose Upper Confidence Bounds (UCB) [1] as MCTS's tree policy to do simulations, and it has been proved that UCT definitely finds the optimal solution for the infinite memory case [11]. UCT evaluates states by simulating the real outcomes by fast rollouts in a Monte Carlo manner, which provides reliable information for making decisions. However, feedback value used in neural-guided MCTS in recent RL algorithms [23, 19] is provided by the predictive value network. The decision's quality highly depends on the guidance network. The optimality of MCTS is only established for part of heuristic functions and neural-guided MCTS will hardly find the optimal solution if the value estimation is poor. Though originated from A ∗ search, PC was also promising on MCTS with deep RL algorithms like AlphaZero [29, 33]. In this paper, we prove that when constraining value estimator with PC, neural-guided MCTS used in those modern RL algorithms is guaranteed to find the optimal solution.

3 Preliminary

3.1 Path consistency

For a Markov decision process (MDP), the agent observes state s t from state space S , chooses an action a t +1 from action space A , and receives reward r t +1 according to the mapping S × A → R . Starting from s 0 , a playing path L consisting of n L +1 states is generated while the agent interacts with the environment for n L steps:

If s n L is the preferred termination state, i.e., the path L receives the most accumulated reward, then L becomes the optimal path L ∗ . In Eq. (1), g ( s t ) is the accumulated reward from s 0 to s t and the f value can be reformulated as

where v ( s t ; θ ) is the state value estimated by the policy-value neural network with parameter θ . The general PC is defined as the consistency of f values.

Definition 3.1. Path consistency (PC for short) is that f values of states along any optimal path in a search graph should be identical, i.e.

3.2 The neural-guided Monte Carlor Tree Search (MCTS)

In MCTS, for each possible action a ∈ A in a given state s , a set of statistics is used to record the situation, including visit count N ( s, a ) , prior policy p ( a | s ) , cumulative reward from multiple simulations W ( s, a ) , and reward r ( s, a ) . The action value Q ( s, a ) is calculated as W ( s, a ) /N ( s, a ) . Starting from the current root state s t , each simulation first traverses to a leaf node s t + ℓ by maximizing over a probabilistic upper confidence tree (PUCT) bound at each time-step, i.e.,

The children states of s t + ℓ are expanded and v ( s t + ℓ ) is backed up to all edges ( s i , a i +1 ) in the traversing path { ( s t , a t +1 ) , ( s t +1 , a t +2 ) , · · · , ( s t + ℓ -1 , a t + ℓ ) } via

where γ is the discount factor. If γ is set to 1 . 0 , the above equation is rewritten as:

After K time simulations, the action with the maximum visit count will be selected as the next move, and the selected child will become the new root of the search tree.

3.3 Reanalyze

To improve the sample efficiency, MuZero [19] proposes a reanalyze technique to revisit its past time steps. MCTS is re-executed with the latest model parameters, potentially resulting in a better-quality policy π current than the original policy π old . The l -step bootstrapped value target is estimated based on multi-step TD value relationships

The reanalyze technique updates z t with the most recent value network parameters θ current . EfficientZero [32] employs an off-policy value correction method, which conducts an additional MCTS to estimate the value as v MCTS ( s t + l ; θ ) and computes z t by replacing v ( s t +1 ; θ ) with v MCTS ( s t + l ; θ ) in Eq. (9). Enabling off-policy value correction for reanalysis requires running two MCTS for each sample, resulting in a doubling of the computational cost.

4 Methodology

The proposed GW-PCZero is built on EfficientZero with a generalized PC constraint. The PC constraint enables our GW-PCZero to not only remove the time-consuming off-policy value correction which has a significant contribution to the performance of EfficientZero, but also improve the performance to surpass EfficientZero with theoretical guarantee on the optimality. The loss function for learning the policy-value network is taken as the same form of Eq. (2). Different from PCZero, here L RL ( θ ) is set to be the same as EfficientZero's loss function, consisting of policy loss, value loss, value prefix loss, and self-supervised consistency loss [32], while L PC ( θ ) is evaluated by the deviation of f value from its mean along the path. Specifically, on the state s t , we can rewrite the squared L 2 deviation [ f ( s t ) -¯ f ( s t )] 2 by noticing v ( s t ; θ ) = h ( s t ) in Eq. (1) as follows:

Built on the basis of MuZero, EfficientZero has achieved superhuman performance on Atari games [32] with limited 100 k training data. EfficientZero has made three critical changes to MuZero. First, a self-supervised consistency loss was proposed to help model the environment's state transition dynamics. Second, one-step reward prediction was replaced by the estimation of value prefix ∑ l j =1 γ j -1 r t + j . Third, MCTS value target correction was executed by replacing the policy-value network's output with root values estimated by MCTS. To take advantage of the merits of EfficientZero, we keep the first two changes and disable the third one, because the MCTS's root value target correction doubles the need of computing resources. Experimental results indicate that values estimated by the network trained under the PC constraint is more reliable than this time-consuming value correction method. With the help of the PC constraint, our method is able to improve the efficiency of the policy-value network parameter learning and save computing resources.

4.1 The optimality of the PC loss

It is noted from Eq. (10) that L PC ( s t ) is the square error loss and L PC ( s t ) ≥ 0 holds. We present the following theorem for the optimality of the PC loss.

Theorem 4.1. The PC loss of state s t achieves the optimal value when the following equation holds:

̸

The above Eq. (11) actually represents the multi-step TD value estimation relationships between s t and all other states { s i } i = t in the same path.

The estimated values given by an effective value function predictor should satisfy the multi-step TD value relationships specified by the Bellman equation. Theorem 4.1 indicates that L PC will be minimized if multi-step TD value relationships are established for the estimated values, indicating that path consistency is a nature that a well-learned value predictor should have. The term of PC loss guides the learning process towards it proactively, which gives an explanation about why the adoption of PC improves learning efficiency. State values learned by minimizing L PC will establish the relationship among them, rather than learning the value predictions for different states independently. In particular, by setting discount factor γ = 1 in Eq. (9), the bootstrapping target for the state s i 's value learning in MuZero becomes the right formula of Eq. (11), implying that PC is beneficial to the learning of value estimation. It should be noted that Theorem 4.1 is also established for any L p norm deviation loss with p ≥ 1 , not limited to L 2 norm used in this paper.

To prove Theorem 4.1, we define t PC ( s t ) = ¯ f ( s t ) -g ( s t ) as the learning target for the value network in Eq. (10), and let ¯ f ( s t ) be the mean over all states in the path L . Based on Eq. (3) & (4), L PC ( s t ; θ ) is equivalently derived to be

According to Eq. (12), the PC loss is a square error function with L PC ( s t ; θ ) ≥ 0 . When the multi-step TD value relationships by Eq. (11) are satisfied, each term in the square brackets of Eq. (12) is equal to zero. Then, L PC ( s t ; θ ) is minimized at zero, and it implies that PC is established, which suggests that Theorem 4.1 holds. The derivation details of Eq. (12) are given in Appendix 1 .

Previous research extensively employed TD error for training value function models in various algorithms. Minimizing TD error is a commonly utilized learning objective among model-free RL algorithms, such as Q learning [28] and its variants [15, 27, 6, 26]. For model-based RL algorithms, the TD error is employed as one of the auxiliary training objective during the learning process of the environment model. Dyna [25] updated the estimated values to be consistent with the environment model by minimizing the TD( 0 ) error, while TD( k ) is adopted by value equivalent models, such as MuZero [19] and Muesli [7]. SCZero [4] utilizes the TD( 0 ) error to update the environment model and value function simultaneously to encourage them to be jointly self-consistent, demonstrating that the TD constraint on value estimations is beneficial for both policy evaluation and environment model construction. Compared with those TD based algorithms, PC is a global constraint for all states on the optimal path. As illustrated in Eq. (12), PC loss of s t is minimized if the TD errors for state s t with all states on the same path are minimized to be 0 , while TD error itself represents a local constraint that pertains to the relationship between two states. The PC constraint on state s t encompasses both consistency with subsequent states and consistency with preceding states. In contrast, the TD error only takes into account the consistency with subsequent states. Both SCZero and our GW-PCZero propose a consistency constraint to improve the learning efficiency. SCZero [4] is built on model-based algorithms, encouraging the learned environment model and value function to be consistent by minimizing the TD( 0 ) error. The concept of path consistency, explored in this paper, is conceptually different from the notion of self-consistency discussed in SCZero. PC requires that estimated f values on the same optimal path are consistent, which is applicable to all value-based RL algorithms, regardless of their reliance on a model-based framework.

We further prove that the PC constraint on the estimated values guarantee the neural-guided MCTS to find the optimal solution, under an assumption below, which is made from the perspective of probability based on Definition 3.1. We use the notation N ( µ, σ 2 ) to denote a normal distribution and use Pr ( µ, σ 2 ) to denote an arbitrary distribution, where µ denotes mean and σ 2 denotes variance.

Assumption 4.2. For the state s on the optimal path, suppose f ( s ) = µ f 0 ; For every state s ′ not on the optimal path, { f ( s ′ ) } are assumed to be sampled from an i.i.d. Pr ( µ f 1 , σ 2 1 ) .

The µ f 0 and µ f 1 are the expected total rewards for optimal and non-optimal solutions respectively according to the definition of f , and the inequality µ f 0 > µ f 1 holds.

Theorem 4.3. Under the PC condition, P g , defined as the probability of finding the optimal path with the neural-guided MCTS, satisfies the following inequality as the number of simulations K goes sufficiently large,

where b t is a constant, and K ′ is the minimum simulation times for nodes in the search tree.

The proof of Theorem 4.3 is given in Appendix 2 . Intuitively, if K goes large ( K → + ∞ ) , every child in the search tree will be visited enough times because of the exploration term in Eq. (6), leading to K ′ → + ∞ . Then, the lower bound of Eq. (13) approaches one, which pushes the probability of finding the optimal path to be one. Formally, we have the following result.

Theorem 4.4. For any Markov decision process, the neural-guided MCTS with path consistency as the constraint of the estimated values is guaranteed to find the optimal solution, when the number of simulations K is sufficiently large,

4.2 Empirical computation of PC on f consistency

Considering that interaction sequences in practical applications like Atari games are quite long, the generated samples are used to update parameters before reaching the termination. In practice, the state s t and its ℓ following unrolled states are sampled from the replay buffer as a small training batch L b = { s t , s t +1 , · · · , s t + ℓ } . Reanalyzing is adopted to re-estimate the learning target with the latest policy-value network for all states in L b . States in L b share part of the cumulative reward before s t . The PC target t PC for the value network in Eq. (10) is calculated as

Only s t is considered to be constrained with PC loss in real implementation because the sampled batch L b is too short to deal with the subsequent states in the same way. The reanalyze technique is used to prepare the learning targets for policy and value without enabling off-policy value correction. All target preparations are summarized in Algorithm 1.

Algorithm 1: Sample Preparation for GW-PCZero

Input : Replay buffer

R , Unrolled steps l , Batch size N .

Output :

( π, z, t PC ) .

1: Sample N unrolled sequences from R and each sequence consists of l +1 consecutive states.

2: for each sampled sequence L i = { s i t , s i t +1 , · · · , s i t + l } do

3: Reanalyze policy target π by performing one iteration MCTS for each state in L i .

4: Recalculate value target z by bootstrapping in Eq. (9) for each state in L i .

5: Estimate PC target t PC ( s t ) according to Algorithm 2.

6: end for

7: return Tuple ( π, z, t PC ) .

4.3 Weighting the PC target

The estimation of t PC ( s t ) in Eq. (15) takes the mean of all f values, treating all states equally. However, the states far away from s t are less reliable than the ones near s t , which is mainly caused by two operations, as shown in Figure 1. One is the exploration uncertainty during the iteration process with the environment. As s t + i is sampled from the policy distribution injected with noise, it may not be the optimal choice. The other more important reason is policy reanalyzing. As illustrated in

the right of Figure 1, s t +1 is the following state of s t in the real iteration sequence collected with an older policy π old , but the optimal choice of the latest reanalyzed policy π current may be another state s ∗ t +1 . According to Definition 3.1, PC requires that all states should be on the same optimal path, and it will be violated due to the above two operations. Especially, the probability that s t + i and s t are not in the same optimal path grows as i increases.

π(St+1)

St

π(st+1)

St+1

*

St+2

*

St+l

*

π(S2+1)

St+1

1

st+1

2

n-1

St

St+1

St+l

π(st+1)

= N(st+1)/N(st)

State collected with πold

i

1 ~ N(μ,o²)

Optimal state under Tπcurrent

t+1

In order to mitigate the high variance brought by the above uncertainty, a weighting mechanism is devised to give a larger discount to farther states. Specifically, t PC ( s t ) is calculated by a weighted summation and the weight w i decreases linearly with index i , i.e.,

where c a and c b are two constants.While there are numerous algorithms available for weighting, the linear weighting method employed in this paper is simple and highly interpretable, which deserves further investigation in future research. The PC target t PC is calculated as:

where ˜ w i = w i / ∑ l i =0 w i is the normalized weight. The details are summarized in Algorithm 2.

Algorithm 2: Weighted PC target t PC estimation

Input : S

= { s t , r t +1 , s t +1 , · · · , s t + l } , value function v ( s ) .

Output

: Target t PC ( s t ) .

1: Initialize the value summation T ← 0 and weight summation W ← 0 .

2: for each state s t + i in S do

3: Calculate weight w i for state s t + i according to Eq. (16).

5: end for

6: return target t PC ( s t ) ← T/W .

4.4 The coefficient for the PC loss

PC is incorporated into the policy-network learning as a regularization term in Eq. (2), and there is a non-negative coefficient λ to be determined. GW-PCZero degenerates back to a variant of EfficientZero (without MCTS value target correction) when λ = 0 . In general, an appropriate λ is required to balance between maximizing the accumulated reward and satisfying the PC constraint. According to the empirical experience of PCZero on Hex game [33], it has been found that PC improves the generalization ability for testing while sacrificing the prediction accuracy for training. For games with low complexity, it is easy to generate all possible states for training and λ should be small to improve the training accuracy. For games with high complexity, the generalization ability should be paid more attention by giving a larger λ . See the details in Appendix 3 .

Therefore, λ is adjusted with the game complexity, which is evaluated by two criteria. One is the action space size |A| , which determines the number of brunch factors for tree search. The other is the difficulty for the computer program to master, which is approximately evaluated by D m = MuZero Score / Human Score in Atari games, where MuZero is trained without 100 k limitation. In the experiment, if |A| < 18 , λ is set to be 0 . 2 . For games with |A| ≥ 18 , if D m is larger than 100 , indicating that this game is easier to master, λ will be set to be 0 . 35 . Otherwise, this game is difficult to master, and λ = 0 . 4 .

5 Experiments

Experiments are conducted on Atari 100 k benchmark containing 26 games [8]. Only 100 k interaction steps are allowed to be collected, i.e., 400 k frames based on a frameskip of 4 , which is roughly the number of experiences played by a human in two hours. The mean performance of human players was collected [17], which is used to evaluate the model's performance against the baseline of the random player's score. The human normalized score of a learned agent is defined as ( Scoreagent -Scorerandom ) / ( Scorehuman -Scorerandom ) .

In EfficientZero [32], MCTS consumes the majority of computing resources, much more than network updating. The number of MCTS is determined by two factors: data collection, which requires 100 k MCTS for 100 k steps, and data reanalysis. 120 k updating steps are performed with a batch size of 256 . A total of 30 . 72 million samples are reanalyzed, with each sample requiring two MCTS runs: one for policy re-estimation and the other for MCTS value correction. Thus, a total of 61 . 44 million MCTS runs are used to prepare the training data, and the computational cost for data generation is relatively negligible. Therefore, the limit of 100 k interaction steps does not fully constrain the computational resources used, and the total number of MCTS runs should be considered. Due to the reanalysis operation, EfficientZero requires a total of 61 . 54 million MCTS runs, which accounts for the majority of the computational resources consumed.

In this paper, we evaluate our GW-PCZero's learning efficiency under a more difficult setting than the original EfficientZero [32]. As summarized in Table 1, the updating steps are limited to 60 k , and MCTS off-policy value correction is disabled. Models in our experiment only consume 25% computational resources of the original EfficientZero. G-PCZero is included for comparisons by using the unweighted computation of the PC target in Eq. (15). EfficientZero † is retrained under identical conditions. Experiments are conducted on 4 NVIDIA Tesla A 100 GPUs with 16 CPU cores. We set c b = 1 . 0 and c a = 0 . 1 in Eq. (16). Totally 32 of different random seeds are used. Other hyperparameter settings are the same as EfficientZero, as summarized in Appendix 5 .

The mean testing results are reported in Table 2. GW-PCZero performs better than EfficientZero † in 19 out of all 26 Atari games. The mean of the human normalized score of GW-PCZero achieves 198 . 0% , much higher than EfficientZero † 's 121 . 2% . Moreover, GW-PCZero slightly outperforms the original EfficientZero [32], which achieved a score of 194 . 3% . It should be noted that GW-PCZero only consumes 25% computational resources of the original EfficientZero, as shown in Table 1. Experiment results demonstrate the effectiveness of our GW-PCZero, and indicate that PC improves the learning efficiency remarkably in Atari games. Notice that GW-PCZero performs poorly in several games requiring great exploratory, such as Freeway, probably due to the quick convergence of PC. Moreover, examples of learning curves of GW-PCZero and EfficientZero ∗ on Atari games are displayed in Appendix 6. Analysis of test variances is included in Appendix 4 .

GW-PCZero outperforms G-PCZero in 15 games and achieves a significantly higher mean normalized score. This ablation study indicates that the weighting mechanism is important to provide a more accurate target for PC learning. The performance is greatly improved by considering different uncertainty levels in the states along the path while calculating the mean f values. It is critical to observe that G-PCZero outperforms EfficientZero † in 16 Atari games, and its mean score of 137 . 8% is slightly higher than that of EfficientZero † , indicating that PC improves the model's performance even without the weighting mechanism. We also employ the weighting mechanism in PCZero [33] and conduct experiments on board games. The game performance exhibits improvement as well. Details are included in Appendix 7 .

Path consistency requires that f values on the optimal path should be identical. The selection of the optimal path from the collected interaction experiences is not only necessary but also presents a significant challenge. Due to the difficulty of definitively determining the optimality of a particular path, one alternative approach is to strive for the collection of training paths that are near-optimal, thus

satisfying the requirement of PC. In on-policy reinforcement learning, game experiences collected through interactions between the updated policy and the environment improve gradually and approach a near-optimal level throughout the training process. Alternatively, near-optimal paths obtained from human experts or other high-level computer players can also fulfill the requirement of PC, allowing the model to be trained in an off-policy manner. Both aforementioned training approaches have been demonstrated to be effective in PCZero [33]. The critical factor for the success of PC lies in the selection of the optimal path, and is relatively unaffected by whether the path is collected through on-policy or off-policy methods. However, in many scenarios, the availability of near-optimal paths provided by experts is limited, and on-policy learning methods demonstrate low sample efficiency. Therefore, in addition to the aforementioned methods, it is crucial to investigate the training of PC using suboptimal paths. For example, game frames in Atari 100 k are collected using a poorly trained model in the early state, which are far from near-optimal. GW-PCZero implements a weighting mechanism to address the limitation of suboptimal paths by excluding unreliable nodes along the collected path from consideration, and experimental results in Table 2 demonstrate the effectiveness of this node filtering approach.

We further investigate the weighting mechanism in more detail. As shown in Eq. (16), linear decay is adopted from the initial value c b = 1 . 0 , with the decay rate controlled by c a . A larger c a gives less weight to the latter states, while c a = 0 represents the situation without weighting. The effects of c a are illustrated in Table 3. The optimal c a varies across different games, indicating that there is still room for improving the weighting strategy. On one hand, more reliable neighboring states should be considered to reduce the variance of the mean f value. On the other hand, unreliable neighbors increase uncertainty in the estimation of the mean f value. c a is to achieve a trade-off between including more neighbors and reducing unreliability. Notice that when considering Asterix, Alien, BattleZone, and Kangaroo, GW-PCZero is inferior to EfficientZero in Table 2, but it becomes better than EfficientZero under appropriate values of c a . When c a is adjustable, GW-PCZero( c a ) achieves a mean score of 2 . 066 , with only 2 out of 26 games performing worse than EfficientZero † . These results suggest that there is potential for further improvement in the weighting strategy and that more effort should be devoted to exploring this area in future research.

6 Conclusion

In this paper, we have proposed GW-PCZero to consider path consistency (PC) in a general setting and provided a theoretical analysis of the convergence property of MCTS with the PC constraint. GW-PCZero extends the previous PCZero from board games to more practical applications, for which the environment emits immediate rewards, such as Atari games. An uncertainty-aware weighting mechanism, which gives a larger discount to the farther states along the searching path, is devised to compute a more reliable learning target for PC. What's more, we have theoretically proved that MCTS is guaranteed to find the optimal solution under the constraint of PC. Experiments demonstrate that PC is beneficial to obtain strong artificial intelligence programs but consumes fewer computational resources. Neural-guided MCTS has been extensively utilized to address optimization problems in various practical applications, such as de novo drug molecular generation [18], organic molecule retrosynthesis [21] and so on. There is potential for the development of PC to contribute to addressing these critical problems by providing more robust value estimators. This work is still in the nascent stages without further applications related to people's daily lives currently and thus there is no immediate ethical or harmful social impacts.

References

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games , pages 72-83. Springer, 2006.

Greg Farquhar, Kate Baumli, Zita Marinho, Angelos Filos, Matteo Hessel, Hado P van Hasselt, and David Silver. Self-consistent models and values. Advances in Neural Information Processing Systems , 34:1111-1125, 2021.

Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics , 4(2):100-107, 1968.

Hado Hasselt. Double q-learning. Advances in neural information processing systems , 23, 2010.

Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy optimization. In International conference on machine learning , pages 4214-4226. PMLR, 2021.

Łukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Bła˙ zej Osi´ nski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In International Conference on Learning Representations , 2019.

Kacper Piotr Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? 2019.

Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning , pages 282-293. Springer, 2006.

Levente Kocsis, Csaba Szepesvári, and Jan Willemson. Improved monte-carlo search. Univ. Tartu, Estonia, Tech. Rep , 1, 2006.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning , pages 5639-5650. PMLR, 2020.

Alexander Long, Alan Blair, and Herke van Hoof. Fast and data efficient reinforcement learning from pixels via non-parametric value approximation. 2022.

Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523-562, 2018.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning , pages 16828-16847. PMLR, 2022.

Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado Van Hasselt, John Quan, Mel Veˇ cerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv:1805.11593 , 2018.

Hao Qian, Cheng Lin, Dengwei Zhao, Shikui Tu, and Lei Xu. Alphadrug: protein target specific de novo molecular generation. PNAS nexus , 1(4):pgac227, 2022.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations , 2020.

Marwin HS Segler, Mike Preuss, and Mark P Waller. Planning chemical syntheses with deep neural networks and symbolic ai. Nature , 555(7698):604-610, 2018.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin , 2(4):160-163, 1991.

Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016.

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning , pages 1995-2003. PMLR, 2016.

Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.

Lei Xu. Deep bidirectional intelligence: Alphazero, deep ia-search, deep ia-infer, and tpc causal learning. In Applied Informatics , volume 5, pages 1-38. Springer, 2018.

Lei Xu, Pingfan Yan, and Tong Chang. Algorithm cnneim-a and its mean complexity. In Proc. of 2nd international conference on computers and applications. IEEE Press, Beijing , pages 494-499, 1987.

Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations , 2020.

Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems , 34:25476-25488, 2021.

Dengwei Zhao, Shikui Tu, and Lei Xu. Efficient learning for AlphaZero via path consistency. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 26971-26981. PMLR, 17-23 Jul 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed Generalized Weighted PCZero (GW-PCZero), which builds on EfficientZero and PCZero. The goal is to generalize the implementation of PCZero from board games to Atari games, which is achieved by adding the theorem “path consistency” to EfficientZero, extending the previous idea from PCZero. This paper further applies a weighting mechanism to path consistency, which calculates more accurate targets for agents to learn. Furthermore, this paper proves that neural-guided MCTS is guaranteed to find the optimal solution under PC constraint, providing a theoretical foundation for PC. The experiments show that under Atari 100k benchmark, GW-PCZero achieves 198% mean human normalized performance, slightly higher than the SOTA EfficientZero (194%). More importantly, GW-PCZero only consumes 25% computational resources of EfficientZero.\n\nSTRENGTHS:\n1. GW-PCZero achieves slightly higher performance under Atari 100k benchmark than the SOTA EfficientZero while consuming only 25% computational resources of EfficientZero.\n2. GW-PCZero generalizes the implementation of path consistency from board game to the case where the environment emits immediate reward, such as Atari game.\n3. This paper proves that neural-guided MCTS is guaranteed to find the optimal solution under PC constraint.\n4. The paper provides rich experimental data. In addition to the Atari 100k benchmark, the authors conducted experimental analyses on board games (Hex) by combining PCZero with the weighting mechanism. Furthermore, experiments were conducted on the Classic control problem - Cartpole. The diverse range of experimental environments shows the generality of this method.\n\nWEAKNESSES:\nIt appears that the proposed method is primarily a combination of two previous works, PCZero and EfficientZero, without introducing significantly novel ideas or concepts in the algorithm itself. However, overall, I would still think it is valuable to obtain the state-of-the-art with the combination.\n\nQUESTIONS:\n1. In Appendix 7.1, regarding the board game Hex, you mentioned that \"Weighted PCZero beats the original PCZero with a score of 175:163.\" However, based on the score of 175:163, it seems that the two are relatively close in performance, and it is not evident from this result alone the advantage of the Weighted mechanism. Probably, you want to show some confidence error. \n2. In Appendix 7.2, the experiments for the Classic control problem only include Cartpole, which is a relatively simple environment. Are there any other experiments that demonstrate the applicability of GW-PCZero to more complex control problems? For example, EfficientZero applied their work to Deepmind Control 100k, and it would be beneficial if you could apply your method to Deepmind Control 100k and compare it with their results.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GW-PCZero,  an RL algorithm based on neural-guided Monte Carlo Tree Search (MCTS). GW-PCZero adopts the idea of Path Consistency (PC) from prior work, i.e., a regularizer that encourages evaluation function to be consistent throughout an optimal path, to improve sample efficiency. Beyond this, GW-PCZero generalizes to the environment with reward given before the end of the episode, and appends a weight in the regularizer that decreases with the number of steps of the episode to account for increasing uncertainty later in the episode. The paper proves that the probability of finding the optimal path is lower-bounded, and achieves comparable / marginally better performance than the state-of-the-art (EfficientZero) with much less computational cost on many Atari environments.\n\nSTRENGTHS:\n**1. The writing is easy to follow and the idea is clearly conveyed.** There are many designs of the prior work that needs to be introduced, such as neural MCTS, path consistency, and re-analyzation of both MuZero and EfficientZero; moreoever, the motivations of PCZero and EfficientZero have to be clearly stated for the readers to understand the line of work. This paper does a good job in the first three section where the concepts are well-explained to the readers. The theorems and assumptions are also clearly stated in the methodology section.\n\n**2. The experiment results are solid and convincing.** The superior performance of GW-PCZero has been evaluated on Hex and many Atari environments, and the ablation on the most important hyperparameters, which are the PC weight factor and $\\lambda$, are provided in detail. Furthermore, the authors have submitted the code, which makes the result more convincing.\n\n\n**3. The algorithm has theoretically proved performance, which is a valuable contribution.** The paper has proved that GW-PCZero has a bound for probability of finding the optimal path, which is the first theoretical result in this line of work. I am convinced that the proving technique utilized in the paper would be beneficial for further theoretical research into neural MCTS.\n\nWEAKNESSES:\n**1. It seems that the result is somewhat sensitive to the choice of weighting hyperparameters.** In table 3, the scores are quite different with adjacent choice of $c_a$, and different environment seems to have very different optimal $c_a$. This is also the case in the selection of $\\lambda$ in table 1 of the appendix, and there is no monotonicity exhibited on either side of the optimal hyperparameter (e.g. 8*8 and 9*9 MCTS player).\n\n**2. Other minor problems:**\n\na) The second part of Table 3 in the main paper should also have a horizontal line between the first row and the rest of the table. \n\nb) Should Eq. 16 be clipped with 0 from below? by line 281 we know that when $i>10$ the weight will be negative, but it does not make sense to discourage path consistency at any time.\n\nQUESTIONS:\nI have four questions for this paper:\n\n1. The experiment results only show the effect of How sensitive the $\\lambda$ is on the board games, which is a special case. How sensitive would $\\lambda$ on the Atari environment be?\n\n2. In section 4.4, the $\\lambda$ is chosen according to the ratio between prior work performance and expert (human) performance. However,  the evaluation of expert (human) performance might not be available in real-life applications. Have the author considered to use a adaptable $\\lambda$ that dynamically changes during the training process? (I noticed that the authors mention \"systematic investigation on automatic weighting methods in the future\", but this is about a different hyperparameter.)\n\n3. The author claims that they can do marginally better than EfficientZero with a quarter of MCTS runs. While this is exciting, a constant factor improvement in the time complexity might be cancelled out by implementation and more values to calculate (e.g. the extra regularizer term). Is it possible for the authors to provide actual wall clock time for each method, or to explain that your method do not have a significant overhead compared to EfficientZero for each MCTS run? \n\n4. It is a little surprising that in the Appendix section 7.2, MuZero without Path Consistency (PC) cannot deal with environments as simple as cartpole, either with or without reanalyzation. Also, there are two interesting phenomenon about this figure, which are 1) for all curves, the rewards first increase quickly, then decrease to a relatively low level, and 2) with no reanalization, MuZERO without PC even seems to work better. Could the authors explain the figure more carefully?\n\nThere are also three suggestions besides addressing the questions above and in the weakness section: \n\n1. Limitation section are missing in the paper. While there are some places that implicitly mentions the limitation (e.g. line 327, \"it deserves a systematic investigation on better automatic weighting ... in the future\"), I suggest the author to think of more possible limitations (see limitation section) and summarize them in a separate paragraph.\n\n2. I suggest the authors to append a pseudocode in the appendix to more clearly shows how GW-PCZero works; it is even better if the authors could highlight the difference in the pseudocode with prior work such as PCZero. \n\n3. The authors claim that there is no immediate ethical or social impact of this work. While this is true, there are still broader impacts of the paper that still needs to be considered, such as potential misuse of the automated technology and potential job loss that needs to be considered.\n\nLIMITATIONS:\nThe authors implicitly mentioned limitations (line 327, \"it deserves a systematic investigation on better automatic weighting ... in the future\"). However, I suggest the authors to think more about limitations, such as hyperparameter sensitivity, manual selection of hyperparameters, possible limitation on future application by the nature of neural-guided MCTS, etc., and summarize them into a separate paragraph. As for potential negative societal impact, the author claims that there are no immediate negative societal impact. While this is true, I encourage the authors to be aware of the broader impact of their work on automated decision making.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a model-based RL method called GW-PCZero, which is built on EfficientZero and generalizes the path consistency (PC) constraint from board games with zero immediate rewards to environments with non-zero immediate rewards. The authors introduce a weighting average mechanism and use the mean f value of states along the path as the target for the PC constraint. \n\nAlthough the authors did comprehensive experiments to demonstrate that GW-PCZero slightly outperforms EfficientZero on the Atari 100k benchmarks with 26 games with less computation, they missed a published paper SCZero whose idea is essentially the same as the proposed GW-PCZero. This makes the novelty and comprehensiveness of GW-PCZero should be questioned. \n\nMeanwhile, since the author uses PC constraint on a off-policy RL algorithm EfficientZero, the off-policy issues may have significantly negative impacts on the effectiveness of PC loss. In the setting of off-policy RL, the author's claim on global constraint of PC loss should be questioned. The authors only make limited discussion and proposed a simple linear weighting trick, which causes PC loss to degenerate into the SC loss.\n\nSTRENGTHS:\n1. The paper extends the path-consistency to environments with non-zero immediate rewards.\n2. The paper uses significantly less computational resources.\n\nWEAKNESSES:\nThe most critical issues of this paper:\n1. The proposed GW-PC loss is extremely similar to a published paper 'Self-Consistent Models and Values'(Neurips 2021). The PC loss can be converted to SC loss easily as\n$l^{SC-residual}=\\sum_{k=0}^K (\\hat{r}(s_k, a_k)+\\gamma \\hat{v}(s_{k+1})-\\hat{v}(s_k))=\\sum_{k=0}^K (g(s_{k+1})+\\hat{v}(s_{k+1})-g(s_k)-\\hat{v}(s_k))=l^{PC}$\nHowever, it seems that the author did not notice SCZero paper at all, no citing or taking SCZero as the most important baseline.\n2. Setting path consistency on a off-policy algorithm like EfficientZero is not reasonable. For tasks with nonzero immediate rewards like Atari Games, the old state transitions collected by the old policy brought significant off-policy issue, which makes the proposed PC loss  hardly be a global constraint as the authors claimed. Considering the off-policy issue and computational limitations, authors proposed a sliding window and a linear weighting trick to reduce the impact of the marginal states within the sliding window. This makes the proposed PC loss has little improvement compared to the previous SC loss, no matter in terms of fundamental ideas or mathematics.\n\nMinors:\n1. The paper didn't report the median human normalized score.\n2. The coefficient of PC loss is required to be adjusted according to the task. Generally, this is not an common hyperparameter in most RL settings. And the performance should not be sensitive to the coefficient of PC loss.\n3. The paper didn't compare it with other data-efficient RL algorithms like IRIS and Dreamer.\n4. The paper over-claims the contribution. GW-PCZero only slightly outperform the full-version EfficientZero(with 120k training steps).\n\nQUESTIONS:\n1. Can you report the median human normalized score?\n2. Can you report the performance of GW-PCZero(c_\\alpha)? and also the performance of Dreamer and IRIS if possible.\n3. Can you report a performance with fixed coeff of PC loss? like \\lambda=0.2 or 0.4\n4. Can you report the performance of GW-PCZero with 120k training steps?\n5. Can you provide some experiment results that shows PC loss better than SC loss within Atari 100K setting?\n\nLIMITATIONS:\nThe paper's novelty should be questioned, since its PC loss is basically a 'reward-sum' version of DeepMind's SC loss published in 'Self-Consistent Models and Values', Neurips 2021.\nThe coefficient of PC loss needs to be adjusted according to the task. Generally, this is not an common hyperparameter in most RL settings. And the performance should not be sensitive to the coefficient of PC loss.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GW-PCZero, a reinforcement learning method, extending the technique of PCZero which is currently limited to board games and lacks theoretical backing. The GW-PCZero is designed for environments with non-zero immediate rewards, such as Atari games. It maintains path consistency by regularizing value estimation with the deviation from the mean value along the path, while a new weighting mechanism minimizes scouting variance. The paper provides the first theoretical proof that a neural-guided Monte Carlo Tree Search can guarantee finding an optimal solution under path consistency. And it reaches better performance on the Atari 100k benchmark with 26 games compared to the previous SoTA EfficientZero.\n\nSTRENGTHS:\n1. The paper is clear and understandable, especially in the details in Preliminary Section.\n2. The paper provides a theoretical guarantee under the constraint of path consistency.\n3. The performance increase in the experiments shown is laudable.\n\nWEAKNESSES:\n1. The novelty seems limited. Compared to PCZero, it modifies the PC target through a linear weighting method, which is a little tricky and common. \n2. The authors claimed a theoretical guarantee for path consistency (PC) for the first time. But the PC is the main contribution of PCZero instead of GW-PCZero, which is confusing. However, there is no guarantee for the weighting method, concerning convergence rate or optimality.\n3. No ablation for different weighting methods. Only the tradeoff c is considered. For example, as illustrated in Figure 1(left), why the weights are not exponentially decayed?\n\nQUESTIONS:\n1. As you mentioned in Definition 3.1 (L125), PC gives the constraint of the optimal path for the f values. For the optimal path, the f values keep the same in the path. But, I am wondering that \"when f values match in the path, is this path optimal? especially when neural nets estimate the f values.\"\n\n2. For tasks with much longer horizons, the computation cost of the path consistency can be much higher. How to reduce the corresponding cost in such cases?\n\nLIMITATIONS:\n1. The novelty of introducing a weighting mechanism without further clarification is limited. The authors claim that the weighting mechanism can mitigate the uncertainty/variance in L234, but no experiments or theorems are provided regarding the uncertainty/variance. \n\n2. The experiments and analysis of the weighting mechanism are not enough, which is the main contribution of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends PCZero to more general games whre the environment emits non-zero immediate rewards and proposes Generalized Weighted PCZero (GW-PCZero). GW-PCZero is built on EfficientZero with a generalized PC constraint. Specifically, GW-PCZero add an additional value consistence loss alone the sampled path, i.e., $L_{PC}(\\theta) = \\left(v(s_t;\\theta)-\\frac{1}{l+1} \\sum_{i=0}^l\\left[\\sum_{j=1}^i r_{t+j}+v\\left(s_{t+i} ; \\theta\\right)\\right]\\right)^2$, the MSE loss between $v(s_t;\\theta)$ and the average value of {$0$-step td-target, $1$-step td-target, ..., $l$-step td-target}. To reduce the bias and variance of the PC targets when training with off-policy data, GW-PCZero devise a weighting mechanism to give larger discounts to farther states, which is very similar to the idea of `td-lambda`. So, this paper can be considered as adding an additional `td-lambda` style value loss to EfficientZero. Experiments on the Atari-100k benchmark validate the superiority of GW-PCZero over EfficientZero.\n\nSTRENGTHS:\n* The paper is written clearly.\n* The proposed GW-PCZero becomes a new SOTA on the Atari-100k benchmark, which achieves a higher human normalized score than EfficientZero but with much less computational cost.\n\nWEAKNESSES:\n* 1. The path consistency exists not only on the `optimal` path but on any `on-policy` path.\n* 2. Although the paper has done a lot of theoretical analysis, the proposed method can be considered as  adding an additional `td-lambda` style value loss to EfficientZero. Therefore, the differences between the new proposed value loss functions in Eq. (15) and (17) with other typical value loss functions, e.g., td-lambda, v-trace, etc, should be discussed.\n* 3. For EfficientZero, the policy target reanalyze process and the value target reanalyze process are not necessarily separated into 2 independent passes. \n    * We can sample a slightly longer sub-trajectory $L_b=\\\\{s_t, s_{t+1}, \\cdots, s_{t+H}\\\\}$ for each item in a batch (but keep the total transition number in a batch the same), where $H>l$ and use a single batch of MCTS to get $\\pi^{MCTS}$ and $v^{MCTS}$ for all states in $L_b$. To compute $z_t, \\ldots, z_{t+H-l}$, we follow Eq. (9) in this paper but replacing $v$ with $v^{MCTS}$. For $z_{t+H-l-1}, \\ldots, z_{t+H}$, we set the $n$-step td targets with smaller $n \\in \\\\{l-1,\\ldots, 0\\\\}$, e.g., we set $z_{t+H}=v^{MCTS}(s_{t+H})$.\n    * Therefore, the computational cost (the number of MCTS runs) of EfficientZero is not necessarily much greater than GW-PCZero.\n* Minors:\n  * $v(s_{t+1};\\theta)$  (line 146) should be $v(s_{t+l};\\theta)$.\n  * $L_p$ (line 221) should be $L_b$.\n\nQUESTIONS:\n* What's the performance of GW-PCZero if setting the updating steps to 120k?\n* What's the performance of EfficientZero if directly using Eq. (9) as the value target (do not replace $v(s_{t+l};\\theta)$ with $v^{MCTS}(s_{t+l};\\theta)$? \n* For $L_b=\\\\{s_t, s_{t+1}, \\cdots, s_{t+H}\\\\}$, only $s_t$ is considered to be constrained with PC loss in real implementation because the sampled batch $L_p$ is too short to deal with the subsequent states in the same way (line 220-221). Why not directly using Eq. (12) as the loss function (which can utilizing all states in $L_b$)?\n* What if removing the value loss (of EfficientZero) in GW-PCZero and only keeping the PC-constant loss shown in Eq. (12)?\n* What if removing the PC-constant loss of GW-PCZero and replacing the value loss (of EfficientZero) with a `td-lambda` style or a `v-trace` style value loss?\n\nLIMITATIONS:\nPlease see the weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}