Title: Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

mento carlo tree search

state abstraction
Abstract: Monte Carlo Tree Search (MCTS) algorithms such as AlphaGo and MuZero have achieved superhuman performance in many challenging tasks. However, the computational complexity of MCTS-based algorithms is influenced by the size of the search space. To address this issue, we propose a novel probability tree state abstraction (PTSA) algorithm to improve the search efficiency of MCTS. A general tree state abstraction with path transitivity is defined. In addition, the probability tree state abstraction is proposed for fewer mistakes during the aggregation step. Furthermore, the theoretical guarantees of the transitivity and aggregation error bound are justified. To evaluate the effectiveness of the PTSA algorithm, we integrate it with state-of-the-art MCTS-based algorithms, such as Sampled MuZero and Gumbel MuZero. Experimental results on different tasks demonstrate that our method can accelerate the training process of state-of-the-art algorithms with 10%-45% search space reduction.

Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction

Abstract

Monte Carlo Tree Search (MCTS) algorithms such as AlphaGo and MuZero have achieved superhuman performance in many challenging tasks. However, the computational complexity of MCTS-based algorithms is influenced by the size of the search space. To address this issue, we propose a novel probability tree state abstraction ( PTSA ) algorithm to improve the search efficiency of MCTS. A general tree state abstraction with path transitivity is defined. In addition, the probability tree state abstraction is proposed for fewer mistakes during the aggregation step. Furthermore, the theoretical guarantees of the transitivity and aggregation error bound are justified. To evaluate the effectiveness of the PTSA algorithm, we integrate it with state-of-the-art MCTS-based algorithms, such as Sampled MuZero and Gumbel MuZero. Experimental results on different tasks demonstrate that our method can accelerate the training process of state-of-the-art algorithms with 10% -45% search space reduction.

1 Introduction

With the advancement of reinforcement learning (RL), AlphaGo is the first algorithm that can defeat human professional players in the game of Go [1]. With no supervised learning of expert moves in AlphaGo Zero [2] to restructured self-play in AlphaZero [3], until the recent MuZero that conducts Monte Carlo Tree Search (MCTS) in hidden state space with a learned dynamics model [4]. MuZero presents a powerful generalization framework that allows the algorithm to learn without a perfect simulator in complex tasks. Recently, EfficientZero [5] has made significant progress in the sample efficiency of MCTS-based algorithms. This development has opened up new possibilities for real-world applications, including robotics and self-driving.

When dealing with complex decision-making problems, increasing the search depth is necessary to achieve more accurate exploration in the decision space, but this also leads to higher time and space complexity [6, 7]. For instance, MuZero trained for 12 hours with 1000 TPUs to learn the Go game, and for a single Atari game, it needs 40 TPUs for 12 hours of training [4]. One approach to reduce the computation is the state abstraction method, which aggregates states based on a certain similarity measure to obtain a near-optimal policy [8, 9, 10]. state abstraction is a crucial technique in reinforcement learning (RL) that enables efficient planning, exploration, and generalization [11].

To reduce the search space of MCTS, previous studies have shown the potential of specific state abstraction techniques [12, 13, 14, 15]. However, finding the minimum abstract state space in these studies is an NP-Hard problem [16]. To our knowledge, this work is the first to define path transitivity in the formulation of tree state abstraction, which enables the discovery of the minimum abstract state space in polynomial time. Additionally, recent MCTS-based algorithms utilize neural networks to

estimate the value or reward of states, which may lead to errors in aggregating states with deterministic state abstraction functions. To address this issue, we proposed a probability tree state abstraction function that aggregates states based on the distribution of child node values, which enhances the robustness of aggregation and ensures transitivity.

This paper proposes the probability tree state abstraction ( PTSA ) algorithm to improve the tree search efficiency of MCTS. The main contributions can be summarized as follows: i) A general tree state abstraction is formulated, and path transitivity is also defined in the formulation. ii) The probability tree state abstraction is proposed for fewer mistakes during the aggregation step. iii) The theoretical guarantees of the transitivity and aggregation error bound are justified. iv) We integrate PTSA with state-of-the-art algorithms and achieve comparable performance with 10% -45% reduction in tree search space.

2 Related Work

2.1 MCTS-based Methods

MCTS is a rollout algorithm for solving sequential decision problems [17]. The fundamental idea of MCTS is to search for the most promising actions by randomly sampling the search space, and then expanding the search tree based on those actions [18]. The computational bottlenecks arise from the search loop, especially interacting with the real environment model of each iteration.

Combined with deep neural networks, MCTS-based methods have achieved better performance and efficiency in various complex tasks, such as board games [2], autonomous driving [19], and robot planning [20]. The model-based algorithm MuZero [4] predicts the environmental dynamics model for more efficient simulation. Based on MuZero , EfficientZero [5] is proposed for the training with limited data, which achieves super-human performance on Atari 100K benchmarks. However, both MuZero and EfficientZero require high computational consumption when dealing with complex action spaces. To address arbitrarily complex action spaces, the sample-based policy iteration framework [21] is proposed. Sampled MuZero extends MuZero by sub-sampling a small fraction of possible moves and achieves higher sample efficiency with fewer expanded actions and simulations. The experimental results show that planning over the sampled tree provides a near-optimal approximation [21]. To further reduce the number of simulations, Gumbel MuZero utilizes the Gumbel-Top-k trick to construct efficient planning [22].

2.2 State Abstraction

State abstraction is aimed at reducing the complex state space by aggregating the similar states [11]. The original state space S can be mapped into a smaller abstract state space S ϕ by state abstraction. By grouping similar states together, state abstraction can help to identify patterns and regularities in the environment, which can inform more effective decision-making [12].

There are two main challenges when applying state abstraction to RL problems. The first challenge is to decrease the value loss between S and S ϕ . With bounded value loss, the approximate state abstractions allow the agent to learn a near-optimal policy with improved training efficiency [11, 23]. The second challenge is to compute the smallest possible abstract state space, which is proven that the computational complexity is NP-hard [16]. The transitive state abstraction [11] is defined to efficiently compute the smallest possible abstract state space. However, most transitive state abstractions with deterministic predicates have low fault tolerance. To improve the robustness of aggregation, we measure the abstraction probability of the state pairs based on the expected value distributions. In addition, some previous studies have analyzed and discussed some specific state abstractions in tree structure [12, 13, 14, 15], but there is no formal definition and analysis for general tree state abstractions. To our knowledge, PTSA is the first method that defines a general formulation of tree state abstractions for deep MCTS-based methods and analyzes the transitivity and aggregation error under balanced search.

3 Preliminaries

In this section, the prerequisites for our proposed method are introduced. We consider an agent learning in a Markov decision process (MDP) represented as ⟨S , A , R, T , γ ⟩ , where S denotes the

state space, A denotes the action space, R : S × A ↦→ R denotes the reward function, T : S × A ↦→ P ( S ) denotes the transition model, and γ ∈ [0 , 1] is the discount factor. The goal is to learn a policy π : S ↦→ P ( A ) that maximizes the long-term expected reward in the MDP.

3.1 Monte Carlo Tree Search

MCTS-based algorithms typically involve four stages in the search loop: selection, expansion, simulation, and backpropagation. After N iterations of the search loop, MCTS generates a policy based on the current states. In the selection stage of each iteration, an action is selected by maximizing over UCB. Notably, AlphaZero [2] and MuZero [4], two successful RL algorithms developed based on a variant of Upper Confidence Bound (UCB) [24] called probabilistic upper confidence tree (PUCT) [25], have achieved remarkable results in board games and Atari games. The formula of PUCT is given by Eq. (1):

where k is the index of iteration, Q ( s, a ) denotes the value of action a in state s , c ( s ) is a hyperparameter for balancing the value score with the visiting counts N ( s, a ) , and P ( s, a ) is the policy prior obtained from neural networks.

3.2 State Abstraction in RL

State abstraction methods aggregate similar environment states to compressed descriptions [26], which simplify the state spaces and significantly reduce the computation time [27]. The state abstraction type is formulated as below [10]:

Definition 3.1. (State Abstraction Type) A state abstraction type is a set of functions ϕ : S ↦→ S ϕ related to a fixed predicate on state pairs: p M : S × S ↦→ { 0 , 1 } . When function ϕ aggregates the state pair ( s 1 , s 2 ) in MDP M , the predicate p M must be true: ϕ ( s 1 ) = ϕ ( s 2 ) = ⇒ p M ( s 1 , s 2 ) .

The conditions for state abstraction are usually strict, which may cause insufficient compression in state spaces. Recent studies have shown that transitive state abstraction [23] is computationally efficient and can achieve near-optimal decision-making, which is defined as:

Definition 3.2. (Transitive State Abstraction) For a given state abstraction ϕ with predicate p M , if ∀ ( s 1 , s 2 , s 3 ) ∈ S satisfies [ p M ( s 1 , s 2 ) ∧ p M ( s 2 , s 3 )] = ⇒ p M ( s 1 , s 3 ) , the state abstraction ϕ is a transitive state abstraction.

Some previous state abstraction functions are shown in Table 1, which aim to abstract similar states in general reinforcement learning methods. For instance, abstraction ϕ a ∗ considers the optimal actions and values of states, which is also widely studied in MCTS-based methods [28].

Transitive state abstraction can reduce the computational cost of finding the smallest abstract state space. As shown in Table 1, most approximate state abstractions are not transitive. However, transitivity is challenging to MCTS-based methods due to the tree structure. In MCTS, a path may contain multiple states, so it is necessary to derive the transitivity of the path from the transitivity of the states. Finding the smallest abstract space in the search tree becomes an NP-hard problem if the state abstraction function lacks transitivity in the path.

4 PTSA-MCTS

In this section, we introduce the proposed probability tree state abstraction ( PTSA ) algorithm, which improves the tree search efficiency for MCTS. As described in Figure 1, PTSA algorithm improves the search efficiency of MCTS in two aspects: one is reducing the search space by abstracting the original search space, and the other is finding the smallest abstract space efficiently by transitive probability tree state abstraction ϕ Q ψ α . The organization of this section is as follows: Subsection 4.1 gives the formulation of general tree state abstraction. Subsection 4.2 presents our proposed probability tree state abstraction. Subsection 4.3 presents the PTSAZero algorithm, which integrates PTSA with Sampled MuZero, and more information can be found in the Appendix. Subsection 4.4 provides the proofs of transitivity in tree structures and bounded aggregation error under balanced exploration.

4.1 Tree State Abstraction Formulation

The formulation of our tree state abstraction is provided with a general abstraction operator for the tree structure, which can be utilized with arbitrary state abstraction types. To facilitate the derivation of theorems and properties, the definition of tree state abstraction is given:

Definition 4.1. (Tree State Abstraction)

For a given tree, V and B denote the node set and path set respectively. A tree state abstraction is a function ϕ : V ↦→ V ϕ & B ↦→ B ϕ with node predicate p vM and path predicate p bM on the sibling path pair:

In the tree structure, a path is a sequence of nodes and a node denotes the representation of the corresponding state. The path and node predicates abstract corresponding states of the path and node respectively. For a given path pair ( b 1 , b 2 ) with the same length l , the predicate on this path pair can be decomposed as:

where v b i -1 ( i = 1 , ..., l ) is the i -th node of path b with length l . When function ϕ aggregates the path pair ( b 1 , b 2 ) in MDP M , the predicate p M must be true:

and path b 1 and path b 2 belong to the same abstract cluster.

The tree state abstraction is applied to two search paths of equal length that start from the same parent node. This ensures that each node along the paths has a corresponding potential abstracted node, while preserving the Markov property after aggregation. For instance, the path pair ( b 1 , b 2 ) from different parent nodes v a , v b can not be aggregated, which violates the Markov property in the state transitions v a → v b 1 0 → . . . → v b 1 l -1 and v b → v b 2 0 → . . . → v b 2 l -1 .

Previous studies have proven that tree state abstraction can be an efficient approach to reducing branching factors in MCTS [12, 27, 14]. However, finding the smallest abstract state space is an NP-Hard problem in the previous studies [16]. Following the definition of transitive state abstraction [11], we define the path transitivity in the tree state abstraction formulation:

Root

1. Selection

Il. Expansion(

IIl. Simulation（—.·→）

aK

IV.Backpropagation

V.Abstraction:

AbstractedPaths

AggregatedPaths

Transitive

Definition 4.2. (Path Transitivity) For a given tree state abstraction ϕ with predicate p bM , if ∀ ( b 1 , b 2 , b 3 ) ∈ B satisfies [ p bM ( b 1 , b 2 ) ∧ p bM ( b 2 , b 3 )] = ⇒ p bM ( b 1 , b 3 ) , the state abstraction ϕ has path transitivity.

The definition of path transitivity extends the equivalence of state abstraction in tree state space and general RL state space. Tree State abstraction for MCTS maps the original path space B into the abstract path space B ϕ . For non-transitive tree state abstraction, it is necessary to determine whether all possible path pairs belong to the same abstract cluster, and paths may appear repeatedly in path pairs, which requires a massive computation cost to obtain the smallest B ϕ .

4.2 Probability Tree State Abstraction

State-of-the-art MCTS-based algorithms utilize neural networks to estimate the value or reward of states. However, hard constraints from previous state abstractions can lead to incorrect aggregation during the early training stage. To reduce the probability of states being mapped to the incorrect abstract space due to bias in network prediction, a novel probability tree state abstraction ϕ Q ψ α is proposed in this work:

Definition 4.3. (Probability Tree State Abstraction ϕ Q ψ α ) For a given α ∈ [0 , 1] with node predicate p vM and path predicate p bM , the aggregation probability of ϕ Q ψ α is defined as:

where P { Q ψ ( v, a ) } = exp ( Q ∗ ( v,a )) ∑ j ∈A exp ( Q ∗ ( v,j )) , and D JS is the Jensen-Shannon divergence.

ϕ Q ψ α encourages nodes that have the same candidate actions with similar value distribution expectations to be aggregated. Using Jensen-Shannon divergence instead of Kullback-Leibler divergence is more advantageous for numerical stability during computation.

4.3 Integration with Sampled MuZero

Our proposed PTSA algorithm can be integrated with state-of-the-art MCTS-based algorithms. The integration includes two main components: offline learning and online searching. Offline learning involves updating the dynamics, prediction, and value networks by sampling trajectories from a buffer. Online searching involves interacting with the environment to obtain high-quality trajectories, similar to MuZero algorithm. Algorithm1 shows how PTSA can be integrated with Sampled MuZero [21] during the searching stage. Compared with the original Sampled MuZero , lines 4-12 describe how to collect all searched paths and update the corresponding node values during the multiple iterations. Lines 13-19 describe how tree state abstraction reduces the search space efficiently. Based on Theorem 4.4, abstracting the most recently searched path is enough to find the smallest abstract space for MCTS-based methods. ( ϕ ( b i ) = ϕ ( b s )) returns a boolean value, where "true" denotes aggregating b i and b s . This boolean value is determined by calculating the probability P ( ϕ ( b i ) = ϕ ( b s )) based on Eq.(5) and Eq.(6).

S L list records the searched paths in the current search tree. S L .delete ( b ) and S L .add ( b ) refer to removing and recording path b in S L respectively. The pruning ( b j ) action denotes removing unique nodes of path b j compared to the other abstracted path in the search tree. h denotes the hidden feature of the original real environment state. Algorithm 1 can be generalized to all tree state abstraction functions by replacing ϕ Q ψ α . The selection, expansion, and backpropagation steps are the same with Sampled MuZero [21]. To maintain the balance of ∑ b N ( s, b ) , the visit count of the aggregate node needs to be accumulated into the corresponding state pair. Furthermore, aggregated nodes with different sets of legal actions could lead to unnecessary exploration of invalid or irrelevant parts of the search space, slowing down the search and potentially reducing the quality of the results. In our implementation, we ensure that the aggregated node only expands legal actions for the abstracted state, thus avoiding any negative impacts caused by illegal actions. An undoing aggregation operation has been considered in 1. As the value estimation becomes more accurate, some previously aggregated

Algorithm 1 PTSAZero

nodes in the search will no longer be aggregated in the new search. With each new timestep, the value of the search tree nodes is re-evaluated, leading to changes in the following aggregation results.

In Sampled MuZero algorithm, the computational complexity of simulating from the root node can be expressed as O ( N s · ( O ( S ) + O ( D ) + O ( P ) + O ( V ))) . N s represents the number of simulations, and O ( S ) denotes the computational complexity of the simulation process, which includes selecting children, expanding, and backpropagating. Additionally, O ( D ) , O ( P ) , and O ( V ) denote the computational complexities of the dynamics network d θ , policy network p θ , and value network v θ , respectively. The dynamics network predicts the next hidden state z and corresponding reward r based on the current hidden state h and action a . In Algorithm 1, the time complexity is given by O ( N s · (log |A| N s · c p + O ( S ) + O ( D ) + O ( P ) + O ( V ))) under balanced search. The balanced search term log |A| N s · c p accounts for the exploration of child nodes, where |A| represents the number of possible actions, and c p is a constant controlling exploration. Since tree state abstraction reduces the branching factor, our algorithm enhances the efficiency of selecting child nodes with a smaller N s . The specific computational time of different methods can be found in Appendix F.

4.4 Theoretical Justification

Following the formulation of the tree state abstractions, the theoretical analysis is conducted from the following perspectives: i) Transitivity; ii) Aggregation error.

4.4.1 Transitivity

To abstract tree paths efficiently, our next result shows the relationship between path transitivity and node transitivity:

Proof . See Appendix A.

Theorem 4.4 indicates that path transitivity is a sufficient and necessary condition for node transitivity. Compared with the previous transitive state abstractions, the proposed ϕ Q ψ α is also transitive for paths as the following proposition given:

Proposition 4.5. The probability of transitivity for ϕ Q ψ α can be computed as:

Proof . See Appendix B.

The computational complexity of computing the smallest possible abstract state space for transitive state abstraction is O ( |S| 2 · c p ) [11], where c p denotes the computational complexity of evaluating for a given state pair.

4.4.2 Aggregation Error Bound

Most approximate state abstractions are non-transitive since their cumulative aggregation value errors are unbounded. We define the aggregation error E ϕ in path set B as:

For instance, consider a commonly used approximate state abstraction ϕ Q ∗ ε [23]:

with the value loss L ϕ Q ∗ ε = V ∗ ( s ) -V π ϕ Q ∗ ε ( s ) ≤ 2 εR max (1 -γ ) 2 . The aggregation error can be bounded as 6 εR max (1 -γ ) 2 if two paths of length 3 are abstracted.

In Algorithm1, the aggregation times must be less than the current branching factor 2 . The next result extends to the general tree state abstractions in Algorithm1. The path transitivity implies that the abstracted paths present in the current B ϕ must belong to different abstract clusters, which can give the following theorem:

Theorem 4.6. Considering a general tree state abstraction ϕ with a transitive predicate p ( L ϕ ≤ ζ ) , the aggregation error in Alg. 1 under balanced search is bounded as:

If predicate p ( L ϕ ≤ ζ ) is not transitive, the aggregation error is bounded as:

Proof. See Appendix C.

Theorem 4.6 provides a theoretical guarantee for finding the smallest B ϕ within aggregation error bound in Algorithm1. At the same time, this theorem also points out that the transitivity has an important effect on the tree structure: as the size of the action space |A| increases, the aggregation error upper bound for transitive tree state abstraction will decrease, while the upper bound for non-transitive state abstraction will increase.

5 Experiments

In this section, experiments focus on tree state abstraction for computational efficiency improvement. Firstly, PTSA is integrated with state-of-the-art MCTS-based RL algorithms Sampled MuZero and Gumbel MuZero and evaluated on various RL tasks with 10 seeds. The aggregation percentages are also evaluated, which reflects the search space reduction in different tasks. In addition, the comparison between probability tree state abstraction and other state abstraction functions is also conducted.

5.1 Results on Performance

To demonstrate the improvement in computational efficiency, the PTSA is integrated with Sampled MuZero in Atari and classic control benchmarks, as well as with Gumbel MuZero in the Gomoku benchmark. The performance is compared against several MCTS-based RL algorithms, including (i) MuZero [4], (ii) Sampled MuZero [21], (iii) EfficientZero [5], (iv) Gumbel MuZero [22].

5.1.1 Atari

Atari games are visually complex environments that pose challenges for MCTS-based algorithms[4, 30]. The results for each benchmark and the normalized result are shown in Figure 2, and the shaded intervals represent the standard deviation of the performance across the different random























































PTSA



(a) Alien













































PTSA



(b) Boxing











































PTSA



(c) Freeway







































PTSA



(d) Pong















































PTSA



(e) Tennis













Time











Time Return

MuZero N

MuZero N=18

SMuZero N=30

SMuZero N=18

EfficientZero N=30

PTSAZero N=18



=30

(f) Normalized Score

seeds. As Gumbel MuZero does not require large simulations for Atari and control tasks, we only compare its performance in the Gomoku game. When the simulation times N = 18 , MuZero and Sampled MuZero fail to converge within the maximum training time in some tasks. Since the search space of MCTS is mapped into the abstract space with a smaller branching factor, PTSAZero ( α = 0 . 7 ) can converge rapidly with fewer simulations. Although EfficientZero improves the sampling efficiency of MuZero, the increased complexity of the network entails more time to converge with the same computational resources. EfficientZero is tested with different simulation times, and the best case in time efficiency with N = 30 is shown. The normalized score is computed by s norm = ( s agent -s min ) / ( s max -s min ) , and the normalized time is computed by t norm = ( t agent -t min ) / ( t max -t min ) . The experiment results on Atari benchmarks indicate that Sampled MuZero with PTSA can achieve comparable performance with less training time. For MuZero-based algorithms which require massive computational resources and parallel abilities, PTSA provides a more efficient method with less computational cost.

5.1.2 Control Tasks

In the comparison experiments conducted on Gym benchmarks, including classic control and box2d tasks, certain modifications are made to increase the tree search space in control tasks. Specifically, the action spaces of CartPole-v0 and LunarLander-v2 are discretized into 100 and 36 dimensions, respectively. This discretization of the action space necessitates MCTS-based algorithms to run a larger number of simulations and increases the number of sampled actions in Sampled MuZero.

For both SMuZero and PTSAZero, the number of sampled actions is set to 25 in CartPole-v0 and 12 in LunarLander-v2. Figure 3 demonstrates that PTSAZero exhibits superior computational efficiency compared to other methods. In the LunarLander-v2-36 task, PTSAZero significantly improves the training speed of MuZero by a factor of 3.53. These results highlight the effectiveness of PTSAZero in enhancing the efficiency and performance of MuZero-based algorithms in various control tasks.

5.1.3 Gomoku

Gomoku is a classic board game with multi-step search, where the agent is asked to beat an expert opponent on the 15 × 15 and 19 × 19 boards. Gumbel MuZero replaces heuristic mechanisms in original MCTS algorithms for a smaller number of simulations [22]. In board game tasks, we integrate PTSA algorithm with Gumbel MuZero: tree state abstraction ϕ Q ψ α is utilized to reduce the search space, as described in Algorithm 1. The observation of the Gomoku task is a 2-dimensional matrix describing the distribution of the pieces, and the number of sampled actions for SMuZero,





























 











PTSA









e(min)

(a) CartPole

















(min)



















 

 







 

PTSA





(b) Lunarlander

















(min)

































PTSAG



(c) Gomoku-15

×

15





















(min)





















 

 





 

P



T



A





(d) Gomoku-19

×

19















































(a) Atari













































&

Control

Gumbel MuZero, and PTSAGZero is 30. The training returns of different methods w.r.t. training steps are shown in Figure 3, where a value of 1 represents a win and -1 represents a loss. The results demonstrate that Gumbel MuZero can speed up the training process of Sampled MuZero, and PTSA can provide incremental improvement of Gumbel MuZero . In the Gomoku19 × 19 task, despite accelerating the convergence speed of the original algorithms, PTSA does not significantly improve the optimal performance of the algorithms. Experimental results also show that MuZero-based algorithms require a larger number of simulations to learn an effective policy due to the increased size of the action space.

5.2 Search Space Reduction

To analyze the abstracted tree search space, Figure 4 shows the aggregation percentage (the average proportion of aggregated paths) during the training process of PTSAZero on Atari, Control, and Gomoku tasks. Results demonstrate that tree state abstraction reduces the original branching factors by 10% up to 45% . As the network parameters are updated during the training process, the aggregation percentage tends to stabilize. As the training progresses towards convergence, the aggregation percentage decreases from 23 . 7% to 8 . 1% in the Freeway task, while in the Gomoku 19x19 task, it increases from 10 . 1% to 27 . 8% . The trend of the aggregation percentage is influenced by the initialization of network and the initial values for the nodes. The converged aggregation percentage may vary depending on specific algorithm parameters and task environments, and it does not imply a certain range of branching factors for all tasks. Each task has unique characteristics and complexities, which can influence the effectiveness of different abstraction functions. It should be emphasized that a larger reduction in the state space does not guarantee improved training efficiency. While a higher aggregation percentage indicates a greater reduction in the search space, the aggregation quality of the abstractions also determines the impact on training efficiency. Improving the accuracy and quality of tree state abstraction during the training process is a future challenge that deserves further research. Furthermore, since the search space of MCTS is reduced by tree state abstraction, the search depth of PTSAZero is deeper than that of SMuZero with the same number of simulations. More results of search depth can be found in Appendix F.

5.3 Comparison with Other State Abstraction Functions

We compare the effectiveness of our proposed probability tree state abstraction with other state abstraction functions on Atari and gym benchmarks by integrating them into PTSA algorithm. Referring to the speedup evaluation method in Gumbel MuZero [22], Table 2 shows the speedup of PTSAZero with different state abstraction functions compared to the Sampled MuZero . The specific usage and properties of other state abstraction functions can be found in previous works [23, 11, 29]. We adjusted hyperparameters for different state abstraction functions and selected the best values ( ϵ and d are set to 0.5 and 0.2, respectively). The results demonstrate that the proposed probability tree state ab-





































a

*





a

*





Q

*





Q

*





Q

*

d





Q



straction ϕ Q α ψ can achieve better speedup performance on average compared to other state abstraction functions. To evaluate the robustness of ϕ Q ψ α , PTSAZero with different state abstraction functions with N = 10 is compared in the Pong task. As shown in Figure 5, SMuZero and MuZero can not learn an effective policy, and PTSAZero with ϕ ε Q ∗ and ϕ a ∗ may aggregate incorrect states, which leads to a decrease in performance. We also find that the state abstraction function ϕ a ∗ and ϕ Q ∗ have strict requirements for the abstract conditions, resulting in a small search space reduction ( 1 . 5% and 2 . 8% respectively). Compared with other state abstraction functions, ϕ Q ψ α can abstract the original search space more accurately and robustly, thus accelerating the training process.

6 Conclusion

This paper introduces PTSA algorithm for improving the computational efficiency of MCTS-based algorithms. For efficient abstraction in tree search space, we define path transitivity in the formulation of tree state abstraction. Furthermore, we evaluate that the proposed probability tree state abstraction has a better performance compared with previous state abstraction functions. The experimental results demonstrate that PTSA can be integrated with state-of-the-art algorithms and achieve comparable performance with 10% -45% reduction in tree search space. However, the main limitation of the proposed method is that the parameters of some state abstraction functions need to be manually designed to obtain a more accurate abstract state space. Furthermore, selecting an appropriate state abstraction function based on the characteristics of the state space and transition model is also a potential challenge. Further research will be conducted to address these issues for better performance.

References

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 , 2017.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems , 34:25476-25488, 2021.

Yuxiang Sun, Bo Yuan, Yongliang Zhang, Wanwen Zheng, Qingfeng Xia, Bojian Tang, and Xianzhong Zhou. Research on action strategies and simulations of drl and mcts-based intelligent round game. International Journal of Control, Automation and Systems , 19(9):2984-2998, 2021.

Debin Zhao, Zhengyuan Hu, and Yinjian Yang. An mcts-based recommender system for education complex. In 2022 International Conference on Machine Learning and Knowledge Engineering (MLKE) , pages 323-326. IEEE, 2022.

Sultan Javed Majeed and Marcus Hutter. Performance guarantees for homomorphisms beyond markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 7659-7666, 2019.

Marcus Hutter. Extreme state aggregation beyond markov decision processes. Theoretical Computer Science , 650:73-91, 2016.

David Abel, Dilip Arumugam, Lucas Lehnert, and Michael L Littman. Toward good abstractions for lifelong learning. In Proceedings of the NIPS workshop on hierarchical reinforcement learning , page 92, 2017.

David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for lifelong reinforcement learning. In International Conference on Machine Learning , pages 10-19. PMLR, 2018.

Jesse Hostetler, Alan Fern, and Tom Dietterich. State aggregation in monte carlo tree search. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, 2014.

Aijun Bai, Siddharth Srivastava, and Stuart Russell. Markovian state and action abstractions for mdps via hierarchical mcts. In IJCAI , pages 3029-3039, 2016.

Samuel Sokota, Caleb Y Ho, Zaheen Ahmad, and J Zico Kolter. Monte carlo tree search with iteratively refining state abstractions. Advances in Neural Information Processing Systems , 34:18698-18709, 2021.

Alexander Dockhorn, Jorge Hurtado-Grueso, Dominik Jeurissen, Linjie Xu, and Diego PerezLiebana. Game state and action abstracting monte carlo tree search for general strategy game-playing. In 2021 IEEE Conference on Games (CoG) , pages 1-8. IEEE, 2021.

Eyal Even-Dar and Yishay Mansour. Approximate equivalence of markov decision processes. In Learning Theory and Kernel Machines , pages 581-594. Springer, 2003.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games , 4(1):1-43, 2012.

Jienan Chen, Cong Zhang, Jinting Luo, Junfei Xie, and Yan Wan. Driving maneuvers prediction based autonomous driving control by deep monte carlo tree search. IEEE transactions on vehicular technology , 69(7):7146-7158, 2020.

Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems , 34:27580-27591, 2021.

Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning , pages 4476-4486. PMLR, 2021.

Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference on Learning Representations , 2022.

David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning , pages 2915-2923. PMLR, 2016.

Sylvain Gelly and Yizao Wang. Exploration exploitation in go: Uct for monte-carlo go. In NIPS: Neural Information Processing Systems Conference On-line trading of Exploration and Exploitation Workshop , 2006.

Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence , 61(3):203-230, 2011.

David Andre and Stuart J Russell. State abstraction for programmable reinforcement learning agents. In Aaai/iaai , pages 119-125, 2002.

Ankit Anand, Ritesh Noothigattu, Parag Singla, et al. Oga-uct: On-the-go abstractions in uct. In Twenty-Sixth International Conference on Automated Planning and Scheduling , 2016.

Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng-Yu Tsai, I Wu, Cho-Jui Hsieh, et al. Are alphazero-like agents robust to adversarial perturbations? Advances in Neural Information Processing Systems , 35:11229-11240, 2022.

Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. In AI&M , 2006.

Weirui Ye, Pieter Abbeel, and Yang Gao. Spending thinking time wisely: Accelerating mcts with virtual expansions. arXiv preprint arXiv:2210.12628 , 2022.

Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging { AI } applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) , pages 561-577, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach called Probability Tree State Abstraction (PTSA) to improve the efficiency of Monte Carlo Tree Search (MCTS) algorithms, which have shown remarkable performance in challenging tasks. The computational complexity of MCTS algorithms is influenced by the size of the search space, and the proposed PTSA algorithm aims to address this issue. The algorithm introduces a general tree state abstraction with path transitivity, which helps in reducing the number of mistakes during the aggregation step. The theoretical guarantees of transitivity and aggregation error bound are also provided. The PTSA algorithm is integrated with state-of-the-art MCTS-based algorithms, including Sampled MuZero and Gumbel MuZero, and experimental results on various tasks demonstrate its effectiveness. The PTSA algorithm accelerates the training process of these algorithms, achieving a search space reduction of 10% to 45%.\n\nSTRENGTHS:\n1. The approach of aggregation considers the entire path, not only a state, is novel and unique.\n\n2. The PTSA algorithm presented in this paper can be applied with any other state abstraction functions mentioned in previous studies, in a general way.\n\n3. The paper provides extensive experimental data. It includes environments such as Atari games, as well as tasks with continuous action spaces like CartPole and LunarLander, and board games like Gomoku. The rich variety of experimental environments demonstrates the effectiveness of the proposed method across various tasks.\n\n4. Integrate PTSA with state-of-the-art algorithms can achieve comparable performance with smaller branching factors. In other words, PTSA provides a more efficient method with less computational cost.\n\nWEAKNESSES:\n1. The meaning of probability in PTSA (in Definition 4.3) is not well-defined and requires further clarification. This will be addressed in the Questions section below.\n\n2. There are some errors in the proofs presented. This will be discussed in detail in the Questions section as well.\n\nQUESTIONS:\n1. Why does $v_0.pruning$ do in line 17 in Algorithm 1? Any difference from $S_L.delete(b_j)$. \n\n2. What role does the probability $\\mathbb{P}$ in Definition 4.3 play in Algorithm 1?And, how to calculate $\\phi$ in line 15 in Algorithm 1? In other words, when does $\\phi(b_i)=\\phi(b_s)$ hold true? Are both related? \n\n3. In your paper, you mentioned a previous work titled \"Monte Carlo Tree Search with Iteratively Refining State Abstractions.\" That method directly calculates the distance between states and performs aggregation if the distance, denoted as $d(s_1, s_2)$, is below a threshold. This approach differs from the method proposed in your paper, but both aim to reduce the branching factor of MCTS. Have you conducted any experiments comparing your method with the approach mentioned above? I couldn't find any analysis of that method in Table 1 or the experimental section below. Some insight into the reason for this omission should be provided. \n\n4. This paper mentioned “reduce the computation time” with abstraction. My question (or curiosity) is how much overhead the checking operations (in Lines 14 and 15) incur. Note that in line 207 there is a time complexity which is required to be described in more detail, like $\\log N_s$. \n\n5. Equation (19) in the appendix is written incorrectly. Need to be fixed. For example, the final $p_{bM}(b_2, b_3)$ should be $p_{bM}(b_1, b_3)$. Also fix some other wrong indices in (19).\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel search algorithm, PTSA, to improve the search efficiency of MCTS. Empirical results show that PTSA can be integrated with Sampled MuZero and Gumbel MuZero and can reduce the original branching factor by 10% up to 45%.\n\nSTRENGTHS:\nThe proposed PTSA algorithm can reduce the branching factor of MCTS and improve the computational efficiency of MCTS-based algorithms. The authors also provide both theoretical and empirical analyses.\n\nWEAKNESSES:\n* The author claims that the proposed method can reduce the branching factor by 10% up to 45%. However, the result is based on only five Atari games. Based on Figure 3, the aggregation percentage varies across different Atari games. Can these five games represent all Atari-57 games? It would be more convincing to run more Atari games to support the claims.\n\n* Moreover, it is unclear for the aggregation percentage on control tasks and Gomoku experiments. Without these experiments, it is inappropriate to claim “reduce branching factor by 10% up to 45%”.\n\n* The time complexity of the proposed approach is higher than the original MCTS. It is unclear whether PTSAZero will still improve its efficiency when running under a larger simulation number. Currently, the authors only run “PTSAZero N=18” in Atari experiments. Will “PTSAZero N=30” perform better than “PTSAZero N=18”?\n\n* Besides, in the board games such as Gomoku or Go, it is common to run large simulation numbers such as N=400 or N=800 during evaluation. It would be better to provide additional experiments/analyses to demonstrate the scale-up ability for PTSAZero. For example, providing the aggregation percentage/time usage/strength when using different simulation numbers.\n\nQUESTIONS:\n* In Algorithm 1, line 15, if $b_i$ and $b_s$ have different lengths, will their $\\phi_{Q_{\\alpha}^{psi}}(b)$ be different? In addition, what is the definition for $\\phi_{Q_{\\alpha}^{psi}}(b)$? Definition 4.3 only shows the probability. \n* In Algorithm 1, line 17, $v_0$ is root node and $b_j$ is a selection path. what does $v_0$.prunning($b_j$) mean?\n* In Figure 2, will PTSA get better performance when using a larger simulation (N=30)? Current experiments only used N=18. It would be better to add another experiment with a larger simulation to show the scale-up ability of PTSA.\n* In the Gomoku experiment, what does the expert opponent stand for? How many simulations are used in the Gomoku evaluation? As Gomoku is a two-player game, why not compare PTSAZero to other methods directly?\n* line 302: “The winning rates of different methods w.r.t. training time are shown in Figure 4”. Should the range of the win rate be between 0 and 1 in Figure 4?\n* In Figure 3, it seems that the aggregation percentage varies across different Atari games. Which type of game may have a higher aggregation percentage? Why do you choose these games? Can these five games represent Atari-57 games? Do you have more experiments on other Atari games?\n* In Atari experiments, “As Gumbel MuZero does not require large simulations for Atari and control tasks”. In fact, Gumbel MuZero improves training efficiency by only using N=2 in Pacman, and the result is comparable to N=50. It would be more convincing to add additional experiments to compare the training efficiency between “Gumbel MuZero N=2” and “PTSAGZero N=2“ in Atari experiments.\n* In Figure 2 (f),  the label of the green curve is “MuZero N=50”, should it be “MuZero N=30”?\n* Line 17, typo: Muzero -> MuZero.\n* Figure 2, typo: state-of-art -> state-of-the-art.\n* Figure 3 is shown after Figure 4. Please fix the order of these figures.\n\nLIMITATIONS:\nThe authors have addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel tree state abstraction function (PTSA) for use during MCTS. The primary contributions of the paper are algorithmic and empirical. The key idea involves aggregating paths in the tree if their Q-values (as probabilities) along the path closely match an existing path with the same parent node. An analysis of the abstraction quality and error bounds are included. Experiments on Atari and Gym environments show that a recent MCTS variant leveraging PTSA outperforms a number of strong baselines.\n\nUPDATE: I thank the authors for their detailed response. After reading the other reviews and comments, I'm more inclined to recommend acceptance and have updated my score to reflect that.\n\nSTRENGTHS:\n+ The paper tackles an important problem of accelerating MCTS search. It does so using tree state abstraction. The approach is intuitively clear and is also supported by analysis. \n\n+ The paper proposes a novel tree state abstraction function based on path transitivity. The abstraction function is based on the difference in the Q values of the nodes (converted to probabilities) in the path. Although important implementation details are not clear to me, the intuition that abstracting entire paths accelerates search makes sense as does the abstraction of only the most recent path during search leading to smaller trees during online search. The paper is accompanied by analysis showing the correctness of the approach and an error bound under certain conditions. Overall, the algorithm seems to have high novelty.\n\n+ The experiments are conducted on a number of Atari and Gym environments. Sampled MuZero with the proposed abstraction (PTSA) outperforms a number of strong baselines by a significant margin. The implementation seems to work very well. This seems to be a new state of the art in state abstraction for modern MCTS variants.\n\nWEAKNESSES:\n- The approach is intuitively clear and seems to perform well empirically, which increases confidence. However, I found the description of the implementation details of Algorithm 1 difficult to follow. Please consider including a more careful description of the implementation in Section 4. The issue is exacerbated by the absence of code. This is currently preventing me from giving the paper a higher score.\n  - For example, the actual implementation of the algorithm in L15 of Algorithm 1 is unclear to me. I expect it to involve Eq 5 with some value of $\\alpha$ like 0.7. But Eq 5 returns a real-valued prob estimate for a path pair (b_i, b_s). How is that turned into a boolean value (True / False) in L15? It's probably not real-valued equality. This is a key detail so please explain.\n  - There are a number of other implementation details that are difficult to find or missing. See the questions for examples.\n\n- Given that the primary contribution is algorithmic and empirical, I'd have hoped to see the source code included. Reproducibility is going to be challenging without it and since this paper seems to establish a new state of the art, I'd encourage the authors to release their code.\n\nQUESTIONS:\n- I had a number of questions about the exact implementation\n  - What exactly is the implementation of the branching condition of L15 of Algorithm 1? How does it relate to Eq 5 and 6 which is defined as a function taking two inputs (b_i, b_j) and returning a probability?\n  - What exactly is learned during offline learning vs online search? $d, v, p$ are likely learned offline. What about the abstraction function $\\phi$? This seems online to me. Correct?\n  - What is $l$ set to in the implementation? How does it value impact performance?\n  - What is the implementation of the pruning function in L17 of Algorithm 1?\n  - How are the legal actions for the abstracted state computed?\n  - What is the size of $S_L$? How was it chosen? How does varying it affect performance?\n\n- As described in L346-349, there seem to be a number of choices for the designer to make. These are not clear to me at the moment besides the obvious ones (e.g., $\\alpha, N$). Please enumerate what exactly needs to be hand-designed or set manually for a given domain and what can be used off-the-shelf.\n\n- Is there a reason to not include code? Will code be included in the final version?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
To accelerate MCTS, the paper proposed a novel probability tree state abstraction (PTSA) algorithm to improve the search efficiency of MCTS. 
They define states that are similar by using path transitivity and claim that such a method can have fewer mistakes. According to the results of Atari and Gomoku, the method can be 10% ~ 45% faster.

STRENGTHS:
1. The method provides some theoretical guarantee.
2. The experiments take place in many environments.
3. The ablation studies have tried many abstraction functions.

WEAKNESSES:
1. The intuition of the paper is weird.  The method required of all states on the paths needs to be similar. However, there are two problems here. First, the V value might be more incorrect at the beginning. Second, even if the V value is correct for the whole path, this method reduces the chance of pruning more nodes. For example, in Atari, agents can reach the exact same state with different paths. Since the environment is MDP, we should merge those two states. 
 
2. It is unknown for the performance when the simulation is higher. The abstract error normally increases when the simulation increase. The method might delete some good paths that can only be identified after numerous simulations.

QUESTIONS:
1. How do you prune a path from a tree? What will happen to those branches that are on the path?
2. Have you considered abstraction functions that also require the best action should be the same[1]?
[1] Are AlphaZero-like Agents Robust to Adversarial Perturbations?

LIMITATIONS:
Stated in the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggests a method of abstracting the state space explored by a Monte Carlo Tree Search (MCTS) algorithm, in order to reduce the complexity of the search. We can create an abstraction of the state space for MCTS by considering an abstraction over entire paths in the tree - two paths of equal length, that start from the same node in the tree, can be aggregated into a single abstract state, thus reducing the search space. The paper proposes to use a probabilistic approach to the abstraction process, using the justification that this enables the algorithm to recover from aggregation errors that it commits early on. The specific probabilistic approach discussed relies on a divergence measure between the distribution of the value functions across the entire two paths, thus merging together with high probability actions that lead to similar distributions of the value function. This abstraction helps mitigate the worst weakness of MCTS - it reduces the large search space. Some theoretical guarantees are provided, as well as several experimental results for different game problems and for control tasks.\n\nSTRENGTHS:\nThe paper deals with the very important challenge of improving MCTS techniques. This type of research is applicable in many domains, as this is a well known and well used algorithm. \n\nThe experimental results looks extensive and well-presented, and are the main strength of the paper. I especially liked the comparison of different state abstraction functions, as it showcases the contribution of the paper in coming up with a specific one that seems to work well. Adding a successful novel approach to a well established algorithm is not a trivial task, and experimental results seem very promising. This seems like a strong enough reason to publish on its own.\n\nWEAKNESSES:\nI thought the main weakness of the paper is its readability. I had a tough time understanding the approach and the logic behind it, even though I have some experience with MCTS specifically (though admittedly, it had been awhile). Some more careful attention can be given to notation and explanations. The math in this paper requires close scrutiny and some of the explanations seem to assume a close familiarity with the specifics of MCTS, as well as state abstraction functions. This results in a reduced reading experience and lower comprehension. \nSome examples: \n1. In equation (1) Q is never explicitly defined, figure 1 appears much earlier in the paper than the definition of the probability state abstraction \n2. The complex distinction between paths, states and nodes is not explicitly stated, and sometimes ignored - table 1 is referenced during the general RL discussion that has a state notation (s1, s2) but uses a different notation, that is later used for nodes (v1, v2). \n3. Some of the notation within Algorithm 1 is never explained (e.g., actions like pruning/delete/add and the usage of a hidden state h). \n4. Q* usage is never explained \n5. In the explanation after definition 4.3 - encourages nodes that have the same candidate actions with similar value distribution expectations to be aggregated - should that be be encourages paths ? The entire definition seems to be about aggregating paths instead of specific states, but paths that start from the same node. \n\nIt is fine to delegate some details to referenced works, but a paper should at least succinctly explain its own notations and be as self-contained as possible. I trust these weaknesses in explanations and paper organization can be fixed by the authors.\n\nQUESTIONS:\n1. Are you planning to publish the code you used? \n2. Please check your math for some typos - eq. 19 in appendix A.\n\nLIMITATIONS:\nSome limitations are briefly addressed, namely the need for hyper-parameter tuning and manually selecting the underlying abstraction function. I believe another limitation may lie in the added computational complexity of this method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}