Left Heavy Tails and the Effectiveness of the Policy and Value Networks in DNN-based best-first search for Sokoban Planning

Abstract:  Despite the success of practical solvers in various NP-complete domains such as SAT and CSP as well as using deep reinforcement learning to tackle two-player games such as Go, certain classes of PSPACE-hard planning problems have remained out of reach. Even carefully designed domain-specialized solvers can fail quickly due to the exponential search space on hard instances. Recent works that combine traditional search methods, such as best-first search and Monte Carlo tree search, with Deep Neural Networks' (DNN) heuristics have shown promising progress and can solve a significant number of hard planning instances beyond specialized solvers. To better understand why these approaches work, we studied the interplay of the policy and value networks of DNN-based best-first search on Sokoban and show the surprising effectiveness of the policy network, further enhanced by the value network, as a guiding heuristic for the search. To further understand the phenomena, we studied the cost distribution of the search algorithms and found that Sokoban instances can have heavy-tailed runtime distributions, with tails both on the left and right-hand sides. In particular, for the first time, we show the existence of \textit{left heavy tails} and propose an abstract tree model that can empirically explain the appearance of these tails. The experiments show the critical role of the policy network as a powerful heuristic guiding the search, which can lead to left heavy tails with polynomial scaling by avoiding exploring exponentially sized subtrees. Our results also demonstrate the importance of random restarts, as are widely used in traditional combinatorial solvers, for DNN-based search methods to avoid left and right heavy tails.

Reviews and author responses:

 > In this paper, do you consider planning problems as finding a satisficing plan?

In this paper, yes. But finding shorter plans instead of an arbitrary one is a way more important research question. Not only because shorter plans are combinatorially harder to find, but also training on shorter plans can exploit sharper combinatorial structure and provides better generalization power. These will be further addressed in our following papers.

> How did you obtain h(n) in a policy-guided heuristic search? I couldn't find the details other than the value will be the predicted outcome from the value head of the NN. Are they admissible heuristics?

Please check line 163-165. For each found plan, we use the remaining distance to the goal as the label for h(n). This heuristic is not admissible, which means optimality is not guaranteed by deep-net based best-first search. However, the predicted heuristic is still better than manually crafted admissible heuristic since the latter one is usually too coarse inaccurate.

> Could you explain more about Figure 1? There are heavy/exponential tail cases. Does each correspond to 1 Sokoban instance or group of instances? From the caption, it sounds like there were 4 Sokoban instances. What is the right tail survival function/left tail cumulative distribution function? I think the message is that random restarting helps if it appears on the left side of the mean. What is the right way to read those plots?

They are 4 instances showing different shapes of tails. One quick way to understand the figure is to focus on "mean" line (red) and "expected sample total runtime l^S" line (yellow). The red line basically means a naive run of the algorithm while yellow line means using restarts. For heavy tails, we can observe yellow lines are orders of magnitude shorter than the red lines, while for exponential tails they are very close. This means we can only achieve acceleration with restarts for instances with heavy tails.

> Figure 2 mentions areas with colors, but I only see red dots and no cyan areas.

It is a visual bug if you directly read the paper online inside OpenReview... If you download the paper and use your local pdf reader software then you can see the cyan areas.

> Since this is very empirical work for discovering left-heavy tail, could you elaborate more on how experiments were done, including trials, running times, and NN training in the Appendix? Probably, it is a huge amount of work to "reproduce" the result. I think it is important to know what amount of experiment effort lead to this finding.

Thanks for the advice. We will add more training details to the appendix in our revision.

> I found this paper more interesting after rereading it. One minor issue is the clarity of the paper. Besides, the sentence in Line 214-217 doesn't read as authors explained, there are some less common word choices. In Table 3, "Solving ratio" wouldn't heart clarity too much, but we often say "coverage" of search methods. In Table 2, a common choice would be "search statistics" instead of "solver statistics." Could you check if -log(p_{ls}) is cross entropy?

We thank the reviewer so much for these detailed and constructive comments! We will carefully rewrite the parts you mentioned in our revision. The cross entropy term is correct. We omitted the softmax part (e^{x_k} / \sum_i e^{x_i}) in the paper.

 Thank you very much for your clarifications!
Here are a few more clarification questions.

In this paper, do you consider planning problems as finding a satisficing plan? 

How did you obtain h(n) in a policy-guided heuristic search?
I couldn't find the details other than the value will be the predicted outcome from the value head of the NN. 
Are they admissible heuristics? 

Could you explain more about Figure 1?
There are heavy/exponential tail cases.
Does each correspond to 1 Sokoban instance or group of instances?
From the caption, it sounds like there were 4 Sokoban instances.
What is the right tail survival function/left tail cumulative distribution function?
I think the message is that random restarting helps if it appears on the left side of the mean. 
What is the right way to read those plots?

Figure 2 mentions areas with colors, but I only see red dots and no cyan areas.

Since this is very empirical work for discovering left-heavy tail, 
could you elaborate more on how experiments were done, including trials, running times, and NN training in the Appendix? 
Probably, it is a huge amount of work to "reproduce" the result. 
I think it is important to know what amount of experiment effort lead to this finding.

I found this paper more interesting after rereading it. One minor issue is the clarity of the paper. 
Besides, the sentence in Line 214-217 doesn't read as authors explained, 
there are some less common word choices.
In Table 3, "Solving ratio" wouldn't heart clarity too much, but we often say "coverage" of search methods.
In Table 2, a common choice would be "search statistics" instead of "solver statistics."
Could you check if -log(p_{ls}) is cross entropy?

I think this paper is a consequence of a huge amount of work, so spending a little bit more time to improve those minor issues may improve the quality of this paper.

 > One very minor comment is that the acronym of DBFS has been already used in the following paper well known in the planning community. Therefore, I would recommend them to use a different acronym if possible. https://ojs.aaai.org/index.php/AAAI/article/view/8004. As a side node, this DBFS (Diverse Best-First Search) uses randomization to avoid the plateau of search due to heuristic values. It uses neither the notions of the left-heavy tails nor critical nodes. But perhaps Diverse Best-First Search might have some relations (e.g., why does avoiding the plateau leads to an improvement?) to these notions investigated by the authors, which might generate some important future research.

We thank the reviewer so much for these valuable suggestions. We will pick another acronym in our revision. Diverse Best-First Search manually creates randomness while we merge randomization into the learning procedure, which we think might have more generalization power. For example, for those easy board inputs the network will learn small randomization while the randomization will be larger for those out-of-distribution and harder boards.

 > How is the randomization introduced by dropout useful? (I'm guessing it allows exploring a different part of the search space after a restart. Is this correct?) How important is dropout to the overall approach? Are there any other randomization schemes that can be considered besides dropout? How might they perform?

Figure (b) in Page 9 provides evidence showing a single deterministic run (blue line) will be gradually out-scaled by multiple random runs as the search budget increases. There are other randomization works, e.g., Baysian networks, which, however, will greatly change the architecture of the original deterministic network structure. Dropout only adds minor change to the architecture and will serve as a fair baseline for the performance comparison between deterministic/non-deterministic search.

> Besides Sokosolution and knowledge-guided BFS, are there any other known strong Sokoban solvers? (If not, this could be made clearer.)

Table 1 in Pages 4 shows some board statistics about previous works. Previous deep learning based works are studying small scaled Sokoban instances (number of boxes <= 4), while our work studies way harder instances which contain 20 boxes in average. You need harder instances to make statistic results on a combinatorial domain shaper and significant.

> (Minor). Figure 5 (a,b) are missing their labels.

Thanks for pointing them out. Will be fixed in the revision.

 > The main weakness is the limited scope of work: (1) The paper focuses on a single planning domain. Sokoban is a well-known planning domain but as the paper seems to suggest left heavy tails are likely to occur in other practical planning problems (right heavy tails have been observed in many problems), experiments on other domains would significantly strengthen the work. (2) Unlike the results on the effectiveness of DNN-based best-first search, the results on the left heavy tails are not shown for multiple evaluation functions and it they appear across different evaluation functions.

We have been careful in stating our findings and the scope. Also notice that right-hand heavy tails were first observed and reported for SAT on a single combinatorial task, quasigroup completion (underconstrained instances), due to limited computational power. Later they were confirmed in many other domains. Unlike right heavy tails (majority polynomial runtime and rare exponential runtime), left heavy tails (majority exponential time and rare polynomial time) require extremely extensive computational budget to verify, even for a single experiment setting.

> In addition, I think some of the claims require further justification: (1) I think the claim that left heavy tails are shown for the first time needs to be more clearly justified in the paper as there are existing works that discuss left heavy tails (e.g., reference [18] in the paper). (2) I think the claim that the paper shows the importance of using uncertainty-aware networks in planning should be more carefully stated due to the narrow scope described above as well as the fact that experiments report results on only one dropout value (30%).

Left-hand heavy tails have been conjectured to exist for SAT but not observed (to our knowledge). The dropout value was found by a grid search and was shown to be effective consistently. As the result, we picked this specific value. Figure (b) in Page 9 uses the number of restarts to show the importance of using uncertainty-aware networks. Also, due to the large test set (we have tested on ~10000 instances), a single experiment lasts for weeks which limits the number of parameter settings used in this paper.

> Minor points: (1) I think the citation style in the paper ("[10] used ...", "[34] proposed ...") is hard to read and it would be better to use the author names. (2) The supplementary material has some missing references presented as "?" such as "Proof of ??", "By the result of (author?)"

Thank you so much for the careful advices. We will fix them in the revision.

 > In policy guided search, how did you compute p(n | parent(n) ) ?

Given any state s, the policy head predicts a probability distribution among outgoing actions p(a | s). So p(n | parent(n)) = p(action leading to n | parent(n)).

> In network architecture and training details, can you explain each term in the loss function?

$loss = (\log(h) - \log(v_s))^2 - \log(p_{l_s}) + c|| \theta ||^2$, where the first term is mean squared loss for log scaled remaining distance, the second term is cross entropy loss for next action prediction, and the last term corresponds to l_2 weight normalization.

> On the contribution of this paper on the "uncertainty aware network," Is there some part in the paper other than Monte Carlo dropout (line 185-188) explaining the details? The conclusion is the drop-out ratio of 30% is the best after the grid search. If we choose other ratios, would it change the analysis on left-heavy tails? I cannot find how this selection effectively introduced randomization to search.

Picking the best ratio is actually extremely computational intensive since a single run for single ratio lasts for weeks. Figure (b) in Page 9 does provide results w/wo the uncertainty component. As shown in the figure, without randomization, the blue curve scales poorly as the total computational budget increases. 

> In the effectiveness of policy and value networks, I was not able to understand the last sentence (lines 214-217). Can you explain how to detect dead-end children? what does it mean by "predict a higher/lower policy/value to the child"?

Use the policy as the example. If there is a dead-end child node, then the predicted action probability is supposed to be lower than all other non-dead-end children, otherwise it is considered to fail detecting dead-end.

> In sample statistics about the heaviness of tails, how t^{\mathcal{S}} (lin 227) was given as an approximation of the optimal sample restart time?

Run the algorithm multiple times to collect a runtime sample S, then approximate the real runtime distribution by the discrete distribution on S. And then you have both equations.

> In the abstract search tree model, is n the length of the plan for Sokoban problems or a total number of states? What is the implication of the number of critical nodes being O( log(n) )? Are the critical nodes must be nodes along the path from the root to the goal?

n is the length of the plan. Informally, a critical node means network prediction is inaccurate on this node and will incur exponential search space. O(log(n)) critical nodes provide a non-negligible probability for polytime search. Yes. all critical nodes are along the path on the solution path.

> ...... How to apply this idea to designing a random restarting strategy? Is it mandatory to have strong heuristics such as policy networks? Is this implies dead-end detection is more important than estimating the cost to go more precisely?

This is a fantastic question which also bothered us for a long time. This is an analyzing paper which is done months ago. Our current intuition is that the way we train the model and the way we utilize the prediction (e.g., variations of A*) during the search are separated. A fully differential framework in which the training procedure is aware about the search would be a better strategy. Please check another work "MCTSnets" for an example. For restarting, multiple equal-length runs are proved to be robust and effective in the literature.

 This paper analyzes the behavior of DNN-based best-first search for solving Sokoban planning problems and discovers "left heavy tails" on the running time distribution from hard Sokoban instances. The main contribution of this paper is to show the existence of left heavy tails and the abstract search tree model that characterizes such behavior. There are also relatively minor contributions, such as modifying the f(n) and systematic comparison of various search configurations.
 The strength of this paper is through experiments on a very large number of Sokoban instances (over 10,000) and evaluations of various search configurations, which provides some evidence and explanations why search with DNN-based heuristic works well in practice.

The weakness of this paper is missing technical details.
 In policy guided search, how did you compute p(n | parent(n) ) ?

In network architecture and training details, can you explain each term in the loss function? 

On the contribution of this paper on the "uncertainty aware network," Is there some part in the paper other than Monte Carlo dropout (line 185-188) explaining the details? The conclusion is the drop-out ratio of 30% is the best after the grid search. If we choose other ratios, would it change the analysis on left-heavy tails? I cannot find how this selection effectively introduced randomization to search.

In the effectiveness of policy and value networks, I was not able to understand the last sentence (lines 214-217). Can you explain how to detect dead-end children? what does it mean by "predict a higher/lower policy/value to the child"?

In sample statistics about the heaviness of tails, how t^{\mathcal{S}} (lin 227) was given as an approximation of the optimal sample restart time?

In the abstract search tree model, is n the length of the plan for Sokoban problems or a total number of states?
What is the implication of the number of critical nodes being O( log(n) )? Are the critical nodes must be nodes along the path from the root to the goal?

The proof in the Appendix, as well as the abstract search tree model, assumes the number of critical nodes is in O( log(n) ) and the non-critical node has a search space of size O( poly(n) ). From this assumption, the optimal expected halting time with restart is shown to be O( poly(n) ).
Figure 1 also shows that the l^{\mathcal{S}} (yellow) is much smaller than the mean running time (red line). Figure 5 (caption is missing) shows the abstract search tree model matches the data. How to apply this idea to designing a random restarting strategy? Is it mandatory to have strong heuristics such as policy networks? Is this implies dead-end detection is more important than estimating the cost to go more precisely?
 I think this work is not relevant to this section.

 The paper investigates DNN-based best-first search with policy and value networks for Sokoban planning. First, the paper studies PHS and PHS* and a variant proposed in this paper and find that policy networks learn effective heuristics for search and can be further enhanced by the use of value networks. Then, the paper empirically studies left heavy tails from experiment data and develop an abstract model that show how left heavy tails can arise during the search.


----
I thank the authors for their response.

I think the paper would benefit from showing this phenomenon extends beyond a single domain and a single heuristic function, however I understand the limitation on computational resources and I think these results are of interest. I therefore increase my score by one point.

I think the paper should include a discussion on the previous works discussing left heavy tails (including conjectures). Strengths:
- Interesting empirical results on effectiveness of DNN-based best-first search in Sokoban planning.
- The discovery of left heavy tails in DNN-based best-first search in Sokoban planning.
- A novel theoretical results that explains the occurrence of left heavy tails.


Weaknesses:
- The main weakness is the limited scope of work: 
	* The paper focuses on a single planning domain. Sokoban is a well-known planning domain but as the paper seems to suggest left heavy tails are likely to occur in other practical planning problems (right heavy tails have been observed in many problems), experiments on other domains would significantly strengthen the work.
	* Unlike the results on the effectiveness of DNN-based best-first search, the results on the left heavy tails are not shown for multiple evaluation functions and it they appear across different evaluation functions.
- In addition, I think some of the claims require further justification:
	* I think the claim that left heavy tails are shown for the first time needs to be more clearly justified in the paper as there are existing works that discuss left heavy tails (e.g., reference [18] in the paper).
	* I think the claim that the paper shows the importance of using uncertainty-aware networks in planning should be more carefully stated due to the narrow scope described above as well as the fact that experiments report results on only one dropout value (30%).
- Minor points:
	* I think the citation style in the paper ("[10] used ...", "[34] proposed ...") is hard to read and it would be better to use the author names.
	* The supplementary material has some missing references presented as "?" such as "Proof of ??", "By the result of (author?)"
 - Did you experiment with other domains?
- Did you find left heavy tails across different evaluation functions?
- Did you experiment with different dropout values? I did not identify any issues related to limitations and potential negative societal impact.

 The paper analyzes the performance of learned heuristics in best-first search for planning in Sokoban. The paper proposes a variant of the policy-guided best-first search for its experiments. The algorithm requires a node evaluation function and this paper proposes a simplified variant of the evaluation function in PHS* which requires, both, a value (distance to goal) prediction and an action prediction (P(n | parent(n))). The experimental setup uses a dataset of Sokoban instances designed by humans. Supervised training data for learning the distance to goal heuristic is generated by running an off-the-shelf Sokoban planner on these instances. A DNN with dropout (for randomization) (and policy and value heads) is used to learn the heuristic. The experiments compare different evaluation functions (with and without learned policies) and contain a number of interesting results. Using a learned policy significantly improves performance on hard Sokoban instances, outperforming the hand-crafted heuristic. Next, the use of action probability in the evaluation function improves performance over just using the learned values (distance to goal). Finally, the paper analyzes the runtimes for unsolved instances and discovers the existence of heavy tails on the left (mostly-long runs, few short runs). The paper develops some formalisms to characterize this behavior. The analysis and experiments suggest that a strategy involving more frequent restarts can improve overall performance in such cases.

---------

UPDATE: I thank the authors for their responses. After reading the other reviews, I'm more positively inclined towards the paper and have updated my scores to reflect this. Strengths
  + The paper investigates into the performance of best first search with learned knowledge (DBFS) on challenging planning problems. This is an important and potentially impactful analysis of a popular learning to search method on a challenging problem.
  + The paper includes a small algorithmic contribution (simplified eval function), a number of empirical insights into the performance of DBFS in Sokoban and an analysis of the phenomena of left heavy tails to better understand the unsolved problem instances.
  + The DBFS algorithm can solve a significant fraction of hard problem instances, which the state of the art solver fails on. This seems like a significant performance improvement on hard Sokoban instances. (Right?)
  + The paper investigates the hardest Sokoban instances and discovers the existence of left heavy tails. This seems like a novel insight, as far as I can tell.

Weaknesses
  - One of the main claims of the paper is that the randomization injected by dropout is important. However, I was unable to find any supporting evidence for the claim other than a single sentence ("We did a grid search ... and found 30% reached the best performance") The size of the performance gap with and without dropout is unclear to me. What have I missed?
  - The paper is reasonably well written but I thought it didn't quite flow on my first reading. For example, the concepts of optimal speedup of Las Vegas algorithms and right heavy tails are briefly introduced early in the paper (Section 2) but only used to analyze the unsolved problem instances much later in Section 5. Introducing these concepts in Section 5 might improve the flow.
  - The empirical contributions of DBFS could be made clearer. For example, is there any other solver which does better than DBFS on the hard dataset task in Table 2?   1. How is the randomization introduced by dropout useful? (I'm guessing it allows exploring a different part of the search space after a restart. Is this correct?) How important is dropout to the overall approach? Are there any other randomization schemes that can be considered besides dropout? How might they perform?

  2. Besides Sokosolution and knowledge-guided BFS, are there any other known strong Sokoban solvers? (If not, this could be made clearer.)

  3. (Minor). Figure 5 (a,b) are missing their labels. The authors have not discussed the limitations or social impact of the work. Since this work explores general search algorithms, I don't see any major negative social impact.


 The authors attempt to understand the behavior of search in the planning domain when it is combined with a DNN-based heuristic function, namely in Sokoban. Their contributions include revealing the existence of left and right heavy tails, proposing an abstract tree model which explains this phenomenon and providing theoretical analysis. Their experiments show that policy networks can lead to avoiding exponentially large trees, which is an explanation to the efficiency of the DNN-based heuristic function. They finally show that random restarting plays an important role to avoid both tails. 
 The heavy-tailed distribution is a well-known phenomenon observed in many combinatorial search domains including SAT and CSP. This paper has a very interesting observation which does not occur in those domains but which newly occurs in the planning domains.

One very minor comment is that the acronym of DBFS has been already used in the following paper well known in the planning community. Therefore, I would recommend them to use a different acronym if possible. 

https://ojs.aaai.org/index.php/AAAI/article/view/8004

As a side node, this DBFS (Diverse Best-First Search) uses randomization to avoid the plateau of search due to heuristic values. It uses neither the notions of the left-heavy tails nor critical nodes. But perhaps Diverse Best-First Search might have some relations (e.g., why does avoiding the plateau leads to an improvement?) to these notions investigated by the authors, which might generate some important future research.

In summary, this is a very interesting paper and I enjoyed reading it.  I do not have any specific question.
 N/A


Meta-review: The reviewers found this empirical paper's observations and explanations regarding search guided by a DNN-based heuristic insightful and definitely worth publishing, despite the fact that the study was done on only one benchmark, Sokoban.

Decision: Accept