SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Acknowledgement. We thank anonymous reviewers for their valuable comments. We gratefully acknowledge support from ONR MURI grant N00014-16-1-2007; from the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF-1231216); from NSF grant 2214177; from Air Force Office of Scientific Research (AFOSR) grant FA9550-22-1-0249; from ONR MURI grant N00014-22-1-2740; from ARO grant W911NF-23-1-0034; from the MIT-IBM Watson AI Lab; from the MIT Quest for Intelligence; and from the Boston Dynamics Artificial Intelligence Institute. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors.
Ron Alford, Vikas Shivashankar, Mark Roberts, Jeremy Frank, and David W Aha. Hierarchical Planning: Relating Task and Goal Decomposition with Task Sharing. In *IJCAI*, 2016. 10
Waiss Azizian and Marc Lelarge. Expressive Power of Invariant and Equivariant Graph Neural Networks. In ICLR, 2021. 10
Pablo Barceló, Egor V Kostylev, Mikael Monet, Jorge Pérez, Juan Reutter, and Juan Pablo Silva. The logical expressiveness of graph neural networks. In ICLR, 2020. 6, 10
Anthony Barrett and Daniel S Weld. Characterizing Subgoal Interactions for Planning. In *IJCAI*, 1993. 10
Davide Buffelli, Pietro Lio, and Fabio Vandin. SizeShiftReg: a Regularization Method for Improving Size-Generalization in Graph Neural Networks. In NeurIPS, 2022. 10
Tom Bylander. The Computational Complexity of Propositional STRIPS Planning. Artif. Intell., 69(1-2): 165-204, 1994. 1, 10
Jin-Yi Cai, Martin Fürer, and Neil Immerman. An Optimal Lower Bound on the Number of Variables for Graph Identification. Combinatorica, 12(4):389-410, 1992. 6
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning. arXiv:2302.02662, 2023.1
Hubie Chen and Omer Giménez. Act Local, Think Global: Width Notions for Tractable Planning. In ICAPS, 2007. 1, 4, 5, 10, 15
Joseph Culberson. Sokoban is PSPACE-complete. Technical report, University of Alberta, 1997. 2, 10
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural Logic Machines. In ICLR, 2019. 1, 9, 10, 21
Dominik Drexler, Jendrik Seipp, and Hector Geffner. Learning Sketches for Decomposing Planning Problems into Subproblems of Bounded Width. In ICAPS, 2022. 10
Kutluhan Erol, James Hendler, and Dana S Nau. HTN planning: Complexity and Expressivity. In AAAI, 1994. 10
Richard E Fikes and Nils J Nilsson. STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving. Artif. Intell., 2(3-4):189-208, 1971. 2
Eugene C Freuder. A Sufficient Condition for Backtrack-Free Search. JACM, 29(1):24-32, 1982. 15
Naresh Gupta and Dana S Nau. On the Complexity of Blocks-World Planning. Artif. Intell., 56(2-3):223-254, 1992. 10
Zhengyao Jiang and Shan Luo. Neural Logic Reinforcement Learning. In ICML, 2019. 9
Richard M Karp. Reducibility among Combinatorial Problems. Springer, 1972. 10
Richard E Korf. Planning as Search: A Quantitative Approach. Artif. Intell., 33(1):65-88, 1987. 10
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards Practical Multi-Object Manipulation Using Relational Reinforcement Learning. In ICRA, 2020. 1, 9, 10
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control. arXiv preprint arXiv:2209.07753, 2022. 1
Vladimir Lifschitz. On the Semantics of STRIPS. In Reasoning about Actions and Plans. Morgan Kaufmann, 1987. 2
Nir Lipovetzky and Hector Geffner. Width and Localization of Classical Planning Problems. In ECAI 2012. IOS Press, 2012. 5, 10, 15
Zhezheng Luo, Jiayuan Mao, Joshua B Tenenbaum, and Leslie Pack Kaelbling. On the Expressiveness and Generalization of Hypergraph Neural Networks. In LoG, 2022. 6, 10
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks. In AAAI, 2019. 5, 6
Earl D Sacerdoti. Planning in a Hierarchy of Abstraction Spaces. Artif. Intell., 5(2):115-135, 1974. 10
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In NeurIPS, 2017. 1, 5
William Vega-Brown and Nicholas Roy. Task and Motion Planning is PSPACE-Complete. In AAAI, 2020. 2
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning Structured Policy with Graph Neural Networks. In ICLR, 2018. 1
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In ICLR, 2019. 10
Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. What can Neural Networks Reason About? In ICLR, 2020. 10
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks. In ICLR, 2021. 10
The appendix of the paper is organized as the following. In Appendix A, we provide the proof details and more discussion with related work on search complexity. In Appendix B, we provide the proof details and more discussion on policy circuit complexity. Next, in Appendix C, we include a detailed analysis of regression widths and circuit complexities for empirical planning problems. Finally, in Appendix D, we describe environmental setups and implementation details for the experiments.
A Proofs and Discussions for Search Complexity
A.1 Complete and Optimal Version of S-GRS
Algorithm 3 shows a complete and optimal version of the serialized goal regression search algorithm. The key difference between this variant and the simplified one presented in the paper is that now we keep track of multiple possible trajectories that can achieve the specified goal. The algorithm is complete because for any set of precondition atoms, in the optimal trajectory, there will always be an order in which they have been achieved (note that some of the sub-trajectories can be empty, indicating that while planning for the prefix the next subgoal has already been achieved; this covers the "parallel" precondition achievement case.). We show an example of this branching factor in Figure 4. Note that, to make sure that the algorithm eventually terminates, we need to make sure that all trajectories do not have loops.
However, this bare algorithm can be very slow: the worst time complexity of it is  $O(3^{N})$  with respect to  $N$ , the number of atoms in  $\mathcal{P}_0$  — in contrast to  $O(2^{N})$  for a simpler backward search algorithm. This is because the possible state set for different subgoals and different constraints may overlap, so it will "waste" time during the search. Therefore, this algorithm may not be of practical use.
Algorithm 3 Serialized goal regression search with multiple optimal path tracking.
function GRSOPT(s0, g, cons)  
if  $g \in s_0$  then return (s0)  
all POSSIBLE_t = empty_set()  
for  $r \in \mathcal{R}_0$ : cons  $\cap$  eff_(action(r)) =  $\emptyset$ $\land$  goal(r) = g do  
 $p_1, p_2, \dots, p_k = \text{subgoals}(r)$   
if  $\exists p_i$  s.t.  $p_i \in$  goal stack then  
continue  
possible_t = \{\emptyset\}  
for  $i$  in  $1, 2, \dots, k$  do  
next POSSIBLE_t = empty_set()  
for prefix_t in possible_t do  
imm_state = last_state(prefix_t)  
for  $nt \in \text{GRSOPT}(imm\_state, p_i, cons \cup \{p_1, \dots, p_{i-1}\})$  do  
next POSSIBLE_t = next POSSIBLE_t  $\cup$  {prefix_t + nt}  
possible_t = next POSSIBLE_t  
all POSSIBLE_t = all POSSIBLE_t  $\cup$ $\{t + T(last\_state(t), action(r))| t \in possible_t\}$   
return all POSSIBLE_t
A.2 Proof for Theorem 3.1: S-GRS Optimality and Completeness
We define the set of optimally serializable rules for a state  $s$  as  $OSR(s)$ , and the set of single-literal goals that can be solved with  $OSR(\cdot)$  from  $s_0$  as  $OSG(s_0)$ .
Proof. Completeness: the definition of  $OSG(s_0)$  asserts that  $g$  can be solved by S-GRS.
Optimality: proof by induction. In the following, we prove that for any  $g \in OSG(s_0)$ , the trajectory returned by S-GRS is optimal.
Note that constraints is always a subset of state state, so the worst complexity is  $O(3^{N})$ .
Figure 4: Illustration of the branching factor caused by tracking multiple resulting states after achieving a subgoal.
First, we consider the set of all S-GRS calls in the regression tree that contributes to the returned trajectory. We do induction based on a distance metric. Consider any goal atoms  $g \in OSG(s_0)$ , let  $Distance(s_0, g, cons)$  be the length of the path returned by S-GRS. First, for all argument tuples with distance 0 (i.e.,  $g \in s_0$  and  $cons \subseteq s_0$ ), S-GRS will return the optimal trajectory.
Next, assuming that S-GRS will return optimal trajectories for all argument tuples with distances strictly smaller than  $d$ . Consider a new argument call  $grs(s_0, goal, cons)$  with distance  $d$ . Consider now all optimally serializable and goal-achieving rules  $r \in OSR(s_0)$  such that  $goal(r) = g$  and  $cons \cap eff_{-}(action(r)) = \emptyset$ . Let  $r_g$  be the rule that returns the plan during search, and  $p_1 \cdots p_k$  be the serialized precondition list. First, all sub-function calls to S-GRS will return optimal plans by induction. Second, by optimal serializability of the rule  $r$ , we know that concatenating the sub-plans will yield an optimal plan to achieve  $g$  while preserving  $g$ . Therefore, our induction holds.
A.3 Making Non-SOS Problem Serializable
Here we discuss a general strategy to make non-SOS problems serializable by introducing super predicates. In particular, for a given operator, if two of the preconditions (without loss of generality, say,  $p(x)$  and  $q(x)$ ) are not serializable, we can introduce a new predicate  $p$  and  $q(x)$ , and rewrite the precondition of the operator with this new super predicate.
The disadvantage of this encoding is that it will enlarge not only the predicate set but also the operator set. In particular, for any operator  $o$  such that  $p(x) \in \text{eff}_{+}(o)$  or  $p(x) \in \text{eff}_{-}(o)$ , we need to break the operator into two sub-operators. For example, if  $p(x) \in \text{eff}_{+}(o)$ , we need to create a new operator  $o_1$  that has  $\neg q(x)$  in its precondition, and set  $\text{eff}_{+}(o_1) = \text{eff}_{+}(o)$  and  $\text{eff}_{-}(o_1) = \text{eff}_{+}(o)$ . For the second operator  $o_2$ , it should have  $q(x)$  in its precondition and  $\text{eff}_{+}(o_1) = \text{eff}_{+}(o) \cup \{p_{-} \text{and}_-q(x)\}$  and  $\text{eff}_{-}(o_1) = \text{eff}_{-}(o)$ . As the number of predicates in this super predicate increases, the number of additional operators grows exponentially.
This provides a characterization of "what contributes to the (forward) width of a problem." Specifically, we have identified two sources of forward widths: non-serializability of certain operators, and the need to track constraints while doing a regression search.
A.4 Proof for Theorem 3.2: Polynomial Hardness for SOS-k problems.
If a problem  $P$  is of SOS width  $k$ , it can be solved in time  $O(N^{k + 1})$ , where  $N$  is the number of atoms. Here we have omitted polynomials related to enumerating all possible actions and their serializations (which are polynomial w.r.t. the number of objects in  $\mathcal{U}$ ).
Proof. First, if we are given the rule set  $\mathcal{R}$ , then the proof can be simply done by analyzing function calls: it is important to observe that there are at most  $O(N^{k + 1})$  possible argument combinations of (goal, cons) to function grs.
First, due to SOS condition, we only need to maintain one single optimal trajectory for each set of argument values  $(s_0, goal, cons)$ . Next, we consider we only maintain one single optimal trajectory for each pair of (goal, cons), ignoring the first argument  $s_0$ . Note that, even if the "cached" trajectory does not include the input state  $s_0$  we can still return the cached trajectory. This is because the optimal serializability suggests that any trajectories for achieving (goal, cons) can be extended to future preconditions.
Finally, we consider removing the assumption of knowing  $\mathcal{R}$ . Note that the enumeration of all possible goal width- $k$  regression rules can be done in  $O(2^k)$ , which is considered constant when  $k$  is small.
Therefore, overall the algorithm runs in polynomial time and the solver does not need to know the set of optimally serializable regression rules  $\mathcal{R}$  a priori.
First, the fact that the algorithm does not need to know  $\mathcal{R}$  a priori is important because it suggests that, as long as there exists a set of optimally serializable rules for the problem we are trying to solve, the search algorithm will find a solution, without having any additional knowledge other than the operator definitions. Intuitively, this is because doing S-GRS with additional non-optimally serializable rules (e.g., using all rules in  $\mathcal{R}_0$ ) will not falsify the completeness and optimality of the algorithm.
Second, unfortunately, the SOS condition can not be relaxed. That is, if we only assume optimal serializability of generalized regression rules and all rules have width  $k$ , the proof will fail. This is because even if regression rules have a small width, there will be a possibly exponential branching factor caused by "free variables" in regression rules that do not appear in the goal atom. Concretely, let us consider the following rule  $g(x) \Leftrightarrow p_1(x,y)^\emptyset, p_2(x)^\emptyset \parallel a_1(x,y)$ . This rule is of width 0; however, since there is no guarantee that any optimal trajectory for achieving  $p_2(x)$  will also achieve  $p_1(x,y) \wedge p_2(x)$ , during search, we can not use the cached trajectory for  $(p_2(x),\emptyset)$  as the return value when we solve for  $p_2(x)$  after achieving  $p_1(x,y)$ , which breaks the complexity analysis.
A.5 Additional Discussion
Deriving optimally-serializable, small-width rules. For a practical system, if we can know the set of regression rules needed, we can accelerate the search significantly. Here we discuss a strategy for deriving such rules by analyzing the domain definition itself.
If we ignore the optimality, a sufficient condition that a rule  $r$  is not strong optimal serializability for a given state  $s_0$  is that, there exists a precondition  $p_i$  such that there is no rule  $r'$  such that  $goal(r') = p_i$ ,  $eff_{-}(action(r')) \cap p_i = \emptyset$  and  $goal(r) \notin subgoals(r')$ . The last condition is the goal-stack checking: if achieving a goal  $g$  using rule  $r$  requires achieving goal  $g$  itself, then the rule is not serializable.
Putting this into practice, consider all rules for achieving clear(A). If there is another object B on A in  $s_0$ , then only the rule derived from unstack(B,A) will be SOS serializable. This is because, for any other rules unstack(y,A), where  $y \neq B$ , achieving on(y,A) will involve achieving clear(A) itself. Note that this analysis can be done by static (lifted) analysis of the domain.
This analysis can be viewed as simulating a one-step goal regression search in a lifted way, and it can be extended to more steps. In some domains, this strategy will help rule out some serialization of the preconditions (such as the BlocksWorld example), but in general, whether a rule is strong optimally serializable is state-dependent and can be hard to compute.
Connection with other concepts in search. Another view of the regression width is the maximum number of constraints one needs to track while applying S-GRS recursively. This notion is analogous to the treewidth of a constraint graph in constraint satisfiability problems (CSPs): intuitively, the number of backward-pointing edges [Freuder, 1982].
As also briefly discussed in Lipovetzky and Geffner [2012], delete-relaxation heuristics are also related. In particular, the hFF heuristic can be interpreted as applying backward search by considering "parallelizability" in contrast to "serializability." Informally, for a given state  $s$  and a sequence of preconditions  $p_i$ , we compute trajectories for each individual  $p_i$  separately and concatenate them.
Generalization to  $\forall$ -quantified preconditions. It would be possible to extend the definition to  $\forall$ -quantified preconditions by simply allowing regression rules to have an unbounded number of subgoals (e.g., to handle goals such as  $\forall xon-table(x)$ ). However, this is not helpful in general because the constraints will accumulate as the number of subgoals becomes larger, and enumerating subgoal serialization will now take  $O(m!)$  where  $m$  is the number of subgoals. This notion will be more helpful when we have a "goal regression rule selector."
A.6 Connection with Width-Based Forward Search IW(k)
Wecall the width of a problem defined in Chen and Giménez [2007] and the Iterated Width algorithm  $IW(k)$  [Lipovetzky and Geffner, 2012] as forward width. Then we have:
Theorem A.1 (Sufficiency condition for forward width). Any planning problem that has SOS-width  $k$  has a forward width of at most  $k + 1$  and, hence, can be solved by the algorithm  $IW(k + 1)$ .
Proof. We prove that for any trajectory returned by the S-GRS algorithm for an SOS width- $k$  problem  $P$ , all actions in the trajectory achieve a novel  $k + 1$ -sized atom set. If this is true, then the final goal will be reachable from  $s_0$  in the size  $k + 1$  tuple graph in IW search.
Consider any action  $a$  in the returned plan, it corresponds to a unique rule for achieving a subgoal  $g$  and a set of constraints  $cons$ . If the size  $k + 1$  tuple  $\{g\} \cup cons$  has already been achieved before  $a$ , then due to the SOS condition, we can make the returned plan shorter. This violates the optimality of the S-GRS algorithm.
Unfortunately, these two definitions are not exactly equivalent to each other. For example, consider the Blocks World domain and a state of three stacked blocks A, B and C, the goal regression rule
$$
c l e a r (A) \leftrightarrow o n (B, A), c l e a r (B) ^ {\emptyset}, h a n d s f r e e (\{^ {c l e a r (B)} \} \| u n s t a c k (y, x)
$$
is SOS. It has width 1 because while achieving handsfree(), we have to explicitly maintain clear(B); otherwise, it is possible that we directly put  $C$  back onto  $B$  instead of putting  $C$  onto the table (both plans are optimal for achieving handsfree()) after achieving clear(B). By contrast, in IW, the Blocks World domain has width 1 instead of 2 if the goal predicate is clear, because the optimal plan for achieving handsfree({clear(B)}) happens to be the optimal plan for achieving on-table(C).
Proofs for (forward) widths of planning problems in IW were mostly done by analyzing solution structures. By contrast, the constructions in this paper form a new perspective to interpret planning problem width: they are subgoals and constraints that need to be satisfied during a goal regression search. This view is helpful because our analyses focus on analyzing concrete goal regression rules, and the derived results are compositional. Unfortunately, not all problems that can be solved by  $IW(k)$  have a regression width of  $k - 1$ .
B Proofs and Discussions for Policy Circuit Complexity
B.1 Proof of Theorem 4.1: Compilation of BWD
Given a planning problem  $P$ , let  $T$  be the length of the optimal trajectory as (the planning horizon),  $k_{\mathrm{BWD}}$  be the maximum number of atoms in the goal set in  $\mathsf{BWD}$ , and  $\beta$  be the maximum arity of atoms in the domain. The backward search algorithm  $\mathsf{BWD}$  can be compiled into a relational neural network in RelNN[O(T),  $\beta \cdot k_{\mathrm{BWD}}$ ].
Goal is a set of conjunctive formula.
Figure 5: Illustration of the compilation of backward search into a RelNN policy.
Proof. There is a proof by construction. As illustrated in Fig. 5, a relational neural network can use its intermediate representations to keep track of a set of subgoals (i.e., a set of sets of ground atoms)  $Goal^d$  at each layer  $d$ . Initially,  $Goal^0$  contains only the goal atom. Then,  $sg \in Goal^d$  if there is a path of length  $d$  from any state  $s$  that satisfies  $sg \subseteq s$  to the final goal of the planning problem. For all layers  $d$ , for any subgoals  $sg$ , for all ground actions  $a \in \mathcal{A}$ ,  $sg \in Goal^d \wedge (eff_{-}(a) \cap sg = \emptyset) \Rightarrow (sg \cup pre(a) \setminus eff_{+}(a)) \in Goal^{d + 1}$ . Since the maximum number of atoms in  $BWD$  is  $k_{BWD}$ , we only need to keep track of ground atom conjunctions that involve at most  $k_{BWD} \cdot \beta$  distinct objects. Therefore,  $BWD$  search can be compiled into a relational neural network in  $RelNN[O(T), k_{BWD} \cdot \beta]$ .
A more detailed construction is the following. Let  $k' = k_{\mathrm{BWD}} \cdot \beta$ . Let  $x_1, \dots, x_{k'}$  be variables.  $\{q_i(x_1, \dots, x_{k'})\}$  is the set of all possible conjunctive expressions that involve variables  $\{x_1, \dots, x_{k'}\}$ . Let  $q_i^{(d)}(x_1, \dots, x_{k'})$  be the value of  $q_i(x_1, \dots, x_{k'})$  at layer  $d$ , and  $expr(q_i)$  be the conjunctive expression of  $q_i$ . For all  $d$ ,  $q_i$ ,  $q_j$ , and for all actions  $a$  that are grounded on variables from  $\{x_1, \dots, x_{k'}\}$ , if  $(eff_{-}(a) \cap expr(q_i) = \emptyset) \land (expr(q_j) = expr(q_i) \cup pre(a) \setminus eff_{+}(a))$ , there is a logical rule:  $\forall x_1, \dots, x_{k'}$ .  $q_i^{(d)}(x_1, \dots, x_{k'}) \Rightarrow q_j^{(d + 1)}(x_1, \dots, x_{k'})$ . Furthermore, for all  $d$ ,  $q_i$ , if  $(eff_{-}(a) \cap expr(q_i) = \emptyset) \land (expr(q_i) \cup pre(a) \setminus eff_{+}(a) \subset s)$ , where  $s$  is the current state, we have the following rule that outputs the next action to take,  $\forall x_1, \dots, x_{k'}$ .  $q_i^{(d)}(x_1, \dots, x_{k'}) \Rightarrow a^{(d + 1)}$ , where  $a^{(d)} = 1$  indicates that there exists a length  $d$  trajectory for achieving the goal. In practice, we will the shortest one. Note that all these rules are fully quantified by  $\forall x_1, \dots, x_{k'}$  so they are lifted FOL rules and can be implemented by a RelNN circuit.
B.2 Proof of Theorem 4.2: Compilation of S-GRS
Given a planning problem  $P$  of SOS width  $k$ , let  $T$  be the length of the optimal trajectory, and  $\beta$  be the maximum arity of atoms in the domain. The serialized goal regression search S-GRS can be compiled into a RelNN[ O(T), (k + 1) \cdot \beta ], where  $k + 1 \leq k_{\mathrm{BWD}}$ .
Proof. There is a proof by construction. The easiest construction is based on the sufficient condition between regression width  $k$  and  $IW(k)$ . In particular, based on Theorem 3.3, we know that  $IW(k + 1)$  can solve the planning problem  $P$ . Therefore, we can use the following construction to simulate  $IW(k + 1)$ .
The most important trick in the construction is that a relational neural network can use its intermediate representations to keep track of two sets of atoms. First, for each atom tuple  $t$  up to size  $k + 1$ , whether  $t$  is reachable from  $s_0$  within  $d$  steps. Second, let  $s_t$  be the last state of the optimal trajectory associated with  $t$ ; for each atom tuple  $t$  up to size  $k + 1$  and for a predicate  $p(x_1, x_2, \dots, x_\beta)$ , we
keep track of whether  $p(x_{1},x_{2},\dots ,x_{\beta})$  is true in  $s_t$ . At each layer  $d$ , we consider all possible ground actions  $a\in \mathcal{A}$ , if there exists such  $(t_1,t_2)$  such that  $t_2$  is not reachable within  $d$  steps and  $t\subseteq \mathcal{T}(s_{t_1},a)$ , we set  $t_2$  to be reachable within  $d + 1$  and  $s_{t_2} = \mathcal{T}(s_{t_1},a)$ .