Learning Generalised Policies for Numeric Planning

Venue: icaps-conference_org_ICAPS_2024_Conference
Published: 2023-12-07 15:06:06

Abstract: We extend Action Schema Networks (ASNets) to learn generalised policies for numeric planning, which features quantitative numeric state variables, preconditions and effects. We propose a neural network architecture that can reason about the numeric variables both directly and in context of other variables. We also develop a dynamic exploration algorithm for more efficient training, by better balancing the exploration versus learning tradeoff to account for the greater computational demand of numeric teacher planners. Experimentally, we find that the learned generalised policies are capable of outperforming traditional numeric planners on some domains, and the dynamic exploration algorithm to be on average much faster at learning effective generalised policies than the original ASNets training algorithm.

Main paper: 
 introduction 15: 
 Generalised planning is broadly concerned with the representation, synthesis, and learning of plans, policies, heuristics, and other forms of control knowledge applicable to many problem instances (Srivastava, Immerman, and Zilberstein 2011; Hu and Giacomo 2011; Celorrio, Aguas, and 20 Jonsson 2019). Interest in generalised planning has steadily increased in recent years, fueled in part by advances in machine learning, and by the development of new formalisms to represent and reason about generalised planning tasks and their solutions (Toyer et al. 2018; Francès et al. 2019; Garg, Bajpai, and Mausam 2019; Bonet and Geffner 2020; Aguas, Jiménez, and Jonsson 2020; Toyer et al. 2020; Shen, Trevizan, and Thiébaux 2020; Karia and Srivastava 2021; Aguas, Jiménez, and Jonsson 2021; Ståhlberg, Bonet, and Geffner 2022; Lin et al. 2022 ).An important limit of existing work on generalised planning is that it only allows for primitive forms of quantitative information to be modelled, 1 even though such information is core to many real world problems -for example, modelling a delivery robot requires modelling how much weight it can hold, and modelling flights require reasoning about the the product of distance travelled and fuel consumption per unit of distance. This is despite the existence of the vibrant field of Numeric planning, which extends classical planning formalisms to allow modelling numeric fluents, conditions and effects (Fox and Long 2003) , and typically handles 1 We are only aware of a single exception (Lin et al. 2022) .them using new heuristic search, optimisation, or satisfiability modulo theory based techniques (Hoffmann 2003; Coles et al. 2013; Scala et al. 2016a Scala et al. ,b, 2020;; Kuroiwa et al. 2022; Leofante 2023) .In this paper, we extend a state of the art generalised planning approach, namely Action Schema Networks (ASNets) (Toyer et al. 2018 (Toyer et al. , 2020) ) to handle numeric planning problems described in PDDL2.1 (Fox and Long 2003) . ASNets is a recent deep learning architecture capable of representing policies for generalised planning, and designed to learn from smaller planning tasks and then apply that knowledge to tackle larger, more complex challenges within the same domain. The network's contruction exploits the relational structure of planning problems and domains, and its connectivity reflects the precondition-effect relationships captured in the domain's action schemas. This scheme makes it possible to share weights between policy networks instantiated for different problems in a domain, and learn a single set of parameters which can be transferred to problems of arbitrary size in that domain. ASNets are trained using an imitation learning algorithm, which iteratively explores the state space of the training problems using a teacher planner. Experiments with classical and probabilistic domains have shown that ASNets can outperform conventional planners when the domain has simple tricks that are key to solving larger problems but which can be learned from small problems.To extend ASNets to numeric planning, We first propose a network module that enables ASNets to directly reason about numeric fluents. Then, we illustrate why such reasoning is not always sufficient for learning effective generalised policies. We argue that reasoning about the interaction between individual fluents is crucial, and allow ASNets to perform this interaction reasoning through numeric comparisons in the problem. To cope with the increased length of numeric plans and run-time of numeric planning teachers in comparison with their classical planning counterparts, we also propose a new training algorithm which offers greater control over the exploration versus learning balance.Finally, we evaluate our proposed techniques on a representative set of benchmarks from the latest International Planning Competition, featuring both simple and linear numeric planning domains. Our results show that our extensions to ASNets allow it to learn generalised policies capa-ble of solving problem instances significantly more complex than those seen in training, and outperform non-learning planners in several domains. We also find that the greater control offered by our new training algorithm allows ASNets to be trained much more quickly without compromising the effectiveness of the learned generalised policies.As in PDDL2.1 (Fox and Long 2003) , a numeric planning problem, denoted as P = ⟨D, I⟩, consists of a domain D and an instance I. The instance I, in full, is a tuple I = ⟨O, s 0 , G, M ⟩. The initial state s 0 is any state in S, the goal G is a conjunction of comparisons and propositions, and the plan metric M is an optional arithmetic expression over F . An action is applicable in a state when its precondition is satisfied, and its application yields a new state according to its effect. Propositions and fluents not included in the effect remain unchanged. A 125 plan is a sequence of actions. It is an executable plan if when iteratively applied in s 0 , each subsequent state satisfies the next action precondition, and it is a goal achieving plan if the final state satisfies G. A valid plan is an executable goal achieving plan. The cost of a valid plan is the value of M at the final state if M is provided, or otherwise the number of actions in the plan. For this paper, a generalised policy for a domain D is a mapping from instances to executable (but not necessarily goal achieving) plans. The effectiveness of a generalised policy over a finite set of instances measuresAction Schema Networks are a state of the art approach for learning generalised policies for classical planning problems (Toyer et al. 2020) . Core to its effectiveness is its ability to generalise from a small set of training problems to much larger and unseen problems in the same domain, thereby amortising training time. This original approach is unable to perform numeric reasoning effectively due to a lack of architectural components dedicated to numeric reasoning. In this section we propose Numeric Action Schema Networks (ν-ASNets) for learning generalised policies for numeric planning.A ν-ASNet is a neural network with weights θ that takes in input vectors describing the current state s and outputs a probability distribution π θ (a|s) over all applicable actions a. For each problem instance, a ν-ASNet is constructed with the weights θ shared between all instances in the same domain. That is, the network architecture is instancedependent, but the weights are instance-agnostic through a weight-sharing mechanism that we describe later.For each instance, the ν-ASNet architecture includes layers of network modules that alternate between encoding action and state information, as shown in Figure 1 . Each action layer contains an action module for each action a ∈ A. The last layer of a ν-ASNet is always an action layer whose outputs determine π θ , and we inherit the assumption from ASNets that the first layer is also always an action layer. Each state layer contains one state module for each piece of state information, namely propositions, fluents, and comparisons. Each network fixes a hidden dimension d h , and networks modules propagate forward a hidden representation vector in R d h to connected modules in the next layer. Network modules are connected sparsely to modules in adjacent layers through a notion of relatedness. Definition 1 (relatedness) An action a is related to a proposition p, fluent f , or comparison c at position k, denoted by R(a, p/f /c, k), if p/f /c is a ground instance of the kth unique predicate/function/comparison schema appearing in the action schema of a, respectively. Example 1 Consider the following action schema for a robotic arm picking up an object of a given weight, subject to a limit on the total load carried by the arm: The notion of relatedness extends naturally to the various network modules introduced below.Action modules. The action module for a ∈ A in the lth action layer takes an input vector u l a and produces a hidden representation where W l a ∈ R d h ×d l a and b l a ∈ R d h are the learnt weight matrix and bias vector for the action module, σ is a nonlinearity, d h is the fixed hidden representation size, and d l a is the dimension of u l a . The input vector is constructed by concatenating the hidden representation of all related state modules in the previous layer, ignoring relatedness positionϕ l a = σ(W l a • u l a + b l a )u l a = ψ l-1 1 T . . . ψ l-1 M T Twhere ψ l-1 j is the hidden representation produced by a related state module in the preceding state layer. Since all ψ l-1j 200 have dimension d h , u l a has dimension M • d h .The related state modules of an action a ∈ A can be determined by enumerating all the predicates, functions, and comparison schemas in its action schema grounding them using the same objects used to ground a. If we impose an 205 ordering on these constructs (e.g. using position), the structure and dimension of u l a across all actions with the same schema is fixed. Such structure is the key to weight-sharing. For a given domain, all action modules across different ν-ASNets at the same layer l with the same action schema α 210 share the same weight matrix W l α and bias vector b l α . This allows us to apply the same set of weights to any instance in a domain, as all actions in these problem instances are grounded from the same set of action schemas.Fluent modules. In each state layer, there is one fluent module for each fluent in the problem. Fluent modules allow the network to reason directly about the quantitative components of the state space. Like action modules, a fluent module for fluent f ∈ F in the lth state layer computes a hidden representationψ l f = σ(W l f • v l f + b l f )whereW l f ∈ R d h ×d l f and b l f ∈ R d hare the learned weight matrix and bias vector, σ is the same nonlinearity as before, v l f is the input feature vector, and d l f is the dimension of v l f . Like action modules, weight-sharing requires that the input vectors v l f have a similar structure for fluents derived from the same function. Unlike action modules, the number of actions related to a fluent is not instance-agnostic, so simple concatenation of hidden representations of related action modules from the preceding layer is not sufficient. We treat this similar to how proposition modules are constructed in the original ASNets. From all the actions related to f at various positions, we extract their action schemas and enumerate all unique pairs {(α 1 , k 1 ), . . . , (α S , k S )} of action schemas and position pairs. These pairs are only dependent on the function of f . We then construct the input feature byv l f =    pool({ϕ l a | op(a) = α 1 ∧ R(a, f, k 1 }) . . . pool({ϕ l a | op(a) = α S ∧ R(a, f, k S })   where op(a) is the action schema of a and pool is a pooling function to combine multiple R d h vectors into one R d h vector. Like the original ASNets we use the element-wise max function for pool. The structure of the resulting input feature is only dependent on domain information, namely the action schemas and functions, and hence enables weight-sharing -all fluent modules across different ν-ASNets at the same layer l with the same function share the same weight matrix and bias vector.Proposition modules. Like the original ASNets, in each state layer we include a proposition module for each proposition in the problem. Proposition modules are almost identical to fluent modules, with the same computation for hidden representation, construction of input feature, and weightsharing property.Comparison modules. By including fluent modules in the network and fluent values in the network input, ν-ASNets are able to learn generalised policies that reason directly on the value of each fluent. Such reasoning is unfortunately not always sufficient. Consider a domain D whose sole numeric component involves robotic arms with load limits lifting up items of varying weight, and suppose two problem instances I and I ′ differ only in that all the load limit and item weights in I ′ are double their counterparts in I. There is no practical difference between the problem P = ⟨D, I⟩ and P ′ = ⟨D, I ′ ⟩, and an ideal generalised policy should produce the same plan for both problems. Fluent modules are unable to recognise this "symmetry" between P and P ′ -weights learned by training on P would not apply directly to P ′ .More generally, the value of fluents are often only meaningful in the context of other fluent values, and it is valuable to allow learned generalised policies to reason about the interaction between fluents. In particular, fluents interact in comparisons in action preconditions, which we capture through comparison modules. Let comp(a) denote the comparisons in action a and C = a∈A comp(a), in each state layer there is one comparison module for each comparison. A comparison module for the comparison c ∈ C in the lth state layer computes a hidden representationψ l c = σ(W l c • v l c + b l c )where W l c ∈ R d h ×d l c and b l c ∈ R d h are the learned weight matrix and bias vector, σ is the same nonlinearity as before, v l c is the input feature vector, and d l c is the dimension of v l c . Similar to fluent modules, a pooling mechanism is employed to enable weight-sharing between comparisons that share the same schema. From all actions related to c at various positions, we extract their action schemas and enumerate all unique pairs {(α 1 , k 1 ), . . . , (α S , k S )} of action schema and position pairs. Again, these pairs only depend on the comparison schema of c, and we construct the input feature using them byv l c =    pool({ϕ l a | op(a) = α 1 ∧ R(a, c, k 1 }) . . . pool({ϕ l a | op(a) = α S ∧ R(a, c, k S })   where pool is the same pooling function as before. The resulting input feature v l c is again only dependent on domain information, and hence enables weight-sharing where all comparison modules across different ν-ASNets at the same 290 layer l with the same comparison schema share the same weight matrix and bias vector.Input. The first and last action layers take minor exceptions to the above as they are the input and output layers of the network. For the first layer, there is no preceding state layer and the input vector u 1 a is a vector encoding state and heuristic information relevant to the current action. Specifically,u 1 a = v T p v T f v T c g T p g T f m c a c T L Twhere v p , v f , v c are the values of the related propositions, fluents, and comparisons of the action in the current state respectively, g p and g f indicate if the related propositions and fluents appear in the goal or not, m is a boolean value indicating if the action a is applicable in the current state, c a is the number of times a has been applied so far, and c L is a boolean vector encoding landmark information. For v c , we treat the value of a comparison as the boolean value indicating if it is satisfied. The lack of goal input g c for comparisons is a consequence of the lack of overlap between comparisons in action preconditions and comparisons in the goal. We also cannot include goal comparisons directly in the input as there is no notion of relatedness between them and actions. In domains where all instances of interest have goals with the same structure, one can define a "reach" action whose precondition is the original goal and effect is a proposition "goalreached" which also replaces the goal. This special action would allow ν-ASNets to reason about goal comparisons.The inclusion of c a and c L is to compensate for the receptive field problem discussed in the original paper (Toyer et al. 2020) . Essentially, longest chain of related action and state modules the network can reason about is limited by its fixed and finite depth. The inclusion of heuristic information can effectively address this problem. The action count c a helps the network avoid cycling between adjacent states. The numeric landmark encoding c L in ν-ASNets is derived from hybrid landmarks extracted from an AND/OR graph structure (Scala et al. 2017) . Each such landmark ℓ has a target t ℓ along with a set of actions A ℓ and contributions for each action {λ a ℓ | a ∈ A ℓ }, and represents the inequalitya∈A ℓ λ a ℓ y a ≥ t ℓwhere y a is the number of times action a is applied from the current state. Given a set of hybrid landmarks, the resulting landmark encoding c L is a vector in {0, 1} 3 , where c(1) L = 1 if the action a appears as the only action in A ℓ for any landmark, c(2) L = 1 if the action a appears in any A ℓ with other actions, and c(3) L = 1 if a does not appear in any A ℓ .We have also experimented with other encodings of hybrid landmarks, specifically encodings that take into account the contribution and target of landmarks. We additionally experimented with removing all numeric components of the numeric problem and encoding the LM-cut landmarks of the resulting classical planning problem (Helmert and Domshlak 2009) , which were used in the original ASNets. We did not find experimental success for either.Output. The last layer of the network is the output layer. The output ϕ L+1 a of each action module in the last layer is only a single real number, and the resulting output of the network is the masked softmax of all individual outputs,π θ (a | s) = m a exp(ϕ L+1 a ) a ′ ∈A m a ′ exp(ϕ L+1 a ′ )where m a is a boolean mask of whether the action a is applicable in the current state s, and π θ (a | s) is the probability of selecting action a in state s.Given weights θ for a domain, the resulting generalised policy for a domain produces a plan for each instance by repeatedly selecting and applying actions using π θ , starting at the initial state, and terminating upon reaching a goal state, a state with no applicable action, or a fixed length limit. Like ASNets, we use π θ during training by sampling from it, and during evaluation by greedily selecting the action with maximum probability and breaking ties deterministically.Miscellaneous. We have introduced comparison modules and fluent modules together, along with their implications for network input and action modules. It is important to note that they can and are designed to be applied separately. We term the network with only comparison modules C-ASNets and the network with only fluent modules F-ASNets. This specialisation allows the network to focus on a particular form of reasoning, reduce computation burden, and potentially reduce overfitting. To disambiguate, we will use B-ASNets to refer to the network with both modules, and ν-ASNets to refer to the collection of architectural variations.It is worth noting that ν-ASNets make no requirement on the form of comparisons or numeric effects appearing in the Figure 2 : The exploration algorithm used by the original ASNets, where all states from the sampled trajectories are explored with the teacher planner. For all states shown, the teacher planner is called on all applicable actions to label the optimal actions problem, and can be applied to the full numeric fragment of PDDL 2.1 (Fox and Long 2003) . Additionally, we also include skip connections between modules that correspond to the same action, proposition, etc. (Toyer et al. 2020) , 375 which make it easier for the network to propagate information across layers.Like the original ASNets, for each domain, we train ν-ASNets on a small number of training problem instances under the supervision and guidance of a teacher planner. We use the state-of-the-art ENHSP numeric planner as our teacher planner (Scala et al. 2020) . The original algorithm trains over a number of epochs. Each epoch involves an exploration phase and a learning phase. The exploration algorithm used by the original ASNets first uses the current network weights θ to sample a number of trajectories from π θ , then explores all the states from these trajectories by calling the teacher planner on them, and adding all the resulting states to a multiset state memory S mem , as illustrated in Figure 2. For each state added to S mem , the teacher planner is called on the resulting state of applying each applicable action, and the actions leading to the lowest-cost plans are labelled optimal. The learning phase then updates the weights through mini-batch gradient descent to choose the optimal actions.In our preliminary experiments we find that the original exploration algorithm is inadequate for effectively learning generalised policies for numeric planning. We observe that for numeric planning, the plan lengths of even simple train-400 ing problems tend to be longer than that of classical planning, and ENHSP to be much slower than teachers used by ASNets for classical planning. In each epoch, the number of states added to S mem in the original exploration algorithm is quadratic in the plan lengths of the training problems, resulting in a large number of states added for numeric planning. This has a number of downstream consequences in the original exploration algorithm: 1. For each state s added to S mem , the teacher is called for each applicable action in s. An increased number of states added to S mem therefore leads to a significant increase in calls to the teacher planner, especially in domains where states tend to have many applicable actions. 2. The learning phase uses states in S mem for training. We observe that the size of S mem is sometimes a few orders of magnitude larger than the number of states used in the learning phase. In this case many states added to S mem are rarely used, wasting memory and exploration effort. 3. The alternating exploration and learning phases mean that states added to S mem early are explored using outdated network weights (stale) and hence less useful for learning than more recently added states.We propose a dynamic exploration algorithm to address these problems by removing stale states and dynamically adjusting the amount of exploration performed based on the time spent on the learning phase, as shown in Algorithm 1. In each exploration phase, dynamic exploration first uses the current network weights to generate T gen trajectories for each training problem, i.e. calling the network iteratively starting at the initial state and sampling the action to apply from π θ (line 8). Each such trajectory terminates upon reaching a goal, a fixed length limit, or a state with no applicable actions. The states in these trajectories are added to an initially empty S traj multiset (line 3). We then repeatedly randomly remove (or explore) states from S traj and call the teacher planner for each removed state until a termination condition is met (lines 12 to 13). All states in the resulting teacher plan are added to S mem (line 14). This can be understood as asking the teacher planner to guide the network back onto a valid trajectory. If S traj ever becomes empty, it is refilled by generating one trajectory from each training problem from the initial state.The termination condition (line 9) is based on the average time t learn spent on recent learning phases and an hyperparameter r to control the ratio between spent on exploration and learning. We terminate exploration when either r • t learn time has elapsed in the current exploration phase or the number of states explored reaches an upper bound e max , but never before at least e min states have been explored. For the first exploration phase where t learn is undefined, we terminate once at lease one state from each problem and at least e min states overall have been explored.To avoid a size explosion of S mem and ensure its states are recent, we group states in S mem by the epoch they are added in. Whenever the number of states in S mem exceeds a limit N mem , we repeatedly remove the oldest group till the size of S mem falls back under the limit (line 16).To address point 1, we also enable an option in the original ASNets implementation to approximate action optimality. Whenever a state s is added to S mem , this option approximates the optimal action by only calling the teacher planner on s instead of calling the teacher planning for each applicable action. The approximated optimal action is simply the action selected by the teacher planner on s.For evaluation, we implemented ν-ASNets and dynamic exploration based on the original implementation of ASNets. The code will be publicly released when the paper is published.Benchmark domains and teacher planner. We use benchmarks from the International Planning Competition 2023 Numeric Track 3 , and use the state-of-the-art numeric planner ENHSP-20 (Scala et al. 2020) as the teacher planner. These domains only include simple or linear numeric planning problems, and do not fully demonstrate the applicability of ν-ASNetsto the entire numeric fragment of PDDL 2.1. For each domain, we use the 3 to 6 smallest instances for training. ENHSP has a wide set of configurations based on heuristic, search algorithm, and the use of methods such as redundant constraints and helpful actions. From these configurations, we select as the teacher ENHSP configuration one that produces short plans quickly (within one second) for the training instances. We do not experiment with domains where such a teacher configuration could not be found. The resulting benchmark domains and teacher configuration used for each domain are shown in Table 1 . We classify domains by the proportion of reasoning that is numeric versus propositional into heavily numeric and hybrid domains.For the domain Counters, we do not use the IPC instances, but instead use a set of evaluation instances with 2 to 60 counters where all the counters start with value 0. We only use one of these evaluation instances for training, and include for training another two instances similar to it but with different initial states. This set up allows us to better understand how the network would generalise across dimensions (number of counters) unvaried during training.3 https://ipc2023-numeric.github.io/ Baselines and ν-ASNet variations. We use ENHSP as the baseline for comparison using all configurations that are used as teacher for at least one domain, and report its results for the best and teacher configurations of each domain. We also compare within the ν-ASNets variations, namely the baseline network without either fluent or comparison modules (N-ASNets), F-ASNets, C-ASNets, and B-ASNets. We also compare the original ASNets training algorithm and the dynamic exploration algorithm for training F-ASNets and C-ASNets, and use superscripted O or D on the ν-ASNets variation to denote them. For fairness, we enable action optimality approximation with the original algorithm.Hyperparameters. For each domain, we train the network and evaluate it once on each problem instance. Unless otherwise specified, the hyperparameters for ν-ASNets are fixed across domains and architectural variations. We use three action layers and two state layers, with a hidden representation size (d h ) of 15 and an ELU as the non-linearity σ (Clevert, Unterthiner, and Hochreiter 2016). When using dynamic exploration, in each exploration phase we collect T gen = 2 trajectories initially, terminate exploration with parameters r = 1, e min = 10, and e max = 1000, and impose an N mem = 15000 limit on the size of S mem . When using the original algorithm, we collect two trajectories per problem and explore all states within the collected trajectories. After exploring, the learning phase performs weight optimisation using the Adam optimiser (β 1 = 0.9, β 2 = 0.99, and ϵ = 10 -7 ). Mini-batch gradient descent is performed with a learning rate of 0.0003, batch size of 50, and 60 batches per epoch. We additionally apply an ℓ 2 regulariser with a coefficient of 0.005 and a dropout probability of 0.1. We stop training when all collected trajectories reach the goal for 20 consecutive epochs.Computational limits. When training, we apply a time limit of 8 hours for dynamic exploration and 24 hours for the original training algorithm. We apply a 1800 seconds time limit per problem for ENHSP and ν-ASNets during evaluation. Training of ν-ASNets and evaluation of ENHSP is performed on a virtual machine with 32GB of memory and a single dedicated core clocked at 4.5 GHz. Evaluation of ν-ASNets is performed on the same virtual machine with only 8GB of memory.Table 1 shows the coverage achieved by the ν-ASNets variations and ENHSP by domain. The learned generalised policies are able to achieve coverages competitive with ENHSP, and outperform it on several domains, namely Delivery, FO-Counters, MPrime and TPP. Interestingly, except for FO-Counters, the other three domains all involve some forms of graph traversal and logistics. On Block Grouping, Rover, and Zenotravel, the ν-ASNets achieve coverages commensurate with ENHSP. On the remaining two domains, Counters and Drone, ν-ASNets are able to generalise from the small training problems to bigger problems and perform similarly with or outperform its teacher, but not the best ENHSP configuration. Figure 3 shows the plan cost and evaluation runtime produced by the learned generalised policies and ENHSP with the teacher configuration. When both produce valid plans, they produce plans with similar costs. On Zenotravel ν-ASNets tend to produce better plans, while on Block Group-ing ENHSP tends to produce better plans. For runtime, when both produce plans quickly (less than 10 seconds), ENHSP tends to be quicker than the generalised policies. This is likely due to the higher constant overhead required by ν-ASNets to construct the network and load the weights. On more complex problem instances, ν-ASNets tend to produce plans faster than the ENHSP teacher configuration. The large number of points on the top line in the runtime plot demonstrates that the learned generalised policies are able to solve many instances the teacher cannot solve.Why do we need F-ASNets or C-ASNets? Results in Table 1 show that the specialisation in reasoning offered by F-ASNets or C-ASNets often allow one of them to perform better than if they are both included. For example, comparison modules alone on Delivery or fluent modules alone in MPrime achieve notably higher coverages than when the other is included.Why do fluent modules result in coverages of 0 in TPP and Zenotravel? In both domains, whenever fluent modules are included in the network, the learned generalised policies fail to solve any problem. In these domains, except for fluents used to help the plan metric, all the other fluents are only meaningful in context of each other. We suspect that when fluent modules are included, the network attempts to learn to reason directly on the fluent values, but receive conflicting information on how to do so on the different training instances. This results in training never being able to converge, and ultimately the coverage of 0.Which ν-ASNets variation is the best? The best ν-ASNets variation by coverage depends on the nature of the domain. Generally, C-ASNets tend to perform well on all benchmark domains, while F-ASNets and B-ASNets perform well on particular domains such as MPrime and Rover 625 respectively.How can N-ASNets perform well on some domains? N-ASNets is not equipped with network modules for numeric reasoning, but it still has the capability for classical planning reasoning. Unsurprisingly, this allows it to still be effective on hybrid domains where there is a sizeable classical planning component. However, on heavily-numeric domains its unsuitability for numeric reasoning is clear from the poor coverage it achieves.How do ν-ASNets compare with the IPC 23 competition 635 planers? On domains where we use the same problem sets as IPC 23 (i.e. all but counters), ν-ASNets achieve better coverages than the reported coverage4 of the IPC 23 competition planners except on Drone and Rover.Existing work on generalised planning is severely limited when it comes to dealing with numeric information. Popular approaches based on Qualitative Numeric Planning (QNP) (Srivastava, Immerman, and Zilberstein 2011; Bonet and Geffner 2020) , can represent a fixed number of positive numeric variables that can only be increased or decreased by a positive non-deterministic amount in action effects, and boolean combinations of comparisons of these variables with 0 in action preconditions, initial states and goals. Other approaches allow for incrementing or decrementing a finite set of positive counters by a constant in a deterministic fashion (Srivastava, Immerman, and Zilberstein 2010; Srivastava et al. 2015) . The more recent Generalised Integer Numeric Planning (GLINP) (Lin et al. 2022 ) supports non-simple numeric effects, but is limited to integer variables. In contrast our work support the full numeric fragment (level 2) of PDDL2.1 (Fox and Long 2003) , including nonlinear effects (Scala et al. 2016a ) and numeric variables whose number grows with the number of objects. On the other hand, the above works provide guarantees on the effectiveness of generalised policies, whereas our learning approach cannot.Independently and concurrently to our work, Tariq, Valenzano and Soutchanski (2023) experimented with handling numeric planning problems with the original ASNet policy representation. They reduced the set of values each fluent takes to a finite range, which they then manually discretised into consecutive intervals, each represented by a new predicate. Numeric conditions in the action schemas are then compiled into a disjunction over these predicates. As Tariq et. al observe, this approximation of the original numeric problem creates a large number of related propositions for each action, which leads to impactically large networks and compromises the sparseness of the ASNets policy representation. The empirical evaluation conducted by Tariq et al. only used four unseen test instances per domain. These are only marginally larger than the training instances and solved within less than a second by both ENHSP and ASNets. In contrast, we have proposed an architecture that treats fluents and comparisons as first-class citizens and explicitly reasons about them. Its performance is competitive with ENHSP over the latest numeric planning competition instances.We have introduced ν-ASNets and its variations, neural network architectures for learning generalised policies for numeric planning based on ASNets. The network is able to reason directly about numeric values through fluent modules, and about numeric contexts through comparison modules. We also introduced dynamic exploration, which trains ν-ASNets much faster than the original algorithm used by ASNets, without harming the effectiveness of the learned generalised policies.Our work leaves significant room for future research. A common trait in plans for numeric planning problems is the repetition of actions. We believe that a network architecture capable of not just predicting the action to apply, but also the number of times to apply it, can be highly effective. Additionally, while our work has focused on numeric planning, the method we use to construct comparison modules can in principle be applied to other components of actions modelled in PDDL, to include constructs such as action effects and universal/existential quantifiers. This leads to a network that can potentially learn generalised policies for a much more expressive class of problems than numeric planning.the proportion of instances the policy maps to goal-acheving plans. The more effective a generalised policy is, the larger this number is.2 The value of n, or arity, is dependent on the particular predicate or function.https://ipc2023-numeric.github.io/results/presentation.pdf 
 
 Appendix:  


Decision: The paper presents an interesting extension of the ASNet architecture for generalizing numerical planning. Please, read the reviews as some details of the architecture, and the experiments are not fully clear. 

Also, the loss function being minimized is not given, only briefly mentioned in the last sentence of the first paragraph of the section "Dynamic Exploration". I believe it is the same as in the ASNet (journal) paper, but this must be given for the paper to be self-contained and reproducible.

Reviews: review: strength_weakness: strength_weakness: # MAJOR
  * Lines 156-158, if the network architecture is instance dependent, the weights can not be instance-agnostic (at least the number of weights changes if the architecture changes), so this point is not very clear.
  * Line 305, first time that uses landmarks without any previous definition, citation or how they were computed. Also, it is hard to know the impact of those landmarks in the resulting architecture.
  * Line 358, ties are deterministically broken, but it does not explain how this deterministic process works.
  * The article explains that focus on specialized modules like C-ASNets or F-ASNets reduces the computational burden and potentially the overfitting, however, there are no justifications. Regarding computational burden, it is not explained if it applies to training, evaluation or both. Regarding overfitting, it seems wrong to consider that only "fluent modules" should overfit less than "fluent+comparison modules", e.g. let's pose a problem that can be generally solved only with comparison modules, then only fluent modules should overfit more to the training instances than "fluent+comparison modules" which may focus, during the learning phase, on the comparison modules of the architecture to generally solve the problem.
  * Three action layers & two state layers do not look very deep, however, the depth is one of the key factors that affect the reasoning performance in previous ASNets (receptive field problem). It would be good to justify in more detail this design choice.
  * When combining both (fluent+comparison modules) the performance is not as good as expected. I cannot get why the neural network does not learn to discard the modules that do not help to generalize. In my opinion, this experiment should outperform the specialized ones. (Similar comment for the question/answer between Lines 610-620).

# MINOR
  * The article explains each architecture module and then the input and output to the network as special cases. I find this order a bit confusing, because the input to the network already appears for the first action layer, but input to action modules are explained as intermediate hidden representations. Introducing the original input before action modules would help the reader.
  * In definition of v^l_f (after line 235), there are missing ')' at the end of 'R' symbol
  * "v-ASNetsto" -> "v-ASNets to"
  * In the experimental setup, when only comparing heavily numeric & hybrid domains, I guess this is because there are no "heavily propositional" (i.e. classical) domains.
  * "For fairness, we enable action optimality approximation with the original algorithm." -> I did not understand this part.
questions: questions: Q1. Line 153, when the policy is defined as \pi^\theta(a|s) over *all* applicable actions. Does the output size require to be at least as large as the number of ground actions?
Q2. What do you mean by different v-ASNets at the same layer l (Lines 209-210)?
Q3. Why do you say that same weights are used to any instance in a domain (Lines 211-213)? The number of weights seems to depend on the number of grounded actions, but this changes from instance to instance (Lines 190-196).
Q4. The "action count c_a helps the network avoid cycling between adjacent states" -> is this included in any kind of loss function? otherwise, why a counter prevents cycling between adjacent states?
Q5. Line 374, what do you mean by "skip connections"? how is this implemented?
Q6. "Actions leading to the lowest-cost plans are labelled optimal" -> How do you know an action belongs to a lowest-cost plan? Are you running an optimal planner?
Q7. I don't find the difference between the "stale" problem (Line 421), and "repeatedly remove the oldest group" (Line 456). The latter will also consider the more recent states instead of the old ones, so that looks like the same problem.
strength_weakness: strength_weakness: Authors mentioned they didn’t experiment with IPC-2023 domains where they cannot find a configuration for the teacher planner. On the one hand, the numeric track repository contains 20 domains, so authors could have discussed the limitation that evaluation only shows part of the picture.  
On the other hand, the pre-selection of domains is related to the availability of good training instances and not from a limitation of the system itself. In general, IPC problem sets are designed to be challenging for state-of-the-art planners, so using the first instances may not be feasible if they cannot be solved for exploration.  Instead, one could create instances specifically for training purposes. For instance, selecting a sub-set of goals, discarding part of the objects, or using the random problem generator, which is available for some cases. As an example, the Counters domain was run with a different set. 

Minor details:
L.672: impactically -> impractically
Line 451: at lease one state -> at least one state
Figure 3: The legend, shared by the two images, can be placed outside to avoid hiding some points behind it.
questions: questions: The hidden representation size was set to 15 and no further discussed. 
- How this size relevant regarding the amount of information the ASN can encode? 
- Should we expect very different results with other d_h values?
strength_weakness: strength_weakness: 1 (Minor). The work is mostly empirical, and there is a large number of parameters/design choices whose setup is not always easy to understand. However, I think this is an inherent limit of any neural-network learning mechanism. It is hard to find some general principle from this work. 
2 (Minor). Redundancy in the description of the modules. All modules share the very same structure (the sigma function relating the input with the weights and the bias); I feel this can be factorised and space can be recovered to better explain some design choice.
3 (Minor). The work may be a little incremental w.r.t. ASNET for classical planning
questions: questions: 1.The authors seem to assume that all preconditions and goals are expressed as conjunctions, but block-grouping is a domain which makes heavy use of disjunction. How is that accounted for?
2. How are ties broken when actions have the same probabilities?
3. Is it possible that the network produces a non zero probability for inapplicable actions? If not, can you please detail where this is enforced?