Action Model Learning from Noisy Traces: a Probabilistic Approach

Venue: icaps-conference_org_ICAPS_2024_Conference
Published: 2023-12-07 15:09:54

Abstract: We address the problem of learning planning domains from plan traces that are obtained by observing the environment states through noisy sensors. In such situations, approaches that assume correct traces are not applicable. We tackle the problem by designing a probabilistic graphical model where preconditions and effects of every planning domain operators, and traces’ observations are modeled by random variables. Probabilistic inference conditioned by the observed traces allows our approach to derive a posterior probability of an atom being a precondition and/or an effect of an operator. Planning domains are obtained either by sampling or by applying the maximum a posteriori criterion. We compare our approach with a frequentist baseline and the currently available state-of-the-art approaches. We measure the performance of each method according to two criteria: reconstruction of the original planning domain and effectiveness in solving new planning problems of the same domain. Our experimental analysis shows that our approach learns action models that are more accurate w.r.t. state-of-the-art approaches, and strongly outperforms other approaches in generating models that are effective for solving new problems.

Main paper: 
 : 
 preconditions and effects of every planning domain operators, and traces' observations are modeled by random variables. Probabilistic inference conditioned by the observed traces allows our approach to derive a posterior probability of an atom being a precondition and/or an effect of an operator. Planning domains are obtained either by sampling or by applying the maximum a posteriori criterion. We compare our approach with a frequentist baseline and the currently available stateof-the-art approaches. We measure the performance of each method according to two criteria: reconstruction of the orig-The problem of learning symbolic planning domains, aka action model learning, (Aineto, Celorrio, and Onaindia 2019; Grand, Pellier, and Fiorino 2020; Mordoch, Juba, and Stern 2023 ) consists of inducing a symbolic planning domain from a set of observations of the executions of domain actions. Most of the proposed approaches start from a set of plan traces (Arora et al. 2018) , i.e., sequences of (partial) descriptions of the states of the environment interleaved by the executed actions. Moreover, state descriptions are (often implicitly assumed to be) the result of decoding the subsymbolic signals acquired through the agent's sensors.Assuming that the interpretation of the signal leads to a semantically correct description of the environment's state is often unrealistic. This is particularly evident when sensors generate high-throughput data, such as images, audio signals, natural language, point clouds, etc. In these cases, the mapping from perceptions to symbolic states is typically provided by deep neural networks, which return the truth value of state variables starting from the signals. For example, starting from the picture of an object book 1 , a neural network returns if the object is open or closed, i.e. open(book 1 ) = true or open(book 1 ) = false.In such situations, action model learning approaches that assume correct traces, such as (Yang, Wu, and Jang 2007; Cresswell and Gregory 2011; Cresswell, McCluskey, and West 2013; Stern and Juba 2017; Lamanna et al. 2021) , are not applicable. There are existing approaches designed for handling noisy traces (Rodrigues, Gérard, and Rouveirol 2011; Mourão et al. 2012; Segura-Muros, Pérez, and Fernández-Olivares 2018; Grand, Pellier, and Fiorino 2020) ; however, we believe that the problem of action model learning from noisy plan traces cannot be considered fully addressed; especially when the noise level is particularly high, where current state-of-the-art approaches do not provide a satisfactory performance.We posit that adopting a more principled approach involving a probabilistic model, which explicitly represents the components of the problem along with the associated hypotheses and priors, has the potential to provide superior results. Consequently, we design an approach, called Noisy Offline Learning of Action Models (NOLAM), that is based on a probabilistic graphical model, where preconditions and effects of every planning domain operators, and traces' observations are modeled by random variables. More precisely, for each operator, every atom involving a subset of object types that are involved by the operator is associated with a boolean random variable that models whether the atom belongs to the set of positive (or negative) preconditions of the operator; similarly for the sets of positive and negative effects of the operator. Therefore, NOLAM learns a probability distribution of the preconditions and effects of each operator, which is conditioned on the input noisy plan traces and maximum noise level. Probabilistic inference on the probabilistic model adopted by NOLAM, conditioned by the observed traces, allows NOLAM to derive a posterior probability of an atom being a precondition and/or an effect of a planning domain operator. Planning domains are obtained either by sampling or by applying the maximum a posteriori criterion.The advantage of employing probabilistic models for solving the action model learning problem lies in their ability to explicitly represent the (in)dependency hypotheses between random variables (associated with operators preconditions and effects) and explicitly model prior knowledge about their distribution. For example, the probabilis-tic model adopted by NOLAM considers the dependence between an atom being a precondition and/or an effect of an operator. Moreover, NOLAM also models the prior knowledge of an atom being true/false in a given state. This approach offers the flexibility to explore various adaptations of the probabilistic model of NOLAM by adjusting priors and (in)dependence hypotheses.The practical effectiveness of the proposed approach has been proved in an extensive experimental analysis on 23 past International Planning Competitions (IPCs) domains (Vallati et al. 2015) . We compare NOLAM with a frequentist baseline and currently available state-of-the-art approaches. We measure the performance of each approach according to two criteria: reconstruction of the original planning domain and effectiveness in solving new planning problems of the same domain. Our experimental analysis shows that NOLAM outperforms all other approaches in terms of both accuracy of the learned action models and in generating models that are effective for solving planning problems.The paper is structured as follows, we firstly discuss existing approaches for learning action models offline from noisy plan traces. Next, we provide some basic notions of planning domains and formally define plan traces with noisy states. Then, we describe in detail the probabilistic model adopted 110 by NOLAM in terms of random variables and (in)dependence assumptions. Afterward, we provide an ablation study of NOLAM where we compare different assumptions about the prior knowledge and different criteria for performing probabilistic inference conditioned by the observed traces. Finally, we perform a comparison between NOLAM, a baseline, and two state-of-the-art approaches, by evaluating the accuracy of the learned models and the performance in generating models that are effective for solving planning problems.The action model learning problem has been widely studied under different assumptions on the environment states and agent actions (Zhuo et al. 2010; Aineto, Celorrio, and Onaindia 2019; Lamanna et al. 2021; Verma, Marpally, and Srivastava 2021; Juba and Stern 2022; Mordoch, Juba, and 125 Stern 2023). We discuss related approaches by focusing on the ones that learn action models offline from an input set of noisy plan traces. In (Mourão et al. 2012) , the authors propose a method, namely ALICE, for learning action models from traces with noisy and incomplete states. ALICE trains 130 a set of classifiers for predicting the action effects, and then extracts an action model from the classifiers in the form of a set of rules. However, the method proposed in (Mourão et al. 2012 ) is tested only with large amounts of data (i.e. about 20000 transitions per domain) and small noise levels (i.e. 135 from 0 to 0.05). On the contrary, NOLAM achieves high performance with fewer data (i.e. few hundred of transitions per domain) and works with a noise rate ranging from 0 to 0.4.Another approach for learning action models from plan traces with noisy actions is proposed in (Zhuo and Kambhampati 2013) . The proposed approach takes as input a set of plan traces with no intermediate states, computes a set of "candidate" action models consistent with the input traces, and defines the probability distribution of the "candidate" action models conditioned on the input traces. They learn the parameters of such a probability distribution by means of a policy gradient algorithm (Sutton and Barto 2018) . The approach in (Zhuo and Kambhampati 2013) In (Segura-Muros, Pérez, and Fernández-Olivares 2018), authors propose planminer, a method for learning action models from traces with noisy and incomplete states. For each planning domain operator, they build a dataset consisting of the transitions where the operator appears and then learn a set of logical rules by applying an inductive learning algorithm on the operator dataset. The learned rules can be related to either preconditions or effects of the operator. Finally, the rules are converted to a PDDL (McDermott et al. 1998 ) action model.The work by (Agravante, Kimura, and Tatsubori 2023) learns action models from traces (with possibly noisy states) by means of a neuro symbolic model that relies on inductive logic (Riegel et al. 2020) . They model the preconditions and effects of an operator as a set of logical rules, where each atom is associated with a binary weight. Then, they learn the binary weights and translate the weighted rules into a PDDL action model. However, no experimental analysis shows how introducing noise in the intermediate states affects the performance of this method.Another method for learning action models from traces with noisy states is proposed in (Rodrigues, Gérard, and Rouveirol 2011) . The proposed method learns a set of logical rules associated with preconditions and effects of actions, and then incrementally specialize or generalize the learned rules as new transitions are given as input. However, the method in (Rodrigues, Gérard, and Rouveirol 2011) is evaluated on a single domain (i.e. blocksworld) for learning the preconditions and effects of a single operator from a few hundred operator transitions. Our approach does not require such a large amount of data, i.e. it learns operator preconditions and effects from a few dozen of operator transitions.Most importantly, the majority of the above approaches is based on inductive logic, which makes these approaches strongly depend on the input plan traces and less robust to high levels of noise in the states of the traces. On the contrary, NOLAM does not have this limitation. Furthermore, NOLAM uses a novel approach based on a probabilistic graphical model that allows to express a variety of (in)dependence assumptions between preconditions and effects of the planning domain operators. All of the approaches above are evaluated in a limited number of IPC domains ranging from 1 to 6, using an error metric that does not differentiate between the completeness and correctness of the learned models. On the contrary, we empirically prove the effectiveness of NOLAM by conducting an extensive experimental evaluation on 23 IPC domains; we evaluate the learned action models by means of standard precision and recall metrics adopted in (Aineto, Celorrio, and Onaindia 2019; Lamanna et al. 2021) , and the validity of the plans computed with the learned action models.There is a set of approaches that learn symbolic action models from continuous high-dimensional observations 205 such as RGB images (Kurutach et al. 2018; Konidaris, Kaelbling, and Lozano-Pérez 2018; Asai and Fukunaga 2018; Hafner et al. 2019; Asai 2019; Asai and Muise 2021; Sartor et al. 2023 ). These approaches map noisy perceptions into latent propositional states by means of deep neural networks, and learn action models where preconditions and effects are specified in terms of latent state variables. However, the learned models can be used only for planning in the latent space, and the latent states need to be decoded back to human comprehensible representations. NOLAM does not suffer from these drawbacks, and could in principle learn action models from continuous high-dimensional observations by integrating deep neural networks that map the continuous observations into (possibly noisy) symbolic states.Let P be a set of predicates with associated arity, and O be a set of operator names with associated arity. Predicates and operators of arity n are called n-ary predicates and n-ary operators. Given an n-tuple x = ⟨x 1 , . . . , x n ⟩ of distinct symbols (constants or variables), let P(x) be the set of atomic formulas p(x i1 , . . . , x im ) obtained by applying the m-ary predicate p ∈ P to any m-tuple of symbols ⟨x i1 , . . . , x im ⟩ in x (with 1 ≤ i 1 , . . . , i m ≤ n). For instance, if P contains the single binary predicate on, and x = ⟨x 1 , x 2 , x 3 ⟩. Then, We use the term lifted, as the opposite of grounded, to refer to expressions and actions where constants have been replaced with parameters.P(x) = {on(x i , x j ) | 1 ≤ i, j ≤ 3}.Definition 3 (Action model) An action model M is a triple ⟨P, O, H⟩ where P is a set of predicates, O is a set of operator names with their arity and, for every op ∈ O, H is a function mapping an operator name op into an action schema.A trace t is a set of n transitions {⟨s i , a i , s ′ i ⟩} n i=1 where s ′ i = s i \ eff -(a i ) ∪ eff + (a i ). Given a plan π = (a 1 , . . . , a n ), the plan trace of π is a trace t π = {⟨s i-1 , a i , s i ⟩} n i=1where the transitions are generated by executing π from s 0 . Similarly, a set T of m traces can be defined as m i=1 t i .Definition 5 (Noisy trace) Given a trace t, a noisy trace t of t is a set of n transitions {⟨ŝ i , a i , ŝ′ i ⟩} n i=1 where ŝi (resp. ŝ′ i ) is obtained by changing the truth value of some atom in s i (resp. s ′ i ).Notice that the only difference between a trace t and a noisy trace t generated from t is in terms of truth values of state atoms, whereas t and t shares the same set of ground actions. We denote by T a set of noisy traces generated from T , where every ti ∈ T is a noise trace of t i ∈ T .Our method is based on a probabilistic model that explains how a given set of observations, i.e. the state transitions in a set of noisy traces, can be generated by the execution of a set of operators with a certain action schema. Given a set of observations of the state transitions generated by executing an operator op ∈ O, we want to find, for every atom p(x) ∈ P(par(op)), the probability that p(x) is a positive/negative precondition and/or a positive/negative effect of op.We start by introducing the random variables that model the sets of preconditions and effects of an operator, the truth values of propositions in the state transitions generated by executing the operator, and the noisy truth values of propositions in the same state transitions observed through noisy sensors. Then, we describe the probabilistic model adopted by NOLAM for inferring the probability of an atom being a positive/negative precondition (or effect), given the truth values of such atom in the observed state transitions.For each operator op ∈ O of an action model M, and for every atom p(x) ∈ P(par(op)), we introduce two random variables R op,p(x) and E op,p(x) that take values in {+, -, ∅}. The variable R op,p(x) (resp. E op,p(x) ) indicates if p(x) is a positive, a negative, or is not a precondition (resp. effect) of op. For example, R op,p(x) = + indicates that p(x) ∈ pre + (op).We make the following independence assumptions:• For every pair of distinct operators op, op ′ ∈ O and for every atom p(x), we assume that R op,p(x) and R op ′ ,p(x) are independent, similarly for R op,p(x) and E op ′ ,p(x) , E op,p(x) and E op ′ ,p(x) , and E op,p(x) and R op ′ ,p(x) . This assumption is typically adopted in PDDL, where the specification of the sets of preconditions and effects of an operator does not depend on the sets of preconditions and effects of a different operator.• For every operator op ∈ O and pair of distinct atoms p(x), p ′ (x ′ ) ∈ P(par(op)), we assume that R op,p(x) and R op,p ′ (x ′ ) are independent; E op,p(x) and E op,p ′ (x ′ ) are independent, and R op,p(x) and E op,p ′ (x ′ ) are independent. This assumption is rather strong; however, it is an acceptable trade-off between complexity and performance. Indeed, considering dependencies between multiple preconditions and effects would result in an exponential increase in the inference complexity. Notice that, we do not assume that R op,p(x) and E op,p(x) are independent. The above assumptions allow us to describe NOLAM by focusing on a single operator op ∈ O and atom p(x) ∈ P(par(op)), though NOLAM can be applied to a set of operators O and set of predicates applied to the operators parameters. For the sake of readability, in the following, we refer to R op,p(x) and E op,p(x) as R and E, respectively.The second group of random variables concerns the (observed) truth values of ground atoms in the state transitions obtained by executing an operator op ∈ O. Let T be a set of n transitions of op and T be a set of noisy traces generated from T . For each ⟨s i , op(c i ), s ′ i ⟩ ∈ T , we introduce two boolean random variables X i and Y i that represent the truth values of the ground atom p(c i ) in s i and s ′ i , respectively. For example, both X i and Y i being true indi-cates that p(c i ) ∈ s i ∩ s ′ i .Similarly, for the noisy transition ⟨ŝ i , op(c), ŝ′ i ⟩ ∈ T , we introduce two boolean random variables Xi and Ŷi that represent the truth values of the ground atom p(c i ) in ŝi and ŝ′ i , respectively. We model noise in the sensors by the conditional probabilities P ( Xi | X i ) and For each combination of values (x, y) ∈ {0, 1} 2 , we introduce a random variable N (x,y) that counts the number of transitions in T where ( Xi , Ŷi ) = (x, y):P ( Ŷi | Y i ),N (x,y) = n i=1 1 ( Xi, Ŷi)=(x,y) .(1)Let N = ⟨N (1,1) , N (1,0) , N (0,1) , N (0,0) ⟩ be a random vector of variables that take values in {0, . . . , n}.The dependencies among the previously introduced random variables are described in the graphical model in Figure 1 . Notice that, since we do not assume R and E to be independent, we consider (R, E) as a unique variable that takes values in {+, -, ∅} 2 . Similarly, since X i and Y i are not independent, we consider (X i , Y i ) as a single variable that takes values in {0, 1} 2 . The variables X i and Y i depend on R and E since, e.g., when (R, E) = (+, -) then (X i , Y i ) = (1, 0); whereas X depends on X since, with probability 1 -e, X = x when X = x; similarly for Ŷ and Y . Finally, N depends on Xi and Ŷi , as of Equation (1).We assume that P (X i , Y i | R, E) are equal for every i in {1, . . . , n}. Furthermore,P ( Xi | X i , Y i ) = P ( Xi | X i ) (R, E) = (∅, +) (X, Y ) = (0, 0) ( 1 - k ) • 0 (X, Y ) = (0, 1) ( 1 -k ) • 1 (X, Y ) = (1, 0) k • 0 (X, Y ) = (1, 1) k • 1 ( X, Ŷ ) = (0, 0) ( X, Ŷ ) = (0, 1) ( X, Ŷ ) = (1, 0) ( X, Ŷ ) = (1, 1) k • 1 (1 -e) 2 ( 1 -k ) • 1 e ( 1 -e ) P ( X = 1, Ŷ = 1 | R = ∅, E = +) = (1 -k)e(1 -e) + k(1 -e) 2 Figure 2: Computing P ( X = 1, Ŷ = 1 | R = ∅, E = +). and P ( Ŷi | X i , Y i ) = P ( Ŷi | Y i ) since Xi (or Ŷi ) only de- pends on X i (or Y i ).Finally, we assume that P ( Xi | X i ) = P ( Ŷi | Y i ), and they are equal for every i in {1, . . . , n}, i.e. the probability of correctly observing Xi (or Ŷi ) does not 355 change in different state transitions of T . This is reasonable when, e.g., the value of X is observed through a noisy sensor with a fixed noise level. Therefore, in the following we omit the index i from X i , Y i , Xi , and Ŷi . The independence of P (X i , Y i | R, E) from i implies that P r(N = n | R, E) 360 factorizes as:EQUATIONwhere P r( X, Ŷ | R, E) is obtained by marginalizing P r( X | X)P r( Ŷ | Y )P r(X, Y | R, E) over X and Y , i.e.:x,y P r( X|X = x)P r( Ŷ |Y = y)P r(X = x, Y = y|R, E)We finally need to provide a model forP (X, Y | R, E). No- tice that P (X, Y | R, E) is deterministic when R is differ-365 ent from ∅. For instance, if R = + and E = -, i.e. if p(x)is a positive precondition and a negative effect, then p(x) must be true before and false after executing the operator, and therefore P r(X = 1, Y = 0 | R = +, E = -) = 1.When R = ∅ and E = -, i.e. p(x) is neither a positive nor 370 a negative precondition, X can be either 0 or 1. The value of Y instead is determined by X and E. For instance, if X = 1 and E = ∅, then Y must be 1. We therefore need to introduce a prior k ∈ [0, 1] of X being true. In our experiments, we set k = 0.5 since changing the value of k had no signifi-375 cant impact on the performance.Example 1 An example where we compute P ( X = 1, Ŷ = 1 | R = ∅, E = +) is shown in Figure 2 and detailed in the following. We compute P r(X = 1, Ŷ = 1 | R = ∅, E = +)by marginalizing over X and Y :x,y∈{0,1}P r( X = 1, Ŷ = 1|X = x, Y = y) •P r(X = x, Y = y|R = ∅, E = +) Since P r(X = x, Y = 0 | R = ∅, E = +) = 0 for every x ∈ {0, 1}, we have that the above summation can be written as:P r( X = 1, Ŷ = 1 | X = 0, Y = 1) • P r(X = 0, Y = 1 | R = ∅, E = +) + P r( X = 1, Ŷ = 1 | X = 1, Y = 1) • P r(X = 1, Y = 1 | R = ∅, E = +)and the above sum is equal to:e(1 -e)P r(X = 0, Y = 1 | R = ∅, E = +) + (1 -e) 2 P r(X = 1, Y = 1 | R = ∅, E = +)which can be formulated as:e(1 -e)P r(X = 0) + (1 -e) 2 P r(X = 1)and we finally obtain:e(1 -e)k + (1 -e) 2 (1 -k)we can derive P r( X = x, Ŷ = y | R = r, E = e) for (x, y) ∈ {0, 1} 2 and (r, e) ∈ {+, -, ∅} 2 in a similar way. We report all possible values of P r( X, Ŷ | R, E) in the supplementary material.Our objective is to estimate the posterior probability of (R, E) given the observations N , i.e. P r(R, E | N ).The posterior P r(R, E | N ) can be exploited for obtaining an approximation of an action model that generates the traces T . By the Bayes Theorem, P r(R, E | N ) can be formulated as:EQUATIONwhere P r(N | R, E) is typically referred to as likelihood, whereas P r(R, E) is a prior, and P(N ) is the probability of the observations N .The probability P r(N ) can be computed by marginalizing over R and E as follows: r,e∈{+,-,∅} P r(N , R = r, E = e).(4)The joint probability P r(N , R, E) can be decomposed as:EQUATIONwhere the prior P r(R, E) is given as input, and the likelihood P r(N | R, E) is computed according to Equation (2). We evaluate the accuracy of the action models learned by NOLAM from a set of noisy traces and the effectiveness of the learned action models for solving planning problems.We firstly perform an ablation study of NOLAM, and then a comparison with state-of-the-art approaches.Benchmarks We conduct an experimental analysis on 23 classical planning domains taken from past IPCs. To generate the noisy traces used for learning, we proceeded as follows. For each domain, we generated 10 random problems using the generators available from past IPCs. The statistics 420 about the domains are reported in Table 1 . For every generated problem, we computed a solution plan with the ground truth action model provided in the IPCs. As a planner we adopted FastDownward (Helmert 2006 ) and a lazy greedy best-first search with context-enhanced additive (Eyerich, 425 Mattmüller, and Röger 2012) and FastForward (Hoffmann 2001) heuristics. By executing every solution plan from the initial state of the solved problem, we obtained a set T of 10 traces for each domain. The generated traces contain a number of transitions ranging from 1 to 70, a number of ob-430 jects ranging from 3 to 58, and a number of ground atoms ranging from 8 to 687. Finally, the set T of noisy traces has been obtained by randomly changing the truth value of each ground atom in every state in T with probability e ∈ {0, 0.1, 0.2, 0.3, 0.4}.To evaluate the learned action models we use two main criteria. A first criterion measures the capability of reconstructing the ground truth model, by comparing the action model M, learned from the set of noisy traces T , with the ground truth model M, which generated the set of traces T . We compare the learned and ground truth action models using the well-known precision and recall metrics on the preconditions and effects of the operators. The precision P pre + (op) and recall R pre + (op) of the positive preconditions of an operator op ∈ O are defined as:P pre + (op) = | pre + (op) ∩ pre + (op)| | pre + (op)| R pre + (op) = | pre + (op) ∩ pre + (op)| |pre + (op)|where pre + (op) and pre + (op) are the sets of positive preconditions of op in M and M, respectively. Similarly, we define the precision and recall for the sets pre -, eff + and eff -. The overall precision P op for an operator op is defined by considering the sets pre(op) = pre + (op) ∪ pre -(op) andEQUATION:P op = | pre(op) ∩ pre(op)| + | êff(op) ∩ eff(op)| | pre(op)| + | êff(op)| R op = | pre(op) ∩ pre(op)| + | êff(op) ∩ eff(op)| |pre(op)| + |eff(op)|We explicitly define P op instead of averaging P pre + (op) , P pre -(op) , P eff + (op) , and P eff -(op) , because for domain without negative preconditions P pre -(op) and R pre -(op) are not significative. However, we want to evaluate the impact of learning wrong negative preconditions. Therefore, we adopt the above definition of P op in our ablation study for measuring the impact of wrong predictions of negative preconditions. The average precision P (or recall R) of M w.r.t. M is defined by averaging P op (or R op ) for all the operators op ∈ O.A second criterion concerns the capability of the learned models of solving problems. There are situations where it is preferable to learn an action model capable of solving more problems, than an action model more similar to the ground truth model. Indeed, an error in the prediction of a precondition or an effect can have a substantially different impact on the learned model's capability of solving problems. For example, if the learned model has a precondition that is not in the ground truth model but is implied by another precondition in the ground truth model, such a precondition would not have a negative impact on solving problems effectively. Instead, a learned model that contains an effect that is not in the ground truth model may produce non-executable plans.Therefore, we also evaluate the capability of a learned action model to generate valid plans, i.e. plans that are executable and actually achieve the goal according to theV v + - ∅ R 2(n (1,1) +n (1,0) ) 3n2(n (0,1) +n (0,0) ) 3n1 3E n (0,1) n n (1,0) n n (1,1) +n (0,0) n Table 2: P r(V = v) with V ∈ {R, E} and v ∈ {+, -, ∅}.ground truth model. We measure such capability by computing two ratios EP and EV of solved problems and valid plans to a set of problems: Ablation study We firstly perform an ablation study of NOLAM to investigate how different methods for deriving M from P r(R, E | N ) and different assumptions about the 485 prior P r(R, E) affect the precision and recall of M.We compare two methods for deriving M from the posterior P r(R, E | N ): maximum a posteriori (MAP) and sampling (sample).We also consider an orthogonal analysis that distinguishes 490 the situations where the absence of negative preconditions is given as input knowledge by setting P r(R = -, E) = 0, and the situation where P r(R, E) admits also negative preconditions. We refer to the situation where P r(R = -, E) = 0 by index pre + . The resulting variants of NO-495 LAM are denoted by MAP, sample, MAP pre + , and sample pre + . We factorize the prior P r(R, E) as P r(R) • P r(E), where P r(R) and P r(E) are estimated from T according to the formulas reported in Table 2 . For example, the prior P r(E = +) is obtained as the ratio of transitions where the 500 ground atom is false in the previous state and true in the destination state. It is worth noting that having P r(R = ∅) = 0 would make every atom either a positive or a negative precondition. To avoid this situation, we set P r(R = ∅) = 1 3 . We report the average precision and recall of the action 505 models learned by MAP, MAP pre + , sample and sample pre + in Figures 3 and 4 , respectively. The measures of precision and recall are averaged over 23 domains. Not surprisingly, MAP pre + (or sample pre + ) achieves higher precision than MAP (or sample), while the recall is comparable. This 510 is because, for every domain, the ground truth model has no negative preconditions, and MAP pre + (or sample pre + ) cannot introduce negative preconditions in the learned model. However, we noticed that most of the negative preconditions learned by MAP and sample for a single operator are 515 implied by some positive preconditions of such an operator. For example, in the blocksworld domain, the operator PUT-DOWN(x) has the positive precondition HOLDING(x), indicating that to put down a block x the agent must be holding x. However, when e.g. e = 0 then MAP and sample learn the negative precondition ¬HANDEMPTY(), indicating that, to put down x, the agent hand cannot be empty, which is implied by HOLDING(x) being true. It is worth noting that learning negative preconditions implied by positive ones decreases the precision P but does not affect the effectiveness 525 of the learned action models for computing valid plans. Interestingly, the recall of MAP (or MAP pre + ) is higher than the recall of sample (or sample pre + ), and the gap in terms of recall tends to increase as the error e increases. These results show that the maximum a posteriori criterion 530 provides action models that are more accurate in terms of recall and better or comparable in terms of precision than the sampling criterion, regardless of the admittance of negative preconditions.Comparison with state-of-the-art approaches We com-535 pare NOLAM with a baseline and two state-of-the-art approaches2 : planminer (Segura-Muros, Pérez, and Fernández-Olivares 2018) and ALICE (Mourão et al. 2012) . Since plan-miner and ALICE learn action models with negative preconditions, for a fair comparison, we adopt the MAP variant of NOLAM. We also compare with a frequentist baseline, which we refer to as baseline, that considers only the priors P r(R) and P r(E) defined in Table 2 and selects the values v for R and E that maximize the priors.We report the average precision and recall grouped by pre + , eff + and eff -for error e ranging from 0 to 0.4 (Table 3). The metrics P pre -and R pre -are not reported since the considered domains have no negative preconditions. NO-LAM always outperforms planminer and ALICE in both precision and recall, in many cases, by a large margin (e.g. when 0.2 ≤ e ≤ 0.4). On average, NOLAM improves the precision and recall w.r.t. planminer of 0.29 and 0.25, respectively; similarly, NOLAM improvements of precision and recall w.r.t. ALICE are respectively 0.25 and 0.36 on average. These results show that NOLAM learns action models that are much more accurate than the ones learned by current state-of-the-art approaches.Surprisingly, our baseline achieves competitive performance w.r.t. NOLAM. The precision P pre + achieved by NO-LAM is always greater than the one achieved by baseline. However, when 0.2 ≤ e ≤ 0.4, the recall R pre + is better for baseline. We believe this is because baseline adds more positive preconditions to every operator, introducing positive preconditions that are not in M (which reduces P pre + ), but reducing the number of positive preconditions that are in M and not in M (which increase R pre + ). The precision P eff + of NOLAM is better than or comparable to P eff + achieved by baseline for all considered values of e; similarly for R eff + , P eff -and R eff -. Notably, with high error e, i.e. e ∈ {0.3, 0.4}, NOLAM outperforms baseline in terms of (positive and negative) affects precision and recall by a large average margin (0.21 and 0.35, respectively). This is empirical evidence that, with high levels of noise, NOLAM is more robust than baseline in learning accurate action models.We further evaluate the action models learned by NOLAM, baseline, planminer, and ALICE on their capability of solving new planning problems, i.e., problems that were not used for generating the set of traces T . For each domain, we randomly generated a new set of 10 problems, and solved them with the learned models. For solving each problem, we set a CPU time limit of 60 seconds, which was sufficient for solving every problem with the ground truth action model. Figure 6 and 5 report the measures of EP and EV averaged over the 23 considered domains. We evaluated the validity of the plans by means of the validation tool adopted in past IPCs (Howey, Long, and Fox 2004) .NOLAM strongly outperforms ALICE in terms of EV, for all considered values of e; similarly, NOLAM strongly outperforms planminer for all considered values of e but e = 0, where planminer learns action models that produce plans as valid as NOLAM ones. However, the valid plans ratio achieved by planminer drastically decreases when e > 0. Surprisingly, the EV achieved by ALICE is always lower than 0.06. We noticed that the action models learned by ALICE lack of static preconditions, even when e = 0, which is a possible reason that causes them to not produce valid plans. Interestingly, the failures of planminer and ALICE are not mainly due to the fact that they do not find solution plans. Indeed, the values of EP achieved by planminer and ALICE are always higher than 0.27 and 0.21, respectively.The gap between EP and EV is lower for NOLAM and baseline than for ALICE and planminer. Hence, the action models learned by NOLAM and baseline produce a lower number of invalid plans than the action models learned by ALICE and planminer.Notably, despite the performance of the action models learned by NOLAM and baseline are comparable in terms of precision and recall, the models learned by NOLAM are much more effective for solving planning problems. The improvement of NOLAM w.r.t. baseline over plans valid-610 ity ranges from 0.04 to 0.26, and decreases as the error e increases. Even though baseline achieves higher R pre + and higher/comparable P eff -and P eff + than NOLAM when e ≤ 0.2, then EV is significantly higher for NOLAM than for baseline. This is an empirical evidence that metrics precision 615 and recall can be misleading when evaluating the learned action models quality. These results show that the probabilistic model adopted by NOLAM allows to learn action models that are much more effective for solving planning problems than current state-of-the-art approaches.We presented an approach for learning action models offline from an input set of plan traces with noisy states. Our approach, namely NOLAM, models the fact that an atom is a precondition or effect of an operator by means of a 625 probabilistic model. NOLAM performs probabilistic inference given a set of observations consisting of the transitions in the plan traces. We compare NOLAM with two state-ofthe-art approaches on 23 IPC planning domains, by evaluating the accuracy of the learned models and the effectiveness 630 of the learned models for solving planning problems. In our experimental analysis, NOLAM strongly outperforms other approaches. In future work, we will extend NOLAM to an online setting, where the traces are generated online by executing action and observing the environment state through 635 noisy sensors.Between AI Planning and Reinforcement Learning. Aineto, D.; Celorrio, S. J.; and Onaindia, E. 2019. Learning action models with minimal observability. Artificial Intelligence, 275: 104-137. Arora, A.; Fiorino, H.; Pellier, D.; Métivier, M.; and Pesty, the context-enhanced additive heuristic for temporal and numeric planning. In Towards Service Robots for Everyday Environments: Recent Advances in Designing Service Robots for Complex Tasks in Everyday Environments, 49-64. Springer.For other approaches there is no available code, which prevents us from further comparisons 
 
 Appendix:  


Decision: There is consensus on the merits of the paper to be accepted. Please, read the reviews and revise the paper accordingly.

Reviews: review: strength_weakness: strength_weakness: The main weakness that I see are that some of the assumptions are strong. In particular the assumption that the noise model is identical for all variables at all times does not seem very realistic. What would be involved in relaxing this assumption? I assume the fact that it is identical for all variables is easy to relax? Given it takes 0.2s to learn the model of an action, you could probably afford a bit more expressiveness and more computation time.


I only have minor comments:

* Definition 1, I's say par(op) is a n-tuple of variables (sorry for being pedantic).
* I cannot understand the n^{(x,y)} in equation (2).  n^{(x,y)} is not defined. Is it the value of the random variable N(x,y)? It's a bit confusing given that n is used in the equation for something else. Idem in table 2.
* typos: line 400: “and P(N) is the probability of …” (should be Pr(N)). Line 565: “that are in M and not in M” (I think you mean in \hat{M}and not in M).
* After equation 5: given that you have spelt things out in so much detail, it looks strange not to spell out the MAP criterion.
* Figure 3: I am surprised that the precision for MAP/sample Pre+ is only 0.9 with e=0. Why is that and are there particular domains that are responsible?
* If you have space in the final version, it would be interesting to show per domain accuracy results with extreme values of e.
* Supplementary material: why not use VAL as the validator?
questions: questions: 1. Please explain what would be required in order to relax the assumptions that the noise model is identical for all variables and for all time steps.
2. Please correct or explain/define the n^(x,y) in equation (2).
3. Why is the precision for MAP/sample Pre+ only 0.9 with e=0. And are there particular domains that are responsible?
4. You say you could in principle integrate neural networks to NOLAM. Can you confirm that what you're talking about is taking a neural network that has been already trained to predict the value of atoms, and using these predictions as observations in  NOLAM, rather than having a neuro-symbolic model trainable end-to-end? And if you are talking about the latter, please breifly explain how this would work and how gradients would be propagated back?
strength_weakness: strength_weakness: The second independent assumption is indeed rather strong (as acknowledged by the authors). It is unclear to me how the sets x and x' compare. Specifically, I wonder whether the two tuples of symbols are disjoint or if may they share some symbols. I am assuming that x and x' are tuples that can contain some common symbols to denote, for example, one precondition and one effect of an operator (e.g., in the blocksworld domain, op=unstack, p(x)=on(x1,x2), p'(x')=holding(x1)). How does this independence assumption affect the generation of a noisy trace \hat{t} from 't'? Does this mean that the sensor may observe either on(x1,x2) or (not (on(x1,x2)) in the pre-state of 't' and holding(x1) or (not(holding (x1)) in the post-state of 't' independently of each other? Additionally, authors claim they do not assume that that R_op,p(x) and E_op,p(x) are indepedent. I don't understand what it means to have the same atomic formula in the preconditions and effects of an operator unless it refers to a positive/negative formula in the preconditions and the reverse (negative/positive) formula in the effects (e.g. (not (holding ?param_1)) in the preconditions of operator pick-up, and (holding ?param_1) in the effects of pick-up). If this is the case, what does it mean they are not indepedent? That if (not (holding b1)) is observed in the pre-state of action pick-up(b1) then (holding b1) is observed in the post-state of pick-up(b1) and viceversa? I think this issue is very relevant to fully understand how the noisy traces are generated. (1)

The formalization is a little bit messy at some points:

1) A noise trace \hat{t} is defined over a set of 'n' transitions, the same number of transitions of the given trace 't'. Not sure if authors assume that a noise level is introduced in every transition or there may be some non-noisy transitions in a trace.
2) In the definition of the graphical model, the index 'i' is being used for two purposes: (a) to refer to the symbols of the m-tuple 'c_i' and (b) to the transitions of the set \hat{T}. Furthermore, sometimes the text says op(c_i) and others op(c).
3) If I understood well, the probability of correctly observing the truth values of the ground atoms of the pre-state and post-state of a transition of \hat{T} is the same. Consequently, the same probability is given to all the state transitions of one operator 'op'. Authors justify this by saying this is reasonable when the truth values of a state is observed through a sensor with a fixed noise level. But then this applies also to the pre-state and post-state of the transitions of every operator in the trace as I assume the same noise level is applied all along the trace. Moreover, since a state 's' in the author's approach is a full assignment of truth values to propositions, every si and s'i of all the transitions of a trace have the same size. My point is that if the level of noise is, for instance, e=0.3, then every proposition of X and Y in all the states of the plan trace will be a noisy proposition with probability 0.3 and non-noisy with probability 0.7 regardless the operator. (2)

In general, I think that the section devoted to the generation of noisy traces would benefit from clarification and, particularly, from a proper explanation of the implications of the adopted assumptions in the authors' approach.

The theoretical foundation of the prediction model is pretty straightforward. I miss some indication of the threshold of MAP that is used to derive the sets of positive/negative preconditions/effects.

I wish the authors had included elaboration on the application of the theoretical framework to their approach.

Some remarks concerning the experimental evaluation:

1) The sampling method to derive the learned model \hat{M} is never explained.
2) How is MAP used to derive the learned model? What MAP values are used to include or not a precondition/effect in an operator of \hat{M}? 
3) A brief explanation of what a frequentist model is should be included.

Overall, the superior performance of NOLAM is shown in the results. I agree with the authors that the metrics of precision and recall can be misleading when evaluating the quality of the learned action models, and the reason is that they are syntactic metrics that measure some sort of distance to the ground-truth model. To complement a proper evaluation and introduce a semantic evaluation, authors include results of the number of solved problems by \hat{M} and how many of the solution plans are valid in the original model M. Still, this evaluation is dependent on the existence of the ground-truth model M. In many practical applications, we want to learn an action model from observations where no ground-truth model is available as it happens in many Machine Learning scenarios. In my opinion, evaluating the quality of a learned model wrt to a supposedly existing true model is a limitation of the approach.
questions: questions: I'd like if authors could clarify my observations and comments labeled in the review as (1) and (2).


**** POST-REBUTTAL ****

I thank the authors for their responses to my concerns. I would have expected, though, more concise answers to my specific questions. In particular, I think the paper should include a more straightforward explanation of how the MAP of R and E impacts the noise of the states and the limitations of assuming a fixed noise level. It is important that the paper clearly states the problem that NOLAM is capable of solving highlighting not only the advantages of the probabilistic model but also the limitations of the approach due to the noise/independence assumptions.
strength_weakness: strength_weakness: While I eventually managed to understand the content of the paper, it required multiple readings of certain sections due to issues affecting clarity. Notably, numerous typos, some leading to confusion, were present. Additionally, a considerable number of sentences were excessively lengthy or lacked clarity. Specific examples of these problems are provided below. Furthermore, the problem at hand was not adequately defined. Particularly, the background section failed to formally specify what is an input to the learning problem and what is the output. I assume that the operator names and arity are given, and only the action schema is being learned, but it is never mentioned explicitly. Moreover, there was no explicit discussion regarding the scope of the problem with respect to PDDL capabilities. For instance, it is clear that numeric variables are not supported, but if I understand correctly, conditional effects are also not supported.

A second notable weakness pertains to the assumptions made in the paper. While the motivation convincingly establishes the necessity for learning from noisy transitions, particularly in scenarios where sensors are translated into symbolic representation using deep neural networks (DNNs), the paper makes a questionable assumption of a fixed noise level. This assumption lacks practicality in the context of DNNs, where the noise level is unlikely to remain constant, and is expected to drastically change in different areas of the state space. Moreover, the algorithm's evaluation falls short in examining more realistic scenarios where this assumption may not hold. Another assumption involves the independence of $\hat{X}_i$ and $\hat{Y}_i$. It is probable that the sensor would exhibit similar levels of noise in two consecutive states, especially a DNN-based sensor.

Finally, the empirical evaluation uses a fixed number of transitions (ten) and does not perform an ablation study to see the quality of the learning with a different number of input trajectories (maybe on the harder domains in the set). Personally, I don’t find the EP measure very interesting. “learning” an action model with no preconditions and all possible effects result in 100% EP. Finally, figures 3 and 4 should also show the std (e.g., as a shaded area around the mean) and not only the average. Finally, there is no explanation for why NOLAM solves more problems than the baseline, even when their P and R metrics are comparable. The authors mention that these metrics can be misleading, but do not provide a clear explanation for this phenomenon.

I am deliberating on whether to recommend accepting or rejecting the paper based on these weaknesses. At this point, I am leaning towards recommending acceptance; however, it's crucial to note that my decision is subject to change based on the author's response.

Detailed clarity issues:
1.	Line 41: “starting from the signals”. The term starting is confusing. Originally, I thought that you meant that a sequence of signals generates a state, but after additional reading, I now assume you mean “based on the signal”, or “using the signal as an input”.
2.	The sentence that spans from line 66 to line 72 (!) is very hard to follow.
3.	Line 238: triple = tuple
4.	Lines 276-281: long sentence that is hard to follow.
5.	Line 323: X_i and Y_i are not very indicative names. Why not use S_i and S’_i to indicate the meaning of the variables?
6.	Line 343: “similarly”. The word similarly is confusing, since R and E are dependent while X_i and Y_i are independent.
7.	Line 360 and Eq.2: you use n as the number of examples and \textbf{n} as the value of N. It is hard to differentiate between these symbols when they are in the same equation (e.g., n \choose \textbf{n}).
8.	Line 473: “Instead” should be “In contrast”.
9.	Line 520: “However, when e.g. e=0 then MAP” => “However, when e=0, for example, MAP and sample learn..”
10.	Figures 5 and 6: use the names EV and EP in the caption.
11.	Lines 565-566: “but reducing the number of positive preconditions that are in M and not in M”. I understand why the precision is decreased and the recall is increased based on the previous sentence, but I could not interpret what you meant here.
12.	Line 615: “that metrics precision and recall can” => “that the precision and recall metrics can”.

------------------------------------------------------------------------------------------------------------------------------------------------------
#Post rebuttal:
I appreciate the authors' response. However, certain weaknesses of the paper, especially regarding the assumption of a fixed noise level, were not fully addressed. I find this assumption unrealistic, particularly considering the motivation for utilizing DNNs to generate symbolic input from images. This concern is echoed by reviewer gLvw. Additionally, the authors' response regarding the effect of different numbers of trajectories on the results is not fully satisfactory. While they provided additional results, these only include their algorithm and lack a comparison with the baselines.

Despite these issues, I recognize the merit of this work. However, I suggest the authors adopt a more modest tone regarding their contribution, framing it as a step toward handling noisy trajectories rather than a comprehensive solution to the more realistic problem of dependent noise without prior information.

In summary, I maintain my weak acceptance recommendation but recommend that the authors refine their claims regarding the problem that their approach is capable of solving and provide a more in-depth analysis of the impact of varying the number of trajectories in the final submission.
questions: questions: 1. Can you justify the assumptions of the paper, given the motivation?
2. Can you discuss the problem with respect to the PDDL requirements? (quantifies, conditional effects, etc.)
3. Do the trends reported in the results persist when the number of trajectories changes?
4. Can you conjecture on why the baseline solves less problems?