A Case for Robust AI in Robotics

Researchers envision a world wherein robots are free to interact with the external environment, thereby including human beings, other living creatures, robots and a variety of inanimate objects. It is always tacitly assumed that interactions will be smooth, i.e., they will fulfill several desirable properties ranging from safety to appropriateness. We posit that a reasonable mathematical model to frame such vision is that of Markov decision processes, and that ensuring smooth interactions amounts to endow robots with control policies that are provably compliant with side conditions expressed in probabilistic temporal logic. 1 Context, Motivation, Objectives A (stationary, discrete time) Markov decision process is a collection of objects M = {T, S,As, p(·|s, a), r(s, a)} where T ⊆ N is a set of time points over an infinite horizon; S is a finite, time-independent set of states which the system can occupy at each t ∈ T ; As is a finite time-independent set of actions allowable in some state s ∈ S where A = ⋃ s∈S As collects all actions; p(j|s, a) is a (nonnegative) transition probability function such that ∑ j∈S p(j|s, a) = 1, denoting the probability that the system is in state j at time t+ 1 when action a ∈ As is accomplished in state s at time t; r(s, a) is a function |r(s, a)| ≤M for some finite M ∈ R; r is defined for all states and actions, and it denotes the value of the reward received when executing an action in a state. A labelling function L : S → AP can be used to express additional properties of states by labelling them with atomic propositions from a set AP . A (stationary) policy (also scheduler or controller) specifies a procedure for action selection in each state of a Markov decision process. A policy π is deterministic if it is a function π : S → As, and is stochastic if it is a function π : S × A → [0, 1] with π(s, a) = 0 for all a / ∈ As and s ∈ S, and ∑ a∈As π(s, a) = 1 for all s ∈ S. The well known Markov decision problem (MDP) amounts to compute a policy which fulfils some optimally criterion within a Markow decision process M. Usually the criterion is linked to the rewards that can be accrued when acting in M according to a policy π starting from some state s, i.e., the value Vπ(s) of state s under policy π. A policy π is better than or equal to π′ (π ≥ π′) exactly when Vπ(s) ≥ Vπ′(s) for all s ∈ S. Given some reasonable definition of value V·(s) for all s ∈ S, solving an MDP amounts to find π∗ such that π∗ ≥ π for all possible π — more about MDPs and related decision problems can be found in [1]. From a modeling point of view, Markov decision processes and associated optimization problems, capture a broad set of approaches to the analysis and synthesis of intelligent behavior for autonomous agents which can be put to good use in the field of Robotics. For instance, in [2] it is argued that many problems of AI planning under uncertainty can be modeled as MDPs. In the learning community — see [3] for a recent persective — reinforcement learning (RL) is viewed as one of the key techniques to synthesize intelligent behavior for interactive agents, and the mathematical underpinning of RL is also given by Markov decision processes. Even closer to field robotics, the area of optimal control has a long tradition of leveraging MDPs for problems involving sequential decision making under uncertainty — see, e.g., [4]. Given the widespread adoption of MDPs in AI and related (sub)fields, proposing techniques to achieve robustness of autonomous agents based on Markov decision processes is bound to have a broad impact. We believe that Robotics might benefit the most from robust AI techniques, since robots ought to be functional but also dependable, and the trade-off between these two aspects need to be fully understood and explored. Our key proposition is to extend the modeling framework of MDPs to one that includes explicit side conditions expressed in probabilistic computation tree logic (PCTL). The syntax of PCTL is defined considering the set Σ of state formulas, and the set Π of path formulas. Given a set of atomic propositions AP , Σ is defined inductively as: (i) if φ ∈ AP then φ ∈ Σ; (ii) > ∈ Σ; if α, β ∈ Σ then also α ∧ β ∈ Σ and ¬α ∈ Σ; and (iii) P./p[ψ] ∈ Σ where . /∈ {≤, }, p ∈ [0, 1] and ψ ∈ Π, where P./p[ψ] is the probabilistic path operator. The set Π contains exactly the expressions of type Xα (next), αU≤kβ (bounded until) and αUβ (until) where α, β ∈ Σ and k ∈ N — more on PCTL can be found in [5]. Given an MDP M, a definition of value V·(s) for all states of M, and a PCTL formula φ, the Markov decision problem with probabilistic side condition (mdpp) can be defined as the problem of finding π∗ such that π∗ ≥ π for all policies π and DM,π∗ |= φ, i.e., φ is always satisfied in the discrete-time Markov chain (DTMC) DM,π∗ corresponding to the combination of M and π∗ — see [5] for details about DTMCS and PCTL semantics, and [6] for details about combining MDPs with policies to yield DTMCs.