In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications

1 INTRODUCTION . Building agents that generalize their learning to satisfy new specifications is a crucial goal of artificial intelligence ( Oh et al. , 2017 ; Bahdanau et al. , 2019 ; Hill et al. , 2021 ) . Deep reinforcement learning ( DRL ) holds promise in autonomous agents that tackle complex scenarios ( Silver et al. , 2017 ; Samvelyan et al. , 2019 ) , which motivates ongoing research with DRL algorithms following human instructions expressed in natural language ( Yu et al. , 2018 ; Mao et al. , 2019 ) . Unfortunately , generalization in DRL is linked to training autonomous agents with large numbers of instructions requiring to build manually those natural language expressions and their corresponding reward functions , which prevents such methods from scaling efficiently ( Lake , 2019 ; Vaezipoor et al. , 2021 ) . These considerations inspired research on agents learning from formally specified instructions ( Wen & Topcu , 2016 ; Alshiekh et al. , 2018 ; Simão et al. , 2021 ) as a substitute for natural language . Formal languages ( Huth & Ryan , 2004 ) offer desirable properties such as unambiguous semantics and compositional syntax , allowing to automatically generate large amounts of training instructions and their corresponding reward functions . Earlier contributions in this area rely on the compositional nature of formal languages , often employing multiple policy networks to execute temporal logic ( TL ) instructions ( Andreas et al. , 2017 ; Icarte et al. , 2018 ; Kuo et al. , 2020 ) . However , these approaches do not scale well with the number of instructions since policy networks are a computationally costly resource and , consequently , these earlier studies are restricted to relatively small state-spaces environments that require less computation , e.g. , non-visual settings . More recent contributions ( León et al. , 2020 ; Vaezipoor et al. , 2021 ) have presented DRL frameworks that are capable of generalizing to large numbers of out-of-distribution ( OOD , i.e. , never seen during training ) instructions while relying on a single policy network . Those latter studies evidence that deep learning architectures within DRL agents are key towards the ability of agents to generalize formal instructions . However , network architectures in previous literature have mostly followed a standard configuration , where all the network ’ s layers except for those encoding the input ( e.g. , convolutional layers for images , recurrent layers for text ) have equal access to both human instructions and environment ’ s observation . We propose a novel configuration to help agents generalize better by having a task-agnostic representation of the current state concatenated to a latent goal that is form by processing both observation of the agent and human instruction . As a motivation example , let us consider two scenarios : 1 ) a robot that is at the center of an empty room with a red square at the bottom right corner , 2 ) a robot at the same position in a room that is identical to the one in ( 1 ) except that the red square is now green . Intuitively , we can say that , if we give the instruction “ go to the red square ” in ( 1 ) , the goal of the robot is the same as if we say “ go to the green square ” in ( 2 ) , because the two tasks can be abstracted as “ turn to face the bottom right of the room , then move straight ” . In a nutshell , computing the human instruction together with the current state of the environment – being at the center of the room , with the object asked by the human in the bottom right corner – allows to deduce that the optimal policy is the same in both scenarios . Our contributions are listed as follows : • We propose a new deep learning configuration ( the latent-goal architecture ) that helps agents to generalize better when solving multi-task instructions in OOD scenarios . • We are first to evaluate the performance of multiple state-of-the-art ( SOTA ) neural networks targeting generalization when following temporal logic instructions . • Through ablation studies , we find that networks generating independent pieces of information within their output are specially benefited from being used within latent-goal architectures . The remaining of the paper is structured as follows : Section 2 briefly introduces the key concepts needed to follow this work . Section 3 presents the formal language used to generate instructions and the neuro-symbolic framework that agents use in our experiments . Then , Section 4 details the proposed network configuration . Section 5 includes the experimental settings , empirical results and ablation studies . Last , Sections 6 and 7 discuss related works and conclusions , respectively . 2 BACKGROUND . We develop agents aimed to execute OOD instructions expressed in TL while navigating in partially observable ( p.o . ) environments . Specifically , agents operate with a fixed perspective and a limited visual range . Below we introduce the basic concepts used in the rest of the paper . Reinforcement learning . Our p.o . environment is modelled as a p.o . Markov decision process ( POMDP ) . A POMDP is a tupleM = 〈S , A , P , R , Z , O , γ〉 where ( i ) S is the set of states s , s′ , . . . ; ( ii ) A is the set of actions a , a′ , . . . ; ( iii ) P : S × A × S → [ 0 , 1 ] is the ( probabilistic ) transition function ; ( iv ) R : S ×A× S → R is the reward function ; ( v ) Z is the set of observations z , z′ , . . .. ( vi ) O : S × A × S → Z is the observation function . ( vii ) γ ∈ [ 0 , 1 ) is the discount factor . At each time step t , the agent chooses an action at ∈ A triggering an update in the current state from st to st+1 ∈ S according to P . Then , Rt = R ( st , a , st+1 ) provides the reward associated with the transition , and O ( st , at , st+1 ) generates a new observation ot+1 for the agent . Intuitively , the goal of the learning agent is to choose the policy π that maximizes the expected sum of discounted rewards : V π ( x ) def = Eπ [ ∑ t≥0 γ trt ] , where γ ∈ [ 0 , 1 ) is the discount factor ( see Sutton & Barto ( 2018 ) ) . Relational networks . Relational networks ( RelNets ) are a particular kind of neural network whose structure is explicitly designed for reasoning about relations ( Kemp & Tenenbaum , 2008 ) . Previous contributions on solving formal instructions with DRL have been mainly focused on multi-layer perceptrons ( MLPs ) and recurrent networks ( Goodfellow et al. , 2016 ) , yet RelNets are of particular interest in our context since they hold promise to improve generalization in DRL ( Santoro et al. , 2018 ) . Their central principle is to constrain the functional form of the neural network in the relational module so that it captures the core common properties needed to reason about the relations between entities ( e.g. , objects ) and their properties ( e.g . color ) , see Santoro et al . ( 2017 ) . Recently , relational networks have been enhanced with the inclusion of the widely-known key-value attention mechanism ( Bahdanau et al. , 2015 ) . This method relies on the computation of attention scores that are used to decide in which portions of the input data the layer using this mechanism should focus on . 3 LEARNING TO SOLVE SATTL INSTRUCTIONS . In this section we detail the shared features of our agents . Concretely , Sec . 3.1 introduces the formal language we use to procedurally generate instructions , while Sec . 3.2 details the neuro-symbolic framework that we use while testing the different neural network configurations . 3.1 SAFETY-AWARE TASK TEMPORAL LOGIC . We investigate agents learning compositionally instructions in temporal logic . Below , we formally define the language from which we generate instructions to be executed by our agents . Specifically , we extend task temporal logic ( TTL ) , defined in León et al . ( 2020 ) to study the ability of neural networks to learn systematically from instructions about reachability goals such as `` eventually reach a sword '' . In this work , we evaluate agents that handle safety constrains as well , e.g . `` walk on soil or stone until you reached a sword '' . Thus , we extend TTL to safety-aware task temporal logic ( SATTL ) . Definition 1 ( SATTL ) . Let AP be a set of propositional atoms . The sets of literals l , atomic tasks α , and temporal formulas T in SATTL are inductively defined as follows : l : := +p | −p | l ∨ l α : := lUl T : := α | T ; T | T ∪ T. Literals l are positive ( +p ) or negative ( −p ) propositional atoms , or disjunctions thereof . Atomic tasks α are obtained by applying the temporal operator until ( U ) to pairs of literals . An atom α = lUl′ is read as it is the case that l until l′ holds . Temporal formulas T are built from atomic tasks by using sequential composition ( ; ) and non-deterministic choice ( ∪ ) . Note that U and ∪ are different operators . We use an explicit positive operator ( + ) so that both positive and negative tasks have the same length . This is beneficial for learning negative literals ( −p ) when instructions are given visually , as highlighted in ( León et al. , 2020 ) . The temporal operators eventually and always 2 can be respectively defined as l ≡ trueUl and 2l ≡ lUend , where end is a particular atom true only at the end of the episode . As TTL , SATTL is interpreted over finite traces λ , which in this context refers to finite sequences of states and actions . We denote time steps , i.e. , instants , on the trace as λ [ j ] , for 0 ≤ j < |λ| , where |λ| is the length of the trace . While , λ [ i , j ] is the ( sub ) trace between instants i and j . A model is a tuple N = 〈M , L〉 , whereM is a POMDP , and L : S → 2AP is a labelling of states in S with atoms in AP . Definition 2 ( Satisfaction ) . Let N be a model and λ a finite path . We define the satisfaction relation |= for literals l , atomic tasks α , and temporal formulas T on path λ as follows : ( N , λ ) |= +p iff p ∈ L ( λ [ 0 ] ) ( N , λ ) |= −p iff p /∈ L ( λ [ 0 ] ) ( N , λ ) |= l ∨ l′ iff ( N , λ ) |= l or ( N , λ ) |= l. ( N , λ ) |= lUl′ iff for some 0 ≤ j < |λ| , ( N , λ [ j , |λ| ] ) |= l′ , and for every t ∈ [ 0 , j ) , ( N , λ [ t , j − 1 ] ) |= l ( N , λ ) |= T ; T ′ iff for some 0 ≤ j < |λ| , ( N , λ [ 0 , j ] ) |= T and ( N , λ [ j + 1 , |λ| ] ) |= T ′ ( N , λ ) |= T ∪ T ′ iff ( N , λ ) |= T or ( N , λ ) |= T ′ Intuitively , by Def . 2 an atomic task α = cαUgα is satisfied if the “ safety ” condition cα remains true until goal gα is fulfilled . In the context of this work we are not interested in strict safety warranties but in agents trying to reach a goal while aiming to minimize the violation of additional conditions . Thus , we may say our agent has satisfied α even though it has not fulfilled cα at every time step before gα . Formally , in those cases we consider traces that are not starting at the beginning of the episode , but from the state after the last violation . The agent is penalised for this behaviour through the reward function . Examples of tasks we use are α1 = −grassU ( +axe ∨ +sword ) “ avoid grass until you reach an axe or a sword ” and α2 = ( +soil ∨+mud ) Uaxe “ move through soil or mud until reaching an axe ” . It is not difficult to prove that SATTL is a fragment of the well-known Linear-time Temporal Logic over finite traces ( LTLf ) ( De Giacomo & Vardi , 2013 ) . We provide a translation of SATTL into LTLf and the corresponding proof of truth preservation in Appendix F .

Summary: This work focuses on training (via deep RL) task conditioned agents for a given class of workspaces. Tasks are formalized as a variant of finite temporal logic (and thus encode a regular language). Tasks are either provided as text or via visual diagrams. A key goal of this work is to perform well on tasks that lie outside of the training distribution. The paper illustrates on two common grid like domains with the results of a number of various architectures and ablations reported.  The approach taken is an adaptation of deep reinforcement learning, with the primary contribution being a slight, but seemingly important, modification of an existing architecture to create a separate embedding for (i) a task agnostic state encoding and (ii) create a task specific state encoding. Furthermore, in order to encourage the network to generalize between tasks, an information bottleneck is placed on the task specific encoding.