Trajectory representation learning for Multi-Task NMRDPs planning

1 INTRODUCTION . One of the major objectives in reinforcement learning is to identify optimal policies of a given Markov Decision Process ( MDP ) , whose existence is typically ensured if the environment is Markovian ( Sutton et al. , 1998 ) . However , this assumption is usually not fulfilled in many real life problems . A first possible cause is “ partial observability ” , when the real state space is observable only through a projection on a sub-space . For instance , consider a ( cleaning ) robot that uses the distance from the walls in four directions as a state space . This representation of the environment does not necessarily correspond to the exact location in the house as different places share the exact same distances to walls . This setting is usually framed as a Partially Observed Markov Decision Process ( POMDP ) ( Shani et al. , 2013 ; Cassandra , 1998 ; Murphy , 2000 ; Hausknecht & Stone , 2015 ; Zhu et al. , 2018 ) . A second possible cause is that reward functions might depend on the whole trajectory . Consider the example of a waiter/server robot whose state space is the current position in the coffee shop and the current interaction with the clients ( taking orders , getting and delivering beverages ) . The objective is to train this robot to bring coffee when ordered . A possible reinforcement signal is to reward the agent when it processes an order that was not satisfied in the past . As the states do not include past requests , nor their fulfilment , the environment is not Markovian and the proposed reward is a function of the trajectory up to the present . This setting is a Non-Markovian Reward Decision Process ( NMRDP ) ( Bacchus et al. , 1997 ; Thiébaux et al. , 2006 ; Bakker , 2002 ; Bacchus et al. , 1996 ) . Both POMDP and NMRDP are special cases of Non Markovian Decision Processes . The second above example ( waiter robot ) is actually a multiple task problem . Planning in this setting is a challenging problem because it requires more than optimally and independently solving each task ( Toro Icarte et al. , 2018 ) . It is possible that a global optimal policy for the whole sequence of tasks induces sub-optimal policies for each one of the sub-tasks . Let us take a naive example of an agent building a shelter which requires gathering wood and stones from nearby resources . To gather wood/stones optimally , the agent should look for the nearest forest/quarry . However , if it wants to optimally gather all the resources , it should not just consider how close the forests or the quarries are , but how close they are to each other as it will have to walk from one to the other . Notice that this example shares similarities with the k-server problem ( Manasse et al. , 1990 ) . In those cases , the nature of the task is not encoded in the state space , so the Markov assumption is not verified . Casting the problems as an NMRDP requires using a ( stage by stage ) reward function that encodes the tasks specifications . In the shelter building scenario , the reward as a function of the states , while none of the tasks is achieved , is different from when stones or wood have been collected . If it is possible to map the previously observed states ( the trajectory up to the present ) into the current task , the problem can be expanded into a standard MDP , where it is known how to identify optimal policies . Motivated by the shared similarities of the described problems , we shall focus on learning to identify current tasks from the trajectory of visited states ( Wilson et al. , 2007 ) . Ideally , if annotated trajectory data sets are available , learning a trajectory representation function that expands the NMRDP into an MDP would be a supervised task . However this is a strong assumption . Indeed if a simple way of creating such data sets existed , it would be used to expand the NMRDP ’ s state space . Another issue , is that given a set of tasks , there might exist multiple non Markovian reward functions of interest . For instance , consider an agent whose goal is to maximise customers satisfaction . In this case the reinforcement signals are the declared satisfaction of said customers ( in the form of a rating for example ) . However even if different customers expect the same goals to be accomplished by the robot , their perception of the performance might differ . This example can be generalised to the case where several experts ( or simply users ) are training a robot to perform some tasks , possibly different from one expert to the other . They share a common NMRDP environment with different reward/reinforcement functions as the set of tasks ( or their appreciations/preferences ) might be different from one expert to the other . The same augmented MDP can be used for all of their respective NMRDP as they share the same latent global set of tasks ( more or less whatever the robot can physically do ) . This training can be made by the expert independently but it will certainly be faster and more efficient if made collectively . In these cases , in order to identify the full set of tasks and annotate the trajectories , a central authority must be aware of the reward function of each expert . This might be impossible due to tractability reasons , or privacy constraints . Fortunately , the experts can easily provide “ similarity ” between trajectories with respect to the latent task , i.e. , whether the robot was performing the same task or not at that time , without specifying precisely the task . In the customer example , the evaluations given by the same client can be used to compare if two trajectories are associated with the same task or not , but not if those trajectories come from batches associated to two different customers . Indeed , establishing comparisons between the tasks associated to subsets of the observed trajectories without actually knowing the actual task is possible . As a consequence , we are going to assume that learning is done using batches of trajectories , where we will have a task similarity measure within each batch . In order to identify the tasks , standard techniques identify a latent state space in which policies become Markovian , thus optimal ones exist . Unfortunately these approaches require domain specific knowledge in the form of “ relevant propositions ” ( Bacchus et al. , 1997 ; Thiébaux et al. , 2006 ; Bakker , 2002 ; Toro Icarte et al. , 2018 ) , either by estimating the hidden state or by constructing additional features . For example , in the waiter robot case , received orders are the relevant propositions that the agent needs to save in order to identify current tasks . Our major contribution in this paper is a generic approach to discover a latent state space given a set of similar and different trajectories . This assumption requires weaker domain specific knowledge . Once the latent state space is available , multi-task and meta-learning related approaches provide efficient tools to identify optimal policies ( Andrychowicz et al. , 2017 ; Colas et al. , 2019 ) or to learn a general purpose policy from which learning a specific task ’ s optimal policy can be done in few steps ( Duan et al. , 2017 ; Rakelly et al. , 2019 ) . The remaining of the paper is organised as follows : we introduce the mathematical tools used in this paper and we explain how they relate to our work in Section 2 . We specify in Section 3 a subset of NMRDPs that can be expanded with a Markovian latent space into MDPs and we provide a possible structure of such expansion . In Section 4 , we propose an algorithm to learn trajectory representation functions that approximate the tasks ’ latent state space and thus approximates the proposed equivalent MDPs . Section 5 is dedicated to empirical results . We consider toy NMRDPs grid-world environment ( where we solve a multi-task problem using learned representations ) and a real-life data set of tourists visiting Salzburg ( where we learn a trajectory representations that clusters their paths according to their destinations ) . 2 BACKGROUND . 2.1 NON MARKOVIAN REWARD DECISION PROCESS ( NMRDP ) . Both finite NMRDPs and MDPs are defined by a finite state space S , a finite action space A , a transition probability P , a discount factor γ , and a reward function R. In NMRDPs , the reward function maps trajectories into real values while in MDPs the reward function maps states into real values . To the extent of our knowledge , existing methods for planning in NMRDP ( Toro Icarte et al. , 2018 ; Bacchus et al. , 1997 ; Thiébaux et al. , 2006 ) define an augmented state space to build an equivalent MDP to the NMRDP . The most straightforward idea would be to define the augmented state space S∗ as the set of all possible trajectories . However , the number of possible trajectories grows exponentially with the horizon , which makes the numerical complexity intractable for state of the art reinforcement learning techniques . An established approach to expand the state space requires a set of domain-specific propositions . Using temporal logical operator it combines these propositions to automatically construct a set of temporal variables ( Bacchus et al. , 1997 ; Thiébaux et al. , 2006 ) that extends the state space to specify an equivalent MDP . These methods have been successfully used to teach a reinforcement learning ( RL ) agent multiple tasks ( Toro Icarte et al. , 2018 ) . However , the downside is that domain specific propositions are required to disentangle the non Markovianity of the reward function . Other approaches exploit the model free RL techniques by adding a Long Short Term Memory ( LSTM ) layer to the policy network ( Bakker , 2002 ) . The agent policy becomes a function of the full trajectory and the agent has enough information to find the optimal policy . In the tested toy examples of POMDPs , the agent needs undesirable directed exploration techniques to have a good performance ( Bakker , 2002 ) . Moreover , when dealing with similar NMRDPs with different reward functions , we need to start the training from scratch when using this approach . Motivated by the lack of approaches to expand the state space when relevant information about the reward structure are not provided , we investigate the state expansion problem in NMRDPs . 2.2 TRIPLET LOSS AND CONTRASTIVE LEARNING . Learning useful representations of the data has been done in an array of challenging tasks . The access to data in the form of similar and/or dissimilar observations enabled the development of multiple efficient contrastive learning techniques ( Mikolov et al. , 2013 ; Schroff et al. , 2015 ; Kilian & Lawrence , 2009 ) . Recent work ( Arora et al. , 2019 ) establishes a link between contrastive learning and efficient linear separability of the data . The core idea is to find a representation in a metric space so that similar samples have nearby representations and vice versa . The triplet loss is a clever evaluation of this idea . Multiple formulations have been suggested in the literature for both the loss definition and the batching procedure ( Kilian & Lawrence , 2009 ; Arora et al. , 2019 ; Ding et al. , 2015 ; Oh Song et al. , 2016 ; Hermans et al. , 2017 ) . A recent success of this approach lies in a new formulation ( Hermans et al. , 2017 ) that outperforms existing techniques in the person re-identification task ( i.e. , representing pictures of the same person from different angles as similar embedding vectors , and representing pictures of different persons as different embedding vectors ) . The suggested batching ( called PK Batch-hard ) selects P classes ( person identities ) and for each of these , K examples ( images from different angles ) . Each of the PK images is selected as an anchor , and each of the anchors is associated with the hardest positive ( the same person ’ s image from an angle which representation is the furthest to the anchor ’ s representation ) and with the hardest negative ( a different person ’ s image which representation is the closest to the anchor ’ s ) . This allows to build a batch of anchors , positives and negatives of size PK . Formally , let X be the set of the PK sampled elements , with Xji being the i th sample of the jth class , and fθ ( X j i ) be a representation parameterised by θ in a metric space R d where d is the representation dimension . The metric of the representation space is denoted by D ( u , v ) . The proposed loss is : LBH ( θ , X ) = P∑ j=1 K∑ i=1 [ m+ max p≤K D ( fθ ( X j i ) , fθ ( X j p ) ) − min p≤K , c≤P , c6=j D ( fθ ( X j i ) , fθ ( X c p ) ) ] + . ( 1 ) The margin m is a constraint on how far the negative samples should be from each other , and the relu function [ x ] + avoids correcting already well defined representations . The relu function can be substituted by any smooth approximation such as log1p ( . ) defined as log1p ( x ) = log ( 1 + ex ) . In multi-task problems , a representation that identifies the current task is sufficient to expand the state space and build the equivalent MDP . Given trajectory batches with similar and dissimilar tasks , the contrastive learning approaches , and particularly the triplet loss , lend themselves to construct trajectory representations that translate the task similarity . In these approaches , no information about the reward structure is required .

Summary: The paper studies a specific class of Non-Markovian Reward Decision Processes (NMRDPs). In general, in NMRDP the dynamics is Markovian but the reward function may depend on the entire trajectory. The authors consider a sub-case where the trajectory can be mapped to a specific "task" and the reward function can be formalized as a mapping between the state-task pair to the reward. This greatly simplifies the problem that can be mapped onto an augmented MDP and solved using standard RL tools. The authors focus on the representation learning problem of the mapping between trajectories to an embedding of the task itself. In particular, they consider contrastive learning to find a representation that discriminate between trajectories associated to the same task and trajectories coming from different tasks. The resulting LSTM-based architecture is then evaluated on a grid-world synthetic problem and on GPS trajectories from tourists in the city of Salzburg.