Title: UNIVERSAL AGENT FOR DISENTANGLING ENVIRONMENTS AND TASKS

Abstract: Recent state-of-the-art reinforcement learning algorithms are trained under the goal of excelling in one specific task. Hence, both environment and task specific knowledge are entangled into one framework. However, there are often scenarios where the environment (e.g. the physical world) is fixed while only the target task changes. Hence, borrowing the idea from hierarchical reinforcement learning, we propose a framework that disentangles task and environment specific knowledge by separating them into two units. The environment-specific unit handles how to move from one state to the target state; and the task-specific unit plans for the next target state given a specific task. The extensive results in simulators indicate that our method can efficiently separate and learn two independent units, and also adapt to a new task more efficiently than the state-of-the-art methods.

1. INTRODUCTION:

Let's imagine ourselves learning how to play tennis for the first time. Even though we have never played tennis before, we already have a good understanding of agent and environment dynamics related to tennis. For example, we know how to move our arm from one position to another and that a ball will slow down and bounce back from the ground. Hence, we just need to learn the tennis specific knowledge (e.g. its game rule and a relationship between an arm control and a tennis racket). Just like this example, when we learn to complete a new task, we utilize the prior knowledge that is disentangled from the task and acquired over our lifetime. From a reinforcement learning perspective, this brings a very interesting question -how can agents also obtain and utilize such disentangled prior knowledge about the environment? Most of today's deep reinforcement learning (DRL) models Mnih et al. (2015; 2016) ; Schulman et al. (2015a; 2017) are trained with entangled environment-specific knowledge (e.g. transition dynamics) and taskspecific knowledge (e.g. rewards), as described in Figure 1a However, as described earlier, humans τ τ have an innate ability to obtain a good understanding about the environment dynamics, and utilize them in a newly given task. Motivated from this, we introduce a new scheme to disentangle the learning procedure of task-independent transition dynamics and task-specific rewards, as described in Figure 2 . This will help an agent to adapt to a new task more efficiently and also provides an extra interpretability.

The idea of disentangling a model into two components can be related to hierarchical RL approaches Sutton et al. (1999) ; Parr & Russell (1998) ; Oh et al. (2017) . However, to the best of our knowledge, there has not been a work that separating units by the natural criteria of environment and task specific knowledge, for the goal of transfer learning.

To this end, we introduce a model that consists of two major units: a PATH function and a goal generator. This is illustrated in Figure 2 . The key intuition is as the following. PATH function handles the environment specific knowledge, and a goal function handles the task specific knowledge. We design (1) PATH function to learn how to move from one state to another -a lower-level controller, which is independent from the task and only depends on the environment, and (2) the goal function τ to determine the next state given a target task -a higher-level planner. Thus, PATH function can be shared across different tasks as long as it is under the same environment (e.g. the physical world).

We evaluate our method to answer the following two questions: (1) how a good PATH unit can benefit the task learning, and (2) how efficient is our model for learning a new task in the same environment.

We analyze the behavior of our method on various environments including a maze world and multiple Atari 2600 games. Our study shows that a good PATH unit can be trained, and our model has a faster convergence compared to the state-of-the-art method Mnih et al. (2016) in most of the tasks, especially on transfer learning tasks.

In summary, we introduce an RL model with disentangled units for task-specific and environmentspecific knowledge. We demonstrate in multiple environments that our method learns environmentspecific knowledge, which further enables an agent adapting to a new task in the same environment.

2. FRAMEWORK:

Our goal in this paper is to introduce a reinforcement learning model that disentangles environment and task-specific knowledge. The advantage of such disentangling model is that we only need to learn task-specific knowledge when a goal of a task changes, or environment-specific knowledge when an environment of a task changes. This essentially is one form of generalization techniques across tasks or environments.

To this end, we introduce a modularized neural networks model -one unit handling a task-specific knowledge, and the other unit handling an environment-specific knowledge. For this, our model introduces two sub-functions: a goal function (τ ) and a path function (PATH). Our goal function (τ ) determines and generates the next target state for a given task (task-specific unit), and our path function (PATH) will provide the way to get to the target state (environment-specific unit). In this setup, we only have to change τ function when the goal of a task changes (e.g. hitting 5 pins instead of 10 pins) in the same environment, because the way to get from one state to another state stays the same within the same environment (i.e. PATH function can remain the same). On the other hand, we only need to change PATH function if the environment changes (e.g. faster deceleration of a bowling ball) while the task is the same, because the way to go to one state from the current state changed (e.g. due to deceleration) while the goal of the task itself is the same.

2.1. PATH FUNCTION:

We design our path function (PATH) to determine how to get to a state from another state, which represents environment-specific knowledge. Mathematically, we define: PATH(s, s ) → P(A).

Given a (starting state, target state) pair, PATH function outputs a probability distribution over action space for the first action to take at state s in order to reach state s . For discrete action spaces

A = {a 1 , a 2 , • • • , a k })

, the output is a categorical distribution, and for continuous action space: We add the -1 term in reward function to enforce the actor to choose the shortest path getting from s to s . PATH function can be learned with any RL algorithms on the defined game instances.

A = R k ,

The learning process of the PATH function is agnostic of any underlying task in the environment.

When the state space is a prior knowledge or is easy to explore, the state pairs can be directly sampled.

For complex environments where the state space is not easily tractable, one can utilize exploration strategies (e.g., curiosity-driven RL Pathak et al. (2017) ). Meanwhile, the PATH function, as a collection of "skills", can be jointly learned with one or more tasks. For example, PATH function and τ function are first jointly optimized on a source task. The PATH function is then reused in a new target task. In such cases, we sample state pairs from agents' historic experiences, which requires the restoration of an arbitrary state (see Atari games for detail). We leave other approaches for the PATH training as future work.

2.2. GOAL FUNCTION:

We now design a goal function (τ ), which determines what the goal state should be for a given task, and thus task-specific knowledge. We define our goal function for a given task: τ (s; θ g ) ≈ φ(s ) to compose a target state in feature space given the current state. PATH function is then inferred to get the primitive action to be taken.

Train with back-propagation The simplest way to train the τ function is by back-propagation through the PATH module. As in A3C, actor corporates with a task-specific critic network which maintains a value estimation for any state s. Fixing the parameters for PATH θ p , the gradient for θ g is computed as:

∇ θ g log π(a|s; θ p , θ g )A(s, a; θ p , θ g , θ v ) = ∂s ∂θ g ∇ s log PATH(a|s, s ; θ p )A(s, a; θ p , θ g , θ v ),

where s = τ (s; θ g ) and A is the advantage computed by

k-1 i=0 γ i r t+i + γ k V (s t+k ; θ v ) -V (s t ; θ v

) and γ is the discount factor Mnih et al. (2016) .

Reduce to a continuous control problem Another way to train the τ function is by viewing it as the response made by the agent and optimize it using the reward signal. Since the output lies on the manifold of φ(S), we can model it as a Gaussian distribution (as in continuous control RL). Any RL algorithms operated on continuous action space Schulman et al. (2015a; b); Lillicrap et al. (2015) can thus be applied to optimize the τ function.

A natural question would be: how to better choose the manifold of φ(S), which certainly affects the learning of τ ? For example, can we use β- VAE Higgins et al. (2016) which enforces the distribution of φ's output to be close a Gaussian distribution to help? In the experiment section, we will study several choices of the φ(S) space including auto-encoder Hinton & Salakhutdinov (2006) , beta- VAE Higgins et al. (2016) , forward dynamics Chiappa et al. (2017) , inverse dynamics (one-step PATH function) Pathak et al. (2017) , and the feature space jointly learned with PATH function.

Inspired by the recent advances in low-bits neural networks Rastegari et al. (2016) ; Zhou et al. (2016) , to eliminate d.o.f. brought by the Gaussian distribution, we conduct experiments where we binarize the output coding of φ. Specifically, we first train φ with a tanh activation at the last layer, then binarize the output, and finally fine-tune the φ again. By doing so, the output distribution of τ function becomes a binomial distribution.

Notes on the τ function learning The major difference between two methods lies in the position where the sampling and the supervision of reward signals take place: at the output of PATH or the output of τ . The first approach relies on PATH to update the planning made by τ using gradients back-propagated through the PATH module (environmental reward signals are used to optimize primitive actions that the model outputs). On the other hand, the second approach based on τ directly optimizes the state planning in the (latent) state space (environmental reward signals are directly used to optimize the goal state generated by τ ). We slightly prefer the first approach as it is easy to implement and is more stable when the (latent) state space is changing (because in some settings we are jointly training the PATH module and τ module).

2.3. TRAINING DETAILS:

Reinforcement learning model We implemented batched A3C as our base model following same settings as original paper Mnih et al. (2016) . The discount factor is chosen as γ = 0.99 for all our experiments. We use a convolutional neural network (CNN) structure (2-4 layers depending on the task) instead of LSTM as feature extractor for φ and τ , while PATH function is a 3-layer MLP. To preserve historical information, we concatenate four consecutive frames in channel dimension as input to our network (as state s). The critic network shares feature with τ (they are both task-specific) and has two extra fully-connected layers for value estimation.

Curriculum learning for Path function The game defined in Section 2.1 is a game with very sparse reward. To accelerate the training process, we employ curriculum learning to help. In particular, the max distance between starting state and final state is initialized as one at the beginning of the training process. After every K iterations of updating, the max distance is increased by one. This max distance can be either accurate (by finding the shortest path from s to s ), heuristic (e.g., Manhattan distance on a 2D maze), or determined by agents' experience (see experiments).

Compared to other pre-training methods or hierarchical RL methods, the proposed universal agents shows following two main advantages:

1. PATH function can be obtained without task specification. With only exploration (e.g., naive random walk or curiosity-driven RL Pathak et al. ( 2017)), PATH function can be trained using the experience.

2. PATH function encodes no information about the task. When jointly learned on one or more tasks, PATH function shows better generalization to new tasks.

3. EXPERIMENTS:

We begin our experiments with a small environment: Lava World. Rich ablation experiments are conduced to study the choice of feature space, optimization strategy and its knowledge transfer ability. We further extends our result to a more realistic environment: Atari 2600 games. More experiments such as continuous control problem or imitation learning would be shown in the appendix. Therefore, to go from one room to another, the agent is forced to find the door connecting two rooms and go through it, which is harder than random generated 2D maze.

Path function The set of states in lava world is definite, we directly use randomly generated pairs of starting state and final state to train the path function. During this training phase, the agent understands how to move as well as how to go from one room to another.

We define the generalization ability of the PATH function as the ability to reach the target states whose distance is larger than training samples. We first test this by setting a max distance between the starting state and the final state during training but testing it on harder instances (the distance is longer). Figure 4a shows the result. When trained with very simple instances (distance is less than 7), the PATH function shows very limited generalization ability. When the training distance is increased to 7, it begins to generalize. This phenomenon was also witnessed in other experiments: too short training distance leads to pool generalization.

(a) Generalization analysis of PATH function. Task: Reachability In this task, agents are asked to move to the target position presented on the map as a special mark. In all experiments, agents are provided with dense reward: when the agent is getting closer to the target position, it can receive positive reward. Shown in figure 5 , universal agent demonstrates faster learning speed over pure A3C models with no prior knowledge. Furthermore, the learned τ function is well interpretable and we provide the visualization of the goal state composed by the agent in the appendix A.

In figure 5 , the τ function is optimized by the gradients back-propagated through the PATH module, and the PATH function is almost "perfect". In other words, the PATH function can resolve the path between every pair of states. The goal generator only focuses on the goal state composition.

Feature space and training strategy We now go deeper into the choice of feature space of φ and the training strategy for τ .

We explored following feature spaces for the state encoder φ:

• AutoEncoder: Encode the input state with an information bottleneck proposed in Hinton & Salakhutdinov (2006) . • β-VAE: Encode the input state with a disentangled coding by best compressing the information. We adjusted β such that the information is reduced as much as possible under the constraint that the reconstruction is perfect. From another point of view, β-VAE also enforces the feature space to be close to a Gaussian distribution. Higgins et al. (2016) .

• Forward dynamics: Similar to Chiappa et al. (2017) , we trained an neural network to approximate the dynamics of the environment: p(s t+1 s t , a t ).

• Inverse dynamics: Also used by Pathak et al. (2017) , inverse dynamics is an one-step PATH function: p(a t s t , s t+1 ).

• No pre-training (shown as "Scratch" in the figures). The feature space is jointly learned with PATH function.

We first train PATH function for certain φ space with sufficient amount of data. Shown in Figure 4b , auto-encoder and β-VAE perform worse than other feature spaces due to their ignorance of the underlying environment's dynamics. The interesting "odd-even" pattern, we hypothesis, is attributed to the feature space for pixel-level reconstruction. For all feature space mentioned, we model the output distribution of τ function as multi-dimensional Gaussian distribution with a scalar variance σ 2 . For their binarized variants, we model the distribution as binomial distribution. We also investigate two variants of the distribution modeling: Gaussian-Additive (φ(t) ≈ φ(s) + N (µ, σ 2 )) and Binomial-Multiplicative (φ(t) = φ(s) * Binomial(logits)).

We use proximal policy optimization (PPO) Schulman et al. ( 2017) for the distributional τ learning. Shown in figure 6 , Gaussian, and Gaussian-A have similar behavior while Binomial is typically a better choice than Binomial-B, probably because currently τ (s) does not depend on φ(s), which means the target state space composer has no information about the current state representation φ(s). Gaussian models perform well on all feature spaces with pixel-level information (e.g., β-VAE and dynamics), while fails in the feature space with more high-level, discrete information (inverse dynamics). The hyper-parameters for the distributional models are generally more difficult to adjust, but might be useful in robotics settings where the input space itself is usually a continuous space (e.g., joints of a robot). However, it shows no clear advantage over the results shown in figure 5 on the visual input games. In no-reward exploration setting, PATH function is trained from scratch without pre-trained feature space. Since we can not exactly get the distance between two states in this setting (assuming we have no knowledge about the state space), when picking the starting state and the final state from the experience pool, we set the limit the number of steps that the explorer used as 15. With a same number of frames experienced (∼ 32M frames), the PATH function can reach a comparable accuracy as the original curriculum learning correspondent. We compare the performance of two models To study the transfer across multiple tasks, we define another task in the environment: Taxi. The agent now acts as a taxi driver, who first goes to one position to pick up the passenger (shown on the map), and then goes to another place. The target position will be provided to the agent by two one-hot vectors representing the (x, y) coordinates after our agent picks up the passenger. In order to complete the task, the agent learns how to interpret the task representation (one-hot vectors).

We compare the performance of universal agent with both A3C baseline trained from scratch as well as A3C model pre-trained for reachability game (A3C-L) in Figure 7b . Even though the first phase of the task is exactly the same as reachability game, universal agent excels in the task interpretation and outperforms both baselines.

3.2. ATARI 2600 GAMES:

We tested our framework on a subset of Atari 2600 games, a challenging RL testbed that presents agents with a high-dimensional visual input (210 × 160 RGB frames) and a diverse set of tasks which are even difficult for human players.

Feature spaces and PATH function A critical feature of Atari games is the difficulty of exploration, which is attributed to the complexity of the environment as well as the intricate operation. We use state restoration for the PATH learning, starting state and targeting state are sampled from agents' experience performing tasks or exploration. The state restoration here is an extra requirement for the learning, and we believe that techniques for PATH training without state restoration can be separately addressed.

Also, as the state space is not known as a prior knowledge for Atari games, we directly optimize the state encoder φ jointly with the PATH function.

We prefer to train a PATH function jointly with tasks. We first chose 8 typical Atari games to validate how PATH function can be learned jointly and show its effectiveness in the appendix D and visualization in the appedix F. Also, we provide analysis on using pure exploration for PATH function in the appendix E.

Knowledge transfer from other tasks Universal agent is capable of efficient transfer learning. We further define some new tasks for four of the Atari games. The task specifications are included in the appendix G. All new tasks have large differences with the corresponding original task in the game (E.g., in original Riverraid, the agents should fire the boats. While in the new task, the agents should avoid firing any boats.) In Figure 8 , we demonstrate how our universal agent can benefit from the pre-trained PATH function in the original task (jointly learned with a τ function for the original task). For a fair comparison, in A3C-L, we load all the weights (except the last fully-connected layers for policy and value) from the model trained on original task, while A3C represents A3C trained from scratch.

In both Alien-new and Riverraid-new, UA with pre-trained PATH function outperforms A3C baselines.

We also show two more games where UA fails to perform very well. In game UpNDown-new, A3C succeeds in finding a bug in the game, and use it to escape from other cars. While performing slightly worse than A3C, UA chooses to jump over other cars to avoid being hit, which is more reasonable and rational. In JourneyEscape-new, both UA and A3C-L perform better than A3C trained from scratch.

Task transfer from multiple sources We demonstrate the ability of knowledge transfer from multiple tasks in two Atari games. A set of pre-trained A3C agents for different tasks in same environments are employed to collect experiences for our agent in the PATH function training phase. This method essentially "distills PATHfrom multiple agents". The task specifications can be found in the appendix. The learning procedure corresponds to humans' life-long learning from multiple tasks in the same environment. With this prior knowledge, agents are trained on completely new tasks with PATH function given and fixed. Results for A3C-S and A3C-L are also provided as baseline models.

Another implicit advantage of universal agent is that PATH functions, as sets of skills, will filter out meaningless actions. When we extract PATH function from pre-trained models, PATH function only learns how to reach those meaningful states. This helps the exploration and exploitation for the agents in new target tasks. In these two environments, universal agent shows faster convergence and better performance than a3c with pre-trained models on another task. Currently, the results are not completely comparable with the method for transfer from a single source (e.g., for the Riverraid task). How to effectively learn the PATH function from several tasks, either in a parallel or a sequential manner, falls into the category of continual learning and life-long learning Ring (1994; 1997) . We leave this as a future work. 2017)). In this paper, we address the problem of representing the knowledge about the environment by learning skills (how to reach a given state). As a cognitive science motivation described in Graziano (2006) , humans also store the knowledge of movements or actions by their end-states. Most importantly, this knowledge can be easily utilized by a task-specific module (the goal generator) to exploit novel tasks. 2017), the τ function does not perform option selection but directly composes target state since a general-purpose PATH function is utilized. We do not follow typical HRL methods where once the subtask is selected, the low-level controller will be executed for multiple time steps. In universal agent, for simplicity now, τ function plans the future state at every time step. We leave its adaption to HRL frameworks as a future work.

4. RELATED WORKS AND DISCUSSION:



Model:

Instruction-based reinforcement learning In Oh et al. (2017) , instruction are provided to agents as task description. Trained with the ability to interpret instructions, agents can perform zero-shot generalization on new tasks. However, the performance of this method is still restricted by the set of tasks the agents are trained with. where policy derived from experience, it provides an intuitive way to communicate task information as human. A common solution is to performs supervised learning from observations to actions Pomerleau (1989); Ross et al. (2011) . However, purely supervised learning suffers from the lack of knowledge about domain dynamics, which leads to bad generalization when the environment is complex Hester et al. (2017) . We show in the appendix how we incorporate prior knowledge of environment into an imitation learning framework and improve its generalization and robustness.

5. CONCLUSION AND FUTURE WORKS:

We present a new reinforcement learning scheme that disentangles the learning procedure of taskindependent transition dynamics and task-specific rewards. The main advantage of this is efficiency of task adaptation and interpretability. For this we simply introduce two major units: a PATH function and a goal generator. Our study shows that a good PATH unit can be trained, and our model outperforms the state-of-the-art method Mnih et al. (2016) in most of tasks, especially on transfer learning tasks.

The proposed framework is a novel step towards the knowledge representation learning for deep reinforcement learning (DRL). There are a variety of future research directions. For example, how PATH function can be learned (e.g., in a continual learning manner Ring (1994), with less requirement such as state restoration), how it can better cooperate with the goal generator (e.g., incorporating explicit future planning) and how it can be used for other tasks (e.g., learning from demonstration).

B IMITATION LEARNING IN LAVAWORLD:

In imitation learning (IL), task supervision is provided by the demonstration coming from either human experts or other pre-trained agents. A straightforward way is to minimize E (s,a)∈D log π(a|s; θ). Given a set of demonstration plays (sequences of state-action pair D), we can add extra supervision signal on Universal Agent to make it better understand the goal.

L = L + α E demo KL[τ (s t )|s t+k ],

where s t , s t+k are two sampled state from a trajectory in D. k is a planning horizon factor. Figure 11 shows the empirical analysis of agents' performance in the game with respect to the number of demonstration plays. With a limited number of demonstration plays, universal agent interprets the task and shows better generalization than agents trained from scratch. In lunar lander, the state space is rather difficult to explore by random actions. Thus we make use of a pre-trained A3C model for the landing task and other heuristic agents for imitation learning (described later) to provide demonstration plays for our agent in the PATH function training phase.

To ensure the generality of learned PATH function, a large noise is added to the action performed by the demonstrators (N (0, 0.6) in our setting, compared to the valid value interval [-1, 1]). In Figure 15 , with PATH function learned during exploration, the agents showed faster learning speed on the given task initially. However, their performance (score) is typically worse than the A3C model.

In both tasks, we find that the agents are able to control themselves, but fail to effectively avoid colliding with objects which will lower the score. In path training, because of the absence of reward signal, the PATH function fails to learn how to avoid certain "bad" objects. 

:

For simplicity here, environmental randomness is considered as a part of the transition dynamics. E.g. assuming s contains the random seed of the environment.

Figures and Tables:

Figure 1: Our model disentangles environment-specific information (e.g. transition dynamics) and task-specific knowledge (e.g. task rewards) for training efficiency and interpretability.

Figure 2: Proposed universal agent, which consists of three parts: a φ function mapping raw observation to feature space, a PATH function as an environment actor, and a τ function for future state planning.

Figure 3: Lava World

The quality of path functions learned over different feature spaces.

Figure 4: Experiments showing the performance of the PATH function.

Figure 5: The rate of successfully reaching the final position in game Reachability. The universal agent model is trained through backpropagation.

Figure 6: Different feature spaces affect the learning of τ function

PATH function learned using curiositydriven RL without any reward.(b) The probability of successfully finish the task during testing time in game Taxi. We only measure the probability of agents finishing the whole task.

Figure 7: Knowledge transfer via the PATH function. in Figure 7a. When used with τ function for the reachability task, their performances are almost identical.

Figure 8: Comparison between A3C, A3C with pre-trained model on the original task, and UA with pre-trained PATH function. The translucent regions indicate the standard deviations of multiple runs. All environments are based on the latest versions of Gym Atari.

Figure 9: Comparison between A3C, A3C with pre-trained model on one of source tasks, and UA with pre-trained PATH function from multiple source tasks. The translucent regions indicate the standard deviations of multiple runs. All environments are based on the latest versions of Gym Atari.In these two environments, universal agent shows faster convergence and better performance than a3c with pre-trained models on another task. Currently, the results are not completely comparable with the method for transfer from a single source (e.g., for the Riverraid task). How to effectively learn the PATH function from several tasks, either in a parallel or a sequential manner, falls into the category of continual learning and life-long learningRing (1994;1997). We leave this as a future work.

-based reinforcement learning Typical model-based RL frameworks aim to model the dynamics of the environment. They usually involve search-based algorithms (e.g., Monte Carlo Tree Search) as part of the policy Sutton & Barto (1998) (e.g., Alpha GO with a simulator Silver et al. (2017), Scheme-networks with learned forward dynamics Kansky et al. (

Multi-task learning and transfer learning Multi-task learning and transfer learningDevin et al. (2016);Andreas et al. (2016);Taylor & Stone (2009) provide approaches to transfer knowledge among multiple agents. The methods include the decomposition of value function or task and direct multi-task learning where agents learn several tasks jointly. Contrary to them, universal agent is able to obtain the environment specific knowledge without any specific task supervision.Hierarchical reinforcement learningIn standard hierarchical RL (HRL) Sutton et al. (1999); Hernandez-Gardiol & Mahadevan (2001); Kulkarni et al. (2016); Fox et al. (2017) a set of subtasks are defined by a hyper-controller. Low-level controllers receive subtask signals from the hyper-controller and output primitive actions. Both low-level controllers and the hyper-controller (sometimes called meta-controller) learn from a specific task. Contrary to them, the proposed PATH function can be trained without supervision and provides a more general interface for potential tasks. Unlike Schaul et al. (2015); Vezhnevets et al. (

Figure 11: The probability of successfully reaching the final position during testing time in game Reachability by pure imitation learning.

Figure11shows the empirical analysis of agents' performance in the game with respect to the number of demonstration plays. With a limited number of demonstration plays, universal agent interprets the task and shows better generalization than agents trained from scratch. Figure ?? illustrates consistent results for imitation learning + RL.

Figure 12: Lunar Lander

Figure 14: Comparison between A3C, jointly trained UA, and UA with pre-trained PATH function. With a pre-trained PATH function, a faster convergence speed in universal agent is witnessed.

Figure 15: Knowledge transfer from exploration in Atari games

Figure 16: Visualizations of the saliency maps for the PATH function w.r.t. both inputs in the game Riverraid Scene #1. Columns (left-to-right): input state, the saliency map, the Jacobian. First row: current state. Second row: next state (with certain distances).

Figure 17: Visualizations of the saliency maps for the PATH function w.r.t. both inputs in the game Riverraid Scene #2. Columns (left-to-right): input state, the saliency map, the Jacobian. First row: current state. Second row: next state (with certain distances).

Figure 18: Visualizations of the saliency maps for the PATH function w.r.t. both inputs in the game Alien Scene #1. Columns (left-to-right): input state, the saliency map, the Jacobian. First row: current state. Second row: next state (with certain distances).

Figure 19: Visualizations of the saliency maps for the PATH function w.r.t. both inputs in the game Alien Scene #2. Columns (left-to-right): input state, the saliency map, the Jacobian. First row: current state. Second row: next state (with certain distances).

a multidimensional normal distribution with a spherical covariance is outputted(as Lillicrap et al. (2015)). To generate a trajectory from s to s , one can iteratively apply the PATH function and obtain a Markov chain.Given the state space S, PATH function is trained by sampling a starting state and a final state and teach the PATH function to generate the path from one to another using reward signal. Formally speaking, we define a series of game instances based on the original transition dynamics s = Trans(s, a)1 , where s is the state and a is the action. For sampled state pair (s, s ) ∈ S = S 2 , we construct the following game instance (S , Trans , Reward ):

Table content: <table><tr><td>Trans ((s, s ), a) = (Trans(s, a), s ); Reward ((s, s ), a) =</td><td>1 -1 otherwise if Trans e (s, a) = s</td><td>.</td></tr></table>

Universal agent addresses the generalization among multiple tasks by a general PATH function.Path function PATH function is an instance of universal value function approximatorSchaul et al. (2015). While a set of works use the prediction of future state as knowledge representation for RL agentsChiappa et al. (2017); Dosovitskiy & Koltun (2016) (i.e., trying to approximate the transition dynamics p(s t+1 s t , a t )), the proposed PATH function can be viewed as an inverse and a high-level abstract version of the approximated transition dynamics. From another point of view, PATH function is a variant of the feasibility function which measures the reachability from one state to another. An important property of PATH function is the ignorance of any specific tasks. Therefore, PATH function can be shared among different agents working on different tasks but in the same environment.

Table content: <table><tr><td>Imitation learning Imitation learning Hester et al. (2017); Duan et al. (2017) considers the problem</td></tr><tr><td>of deriving a policy from examples provided by a teacher. Contrary to Reinforcement Learning (RL)</td></tr></table>

While the primitive action policy was obtained in environment-specific training phase, agents are forced to understand how to plan future state (what should happen) instead of primitive actions (what should be done). We show empirically that universal agent demonstrate better generalization and robustness over pure imitation learning.

Table content: <table><tr><td/><td>1.0</td><td/><td/><td/></tr><tr><td/><td/><td>UA</td><td/><td/></tr><tr><td/><td/><td>Imitation</td><td/><td/></tr><tr><td/><td>0.8</td><td/><td/><td/></tr><tr><td>probability of success</td><td>0.4 0.6</td><td/><td/><td/></tr><tr><td/><td>0.2</td><td/><td/><td/></tr><tr><td/><td>0.0</td><td>10 50 100</td><td>300</td><td>500 # of demonstrations</td><td>1000</td></tr></table>

Peer review (iclr2018):
 The authors propose to decompose reinforcement learning into a PATH function that can learn how to solve reusable sub-goals an agent might have in a specific environment and a GOAL function that chooses subgoals in order to solve a specific task in the environment using path segments. So I guess it can be thought of as a kind of hierarchical RL. The exposition of the model architecture could use some additional detail to clarify some steps and possibly fix some minor errors (see below). I would prefer less material but better explained. I had to read a lot of sections more than once and use details across sections to fill in gaps. The paper could be more focused around a single scientific question: does the PATH function as formulated help? The authors do provide a novel formulation and demonstrate the gains on a variety of concrete problems taken form the literature. I also like that they try to design experiments to understand the role of specific parts of the proposed architecture. The graphs are WAY TOO SMALL to read. Figure #s are missing off several figures. MODEL & ARCHITECTURE The PATH function given a current state s and a goal state s*, returns a distribution over the best first action to take to get to the goal P(A). ( If the goal state s’ was just the next state, then this would just be a dynamics model and this would be model-based learning? So I assume there are multiple steps between s and s’?). At the beginning of section 2.1, I think the authors suggest the PATH function could be pre-trained independently by sampling a random state in the state space to be the initial state and a second random state to be the goal state and then using an RL algorithm to find a path. Presumably, once one had found a path ( (s, a0), (s1, a1), (s2, a2), …, (sn-1,an-1), s’ ) one could then train the PATH policy on the triple (s, s’, a0) ? This seems like a pretty intense process: solving some representative subset of all possible RL problems for a particular environment … Maybe one choses s and s’ so they are not too far away from each other (the experimental section later confirms this distance is >= 7. Maybe bring this detail forward)? The expression Trans’( (s,s’), a) = (Trans(s,a), s’) was confusing. I think the idea here is that the expression Trans’( (s,s’) , a ) represents the n-step transition function and ‘a* represents the first action? The second step is to train the goal function for a specific task. So I gather our policy takes the form of a composed function and the chain rule gives close to their expression in 2.2 PATH( s, Tau( s, th^g ), a ; th^p ) d / { d th^g } PATH( s, Tau( s, th^g ), a ; th^p ) = {d / d {s’ } PATH } ( s, Tau( s, th^g ), a ) d / {d th^g} Tau( s, th^g) What is confusing is that they define A( s, a, th^p, th^g, th^v ) = sum_i gamma^i r_{t+1} + gamma^k V( s_{t+k} ; th^v ) - V( s_t ; th^v ) The left side contains th^p and th^g, but the right side does not. Should these parameters be take out of the n-step advantage function A? The second alternative for training the goal function tau seems confusing. I get that tau is going to be constrained by whatever representation PATH function was trained on and that this representation might affect the overall performance - performance. I didn’t get the contrast with method one. How do we treat the output of Tau as an action? Are you thinking of the gradient coming back through PATH as a reward signal? More detail here would be helpful. EXPERIMENTS: Lavaworld: authors show that pretraining the PATH function on longer 7-11 step policies leads to better performance when given a specific Lava world problem to solve. So the PATH function helps and longer paths are better. This seems reasonable. What is the upper bound on the size of PATH lengths you can train? Reachability: authors show that different ways of abstracting the state s into a vector encoding affect the performance of the system. From a scientific point of view, this seems orthogonal to the point of the paper, though is relevant if you were trying to build a system. Taxi: the authors train the PATH problem on reachability and then show that it works for TAXI. This isn’t too surprising. Both picking up the passenger (reachability) and dropping them off somewhere are essentially the same task: moving to a point. It is interesting that the Task function is able to encode the higher level structure of the TAXI problem’s two phases. Another task you could try is to learn to perform the same task in two different environments. Perhaps the TAXI problem, but you have two different taxis that require different actions in order to execute the same path in state space. This would require a phi(s) function that is trained in a way that doesn’t depend on the action a. ATARI 2600 games: I am not sure what state restoration is. Is this where you artificially return an agent to a state that would normally be hard to reach? The authors show that UA results in gains on several of the games. The authors also demonstrate that using multiple agents with different policies can be used to collect training examples for the PATH function that improve its utility over training examples collected by a single agent policy. RELATED WORK: Good contrast to hierarchical learning: we don’t have switching regimes here between high-level options I don’t understand why the authors say the PATH function can be viewed as an inverse? Oh - now I get it. Because it takes an extended n-step transition and generates an action.