Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation

Deep Reinforcement Learning has managed to achieve state-of-the-art results in learning control policies directly from raw pixels . However , despite its remarkable success , it fails to generalize , a fundamental component required in a stable Artificial Intelligence system . Using the Atari game Breakout , we demonstrate the difficulty of a trained agent in adjusting to simple modifications in the raw image , ones that a human could adapt to trivially . In transfer learning , the goal is to use the knowledge gained from the source task to make the training of the target task faster and better . We show that using various forms of fine-tuning , a common method for transfer learning , is not effective for adapting to such small visual changes . In fact , it is often easier to re-train the agent from scratch than to fine-tune a trained agent . We suggest that in some cases transfer learning can be improved by adding a dedicated component whose goal is to learn to visually map between the known domain and the new one . Concretely , we use Unaligned Generative Adversarial Networks ( GANs ) to create a mapping function to translate images in the target task to corresponding images in the source task . These mapping functions allow us to transform between various variations of the Breakout game , as well as between different levels of a Nintendo game , Road Fighter . We show that learning this mapping is substantially more efficient than re-training . A visualization of a trained agent playing Breakout and Road Fighter , with and without the GAN transfer , can be seen in https : //streamable.com/msgtm and https : //streamable.com/5e2ka . 1 INTRODUCTION . Transferring knowledge from previous occurrences to new circumstances is a fundamental human capability and is a major challenge for deep learning applications . A plausible requirement for artificial general intelligence is that a network trained on one task can reuse existing knowledge instead of learning from scratch for another task . For instance , consider the task of navigation during different hours of the day . A human that knows how to get from one point to another on daylight will quickly adjust itself to do the same task during night time , while for a machine learning system making a decision based on an input image it might be a harder task . That is because it is easier for us to make analogies between similar situations , especially in the things we see , as opposed to a robot that does not have this ability and its knowledge is based mainly on what it already saw . Deep reinforcement learning has caught the attention of researchers in the past years for its remarkable success in achieving human-level performance in a wide variety of tasks . One of the field ’ s famous achievements was on the Atari 2600 games where an agent was trained to play video games directly from the screen pixels and information received from the game ( Mnih et al. , 2013 ) . However , this approach depends on interacting with the environment a substantial number of times during training . Moreover , it struggles to generalize beyond its experience , the training process of a new task has to be performed from scratch even for a related one . Recent works have tried to overcome this inefficiency with different approaches such as , learning universal policies that can generalize between related tasks ( Schaul et al. , 2015 ) , as well as other transfer approaches ( Fernando et al. , 2017 ; Rusu et al. , 2016 ) . In this work , we first focus on the Atari game Breakout , in which the main concept is moving the paddle towards the ball in order to maximize the score of the game . We modify the game by introducing visual changes such as adding a rectangle in the middle of the image or diagonals in the background . From a human perspective , it appears that making visual changes that are not significant to the game ’ s dynamics should not influence the score of the game , a player who mastered the original game should be able to trivially adapt to such visual variants . We show that the agent fails to transfer . Furthermore , fine-tuning , the main transfer learning method used today in neural networks , also fails to adapt to the small visual change : the information learned in the source task does not benefit the learning process of the very related target task , and can even decelerate it . The algorithm behaves as if these are entirely new tasks . Our second focus is attempting to transfer agent behavior across different levels of a video game : can an agent trained on the first level of a game use this knowledge and perform adequately on subsequent levels ? We explore the Nintendo game Road Fighter , a car racing game where the goal is to finish the track before the time runs out without crashing . The levels all share the same dynamics , but differ from each other visually and in aspects such as road width . Similar to the Breakout results , an agent trained to play the first level fails to correctly adapt its past experience , causing the learned policy to completely fail on the new levels . To address the generalization problem , we propose a zero-shot generalization approach , in which the agent learns to transfer between related tasks by learning to visually map images from the target task back to familiar corresponding images from the source task . Such mapping is naturally achieved using Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , one of the most popular methods for the image-to-image translation that is being used in computer vision tasks such as style transfer ( Zhu et al. , 2017 ; Kim et al. , 2017 ) , object transfiguration ( Zhou et al. , 2017 ) , photo enhancement ( Ledig et al. , 2016 ) and more recently , video game level generation ( Volz et al. , 2018 ) . In our setup , it is not realistic to assume paired images in both domains , calling for the use of Unaligned GANs ( Liu et al. , 2017 ; Zhu et al. , 2017 ; Kim et al. , 2017 ; Yi et al. , 2017 ) . Using this approach we manage to transfer between similar tasks with no additional learning . Contributions This work presents three main contributions . First , in Section 2 , we demonstrate how an agent trained with deep reinforcement learning algorithms fails to adapt to small visual changes , and that the common transfer method of fine-tuning fails as well . Second , in Section 3 , we propose to separate the visual mapping from the game dynamics , resulting in a new transfer learning approach for related tasks based on visual input mapping . We evaluate this approach on Breakout and Road Fighter , and present the results comparing to different baselines . We show that our visual transfer approach is much more sample efficient then the alternatives . Third , in section 5 , we suggest an evaluation setup for unaligned GAN architectures , based on their achieved performance on concrete down-stream tasks . 2 GENERALIZATION FAILURES OF DEEP RL . Many Breakout variations can be constructed that involve the same dynamics . The main idea is to make modifications that are not critical for a human playing the game but are for the algorithm that relies on visual inputs . We demonstrate the difficulty of deep reinforcement learning to generalize using 4 types of modifications as presented in Figure 1 . 2.1 SETUP . For all the experiments in this section forward we use the Asynchronous Advantage Actor-Critic ( A3C ) algorithm ( Mnih et al. , 2016 ) , taking advantage of being faster than Deep Q-Network ( DQN ) ( Mnih et al. , 2013 ) . The A3C learns the policy and the state-value function using parallel actor-learners exploring different policies for the acceleration and stability of the training . We rescale the image to 80× 80 and keep the RGB colors for more realistic images . We use 32 actor learners , a discount rate of 0.99 , learning rate of 0.0001 , 20-step returns , and entropy regularization weight of 0.01 . The A3C variation we choose is the LSTM-A3C network . We use the standard high-performance architecture implemented in ( Kostrikov , 2018 ) . 2.2 TRANSFER-LEARNING VIA FINE-TUNING . The setup mentioned in 2.1 successfully trains on Breakout , reaching a score of over 400 points . However , when a network trained on the original game is presented with the game variants , it fails completely , reaching to a maximum score of only 3 points . This shows that the network does not necessarily learn the game ’ s concepts and heavily relies on the images it receives . The common approach for transferring knowledge across tasks is fine-tuning . We experiment with common techniques used in deep learning models . In each setting , we have a combination of frozen and fine-tuned layers ( Partial/Full ) as well as layers that are initialized with the target ’ s parameters and layers that are initialized with random values ( Random ) . Our settings are inspired by ( Yosinski et al. , 2014 ) . We train each one of the tasks ( before and after the transformation ) for 60 million frames , and our evaluation metric is the total reward the agents collect in an episode averaged by the number of episodes , where an episode ends when the game is terminated or when a number of maximum steps is reached . We periodically compute the average during training . We consider the following settings : • From-Scratch : The game is being trained from scratch on the target game . • Full-FT : All of the layers are initialized with the weights of the source task and are fine-tuned on the target task . • Random-Output : The convolutional layers and the LSTM layer are initialized with the weights of the source task and are fine-tuned on the target task . The output layers are initialized randomly . • Partial-FT : All of the layers are initialized with the weights of the source task . The three first convolutional layers are kept frozen , and the rest are fine-tuned on the target task . • Partial-Random-FT : The three first convolutional layers are initialized with the weights of the source task and are kept frozen , and the rest are initialized randomly . 2.3 RESULTS . The results presented in Figure 2 show a complete failure of all the fine-tuning approaches to transfer to the target tasks . In the best scenarios the transfer takes just as many epochs as training from scratch , while in other cases starting from a trained network makes it harder for the network to learn the target task . As the graphs show , some of the modification influenced more than others . For example , Figure 2a shows that adding a simple rectangle can be destructive for a trained agent : while training from scratch consistently and reliably achieves scores over 300 , the settings starting from a trained agent struggle to pass the 200 points mark within the same number of iterations , and have a very high variance . We noticed that during training the agent learns a strategy to maximize the score with a minimum number of actions . None of the experiments we performed showed better results when the layers in the network were fine-tuned , and some showed negative transfer which is a clear indication of an overfitting problem . The A3C model learned the detail and noise in the training data to the extent that it negatively impacted the performance of the model on new data . Our results and conclusions drawn from them are consistent with the results shown when a similar approach was used on Pong ( Rusu et al. , 2016 ) . In addition to Breakout , we also experimented transfer between the first and advanced level of Road Fighter , where the backgrounds change but the dynamics remains the same . This experiments resulted with 0 points on each of the levels , a complete failure of the agent to re-use the driving techniques learned on the first levels on the next ones .

Summary: This paper proposes to improve the sample efficiency of transfer learning for Deep RL by mapping a new visual domain (target) onto the training one (source) using GANs. First, a deep RL policy is trained on a source domain (e.g., level 1 of the Atari Road Fighter game). Second, a GAN (e.g. UNIT or CycleGAN) is trained for unsupervised domain adaptation from target images (e.g., level 2 of Road Fighter) to source ones. Third, the policy learned in the source domain is applied directly on the GAN-translated target domain. The experimental evaluation uses two Atari games: i) transfer from Breakout to Breakout with static visual distractors inpainted on the screen, ii) from one Road Fighter level to others. Results suggest that this transfer learning approach requires less images than retraining from scratch in the new domain, including when fine-tuning does not work.