Insights on Visual Representations for Embodied Navigation Tasks

1 INTRODUCTION . Recent advancements in deep reinforcement learning ( deep RL ) have allowed for the creation of systems that are able to out-perform human experts on a variety of different games such as Chess , Go , Dota2 , and Starcraft2 ( Vinyals et al. , 2019 ; Silver et al. , 2017 ; Tian et al. , 2019 ) . These advances have heavily relied on sample-inefficient algorithms that require significant amounts of task-specific training episodes , making them computationally expensive to run . Furthermore , deep RL has been found to be capable of overfitting to the training task , even for complex problems ( Zhang et al. , 2018 ) , or failures when the environment is altered ( even if this , in turn , simplifies the task Ruderman et al . ( 2019 ) ) . These observations call into question whether representations learned with one training task will be reusable for novel tasks . The generality and reuse-ability of representations is a desirable and powerful property as it allows knowledge to be transferred between tasks and can help alleviate a lack of data . In the regime of supervised learning , it is well known that deep neural networks are capable of overfitting on tasks and memorizing random labels ( Zhang et al. , 2016 ) , making it reasonable to expect that representations would be highly tuned to their training task . However , many have shown that representations trained for one task perform well for other tasks , both as an initialization for fine-tuning and as a static featureextractor ( Girshick , 2015 ; Ren et al. , 2015 ; Anderson et al. , 2018b ; Conneau et al. , 2017 ; Kornblith et al. , 2019 ) . Resolving this discrepancy is an area of much debate and active research ( Neyshabur et al. , 2019 ; 2015 ; Keskar et al. , 2017 ; Golowich et al. , 2018 ; Arora et al. , 2018 ; Morcos et al. , 2018b ) . Reusing representations provides a promising avenue for the emerging field of training virtual robots in simulation before transfer learned skills to reality . There have been a number of recent works proposing to train robots as Embodied Agents in simulated environments with the ultimate goal of transferring agents learned in simulation to reality ( Gupta et al. , 2017 ; Zhu et al. , 2017 ; Anderson et al. , 2018c ; Das et al. , 2018 ; Gordon et al. , 2018 ; Wijmans et al. , 2019 ; Savva et al. , 2019 ) . The ability to reuse representations for new tasks and in new environments is of particular concern to the goal of transferring embodied agents from simulation to reality . Once in the real world , an agent should be capable of learning new tasks – such as finding new objects or handling new questions – and be able to cope with the non-stationarity of a changing world . Thus , we seek to answer the following question : Do different embodied navigation tasks induce different visual representations ? Contributions . We study our primary question in the context of the task of Object Navigation ( ObjectNav ) , e.g . ‘ Go to the X ’ . We define two different embodied tasks by constructing disjoint splits of target objects , allowing us to understand the exact differences between our tasks . We then adapt the methodologies proposed in Raghu et al . ( 2017 ) and Morcos et al . ( 2018a ) to examine the impact the task has on the visual representation . We first perform our experiments using SqueezeNet1.2 ( Iandola et al. , 2016 ) as parameter efficient networks would be a good choice for embodied agents deployed on real robots . We find that , surprisingly , differences in the task do not lead to a measurable effect on the visual representation . We leverage this knowledge to show that visual representations trained for one tasks are useful for learning another , and , surprisingly , allow for more sample efficient learning . Next , we design a special case where the different tasks constrain the agent to spatially disjoint locations in the environment , such that the agents should explore different areas during training , resulting in different representations and providing insight on how task independent visual representations emerge . We demonstrate that this task dependence negatively impacts the ability to re-use the representation for new tasks . We then consider how our choice of CNN impacted our findings by performing our analysis on a second CNN , a version of ResNet50 ( He et al. , 2016 ) modified to have a comparable number of parameters to SqueezeNet1.2 , and find similar conclusions for the non-special case set of tasks . For the spatially disjoint tasks however , the modified ResNet50 learns reasonably task-agnostic representations , in contrast to our results for SqueezeNet , suggesting that ResNets contain inductive biases that encourage more task-agnostic representations . Finally , to evaluate the extent to which these results are environment dependent , we generalize our analysis to transfer across multiple permutations of an environment and demonstrate representations learned in one permutation of an environment are effective for the other permutations . 2 RELATED WORK . Representation analysis . Analyzing the representations of deep neural networks has been the subject of many works . Initial works focused on analyzing individual neurons ( Li et al. , 2016 ; Zeiler & Fergus , 2014 ; Bau et al. , 2017 ; Arpit et al. , 2017 ; Morcos et al. , 2018b ) . In this work , however , we examine the entire representation . Our closest related works , Raghu et al . ( 2017 ) ; Morcos et al . ( 2018a ) , propose methods to examine the entire representation of neural networks in the context of standard image classification tasks . We adopt their analysis tools and utilize them to analyze neural networks in the context of embodied-vision tasks and reinforcement learning . See Section 3.2 for a more detailed discussion of the benefits of these methods . Reward-free reinforcement learning . Transfer of knowledge and representations is a paradigm commonly used in task-agnostic and reward-free reinforcement learning . The goal of this paradigm is to allow the agent to interact with its environment such that it gains general knowledge , thereby allowing it to learn downstream tasks with less samples . These works provide the agent with a reward signal such that it will explore its environment ( or state-space ) . This can be formulated from an information theoretic standpoint to provide intrinsic motivation Jung et al . ( 2011 ) ; Eysenbach et al . ( 2018 ) ; Gregor et al . ( 2016 ) ; Haarnoja et al . ( 2017 ) . Others provide a more direct signal in the form of exploration based rewards Burda et al . ( 2018 ) ; Savinov et al . ( 2018 ) . We differ from these works by using representations learned via task-driven reinforcement learning directly for a different task . Transfer Learning . Transfer learning seeks to transfer knowledge between a domain with labeled data to another domain ( Pan & Yang , 2009 ; Luo et al. , 2017 ; Oquab et al. , 2014 ) . Transfer learning has also been studied in the context of reinforcement learning by designing specific objectives or model structures such that knowledge can be transferred between two tasks Oh et al . ( 2017 ) ; Bacon et al . ( 2017 ) ; Taylor & Stone ( 2009 ) . We do not use any specific architecture or objectives and examine task dependence of vanilla architectures . 3 APPROACH . In this section , we outline our approach for answering our core question by describing the task we examine and the method for comparing representations we leverage . 3.1 EXPERIMENTAL SETUP . Task . We examine the task of Object Goal Navigation ( ObjectNav ) due to its reliance on both semantic and spatial understanding . In ObjectNav , an agent is given a token describing an object in the environment , such as fridge , and then must navigate through the environment until it finds a good view of the fridge and calls the stop action . To avoid under-specification of the task , we restrict target objects to have at most two instances for a given class . Note that each target object is specified uniquely by its object ID . The terminal reward given is proportional to how much of the target object is in the agent ’ s field of view . At every time-step , a shaped reward proportional to the agents progress towards the target object is also provided . See the supplementary for more details . Environment . We use the extreme high-fidelity reconstructions in the Replica Dataset ( Straub et al. , 2019 ) and simlate agents utilizing AI Habitat ( Savva et al. , 2019 ) . We utilize these environment so that our analysis will be more applicable to the ultimate goal of agents operating in reality . See Fig . 1a for a top-down view of an environment and the supplement for example agent views . Agent . The agent has 4 primitive actions , move_forward , which moves 0.25 meters forward ; turn_left and turn_right ( which turn 10 degrees left and right , respectively ) , and stop which signals that the agent believes it has completed its task . At every time-step , the agent receives an egocentric RGB image and the token specifying the target object . Policy . We parameterize our agent with 3 components . A visual encoder , a target encoder , and a recurrent policy . The visual encoder utilizes SqueezeNet1.2 ( Iandola et al. , 2016 ) as the backbone architecture as its combination of parameter efficiency and representational power is a logical choice for embodied agents deployed on real robots . The target encoding is a 128 dimensional vector that is learn-able and unique for each target object . The policy consists of a GRU ( Cho et al. , 2014 ) followed by 2 fully connected layers . See the supplementary for more details . Note that the vast majority ( ∼80 % ) of the learnable parameters are in the visual encoder . This is key to our analysis as other the network is able to perform the task with a frozen randomly initialized visual encoder . Training . We use Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) with Generalized Advantage Estimation ( Schulman et al. , 2015 ) and the Adam optimizer ( Kingma & Ba , 2014 ) to train our agent . We train for 15,000 rollouts ( ∼ 61× 106 steps ) to ensure converge across different random seeds . See the supplementary for more details . 3.2 MEASURING THE SIMILARITY OF REPRESENTATIONS . A perhaps straight-forward approach to measuring the similarity of representations would be to simply measure the distance ( e.g. , Euclidean or cosine ) between their representations of the same inputs . However , this approach is ill-suited to neural networks . Consider the following toy example : For a set of inputs X , suppose that function f produces a representation that is uniform on the N-ball and define f ′ = Af for an affine transform A . A simple distance calculation ( or alternatively , dimensionality reduction and clustering ) would report a high distance between the two representations . Accounting for affine transformations is important when analyzing neural networks as , for any given layer , one can apply any affine transformation to the activations and the inverse to the next layer ’ s weights without changing the network . Given two neural networks trained in the exact same way modulo the random seed , there is no reason why their representations would be aligned despite computing very similar ( if not exact the same ) functions ( Li et al. , 2016 ) . Instead , we follow the approach of Raghu et al . ( 2017 ) ; Morcos et al . ( 2018a ) to compare the representations of two deep neural networks . Given two neural networks , A and B , and a set of N inputs , Raghu et al . ( 2017 ) ; Morcos et al . ( 2018a ) compare the representations at layer L of both networks by 1 ) extracting the neuron activation matrix , X , of both networks – where Xi , j is the activation of the ith neuron on the jth input ; and 2 ) compute the distance between the neuron activation matrices using Canonical Correlation Analysis ( CCA ) , a classic statistical technique ( Hotelling , 1936 ) . CCA finds a basis which maximizes the correlation between two matrices and then computes the correlation in that basis , thereby account for any affine transformations between two representations . ( a ) Top-down view in the environment . Circles denote the location of all target objects . Coloring denotes which target set objects are in for the spatially disjoint split : blue for A , red for B , and green is unused . 1- [ s =2 ] 2- [ s =2 ] 3 4 5- [ s =2 ] 6 7 8- [ s =2 ] 9 10 11 12 13 14 15 Layer 0.0 0.1 0.2 0.3 0.4 D pw cc a Randomly Split – SqueezeNet A B A-B ( b ) PWCCA results of comparing networks trained on different embodied task . Down-sampling layers are marked with [ s=2 ] . Shading around the line corresponds to a 95 % confidence interval calculated via empirical bootstrapping . Figure 1 It is worth noting that CCA ( and variants ) do not capture the “ usefulness ” of representation to the downstream task . We follow the technique proposed by Morcos et al . ( 2018a ) to account for differing numbers of noise dimensions between representations . This method weights CCA correlation coefficients by the amount of variance each CCA direction explains in the real data . Given each of the CCA directions hi and correlation coefficients ρi , Morcos et al . ( 2018a ) first computes the projection coefficients αi = ∑ k |〈di , Xk〉| and then computes 1 minus the weighted average of the correlation coefficients , Dpwcca = 1.0− 1∑ k αk ∑ k αkρk , as the distance between representations .

Summary: This article studies the similarities between the learned representations for different tasks when trained using reinforcement learning algorithms. The ultimate question that this study tries to answer is an interesting one. Namely, how much can representations learned by training on one task be beneficial for learning other tasks? A high interdependence between the representations can lead to a more successful transfer of knowledge between tasks.