Expressiveness in Deep Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) algorithms , such as DQN ( Mnih et al. , 2015 ) , A3C ( Mnih et al. , 2016 ) , DDPG ( Lillicrap et al. , 2015 ) , and TRPO ( Schulman et al. , 2015 ) , have been applied in a range of challenging domains including Atari games ( Mnih et al. , 2015 ; Schaul et al. , 2015 ; Van Hasselt et al. , 2016 ) , robot locomotion tasks ( Schulman et al. , 2015 ; 2017 ) and the game of Go ( Silver et al. , 2016 ; 2017 ) . The combination of RL and high-capacity function approximators such as neural networks holds the promise of automating a wide range of decision making and control tasks . The DRL model often contains two parts . First , a deep neural network is used to extract stateful information from raw signals , e.g , convolutional neural network for images-based games ( Mnih et al. , 2015 ; Vinyals et al. , 2017 ) or recurrent neural network for natural language-based games ( Narasimhan et al. , 2015 ; Zhao & Eskenazi , 2016 ) . Without any confusions , we call such raw signals as observation , call the extracted stateful information as state and consider this deep neural network as state extractor . Given the representation of state , a feedforward neural network is further used to select different actions to maximize the potential rewards . From the above description , it is easy to see that the performance of an RL agent depends on two parts . First , it depends on whether the state extractor is good . With a good state extractor , the representation which is a depiction of the observation will retain necessary information for taking actions . Second , it depends on the accuracy of the policy : whether the feed-forward model can correctly take the optimal action given the state . In this paper , we mainly study the relationship between representations extracted by the state extractor and the performance of the RL agents . We observe that when agents achieve better performances , matrices composed by their representations are approximately higher rank.1 Firstly , we study representations assigned by state extractors in different performed RL agents over all Atari games . We find that representations assigned by the better extractor are more scattered . Furthermore , given different trajectories that lead to high/low rewards as inputs , matrices of representations from the better extractor are always higher rank . Secondly , we find changes of the approximate rank are highly 1Approximately low rank means that most of the singular values for a matrix are close to zero while only little of them have large values , and higher rank corresponds to more large singular values . In this paper , low/high rank refer to approximately low/high rank . consistent with changes of rewards , which also demonstrates the positive correlation between the rank and rewards . These observations motivate us to encourage matrices of representations to be high rank during training . Based on our observations , we formally define expressiveness of the state extractor in reinforcement learning . For given markov decision process ( MDP ) and the initialization state , we can get a realization of the MDP , i.e. , a trajectory . The state extractor extracts representations from observations in this trajectory . Representations compose a matrix , which we call it representation matrix . For given MDP with finite state space , expressiveness in RL for the state extractor model is defined as the rank of the representation matrix . According to the definition , expressiveness in RL is both related to the MDP which generates data and the state extractor model . Based on above experimental studies , we can conclude that higher expressiveness will lead to better performances . As we can see that expressiveness is not easy to calculate because the max operator and the infinite number of columns in representation matrix , so we consider the rank of the representation matrix composed by a mini-batch of representations , which is called empirical representation matrix . Empirical expressiveness in RL is defined as the approximate rank of the empirical representation matrix , i.e. , if some singular values are approximately zero , we regard them as zero . We further propose a novel method ExP ( Expressiveness Promoted ) DRL , which aims improve the expressiveness of the state extractor , so as to promote RL algorithm performances . Based on experimental observations and the definition , the empirical representation matrix is encouraged to be high rank in our method . The idea is implemented by adding a regularization term to the loss , which is computationally efficient and can be applied to multiple kinds of DRL algorithms . We applied our method to A3C ( Mnih et al. , 2016 ) and DQN ( Mnih et al. , 2015 ) . Evaluation results on Atari games show that our method outperforms the baseline on most of games . Furthermore , we also demonstrate that the expressiveness of the state extractor is significantly enhanced by our proposed ExP DRL . 2 STATE EXTRACTOR EXPRESSIVENESS . 2.1 EXPERIMENTAL OBSERVATIONS . In image classification tasks , many papers show that the representation of images ( high-level features from top layers ) contains useful and abstract information for decision makings , e.g. , predicting the categorical labels ( Coates et al. , 2013 ) . In reinforcement learning , such high-level features will be used to find better actions in pursuit of larger reward . However , the representation in reinforcement learning is much harder to learn compared to that in supervised learning problems . In image classification task , e.g. , ImageNet , different images with the same label contain the same item and most of the images in different categories contain different items . This makes the neural network can learn discriminative information effectively and the learned features are good with intra-class compactness and inter-class separability ( Liu et al. , 2016 ) . In reinforcement learning , taking shooter games in Atari as an example , the input images which will be fed into the neural network are similar as most of them contains a group of enemies , bullets , the agent and different objects in the game . Based on similar observations , the neural network has to learn to extract fine-grained local features that contain the positions of enemies , the direction of bullets , the current position of the agent and related objects , which are essential for taking actions . Therefore , to investigate the characteristic of good representations assigned by the state extractor , we compare state extractors in RL models with different performances . We find that representations generated by better state extractor are more scattered compared with worse extractors , and the matrix composed by better representation vectors is higher rank . We observe similar phenomena during training process , which shows that when the matrix formed by representations becomes higher rank , the reward of the RL model become higher . Furthermore , we notice that representations are not always getting more discriminative during training , which also motivates our proposed method . 2.1.1 COMPARISON BETWEEN BETTER AND WORSE STATE EXTRACTORS . Comparison between representations assigned by state extractors in RL models with different performances is shown in Fig . 1 and Table 1 . We let two models with different model size but same number of last hidden layer units play the Atari game Gravitar for 200M frames respectively . After training , the larger model achieves 3050 for the game score , while the smaller model gets 600 points . Besides , in order to exclude the bias of the model size , we also observe representations generated by the large model with its randomly initialized weights , which can only gets 0 game point because points are hard to obtained in this game . We refer to these three models with large trained model , small trained model and large random model respectively . Fig . 1 shows the two-dimensional embedding of the representations in the last hidden layer assigned by three models to game observations in the trajectory played by the large trained model . Plots are generated by using SVD dimension reduction on the matrix composed by representation vectors . Points in the embedding figure of the model with higher final reward are more scattered overall . Observations only have subtle differences on the existence of enemies , the position and direction of the controlled agent . More scattered representations indicate that the better model can distinguish these subtle differences well , and can do better in extracting these fine-grained local features . These scattered representations can benefit policies and thus lead to better performances . In other words , representations assigned by the better state extractor is more discriminative . To get more accurate and solid observations , we list some statistics over all Atari games in Table 1 . For each of the Atari game , representations of a large trained model , a small trained model and a large random model are compared . To study the relation between performances and representations , we select games in which the large trained model performs best and the large random model performs worst . In this way , 39 games among 55 games are picked . Then for each game , we generate three trajectories with different final reward by interacting with the environment using three models . Observations in these three trajectories are sent to three state extractor models respectively , and then 9 matrices composed of generated representations are obtained for every game . We sort singular values of the matrix from largest to smallest , and calculate the smallest ordinal number while the cumulative percentage of singular values is above a certain threshold . We list average smallest ordinal numbers of selected 39 games for threshold 80 % and 90 % in Table 1 . These data demonstrate that given different trajectories and model size , the average smallest ordinal number always positively correlates with the model performance : the smallest ordinal number of the better model is always larger then which of the the worse model . This means that the matrix composed of representations generated by the better model is always higher rank than which generated by the worse model , which suggests that regardless of input trajectories and model size , states assigned by the better extractor is more discriminative and expressive comparing with the worse model , because higher rank corresponds to better discrimination . Representations which are not expressive enough cause troubles for decision making . Therefore , representations should be encouraged to be more discriminative and expressive for obtaining good policies . 2.1.2 TRENDS DURING TRAINING . In order to investigate the found phenomena extensively , and also make the observation more convincing , we study representation changes during training in this section . We train a model to play the Atari game WizardOfWor for 200M frames . Each time the model is updated with a mini-batch of transitions , representation vectors in the last hidden layer are regarded as a matrix , and singular values of this matrix are recorded . We calculate the smallest ordinal number while the cumulative percentage of singular values is above a certain threshold ( i.e. , 80 % ) as last section . Large ordinal number means that the matrix composed by representation vectors is high rank , and corresponding representations are discriminative . Two curves are plotted in Fig.2 . The blue one tracks testing rewards , and the yellow one tracks smallest ordinal numbers . It shows that curve trends of testing rewards and smallest ordinal numbers are highly consistent : when smallest ordinal numbers increase/decrease , rewards also become high/low . Same phenomena on more games can also been seen in Fig . 6 . This consistency demonstrates that states assigned by the better extractor , which form high rank matrices , are more discriminative and expressive , and holding discriminative representations is necessary for learning good policies . Besides these same conclusions mentioned in last section , we also notice that during training , the smallest ordinal number is not always increase . But representations which are not expressive enough cause troubles for decision making . Thus , the expressiveness of state extractors should be promoted ( i.e. , formed matrices should be encourage to be high rank ) during training .

Summary: This paper proposes a simple regularizer for RL which encourages the state representations learned by neural networks to be more discriminative across different observations. The main idea is to (implicitly) measure the rank of the matrix which is constructed from a sequence of observations and state feature vectors and encourage the rank to be high. This paper introduces three different objectives to implement the same idea (increasing the rank of the matrix). The experimental results on Atari games show that this regularizer improves A3C on most of the games and show that the learned representations with the proposed regularizer has a high rank compared to the baseline.