Expressiveness in Deep Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) algorithms , such as DQN ( Mnih et al. , 2015 ) , A3C ( Mnih et al. , 2016 ) , DDPG ( Lillicrap et al. , 2015 ) , and TRPO ( Schulman et al. , 2015 ) , have been applied in a range of challenging domains including Atari games ( Mnih et al. , 2015 ; Schaul et al. , 2015 ; Van Hasselt et al. , 2016 ) , robot locomotion tasks ( Schulman et al. , 2015 ; 2017 ) and the game of Go ( Silver et al. , 2016 ; 2017 ) . The combination of RL and high-capacity function approximators such as neural networks holds the promise of automating a wide range of decision making and control tasks . The DRL model often contains two parts . First , a deep neural network is used to extract stateful information from raw signals , e.g , convolutional neural network for images-based games ( Mnih et al. , 2015 ; Vinyals et al. , 2017 ) or recurrent neural network for natural language-based games ( Narasimhan et al. , 2015 ; Zhao & Eskenazi , 2016 ) . Without any confusions , we call such raw signals as observation , call the extracted stateful information as state and consider this deep neural network as state extractor . Given the representation of state , a feedforward neural network is further used to select different actions to maximize the potential rewards . From the above description , it is easy to see that the performance of an RL agent depends on two parts . First , it depends on whether the state extractor is good . With a good state extractor , the representation which is a depiction of the observation will retain necessary information for taking actions . Second , it depends on the accuracy of the policy : whether the feed-forward model can correctly take the optimal action given the state . In this paper , we mainly study the relationship between representations extracted by the state extractor and the performance of the RL agents . We observe that when agents achieve better performances , matrices composed by their representations are approximately higher rank.1 Firstly , we study representations assigned by state extractors in different performed RL agents over all Atari games . We find that representations assigned by the better extractor are more scattered . Furthermore , given different trajectories that lead to high/low rewards as inputs , matrices of representations from the better extractor are always higher rank . Secondly , we find changes of the approximate rank are highly 1Approximately low rank means that most of the singular values for a matrix are close to zero while only little of them have large values , and higher rank corresponds to more large singular values . In this paper , low/high rank refer to approximately low/high rank . consistent with changes of rewards , which also demonstrates the positive correlation between the rank and rewards . These observations motivate us to encourage matrices of representations to be high rank during training . Based on our observations , we formally define expressiveness of the state extractor in reinforcement learning . For given markov decision process ( MDP ) and the initialization state , we can get a realization of the MDP , i.e. , a trajectory . The state extractor extracts representations from observations in this trajectory . Representations compose a matrix , which we call it representation matrix . For given MDP with finite state space , expressiveness in RL for the state extractor model is defined as the rank of the representation matrix . According to the definition , expressiveness in RL is both related to the MDP which generates data and the state extractor model . Based on above experimental studies , we can conclude that higher expressiveness will lead to better performances . As we can see that expressiveness is not easy to calculate because the max operator and the infinite number of columns in representation matrix , so we consider the rank of the representation matrix composed by a mini-batch of representations , which is called empirical representation matrix . Empirical expressiveness in RL is defined as the approximate rank of the empirical representation matrix , i.e. , if some singular values are approximately zero , we regard them as zero . We further propose a novel method ExP ( Expressiveness Promoted ) DRL , which aims improve the expressiveness of the state extractor , so as to promote RL algorithm performances . Based on experimental observations and the definition , the empirical representation matrix is encouraged to be high rank in our method . The idea is implemented by adding a regularization term to the loss , which is computationally efficient and can be applied to multiple kinds of DRL algorithms . We applied our method to A3C ( Mnih et al. , 2016 ) and DQN ( Mnih et al. , 2015 ) . Evaluation results on Atari games show that our method outperforms the baseline on most of games . Furthermore , we also demonstrate that the expressiveness of the state extractor is significantly enhanced by our proposed ExP DRL . 2 STATE EXTRACTOR EXPRESSIVENESS . 2.1 EXPERIMENTAL OBSERVATIONS . In image classification tasks , many papers show that the representation of images ( high-level features from top layers ) contains useful and abstract information for decision makings , e.g. , predicting the categorical labels ( Coates et al. , 2013 ) . In reinforcement learning , such high-level features will be used to find better actions in pursuit of larger reward . However , the representation in reinforcement learning is much harder to learn compared to that in supervised learning problems . In image classification task , e.g. , ImageNet , different images with the same label contain the same item and most of the images in different categories contain different items . This makes the neural network can learn discriminative information effectively and the learned features are good with intra-class compactness and inter-class separability ( Liu et al. , 2016 ) . In reinforcement learning , taking shooter games in Atari as an example , the input images which will be fed into the neural network are similar as most of them contains a group of enemies , bullets , the agent and different objects in the game . Based on similar observations , the neural network has to learn to extract fine-grained local features that contain the positions of enemies , the direction of bullets , the current position of the agent and related objects , which are essential for taking actions . Therefore , to investigate the characteristic of good representations assigned by the state extractor , we compare state extractors in RL models with different performances . We find that representations generated by better state extractor are more scattered compared with worse extractors , and the matrix composed by better representation vectors is higher rank . We observe similar phenomena during training process , which shows that when the matrix formed by representations becomes higher rank , the reward of the RL model become higher . Furthermore , we notice that representations are not always getting more discriminative during training , which also motivates our proposed method . 2.1.1 COMPARISON BETWEEN BETTER AND WORSE STATE EXTRACTORS . Comparison between representations assigned by state extractors in RL models with different performances is shown in Fig . 1 and Table 1 . We let two models with different model size but same number of last hidden layer units play the Atari game Gravitar for 200M frames respectively . After training , the larger model achieves 3050 for the game score , while the smaller model gets 600 points . Besides , in order to exclude the bias of the model size , we also observe representations generated by the large model with its randomly initialized weights , which can only gets 0 game point because points are hard to obtained in this game . We refer to these three models with large trained model , small trained model and large random model respectively . Fig . 1 shows the two-dimensional embedding of the representations in the last hidden layer assigned by three models to game observations in the trajectory played by the large trained model . Plots are generated by using SVD dimension reduction on the matrix composed by representation vectors . Points in the embedding figure of the model with higher final reward are more scattered overall . Observations only have subtle differences on the existence of enemies , the position and direction of the controlled agent . More scattered representations indicate that the better model can distinguish these subtle differences well , and can do better in extracting these fine-grained local features . These scattered representations can benefit policies and thus lead to better performances . In other words , representations assigned by the better state extractor is more discriminative . To get more accurate and solid observations , we list some statistics over all Atari games in Table 1 . For each of the Atari game , representations of a large trained model , a small trained model and a large random model are compared . To study the relation between performances and representations , we select games in which the large trained model performs best and the large random model performs worst . In this way , 39 games among 55 games are picked . Then for each game , we generate three trajectories with different final reward by interacting with the environment using three models . Observations in these three trajectories are sent to three state extractor models respectively , and then 9 matrices composed of generated representations are obtained for every game . We sort singular values of the matrix from largest to smallest , and calculate the smallest ordinal number while the cumulative percentage of singular values is above a certain threshold . We list average smallest ordinal numbers of selected 39 games for threshold 80 % and 90 % in Table 1 . These data demonstrate that given different trajectories and model size , the average smallest ordinal number always positively correlates with the model performance : the smallest ordinal number of the better model is always larger then which of the the worse model . This means that the matrix composed of representations generated by the better model is always higher rank than which generated by the worse model , which suggests that regardless of input trajectories and model size , states assigned by the better extractor is more discriminative and expressive comparing with the worse model , because higher rank corresponds to better discrimination . Representations which are not expressive enough cause troubles for decision making . Therefore , representations should be encouraged to be more discriminative and expressive for obtaining good policies . 2.1.2 TRENDS DURING TRAINING . In order to investigate the found phenomena extensively , and also make the observation more convincing , we study representation changes during training in this section . We train a model to play the Atari game WizardOfWor for 200M frames . Each time the model is updated with a mini-batch of transitions , representation vectors in the last hidden layer are regarded as a matrix , and singular values of this matrix are recorded . We calculate the smallest ordinal number while the cumulative percentage of singular values is above a certain threshold ( i.e. , 80 % ) as last section . Large ordinal number means that the matrix composed by representation vectors is high rank , and corresponding representations are discriminative . Two curves are plotted in Fig.2 . The blue one tracks testing rewards , and the yellow one tracks smallest ordinal numbers . It shows that curve trends of testing rewards and smallest ordinal numbers are highly consistent : when smallest ordinal numbers increase/decrease , rewards also become high/low . Same phenomena on more games can also been seen in Fig . 6 . This consistency demonstrates that states assigned by the better extractor , which form high rank matrices , are more discriminative and expressive , and holding discriminative representations is necessary for learning good policies . Besides these same conclusions mentioned in last section , we also notice that during training , the smallest ordinal number is not always increase . But representations which are not expressive enough cause troubles for decision making . Thus , the expressiveness of state extractors should be promoted ( i.e. , formed matrices should be encourage to be high rank ) during training .

Summary: The authors propose the notion of "expressiveness" of state representation by simply checking the effective rank of the state vectors formed by a sampled trajectory. The authors then propose a regularizer that promotes this expressiveness. Experiments on a large set of Atari games show that this regularizer can improve the performance of reinforcement learning algorithms.