Neuron Activation Analysis for Multi-Joint Robot Reinforcement Learning

1 INTRODUCTION . Convolutional Neural Networks ( CNN ) are well known to demonstrate a strong general feature extraction capability in lower network layers . In these networks feature kernels can not only be visualized , pre-trained general feature extractors can also be reused for efficient network learning . Recent examples propose efficient reusability experimentally for Reinforcement Learning neural networks as well : Networks are pre-trained on similar tasks and continued learning for the goal application . Reusing ( sub ) networks that can be re-assembled for an application never seen before can reduce network training time drastically . A better understanding of uniform or inhomogeneous network structures also improves the evaluation of network performance as well unveils opportunities for the interpretability of networks which is crucial for the application of machine learning algorithms e.g . in industrial scenarios . Finally , methodologies and metrics estimating network intrinsic and inter correlations in artificial neural networks may also enhance the understanding of biological learning . Eickenberg et al . ( 2017 ) could recently demonstrate that layers serving as feature extractors in CNNs could actually be found in the Human Visual Cortex by correlating artificial networks to biological recordings . Successful experiments to re-use end-to-end learned networks for similar tasks leave open whether such networks also self-organize feature extractors or in a dynamical domain motion primitives . Here , we analyze neuron activation in networks in order to investigate activation distribution and mapping between different networks trained on similar robot reaching tasks . In this paper we consider a standard vertical space robot manipulator with variable number of revolute joints as the test setup for target reaching end-to-end Reinforcement Learning ( RL ) experiments . We introduce metrics applied to evaluate individual neuron activation over time and compare activity within individual networks all-to-all ( every neuron is correlated to any other neurons in the network ) and layer wise ( only correlations between networks on the same layer are inspected ) . These metrics are utilized to set up a pruning procedure to maximize the information density in learned neural networks and reduce redundancy as well as unused network nodes . Exploiting these optimization procedure we learn various neural networks with variable dimensions on robot manipulators with two to four joints , representing two to four Degrees of Freedom ( DOF ) . in order to analyze similarities between network activation patterns . As a result we demonstrate experimentally that the introduced pruning process reduces the network size efficiently keeping performance loss in bounds and hereby builds a valid basis for network analysis . We show that the networks trained and iteratively pruned on the robot manipulators form distinct neuron activation . Analyzing neuron activation correlations between different networks of various sizes , mappings between neurons trained on different manipulators can be found . A layer wise interpretation reveals that networks trained for same tasks build similar structures , but we can also discover partially similar structures between networks trained on 3 and 4 joint manipulators . 2 RELATED WORK . The apability of feature extraction in CNNs , alongside with a variety of analysis and visualization tools , serves as a motivation for this work on training , analysis and pruning for networks trained with RL . Analysis methods for CNNs reach from regional based methods , e.g . image occlusion Zeiler & Fergus ( 2014 ) , that aim to expose the region of an image most relevant for classification , to feature based methods , e.g . deconvolution Zeiler & Fergus ( 2014 ) or guided backpropagation Selvaraju et al . ( 2017 ) . Methods combining the described techniques are for example introduced as Grad-CAM in Springenberg et al . ( 2014 ) . These networks demonstrate class discrimination for features of deeper network layers ( Zeiler & Fergus ( 2014 ) ) as a basis to apply such general feature extractors to different applications after pre-training . Pre-trained networks such as ResNet He et al . ( 2016 ) , which has been trained on the ImageNet1 data set , speed up training drastically by initializing CNNs applied for similar tasks . Kopuklu2019 demonstrated that even reusing individual layers in the same network can lead to a performance increase . Recent advances pushed RL agents to reach super human performance in playing Atari video games Bellemare et al . ( 2013 ) Mnih et al . ( 2015 ) , Chess Silver et al . ( 2017 ) and Go Silver et al . ( 2016 ) . These results were extended to cope with continuous action spaces in e.g . Lillicrap et al . ( 2015 ) and demonstrated great performance on highly dynamic multi-actuated locomotion learning tasks such as demonstrated in the NIPS 2017 Learning to Run challenge Kidziński et al . ( 2018 ) . Vuong et al . ( 2019 ) and Eramo et al . ( 2020 ) demonstrate experimentally that knowledge learned by a neural network can be reused for other tasks in order to speed up training and hereby translate modularity concepts from CNNs to RL frameworks . Hierarchical Reinforcement Learning incorporates these ideas , utilizing the concept of subtask solving into neural networks e.g . in Andreas et al . ( 2016 ) for question answering . A successful example of transfer learning to build up a general knowledge base could be demonstrated with RL in Atari games in Parisotto et al . ( 2016 ) . Gaier & Ha ( 2019 ) emphasizes the importance of neural architectures that can perform well even without weight learning . With a main motivation to improve learning efficiency and reduce computational requirements , network pruning is introduced for various network architectures . Early work in LeCun et al . ( 1990 ) utilizes second derivative information as a heuristic to decrease network size , recent work in Livne & Cohen ( 2020 ) introduces network pruning for Deep Reinforcement Learning based on redundancy detection in an iterative process . Li et al . ( 2018 ) 3 EXPERIMENTAL SETUP . In this paper we focus on a robot manipulator with operation limited to a vertical plane . A neural network is trained with end-to-end Reinforcement Learning in order to reach predefined locations in 2D space without prior knowledge of neither robot dynamics nor the environment . Hereby , end-toend refers to a mapping from sensory feedback in terms actual joint positions in cartesian space and the desired goal location to output actions as joint position commands . We apply Deep q-learning , as proposed in Mnih et al . ( 2015 ) , to predict q-values , an action is selected by means to the softmax exploration policy and Gradient descent of the networks weights is handled by the Adam Solver Kingma & Ba ( 2014 ) . For performance reasons our experiments are executed within a simplified simulation environment as shown conceptually in Figure 1 ( right ) , but exemplary behaviors have been successfully trans- ferred to a realistic robotic simulation ( Figure 1 left ) . We simulate robots with 2 to 4 DOF that are implemented as revolute joints restricted to vertical space motions and actuated with PID position controllers . For all experiments , the neural networks originally consist of 6 fully connected hidden layers with ReLU activation functions , but may be reduced in the pruning process we introduce . The network input vector x encodes actual robot joint angles θ̂i as their sine and cosine contribution for every control step t ( control cycle time of 50ms ) as well as the desired goal position in cartesian coordinates [ x∗ , y∗ ] as x ( t ) = [ sin ( θ̂ ( t ) 1 ) cos ( θ̂ ( t ) 1 ) ... sin ( θ̂ ( t ) n ) cos ( θ̂ ( t ) n ) x∗ y∗ ] T . ( 1 ) The output layer contains 3n neurons as the action of every individual joint i is quantized into the three change of motion states { 1 , −1 , 0 } as forward , backward and no motion for each joint with joint angle changes of±0.05rad . The goal state of an agent is a randomly instantiated 2D location to be reached with the robot finger tip in max . 60 control steps , each representing 50ms . The distance between the goal position p∗ and the tip p̂ is mapped into [ 0 , 1 ] and squared to serve as the reward function r ( ti ) : = ( 1 |p̂ ( ti ) −p∗|L2+1 ) 2 . All network results that are presented passed a validation test consisting of 300 test episodes . This test also serves as the pruning baseline : The probability of a type two error for reaching the final reward threshold r̄ = 0.9 with an accuracy ρ̄ = 0.9 lies bellow significance α = 0.05 on the test data . 4 NEURON ACTIVATION ANALYSIS . We first analyze individual neuron activation inside multiple neural networks trained on the introduced target reaching robotic manipulator . This initial analysis servers as a baseline for pruning and projection evaluation , therefore we study only 3 joint robotic manipulators in depth before we investigate a comparison of different kinematic structures . We define a distance metric between neurons that is based on the neuron activation history in scope of every episode in order to account for the dynamics in motion trajectory learning . All neuron activation values over the course of an episode are collected in a vector z ( E ) ni for every neuron ni of the network in Episode E. Utilizing the linearity of applied ReLU activation functions we normalize this activation in range [ 0 , 1 ] in reference to the maximum value attained . For a set of sample episodes E , representing a set of potential robot actions , we define the distance of neurons ni and nj as d ( ni , nj ) : = 1 |E| ∑ E∈E ∣∣∣∣∣z ( E ) niZni − z ( E ) nj Znj ∣∣∣∣∣ L2 , ( 2 ) with z ( E ) ni ∈ RT≥0 denoting the vector containing activation series of neuron ni in episode E and Zni ∈ R > 0 the maximum activation of ni in all episodes E . For a layer wise analysis Equation 2 is adapted accordingly , only considering distances to neurons that belong to the same layer . The upper triangular matrix of a distance matrix D holds all values d ( ni , nj ) with i ? = j . The density distribution of neuron distances can be approximated by collecting all values in the upper triangular matrices of D. Additionally , hierarchical clustering as described in Hastie et al . ( 2009 ) is applied to individual network layers in order to reveal neuron groups that show similar activation behavior . We form groups that minimize the mean cluster distance D ( Cl ) of contained neurons as D ( Cl ) : = 1 |Cl| ( |Cl| − 1 ) ∑ nil∈Cl ∑ njl∈Cl\ { nil } d ( nil , njl ) . ( 3 ) for neuron cluster C of layer l. We conduct an experiment with a set of M = 20 networks ( 48 neurons per hidden layer ) , for the three joint manipulation task . A reference set of untrained networks with identical structure is initialized by Xavier initialization Glorot & Bengio ( 2010 ) . Neuron distances are averaged from a set ofm = 500 sample episodes . The distance distribution in randomized networks forms a bell-shaped distribution globally as well as layer wise ( Figure 2 , top ) . However , the all-to-all distribution of trained networks primarily indicate a lower standard deviation and mean compared to random networks , with a slight distortion at high distances . Layer-wise analysis reveals that these higher distance scores occur increasingly on network layers closer to the output , in particular in the second half of layers . In contrast , lower layers demonstrate close to normal distributions . Clustering reveals a variety in distances for all layers in untrained randomly initialized networks ( Figure 2 bottom ) which is kept on the first layer only in trained networks . In particular on the middle layers clusters with low distances emerge during training . The intrinsic network analysis depicts successful training that visibly changes the neuron activation characteristics which highly depends on the location inside the network .

Summary: The paper presents a technique to compare networks trained to solve similar tasks trained in different context. The considered task is reaching with a robotic planar arm; the considered context is varied varying the robot degrees of freedom. The goal of the paper is to find correlations across neural activity patterns across networks trained to solve the same task in different contexts.