Neuron Activation Analysis for Multi-Joint Robot Reinforcement Learning
Authors: Benedikt Feldotto
Heiko Lengenfelder
Alois Knoll
Venue:  (2021)
Keywords: Reinforcement Learning
Machine Learning
Robot Motion Learning
DQN
Robot Manipulator
Target Reaching
Network Pruning

Abstract: Recent experiments indicate that pre-training of end-to-end Reinforcement Learning neural networks on general tasks can speed up the training process for specific robotic applications. However, it remains open if these networks form general feature extractors and a hierarchical organization that are reused as apparent e.g. in Convolutional Neural Networks. In this paper we analyze the intrinsic neuron activation in networks trained for target reaching of robot manipulators with increasing joint number in a vertical plane. We analyze the individual neuron activity distribution in the network, introduce a pruning algorithm to reduce network size keeping the performance, and with these dense network representations we spot correlations of neuron activity patterns among networks trained for robot manipulators with different joint number. We show that the input and output network layers have more distinct neuron activation in contrast to inner layers. Our pruning algorithm reduces the network size significantly, increases the distance of neuron activation while keeping a high performance in training and evaluation. Our results demonstrate that neuron activity can be mapped among networks trained for robots with different complexity. Hereby, robots with small joint difference show higher layer-wise projection accuracy whereas more different robots mostly show projections to the first layer.

Under review as a conference paper at ICLR 2021
NEURON ACTIVATION ANALYSIS IN MULTI-JOINT ROBOT REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent experiments indicate that pre-training of end-to-end Reinforcement Learn- ing neural networks on general tasks can speed up the training process for specific robotic applications. However, it remains open if these networks form general feature extractors and a hierarchical organization that are reused as apparent c.g. in Convolutional Neural Networks. In this paper we analyze the intrinsic neuron activation in networks trained for target reaching of robot manipulators with in- creasing joint number in vertical plane. We analyze the individual neuron activ- ity distribution in the network, introduce a pruning algorithm to reduce network size keeping the performance, and with these dense network representations we spot correlations of neuron activity patterns among networks trained for robot ma- nipulators with different joint number. We show that the input and output network layers have more distinct neuron activation in contrast to inner layers. Our pruning algorithm reduces the network size significantly, increases the distance of neuron activation while keeping high performance in training and evaluation. Our re- sults demonstrate that neuron activity can be mapped among networks trained for robots with different complexity. Hereby, robots with small joint difference show higher layer-wist wise projection accuracy whereas more different robots mostly show projections to the first layer.
INTRODUCTION
Convolutional Neural Networks (CNN) are well known demonstrate strong general feature traction capability in lower network layers. In these networks feature kernels can not only be visual- ized, pre-trained general feature extractors can also reused for efficient network learning. Recent examples propose efficient reusability experimentally for Reinforcement Learning neural networks as well: Networks are pre-trained on similar tasks and continued learning for the goal application. Reusing (sub) networss that can be re-assembled for an application never seen before can reduce network training time drastically. A better understanding of uniform or inhomogeneous network structures also improves the evaluation of network performance as well unveils opportunities for the interpretability of networks which is crucial for the application of machine learning algorithms e.g. in industrial scenarios. Finally, methodologies and metries estimating network intrinsic and inter correlations artificial neural networks may also enhance the understanding of biological learning. Eickenberg et al. (017) could recently demonstrate that layers serving feature extractors in CNNs could actually be found in the Human Visual Cortex by correlating artificial networks to biological recordings.
Successful experiments to re-use end-to-end learned networks for similar tasks leave open whether such networks also self-organize feature extractors or in a dynamical domain motion primitives. Here, we analyze neuron activation in networks in order to investigate activation distribution and mapping between different networks trained on similar robot reaching tasks.
In this paper we consider standard vertical space robot manipulator with variable number of revo- lute joints as the test setup for target reaching end-to-end Reinforcement Learning (RL) experiments. We introduce metrics applied to evaluate individual neuron activation over time and compare activity within individual networks all-to-all (every neuron is correlated to any other neurons in the network) and layer wise (only correlations between networks on the same layer inspected). These metrics are utilized to set up a pruning procedure to maximize the information density in learned neural networks and reduce redundancy as well as unused network nodes. Exploiting these optimization

---

Under review as a conference paper at ICLR 2021
procedure we learn various neural networks with variable dimensions robot manipulators with two to four joints, representing two to four Degrees of Freedom (DOF). in order to analyze similar- ities between network activation patterns.
As a result we demonstrate experimentally that the introduced pruning process reduces the network size efficiently keeping performance loss in bounds and hereby builds valid basis for network analysis. We show that the networks trained and iteratively pruned on the robot manipulators form distinct neuron activation. Analyzing neuron activation correlations between different networks of various sizes, mappings between neurons trained on different manipulators can be found. A layer wise interpretation reveals that networks trained for same tasks build similar structures, but we can also discover partially similar structures between networks trained on 3 and 4 joint manipulators.
2 RELATED WORK
The apability of feature extraction in CNNs, alongside with a variety of analysis and visualization tools, serves as a motivation for this work on training, analysis and pruning for networks trained with RL. Analysis methods for CNNs reach from regional based methods, e.g. image occlusion Zeiler & Fergus| (2014), that aim to expose the region of an image most relevant for classification, to feature based methods, e.g. deconvolution Zeiler & Fergus (2014) or guided backpropagation Selvaraju et al. (2017). Methods combining the described techniques are for example introduced as Grad-CAM in Springenberg et al. (2014). These networks demonstrate class discrimination for features of deeper network layers (Zeiler & Fergus (2014)) as a basis to apply such general feature extractors to different applications after pre-training. Pre-trained networks such as ResNet (e al.] (2016). which has been trained on the ImageNet1 data set, speed up training drastically by initializ- ing CNNs applied for similar tasks. Kopuklu2019 demonstrated that even reusing individual layers in the same network can lead to performance increase.
Recent advances pushed RL agents reach super human performance in playing Atari video games
Bellemare et al. 2213) Mmih et al. (2013), Chess (iilvrr et al.] (2017) and Go Silver et al.) (2016).
These results were extended to cope with continuous action spaces in e.g. [illicrap et al. (2015)
and demonstrated great performance on highly dynamic multi-actuated locomotion learning tasks
such as demonstrated in the NIPS 2017 Learning to Run challenge Kidziński et al. (2018). Vuong et al. (2019) and Eramo et al. (2020) demonstrate experimentally that knowledge learned by neural network can be reused for other tasks in order to speed up training and hereby translate modularity concepts from CNNs to RL frameworks. Hierarchical Reinforcement Learning incorporates these ideas, utilizing the concept of subtask solving into neural networks e.g. in Anddees et al.) (2016) for question answering. A successful example of transfer learning build up a general knowledge base could be demonstrated with RL in Atari games in Parisotto et al. (2016). 6). Gater & Ha (2019) empha- sizes the importance of neural architectures that can perform well even without weight learning. With a main motivation to improve learning efficiency and reduce computational requirements, net- work pruning is introduced for various network architectures. Early work in LeCun et al. (1990) utilizes second derivative information as heuristic to decrease network size, recent work in Livne & Cohen (2020) introduces network pruning for Deep Reinforcement Learning based on redundancy detection in an iterative process.
Li ei al. 20188)
EXPERIMENTAL SETUP
In this paper we focus on robot manipulator with operation limited vertical plane. A neural network is trained with end-to-end Reinforcement Learning in order to reach predefined locations in 2D space without prior knowledge of neither robot dynamics nor the environment. Hereby, end-to- end refers to a mapping from sensory feedback in terms actual joint positions in cartesian space and the desired goal location to output actions as joint position commands. We apply Deep q-learning, as proposed in Mnih et al. (2015). to predict q-values, an action is selected by means to the softmax exploration policy and Gradient descent of the networks weights is handled by the Adam Solver Kingma & Ba (2014).
For performance reasons our experiments are executed within a simplified simulation environment shown conceptually in Figure T (right), but exemplary behaviors have been successfully trans-
2

---

Under review as a conference paper at ICLR 2021
Figure 1: Neuron activity is analyzed in neural networks trained for target reaching of multi-joint robotic manipulators utilizing end-to-end Deep Q-Learning. We train the network a simplified environment of robot with to 4 controllable joints operating in vertical space (right, initial con- figuration with joints 8₂). Transferability to sophisticated robotic simulation (left) with motions restricted to vertical space and the robot finger tips supposed to reach the green sphere could be demonstrated.
ferred to a realistic robotic simulation (Figure left). We simulate robots with 2 to DOF that are implemented revolute joints restricted to vertical space motions and actuated with PID position controllers. For all experiments, the neural networks originally consist of 6 fully connected hidden layers with ReLU activation functions, but may be reduced in the pruning process we introduce. The network input vector encodes actual robot joint angles 8; as their sine and cosine contribution for every control step (control cycle time of 50ms) as well as the desired goal position in cartesian coordinates [x", yy" as
(t) sin (8())
cos (8(3)
sin
g(L)
(
x
y
(1)
The output layer contains 3 neurons the action of every individual joint is quantized into the three change motion states (1, -1, 0} as forward, backward and no motion for each joint with joint angle changes of #0.05rad. The goal state of an agent is randomly instantiated 2D location to be reached with the robot finger tip in max. 60 control steps, each representing 50ms. The distance between the goal position p" and the tip P is mapped into [0, 1] and squared to serve the reward
All network results that are presented passed a validation test consisting of 300 test episodes. This test also serves as the pruning baseline: The probability of type two error for reaching the final reward threshold F = 0.9 with an accuracy p = 0.9 lies bellow significance 0.05 on the test data.
function r (f.) := (PC))-p'1aa +1
NEURON ACTIVATION ANALYSIS
We first analyze individual neuron activation inside multiple neural networks trained on the intro- duced target reaching robotic manipulator. This initial analysis servers as a baseline for pruning and projection evaluation, therefore we study only joint robotic manipulators in depth before we investigate comparison of different kinematic structures.
We define a distance metric between neurons that is based on the neuron activation history in scope of every episode in order account for the dynamics in motion trajectory learning. All neuron activation values over the course of an episode are collected the network in Episode E. Utilizing the linearity of applied ReLU activation functions
vector (6))) for every neuron n, of normal- ize this activation range [0, 1] in reference to the maximum value attained. For a set of sample episodes &, representing of potential robot actions, define the distance of neurons n; and n, as
(E)
E
d(n,, n2) = E
2n:
Zn;
Zno
Z
BEE
L2
with (E) R denoting the vector containing activation series of neuron n; in episode E and Zn, E Ry00 the maximum activation n; in all episodes E. For a layer wise analysis Equattio adapted accordingly, only considering distances to neurons that belong the same layer. The
3

---

Under review as a conference paper at ICLR 2021
upper triangular matrix of distance matrix D holds all values d(n,, n2) with i? j. The density distribution of neuron distances can be approximated by collecting all values in the upper triangular matrices of D.
Additionally, hierarchical clustering as described in Hastie et al. (2009) is applied to individual network layers in order to reveal neuron groups that show similar activation behavior. We form groups that minimize the mean cluster distance D(C) of contained neurons as
D(C)) = |c|| (1C)) 1)
Σ E BaEC RUEECMmu))
d(n,t, mzz)-
for neuron cluster C' of layer I. We conduct experiment with set of M 20 networks (48 neu- rons per hidden layer), for the three joint manipulation task. A reference set of untrained networks with identical structure is initialized by Xavier initialization Glorot & Bengio (2010). Neuron dis- lances are averaged from set of m 500 sample episodes. The distance distribution in randomized networks forms bell-shaped distribution globally as well as layer wise (Figure 2 top). However, the all-to-all distribution of trained networks primarily indicate lower standard deviation and mean compared to random networks, with a slight distortion high distances. Layer-wise analysis re- veals that these higher distance scores occur increasingly on network layers closer to the output, in particular in the second half of layers. In contrast, lower layers demonstrate close to normal dis- tributions. Clustering reveals variety in distances for all layers in untrained randomly initialized networks (Figure 2jbottom) which is kept on the first layer only in trained networks. In particular on the middle layers clusters with low distances emerge during training. The intrinsic network analysis depicts successful training that visibly changes the neuron activation characteristics which highly depends on the location inside the network.
HEURISTIC NETWORK PRUNING
Non-uniform density distributions and low cluster differences in the inspected neuron activation indicate potential for network pruning. Dense information representation is a requirement for the comparison of different networks. For this purpose we propose pruning procedure that iteratively unifies neurons with similar activation, identified as small cluster distances, and retrains the net- work. Hereby trade-off between reduced network size and maintaining high-performance learning is aspired.
We apply Breadth First Search on the resulting cluster tree of every network layer. The first encoun- tered clusters with distance (3) below threshold d,, which is defined as fraction of the maximum cluster distance, are selected to form the layer segmentation C. Based on this neuron segmentation ( of layer I, reduced network is constructed that represents every cluster a single neuron. Original network weights are reused for the initialization of the reduced network. We exploit the linearity of ReLU activation functions and assume identical neuron behavior only altered by lin- ear scaling inside every cluster. W.l.o.g. cluster activation Çc are defined such that scaling factors 7n > of contained neurons sum to one and Vn E C ÇC 7-.
än with 2n denoting the activation
of neuron n, holds. For cluster C c()) and arbitrary neuron can be rearranged to form the forward propagation of the cluster activation as
C' the forward propagation of "n
Sc ReLU
Σ DECU-1
CD Ta
1
Σ MED
Wmm Tm
(4)
with Wam denoting the weight from neuron m to n. acts an approximation that in practice is only achieved by clusters of dead neurons that are not activated at all. Therefore, in order to improve stability all neurons of cluster contribute to the reduced network weights w as WCD
Emed Wam Scaling factors In are generated from the maximum activation Z,, of the respective neuron n.
In order to evaluate the introduced pruning procedure, we conduct experiments with a set of M 20 neural networks (6 hidden layers, 48 neurons each) trained for the 3 joint manipulation task. Network reduction is applied with a set of m = 300 sample episodes, presented results are averaged over the set of networks which reached sufficient performance. The results presented in Figure 3 (left) show a nearly linear correlation between cluster threshold and resulting pruned network size if networks had identical initial layer size of 48 neurons. In case of only dead neurons
4

---

Under review as a conference paper at ICLR 2021
Layer
Layer 2
Lower 3
201
series Layor
Layor
2013
Layer
Layer
Layer
Layer
Layer 5
Layer
random
trained
F. E
pruned
Figure 2: Neuron activation analysis for randomly initialized, trained and pruned networks three joint manipulator (averages over 500 sample episodes, 20 trained agents). Top: Distance measured between all-to-all (all neurons in network are correlated among each other, left) neurons and layer- wise (for every neuron only neurons on this layer are considered for correlation, right) indicate a bell-shaped distribution with higher mean in the first and last layer. Pruning sharpens the bell-shape, increasing the mean, but reducing very high distance scores. Bottom: Clustering Dendrograms are generated based on the distance measures for exemplary trained network. Untrained networks show very similar clusters, trained network highlights cluster groups and pruning reduces neurons while increasing cluster distance. The first layer generally keeps the most distinct clusters, the penultimate layer the strongest neuron reduction.
reduced, which does not affect the performance of the network. though reduces the network size significantly (initial size all networks: 323 neurons). For values r E (0, 0.1] the network is reduced, but strong effect is apparent initial accuracy [%] and training duration (number of episodes executed until the validation set is passed). We observe interesting behavior in range of (0.1, the initial accuracy decreases significantly, whereas the duration for retraining the networks barely increases. This implicates that the main processes in the network remain in tact after reduction, whereas for 0.2 a strong increase in training duration indicates a loss of relevant information. As trade off between minimal network size and efficient training 7 = 0.2 has been selected as the optimal cluster threshold and was applied for all further experiments. As the pruning process highly depends on the initial network size, we analyze networks of initial hidden layer sizes 32, 48, 128, 256 and 512 within the same test setup. The results shown in Figure (right) emphasize the first reduction step the most dominant. Noticeably large networks of initial layer neuron count of 128, 256 and 512 reach similar pruned network size already the first iteration step. For subsequent reduction steps the network size plateaus. Inspection of neuron per layer counts reveal that small initial networks (32, 48) taper with depth, compared to bigger initial networks that form hourglass shape. The average network shape of 256 and 512 neuron networks after three
5

---

Under review as a conference paper at ICLR 2021
Figure Evaluation of heuristic cluster based network pruning on the example of a three joint manipulator. Left: Even though the initial network accuracy decreases rapidly, the training duration (Number of episodes executed until the validation set is passed) only increases significantly with cluster thresholds larger than 0.3. As a trade off between minimal network size after pruning and efficient training т = 0.2 has been picked as the optimal cluster threshold and was applied for all further experiments. Right: The first reduction step demonstrates the strongest reduction for all networks initialized with different initial neuron count per layer. Layers with more than 128 neurons reduced to very similar neuron count in the first pruning step.
reduction steps turns out as 8 = [51.6 21. 15.9 12.6 10.2 16.7] Network intrinsic neuron distance densities of pruned networks (Figure 2] implicate an increased homogeneous information representation compared to networks trained straight away. The bell-shaped distribution with higher mean shows lower variance, and outliers of high distance scores are reduced. While clusters remain rather similar on the first and last layer, in particular the cluster distances middle layers are drastically increased along with the reduced cluster number. Overall, we find that our pruning process reduces network size efficiently and hereby shows visible effect neuron activation towards a rather uniform distribution and distinct cluster structure.
CORRELATIONS IN NETWORKS TRAINED FOR MULTI-JOINT ROBOTS
Based the both the individual neuron activation analysis and heuristic network pruning, we now investigate mappings of neuron activation between different networks learned on robot manipulators with 2 to 4 joints. Here, the goal trained for the different robot kinematics. For this purpose we construct an unidirectional linear projection between source and target network and analyze its accuracy and structure. Based on the source network neuron activation b E R.K
estimate whether activation patterns are similar networks
20: resulting from input prediction = bb" P of the target activation a E R.M. for the same input x is given by projection matrix R (Figure 4). 20
20 The projection constructed based on a set of N training inputs X that yield activation mainces
Network
Figure 4: Analysis of inter network mappings: Sets of two net- works are trained on robots with different number of joints. A projection matrix P that reflects the network similarity is calcu- lated to compute from the source network neuron activation with minimal difference a.
Acthvations
Prediction diff Activations
NxM and B BNNK of the target and source network, respectively. In order to obtain A E R
procedure invariant to neuron scaling, individual columns of A and B, are normalized to the interval [0, 1] dividing by the maximal values contained. The resulting projection can be adjusted to fit the original training data by Pkm = 3e Pkm- Two approaches for projection construction are considered, Greedy mapping predicts each target neuron from the source neuron with minimal distance 2 every entry of the greedy projection matrix P is 1 if = arg min; [d(m, i)} and otherwise. Linear mapping incorporates source neurons into the prediction of target neuron by linear combination. Projection vectors Pm. predicting the behavior of neuron are given by the solution quadratic optimization with linear boundary constraints for each target neuron individually. Hereby, the mean squared error plus lasso regularization, to enforce sparsity of solution vectors, minimized finding the best projection p, i.e.
a
minimize
Bp
+ Applz
subject to
p p20 0.
(5)
2
Olm
L2

---

Under review as a conference paper at ICLR 2021
B denotes the matrix of souree activations scaled by Bk. am the target activations and X E R20 the regularization strength. As mapping of two networks should invariant to neuron scaling, all individual neuron activations factors 3g and Om for the source and target network neurons, respectively. 30 The solution vectors Pm are stacked to form the linear projection matrix p == [pi deduced from set of sample episodes of the target network without duplicates, In put vectors of robot manipulators with different joint count are transformed by either duplicating best aligning joints or unspecified joints being set to zero, for more or less complex source network, respectively (Figure middle right).
projected into the interval [0, 1] with neuron specific scaling
Par] Input samples X
6.1 EVALUATION METRICS
Projections are evaluated with regard to their goodness to fit set of validation samples X and according to heuristic metrics that directly analyze a projection structure. The mean absolute prediction error is normalized by the prediction enror of the zero projection Po {0} x to construct the normalized error E(P, X) that is invariant to weight scaling and adding dead neurons:
E(P,)) X(P,)) E(Po.X)
1 [A]]
Σ
a Bpm
The entropy of target neuron's 's projection Pm is referred to the saturation of neuron m, projection P is the mean of all neuron saturations. low saturation implies that few neurons suffice to describe the behavior of m. We calculate the overall projection saturation S(P) according to Equation
M Σ
S(P) = M
Pkm logk (Pem) E [0, 1].
The utilization of the source network neurons to describe the target network is indicated by the coverage defined as the entropy of the stochastic process that picks a target neuron m uniformly at random and passes it on to the source network according to the distribution Pm A
[pm L1
low coverage value implies low utilization of the source network.
1 C(P)
1
Pre
K Σ Ki kologk (mk); with K&
(8)
M
Pm 41
The same statistical process is applied to construct a layer-wise projection Pij. It describes the probability of reaching the ith layer _(K)))))) the source network when starting in some uniformly random neuron in the jth layer (M,) of the target network.
Prm
Pij
(9)
1L, (M)
Pm 11
KE
mEL
6.2 RESULTS
For each robot manipulator with and 4 joints, M networks trained, pruned in three steps and we analyze all possible mappings "a-b" between the respective sets. A set of validation inputs X is generated for m 300 sample episodes of the target network and metrics evaluated. As baseline we map all 3 joint manipulator agent networks with an initial neuron count of 256 for each of the 6 hidden fully connected layers, among each other. expected, as a baseline mappings of networks to themselves (referred to as reflexive mapping) show zero error and saturation and coverage 1 (Figure 5) top left). However, greedy mapping shows a high normalized error and low coverage when compared to the linear mapping and thus is considered an inferior approach. In this baseline we extract linear mapping with regularization strength À = 50 as the best metric trained in contrast to random
as it indicates coverage and normalized error most significant networks. Liser-wise linear projection (A = 30) is not optimal but we observe the best mapping to the respective layers, shown the diagonal axis in the table of Figure 5) Hereby, layer and

---

Under review as a conference paper at ICLR 2021
LayenWite Uneer Mapping
Hotra Sanuuation
COTTTA Laurage
0.
1024
0:155
0.21
8:440
0.333
6:39
3.148
0.99
5.004
6.7C
6:44
6:5T
A156
0399
0.30
mapding mwp.impiigg..... 500
napping
ordrr (1+0))
reddom sib
1.0
Saurriie
1.1
P
2
4a
4b
3-- 6.3
3446
192
2:4
294
2:
Figure 5: Projection of neuron activation between networks trained for variable joint robot ma- nipulators. (data averaged with 5 networks each, pruning steps). Top: Benchmark of mapping technique and evaluation metrics: On the example of multiple three joint manipulator networks, we find linear mapping with À 50 the superior mapping approach in contrast greedy mapping. In particular, coverage and normalized error indicate mapping quality well in comparison to untrained networks. A layer-wise linear mapping with À 50 is not optimal, but strongest correlations be found between corresponding layers. This is represented in the higher diagonal values in the table of normalized average layer distances on the right. Here, layer I and 6 show best mapping. (initializa- tion with 6 layers each 256 neurons before pruning, random nets with average pruned network size [46
seven tove
of s = [46 22 16 13 8 20) neurons per layer). Middle: Neuron activation correlations of networks trained on robots with different joint count (2-4 refers to mapping from networks trained for a joint to 4 joint robotic arm): The mapping error gets higher with increased difference in joint numbers, the coverage accordingly decreases. Mapping a network with higher complexity into lower complex ones performs slightly better than vice versa. In this study the mappings 4-2 are closest to the performance of the native 3-3 mappings. This mapping is influenced by proper trans- formation of sensory inputs to the increased number of input neurons on the first layer. The results demonstrated for balanced mapping as 0f 0,, 02 0z, 02 Og 0 (4b) which performs better than in contrast to naive mapping 0f = 0,, 0² = 0z. 0g 0, O₂ = 0 (4a). (results as mean of 25 mappings). Bottom: Mean layer layer projections: Networks trained for more similar robots show better layer to layer mappings. The first layer of the source network shows high utilization for mappings to all other layers, the penultimate layer the most unlikely be utilized. Middle layers map reasonably well for 4-3 and 3-4 mappings, more distinct robots as 4-2 and strongest in 2-4 mostly utilize first layer neurons only.
input and output of the network. six demonstrate the strongest correlation potentially due to increasingly specialized neurons the
8

---

Under review as a conference paper at ICLR 2021
Linear mapping (A 50) has been applied between sets of 2, 3 and 4 joint robot manipulators (Figure 5) middle left), Random networks are initialized by the average network size of the respec- tive joint count as evaluated with pruning. Scenarios 3-4 and 4-3 show similar prediction emrors but indicate a higher mean error compared to 4-2 mappings. Latter mapping performs similar to the baseline, which might be induced by the fact that we transform inputs in balanced way so that the 4 joint arm can act like a 2 joint arm (figure on the right, we choose the transformation 4b). It shows lower coverage of the source network, which is partially related to the fixed input channels for the source networks after input transformation. The worst performance according to the prediction error shown by scenario 2-4 as the two joint manipulator networks are barely able to replicate the behavior of the four joint networks. Generally, the more distinct the robots the worse the mapping, except input transformation is implemented
a meaningful way. More complex networks map slightly better into less complex one, as compared to the opposite way round.
A deeper insight to the source network utilization is drawn from mean layer-wise projections (Fig- ure 5 bottomm) The baseline scenario 3-3 shows more significant correlation to its respective layer the closer it is to the input or output. The first layers of 3-4 and 4-3 mappings seem to follow the behavior of the baseline, whereas the deeper layers show significant correlation. Contrary to the performance of the overall metrics, scenario 4-2 shows no strong layerwise correlation, which is even worse in the inverted 2-4 mapping. If layers do not map well, all target layers tend map to the lower layers especially the first layer (most prominent in 2-4 mappings) of the source network, only a small tendency visible of the output layer mapping to other output layers. We hypothesize this phenomena is credited to first layers having the highest neuron count and activation variance. Overall, we do find that a good mapping correlation when the source network is able imitate the behavior of the target network, a suitable input transformation turned out to be crucial here. 4-2 mappings showed the lowest error, but networks trained on three and four joint networks map better into their respective layer.
CONCLUSION
In this paper analyzed individual neuron activation and correlations between neural networks trained for goal reaching of vertical space robot manipulators with two, three and four joints. We analyzed and classified the activation in order implement pruning algorithm that removes redun- dant neurons and increases information density in the network. Finally, we analyzed correlations between the overall and layerwise neuron activation of networks trained on robots with different joint number by projection mapping. Our results demonstrate that networks develop distinct activa- tion patterns on individual neuron layers with bell-shaped distribution of activation densities. This distribution is compressed by our pruning algorithm that merges similar neuron activation classes mostly on the inner network layers. Networks trained for robots with only small joint number dif- ference show good correlation of neuron activation, for small differences this correlation can be found layer-wise. The more distinct the robot kinematic in terms of joint number, the more impor- tant is proper input transformation that fits the different network input layers. All experiments are benchmarked by comparison against untrained networks and self-correlations for multiple networks trained for the same task. Our results help to improve explainability of reinforcement learning in neural networks for robot motion learning and highlight network structures that can be reused on similar tasks after pre-training. The experiments conducted are limited robot manipulators of 2 to 4 joints acting in vertical space, however the underlying introduced methodologies could be transferred other Reinforcement Learning tasks as well. Analysis of neuron activation has been introduced in other contexts, here here we utilize it for the analysis of the specific use case of verti- cal space robot manipulation. In future work our pruning algorithm can be extended also reduce the number of overall layers, analyze additional network parameters and we will examine reusing network structures with good correlation experimentally.
REFERENCES
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural net- works for question answering. 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 Proceedings of the Conference, pp. 1545-1554, 2016. doi: 10.16653/1/1116-18811

---

Under review as a conference paper at ICLR 2021
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificiil Intelligence Research, 47: 253-279, 2013.
Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152 (September
2017. ISSN 10959572. doi: 10. n..0016/..eurroiaaee.20000
10.001.
Carlo D Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Politecnico Milano, and Jan Peters. S HARING K NOWLEDGE IN M ULTI -T ASK D EEP EINFORCEMENTI L EARN- ING. pp. 1-18, 2020.
Adam Gaier and David Ha. Weight agnostic neural networks, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249 256, 2010.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Eukasz Kidziñski, Sharada Prasanna Mohanty, Carmichael F Ong, Zhewei Huang, Shuchang Zhou, Anton Pechenko, Adam Stelmaszezyk, Piotr Jarosik, Mikhail Pavlov, Sergey Kolesnikov, et al. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromuscu- loskeletal environments. In The NIPS' Competition: Building Intelligent Systems, pp. 121-153. Springer, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6908 2014.
Yann LeCun, John S Denker, and Sara A. Solla. Optimal Brain Damage (Pruning). Advances in neural information processing systems, pp. 505-605, 1990.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes, 2018
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint NtX.vv:10990097.... 2015.
Dor Livne and Kobi Cohen. PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning. pp. 1-14, 2020. URL http::t/taarttt/ org/ abs /0/gg/abs/000... 05012 20010
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):599-333 2015.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic deep multitask and transfer reinforcement learning. 4th International Conference on Learning Representations, ICLR 2016 Conference Track Proceedings, pp. 1-16, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626. 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
10

---

Under review as a conference paper at ICLR 2021
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self- with general reinforcement learning algorithm. arXiv preprint arXtv:1712.01815, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6888 2014.
Tung Long Vuong, Do Van Nguyen, Tai Long Nguyen, Cong Minh Bui, Hai Dang Kieu, Viet Cuong
Ta, Quoc Long Tran, and Thanh Ha Le. Sharing experience in multitask reinforcement learning. IJCAI International Joint Conference on Artificial Intelligence, 2019-August:344-33648, 2019. ISSN 10450823. doi: 00.24933ijeai222115555.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 8 818-833. Springer, 2014.


Reviews:
[{"review_id": "yxS0PGeDbRI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThe paper presents a technique to compare networks trained to solve similar tasks trained in different context. The considered task is reaching with a robotic planar arm; the considered context is varied varying the robot degrees of freedom. The goal of the paper is to find correlations across neural activity patterns across networks trained to solve the same task in different contexts.\n\nTo achieve their goals, authors propose an heuristic network pruning algorithm to reduce the network size while keeping performance in training and evaluation. To correlate different networks, authors propose a technique to project a source network onto a target network.\n#### Clarity\nThe paper is well written and easy to read. \n#### Originality \nThe paper follows a main stream research which aims at pre-training neural networks on general task to speed up learning of specific tasks. As far as I know (I am not an expert in the field), the proposed analysis is original.\n#### Significance \nThe significance of this work is relatively low. The results could be more conclusive with further analysis and experiments.\n#### Major comments\n* **Kinematic redundancy**. Authors have chosen a specific task, nominally reaching a target with a planar manipulator. Within this context, in presence of more than two degrees of freedom (i.e. with kinematic redundancy), the solution of the task itself is non-unique. Therefore fixing the context (i.e. fixing the robot kinematics, the robot geometry, etc) doesn't guarantee that the RL algorithm will find similar solutions across different runs. Actually, given the random training procedure each solution should come up with completely different strategy, exploiting the kinematic redundancy. If so, how do authors compare networks which exploit differently the kinematic redundancy? Are the proposed metrics (greedy mapping, linear mapping, etc) invariant with respect to different solutions to the same task (i.e. invariant to kinematic redundancies)?\n* **Results and conclusions**. Goal of the paper (mentioned in the first two sentences of the abstract) is to progress in understanding if pre-training of end-to-end RL can be used as feature extractors and hierarchical organizations. Despite what claimed in the conclusions (\"Networks trained for robots with only small joint number difference show a good correlation of neuron activation, for small differences this correlation can be found layer-wise.\"), authors fail short in giving a sound explanation of why this is the case.\n#### Major comments\n* **Page 7, line 8 from the top.** \"[..] the reflexive mapping\". This was not mentioned before, authors should give more details.\n* **Page 7, line 1 from the bottom.** \"Are joint numbers very different a proper input transformation is crucial to find correlations\". Please check this sentence. \n* **Page 8, caption of figure 5.** \"balanced mapping θ1′ = θ1 , θ2′ = θ3 (4b) we apply in contrast to the naive mapping θ2′ =θ2,θ2′ =θ2(4a).\" It's unclear what these mappings refer to and how they have been used. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qpRRH9UvWqu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###  Summary\n\nThe authors investigate individual neuron activations over time, and compare the neuron activations within individual networks all-to-all and layer wise. \n\nA distance metric is introduced and utilized to set up a pruning procedure to maximize the information density in learned neural networks and reduce redundancy as well as unused network nodes.\n\nFinally, neuron activations are used to assess the correlations between learned policy networks for manipulators with a varying number of degrees of freedom. A projection mapping between different policy networks is implemented and analysed, as a type of transfer learning between different robot morphologies.\n\n\n###  Review\n\nI believe that this is an interesting work which tries to understand the inner-workings of a robot-control policy network by examining the network activations, and further using this information to prune the unnecessary neurons. Transferring learned network policies between robot morphologies is very useful and preliminary insights seem interesting.\n\nThere are some important flaws that need to be addressed regarding the clarity of the methodology and contributions, as well as the significance of the experimental evaluations. \n\nMy impression is that although the work tackles an important problem with a good idea, this is still an incomplete work, as the presented experimental evaluation is insufficient to draw significant conclusions.\n\nBelow are some of the comments organised by sections, including concrete suggestions for improving the work.\n\n1. INTRODUCTION\n\n The main motivation and goal should be presented explicitly in a separate paragraph. There seems to be a missing link between the neuron activation estimations for pruning and correlation analysis for transfer mapping.\n\n2. RELATED WORK\n\n I do not see the relevance of Bellemare et al. (2013), Mnih et al. (2015), Chess Silver et al. (2017), Go Silver et al. (2016) and Lillicrap et al. (2015) to the specific problems investigated in this paper.\n\n It would be useful to consider other work investigating NN complexity, and adding a discussion on how it relates to this work:\n - Gaier, Adam, and David Ha. \"Weight agnostic neural networks.\" Advances in Neural Information Processing Systems. 2019.\n - Li, Chunyuan, et al. \"Measuring the intrinsic dimension of objective landscapes.\" International Conference on Learning Representations. 2018.\n\n3. EXPERIMENTAL SETUP\n\n “planar space robot manipulator that represents a multitude of real world applications” Do you mean that this task is a surrogate for examining many applications? This seems like a strong statement as it represents a small subset of potential applications.\n\n Moreover, planar space usually refers to having a horizontal plane as a task space. In this case it would be more clear to say “operating in a vertical plane”.\n\n “A neural network is trained with end-to-end Reinforcement Learning” this usually means from input images to output torques, but in the presented approach position control is used, so this should be emphasised.\n\n “physical robotic simulation”, usually it is said “physical robot” referring to a real robot experiment, or a “realistic robot simulation” which refers to a simulation that takes into account the real robot component values (dimensions, mass, inertia…).\n\n Equation 1 is not clear as the text says that $\\textbf{x}$ consists of joint angles $\\hat{\\theta}_i$ but eq 1 shows the sin and cos projections of the angles. Moreover, the index $i$ seems to refer to both the time-step index $t_i$, as well as the joint index $\\hat{\\theta}_i$. Also, $n$ is not introduced as the number of joints.\n\n What is the reason behind mapping the target distance into a (0,1] range?\n\n Another important aspect which I believe should be addressed, is what is the effect of the task which is learned on the activation correlations? Basically, different tasks would have a different state distribution seen at the input of the policy? One simple example would be examining different types of control - position vs velocity vs torque.\n\n\n4. NEURON ACTIVATION ANALYSIS\n\n This section provides an analysis for the 3DOF robot only and should be emphasised.\n\n “We define a distance metric between neurons that is based on the neuron activation history in scope of every episode in order to account for the dynamics in motion trajectory learning.” It is a bit unclear how the activation history is evaluated. This is one of the most important parts of the paper and should be made very clear.\n\n I am not 100% sure that the distance metric should be referred to as Euclidean distance, as it does not operate on euclidean space, so I think it would be sufficient to say “a proposed neural activation distance metric”.\n\n “For a set of sample episodes E representing the agents action space” How do episodes represent the action space exactly, I do not understand this part completely.\n\n The notation is a bit confusing, as $n$ refers both to the number of joints and the number of neurons. Also, what does the superscript $T$ in $R^T$ stand for?\n\n It seems to me that there is another summation missing in equation 2, as the indices of neurons should be more generic like $n_i$ and $n_j$ (unless there are only 2 neurons?). Also, are these neurons form the same layer or all the neurons in the network? This should be clarified in the text and formulated in Eq 2.\n\n What is $C$ in equation 3? I assume it is the cluster and cluster size, but this should be explained explicitly.\n\n Some of the distributions in Fig 2 are a bit skewed and look more like Beta distribution rather than Gaussian. Why is having a Gaussian distribution of distances relevant? \n\n There might be some other visualisation method that could be used here to shed light on the findings. Because currently, it is difficult to see any significant differences between the plots.\n\n The reference to Fig 2 should be improved, the definition of the trained, random and pruned lines is missing (both in the figure and the text). \n\n What does “all-to-all distribution of trained networks” mean (all-to-all neuron comparison)? \n\n The findings for the clustering (Fig 2 bottom) are very interesting! Could you maybe elaborate on these more?\n\n\n5. HEURISTIC NETWORK PRUNING\n\n What is the motivation for retraining after pruning? If neurons that have similar activations are pruned, what would happen if one of them is kept with the corresponding weights? How would this affect the performance? This could be an insightful baseline comparison.\n\n Moreover, what is the advantage of reusing the network weights for initialisation, instead of randomly initialising them? This would also be an interesting experiment to conduct.\n\n Please introduce what are “dead neurons”.\n\n Wrong reference to Equation (5) should be (4)\n\n $\\tau > 2$ → $\\tau > 0.2$ \n\n The accuracy in Fig 3 left, is given in [%], is this a mistake, because then it seems that the initial accuracy is only 0.8%. If this is actually 80%, why is the optimal $\\tau = 0.2$ as the corresponding accuracy has fallen to 20%. How is this evaluated? Does the green label correspond to pruned or initial network size?\n\n\n6. CORRELATIONS IN NETWORKS TRAINED FOR MULTI-JOINT ROBOTS\n\n It would be useful to start with a high level overview of what is the goal of finding these correlations in addition to how they are calculated. It seems that the correlations between networks are not examined, rather the mappings between them. Therefore this is slightly misrepresentative of what is actually being done.\n\n How are activation matrices A and B related to P? Also, what are $\\alpha_m$ and $\\beta_k$ ? \n\n What is the motivation behind making $\\bar{P}^g_{km}$ sparse according to minimal distance in Greedy mapping, or applying L1 regularisation in Linear mapping? I assume your goal is to map joints 1-1 rather than combining them? Please explain this in a more clear way if possible.\n\n Equation 5 does not show the variable which is optimised. What does $\\alpha^{\\downarrow}_m$ stand for?\n\n Equations 6 and 7 are not referred to in the text.\n\n The quantities defined in equations 6, 7, 8 and 9 should be properly introduced as evaluation metrics and named accordingly, in a separate paragraph.\n\n Figure 5 correlations (top right) would be more impactful if represented with a heatmap matrix in addition to the numbers. The graphs on the bottom are not very clear.\n\n The discussion of the mapping results is very interesting and should emphasise the mapping between different robot morphologies. For example the difference between 4 -> 2 and 2 -> 4, where the latter has a higher error which could be expected as there is not enough information stored which can be decoded. Having additional comparisons of robot manipulators with larger differences in DOF would probably emphasise this and support the given conclusions better. This would also strengthen the paper significantly.\n\n Another metric which would be necessary to evaluate the transfer procedure, is to evaluate the mapped network on a test set of the reaching task.\n\n\nOTHER COMMENTS:\n - Figure captions should be larger\n - Several typos\n - Consistency in using “3 joint manipulation task” or “3-DOF manipulator”\n - In the conclusion it is stated: “In this paper we analyzed individual neuron activation and correlations between neural networks trained for goal reaching of a variety of planar space robot manipulators.” Having the same robot structure with 3 different DOFs is not sufficient to be considered as a variety of manipulators.  \n - It would significantly help the clarity of the paper to split certain sections into thematic paragraphs.\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SQ9AiSN-LuA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThe authors present a method for analysing neuron activity in neural networks trained via RL on a multi-joint planar reaching task, as well as correlating neurons between different models trained on tasks with potentially a different number of joints. The methods consists of three steps:\n1. Compare different neurons within a model using normalised activation traces over a number of episodes, and cluster hierarchically based on similar activity.\n2. Use said clusters to prune the networks based on merging neurons within a cluster with an intra-cluster distance below some threshold, alternated with retraining.\n3. Compare different models by optimising a linear projection between neurons, and evaluate reconstruction error, coverage and saturation.\nResults indicate the proposed pruning method is effective in reducing the number of neurons without affecting accuracy, as well as showing correlations between corresponding layers of different models, though these reduce with larger difference in number of joints.\n\n#### Pros\n- The authors perform a sufficiently thorough evaluation, with a large number of models compared and reasonable ablations, baselines and metrics.\n- While descriptions are brief, the method is generally well described and mathematical notation consistent.\n- The proposed heuristic pruning approach seems to perform well in this case, as evident by all model sizes converging to the same size in Fig. 3.\n- The approach of first pruning networks to maximise information content in the activations before correlating different models makes a lot of sense.\n\n#### Cons\n- The authors frame their work within the context of feature reuse and explainability, however the presented work is limited to showing correlations between features learned on identical or very similar tasks. It is unclear how this enables either reuse or explainability and perhaps not surprising that these correlations for very similar or identical tasks exists per se, more interesting would be to see how these can exploited. These correlations also degrade very rapidly with an increasing number of joints. I hypothesise that one would perhaps see stronger correlations between more different tasks if not the morphology was changed but rather the objective / reward. The scope of a planar reacher may also be too limited to draw more general conclusions for other control tasks.\n- Related to task differentiation, a potential weakness in the proposed methods is how the activation traces are generated for source and target model when optimising the projection. Effectively only the target model is evaluated within distribution, after which the inputs observed there are then remapped to the source model. It's unclear what the effect is of potentially evaluating the source model out of its training distribution. I was hoping to see a way to correlate trajectories collected with the respective models independently. Perhaps the type of problem considered only allows a singular solution even across different number of joints, but it would be good to verify this.\n- While the authors do evaluate a large combination of models, only averages are reported. Given how close the results seem to be to random in Fig. 5, it's hard to gauge the significance of the results. Some variance or error metric would be very valuable.\n- While the idea of pruning the networks before correlating intuitively seems like a good idea, this is not experimentally validated. It would be good to add a comparison with and between unpruned models as well.\n- While okay to follow, the text could use a bit more polish.\n\n#### Questions\n- There's currently no mention of how all these models were trained. One caption hints at DQN? Please provide more details.\n- It's unclear how to interpret training duration in Fig. 3. Is this the time required to \"pass\" the validation set again after pruning?\n- What do the values in the table in Fig. 5 represent? Sums of weights in the projection matrix?\n\n#### Conclusion\nWhile overall the method presented makes sense, and the evaluation is relatively thorough, the scope of the problems evaluated is considerably limited to draw any general conclusions of its validity, and some of the framing and details raise questions. As such I'd consider this submission marginally below acceptance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes neuron activations for neural networks trained via RL to perform reaching with planar robot arms. This analysis includes an evaluation of the correlation between neurons of different models trained to control arms with different degrees-of-freedom. In performing these evaluations, the paper proposes a heuristic pruning algorithm that reduces the size of the network and increases information density. Correlation is assessed based on a projection of the source network on the target network.

The paper is well written and considers a challenging problem of interest to the community. The proposed pruning strategy as a means of maximizing information content is reasonable and seems to perform well. However, the significance of the contributions is limited by the experimental evaluation. The experiments consider a large number of models, however the scope of problems on which the method is evaluated is narrow, making it difficult to draw conclusions about the merits and significance of the work. The authors are encouraged to extend the analysis to a more diverse set of problems.
Decision: Reject