Understanding the Evolution of Linear Regions in Deep Reinforcement Learning
Authors: Setareh Cohan
Nam Hee Gordon Kim
David Rolnick
Michiel van de Panne
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
piecewise linear regions
region densities

Abstract: Policies produced by deep reinforcement learning are typically characterised by their learning curves, but they remain poorly understood in many other respects. ReLU-based policies result in a partitioning of the input space into piecewise linear regions. We seek to understand how observed region counts and their densities evolve during deep reinforcement learning using empirical results that span a range of continuous control tasks and policy network dimensions. Intuitively, we may expect that during training, the region density increases in the areas that are frequently visited by the policy, thereby affording fine-grained control. We use recent theoretical and empirical results for the linear regions induced by neural networks in supervised learning settings for grounding and comparison of our results. Empirically, we find that the region density increases only moderately throughout training, as measured along fixed trajectories coming from the final policy. However, the trajectories themselves also increase in length during training, and thus the region densities decrease as seen from the perspective of the current trajectory. Our findings suggest that the complexity of deep reinforcement learning policies does not principally emerge from a significant growth in the complexity of functions observed on-and-around trajectories of the policy.

Understanding the Evolution of Linear Regions in Deep Reinforcement Learning
Setareh Cohan
Nam Hee Kim
Department of Computer Science University of British Columbia setarehc@cs.c wwa.es.bcccc..m
Department of Computer Science
Aalto University namhee. kim@aalto.ff
David Rolnick
Michiel van de Panne Department of Computer Science University of British Columbia vanêcs ubc.
School of Computer Science
McGill University
drolnickécs www.slc....c...................l
Abstract
Policies produced by deep reinforcement learning are typically characterised by their learning curves, but they remain poorly understood in many other respects. ReL.Ubased policies result in a partitioning of the input space into piecewise linear regions. We seek to understand how observed region counts and their densities evolve during deep reinforcement learning using empirical results that span a range of continuous control tasks and policy network dimensions. Intuitively, we may expect that during training, the region density increases the areas that are frequently visited by the policy, thereby affording fine-grained control. We use recent theoretical and empirical results for the linear regions induced by neural networks in supervised learning settings for grounding and comparison of our results. Empirically, we find that the region density increases only moderately throughout training, measured along fixed trajectories coming from the final policy. However, the trajectories themselves also increase in length during training, and thus the region densities decrease seen from the perspective of the current trajectory. Our findings suggest that the complexity of deep reinforcement learning policies does not principally emerge from significant growth in the complexity of functions observed on-and-around trajectories of the policy.
Introduction
Deep reinforcement learning (RL) utilizes neural networks to represent the policy and to train this network optimize objective, typically the expected value of tine-discounted future rewards. Deep RL algorithms have been successfully applied to diverse applications including robotics, challenging games, and increasing number of real-world decision-and-comee control problems [François- Lavet et al., 2018]. For given choice of task, RL algorithm, and policy network configuration, the performance commonly characterised via the learning curves, which provide insight into the learning efficiency and the final performance. However, little has been done to understand the detailed structure of the state-to-action mappings induced by the control policies and how these evolve over time.
In this work, aim to further understand deep feed-forward neural network policies that rectified linear activation functions (rectifier linear units or ReLUs). ReLUs [Nair and Hinton, 2010] are among the most popular choices of activation functions due their practical successes [Montúfar et al., 2014]. For RL, these activations induce a piecewise linear mapping from states to actions,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

80
ke
Figure 1: Schematic illustration of trajectory traversing the piecewise linear regions in the policy state space. So and Sy. indicate the initial and final states the trajectory.
where the input space, i.e., the state space, is divided into distinct linear regions, where within each region, the actions are linear function of the states. Note that the regions are formally affine function of the input, due to the constant-valued bias termss For simplicity and convenience, these are more commonly described as linear regions. Figure provides a schematic illustration of these regions, along with policy trajectory.
The number of distinet regions into which the input space is divided is natural measure of network expressivity. Learned functions with many linear regions have the capacity to build complex and flexible decision boundaries. Thus, the problem of counting the number of linear regions has been extensively studied in recent literature [Montúfar et al., 2014, Raghu et al., 2017, Hanin and Rolnick, 2019a, Serra al., 2018]. While the maximum number of regions is exponential with respect to network depth [Montúfar et al., 2014], recent work has demonstrated that the number of regions is instead typically proportional the number of neurons [Hamin and Rolnick, 2019a].
For RL, we are interested in the local granularity (density) of linear regions along trajectories arising from the policy. Fine-grained regions afford fine-grained control, and thus may hypothesize that region density increases in regions frequently visited by the policy, in order to afford better control. Recent work in supervised learning of image classification is inconsistent with regard to findings about the region density seen the vicinity data points, with some reporting a decrease [Novak al., 2018] provide better generalization and robustness to perturbation, and others not [Hanin and Rolnick, 2019a]. For the RL. setting, we note that counting regions visited along an episode trajectory arguably provides meaningful and task-grounded measurement in contrast to line segments and ellipses that pass through points randomly sampled from training data, which have been used in the prior works mentioned above. We further note that piccewise-affin control strategies are commonly designed into control systems, e.g., via gain scheduling. Understanding how these regions are designed and distributed by deep RL thus helps establish bridges with these existing methods.
To the best of our knowledge, our work is the first to investigate the structure and evolution of linear regions of ReLU-based deep RL policies in detail. We seek to answer several basic empirical questions:
Q1 Do findings for network expressivity. originally developed supervised learning settings, apply to RI. policies? How are the region densities affected by the policy network configu- ration? Do deeper policy networks result in finer-grained regions and hence an increased expressivity?
Q2 How do the linear regions of policy evolve during training? Do greater density of regions emerge along the areas of the state space frequently visited by the episodic trajectories, thereby allowing for finer-grained control? Do random-action trajectories see different densities?
significantly
The key results can be summarized follows, for policies trained using proximal policy optimization (PPO) [Schulman et al., 2017], and evaluated on four different continuous control tasks. Q1: There a general alignment with recent theoretical and empirical results for supervised learning settings. Region density is principally proportional to the number of neurons, with an additional small observed increase in density for deeper networks. Q2: Only a moderate increase of density is observed during training, as measured along fixed final-policy trajectories. Therefore, the complexity of final learned

---

policy does not come principally from increased density on- an-and-around the optimal trajectories, which is a potentially surprising result. In contrast, as measured along the evolving current-police trajectories, decrease in region density is observed during training. Across all settings, we also observe that the region-transition count, as observed during fixed time-duration episodes, grows during training before converging to plateau. However, the trajectory length, as measured in the input space, also grows towards a plateau, although not at the same rate, and this leads to variations in the mean region densities as observed along current trajectories during training.
2 Related Work
Understanding the expressivity of a neural network is fundamental to better understanding its operation. Several works study expressivity of deep neural networks with piccewise linear activations by counting their linear regions [Arora et al., 2016, Bianchini and Scarselli, 2014). On the theoretical side, Pascanu al. [2013] show that in the asymptotic limit of many hidden layers, deep ReLU networks are capable of separating their input space into exponentially more linear regions compared with shallow networks, despite using the same number of computational units. Following this, Montúfar et al. [2014] also explore the complexity of functions computable by deep ferd-forward neural networks with ReLU and maxout activations, and provide tighter upper and lower bounds for the maximal number of linear regions. They show that the number of linear regions is polynomial in the network width and exponential with respect to the network depth. Furthermore, Raghu et al. [2017] improve the upper bound for ReLU networks by introducing a new set of expressivity measures and showing that expressivity grows exponentially with network depth for ReLU and Tanh activated neural networks. Serra et al. [2018) 8) generalize these results by providing even tighter upper and lower bounds for the number of regions for ReLU networks and show that the maximal number of regions grows exponentially with depth when input dimension is sufficiently large.
Recent studies on the expressivity of neural network depth suggest that the actual impact of depth on expressivity is likely far below that of the theoretical maximum proposed by prior literature. Hanin and Rolnick [2019b] study the importance of depth on expressivity of neural networks in practice. They show that the average number of linear regions for ReLU networks at initialization is bounded by the number of neurons raised to the input dimension, and is independent of network depth. They also empirically show that this bound remains tight during training. Similarly. Hanin and Rolnick [2019a] find that the average distance to the boundary of the linear regions depends only on the number of neurons and not on the network depth both at initialization and during training for supervised-leand learninn tasks on ReLU networks. This strongly suggests that deeper networks do not necessarily learn more complex functions in comparison to shallow networks. Prior to this, number of works have shown that the strength of deep learning may arise in part from good match between deep architectures and current training procedures [Mhaskar and Poggio. 2016, Mhaskar et al., 2016, Zhang et al., 1.. Notably, Ba and Caruana [2014] show that once deep networks are trained to perform task successfully, their behavior can often be replicated by shallow networks, suggesting that the advantages of depth may be linked to casier learning.
Another line of work studies function complexity in terms of robustness to perturbations to the input. Sokolić et al. [2017] theoretically study the input-output Jacobian, which is a measure of robustness and also relates to generalization. Similarly, Zahavy et al. [2016b] propose a sensitivity measure in terms of adversarial robustness and provide theoretical and experimental insights on how it relates to generalization. Novak et al. [2018] also study robustness and sensitivity using the input-output Jacobian and the number of transitions along trajectories in the input space as measures of robustness. They show that neural networks trained for image classification tasks are more robust to input perturbations in the vicinity of the training data manifold, due to training points lying in regions of lower density. Several other recent works have also focused on proposing tight generalization bounds for neural networks [Bartlett et al., 2017, Dziugaite and Roy, 2017, Neyshabur et al., 2017].
There are number of works that touch on understanding deep neural networks by finding general principles and patterns during training. Arpit et al. [2017] empirically show that deep networks prioritize learning simple patterns of the data during training. phenomenon in the case of 2-layer networks with Sigmoid activations. Rahaman et al. [2019] study deep ReI.U activated networks through the lens of Fourier analysis and show that while deep neural networks can approximate arbitrary functions, they favor low-frequency ones, and thus, they exhibit a bias towards smooth functions. Samck et al. [2017] present two approaches in explaining predictions
Xu et al. [2019] find similar
3

---

of deep learning models in a classification task, with the first method computing the sensitivity of the prediction with respect to input perturbations and the second method that meaningfully decomposes the decision in terms of input variables.
While deep RL methods are widely used and extensively studied, few works focus on understanding the policy structure in detail. Zahavy et al. [2016a] propose visualization method to interpret the agent's actions by describing the Markov Decision Process as directed graph on t-SNE map. They then suggest ways to interpret, debug and optimize deep neural network policies using the proposed visualization maps. Rupprecht et [2019] train a generative model over the state space of Atari games to visualize states which minimize or maximize given action probabilities. Luo et al. [2018] adapt three visualization techniques the domain of image-basee RL using convolutional neural networks in order to understand the decision making process of the RL agent.
3 Piecewise Linear Regions
Throughout this work, we consider RL policies based on ReLU networks. A ReLU network is ReLU-activated feed-forward neural network, multi-layer perceptron (MLP) which can be formulated as a scalar function f I + Rº defined by neural network with L hidden layers of width d1,..., dz and o-dimensional output. Assuming the output be 1-dimensional, and following the notation of Rahaman et al. [2019), we have
f(x) = ooo 000 o (x),
(1)
where ((k) R -> dd computes the weighted sum (x) (( x + (()) for some weight matrix (K(k) and bias vector (k)), Here, (u) (0, u) denotes the ReLU activation function acting element-wise on vector u = (u,,.., Un.)
Given the ReLU network f from Equation 1, again following Rahaman et al. [2019], piecewise linearity can be explicitly written by
f(x) = E Ei,, (x) b,)
PER
where is the index for the linear region P, and L,, is the indicator function on P.. The x d matrix W. is given by:
W, W
(3)
where Wr(k), is obtained from the original weight (k) by setting its jth column to zero whenever neuron j of layer k-1 is inactive for k >
Counting Linear Regions in RL Policies
To answer the key questions posed in the introduction, we need method for counting the linear regions encountered through the episodic trajectories taken during RL. For each input s, we encode each neuron of the policy network with binary code of 0 if its pre-activation is negative, and with a binary code 1 otherwise. The linear region the input s can thus be uniquely identified by the concatenation of binary codes of all the neurons in the network called an activation pattern. Figure illustrates the activation pattern construction for 2D input space.
Figure 1 provides schematic illustration of the linear regions for 2D state space, together with an example trajectory that consists of sequential states encountered by the policy, connected with straight-ling segments. As seen in the figure, regions can be revisited, which leads to a distinction between the number of transitions between regions and the number of unique regions visited along a trajectory. We compute both of these metries along episodic trajectories, enabled by keeping a list

---

#
c = 8
activation pattern
[11100011101]]
Figure 2: An overview of ReLU-activated policy network and the binary labeling scheme of the linear regions. State [r. ai is within the linear region uniquely identified by the activation pattern computed by concatenating the binary states of the activations of the policy network given input s.
of the regions already visited at any point in time when processing a given episode trajectory. Our region-counting method also includes regions that are crossed by the straight- line segments between successive trajectory states, although the policy itself does not explicitly encounter these regions due to the discrete-time nature of typical RL environments.
To represent the k-th line segment of the episodic trajectories in the input space, we develop parameterized line segment given by s(u) (1- (1-=)ss=1+f usk, where [0. 1] and $;-1, 82 R denote the endpoints of the line segment. We then calculate the exact number of the linear regions over this line segment by considering the hidden layers the policy network one at a time, starting from input towards the output, and observe how each region can be split by the neurons. Starting with the first layer, we consider neurons one by one, and identify the point in the domain of u, if any, that induces change in the binary labeling for that neuron. Each such point subdivides the domain of into two new regions where the pre-activation will be zero for one of them in the next layer. By maintaining a list of these regions and the linear functions defined over them, and whether their pre-activation vanishes in the next layer, we proceed to the neurons of the next layer. This process repeats for each of the regions resulting from the previous layer. We record the activation patterns of all the final regions, to track state visits during all segments of episode trajectories. In the end, for each trajectory т, we compute the total region transitions, (T)( well as the number of unique visited regions, (T)(7), where RT(T) + Z Ry (T). We further compute the trajectory length in the input space, L(r). This allows us compute normalized region densities according to p(T) = PRr(T)/(NLL) in accordance with Hanin and Rolnick [2019a].
where N is the total number of neurons in the policy network,
We use the above metrics along two types of trajectories. First, we consider fixed trajectories, T" as sampled from the final fully-trained policy. Second, we consider current trajectories, т, sampled from the current snapshot of the policy during training. Both these trajectories offer informative views of the evolution of the policy. The former offers direct picture of the linear-region density along a meaningful, fixed region the state space, i.e., that of the final optimized policy. The latter offers view of what the policy trajectories actually encounter during training. In order better understand the evolution of т, we also track its length, L(T), given that the density is determined by the region-transition count well the length of the trajectory. Figure 3 shows the evolution of linear regions and the types of trajectories r" and - for a simple 2D toy environment, and a ReLU-activated policy network of depth 2 and width
Experimental Results
conduct our experiments on four continuous control tasks including HalfCheetah-v2. Walker-v2, Ant-v2, and Swimmer-v2 environments from the OpenAl AI gym benchmark suits [Brockman et al., 2016]. We use the in all of our experiments throughout this work We run each experiment with 5 different random seeds and the results show the mean and the standard deviation across the random seeds. The policy network architecture is MI.P with ReLU activations in all hidden layers in experiments.
implementations of the PPO algorithm [Schulman et al., 2017]
Our code is available attps://githuu: wtt.oa/..oooo/eeeteeeeeeee http://ggttthcooooe/etttteeeeepp
5

---

Epoch 0
Epoch 10
Epoch 20
Epoch 100
Figure 3: State space visualization with current (r) and fixed trajectories (r") of a ReLU-activated policy network with depth 2 and width 8 trained on a 2D toy environment during training. The first row, plots fixed trajectory sampled from the final policy over the state space. The second row plots a trajectory sampled from the current snapshot of the policy. Columns indicate the number of training epochs. Epoch 0 corresponds the policy initialization and Epoch 100 corresponds the mully-trained policy. In this toy environment, an agent starts at the origin indicated with blue dot and the goal is for the agent to reach a target state located at x 100. States are the position æ (horizontal axis) and velocity ă (vertical axis) of the agent, and action is the acceleration.
We train 18 policy network configurations with N E {32, 48, 64, 96, 128, 192) neurons, widths w E (8, 16, 6-4], and depths d E 3, 3.4. We use fixed value function network structure of (64, 64) in all of our experiments. Details of the network configurations used for the policy network are available in Appendix C. These network configurations are chosen such that the network is fully-capable of learning the task and achieves near state-of-the-an cumulative reward the particular task it is trained on. This allows us to study the evolution of linear regions disentangled from properties such trainability, which is outside the scope of this work. The table showing the range of the mean cumulative reward per task, across all network configurations can be found in Appendix B.
We adopt the network initialization and hyperparameters of PPO from StableBaselines3 [Raffin et al., 2021] and train our policy networks on 2M samples (i.e. 2M timesteps in the environ- ment). This initialization scheme also matches the best-practice recommendations for on-policy RL, c.g., [Andrychowicz et al., 2020], although it differs from that used in Hanin and Rolnick [2019a] for theoretical and empirical linear region analysis. A detailed explanation the initialization and choice of hyperparameters is available in Appendix D.
In what follows, we focus on the results for the HalfCheetah-v2 environment as representative example. Similar results for the other environments are provided in Appendix K.
5.1 Evolution of the Density of Linear Regions during Training
We begin by examining the number of transitions encountered as we sweep along the fixed final- trajectory, Ry as shown in Figure 4(a). For clarity, we present results for only three policy network configurations. This plot shows moderate and gradual increase of 50% in the number of region transitions observed fixed-duration RL episodes during training, with larger policy networks having more transitions. Computing the region density requires dividing by number of neurons, N, and the trajectory length, L(r*). which constant for the fixed final trajectory, T". The result is given in Figure 4(b) which shows the mean and standard deviation of the normalized region densities across policy networks with the same depth. This shows a result that is consistent across all policy network dimensions to within +30% and reflects an identical moderate-and antrraaaaanngraauuall increase during training, regardless of the policy network configuration. Thus, in response to Q2, RL policies trained with PPO moderate increase in region density along the parts of the state space that frequently visited by the policy. We later discuss the observed impact of policy-networt depth ($5.2).
We next examine the number of transitions encountered as sweep along the evolving set of current-policy trajectories, Rr(r), shown Figure 4(c). These begin near zero and then rapidly increase as the policy succeeds at exploration. Again, for clarity, show representative sample of

---

90000
Nunler of Laaundiion
Region detuity
Epoch
Epoch
(a) Transition count, trajectory
(b) Region density, fixed trajectory
50200
0.90
di beuurnnn
Number of Loueeliknn
curret
40200
Region deasity
dir-eurreut
30200
d-1-ixed
0.13
dis Hood
20200
0:10
dms dm--fiood
10000
120
Rpoch
Egoch
Transition count, current trajectory T
(d) Region density, and
Number of repoott visites to rogiotts
[86.0]]
23000
Trajectxy jangth
[61.64]
20000
[6664]
25000
11000
10200)
5000
5000
Epoch
Epoch
(e) Trajectory length, current trajectory T
(f) Repeat visits to regions, current trajectory т
0.00)
14.14
Region dennlly
Rlgionn density
0.04
3.1
0.002
0.000
0.9
0,002
20
10
100
1.00
120
Epoch
(g) Density for random lines through the origin
(h) Density for random-action trajectories
Figure 4: Evolution of the number of transitions, lincar region densities, and length of trajectories for HalfChee- tah, during training, Results the rest the environments are provided in Appendix The ranges indicate the standard error across 5 random seeds. [re,,.., raa] in the legend corresponds to network architecture with depth and nı neurons each layer. Summary results are grouped by network depth. d.
three policy network dimensions here. The corresponding region densities shown in Figure 4(d),
7

---

together with the fixed-region densities which are repeated from Figure 4(b) (see the bottom dashed curves). We speculate that the observed densities are higher early on due to the unstructured nature of early exploration and the form of network initialization, and then decrease over time as the trajectory lengths begin to increase, as shown in Figure 4(e).
We compute the number of repeat visits to regions as - i.c., the region transition count minus the unique region count. In Figure 4()) we track this for the current trajectory during training. We speculate that repeat visits are high early in training because of limited exploration. Repeat visits remain later in training because of the cyclic nature of the learned locomotion for HalfCheetah.
5.2 Impact of Policy Network Structure on Expressivity
Our normalized density computation assumes that the number of transitions scales linearly with the number of neurons, N. However, we empirically observe that there remains a moderate dependence on the network depth, as seen in Figure 4(b), with deeper networks resulting in moderately denser regions in the learned policies. We further examine the policy network expressivity using random infinite lines. Following Hamin and Rolnick [2019a], for each policy network, we sample 100 random lines by randomly sampling 100 points from the final trajectory r", and connecting each randomly sampled point the origin of the policy input space. Figure 4(g) shows the mean and standard deviation of the normalized number of transitions, R as computed along these infinite lines in random directions passing through the origin, over the random seeds of each network configuration. These produce approximately uniform results throughout training, showing no overall changes in mean density in various directions. These results are also consistent with the findings of Hanin and Rolnick [2019a] as they show normalized region densities are close to I at initialization and remain roughly constant during training. We repeated the same experiment with 100 random lines through the mean point instead of the origin, by connecting each randomly sampled point to the mean of the 100 sampled points. The difference between the density over lines through the origin and through the mean point is negligible.
To further examine the behavior of the policy network on areas that does not focus on exploring during training, we compute the region density along random trajectories created by taking random actions the environment. For each policy network, we sample a set of 10 such random trajectories 7 and compute the normalized region density over these. Figure 4(h) shows the mean and standard deviation of this metric across this set of random trajectories, over the 5 random seeds of each network configuration. This shows a very moderate decrease during training. with fairly consistent densities for all policy network configurations. We speculate that the slight decrease in density over time arises from the trained policy no longer visiting the same regions the madommacciion policy. Comparing the results of random trajectories with fixed and current (Figure 4(b) and Figure 4(d)), we see that the normalized region density over random-action trajectories and fixed final trajectory are within the same range. Also, although this metric is higher over current trajectories early on during training, it eventually decreases and converges to the same range of region densities as the fixed trajectory. In Appendix G we further directly visualize the regions of a policy network for HalfCheetah, as viewed via an embedding plane that either passes through three points sampled from the final fixed trajectory, or three points sampled from a random-action trajectory. We do not observe significant differences between these two visualizations.
5.3
Additional Experiments
To further study the generalization of our observations, we perform four additional experiments.
5.3.1
RL Algorithm
To test how our findings generalize RL algorithms other than PPO, we repeat our experiments by training deep RL policies with the stochastic actor-critic (SAC) [Haarnoja et al., 2018] algorithm. The experimental setup for SAC and its results are documented in detail in Appendix F. Region densities observed early on during training are quite different for SAC than they are for PPO, exhibiting high observed densities which then rapidly drop, before rising again like the PPO case. We hypothesize this is due to combination of (i) the entropy bonus for SAC, which is then annealed away, and (ii) the different network initialization used for the baseline SAC and PPO implementations. Despite this initial difference, the evolution patterns of densities consistent with the PPO results later on during
8

---

the course of training. Another observation is that similar to PPO results, densities over fixed. current, and sundom-action trajectories are within the same range. This supports our surprising finding that the complexity of RI. policies is not principally captured by increased density on-and-around the optimal trajectories.
5.3.2 Environment
We examine whether a non-cyclic task, such as LunarLander, exhibits different linear-region evolution behavior compared our four default tasks, which are naturally biased towards eyclic locomotion. From the results which are available in Appendix H, we can see that the transition counts and densities along the fixed trajectory are similar in structure to those of the cyclic locomotion tasks, showing that a gradual increase in density appears to be general property for the PPO setting, even for non-cyclic tasks.
5.3.3 Value Network Analysis
To study the difference between the value space and policy space, we repeat our experiments on the value networks trained on HalfCheetah with PPO. Full of results for this experiment is available in Appendix I. Our results show that linear regions in the value functions largely evolve in a very similar way those in the policy.
5.3.4 Behavior Cloning
Deep RL provides us with a grounded setting to explore the impact of non-llI data on decision regions. To test if the non-III setting makes difference, we repeat our experiments by training networks with behavior cloning (BC) using expert data from previously trained policies with PPO on HalfCheetah. The experimental setup for BC and the full results are documented in detail in Appendix J. These results show that the evolution of region densities is different for BC than they for PPO as the general increase in densities is not observed. In addition, number of transitions, density values, and length of current trajectories are significantly smaller for BC-trained policies. We have two hypotheses for these differences: (i) Policy's 's learning history plays a significant role in the evolution of regions, as early trajectories inform the template cell divisions that later evolve with further training. Since BC policies observe the entire state space from the moment training starts, they may be able to better divide their state space and avoid adding too much granularity to certain areas. (ii) Network initialization largely affects the resulting linear regions and their evolution during training.
6 Conclusions
The structure of ReLU-based reinforcement learning policies remains poorly understood. In this work, we explored a key property of such policies, the density of linear regions observed along policy trajectories, as a proxy for the complexity of functions they can learn. We performed a set of experiments to see how this evolves during training and whether and how changes in depth and width of the policy network affect it. The hypothesis that the final policy trajectory sees an increase in region density is supported to a limited degree, as we observe a moderate increase in density during training, i.e., 50%. This result is consistent across tasks and across policy networks of different configurations. When observed from the vantage point of the evolving current trajectory. the region density is high to begin with, due to the limited length of early trajectories, and then decreases asymptotically towards the observed final density. For the tasks we consider, trajectory lengths are observed to grow and asymptotically approach a platcau during training. Overall, the surprising implication, for our experimental setting, is that the complexity of RL policies appears to be captured by fine-tuning during learning, rather than a significant growth in the complexity of the functions observed along trajectories of the final policy.
The key properties of linear regions as demonstrated for supervised learning of image classifiers, e.g.. [Hanin and Rolnick, 2019a], are shown to generally also hold true for reinforcement learning policies. In partieular, their expressivity well modeled as being proportional to the number of neurons, and largely independent of their width and depth, for ranges of these parameters where good policies can still be learned. The number of regions observed for randomly oriented line segments through the origin of the input space is also in line with earlier thcoretical and empirical results, and does not

---

change significantly during training. On a different note, properties of linear regions measured along policy trajectories observed for policies trained with RL. do not hold for policies trained with BC. This emphasizes the unique behavior of RI. compared to its supervised learning counterpart.
Our work comes with a number of limitations. Our experiments mainly focused locomotion- based tasks which produce cyclic trajectories. This could mean that the results may not generalize other types of tasks, For example, some tasks may not result transition counts that grow, as generally observed in our experiments. In addition, our work studies deep RL policies trained with PPO and SAC. Thus, broader analysis of more RL algorithms would shed light on the generality of the observations in this work. Since network initialization largely affects the linear regions, it would be ideal to use the same initialization when comparing different RL algorithms. However, RL algorithms are typically sensitive to network initialization, and changes in common practices in their initialization results in instabilities during training. Currently, we compute only the mean region density for trajectory: it may be useful to further understand the distribution of observed densities across trajectories. We note that it is common practice in RI. to apply input (observation) normalization based rolling average of observations seen so far [Andrychowicz al., 2020], and thus our results also employ this strategy, Applying RL without this normalization may impact both the performance of the RL algorithm and the resulting evolution of transition counts and region density.
Our work suggests opportunities for improving our understanding of deep reinforcement learning through studying how states are mapped to actions and how these mappings evolve during training. For instance, number of questions for future work include understanding the behavior of observed linear regions in the contexts of other RL. algorithms, and behavior cloning. Moreover, we envisage various future directions related to this work, including: (i) understanding the impact of non-llI) data on decision regions more generally; (ii) consideration of RL algorithms that have less "learning path dependence", e.g., via frequent (re)distillation onto randomly reinitialized policies; (iii) using visited regions to efficiently distill more interpretable non-parametric policies; and (iv) in real-world control settings, using knowledge of decision regions, their sizes, and their orderings to better understand exploration and safety issues. Lastly, we believe that exploring possible connections with the lottery-tickel hypothesis [Frankle and Carbin, 2018] will be worthwhile.
As foundational research related to reinforcement learning, this work could be used to develop improved reinforcement learning policy structures and algorithms, with many possible applications, both good and bad in terms of societal impact.
Acknowledgments
This work was supported by the NSERC grant RGPIN-2020-05929, David Rolnick acknowledges support from the Canada CIFAR AI Chairs program. This research was enabled in part by technical support and computational resources provided by Compute Canada (www. cwwputecanada. ca).
10

---

References
Marcin Andrychowicz, Anton Raichuk, Piotr Stañczyk. Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? large-scale empirical study. arXiv preprint arXiv:2006.05990 2020.
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv: 1.01491, 2016.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International Conference on Machine Learning, pages 233-242. PMLR, 2017.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2014.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. neural networks. Advances neural information processing systems, 30, 2017.
margin bounds for
Monica Bianchini and Franco Scarselli. On the complexity of neural network classifiers: compari- son between shallow and deep architectures. IEEE transactions neural networks and learning systems, 25(8): 1553-1565, 2014.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606 01540, 2016.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint vrXiiv:773. 1008, 2017.
Vincent François-Lavet, Peter Henderson, Riashat Islam, Mare G Bellemare, and Joelle Pineau. An introduction to deep reinforcement learning. arXiv preprint arXiv:1811. 2018
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: arXiv:/803.00 1803.80033.333 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference Machine Learning, pages 2596-2604. PMLR, 2019a.
Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems, 32, 2019b.
Jieliang Luo, Sam Green, Peter Feghali, George Legrady, and Cetin Kaya Kog. Visual diagnostics for deep reinforcement learning policy development. arXiv preprint arXiv:/809.0678. 2018.
Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio. Learning functions: when is deep better than 2016.
shallow. arXiv preprint
Hrushikesh N Mhaskar and Tomaso Poggio. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14(06):829-844 848, 2016.
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. arXiv preprint 4rXiv:1402.1869, 2014.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generaliza- tion in deep learning, Advances in neural information processing systems, 30, 2017.
11

---

Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study, arXiv preprint arXiv:1802.08860, 2018.
Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wiss wise linear activations. arXiv preprint arXiv:1312.6098, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Sable-baseeinnes, Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
Maithra Raghu, Ben Poole, Jon Kleinberg. Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference machine learning, pages 2847-2854. PMLR, 2017.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
Christian Rupprecht, Cyril Ibrahim, and Christopher Pal. Finding and visualizing weaknesses of deep reinforcement learning agents. arXiv preprint arX4v:1904.011 2019.
Wojciech Samek, Thomas Wiegand, and Klaus- Müller. Explainable artificial intelligence: Un- derstanding, visualizing and interpreting deep learning models. arXiv preprini arXiv: 170.00/008. 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks. In International Conference on Machine Learning, pages 4558-4566. PMLR, 2018.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65(16):4265-4288, 2017.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In International Conference on Neural Information Processing, pages 264-274. Springer, 2019.
Tom Zahavy, Nir Ben-Zrihem and Shie Mannor. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pages 1899-1908. PMLR, 2016a.
Tom Zahavy, Bingyi Kang, Alex Sivak, Jiashi Feng, Huan Xu, and Shie Mannor. Ensemble robustness and generalization of stochastic deep learning algorithms. arXiv preprini arXi::1602.02388) 2016b.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] Code is available at: https:/ /et:://gttbbbbbt com/ wtwpareeeceeeee..reeiio......
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We show mean/std bars. See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] Our experiments require modest compute resources.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vSQmWl6R6b0", "reviewer": "Reviewer_5UGR", "summary": "The authors aim to describe the evolution of the linear regions that takes place in a policy parameterized by a neural network while being optimized via reinforcement learning. ReLU activations make calculating these regions straightforward, but until now these sorts of analyses have been limited to supervised learning. Reinforcement learning is different in that it involves trajectories, which makes questions around region densities more meaningful (don't need to interpolate between IID datapoints).\n\nThe author investigate the effect of depth on the number of unique regions as well as how these regions evolve over the course of learning. Both of these questions are complicated by the choice of trajectory distributions; the authors do these analyses for both the current trajectory distribution as well as that induced by the final (near optimal) policy.\n\nExperiments are conducted across 4 locomotion tasks, with policies optimized via PPO. They show that while the final trajectories have their region density increase over time, the region density of the current policy's trajectories actually decrease after a large initial spike.", "questions": "1) Would these results hold for other RL algorithms and environments?\n\n(The below questions are more speculative, I don't expect you to run more experiments or anything)\n\n2) How would these results differ for behavior cloning? Do you think this is more more sequence-learning or RL specifically?\n\n3) Would more intelligent (or even optimal) exploration result in current-policy region density increasing monotonically?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1) This is quite a detail-oriented paper, so it's to the authors' credit that I was able to follow along quite easily. The writing quality was great and the related-works section in particular headed off many of my initial questions about this work's relationship to the rest of the field.\n\n2)The high level of idea of bringing analyses developed for supervised deep learning into DRL is well-worth pursuing, particularly given the points made about trajectory-based region density estimation being more natural than in the IID setting.\n\n3) The effect sizes are very strong and quite consistent across architectures. The follow-up experiment on repeat visits to regions is compelling evidence in favor of explanation of the surprising result that region density decreases over time for the current policy (i.e. it's due to poor initial exploration).\n\n# Weaknesses\n\n1) Having a single figure in the main paper that aggregates all of the key results across environments is sorely needed. It's not quite obvious to me how to do this since the y-axis scales might be significantly different, but I'm sure some sort of normalization scheme could allow us to see that the shape of the results are similar across environments.\n\n2) I'm not confident that these results will generalize across environments, which limits the potential impact of this work. To the authors' credit, they acknowledge that focusing on locomotion environments is limiting, but that begs the question as to why they imposed this limitation. I believe the code-base you're using could just as easily be applied to e.g. manipulator environments, and doing so would build confidence in the applicability of your results.\n\n3) Similarly, picking a single RL algorithm feels limiting. Something purely value-based (e.g. Q-learning) would be particularly interesting, as value-space is very differently behaved compared to policy-space. Indeed, even just redoing your analyses on the value network of your PPO agents would be quite interesting.\n\n4) Figure 5 is very pretty, but I'm not sure what it is supposed to be communicating.\n\n5) Some more motivation about why we should care about the outcome of Q1 and Q2 would be nice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pYeIt8m-2BK", "reviewer": "Reviewer_xjsV", "summary": "This paper studies the way neural network policies with RELU activations partition their input space into linear regions. This has been studied before in supervised learning settings, but not for RL policies yet. The authors study 4 continuous control tasks. The main empirical observation is that the complexity of the policies comes from fine-tuning their contribution, not in a significant growth in their number along trajectories of the final policy. The number of regions seems mostly related to the total number of neurons, a finding also observed in supervised learning. ", "questions": "I do not fully understand what I need to learn from the “region densities”, or what you actually compute here. As I understand, you take the total visitation counts per visited region, divide these over (N * L(tau)), and then you average this number? Doesn’t this really just estatimate the number of unique regions you visited? \n* For example, imagine I visited 4 regions, with counts [100,200,300,400], and assume N=10 and L(tau)=10. This would give densities [1,2,3,4], and therefore average density 2.5. \n* Now assume I instead observed counts [200,200,300,300]. This would give densities [2,2,3,3], and therefore still average density 2.5. Whatever way I distribute the counts over the regions, if the total number of regions stays the same, the average density will as well. \n* I think what I would be really interested in is the total number of uniquely visited regions, if possible in a smaller problem with a density-coloured map (instead of averaged over all regions, which throws away a lot of information). \n\n\n", "limitations": "Conclusion: This is a decent paper. I like that the authors focus on a conceptual/insight topic in RL, which is often missing. On the other hand, I do think these results are close to what was already known from supervised learning, and I slightly miss the motivation of the authors why things could be different in RL (nevertheless, always worth investigating). I am also a bit in doubt about the way the results are presented, especially regarding the density estimation: this could maybe be clearer. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong:\n* I think conceptual insight studies are always relevant. As the authors mention as well, in RL we usually only look at the learning curves, which is a pity and probably misses much relevant insight. \n* The overall paper is clearly written and easy to follow. \n* I like Figure 3 (although you could maybe separately draw the Toy environment), it provides good insight. \n\nWeak:\n* See my comment in the Questions section. \n* The authors only study four low-dimensional continuous control tasks, which arguably does not give a lot of variety. I therefore think the conclusions are a bit strong. I would have much preferred if there were 1 or 2 tasks from the Mujoco set (with indeed many cycles), but also some other types of tasks to validate and broader the scope of the conclusions. I think the current conclusions need to be downplayed a bit. \n* The results confirm what we already knew from similar analysis in supervised learning. Of course, one still has to try in the RL setting to confirm this, but I think the paper needs to discuss why we would expect things to be different in the RL case? (I would not a clear reason: RL is mostly a data generation and target computation problem, after that the NN training process is simply a supervising learning problem). \n\nSmaller: \n* Figure 2 needs more explanation ( a better caption) to understand what is going on. \n* I printed your version and all references were invisible (due to the very light green color, I would pick something darker). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mqbwC_AnUnQ", "reviewer": "Reviewer_yuJc", "summary": "Summary\n-------\n\nTo better understand (relu-based) neural network policies in RL, this\npaper proposes to measure the counts and densities of linear regions\nduring training. Using continuous control (MuJoCo) environments, the\nauthors empirically measure the linear regions' visitations (and other\nquantities) while training PPO. They find that the evolution of the\npolicy's linear regions can be seen to increase for fixed trajectory\nlengths, in accordance with results in supervised learning. For\non-policy and current trajectories, however, the density count decreases\nduring training.\n\n\n\n", "questions": "\nDetailed Comments\n-----------------\n\n-   Line 25-26: Some environments do allow for interpreting changes to\n    policy over time. This is usually done for tabular environments\n    however, such as gridworlds, and at the state-level rather than\n    activation-level as in your submission. See, for example, Pardo et.\n    al (2018), \"Time Limits in RL\".\n-   Line 166: Is it more informative to count regions that are not\n    explicitly encountered? This seems like it would lead to an\n    overestimate.\n-   Line 169-184: I think the ideas here would be better described in a\n    figure/diagram or even an algorithm, rather than in a paragraph.\n-   Line 185: Is there a specific reason for using the final policy as\n    the \"fixed\" trajectory? This would presumably be the longest\n    trajectory, but what exactly is a \"fully-trained\" policy.\n-   Line 219: Fixed final trajectory. The policy is fixed, but how is\n    the trajectory fixed? The policy itself is stochastic, so the\n    trajectory cannot be fixed unless you take only the greedy action.\n    Even then, there will be some variation in the start state that\n    introduces variation in the trajectory.\n-   Figure 4b: how is the change in policy measured along a fixed\n    trajectory? The x-axis goes over epochs, which suggest that this is\n    measured during training. Is the \"fixed trajectory\" fixed before\n    training?\n-   Figure 5: Is there any expectation that this coarse grained (only 3\n    points) visualization should show significant differences? I think\n    this may be more clear in a truly 2 dimensional problem, such as an\n    LQR.\n-   Line 243: It is not clear that revisiting a linear region in the\n    neural network is the same as revisiting states, i.e. exploration.\n\nMinor Comments\n--------------\n\n-   Figure 1: Minor, but can you clarify whether this this schematic\n    isrepresenatitive of an actual problem? The 4th and 5th dot occur in\n    the same region and so, should have the same action. And yet the\n    transitions to the next dot look quite different. If this is just an\n    illustration, a clarification would help", "limitations": "The limitations of the work are discussed in some detail in the last few paragraphs. The authors also acknowledge that algorithmic developments can be used for both good and bad, but that this paper does not necessarily contribute to an area that requires ethics review. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Decision\n--------\n\nWhile I find this paper very interesting in the problem it investigates,\nit is currently below the acceptance threshold. The main problem is that\nthe empirical investigation, while interesting, lacks a depth of\nanalysis (especially relative to the paper that it builds upon). A\ntheoretical analysis of the learning dynamics, while outside of the\nscope of this paper, would also serve to strengthen its contribution.\nAnother thing that would help is to better characterize the difference\nbetween RL and supervised learning (this is somewhat addressed in the\ntrajectory lengths, but the larger issue of temporality and changing\nstate distribution is missing). For this paper to meet the acceptance\nthreshold, it would need either: a deeper analysis on perhaps a simpler\nproblem, a broader analysis on more RL algorithms, or a theoretical\nstudy. These changes, however, are outside the scope of a revision.\n\nStrengths\n---------\n\n-   Interesting and novel investigation on the structure of relu-based\n    networks in on-policy and continuous control.\n-   Positioned well with respect to previous work in this area,\n    especially in supervised learning.\n\nWeaknesses\n----------\n\n-   Results, while interesting, do not shed much light on the\n    peculiarities of RL policies. There are many differences between\n    supervised learning and RL that is not well reflected in the problem\n    being studied (namely PPO). There are many environments invesigated\n    (in the appendix, at least). But, for an experimental paper, I would\n    expect a more thorough evaluation using more algorithms or some\n    specific hypothesis/justification for the limitation to PPO.\n-   Lack of theory relative to other work in this area. Compared to\n    supervised learning, this is obviously very difficult to do in RL.\n    With the lack of \"exciting\" empirical results, however, theory could\n    fill in this gap by providing the framework for thinking about\n    relu-based neural networks and their use in RL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PUTUr5KVlkJ", "reviewer": "Reviewer_zgAD", "summary": "The paper attempts to understand how the activations of the agent’s policies (represented as a neural network) evolves during training on standard continuous control tasks. The hypothesis is that the number of regions partitioned (i.e., density) increases in areas where the policy visits often in order to afford finer control for the learning agent in those regions. The paper uses recent theoretical results and empirical observations from supervised learning literature to draw conclusions in RL.\n", "questions": "* The paper and method used to evaluate the neuron activations in a policy network are interesting. But it is hard to understand how to use those observations in informing future research in RL. A discussion on why these findings are important to RL or more generally in Deep RL would be valuable.\n* Were the linear regions different from the earlier layers to the later layers of the policy network? Were they task specific or agnostic to the considered tasks?\n* Can we expect similar observations to emerge on agents like DQN?\n", "limitations": "The paper presents an interesting analysis of existing policy learning algorithms in terms of its neuron activations. But it is unclear on how to apply these findings in future research.\n", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-presented and the results are interesting. It is an analysis method to understand the RL agents. \n* Empirical results delve into how the activations within the neural network evolve with training on standard RL tasks.\n\n\nWeaknesses:\n* It is unclear as to how the observations made in this paper inform future research. A discussion around this would be much appreciated.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept