Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper attempts to understand how the activations of the agent’s policies (represented as a neural network) evolves during training on standard continuous control tasks. The hypothesis is that the number of regions partitioned (i.e., density) increases in areas where the policy visits often in order to afford finer control for the learning agent in those regions. The paper uses recent theoretical results and empirical observations from supervised learning literature to draw conclusions in RL.

strengths_and_weaknesses: Strengths:
* The paper is well-presented and the results are interesting. It is an analysis method to understand the RL agents. 
* Empirical results delve into how the activations within the neural network evolve with training on standard RL tasks.


Weaknesses:
* It is unclear as to how the observations made in this paper inform future research. A discussion around this would be much appreciated.

questions: * The paper and method used to evaluate the neuron activations in a policy network are interesting. But it is hard to understand how to use those observations in informing future research in RL. A discussion on why these findings are important to RL or more generally in Deep RL would be valuable.
* Were the linear regions different from the earlier layers to the later layers of the policy network? Were they task specific or agnostic to the considered tasks?
* Can we expect similar observations to emerge on agents like DQN?

limitations: The paper presents an interesting analysis of existing policy learning algorithms in terms of its neuron activations. But it is unclear on how to apply these findings in future research.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors aim to describe the evolution of the linear regions that takes place in a policy parameterized by a neural network while being optimized via reinforcement learning. ReLU activations make calculating these regions straightforward, but until now these sorts of analyses have been limited to supervised learning. Reinforcement learning is different in that it involves trajectories, which makes questions around region densities more meaningful (don't need to interpolate between IID datapoints).

The author investigate the effect of depth on the number of unique regions as well as how these regions evolve over the course of learning. Both of these questions are complicated by the choice of trajectory distributions; the authors do these analyses for both the current trajectory distribution as well as that induced by the final (near optimal) policy.

Experiments are conducted across 4 locomotion tasks, with policies optimized via PPO. They show that while the final trajectories have their region density increase over time, the region density of the current policy's trajectories actually decrease after a large initial spike.
strengths_and_weaknesses: # Strengths

1) This is quite a detail-oriented paper, so it's to the authors' credit that I was able to follow along quite easily. The writing quality was great and the related-works section in particular headed off many of my initial questions about this work's relationship to the rest of the field.

2)The high level of idea of bringing analyses developed for supervised deep learning into DRL is well-worth pursuing, particularly given the points made about trajectory-based region density estimation being more natural than in the IID setting.

3) The effect sizes are very strong and quite consistent across architectures. The follow-up experiment on repeat visits to regions is compelling evidence in favor of explanation of the surprising result that region density decreases over time for the current policy (i.e. it's due to poor initial exploration).

# Weaknesses

1) Having a single figure in the main paper that aggregates all of the key results across environments is sorely needed. It's not quite obvious to me how to do this since the y-axis scales might be significantly different, but I'm sure some sort of normalization scheme could allow us to see that the shape of the results are similar across environments.

2) I'm not confident that these results will generalize across environments, which limits the potential impact of this work. To the authors' credit, they acknowledge that focusing on locomotion environments is limiting, but that begs the question as to why they imposed this limitation. I believe the code-base you're using could just as easily be applied to e.g. manipulator environments, and doing so would build confidence in the applicability of your results.

3) Similarly, picking a single RL algorithm feels limiting. Something purely value-based (e.g. Q-learning) would be particularly interesting, as value-space is very differently behaved compared to policy-space. Indeed, even just redoing your analyses on the value network of your PPO agents would be quite interesting.

4) Figure 5 is very pretty, but I'm not sure what it is supposed to be communicating.

5) Some more motivation about why we should care about the outcome of Q1 and Q2 would be nice.
questions: 1) Would these results hold for other RL algorithms and environments?

(The below questions are more speculative, I don't expect you to run more experiments or anything)

2) How would these results differ for behavior cloning? Do you think this is more more sequence-learning or RL specifically?

3) Would more intelligent (or even optimal) exploration result in current-policy region density increasing monotonically?
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Summary
-------

To better understand (relu-based) neural network policies in RL, this
paper proposes to measure the counts and densities of linear regions
during training. Using continuous control (MuJoCo) environments, the
authors empirically measure the linear regions' visitations (and other
quantities) while training PPO. They find that the evolution of the
policy's linear regions can be seen to increase for fixed trajectory
lengths, in accordance with results in supervised learning. For
on-policy and current trajectories, however, the density count decreases
during training.




strengths_and_weaknesses: Decision
--------

While I find this paper very interesting in the problem it investigates,
it is currently below the acceptance threshold. The main problem is that
the empirical investigation, while interesting, lacks a depth of
analysis (especially relative to the paper that it builds upon). A
theoretical analysis of the learning dynamics, while outside of the
scope of this paper, would also serve to strengthen its contribution.
Another thing that would help is to better characterize the difference
between RL and supervised learning (this is somewhat addressed in the
trajectory lengths, but the larger issue of temporality and changing
state distribution is missing). For this paper to meet the acceptance
threshold, it would need either: a deeper analysis on perhaps a simpler
problem, a broader analysis on more RL algorithms, or a theoretical
study. These changes, however, are outside the scope of a revision.

Strengths
---------

-   Interesting and novel investigation on the structure of relu-based
    networks in on-policy and continuous control.
-   Positioned well with respect to previous work in this area,
    especially in supervised learning.

Weaknesses
----------

-   Results, while interesting, do not shed much light on the
    peculiarities of RL policies. There are many differences between
    supervised learning and RL that is not well reflected in the problem
    being studied (namely PPO). There are many environments invesigated
    (in the appendix, at least). But, for an experimental paper, I would
    expect a more thorough evaluation using more algorithms or some
    specific hypothesis/justification for the limitation to PPO.
-   Lack of theory relative to other work in this area. Compared to
    supervised learning, this is obviously very difficult to do in RL.
    With the lack of "exciting" empirical results, however, theory could
    fill in this gap by providing the framework for thinking about
    relu-based neural networks and their use in RL.
questions: 
Detailed Comments
-----------------

-   Line 25-26: Some environments do allow for interpreting changes to
    policy over time. This is usually done for tabular environments
    however, such as gridworlds, and at the state-level rather than
    activation-level as in your submission. See, for example, Pardo et.
    al (2018), "Time Limits in RL".
-   Line 166: Is it more informative to count regions that are not
    explicitly encountered? This seems like it would lead to an
    overestimate.
-   Line 169-184: I think the ideas here would be better described in a
    figure/diagram or even an algorithm, rather than in a paragraph.
-   Line 185: Is there a specific reason for using the final policy as
    the "fixed" trajectory? This would presumably be the longest
    trajectory, but what exactly is a "fully-trained" policy.
-   Line 219: Fixed final trajectory. The policy is fixed, but how is
    the trajectory fixed? The policy itself is stochastic, so the
    trajectory cannot be fixed unless you take only the greedy action.
    Even then, there will be some variation in the start state that
    introduces variation in the trajectory.
-   Figure 4b: how is the change in policy measured along a fixed
    trajectory? The x-axis goes over epochs, which suggest that this is
    measured during training. Is the "fixed trajectory" fixed before
    training?
-   Figure 5: Is there any expectation that this coarse grained (only 3
    points) visualization should show significant differences? I think
    this may be more clear in a truly 2 dimensional problem, such as an
    LQR.
-   Line 243: It is not clear that revisiting a linear region in the
    neural network is the same as revisiting states, i.e. exploration.

Minor Comments
--------------

-   Figure 1: Minor, but can you clarify whether this this schematic
    isrepresenatitive of an actual problem? The 4th and 5th dot occur in
    the same region and so, should have the same action. And yet the
    transitions to the next dot look quite different. If this is just an
    illustration, a clarification would help
limitations: The limitations of the work are discussed in some detail in the last few paragraphs. The authors also acknowledge that algorithmic developments can be used for both good and bad, but that this paper does not necessarily contribute to an area that requires ethics review.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the way neural network policies with RELU activations partition their input space into linear regions. This has been studied before in supervised learning settings, but not for RL policies yet. The authors study 4 continuous control tasks. The main empirical observation is that the complexity of the policies comes from fine-tuning their contribution, not in a significant growth in their number along trajectories of the final policy. The number of regions seems mostly related to the total number of neurons, a finding also observed in supervised learning. 
strengths_and_weaknesses: Strong:
* I think conceptual insight studies are always relevant. As the authors mention as well, in RL we usually only look at the learning curves, which is a pity and probably misses much relevant insight. 
* The overall paper is clearly written and easy to follow. 
* I like Figure 3 (although you could maybe separately draw the Toy environment), it provides good insight. 

Weak:
* See my comment in the Questions section. 
* The authors only study four low-dimensional continuous control tasks, which arguably does not give a lot of variety. I therefore think the conclusions are a bit strong. I would have much preferred if there were 1 or 2 tasks from the Mujoco set (with indeed many cycles), but also some other types of tasks to validate and broader the scope of the conclusions. I think the current conclusions need to be downplayed a bit. 
* The results confirm what we already knew from similar analysis in supervised learning. Of course, one still has to try in the RL setting to confirm this, but I think the paper needs to discuss why we would expect things to be different in the RL case? (I would not a clear reason: RL is mostly a data generation and target computation problem, after that the NN training process is simply a supervising learning problem). 

Smaller: 
* Figure 2 needs more explanation ( a better caption) to understand what is going on. 
* I printed your version and all references were invisible (due to the very light green color, I would pick something darker). 

questions: I do not fully understand what I need to learn from the “region densities”, or what you actually compute here. As I understand, you take the total visitation counts per visited region, divide these over (N * L(tau)), and then you average this number? Doesn’t this really just estatimate the number of unique regions you visited? 
* For example, imagine I visited 4 regions, with counts [100,200,300,400], and assume N=10 and L(tau)=10. This would give densities [1,2,3,4], and therefore average density 2.5. 
* Now assume I instead observed counts [200,200,300,300]. This would give densities [2,2,3,3], and therefore still average density 2.5. Whatever way I distribute the counts over the regions, if the total number of regions stays the same, the average density will as well. 
* I think what I would be really interested in is the total number of uniquely visited regions, if possible in a smaller problem with a density-coloured map (instead of averaged over all regions, which throws away a lot of information). 



limitations: Conclusion: This is a decent paper. I like that the authors focus on a conceptual/insight topic in RL, which is often missing. On the other hand, I do think these results are close to what was already known from supervised learning, and I slightly miss the motivation of the authors why things could be different in RL (nevertheless, always worth investigating). I am also a bit in doubt about the way the results are presented, especially regarding the density estimation: this could maybe be clearer.
review_title: null

Meta-review: metareview: This paper provides an empirical analysis of RL methods via the lens of linear regions induced by NN-based agents. All the reviewers agreed the paper was well written and it was wonderful to see an understanding based paper. The paper is clear about its limitations, very detail oriented & scientific, and borrows & checks the validity of an analysis technique from supervised learning. The authors nicely added an additional environment and agent to the paper at the reviewers request.

This paper is very close. Several reviewers asked in one way or another "where is the evidence that we should care about linear regions" and "how will this inform future research?". The majority of the reviewers found the response and updated paper did not clearly address these questions. Indeed, even the most positive reviewer pointed out that the 2 research questions posed in the introduction were not well motivated. This is concern #1.

Concern #2 regards the generality of the results: what trends we see across agents and environments. With the added experiments the trends were not clear. This can be ok especially when testing an idea shown to be important in SL. It can be useful to report such a result. However, in this case we might need a good variety of agents and environments (not necessarily large scale environments, just different ones).

In the end for such a paper to have impact we need either: (a) interesting trends to come out of the results and a clear articulated connection to future algorithmic development or (b) a clear body empirical results that say "yep we tried this and its not clear it helps". The current paper does a bit of both.
confidence: Less certain
Decision: Accept