The Case for Gradual Structured Pruning in Image-based Deep Reinforcement Learning

Authors: Raphael Trumpp, Mirco Theile, Marco Caccamo
Primary area: reinforcement learning
Keywords: gradual structured pruning, deep reinforcement learning, Procgen benchmark

Abstract:
Scaling neural networks in image-based deep reinforcement learning often fails to improve performance. While it was shown that unstructured pruning of scaled networks can unlock performance gains, we find that refining the architecture of the scaled network yields even greater improvements. However, scaled networks in deep reinforcement learning present a practical challenge: the increased computational demands can hinder deployment on embedded devices, as commonly encountered in robotics applications. To address this, we propose a novel gradual group-structured pruning framework that allows performance gains through scaling while maintaining computational efficiency. Our method preserves the network's functional integrity of inter-layer dependencies in groups, such as residual connections, while seamlessly integrating with standard deep reinforcement learning algorithms. Experiments with PPO and DQN show that our approach sustains performance while significantly reducing inference time, making it the preferred approach for resource-limited deployment.

Decision: reject
Reviewer scores: 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730072252420

thread_type:
official_review

<thread_object>
creator: reviewer_dak6

<content>
summary:
Scaling neural networks has generally been ineffective for deep reinforcement learning, as larger networks do not necessarily improve performance. Recent work by Obando-Ceron et al. (2024) suggests that pruning a large network could overcome this limitation. By initially increasing the network size and applying pruning during training, they achieved performance gains beyond those possible with a standard dense network. However, their unstructured sparsity approach often fails to generalize well outside the original setting. This paper introduces an improved CNN architecture, called Impoola, along with a structured pruning algorithm that outperforms unstructured pruning in both generalization and computational efficiency.

strengths:
- Exploring scaling laws for deep RL is an important challenge.
- New CNN architecture and structured pruning algorithm are proposed.
- Outperform both unstructured and naive structured pruning baselines.

weaknesses:
**Clarification of objectives and contribution**

The primary goal of this paper—whether to build upon Obando-Ceron et al.'s (2024) approach for finding scalable architectures or to develop a structured pruning method specifically for deep RL—is somewhat unclear. If the objective is scaling, adding experiments that compare the proposed method with the original dense network and unstructured pruning across different scenarios would strengthen its claims. On the other hand, if structured pruning is the focus, revisiting the introduction and motivation to reflect this intent could help clarify the paper’s contributions in this area.

---
**Need for broader benchmark comparisons**

If this paper aims to address scaling issues in deep RL, it would benefit from broader comparisons with relevant methods, such as those presented in [1] and [2]. Including at least one of these as a baseline would provide a more comprehensive context for the paper’s approach. If these methods are excluded, providing a brief rationale would clarify the baseline selection choices for readers.

---
**Potential for alternative focus on structured pruning**

An alternative approach for the paper could be to frame it as a structured pruning study specifically for deep RL, rather than as an extension of scaling work. The proposed Impoola architecture appears specialized, so comparing it with existing structured pruning methods for deep RL, such as [3], would reinforce its relevance in this area. Additionally, if the technique is intended to be broadly applicable to CNNs, testing it on traditional CNN tasks (e.g., video classification) would better demonstrate its versatility. Including comparisons with CNN-focused pruning techniques, such as [4], would further highlight the method’s adaptability across different applications.



[1] Obando-Ceron et al. Mixtures of Experts Unlock Parameter Scaling for Deep RL. ICML 2024.\
[2] Sokar et al. Don't Flatten, Tokenize! Unlocking the Key to SoftMoE's Efficacy in Deep RL. arXiv 2024.\
[3] Su et al. Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving. arXiv 2024.\
[4] He & Xiao. Structured Pruning for Deep Convolutional Neural Networks: A Survey. TPAMI 2023.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730433013447

thread_type:
official_review

<thread_object>
creator: reviewer_a9a9

<content>
summary:
This paper proposes a gradual structured pruning method for deep reinforcement learning. The authors firstly analyzes the drawbacks of the unstructured pruning, then introduces a improved impoola-CNN model to replace the previous impala-CNN model, and a gradual structured pruning for image-based deep reinforcement learning tasks. The experimental results show that the proposed method has compariable perfermances comparing with unstrutured pruning mehtod, and it has good robustness when intruducing nosies. Moreover, the proposed pruning method results in less latency times comparing with original dense model and unstructured pruned model.

strengths:
1. The gradual structured pruning method is well-defined, and the experimental results show that is has good computation efficiency

weaknesses:
1. The experimental results did not include the comparsion of impala-CNN and impoola-CNN, thus the advantages of the new network cannot be proved. The authors may consider to provide a direct comparison of performance between Impala-CNN and Impoola-CNN under the same pruning conditions across multiple environments, as well as the un-pruned editions. This would give clearer evidence for the claimed advantages of the new architecture.

2. The advantages of the scoring function are not proved. The reason that use ||w||_1 should be carefully analyzed. For instance, the authors may consider to compare L1-norm, L2-norm or other regularization methods to show the advantages of the proposed scoring function.

questions:
1. Is their some comparsion between impala-CNN and impoola-CNN when both of them are processed by using gradual structured pruning? The authors may use the proposed pruning mehtod to both impala-CNN and impoola-CNN under different configurations to show the improvements introduced by impoola-CNN. 

2. Is their any experimental results to compare the performance of using different scoring functions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730460144954

thread_type:
official_review

<thread_object>
creator: reviewer_xmu9

<content>
summary:
The paper introduces a group-structured pruning framework for image-based deep reinforcement learning (DRL), specifically designed to maintain performance while reducing computational costs. The authors first demonstrate that the benefits of unstructured pruning diminish when making architectural improvements to neural networks. They then propose a structured pruning approach that removes entire network structures (channels/neurons) while accounting for inter-layer dependencies. The method is extensively evaluated on the Procgen Benchmark using PPO and DQN agents, showing comparable performance to unstructured pruning while achieving significant reductions in inference time.

strengths:
1. The overall writing is clear.
2. Efficiency in the domain of RL is interesting.

weaknesses:
1. The motivation is not clear. Although efficiency in the domain of RL is interesting, the author should include the difference between "pruning in image classification" and "pruning in reinforcement learning." This motivation should serve the algorithm.

2. There are some other references [a] to do gradual pruning. What is the difference? Why not cite this?

3. Some arguments are wrong. For example, in line 15, the authors claim, "unstructured pruning merely zeroes out individual weights, the resulting networks usually retain high computational demands despite sparsity". This is not correct. For example, Thinet [b] can achieve realistic acceleration. There are so many other filter pruning methods to achieve practical acceleration.

4. The performance is not good. In table 1, if the up arrow means "higher is better", the proposed method is not as good as others.

[a] H. Wang, C. Qin, Y. Zhang, and Y. Fu, “Neural pruning via growing regularization,” in Proc. Int. Conf. Learn. Represent., 2022
[b] https://github.com/Roll920/ThiNet

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732795155145

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
List of Changes

comment:
Dear Reviewers,

Thank you for your constructive feedback. We uploaded the revision of the paper and hope the reviewers have the time to check the updated manuscript to see whether their concerns were sufficiently addressed. 

**List of overall major changes (chronologically)**:
- Reformulation of the motivation. The preliminary experiment and Figure 1 were updated to emphasize that using Impoola unlocks performance gains through network scaling, but without the need for **un**structured pruning. This serves as motivation for our algorithm, which unlocks performance gains from scaling in combination with lowering compute demand substantially for such scaled networks.
- Extending Section 2. Related Work with a paragraph on Scaling in DRL.
- Table 1 is extended by results for Impala (Dense w/o Impoola) and we added another baseline method (Distillation BC) for better comparison. We also included the training time of each method. This should highlight the performance of group-structured pruning better, as the aim is to develop a method that combines performance improvement (Impoola) and lowering the compute demand of such scaled networks (group-structured pruning).
- Section 5.4 Ablations included several new experiments. There is a new ablation on the importance score function (L1, L2, Taylor, random). We provide further results for a comparison of pruned Impoola and Impala networks.
- We conducted additional experiments of our method for Atari games, confirming the results for Procgen. This demonstrates the broad applicability of our approach.
- **Important note**: We updated the paper's title to reflect our focus on efficient scaling: Gradual Structured Pruning for Efficient Network Scaling in Image-Based Deep Reinforcement Learning.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission proposes a gradual structured pruning method for deep reinforcement learning. After analyzing the drawbacks of prior work and unstructured pruning, they propose an improved Impoola-CNN model and demonstrate comparable performance to unstructured pruning.
The submission received ratings of 3, 5, 6.
Multiple reviewers highlighted issues with the organization of the paper, making its key goals and objectives unclear.
Even after the rebuttal, the reviewers highlighted the issues during the AC discussion. After reading through the submission, the ACs concur with the reviewers regarding it needing significant changes to reach the bar for acceptance.
The ACs did not find enough reason to overturn the overall negative consensus and recommend rejection.