# Policy Compression for Intelligent Continuous Control on Low-Power Edge Devices
## 7 Conclusions  
Deploying intelligent agents for continuous control tasks, such as drones, AMRs, or IoT devices, directly on low-power edge devices is a difficult challenge, as their computational resources are limited, and the available battery power is scarce. This paper addressed this challenge by proposing a novel approach for compressing such DRL agents by extending policy distillation to support the distillation of stochastic teachers that operate on continuous action spaces, whereas existing work was limited to deterministic policies or discrete actions. Not only does this compression increase their applicability while reducing associated deployment costs, but processing the data locally eliminates the latency, reliability, and privacy issues that come with wireless communication to cloud-based solutions. To this end, we proposed three new loss functions that define a distance between the distributions from which actions are sampled in teacher and student networks. In particular, we focused on maintaining the stochasticity of the teacher policy by transferring both the predicted mean action and state-dependent standard deviation. This was compared to a baseline method where we only distill the mean action, resulting in a completely deterministic policy. We also investigated how this affects the collection of transitions on which our student is trained by evaluating our methods using both a student-driven and teacher-driven control policy. Finally, the compression potential of each method was evaluated by comparing the average return obtained by students of ten different sizes, ranging from 0.6% to 189% of their teacher's size. We then showed how each of these compression levels translates into improvements in real-world run-time performance. Our results demonstrate that especially our loss based on the KL-divergence between the univariate normal distributions defined by \(\mu\) and \(\sigma\) is highly effective at transferring the action distribution from the teacher to the student. When distilling an SAC teacher, it outperformed our baseline where only the mean action is distilled on average by 8% on the HalfCheetah-v3 environment and 34% on Ant-v3. This effect is especially noticeable in the student-driven setting, but we were also able to observe a significant increase in sample efficiency in the teacher-driven setup. When a less stochastic PPO teacher was used, all our proposed methods performed equally well, managing to maintain or even outperform their teacher while being significantly smaller. This also confirms that the regularization effect of policy distillation that was observed in the setting for discrete action spaces still holds for the continuous case. In general, we recommend a student-driven distillation approach with our loss based on the KL-divergence between continuous actions as the most effective and stable compression method for future applied work. Through this method, DRL agents designed to solve continuous control tasks were able to be heavily compressed by up to 750% without a significant penalty to their effectiveness. Conceptualization, T.A. and K.M. ; methodology, T.A. and K.M. ; software, T.A. ; validation, T.A. ; formal analysis, T.A. ; investigation, T.A. ; resources, T.A. ; data curation, T.A. ; writing--original draft preparation, T.A. ; writing--review and editing, K.M. ; visualization, T.A. ; supervision, K.M. and T.D.S. ; project administration, T.A. ; funding acquisition, T.A. and K.M. All authors have read and agreed to the published version of the manuscript. The work by [PERSON] on this paper was funded by Research Foundation Flanders [No. 1 SD9523N]. It was also supported by the euROBIN (European ROBotics and AI Network) [No. 101070596] and OpenSwarm [No. 101093046] projects under the Horizon Europe Framework Programme. Data Availability Statement: The environments used in this study are part of the Gymnasium project [29] (version 2.1.0), and the pre-trained teacher models are part of the Stable Baselines3 project [31] (version 0.29.1).