# Policy Compression for Intelligent Continuous Control on Low-Power Edge Devices
## 4 Methodology
### Distilling the Action Distribution
Instead, we derive the KL-divergence between two absolutely continuous univariate normal distributions, starting with the general definition of the KL-divergence for distributions P and Q of continuous random variables:  
\[\text{KL}(P,Q)=\int p(x)\log\Big{(}\frac{p(x)}{q(x)}\Big{)}dx \tag{5}\]  
In our setting, P and Q are normal distributions defined by \(\mu\) and \(\sigma\), for which the probability density function is defined as follows:  
\[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]  
To substitute this in Equation (5), we first focus on the log division:\[\log\big{(}\frac{p(x)}{q(x)}\big{)} =\log p(x)-\log q(x)\] \[=\Big{[}\log\big{(}\frac{1}{\sigma_{p}\sqrt{2\pi}}e^{-\frac{1}{2} \big{(}\frac{x-\mu_{p}}{\sigma_{p}}\big{)}^{2}}\big{)}-\log\big{(}\frac{1}{ \sigma_{q}\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu_{q}}{\sigma_{q}}\big{)} ^{2}}\big{)}\Big{]}\] \[=\Big{[}-\frac{1}{2}\log\big{(}2\pi\big{)}-\log\big{(}\sigma_{p} \big{)}-\frac{1}{2}\big{(}\frac{x-\mu_{p}}{\sigma_{p}}\big{)}^{2}+\frac{1}{2} \log\big{(}2\pi\big{)}+\log\big{(}\sigma_{q}\big{)}+\frac{1}{2}\Big{(}\frac{x- \mu_{q}}{\sigma_{q}}\big{)}^{2}\Big{]}\] \[=\Big{[}\log\big{(}\frac{\sigma_{q}}{\sigma_{p}}\big{)}+\frac{1}{ 2}\big{[}\big{(}\frac{x-\mu_{q}}{\sigma_{q}}\big{)}^{2}-\Big{(}\frac{x-\mu_{p }}{\sigma_{p}}\big{)}^{2}\big{]}\Big{]}\]  
The full equation then becomes:  
\[\text{KL}(P,Q)=\int\frac{1}{\sigma_{p}\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{ x-\mu_{p}}{\sigma_{p}}\big{)}^{2}}\Big{[}\log\Big{(}\frac{\sigma_{q}}{\sigma_{p}} \Big{)}+\frac{1}{2}\big{[}\big{(}\frac{x-\mu_{q}}{\sigma_{q}}\big{)}^{2}- \Big{(}\frac{x-\mu_{p}}{\sigma_{p}}\big{)}^{2}\big{]}\Big{]}dx \tag{7}\]  
We then rewrite this using the expectation with respect to distribution P:  
\[\text{KL}(P,Q) =E_{p}\Big{[}\log\Big{(}\frac{\sigma_{q}}{\sigma_{p}}\Big{)}+ \frac{1}{2}\big{[}\big{(}\frac{x-\mu_{q}}{\sigma_{q}}\big{)}^{2}-\Big{(}\frac {x-\mu_{p}}{\sigma_{p}}\Big{)}^{2}\big{]}\Big{]}\] \[=\log\Big{(}\frac{\sigma_{q}}{\sigma_{p}}\Big{)}+\frac{1}{2\sigma _{q}^{2}}E_{p}\big{[}(X-\mu_{q})^{2}\big{]}-\frac{1}{2\sigma_{p}^{2}}E_{p} \big{[}(X-\mu_{p})^{2}\big{]}\] \[=\log\Big{(}\frac{\sigma_{q}}{\sigma_{p}}\Big{)}+\frac{1}{2\sigma _{q}^{2}}E_{p}\big{[}(X-\mu_{q})^{2}\big{]}-\frac{1}{2}\]  
Note that we could rewrite \((X-\mu_{q})^{2}\) to:  
\[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]0 \[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]1 \[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]2  
Substituting this results in:  
\[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]3 \[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]4 \[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]5  
Finally, we apply this for policy distillation of continuous action spaces:  
\[f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{x-\mu}{\sigma} \big{)}^{2}} \tag{6}\]6  
This should allow for a smoother optimization objective than learning both values using the Huber loss, similar to the distillation of discrete actions.