# Fast Minimum Error Entropy for Linear Regression
## 3 Methodology  
This section derives the relation between the differential entropy and the Gram-Charlier expansion of the PDF of the error for linear regression. Moreover, the assumption that the PDF of the regression error is near the Gaussian distribution \(\varphi(\xi)\) that has the same mean and variance as the error is implemented. The differential entropy [42] of a random variable \(\mathbf{X}\) is shown in Equation (10):  
\[H(x)=-\int p(x)\log p(x)\mathrm{d}x. \tag{10}\]  
Substitute Equation (9) into Equation (10); the result can be derived in Equation (11):  
\[ \begin{split} H(x)\approx&-\int\hat{p}_{x}(\xi)\log \hat{p}_{x}(\xi)\mathrm{d}\xi\\ =&-\int\frac{1}{\sqrt{2\pi\sigma}}\exp\biggl{[}- \frac{(\xi-\mu)^{2}}{2\sigma^{2}}\biggr{]}*\\ &\biggl{[}1+\frac{\kappa_{3}(x)}{3!\sigma^{3}}H_{3}\biggl{(} \frac{\xi-\mu}{\sigma}\biggr{)}+\frac{\kappa_{4}(x)}{4!\sigma^{4}}H_{4}\biggl{(} \frac{\xi-\mu}{\sigma}\biggr{)}\biggr{]}*\\ &\log\biggl{[}\frac{1}{\sqrt{2\pi\sigma}}\exp\big{[}-\frac{(\xi- \mu)^{2}}{2\sigma^{2}}\bigr{]}*\\ &\bigl{[}1+\frac{\kappa_{3}(x)}{3!\sigma^{3}}H_{3}\biggl{(} \frac{\xi-\mu}{\sigma}\biggr{)}+\frac{\kappa_{4}(x)}{4!\sigma^{4}}H_{4}\biggl{(} \frac{\xi-\mu}{\sigma}\biggr{)}\bigr{]}\biggr{]}\mathrm{d}\xi.\end{split}  \tag{11}\]  
This integral is rather difficult to compute. However, notice that if the PDF of \(x\) is near the normal density as assumed, it can be inferred that \(\kappa_{3}(x)\) and \(\kappa_{4}(x)\) are very small. Then, the approximation in Equation (12) can be used:  
\[\log(1+\epsilon)\approx\epsilon-\frac{\epsilon^{2}}{2}, \tag{12}\]  
where \(\epsilon\) denotes the infinitesimal. Then, Equation (11) can be transformed into Equation (13):\[H(x)= -\int\varphi(\xi)* \tag{13}\] \[\big{[}1+\frac{\kappa_{3}(x)}{3!\sigma^{3}}H_{3}\bigg{(}\frac{\xi- \mu}{\sigma}\bigg{)}+\frac{\kappa_{4}(x)}{4!\sigma^{4}}H_{4}\bigg{(}\frac{\xi- \mu}{\sigma}\bigg{)}\big{]}*\] \[\bigg{[}\log\varphi(\xi)+\frac{\kappa_{3}(x)}{3!\sigma^{3}}H_{3} \bigg{(}\frac{\xi-\mu}{\sigma}\bigg{)}+\] \[\frac{\kappa_{4}(x)}{4!\sigma^{4}}H_{4}\bigg{(}\frac{\xi-\mu}{ \sigma}\bigg{)}-\frac{1}{2}\bigg{(}\frac{\kappa_{3}(x)}{3!\sigma^{3}}H_{3} \bigg{(}\frac{\xi-\mu}{\sigma}\bigg{)}+\] \[\frac{\kappa_{4}(x)}{4!\sigma^{4}}H_{4}\bigg{(}\frac{\xi-\mu}{ \sigma}\bigg{)}\bigg{)}^{2}\bigg{]}\mathrm{d}\xi.\]  
As \(H_{i}\) forms an orthogonal system, Equation (13) can be simplified as Equation (14),  
\[H(x)=\frac{1}{2}\log 2\pi\mathrm{e}\sigma^{2}-\frac{\kappa_{3}^{2}(x)}{2*3! \sigma^{6}}-\frac{\kappa_{4}^{2}(x)}{2*4!\sigma^{8}}. \tag{14}\]  
The details of the simplification process of Equation (14) are illustrated as follows. This part is to derive Equation (14). First, it is needed to show that  
\[\int\varphi(\xi)(\frac{\xi-\mu}{\sigma})^{2}H_{3}(\frac{\xi-\mu}{\sigma}) \mathrm{d}\xi=0 \tag{15}\]  
and  
\[\int\varphi(\xi)(\frac{\xi-\mu}{\sigma})^{2}H_{4}(\frac{\xi-\mu}{\sigma}) \mathrm{d}\xi=0 \tag{16}\]  
For Equation (15), suppose that \(t=(\xi-\mu)/\sigma\),  
\[ \begin{split}&\int\varphi(\xi)(\frac{\xi-\mu}{\sigma})^{2}H_{3}( \frac{\xi-\mu}{\sigma})\mathrm{d}\xi\\ &=\int\frac{1}{\sqrt{2\pi}\sigma}\exp\bigg{[}-\frac{(\xi-\mu)^{ 2}}{2\sigma^{2}}\bigg{]}\bigg{[}\frac{(\xi-\mu}{\sigma})^{3}-3(\frac{\xi-\mu}{ \sigma})\bigg{]}\mathrm{d}\xi\\ &=\int\frac{1}{\sqrt{2\pi}}t^{2}\exp(-\frac{t^{2}}{2})(t^{3}-3t) \mathrm{d}t\end{split}  \tag{17}\]  
As \(\exp(-\frac{t^{2}}{2})\) is an even function and \(t^{5}-3t^{3}\) is an odd function, then  
\[\int\frac{1}{\sqrt{2\pi}}t^{2}\exp(-\frac{t^{2}}{2})(t^{3}-3t)\mathrm{d}t=0 \tag{18}\]  
and Equation (15) holds. For Equation (16), suppose that \(t=(\xi-\mu)/\sigma\),  
\[