# Improved Neural Networks Based on Mutual Information via Information Geometry
## 2 Materials and Methods
Considering that the activations of the hidden neurons are independent, the probability of \(z\) is calculated as  
\[p(z|x_{i},w)=p(z|\theta)=\prod\limits_{j=1}^{l}p(z|\theta^{j})=\prod\limits_{ j=1}^{l}\exp(z^{j}\cdot\theta^{j}-\varphi^{j}(\theta^{j}))=\exp(\sum \limits_{j=1}^{l}z^{j}\cdot\theta^{j}-\sum\limits_{j=1}^{l}\varphi^{j}(\theta ^{j}))=\exp(z\cdot\theta-\varphi(\theta)) \tag{9}\]  
Thus, \(p(z|\theta)\) is from the exponential family, while \(\theta\) is a function of the linear input \(x\cdot w\) of \(z\). In addition, \(\varphi\) is a function of \(\theta\). A classic parametric space for this family of probability density functions (PDFs) is  
\[H=\left\{\eta=\frac{\partial\varphi(\theta)}{\partial\theta}\right\} \tag{10}\]  
For the distributions in the exponential family, there are two dual coordinate systems \(\theta\) and \(\eta\) on the manifold (which is defined as a topological space that locally resembles Euclidean space near each point) of the parameters [28]. Here, \(\eta\) is given by  
\[\eta=\frac{\partial\varphi(\theta)}{\partial\theta}=E(z) \tag{11}\]  
With (9) and (11), we calculate the \(\eta_{i}\) for each \(z_{i}\). From (6), it is evident that the \(p(z)\) is the mean of the conditional probability over each input \(x_{i}\). Thus, with (11), the parameters in the density in (6) are approximated by  
\[\overline{\eta}=\frac{1}{N}\sum\limits_{i=1}^{N}\eta_{i} \tag{12}\]  
For the distribution in the exponential family, we have  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]0  
For distribution with \(n\) parameters, the Fisher distance between two points \(\theta^{\prime}\) and \(\theta\star\) in the half-plane \(H\) reflects the dissimilarity between the associated PDF's. It is defined as the minimal integral [29]  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]1where \(\theta(t)\) represents a curve that is parameterized by \(t\), while \(G(\theta)\) is the Fisher information matrix which is defined as  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]2  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]3  
where \(a\in\{1,2,\ldots,n\}\) and \(b\in\{1,2,\ldots,n\}\) are the indexes of the elements in \(G\). The curve that satisfies the minimization condition in (14) is called a geodesic line. **Theorem 1**[30]. : _For the distributions in the exponential family, the Fisher distance is the Kullback-Leibler divergence given by_  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]4  
With (11) to (13), we obtain a dual description for \(p(z)\), as follows:  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]5  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]6  
Let \(\theta^{\prime}=\theta\) and \(\eta\star=\overline{\eta}\) in (14). With (5), we have  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]7  
Then, we substitute (20) into (3) to obtain the second item in (2). By the definition of the Fisher distance (14) and Theorem 1, we know that minimizing the second item in (2) is equivalent to minimizing the Fisher distance between the estimated distribution \(p(r|x_{i})\) and the original distribution \(p(r)\) along the shortest path (geodesic line). To describe the geodesic line using local coordinates, we must solve the geodesic equations given by the Euler-Lagrange equations, as follows:  
\[\frac{\partial^{2}\theta_{b}}{d\overline{t}^{2}}+\sum\limits_{a=1}^{n}\sum \limits_{b=1}^{n}\left(\frac{1}{2}\sum\limits_{a=1}^{n}g^{kl}(\frac{\partial g _{kl}}{\partial\theta_{b}}+\frac{\partial g_{kl}}{\partial\theta_{a}}-\frac{ \partial g_{kl}}{\partial\theta_{b}})\right)\frac{d\theta_{a}}{d\overline{t} }\frac{d\theta_{b}}{d\overline{t}}=0,\forall a,bIn these samples, the percent of labeled training samples \(\alpha\) samples ranges from 10% to 50%. In the remainder of the dataset, 10% of the samples are randomly selected as test data. The constant \(\lambda\) is set to \(-0.1\).