# Improved Neural Networks Based on Mutual Information via Information Geometry
## 2 Materials and Methods  
Let us consider a neural network with one input layer, one output layer, and one hidden layer. The weights for the input are denoted by \(w\). The weights for the hidden neurons are \(v\). Here, \(x\) is the input, while \(\overline{y}\) is the actual output, and \(z\) is the activation of the hidden neurons. The concatenation of the output and hidden neurons is represented as a vector \(r=[\overline{y},z]\). Figure 1 illustrates the architecture of the neural network. Figure 1: The architecture of the neural network. The neural network is a static and usually nonlinear model  
\[p(r|x)=p(\overline{y},z|x,u)=p(\overline{y}|z,v)p(z|x,w) \tag{1}\]  
where the parameters of the neural network are denoted by \(u=[w,v]\). The mutual information (MI) [23] of two random variables quantifies the \"amount of information\" obtained about one random variable, through the other random variable. In the proposed algorithm, we treat the input \(x\) and the combination of the output and hidden neurons \(r\) as two random variables. Correspondingly, the MI quantifies the \"amount of information\" obtained by the output and hidden variable from the input. The learning of the neural network attempts to keep as much information as possible while maximizing the likelihood. We define a novel objective function as  
\[J=E_{x}[(\overline{y}-y)^{2}]+\lambda I(x;r) \tag{2}\]  
where \(y\) is the desired output (i.e., the true value of the label). The first item is the objective function of traditional backpropagation neural networks, while the second item is the MI of \(x\) and \(r\). With the first item, we maximize the likelihood similar to what traditional neural networks do. With the second item, the proposed method helps the neural networks to keep information from the input. Here, \(\lambda\) is a constant determined by experience. Many methods exist for minimizing the first item, such as conjugate gradient [24]. Therefore, we just discuss how to work with the second item in this paper. By the definition of MI and Kullback-Leibler divergence (KLD) [25], the mutual information of \(x\) and \(r\) is  
\[\begin{array}{ll}I(x;r)&=\int\limits_{x\in X}\int\limits_{r\in R}p(x,r)\log \frac{p(x,r)}{p(x)p(r)}drdx\\ &=\int\limits_{x\in X}p(x)\int\limits_{r\in R}p(r|x)\log\frac{p  
where \(z_{i}\) is the value of the vector of hidden neurons when the input is \(x_{i}\), while \(Z\) is the value domain of \(z\), and \(p(\overline{y}|z_{i},v)=p(\overline{y}|z_{i})\) so that \(KLD(p(\overline{y}|z_{i},v)||p(\overline{y}|z_{i}))=0\).  
The original distribution \(p(z)\) can be derived from a finite-sample approximation, which gives us the following:  
\[p(z)=\frac{1}{N}[p(z|x_{1},w)+p(z|x_{2},w)+\ldots+p(z|x_{N},w)]=\frac{1}{N} \sum\limits_{i=1}^{N}p(z|x_{i},w) \tag{6}\]  
Next, we will show an effective way to calculate the \(KLD\) in (5) with the theory of information geometry. Any function that can be written in the following form is from the exponential family [26]:  
\[p(x|\theta)=\exp(x\cdot\theta-\varphi(\theta)) \tag{7}\]  
where \(\varphi\) is a function of \(\theta\). Without loss of generality, the probability of the activation of each hidden unit can be approximated with a function from the exponential family [27]. Here, \(z^{j}\) represents the \(j\)th hidden unit; then,  
\[p(z^{j}|x_{i},w)=p(z^{j}|\theta^{j})=\exp(z^{j}\cdot\theta^{j}-\varphi^{j}( \theta^{j})) \tag{8}\]  
where \(\theta^{j}\) is a function of \(x\cdot w^{j}\) (i.e., the linear weighted input of \(z^{j}\)), while \(w_{j}\) is the \(j\)th column of \(w\), for \(j\in\{1,2,\ldots,l\}\), and \(l\) is the size of the hidden layer. Additionally, \(\varphi^{j}\) is a function of \(\theta^{j}\).