# Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
## Natural Gradient Learning  
### Natural Gradient for Online Learning Mode  
The natural gradient, which was originally proposed by [PERSON] [14], is based on the information geometry considering theoretical properties of the space of stochastic neural networks. Based on the investigation that the space of an arbitrary stochastic model is not Euclidean but Riemannian,Amari [14] called the space as neuromanifold, and proposed an appropriate Riemannian metric using Fisher information matrix \(G(\mathbf{\theta})\) of probability density \(p(\mathbf{x},\mathbf{y},\ \mathbf{\theta})\), which can be written as  
\[G(\mathbf{\theta})=\mathbf{E}_{\mathbf{x}}\Bigg{[}\mathbf{E}_{\mathbf{y}|\mathbf{x}}\Bigg{[} \frac{\partial p(\mathbf{y}|\mathbf{x},\mathbf{\theta})}{\partial\mathbf{\theta}}\Bigg{(} \frac{\partial p(\mathbf{y}|\mathbf{x},\mathbf{\theta})}{\partial\mathbf{\theta}}\Bigg{)}^{T} \Bigg{]}\Bigg{]}. \tag{11}\]  
The steepest descent direction on the neuromanifold can then be derived from the new metric, and the new type of gradient descent learning method is obtained in the form,  
\[\mathbf{\theta}_{t+1}=\mathbf{\theta}_{t}-\eta G^{-1}(\mathbf{\theta}_{t})\frac{\partial l (\mathbf{x},\mathbf{y},\mathbf{\theta}_{t})}{\partial\mathbf{\theta}}. \tag{12}\]  
This is called natural gradient descent learning method. [PERSON] [14] introduced the basic form of online natural gradient learning, and proved its efficiency in asymptotic convergence properties. There have also been several works showing that the natural gradient can solve plateau problem revealing ideal learning dynamics [16; 17; 18; 19; 25; 26]. However, the pure natural gradient cannot be applied to the learning of neural networks for real-world problems. As shown in the Equation (11), the explicit form of \(G(\mathbf{\theta})\) is obtained through two expectations over the conditional probability density \(p(\mathbf{y}|\mathbf{x},\mathbf{\theta})\) and the true input distribution \(q(\mathbf{x})\). Since the expectation over \(p(\mathbf{y}|\mathbf{x},\mathbf{\theta})\) depends on the specific model of stochastic neural networks, it can be written explicitly with some assumptions on the model. Typically, under the assumption that \(p(\mathbf{y}|\mathbf{x},\mathbf{\theta})\) is subject to the standard Gaussian distribution with zero mean and identity covariance matrix, the Fisher information matrix can be written as  
\[G(\mathbf{\theta})=\mathbf{E}_{\mathbf{x}}\Bigg{[}\sum_{i=1}^{L}\frac{\partial f_{i}( \mathbf{x},\mathbf{\theta})}{\partial\mathbf{\theta}}\Bigg{(}\frac{\partial f_{i}(\mathbf{x},\mathbf{\theta}))}{\partial\mathbf{\theta}}\Bigg{)}^{T}\Bigg{]}. \tag{13}\]  
Under the assumption that \(p(\mathbf{y}|\mathbf{x},\mathbf{\theta})\) is subject to the logistic regression model with softmax activation function, the Fisher information matrix can then be written as  
\[G(\mathbf{\theta})=\mathbf{E}_{\mathbf{x}}\Bigg{[}\sum_{i=1}^{L}\frac{1}{f_{i}(\mathbf{x},\mathbf{\theta})}\frac{\partial f_{i}(\mathbf{x},\mathbf{\theta})}{\partial\mathbf{\theta}} \Bigg{(}\frac{\partial f_{i}(\mathbf{x},\mathbf{\theta}))}{\partial\mathbf{\theta}} \Bigg{)}^{T}\Bigg{]}. \tag{14}\]  
One can refer to [20; 21] for detail derivations. Likewise, the expectation over \(p(\mathbf{y}|\mathbf{x},\mathbf{\theta})\) can be obtained under the specific selection of the stochastic model. However, we still have another expectation over true input distribution \(q(\mathbf{x})\), which is not practically given. In addition, there is another practical issue: The necessity of inverse operation of \(G(\mathbf{\theta})\) to obtain the natural gradient as shown in (12). To deal with the two practical issues, [PERSON] et al. [20] proposed an adaptive method for obtaining an approximation of the natural gradient in online learning mode, and [PERSON] et al. [21] extended it to various stochastic models. In this paper, we revisit the previous works and its limitations, and propose a modified version of adaptive natural gradient learning in mini-batch learning mode.