# Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
## 4 Adaptive Natural Gradient Learning in Mini-Batch Mode  
### Adaptive Estimation of Fisher Information Matrix in Mini-Batch Mode  
In order to estimate the natural gradient in mini-batch learning mode, we first consider approximation of Fisher information matrix \(\mathbf{G}(\mathbf{\theta})\). The true Fisher information matrix \(\mathbf{G}(\mathbf{\theta})\) can be calculated when we know the explicit form of true input pdf \(\mathbf{q}(\mathbf{x})\), but it is rarely given in practical situations. As a resolution of this problem, its empirical expectation using given observations is commonly used and then the empirical approximation of the Fisher information matrix can be defined by  
\[\overline{\mathbf{G}}(\mathbf{D},\mathbf{\theta})=\frac{1}{|\mathbf{D}|}\sum_{x\in\mathbf{D}|}^{N }\left[\sum_{i=1}^{L}\left\{\frac{\partial f_{i}(\mathbf{x},\mathbf{\theta})}{\partial \mathbf{\theta}}\left(\frac{\partial f_{i}(\mathbf{x},\mathbf{\theta}))}{\partial\mathbf{ \theta}}\right)^{T}\right\}\right]. \tag{17}\]  
Note that, for simplicity, we assume the additive Gaussian noise model in (17). Since the Fisher information matrix is dependent on parameter vector \(\mathbf{\theta}\), the matrix \(\overline{\mathbf{G}}(\mathbf{D},\mathbf{\theta})\) needs to be calculated at each update of parameters. Also, from the Equation (17), one can easily see that its computational complexity increases as the size of \(\mathbf{D}\) increases. If we take mini-batch mode, the approximation is done by using a smaller subset \(\mathbf{D}_{k}\), which can be written as  
\[\overline{\mathbf{G}}(\mathbf{D}_{\mathbf{k}},\mathbf{\theta})=\frac{1}{|\mathbf{D}_{k}|}\sum_{x \in\mathbf{D}_{k}}\left[\sum_{i=1}^{L}\left\{\frac{\partial f_{i}(\mathbf{x},\mathbf{ \theta})}{\partial\mathbf{\theta}}\left(\frac{\partial f_{i}(\mathbf{x},\mathbf{\theta}))}{ \partial\mathbf{\theta}}\right)^{T}\right\}\right]. \tag{18}\]Note that the Equations (17) and (18) are derived from the additive Gaussian noise model by adopting the Equation (13). For the logistic model defined in (14), the approximation is given as  
\[\overline{G}(\mathbf{D}_{\mathbf{k}},\mathbf{\theta})=\frac{1}{|\mathbf{D}_{\mathbf{k}}|}\sum_{x\in \mathbf{D}_{\mathbf{k}}}\Biggl{[}\sum_{i=1}^{L}\Biggl{\{}\frac{1}{f_{i}(x,\mathbf{\theta})} \frac{\partial f_{i}(x,\mathbf{\theta})}{\partial\mathbf{\theta}}\Biggl{(}\frac{ \partial f_{i}(x,\mathbf{\theta}))}{\partial\mathbf{\theta}}\Biggr{)}^{T}\Biggr{\}} \Biggr{]}. \tag{19}\]  
Although we can reduce computational cost by using a smaller subset \(\mathbf{D}_{\mathbf{k}}\) instead of the whole set \(\mathbf{D}\), use of a small sample set leads to some loss in the estimation accuracy. Moreover, it should be also noted that the rank of \(\overline{G}(\mathbf{D}_{\mathbf{k}},\mathbf{\theta})\) is at most \(|\mathbf{D}_{\mathbf{k}}|\) and \(\overline{G}(\mathbf{D}_{\mathbf{k}},\mathbf{\theta})\) becomes singular, of which inversion does not exist. Therefore, the empirical expectation (18) and (19) are not applicable to calculate natural gradient when the size of subset \(|\mathbf{D}_{\mathbf{k}}|\) is smaller than the number of network parameters. In order to overcome this problem, we propose an iterative process for approximating \(\overline{G}(\mathbf{D},\mathbf{\theta})\) for the whole dataset \(\mathbf{D}\) by repeatedly adding up the current \(\overline{G}(\mathbf{D}_{\mathbf{k}},\mathbf{\theta})\) to the previously obtained \(\overline{G}\bigl{(}\mathbf{D}_{j},\mathbf{\theta}\bigr{)}\) (\(\underline{j}=1\ldots\mathbf{K}-1\)). Staring from an initial matrix \(\overline{G}_{0}=\mathbf{AI}\), the approximated Fisher information matrix \(\overline{G}_{\mathbf{k}}(\mathbf{\theta})\) at \(\mathbf{k}\)-th iteration is given by iterative updates from,  
\[\overline{G}_{\mathbf{k}}(\mathbf{\theta})=\biggl{(}1-\frac{1}{k+1}\biggr{)}\overline {G}_{\mathbf{k}-1}(\mathbf{\theta})+\frac{1}{k+1}\overline{G}(\mathbf{D}_{\mathbf{k}},\mathbf{ \theta}), \tag{20}\]  
where \(\mathbf{k}=1\ldots\mathbf{K}\).