# Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
## 2 Gradient Descent Learning of Stochastic Neural Networks  
### Stochastic Neural Networks  
Since the natural gradient is derived from stochastic neural network models, let us start from the brief description of the two popular stochastic models. Although typical neural networks, including deep networks, are defined as deterministic input-output mapping functions \(\mathbf{y}=f(\mathbf{x},\mathbf{\theta})\) of input \(\mathbf{x}\), output \(\mathbf{y}\), and parameter \(\mathbf{\theta}\), the observed data for training the networks always have inevitable noises and thus the input-output relation can be described in the stochastic manner. In other words, the observed output \(\mathbf{y}\) can be regarded as a random vector that is dependent on the deterministic function \(f(\mathbf{x},\mathbf{\theta})\) and some additional stochastic process that is described by the conditional probability \(p(\mathbf{y}|\mathbf{x},\mathbf{\theta})=p\big{(}\mathbf{y}|\mathbf{f}(\mathbf{x},\mathbf{\theta})\big{)}\). Then the goal of learning is to find an optimal value of parameter \(\mathbf{\theta}^{*}\) that minimizes the loss function defined as negative log likelihood of given input-output sample \((\mathbf{x},\mathbf{y})\). The loss function can then be written as  
\[l(\mathbf{x},\mathbf{y};\mathbf{\theta})=-\log p(\mathbf{x},\mathbf{y};\mathbf{\theta})=-\log p(\mathbf{y }|\mathbf{x},\mathbf{\theta})q(\mathbf{x})=-\log p(\mathbf{y}|\mathbf{x},\mathbf{\theta})-\log q(\mathbf{ x}), \tag{1}\]  
and the optimal parameter is described as  
\[\mathbf{\theta}^{*}=\text{argmin}_{\mathbf{\theta}}\{l(\mathbf{x},\mathbf{y};\mathbf{\theta})\}= \text{argmin}_{\mathbf{\theta}}\{\log p(\mathbf{y}|\mathbf{x},\mathbf{\theta})\}. \tag{2}\]  
Note that the last term \(\log q(\mathbf{x})\) in equation (1) is independent of parameter \(\mathbf{\theta}\) and can be ignored in the objective function for optimization. Based on the general definition, the conventional neural networks can be regarded as a special case with a specific conditional probability distribution, \(p(\mathbf{y}|\mathbf{x},\mathbf{\theta})\). For example, consider that the output \(\mathbf{y}\) is observed with additive noise to the deterministic neural networks function such as  
\[\mathbf{y}=\mathbf{f}(\mathbf{x},\mathbf{\theta})+\mathbf{\xi}, \tag{3}\]  
where \(\mathbf{\xi}\) is a random noise vector. When we assume that the random vector \(\mathbf{\xi}\) is subject to the standard Gaussian distribution, its probability density \(p(\mathbf{\xi})=p(\mathbf{y}-\mathbf{f}(\mathbf{x},\mathbf{\theta}))\) is defined as the normal distribution function, and its loss function becomes the well-known squared error function, which can be written as  
\[l(\mathbf{x},\mathbf{y};\mathbf{\theta})=\|\mathbf{y}-\mathbf{f}(\mathbf{x},\mathbf{\theta})\|^{2}. \tag{4}\]  
Therefore, the stochastic neural network model with additive Gaussian noise is equivalent to the typical neural network model trained with squared error function, which is widely used for regression task. On the other hand, in case that the output \(\mathbf{y}\) is a binary vector, the corresponding probability distribution can be defined by using a logistic model, such as  
\[p(\mathbf{y}|\mathbf{x},\mathbf{\theta})=\prod\
olimits_{i=1}^{L}f_{i}(\mathbf{x},\mathbf{\theta} )^{y_{i}} \tag{5}\]  
where \(f_{i}(\mathbf{x},\mathbf{\theta})\) and \(y_{i}\) are the \(i\)-th component of \(L\)-dimensional vector \(\mathbf{f}(\mathbf{x},\mathbf{\theta})\) and \(\mathbf{y}\), respectively. Since the typical problems with binary target output vector is pattern classification, the logistic model is appropriate for \(L\)-class classification tasks.