# Multiple Network Fusion with Low-Rank Representation for Image-Based Age Estimation
## 2 Multiple Network Learning with Low-Rank Representation
### Multiple Network Learning  
As mentioned in the introduction, multiple feature fusion has been proved to be effective in image representation. In this way, to compute \(\overline{X}\), we propose feature learning by fusing multiple neural networks, which compute features with different neural networks and integrate them to form new features. Neural networks [25; 26] have been widely used to explore hidden representations of images and the effectiveness has been proved. Generally speaking, neural networks compute hidden representation by minimizing the loss function:  
\[\sum_{i}^{n}\parallel x_{i}-\overline{x}_{i}\parallel^{2} \tag{2}\]  
where \(\overline{x}_{i}=Wx_{i}\) is the hidden representation by mapping \(x_{i}\) with weight \(W\). The key to neural networks is optimizing \(W\), which is defined differently by different neural networks. However, they depend on a large number of training data. Usually, age estimation with a single image is achieved with insufficient training samples or classification information. Therefore, we adopt different types of neural networks to extract different types of features and fuse them to improve the descriptive power with a small number of training samples. In MNF-LRR, we use the following neural networks to represent face images. Figure 1: The flowchart of the proposed method (Multiple Network Fusion with Low-Rank Representation (MNF-LRR)). * Autoencoders (AE). Autoencoders are unsupervised to learn the hidden representation. To solve Equation (2), people usually use denoising autoencoders (DAE). In DAE, inputs \(x_{1},...,x_{n}\) are corrupted by randomly removing some features. After corruption, \(x_{i}\) is converted to \(\hat{x}_{i}\) and \(W:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\) is denoted as the transform matrix to reconstruct \(x_{i}\) with \(\hat{x}_{i}\). In this way, the squared reconstruction loss can be defined as \[\frac{1}{2n}\sum_{i=1}^{n}\parallel x_{i}-W\hat{x}_{i}\parallel^{2}.\] (3) The solution to Equation (3) depends on corrupted features of each input. To lower the variance, Marginal Denoising Autoencoders (MDA) [27] utilize multiple epochs with the training set, each epoch with different corruption settings. In this way, the overall squared loss can be transformed to \[loss(W)=\frac{1}{2 mn}\sum_{j=1}^{m}\sum_{i=1}^{n}\parallel x_{i}-W\hat{x}_{i,j }\parallel^{2}\] (4) where \(\hat{x}_{i,j}\) represents the \(j\)th corrupted features, and \(m\) is the number of epochs. To represent features with the matrix form, \(X=[x_{1},...,x_{n}]\in\mathbb{R}^{d\times n}\) is denoted as the data matrix, while the \(m\)-epochs repeated version of \(X\) is denoted by \(\overline{X}=[X,...,X]\) and the corrupted version of \(\overline{X}\) is denoted by \(\hat{X}\). Equation (4) can then be reduced to \[loss(W)=\frac{1}{2 mn}tr[(\overline{X}-W\hat{X})^{T}(\overline{X}-W\hat{X})]\] (5) \[=\frac{1}{2 mn}tr[\overline{X}^{T}\overline{X}-\overline{X}^{T}W \hat{X}-\hat{X}^{T}W^{T}\overline{X}+\hat{X}^{T}W^{T}W\hat{X}]\] We can clearly figure out that Equation (5) is a convex problem, and the global optimal solution to it can be computed by setting its partial derivative for \(W\) to 0. We then need to compute partial derivative of \(loss(W)\), which is defined as \[\operatorname*{arg\,min}_{W}loss(W).\] (6) \[\frac{\partial loss(W)}{\partial W}=0\] is set, and the close form to compute optimal \[W=\overline{X}\hat{X}^{T}(\hat{X}\hat{X}^{T})^{-1}.\] (7)
* Convolutional Neural Networks (CNNs). CNNs are constructed by alternatively stacking convolutional layers and spatial pooling layers. Convolutional layers are key to CNNs since they generate feature maps by linear convolutional filters. The feature maps are then activated by nonlinear functions, which are called activation functions. Different activation functions are defined, such as rectifier, sigmoid, tanh, and so on. Taking the Rectified Linear Units (ReLUs) as an example, the feature maps can be computed by \[F(X;W)=\mathcal{R}(x_{i})=max(0,wx_{i})=\left\{