# A Novel Parallel Auto-Encoder Framework for Multi-Scale Data in Civil Structural Health Monitoring
## 2 Parallel Auto-Encoder Framework
### The Proposed Approach
#### 2.2.4 Relationship Learning  
A stacked deep auto-encoder with multiple non-linear layers (encoding layers) followed by a linear output layer is formed as the relationship learning part in Figure 2. After finishing dimensionality reduction for all the subsets of input, the hidden features taken from the last encoding layer of each auto-encoder model for all the \(r^{th}\) samples are concatenated as one vector \(\overline{c}_{new}\)\({}^{r}\) to form a new feature vector. \(\overline{c}_{new}\)\({}^{r}\) is considered to be a better representation of the original input and it will be fed into the relationship learning part for learning the nonlinear regression against the output \(\overline{o}\)\({}^{r}\) (labelled stiffness elements). A supervised pre-training scheme is performed layer by layer via multiple simple auto-encoders in this part to initialize the layer weights that are close enough to a good solution. The cost function of \(s\)th layer for nonlinear relationship learning is defined as follows. \[J_{cost}^{s}\big{(}W,\overline{b}\big{)}=\ J_{MSE}^{s}\big{(}W,\overline{b} \big{)}+\lambda J_{weight}^{s}\big{(}W,\overline{b}\big{)} \tag{10}\]  
where  
\[J_{MSE}^{s}\big{(}W,\overline{b}\big{)}=\ \sum_{\tau=1}^{N}\ \parallel\overline{o} ^{r}-g_{s}(f_{s}(\overline{h}_{s-1}^{r}))\parallel_{2}^{2} \tag{11}\]  
Equation (10) describes an overall cost function, which includes a L2-weight decay term expressed similarly in Equation (9). Equation (11) denotes the MSE loss function, where s = {1... m} expresses \(m\) layers in the proposed relationship learning model. \(f_{s}\) and \(g_{s}\) are the encoder and decoder mapping functions respectively as mentioned in previous section. \(\overline{h}_{s-1}^{r}\) is the hidden feature vector extracted from \((s-1)\)th relationship learning layer for the \(r\)th sample where \(\overline{h}_{0}^{r}=\ \overline{c}_{new}\)\({}^{r}\). This criterion (10) is used to minimize the difference between the hidden features and the labelled outputs layer by layer. The hidden feature vector, which is in a lower dimensional representation of our original input from each previous layer, compressed gradually and trained to regress to the target. When the pre-training for relationship learning is completed, all the encoding layers are stacked together followed by a linear regression output layer forming the stacked deep auto-encoder model (see Figure 2) for fine-tuning. The final objective function is defined as:  
\[J_{cost}^{F}\big{(}W,\overline{b}\big{)}=\ J_{MSE}^{F}\big{(}W,\overline{b} \big{)}+\lambda J_{weight}^{F}\big{(}W,\overline{b}\big{)} \tag{12}\]  
\[J_{MSE}^{F}\big{(}W,\overline{b}\big{)}=\ \sum_{\tau=1}^{N}\ \parallel \overline{o}^{r}-p(\overline{c}_{new}\)\({}^{r})\parallel_{2}^{2} \tag{13}\]  
where \(p(\overline{c}_{new}\)\({}^{r})=g_{S}(f_{S}(f_{S-1}(\dots(\overline{c}_{new}\)\({}^{r}))))\) is the predicted output vector through all layers in the relationship learning model as shown in Figure 2.  
In summary, the frequencies and mode shapes are trained as input to the proposed framework while the stiffness parameters are the output vector. The relationship between input and output is learned by the proposed framework via two stages as mentioned in Sections 2.2.1 and 2.2.2 respectively. Though the structure of the deep learning framework has changed in this paper, the optimization process is similar as the original one [2] with different cost functions. For completeness, we give a brief description for training and testing as below. * Training: the training of this framework is complete via parallel dimensionality reduction and relationship learning. * Perform dimensionality reduction for several input datasets (e.g., \(\overline{\sigma}^{r}\), \(\overline{m}^{q_{1}r},\dots,\)\(\overline{m}^{q_{1}r}\)) in parallel. An unsupervised pre-training scheme is conducted for each layer of every block via a sparse auto-encoder. * Obtain learnt features from \(k\)th layer of each block and concatenate them as new input vector \(\overline{c}_{new}^{r}\).