# A Novel Parallel Auto-Encoder Framework for Multi-Scale Data in Civil Structural Health Monitoring
## 2 Parallel Auto-Encoder Framework
### The Proposed Approach
#### 2.2.3 Sparse Activation Function and Cost Function  
Inspired by the work of SAF [PERSON] et al. (2017), sparse auto-encoder can also be used for performing dimensionality reduction with learning a sparse representation of modal data. Meanwhile, the learnt representation is considered to be not only sensitive to structural local damage but also robust to system uncertainties and measurement noise. One way to achieve actual zeros in the hidden representations is to use Rectified Linear Unit (ReLUs) [PERSON] et al. (2016) as encoder activation function over the alternative non-sparse activation functions, such as 'tanh' and'sigmoid', in the auto-encoder. In this way, a ReLU can indirectly control the sparsity in the representation [PERSON] et al. (2017). For the proposed framework, we implement the sparse auto-encoder in the parallel dimensionality reduction component to explore the advantages of our method in comparison with SAF. The reconstruction cost function of \(p\)th layer in each parallel block (Figure 2) is defined as follows.  
\[J_{cost}^{p}\big{(}W,\overline{b}\big{)}=J_{MSE}^{p}\big{(}W,\overline{b} \big{)}+\lambda J_{weight}^{p}\big{(}W,\overline{b}\big{)}++\beta J_{sparse} ^{p}\big{(}W,\overline{b}\big{)} \tag{7}\]  
where  
\[J_{MSE}^{p}\big{(}W,\overline{b}\big{)}=\sum_{\tau=1}^{N}\parallel\overline{h }_{p-1}^{\tau}-g_{p}(f_{p}(\overline{h}_{p-1}^{\tau}))\parallel_{2}^{2} \tag{8}\]  
\[J_{weight}^{p}\big{(}W,\overline{b}\big{)}=\frac{1}{2}\sum_{l=p-1}^{p}\sum_{i =1}^{s_{l}}\sum_{j=1}^{s_{l+1}}(w_{ji}^{(l)})^{2} \tag{9}\]  
Equation (7) denotes an overall cost function, which includes a Mean Squared Error (MSE) loss term and a weight decay regularization term, where \(W\) is the weight matrix and \(\overline{b}\) is the bias vector. To optimize the parameters \(W\) and \(\overline{b}\), usually MSE is employed as the loss function in Equation (8), where \(p=\{1\dots k\}\) with k being the number of hidden layers in dimensionality reduction component for each block, N is the total number of training samples. \(f_{p}\) represents the encoder function for \(p\)th layer where a ReLU is used to impose sparsity on the hidden representation. However, applying too strong sparsity may hurt inference performance, since it reduces the effective capacity of the model. Therefore, a sparse regularization term defined in Equation (6) is added to Equation (7), where \(\beta\) is the hyper-parameter to control the trade-off between the reconstruction loss and the applied constraints on the solution. \(g_{p}\) is the decoder function for \(p\)th layer and it is set to be a linear activation function \(g(x)=x\) to reconstruct the real values of input to output \(\overline{h}_{p-1}^{\tau}\) is the compressed hidden feature vector taken from \((p-1)\)th layer in the \(r\)th sample with \(\overline{h}_{0}^{\tau}=\overline{c}_{ori}^{\tau}\). Equation (9) is a L2-weight decay regularization term, where \(w_{ij}^{l}\) is an element in weight matrix \(W\) and \(s_{l}\) denotes the number of hidden neurons in layer \(l\). The L2-weight decay term is added to avoid over-fitting by shrinking the weights on features. In Equation (7) a hyper-parameter \(\lambda\) is utilized to penalize the weight decay term. It is reminded that we carry out dimensionality reduction separately in the proposed model for simplicity with assumption that all blocks have the same number of layers.  
An unsupervised layer-wise pre-training based on sparse auto-encoders is performed for all layers of every block in the parallel structure. The sparse hidden representation of each pre-trained auto-encoder is taken out at the input for the next layer. Finally, the features learnt from \(k\)th layer of each block are concatenated as new input vector \(\overline{c}_{new}\)\({}^{r}\) and fed into a nonlinear relationship learning component as described in Section 2.2.4.