## 4 Development of MLMs for Hybrid Seismic Analysis
### First Screening: Appropriate Type of the Recurrent Unit (Stage 3)  
The architecture of a pure RNN is described mainly by the depth (number of recurrent layers), the width (number of recurrent cells in each layer), and the type of the recurrent cell (or unit). It is very common to add a fully connected Dense layer for the output layer, as presented in the original 1997 LSTM paper [13]. More developed configurations may be considered depending on the problem treated [29]. Since the objective is to predict the shear time history from the displacement time history, a sequence-to-sequence model is adopted. Once trained and tested, the MLM would be implemented to perform hybrid seismicanalyses by predicting the shear force at each integration time step; thus, its architecture should be as small as possible to optimize the computation time. Deep RNNs may not systematically increase the performance of the network [30], as opposed to MLPs and CNNs and as long as overfitting-type problems are treated. When unfolded through time, a shallow RNN is actually a deep network. Therefore, a stacked RNN of only three layers is adopted herein: first recurrent layer as an input layer, second recurrent layer as a hidden layer, and a one-unit Dense layer as an output layer. The network depth being fixed, random searches are performed to optimize the type of the recurrent cell (first screening) and the network width (second screening).  
LSTM [13] and GRU [26] are the most common and widely used recurrent cells (or units) since they outperform traditional RNN cells [31]. They are gated recurrent units capable of learning long-term temporal dependencies, which is important in this study to predict the nonlinear shear force from the displacement time history. The GRU is a recent variant of the LSTM with less trainable parameters. However, the performance of one relative to the other is still an open research field [32]. Both LSTM and GRU cells are used and compared in this study. Box and Whisker plots are visualized to describe the statistical distribution of models' final performances. For each RNN architecture, scores (MSE on validation data after 500 training epochs) are dot marked next to the boxplot, their mean value by a cross mark, and the median value by a horizontal line within the Inter Quartile Range (IQR). Figure 5 shows the performance of LSTM (red color) and GRU (blue color) models, all with the same depth but different widths. The horizontal axis represents the number of RNN cells per each of the two recurrent layers. A 5-unit step is adopted from 10 up to 100 units. For each architecture, 10 independent runs are performed, leading to 10 different evaluations, which attest to the aleatory uncertainty of the training process. MSEs of both LSTM and GRU models are decreasing uniformly up to 35-cells width. From 35 to 100-cells width, the performance of GRU models is more stable compared to LSTM models, which have more outliers and larger IQRs. Therefore, the GRU cell slightly outperforms the LSTM cell for the problem treated in this study. The width range between 60- and 80-GRU cells are selected for the second screening since there is no major gain in accuracy and precision for larger widths, and the minimization of network architecture is a crucial criterion for an efficient implementation of MLMs in hybrid seismic analyses.  
Figure 5: 1 st Screening: Random search for the appropriate architecture (10 runs).