12 Inthispaper,anovelmethodisproposedbasedonawindowedone-dimensionalconvolutional 13 neural network (1D CNN) for multiclass damage identiﬁcation using vibration responses of a 14 full-scale bridge. The measured data is ﬁrst augmented by extracting samples of windows of raw 15 accelerationtime-seriestoalleviatetheproblemofalimitedtrainingdataset. 1DCNNisdeveloped 16 toclassifythewindowedtime-seriesintomultipledamageclasses. Thedamageisquantiﬁedusing 17 the predicted class probabilities, and the damage is localized if the predicted class is equal to 18 the assigned damage class, exceeding a threshold associated with majority voting. The proposed 19 network is optimally tuned with respect to various hyper-parameters such as window size, random 20 initializationofweights,etc.,toachievethebestclassiﬁcationperformanceusingaglobal1DCNN 21 model. The proposed method is validated using the Z24 bridge benchmark data for multiclass 22 classiﬁcation for two diﬀerent damage scenarios, namely, pier settlement and rupture of tendons, 23 1 Sony, September 22, 2021 underthevariousextentofthedamage. Thedamageidentiﬁcationiscarriedoutonvariousbridge 24 componentstocollectivelyidentifythestructuralcomponentwithadamagedsignature. Theresults 25 show that the proposed windowed 1D CNN method achieves an accuracy of 97%, and performs 26 well with diﬀerent types of damage. 27 INTRODUCTION 28 In civil infrastructure, continuously increasing heavy traﬃc, unexpected natural calamities 29 and human-made damages reduce their load-bearing capacity and service life. With ageing, the 30 structures exhibit various damage signatures in several critical locations. In the absence of timely 31 repair and maintenance, progressive damage leads to the collapse of structures. Despite the 32 simplicity,thetraditionalmanualinspectionsuﬀersdiﬃcultywhilescanningtheinaccessibleareas 33 in large structures such as bridges or tall buildings. Over the past few decades, structural health 34 monitoring (SHM) has been a promising tool to supplement the knowledge of structural integrity 35 overtime. However,eﬃcientdiagnosisandprognosisoflarge-scaleinfrastructurerequireareliable 36 assessment of its damage under in-service conditions. SHM aims to provide suitable diagnostics 37 and prognosis, and assist infrastructure owners and decision-makers in maximizing the safety, 38 serviceability, and functionality of critical structures. An autonomous SHM will allow eﬃcient 39 andcost-eﬀectivedisastermanagementandwillleadtoresilientinfrastructurewithfasterrecovery 40 under natural disasters. In this paper, an autonomous multiclass damage identiﬁcation method is 41 proposedbyutilizingartiﬁcialintelligenceinthesequentialdata,suchasvibrationmeasurements. 42 Data-driven damage diagnosis is a critical component of infrastructure asset management 43 (Piryonesi and El-Diraby 2019). Although there is a plethora of research on parametric methods 44 based on time-frequency (TF) decomposition techniques (Staszewski and Robertson 2006, Sadhu 45 et al. 2017, Almasri et al. 2020, Barbosh et al. 2020, Kankanamge et al. 2020, Sony and 46 Sadhu 2020, Sony and Sadhu 2021), non-parametric methods (Nakamura et al. 1998, Wang 47 and Ong 2015, Abdeljaber and Avci 2016) have shown signiﬁcant promises in data-driven SHM 48 methods. Parametric methods include extracting dynamic parameters such as modal parameters, 49 whileinferringthechangeintheseparameterstodetectanypossiblechangesinthestructures. On 50 2 Sony, September 22, 2021 the other hand, non-parametric methods include extracting parameters that are estimated based on 51 the computational models, where the parameters are mathematically derived in a statistical sense. 52 Structuraldamageidentiﬁcationcanbeconsideredasapatternrecognition-basednon-parametric 53 problem,whichisdividedintothreestages,namely,dataacquisition,featureextraction,andfeature 54 classiﬁcation. With proliferation of various machine learning (ML) algorithms, the SHM com- 55 munity has prominently used various supervised learning algorithms (Hou and Xia 2020, Avci et 56 al. 2021). In (Gardner et al. 2020), the authors explained the interface between nondestructive 57 evaluation and machine-learning-based SHM for damage detection. In another study, Su et al. 58 (2020) presented a critical review of ﬁeld monitoring of high-rise structures. The study reviewed 59 techniquesforcomfortassessment,seismicandwindeﬀects,andenvironmentaleﬀectonmonitor- 60 ing of super-tall structures. Recently, the SHM community has explored both vibration and image 61 data for structural damage identiﬁcation and localization. 62 Withadvancementsinartiﬁcialintelligence,image-basedSHMhasgarneredasaninexpensive 63 waytomonitorlargescalestructuresusingConvolutionalNeuralNetworks(CNNs). Whileimage- 64 based 2D CNN techniques remain a popular method for SHM (Kumar et al. 2019, Chang and Chi 65 2019, Gulgec et al. 2019, Sony et al. 2019), they involve signiﬁcant complexity in obtaining a 66 largeamountoflabelleddata,pre-processingandclassifyingtheimages. Asasolution,researchers 67 have studied algorithms that directly operate on the sequential data such as vibration data. Guo 68 et al. (2014) proposed sparse coding as a feature extraction method for unlabeled acceleration 69 measurements obtained from wireless sensors. The damage classiﬁcation was carried out using 70 a CNN, and the results were compared with logistic regression and decision trees. A three-span 71 bridgewasconsideredtoevaluatetheeﬃcacyoftheproposedmethod,anditwasshownthatsparse 72 coding-CNN based method outperforms other methods with an accuracy of 98%. Gulgec et al. 73 (2017) conducted a simulation study on a steel gusset plate connection by varying the size and 74 location of the damage. A CNN was used to classify damaged signals, and the proposed method 75 achieved a testing error of 2% and showed robustness against environmental noise. 76 Fallahian et al. (2018) explored the applicability of dynamic features such as mode shapes, 77 3 Sony, September 22, 2021 frequency response functions, and natural frequencies as damage indicators under varying tem- 78 peratures. The authors used a combination of coupled sparse coding and deep neural network as 79 an ensemble method for damage detection and localization. The proposed method was validated 80 on a numerical truss bridge and experimental benchmark dataset. Bao et al. (2019) proposed a 81 CNN-basedanomalydetectionusingaccelerationmeasurementsbyconvertingthemintograyscale 82 images. The authors used several anomaly parameters such as missing, minor, outlier, square, 83 drift,andtrenddatapointstotrainthedatasetsusingastackedautoencoderarchitecture. Inanother 84 recentstudy,Shangetal. (2020)proposeddeepconvolutionaldenoisingautoencodersforstructural 85 damagedetection. Theproposedmethodextracteddamagefeaturesfromﬁeldmeasurementsunder 86 environmental noise. However, most of these methods are based on 2D CNN (Sony et al. 2021), 87 which are primarily relied on images. 88 RECENT DEVELOPMENT OF 1D CNN-BASED SHM 89 Recently, 1D-CNN has shown promising results in capturing the temporal information and 90 damagedetectionusingvibrationdata. Forexample,Abdeljaberetal. (2017)introduced1DCNN 91 forreal-timevibration-baseddamagedetection. Theauthorstrained1DCNNonavibrationsignal 92 database obtained from a truss, named Qatar Grandstand, by damaging each joint and keeping the 93 other joints undamaged. The proposed model was trained individually on each joint, and near- 94 perfect classiﬁcation accuracy was proposed. However, the proposed method was not tested for a 95 multiclass damage scenario. Zhang et al. (2019) utilized the computational powers of 1D CNN 96 to detect changes in structural parameters such as stiﬀness and mass. Three diﬀerent structural 97 components were used for data acquisition and model validation, namely, T-shaped steel beam, 98 short and long steel girder bridge, and a mean classiﬁcation accuracy of 98% was achieved. In 99 anotherstudy,Nietal. (2019)showedtheapplicabilityof1DCNNwithautoencodersforanomaly 100 detection under data compression. The proposed algorithm was validated using a long-span 101 suspension bridge with an accuracy of 97.53%. 102 A recent study by Azimi and Pekcan (2020) explored the concept of transfer learning in 103 vibration measurements. The authors used a four-story IASC-ASCE SHM model for numerical 104 4 Sony, September 22, 2021 training, and the proposed model was tested on experimental studies using IASC-ASCE SHM 105 benchmarkbuildingandtheQatarUniversityGrandstandSimulatorwithanaccuracyof90-100%. 106 Recently, Sharma and Sen (2020) showed the applicability of 1D CNN for damage detection in 107 the structural frames. Experimental validation was performed on a 2D-steel frame with diﬀerent 108 damage location and severity of the damage. The method was shown to identify diﬀerent damage 109 scenarios and the false-positive rate was also evaluated and found to be well within the acceptable 110 limits. Furthermore, Liu et al. (2020) conducted a study by integrating traditional TF methods 111 with the capability of neural networks. The authors used transmissibility function-based 1D CNN 112 to eﬀectively identify damage at the ASCE SHM benchmark structure. The proposed method was 113 compared with the time-series and fast Fourier transform-based frequency-domain information, 114 where the TF signals exhibited more signiﬁcant damage-sensitive features. 115 Overall, 1D CNN exhibited superior performance over artiﬁcial neural networks (ANNs) in 116 the context of computation eﬃciency and noise insensitivity for big data (Kiranyaz et al. 2019). 117 Recently,Baoetal. (2020)evaluatedacombinationofﬁniteelementmethod(FE)and1DCNNfor 118 localizingdamageforajacket-typeoﬀshorestructure. However,thedatawasgeneratedsynthetically 119 usingaﬁniteelementmodel,whichmightnotresembletheactualreal-worlddatawithoperational 120 andenvironmentalnoisecontamination. Inaddition,thedamagewasinducedartiﬁciallyusingthe 121 FE model that was clearly distinguishable from an undamaged structure that does not concur with 122 thereal-worlddata. Inanotherrecentstudy,Sarawgietal. (2020)used1DCNNforeachjointinan 123 experimentalsetuptoidentifyhypotheticaldamagesimulatedusingremovaloradditionofexternal 124 braces. However, such individual CNN model per damaged node does not scale well to real-world 125 structureswithmulticlassdamagesduetocomputationalcomplexity. Moreover,theneedforlarge 126 datasets and the selection of the appropriate architecture of the network still remain a challenge. 127 Theproposedresearchexplorestheexistingchallengesofmulticlassdamageidentiﬁcationina 128 full-scalebridge. Unlikethesimulateddata,thereal-worlddataislimitedandnoise-contaminated, 129 where multiclass damage localization becomes a signiﬁcant challenge. In this paper, an enhanced 130 windowed-1D CNN is explored for multiclass damage localization with varying damage severity 131 5 Sony, September 22, 2021 underdiﬀerentdamagescenarios. Theissueofthelimiteddatasetissolvedbyaugmentingthedata 132 using windowing the acceleration measurements, and the classiﬁcation results are improved using 133 anovelmajorityvotingapproachofaglobal1DCNNmodel. Theeﬀectivenessoftheproposed1D 134 CNN model is evaluated using a systematic suite of hyperparameter tuning such as window size, 135 randominitializationofweights,andoptimumlearningratestoshowtherobustnessoftheselected 136 optimal architecture of the 1D CNN network. 137 Thepaperisstructuredasfollows. Abriefintroductionofthestructuraldamageidentiﬁcation, 138 its need, and a literature review based on 1D CNN techniques are presented ﬁrst, followed by the 139 theoretical background of the proposed algorithm. The capability of the proposed algorithm to 140 identify multiclass damage, the importance of hyperparameter tuning and various metrics to show 141 the damage parameters of the structures is explained later along with the key conclusions. 142 PROPOSED METHODOLOGY 143 Formulation of 1D CNN 144 Convolutional Neural Networks (CNNs) are a type of feedforward neural network model that 145 isdesignedtoapproximateafunction H=5¹G;\º. Forclassiﬁcation,themodelmapsaninput Gto 146 a category (class) H. The parameters \are learned to best ﬁt a given training dataset by a gradient 147 descentoptimizationalgorithm(Goodfellowetal. 2017). ThemostcommontypeofCNNsare2D 148 CNNs used in the ﬁeld of computer vision for tasks such as image classiﬁcation, where the inputs 149 Gare matrices (2D-shaped) representing images. 1D CNNs are a simpler variant of CNNs, where 150 theinputsGarevectors(1D-shaped),typicallyrepresentingatime-series. Theyarecommonlyused 151 for tasks involving sequential signal processing such as speech recognition (Kiranyaz et al. 2019). 152 Since the last few years, 1D CNNs became popular in SHM due to its computational simplicity in 153 comparison to its parent family of 2D CNNs as it requires simple array application and a shallow 154 network. 155 A typical 1D CNN architecture used in this study is shown in Fig. 1. It consists of an input 156 layer (time-series), multiple alternating convolutional and pooling layers, and one or more fully 157 connected layers at the end. An input time-series Gpresented to the input layer is transformed by 158 6 Sony, September 22, 2021 theforwardpassthroughthehiddenlayersandtheoutputsoftmaxlayerproducestheclasslabel H. 159 When the number of hidden layers is high, this architecture is referred to as a deep convolutional 160 neural network. The convolutional layer is the core building block of a CNN. The parameters of 161 each convolutional layer consist of a set of learnable kernels, which are deﬁned by a kernel length 162 (<). Convolutional layers have a reduced number of parameters in comparison to fully connected 163 layers as a single kernel shares the weights for spatial locations in the input. The convolution 164 process can be expressed as Eq. 1 (Goodfellow et al. 2017): 165 H¹=º=5¹G¹=º ¹<ºº (1) 166 whereG¹=ºistheinputvectoroflength =and¹<ºiskerneloflength <. Thesymbol denotes 167 the traditional 1D convolution between two signals as deﬁned in Eq. 2, 168 G¹=º ¹<º==Õ :=0G¹:º¹=:º (2) 169 Thefunction 5iscalledanactivationfunction,whichistypicallyanon-lineartransformationonthe 170 traditional 1D convolution. Non-linear activations enable the network to learn complex mappings 171 betweentheinputsignalandtheclasslabels. Inthisstudy,RectiﬁedLinearUnit(ReLU)isusedas 172 theactivationfunction,whicheﬀectivelyremovesnegativevaluesfromanactivationmapbysetting 173 them to zero. A pooling layer is added after the convolution layer to sub-sample the convolution 174 output. The pooling operation reduces the dimensionality of a given mapping while highlighting 175 the prominent feature and it also helps to reduce overﬁtting. Max pooling refers to selecting the 176 maximum value in a window that slides over the input map. In Fig. 1, the max pooling layer 177 has reduced the size of each convolution output size by a factor of two. For the output layer, the 178 choice of activation function depends on the type of output. For classiﬁcation problems, SoftMax 179 activations are preferred. SoftMax function for a n-class problem (representing =probabilities of 180 input belonging to each of n-classes) is shown in Eq. 3. 181 7 Sony, September 22, 2021 %¹class =9jIº=4I9 Í= :=14I:(3) 182 whereI9is the input to the softmax node 9from the previous layer. 183 Multiclass damage detection using windowed-voted 1D CNN 184 A method based on a 1D CNN model is proposed to classify the vibration measurement into 185 multiple damage classes (i.e., multiclass classiﬁcation). First, each acceleration signal is scaled to 186 ﬁtastandardnormaldistribution. Thescalingimprovestheconvergencerateofmodelstrainedon 187 thedatasetandpreventsanyoutlierfromdominatingtheinput(IoﬀeandSzegedy2015). Inorderto 188 trainaneuralnetworktoachievehightestaccuracy,alargeamountoftrainingdataisrequired. Due 189 to the scarcity of vibration-based multiclass data for civil infrastructure, it is critical to augment 190 the training dataset. In the proposed method, the dataset of raw acceleration signals is augmented 191 by extracting windows of samples from the original signals, as shown in Fig. 2. Each extracted 192 window is assigned the same damage level as the original time-series. A new dataset is formed by 193 takingtheextractedwindowsandtheirlabelsasthetraininginstances. Inadditiontoincreasingthe 194 number of training instances, this windowing technique also reduces the data dimensionality (i.e., 195 shorter input signals), which allows training machine learning models with less over-ﬁtting. 196 The dataset is then split into training, validation and test sets. A 1D CNN model is trained 197 on the dataset using a standard gradient descent optimizer. Hyper-parameters of the 1D CNN 198 model include the number of layers and number of nodes in each layer, activation function and 199 the kernel size in convolutional layers. Additionally, the length of the extracted windows is also 200 a hyper-parameter. Finding the optimal hyper-parameters (also known as hyper-parameter tuning) 201 is conducted using a random search over the parameter space and selecting the conﬁguration that 202 yields the high accuracy on the validation set (Bergstra and Bengio 2012). 203 In order to classify a new acceleration signal at test time, windows are extracted as before 204 andfedintothetrained1DCNNmodel,whichoutputsasetofclassiﬁcationprobabilitiesforeach 205 window. Thepredictedsetofclassiﬁcationprobabilities %?¹H2ºforafullaccelerationmeasurement 206 8 Sony, September 22, 2021 isobtainedbysummingtheclassprobabilitiesofallthewindowsequencesinasingletime-series. 207 The class with the maximum probability is the predicted damage level classiﬁcation of the series 208 as shown in Fig. 2. It may be noted that this is equivalent to voting for the majority classiﬁcation 209 labeloftheindividualwindowsequencestoarriveatthepredictionbasedontheentiretime-series. 210 Although it is possible for a well-trained model to misclassify some of the individual window 211 sequences that comprise a time-series, the probability of misclassifying a majority of them is 212 very low (Sony 2021). Therefore, the voting process improves the prediction accuracy and other 213 evaluation metrics in the time-series. This result is empirically observed in the full-scale studies 214 and discussed in model performance section. The proposed machine learning pipeline for the 215 multiclass damage classiﬁcation problem is shown in Fig. 3. 216 Performance criteria 217 In machine learning, a number of performance metrics are used to evaluate the eﬃcacy of the 218 computational model. A brief description of metrics used to evaluate classiﬁcation models in the 219 context of SHM is provided below. The confusion matrix is a tabulation of classiﬁcations made 220 by a model, typically with the actual class in rows and predicted class in columns. There are 221 variousmetricsthatarederivedfromconfusionmatrixandarepresentedinTable1. Inthecontext 222 of SHM and multiclass damage detection, ROC-AUC, Accuracy, FNR and F1 score are used to 223 evaluatetheperformanceoftheproposedmethod. Accuracyisaprimaryperformancemetricused 224 to evaluate the ability of a model to correctly classify the data samples into various class labels. 225 AnotherimportantmetricthathasnotbeendiscussedintheliteratureisFNR.IntheSHMcontext, 226 it is critical that a damage detector have a low false negative rate, as a false negative corresponds 227 to the potentially catastrophic case of a damaged signal being classiﬁed as an undamaged signal. 228 Furthermore, a damage detector must have high values for accuracy, and F1 score. Additionally, 229 twocurvesareusedtoevaluatethetrade-oﬀbetweenperformancemetrics. Thereceiveroperating 230 characteristic (ROC) curve shows the trade-oﬀ between True Positive (TP) Rate (TPR) and False 231 Positive(FP)Rate(FPR)asthedecisionthresholdoftheclassiﬁervaries. Theprecision-recall(PR) 232 curve shows the trade-oﬀ between precision and recall as the decision threshold of the classiﬁer 233 9 Sony, September 22, 2021 varies. The area under the curves (AUC) for both ROC and PR is a summary metric that reﬂects 234 the level of possible trade-oﬀ. Both curves are useful for an academic and practicing engineer to 235 ﬁnd a suitable decision threshold. 236 Algorithm for Component-level Damage Identiﬁcation 237 Component level damage is identiﬁed for multi-class problems using Algorithm 1. The entire 238 structureismodeledasoneexperimentratherthanmodelingeachsensorseparatelyasin(Abdeljaber 239 etal. 2017)andpredictionprobabilitiesareacquiredforeachsensorlocation. Thisprovidescontrol 240 over all the features for one experiment and prevents errors arising from multiple models working 241 on diﬀerent sensors. However, as there are multiple sensors covering the entire structure, diﬀerent 242 structural components are collectively used to localize damage using a limited number of sensors 243 for each component. The damage is conﬁrmed if the true predicted probability class is equal to 244 allocated class label for all cumulative windowed series for each sensor location and %?¹H2ºis 245 greater than the threshold. 246 Algorithm 1: Multiclass damage identiﬁcation Input:A signalG¹Cº Output: Prediction probabilities %?¹H2ºfor damage component level identiﬁcation. (a) The acceleration data is pre-processed into multiple windows of time-series and damage class-label is allocated to each windowed data. (b) The structure is modeled as whole (i.e., a global CNN model) as compared to per sensor (Abdeljaber et al. 2017) for computational eﬃciency and ease of modeling. (c) The windowed data is trained using 1D CNN with optimal parameters (e.g., window size, random initialization of weights etc.) and tested on a separate dataset. (d) The probabilities of classiﬁcation are obtained for each sensor of every windowed series and damage is conﬁrmed if true predicted probability class of a sensor is equal to allocated class label, while an average of %?¹H2ºfor various structural components is used as threshold of damage. (h) If the%?¹H2ºthreshold, a localized damaged is conﬁrmed. FULL-SCALE STUDY 247 Details of the Z24 Bridge 248 Damage detection, where classiﬁcation is more than two classes, is considered as a multiclass 249 problem. In this study, two types of damage cases are considered, namely, rupture of tendons, 250 10 Sony, September 22, 2021 and pier settlement of a full-scale bridge, namely, Z24 Bridge (Maeck and Roeck 2003). All the 251 damage classes have multiple damage levels are used to evaluate the performance of the proposed 252 methodformulticlassdamagedetection. ThebridgewaslocatedinthecantonBernnearSolothurn, 253 Switzerland. Itwasaclassicalpost-tensionedconcretetwo-cellbox-girderbridgewithamainspan 254 of 30 m and two side spans of 14 m, as shown in Fig. 4. The bridge was demolished at the end of 255 1998becauseanewrailwayadjacenttothehighwayrequiredanewbridgewithalargersidespan. 256 During the demolition, the bridge data was acquired using 15 accelerometers placed at diﬀerent 257 spansofthebridge,asshowninFig. 5. Thebridgewasexcitedbytwoshakers,oneatthemid-span 258 ofthebridgeandanotherataside-span. Becauseofthesizeofthebridge,responsewasmeasured 259 inninesetupsofupto15sensorseach,withthreeaccelerometersandthetwoforcesensorscommon 260 in all setups. The data was sampled at 100 Hz, and the data was made publicly available by the 261 researchers at the Katholieke Universiteit Leuven (https://bwk.kuleuven.be/bwm/z24). The data 262 was acquired by performing various progressive damage scenarios during the demolition period. 263 Forthebrevityofthisstudy,onlytwodiﬀerentdamagescenariosareconsidered: ruptureoftendons, 264 andpiersettlement. Itmaybenotedthateachdamagescenariohavediﬀerentclassesofdamage,and 265 theywerechosentoevaluatetheperformanceoftheproposedmethodtoclassifyvariousmulticlass 266 damage cases. For example, rupture of tendons have three levels, and pier settlements have four 267 levels, and together they made a case of two separate damage classes. For detailed explanation of 268 howthedamageswereinducedinthebridge,thereadersaresuggestedtorefer(RoeckandTeughels 269 2004). Thereferenceundamagedconditionisconsideredasclass-zeroforallthecasesandtheother 270 damageswereassignedclassesstartingfrom1to =dependinguponthelevelofdamage,asshown 271 inTable2. Forexample,inthecaseofruptureoftendons,thedamagewasinducedatﬁrst,rupture 272 oftwotendons,andsecond,ruptureoffourtendons,third,ruptureofsixtendons,therebycreating 273 threeclassesofdamagesforruptureoftendons. Similarly,therearefourclassesforpiersettlement. 274 Theruptureoftendondatasetcontains1,231time-series(i.e.,vibrationsignals)andtheloweringof 275 pier dataset contains 1,056 time-series. Each time-series contains 65,530 samples. Both datasets 276 areclass-balanced,andtheyaresplitintothreesetsoftrain-validation-testas60%-10%-20%ofthe 277 11 Sony, September 22, 2021 original suite of time-series. 278 Hyper-parameters of the 1D CNN model 279 Avarietyofhyper-parametersisincorporatedtoimprovetherobustnessoftheproposedmodel 280 and avoid overﬁtting. For example, learning rate is evaluated for various scenarios to improve 281 the accuracy while reducing the overﬁtting by empirically changing the gradient during back- 282 propagation. In this study, ﬁrst a search space of hyper-parameters is designed by considering 283 a wide range of values for each hyper-parameter. In order to make the search feasible, extreme 284 values(e.g.,verysmallwindowsizes)thatyieldedpoorperformanceorunstabletrainingdynamics 285 (training error does not decrease) were removed from the search space. A random search is 286 performed on this hyper-parameter space, and the set of hyper-parameters that yielded highest 287 accuracy is selected as the optimal conﬁguration. The evaluation metrics based on the proposed 288 methodaredescribedlaterwithacomparisonbetweenwindow-votedandnon-votedresults. Inthis 289 study, a range of hyper-parameters are selected ﬁrst and tuned using random search algorithm to 290 achieveasetofhyper-parameterthatprovidestheoptimalaccuracy. Therangeofhyper-parameters 291 used for 1D CNN is presented in Table 3. For example, window size is adopted within a range of 292 32-512 samples. Window size is the only external parameter and is decided by the user. Thus, a 293 sensitivity analysis is performed to understand the behavior of the performance evaluation metrics 294 (%<) under diﬀerent window sizes ( F). Two diﬀerent metrics, accuracy and FN, are used for 295 sensitivityanalysisastheyrepresentoverallaccuracyofthemodelandfalse-negativealarmcritical 296 for civil infrastructure. 297 Theoptimalhyper-parametersofthisdatasetareobtainedaftertuningandarepresentedforall 298 themodelsinTable4. Ananalysisisperformedtounderstandtheeﬀectof Fversus%<. Theresults 299 are shown for various damage cases in Fig. 6. For example, Fig. 6 (a-b) shows that the optimal 300 performance is achieved at F=256, with highest ROC and accuracy, and lowest false-negative. 301 Although, the FNR remains consistent after F=512 and other metrics are at their peak, however, 302 due to larger F, the data size reduces per damage class and it leads to over-ﬁtting of the data. 303 12 Sony, September 22, 2021 Random initialization of weights 304 Deep learning algorithms are iterative and require the user to specify value of initial weights 305 of neurons to initiate the iteration and its optimization. In practice, all weights in the model are 306 randomly drawn from a Gaussian or uniform distribution (Goodfellow et al. 2016). However, the 307 scale(loworhighmagnitude)hasalargeeﬀectonboththeoutcomeandoptimizationprocedure. In 308 this study, random initialization with early stopping criteria is used and Adam optimizer (Kingma 309 and Ba, 2014) is used with dropout in each layer for regularization. After acquiring the optimal 310 tunedparameters,aparametricstudyisconductedtounderstandvarianceinthemetricsof1DCNN 311 modelforrandominitializationofweights. Themetricsusedforevaluatingrandominitializationof 312 weightsareROC-AUC,accuracy,FNR,andF1scoreandareshowninTable5. Itcanbeobserved 313 that for pier settlement, the mean ( `) of ROC-AUC is 0.97 with an accuracy of 0.85. The FNR is 314 0.15 and the standard deviation ( f) for all the trials is at its minimal of 1%. Similarly, for rupture 315 of tendons, the ROC-AUC is 0.92 with an accuracy of 0.67 and FNR of 0.33 with minimal fof 316 2%. 317 Eﬀect of window size 318 The window size used to augment the data is an external parameter apart from other model 319 parameters and it is critical to understand its eﬀect on model performance. It can be observed that 320 thebestperformancewithacombinationofmaximumROC-AUCandaccuracyandminimumFNR 321 is achieved at 256 samples per window. It is shown in Fig. 6 (a), ROC-AUC increases to 1.0 at 322 512,800,1024samplesperwindow,however,itleadstooverﬁttingwithincreasedFNR.Asimilar 323 result can be observed from Fig. 6 (b) with optimal performance at 256 samples per window. 324 Model performance 325 The optimal parameters are ﬁrst used to evaluate the performance of the proposed model on 326 an entire series versus windowed and voted-windowed samples. The reason for comparison of the 327 entiretime-series,windowedandvotedwindowsistoshowtheimprovedperformancebyvotingthe 328 windowedsamples. Itshouldbenotedthatmodelperformanceonentireseriesresultsinverypoor 329 accuracyduetononlinearity,andnonstationarityinthesignal. Itmaybenotedthatmicro-averaged 330 13 Sony, September 22, 2021 is the average of area under the curve for all the classes. It can be observed from Figs. 7 (a) & 331 (b) for pier settlement, the ROC-AUC of a full-series signal is merely 0.56 while PR-AUC is only 332 0.21. A similar observation can be deduced in case of rupture of tendon in Figs. 8 (a) & (b), 333 where ROC-AUC and PR-AUC is 0.55 and 0.28, respectively. However, it is observed that voting 334 on windowed dataset increases accuracy considerably and it exhibits in ROC-AUC and precision- 335 recall (PR)-AUC curves, as presented in Figs. 7(c-f), and 8(c-f), respectively. The accuracy in 336 terms of ROC-AUC and PR-AUC increased by 71% and 314% for pier settlement case, and by 337 58% and 153.5% in case of rupture of tendons. Moreover, as shown in Fig. 7, majority voting on 338 sampleshasfurtherimprovedtheAUCforbothROCandPR.Itcanbeobservedthatincaseofpier 339 settlement, there is meager increase on ROC-AUC, however, there is a considerable improvement 340 in the area under the curve for PR. This behaviour can be attributed to a more localized damage 341 in case of pier settlement. Similarly, as observed in Fig. 8, when the damage was considerably 342 distributedincaseofruptureoftendons,majorityvotingonwindowshighlyincreasethePR-AUC 343 for rupture of tendons. 344 It can be observed that majority voting on windows of distributed damage signal increases 345 the probability considerably by allocating the majority class and ignoring the non-prominent class 346 along with augmenting the data samples per class. Another critical performance metric, FNR 347 is used to evaluate the false negative alarm of damage in the proposed model. It can be noted 348 that the FNR reduces as the entire time-series is windowed and further reduces with the majority 349 voting. Forexample,incaseofpiersettlement,FNRreducesfrom0.80to0.17fortheentireseries 350 versus majority voted augmented dataset. Similarly, FNR reduces from 0.71 to 0.34 for rupture of 351 tendons, as shown in Table 6. The label 0, 1, and 2 are used to denote performance metrics of the 352 entire time-series, windowed time-series and majority voted-windowed time series, respectively. 353 The tuning process and the resulting performance improvement provide adequate justiﬁcation to 354 counter any potential errors caused by the windowing. 355 14 Sony, September 22, 2021 Component-level damage identiﬁcation 356 Component level damage identiﬁcation is performed using Algorithm 1, for two multiclass 357 damagescenario,namely,pier-settlementandruptureoftendons. Thesensorlocationsareidentiﬁed 358 ﬁrst, then, three diﬀerent structural components of the bridge are used to localize the damage 359 causedbythepiersettlementandruptureoftendonsduringthedemolitionperiodofthebridge. An 360 undamagedpier(Utzenstorf),bridgedeck,anddamagedpier(koppigen)areusedforrepresentation 361 of predicted probability ( %?) and infer damages in three components. The Koppigen pier is used 362 forinducingthedamagebyloweringitinseveralincrementsstartingwith20mm,40mm,80mm, 363 and moving to 95 mm at the last stage. Twelve diﬀerent sensors are used to identify the location 364 of damage, namely, 4 sensors (411, 421, 431, 441) on the undamaged pier (UDP), 4 sensors (216, 365 221, 226, 231) on the bridge deck (BD), and 4 sensors (511, 521, 531, 541) on the damaged pier 366 (DP), as shown in Fig. 9. 367 The%?is plotted against the sensor index and a dash-dotted average of %?is shown as a 368 representation of combined %?for the corresponding structural component, as shown in Fig. 10 369 for 20 mm, and 40 mm and in Fig. 11 for 80 mm, and 95 mm lowering of pier, respectively. 370 For example, Fig. 10 ¹012ºrepresents%?for undamaged pier (UDP), bridge deck (BD), and 371 damagedpier(DP)for20mmloweringofpiers. Similarly,Fig. 10 ¹34 5ºisfor40mmlowering 372 of piers, respectively. It can be observed that, unlike in pier settlement of 40 mm, the proposed 373 algorithm does not provide conclusive evidence of nominal damage of 20 mm. However, Fig. 11 374 ¹012ºand¹34 5ºshowsidentiﬁcationofdamagefor80mmand95mm,anditcanbeobserved 375 that the identiﬁcation is clearly achieved through the proposed threshold where the %?is highest 376 for DP followed by BD which is aﬀected by diﬀerential settlement of one of the piers. 377 Itmaybenotedthat,astheseverityincreases,thesignalsbecomemoredistinguishableand1D 378 CNN learns the classiﬁcation more eﬀectively. It can be observed from Fig. 12 that UDP shows 379 lowest predicted probability due to its similarity to the response of the undamaged pier, however, 380 both BD, and DP shows higher prediction accuracy. The reason for BD’s highest probability is 381 attributedtothesurfaceareaandlargeraﬀectofdiﬀerentialpiersettlementoftheentirebridge. In 382 15 Sony, September 22, 2021 summary,itcanbeconcludedthattheproposedmethodperformswellfordamagelevelsof40mm, 383 85 mm, and 95 mm, however, does not perform well for a very low level of damage, as shown in 384 case of pier settlement of 20 mm. Finally, Table 7 shows the confusion matrix for the multi-class 385 pier settlement damage classiﬁcation problem with ﬁve classes, starting from no damage (class-0) 386 to 95 mm damage (class 5). 387 Forruptureoftendons,themostaﬀectedareawouldbethebridgedeckandthedamageinduced 388 duetoruptureoftendonswillcreateanon-localizedanddistributeddamagethroughoutthebridge 389 deck in comparison to the damage in the bridge piers. The damage identiﬁcation per sensors is 390 avoided due to non-conclusive inference and a comparison between structural components of the 391 bridge is provided in Fig. 13. It should be noted that rupture of tendons aﬀects bridge deck and it 392 is shown in Fig. 13, however, the proposed algorithm could not clearly show the eﬀect of rupture 393 of two and four tendons, while the rupture of six tendons proves to be the worse damage level 394 scenario. 395 CONCLUSIONS 396 Inthispaper,awindowed-1DCNNisemployedformulti-classdamagedetectionusinglimited 397 datasets. First, the limited dataset is augmented using windowing of the vibration data and the 398 prediction accuracy is improved by a novel majority voting approach on windowed classes. It is 399 observed that due to non-localization of sensors for data acquisition, damage identiﬁcation for a 400 minor level of damage (say, 20 mm of pier settlement) is a challenge to predict. However, the 401 overall accuracy signiﬁcantly improves with the increase in the severity of the damage (i.e., a 402 pier settlement of 40-95 mm). The proposed algorithm is analyzed with a sensitivity analysis on 403 window-size as the external parameter to the model as well as a parametric study to evaluate its 404 sensitivity with random initialization of weights. The improvement in the accuracy is illustrated 405 through a comparison between a single series dataset and windowed-voted time-series for ROC 406 andprecision-recallAUC.Inthispaper,itisdemonstratedthatasimple1DCNNarchitecturewith 407 only one hidden layer is capable of classifying the time-series of vibration data into multi-class 408 withhighaccuracy. Therearestillafewlimitationsoftheproposedalgorithm,whicharereserved 409 16 Sony, September 22, 2021 to be addressed in the future research. 410 •Theobtaineddamageclassiﬁcationprovidedsuperiorresults;however,damagelocalization 411 at the sensor level was not achieved. It may be noted that damage is identiﬁed with the 412 exception of minor level of damage (e.g., 20 mm pier settlement). Future research is 413 reserved to improve the proposed method and accommodate the minor level of damage by 414 conductinganexperimentalsimulationofaseriesofprogressivedamagecaseswithawide 415 range of minor damage. 416 •Theproposed1DCNNmodelbasedonsequentialdataisindependentofanyfeatureselec- 417 tion process and oﬀers robust and accurate approaches to complex damage identiﬁcation. 418 However, like any other supervised technique, the 1D CNN also requires a signiﬁcant 419 amount of training data to classify and predict the damage, which can be considered as a 420 limitation. However, with the recent advancement of remote and autonomous sensors and 421 internet-of-things,long-termSHMtechnologieshaveshownsigniﬁcantpromisetomonitor 422 critical infrastructure in smart cities. It is anticipated that the long-term monitoring will 423 allowtheSHMresearchersandpractitionerstoacquirelow-costperiodicaldatawithmulti- 424 classhealthconditionsofthestructures,servingasthepotentialfuturetrainingdataforthe 425 deep learning techniques. 426 DATA AVAILABILITY STATEMENT 427 All of the data, models, or code that support the ﬁndings of this study are available from the 428 corresponding author upon reasonable request. 429 ACKNOWLEDGMENTS 430 TheauthorswouldliketoacknowledgefundingprovidedtotheﬁrstauthorthroughGovernment 431 of Ontario, Canada as Ontario Graduate Scholarship (OGS), and Queen Elizabeth II Graduate 432 ScholarshipinScienceandTechnology(QEGS-II)forPhDstudies. Theauthorsalsoacknowledge 433 NSERC for providing the ﬁnancial support to conduct this research through the third and fourth 434 author’s NSERC Discovery grant. 435 17 Sony, September 22, 2021