20 Complicated traffic scenarios, including random change of vehicles ’ speed and lane, as well as the simultaneous p resence of multiple 21 vehicles on bridge, are main obstacles that prevents bridge weigh -in-motion (BWIM) technique from reliable and accurate application. To 22 tackle the complicated traffic problems of BWIM, this paper develops a novel BWIM method which integ rates deep -learning -based 23 computer vision technique and bridge influence surface theory. In this study, bridge strains and traffic videos are recorded synchronously 24 as the data source of BWIM. The computer vision technique is employed to detect and track v ehicles and corresponding axles from traffic 25 videos so that spatio -temporal paths of vehicle loads on the bridge can be obtained. Then a novel method is proposed to identify the strain 26 influence surface (SIS) of the bridge structure based on the time -synch ronized strain signals and vehicle paths. After the SIS is identified, 27 the axle weight (AW) and gross vehicle weight (GVW) can be identified by integrating the SIS, time -synchronized bridge strain, and vehicle 28 paths. For illustration and verification, the proposed method is applied to identify AW and GVW in scale model experiments, in which the 29 vehicle -bridge system is designed with high fidelity, and various complicated traffic scenarios are simulated . Results confirm that the 30 proposed method contributes t o improve the existing BWIM technique with respect to complicated traffic scenarios . 31 32 KEYWORDS 33 bridge weigh -in-motion; bridge influence surface; complicated traffic problem; computer vision; deep learning 34 35 1 Introductio n 36 1.1 Motivation and related work 37 The acqu isition of traffic loads on bridge s is essential for determining the structural and maintenance requirements of bridges and road 38 pavements. Besides, traffic information, such as traffic flow, vehicle speed , and so on, is of interest in the planning of traf fic infrastructure, 39 economic statistics as well as enforcement surveys. As a technique invented to measure traffic information on bridges, the bridge weigh - 40 JIAN ET AL . 2 in-motion (BWIM) technique has received considerable attention for its flexibility, durability and u nbiased accuracy in comparison to other 41 weighing methods such as static weigh or pavement -based weigh -in-motion system [1, 2]. 42 During the past four decades, numerous methods for implementing BWIM have appeared, and most of them are built on the static 43 algorithm proposed by Moses [3]. The Moses’s algorithm uses the influence line theory, where vehicle weights can be identified from the 44 influence line value s at the location of axles and the bridge strain readings. In practice, the bridge strain can be easily mea sured by strain 45 gauges attached to longitudinal bridge members , while the influence line values at axles’ location are more difficult to get. In Moses’s 46 pioneering practice of BWIM [4], the vehicle axle spacing and velocity are measured by tape switches placed on pavement s so that the axle 47 location can be estimated by assuming the vehicle moves at a constant speed and invariant transverse location . Though field tests had 48 verified the feasibility of the approach , it is concluded that accurate weight identification cannot be realized in complicated traffic scenarios . 49 For instance, Thillainath and Hood [5] observed a variation in the strain readings of up to 30%, which may introduce a 30% error for 50 weighing with Moses’s algorithm , during a BWIM calibration practice in which the same vehicle travell ed at the same speed, but in diffe rent 51 transverse locations. Now that the strain signals are accurately recorded by strain sensors, the accuracy degradation can be attributed to 52 difficulties in accurately determining the influence line value at the location of each axle when vehicles move in a complicated behavior on 53 the bridge. 54 The accurate location information of axles is the prerequisite for obtaining the corresponding influence value. In order to identify the 55 location of vehicle axles, early BWIM practice made use of traffic sensors suc h as road tubes or pavement -embedded axle detectors [6]. 56 However, any usage of those sensors would diminish BWIM’s advantages over the traditional weigh -in-motion technique . Later new 57 strategies named ‘nothing -on-road’(NOR) or ‘free of axle detector’(FAD) are investigated to locate axles in as pects of signal post - 58 processing [7-9], shear strain method [10-15], and so forth [16, 17]. In addition, some studies [18-21] paid more attention to mitigating the 59 aforementioned complicated traffic problems for BWIM. Nonetheless, these studies still have difficulty in determining the contributions of 60 individual axles to the global bridge response in various complicated traffic scenarios because none of them is able t o continuously and 61 accurately identify the location of every axle at each moment during the vehicle passing the bridge. Therefore, the complicated traffic 62 problems are so far key issues preventing the BWIM technique from a reliable and massive application, and they remain to be thoroughly 63 resolved [1, 2]. 64 In recent years, the continuing progress in computer vision (CV) techniques shed new light on the complicated traffic problem of 65 BWIM research and practice. Previous studies [22-28] have shown that CV can not only recognize vehicle configurations including vehicle 66 type, nu mber , and spacing of axles in a cost -effective manner, but it can accurately identify the spatio -temporal path of every vehicle on the 67 bridge. This advantage is enhanced even further due to the surge of deep learning, which drastically improve s the efficie ncy and robustness 68 in computer visual object detection and recognition [29]. However, those studies do not combine computer vision with BWIM as the vehicle 69 tracks they obtain are not used to identify vehicle weights. Inspired by those studies , Xia and Jian et al. [30, 31] first took advantage of 70 deep-learning -based CV to continuously and accurately locate the vehicles on bridge s. Then the vehicles’ real-time location s are fused with 71 bridge strains to identify vehicle weights. BWIM results obtained from a field test were encouraging despite the presence s of two vehic les, 72 manifesting the feasibility of integrating cost -effective CV technique with BWIM . But the accuracy and efficiency of the computer vision 73 techniques used in those studies remain to be improved. Unacceptable identification errors of vehicle weights stil l occur occasionally when 74 the vehicles deviate from the traffic lane because the usage of the 1-D influence line is inherently unable to consider the transverse movement 75 of vehicles. Besides, the BWIM algorithm they use to weight vehicles only uses peak va lues of bridge strain to weigh vehicles, which can 76 lead to the loss of weighing accuracy. In addition, they cannot identify the axle weight of vehicles either. According to Quilligan et al. [18], 77 extending the 1 -D influence line to the 2-D influence surface is a logical move to handle that problem. Quilligan [32] also pointed out that 78 using an influence surface of the bridge requires that the transvers e and the longitudinal locations of the crossing vehicles be known , but 79 they did not manage to find a n automatic and efficient method to locate vehicles in real time . 80 1.2 Contribution of this work 81 For the purpose of addressing the remaining complicated traffic problems of the BWIM technique, this paper develops a new BWIM 82 JIAN ET AL . 3 method integrating bridge influence surface theory and deep -learning -based CV technique. The method consists of two major part s, namely, 83 detection of vehicles with CV and identificatio n of vehicle weights with strain influence surface (SIS) of bridge structures . The frame work 84 of the proposed method is illustrated in FIGURE 1. The visual se nsing section is designed to detect, track , and locate vehicles from the 85 traffic videos , and in the meantime the strain sensing section is functioning to measure bridge strains that are time -synchronized with video 86 streaming. The complicated traffic proble m of the existing BWIM technique can be effectively resolved with the proposed framework. This is 87 because the computer vision techniques can detect and locate all vehicles on the bridge in a reliable, cost -effective, and real -time manner, and 88 the bridge in fluence surface theory is thereby enabled to weigh vehicles in complicated traffic scenarios by utilizing both the spatial an d temporal 89 strain. 90 91 F IGURE 1 Complicated -traffic -problem oriented framework for BWIM application 92 The main contribution s and novelties of this paper lay in the following aspects: 1) A novel BWIM method integrating the newly 93 released deep -learning -based computer vision technique and influence surface theory is proposed. The usage of computer vision enables 94 the proposed method to accuracy locate vehicles in real time, so that the vehicle weights in complicated traffic scenarios that previous 95 BWIM studies cannot handle well can be reliably and accurately identified. 2) A new optimization -based method is proposed to identify 96 the strain influence surface of bridges in a model -free manner, and the method has advantages on robustness and applicability compared 97 with previous studies. 98 The rem ainder of this paper is organized as follows: In the section of visual sensing , the introduction of YOLO V4, a recently released 99 CV model based on deep learning, is given. Training strategies of the model and its detection results are discussed so that the coordinate 100 JIAN ET AL . 4 transformation can be introduced for the localization of vehicles and axles on the bridge. In the section of strain sensing , an optimization - 101 based method to construct the SIS of bridge structures is proposed first. Then, a method that exploits both spatial and temporal strain to 102 identify the gross weight of vehicles is put forward . At last, a series of experiments on a vehicle -bridge scale model with high fidelity are 103 carried out to illustrate and evaluate the proposed BWIM method in various traffic scenarios. 104 2 Visual sensing 105 2.1 Detect ing vehicles and axles with YOLO V4 106 Comput er vision (CV) is a technique that aims to use computers for understanding and automating tasks from digital images or videos 107 like a human visual system . In the field of computer vision, there are several fundamental visual recognition tasks: image classif ication, 108 object detection, and semantic interpretation . As previously mentioned, one key issue of the BWIM technique is the detection of vehicle s. 109 With the rapid development of hardware facilities and software methods , video surveillance has been ubiquitou sly deployed to monitor the 110 traffic condition of roads and bridges , which paves the way for employing the CV technique to automatically acquire vehicle information 111 such as vehicle position, axle number , and spacing, for BWIM purposes. However, traditional CV methods experience difficulties in getting 112 the desired vehicle information with satisfactory speed, stability , and accuracy , especially when there are complex background, geometric 113 distortion , and illumination variation in traffic videos [33]. 114 Fortunately, the deep learning methods shed new light on the usage of CV for BWIM problems. Over the last few years, deep learning 115 has emerged as a powerful technology, making tremendous prog ress in many fields, with CV being one of the most prominent cases. 116 Numerous scientific studies and engineering practices have validated that the deep-learning -based CV significantly outperform s the 117 traditional CV in a wide range of object detection and cl assification tasks. The ambition and concept behind deep-learning -based CV is to 118 mimic the visual co gnition behavior of human by establishing computational models of multiple processing layers to learn and represent 119 visual data with multiple levels of abst raction. Therefore, in contrast to hand -crafted descriptors used in traditional CV detectors, deep- 120 learning -based CV generate s feature representations from raw pixels to high -level semantic information, which is learned automatically 121 from the training imag e data. Furthermore, benefiting from the learning ability, deep -learning -based CV can obtain better detection results 122 in complex contexts and large dataset s [34]. 123 In this study, a n open -source deep-learning -based CV method , YOLO (You Only Look Once) V4, is adopted to detect and recognize 124 vehicles from traffic videos. The newly released YOLO V4 model can be trained by a single GPU conveniently and achieve real -time, 125 high-quality as well as convincing object detection results. The advantages of YOLO V4 come from its data enhancement tricks and 126 optimization of the neural network structure. The detailed theory of YOLO V4 is not the focus of this paper. M ore details can be found in 127 the literature [35]. The main steps of implementing YOLO V4 for BWIM purposes are illustrated in FIGURE 2 and summarised as 128 follows: 129 JIAN ET AL . 5 130 F IGURE 2 Flowchart of implementing the YOLO V4 networ k 131 (1) Formation of training data. The YOLO V4 finishes object detection tasks in an end -to-end way, which means only the objects 132 YOLO V4 learns from the training data will be detected . As a consequence , training data in diversity must be prepared at first so that the 133 YOLO V4 network can learn representative features of objects and detect them successfully. The application of BWIM is interested in 134 vehicle position, axle number , and spacing. Accordingly, in the images of traffic videos, rears of vehicles are manually labeled for vehicle 135 localization and wheels are annotated for axle detection . Both the labeling data and 200 images are then delivered to the YOLO V4 model 136 for the training process. 137 (2) Training and testing . The training process of YOLO V4 is inte nded to find a set of neural network weights which minimize the 138 error between the detection results and the expected results annotated in the training data . To initiate the training process, training parameters 139 are set at first, including network input siz e, batch size, learning rate, class number of objects, number of convolution kernels, and number 140 of iterations . Values of the training parameters in this study are listed in TABLE 1 below. After the parameters are s et, the training program 141 starts. In this study , the training process took some 4 hours with a single NVIDIA 1080TI GPU , and the training loss curve is plotted in 142 FIGURE 3. Upon the YOLO V4 network is trained well, vehicles and axles in the traffic video can be accurately and stably detected in 143 real-time, regardless of the complicated traffic scenarios . Representative detection results are shown in FIGURE 2. As can be seen, the 144 YOLO V4 outputs the detection results in the form of bounding boxes. The dimensions and pixel coordinates of such bounding boxes are 145 collected to locate the position of vehicles further . 146 TABLE 1 Training parameters Parameters Value Input size 416× 416× 3 Learning rate 0.001 Batch size 64 Classes of l abeled objects 2 (truck and wheel ) Training set / testing set 8/2 Iterations 4000 FIGURE 3 Training loss curve 2.2 Locating vehicles on the bridge 147 The detec tion results of YOLO V4 are in pixel coordinates format, which cannot be directly used for BWIM purposes because BWIM 148 JIAN ET AL . 6 require s the vehicle ’s positions on the bridge . Hence a coordinate transformation is needed to transfer the two-dimensional (2D) pixel 149 coordinates of detected vehicles and axles into three-dimensional (3D) coordinates in real space . Considering the disadvantages of the coordinate 150 transformation method used in previous studies [30], this paper introduces a more accurate and convenient one and modifies it in the context of 151 BWIM. Although computer vision scholars can be expected to be familiar with the transformation , this paper includes the brief description to 152 ensure the accessibility to readers who are not familiar with the computer vision. 153 In order to describe the coordinate transformation mathematically , three coordinate systems are established . They are image coordinate 154 system , camera co ordinate system (in this coordinate system, the location of the camera is the coordinate origin) , and world coordinate system. 155 The three coordinate systems are described through the pinhole perspective model , as illustrated in FIGURE 4 below. 156 157 FIGURE 4 Diagram of the coordinate transformation for BWIM purposes 158 Let a point be denoted as (), p u v= in the image and the corresponding point be denoted as () ,,w w w P X Y Z= in the world 159 coordinate system . Meantime, point P can be denoted as () ,,c c c P X Y Z= in the camera coordinate system. According to [36, 37], the 160 mathematical relation ship between point p and point P can be written as 161 11 12 13 14 21 22 23 24 31 32 33 34 111ww ww c wwXXu m m m mYYZ v m m m mZZm m m m                  ==                     M (1) where M is the perspective matrix, ijm is the eleme nt of M. 162 In the context of BWIM, n ote that the vehicles move on the road surface which is usually an approximate plane , so all points on the plane 163 share a common wZ . Therefore , it is reasonable to describe the vehicle positions wit h 2D coordinates on road surface , instead of 3D coordinates 164 in the space. In light of this planar assumption, a point on the road surface can be denoted as (),ww P X Y= , and equation (1) turns into 165 11 12 13 21 22 23 31 32 33 1 1 1ww c w wu m m m X X Z v m m m Y Y m m m             ==                   M (2) As can be seen from equation (2), the coordinate transformation can be conducted once upon the perspective matrix M is determined. To 166 determine the perspective matri x M, the direct linear transformation (DL T) method [38] is adopted in this study on account of its conciseness 167 and operability . Based on equation (1), the DLT method derives two equations 168 11 12 13 31 32 33 21 22 23 31 32 33w w w w w w w wX m Y m m uX m uY m um X m Y m m vX m vY m vm+ + − − = + + − − = (3) JIAN ET AL . 7 Given n points of which both the spatial coordinates (), ( 1,2, , )wi wiX Y i n= on the road surface and the pixel coordinates 169 (), ( 1,2, , )iiu v i n= in the image are known, there will be 2n linear equations . Matrix form of the 2 n equations can be expr essed as 170 11 1 1 1 1 1 1 12 1 33 1 1 1 1 1 1 13 1 33 21 21 1 1 23 33 1 1 31 33 321 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1w w w w w w w w wn wn wn wn n wn wn wn wn nm X Y u X u Y m u m X Y v X v Y m v m m m X Y u X u X m u m X Y v X v Y m v m m −−      −−     =  =        −− −−    Km 33m    =    U (4) Since equation (4) is linear, it is reasonable to suppose 331 m= so that equation (4) can be denoted as 171 =Km U (5) Equation (5) contains 8 unknowns that are the elements of vector m. In an effort to estimate m, the least square method can be used , 172 provided there ar e at least 4 points of which the coordinates, (),wi wiXY and (),iiuv , are known. The least square solution [39] of m is given 173 by 174 ()-1TT=m K K K U (6) The perspective matrix M is obtained after m is estimated . Next, according to equation (2), the coordinate, (),wwXY , of the vehicle on 175 the road surface can be transformed from the pixel coordinate (),uv , by the equation 176  111TT w w cX Y Z u v−=M (7) Even though cZ is unknown, the coordinate transformation ca n still be complete d under the constraint that the last element of the vector 177  1T wwXY is 1. In this way, position of vehicles on the bridge and axle spacing required by BWIM are identified with the CV technique . 178 The whole transformation does not require any prior knowledge about the camera, so it is rather practical and convenient. 179 3 Strain sensing 180 3.1 Preprocessing of bridge strain signals 181 As discussed earlier, t he BWIM technique is built on the influence line theory [40], which descri bes the relationship among static load, 182 load position , and static bridge responses. Put another way, the influence line theory is a static concept , so the raw strain data measured by strain 183 sensors are not applicable to the influence line theory because br idges are subject to various dynamic loads in operation. To render the influence 184 line theory applicable, it is necessary to preprocess the raw strain signals so that the static strain can be extracted. According to Jian et al. [30], 185 the raw bridge strain signals contain mainly two components, which are non -vehicle component and vehicle -induced component, and the lat ter 186 one can be further divided into veh icle-induced dynamic component and vehicle -induced static component. In order to extract the vehicle - 187 induced static strain from the raw strain signal for BWIM purpose s, the locally weighted regression (LOWESS) [41, 42] algorithm is employed 188 in this study. FIGURE 5 illustrates the process of strain preprocessing, and the details of deploying LOWESS can be found in [30]. The 189 extracted static strain is subsequently used for the identification of strain influence surface and vehicle weight. 190 JIAN ET AL . 8 191 FIGURE 5 Process of extracting vehicle -induced static strain from raw strain signal 192 3.2 Identifying strain influence surface 193 Previous studies [18] on BWIM have shown the inability of influence line theory to realize BWIM in complicated traffic scenarios , as the 194 influence line ignor es the tra nsverse location of the bridge . Therefore , the strain influence surface (SIS) is used in this study for its capacity to 195 reflect both the longitudinal and transverse location of vehicle loads on the bridge . In an effort to identify the SIS of bridge structu res, this study 196 proposes a method that constructs the SIS in two steps . First, the strain influence lines (SIL) at different transverse location s are identified. T hen 197 the SIS is constructed on the basis of multiple identified SILs. 198 The first step starts wi th the influence line depicted in F IGURE 6 below. 199 200 F I G U R E 6 Diagram of the influence line 201 According to [40], when there are vehicles with N axles moving on the bridge, the influence line theory states 202 ()()() ()1 1K k M k p Kpk p k k p kF L y tMt t F L y tEW EW = = = = =   (8) where ()pt is the vehicle -induced static strain at a discrete sampling point pt, ()pMt is the bending moment of the b ridge girder, E is the 203 modulus of elasticity of the bridge material, W is the cross -section modulus of the bridge girder, kF , k=1,2,3, …,K, is the kth static axle load at 204 the location () kpyt on the road surface, K is the number of axles on the bridge, () kpyt are the longitudinal coordinate along the bridge . 205 ()MLy and () Ly are the value of the bending moment and strain influence line at location y, respe ctively. 206 Equation (8) suggests that the SIL is a prerequisite to calculate the axle load . So the first step for implementing BWIM is to identify the 207 SIL by carrying out calibration tests where a vehicle with known a xle weight kF and location () kpyt , which can be identified by CV 208 technique, goes across the bridge . In this way ()pt , kF and () kpyt in equation (8) are all determined, but the ()kp L y t cannot 209 be readily identified yet. This is because in the single equation (8) the unknowns ()kp L y t outnumber the equation, leading to an 210 underdetermined problem. 211 JIAN ET AL . 9 To solve this problem, O ’brien, Karoumi and Quilligan [18, 32, 43] proposed a widely accepted method calculating the bridge influence 212 line from direct measurements by using many time scans . Their method, however, has three limitations. First, since there was no method to 213 directly locate vehicles in real time, they estimated the critical vehicle positions by assuming that the vehicle velocity is constant or the vehicle 214 is in the center of the lane. These assumptions are invalid when the vehicle c hanges its velocity or lane, which may thus make their method 215 inapplicable or lead to a significant error. Second , their method requires to solve a large set of simultaneous equations expressed in matrix form, 216 and the dimensions of the matrices will increa se substantially when the bridge s have longer spans or the vehicle used for identifying the influence 217 line contains a large number of axles, justifying a need for improved computational efficiency. Although improving computability in the BWIM 218 calibration p rocess is not that urgent since the calibration of IL only needs to be done once for the BWIM system, it is helpful to find a method 219 with better computability. Third, as Ieng [44] points out, this method is not robust enough to provide an accurate IL of the brid ge, especially 220 when the bridge strain signals are heavily polluted by noises or dynamic effects of vehicle load. 221 In order t o improve the previous methods for SIL identification , this paper proposes a new method that avoids not only the constant -speed 222 assum ption but also large -scale matrices and is robust to noises . FIGURE 7 illustrates the main steps of the proposed method , and elaborations 223 are given as follows. 224 225 F I G U R E 7 Diagram of the iden tification for the strain influence line 226 1) Estimating a n SIL. 227 Based on the theory of structures [40], it is easy to roughly estimate the SIL of a beam -like structure by cubic interpolation on a set of key 228 points (KP) , ,kp kpyL , selected from the strain curve in a calibration test . As shown in F IGURE 7 (a), (b) and (c ), when a vehicle goes 229 across a three -span beam like FIGURE 6 depicts, the SIL of a fix ed point on the chosen beam cross -section can be roughly estimated with. 230 () 1kp kp K k ky L F ==  (9) It should be noted that Lkp in equation (9) is not the final IL used for BWIM. The objective of equation (9) is to provide an init ial value for 231 the subsequent optimization problem, so it neglects the axle distances that will be considered later. In addition to that, al though equation (9) 232 includes an approximation that introduces errors, the errors will be mathematically minimized thr ough the optimization , which means equation 233 (9) does not affect the weighing accuracy. 234 2) Defining an error function. 235 JIAN ET AL . 10 Obviously, the rough estimation for an SIL in the previous step is far from accurate . This inaccuracy is demonstrated by F IGURE 7 (d) 236 where the strain calculated with equation (8) is compared with the strain measured in the calibration test, and a significant distinction can be 237 observed between those two curves. To quantify this distinction, in this study an error function is defined as 238 ()() ()2 * 11PK kp p k k p pkf L t F L y t  == = −   (10) where P is the total number of discrete sampling points, ()*Ly is the roughly estimated SIL obtained by i nterpolating a set of kpL . 239 3) Optimizing the estimated SIL. 240 After the error function is defined in equation (10), the identification of SIL turns into a multivariable optimization problem that a ims to 241 find a set of key points ,kp kpyL to minimize the defined error function , and the quasi newton method (QNM) [45] is adopted herein to solve the 242 optimization problem. The iterative QNM is often used to find the global minimum of a function that is twice -differentiable, as expressed in equation 243 (10). Procedures for deploying QNM are as follows: 244 i) Choosing and interpolating a set of starting 0 kpL to estimate a n initial ()*0Ly .The 0 kpL can be calculated using equation (9). 245 ii) Calculating search direction by approximating the Hessian matrix 1−H . Details of the approximation can be found in [46]. 246 iii) Calculating the change in kpL using the following equation 247 ()11jj j j kp kp kpL L L+−= − H (11) iv) Determining if method ha s converged using convergence criteria that is defined as 248 ()1thresholdj kpL+ (12) v) Repeat ing from step 2 if not converged. Outputting a set of optimized kpL if converged. 249 By interpolating the optimized kpL , the SIL is identified . FIGURE 7 (f) gives an example of identified SIL with the QNM, and the strain 250 calculated with the optimized SIL is compared with the strain measured in the calibratio n test in FIGURE 7 (e). Since these two curves match 251 very well, it can be concluded that the SIL is successfully identified . 252 As previously mentioned, this study uses the SIS instead of SIL to implement BWIM, so the next step is to construct the SIS from multiple 253 identified SILs of which the transverse locations are different. Provided a calibration vehicle is arranged to go across the bridge on the left, 254 middle , and right side, t he construction can be simply achieved by transverse cubic interpolation on three identified SILs located at the left, 255 middle and right side of the bridge , as shown in FIGURE 8. 256 257 F IGURE 8 Diagram of constructing the strain infl uence surface 258 JIAN ET AL . 11 3.3 Identifying vehicle weight 259 Once the SIS has been identified with calibration tests, the influence surface theory will be applicable so that the bridge strain is written as 260 () ()() 1,K p k k p k p kt F S x t y t  = = (13) where () kpxt and () kpyt are the transverse and longitudinal coordinate where the kth axle locates , respectively. (), S x y are the value 261 of the SIS at location ( x,y). It should be noted that in this paper, () kpxt and () kpyt can be accurately identified by the CV technique in 262 a real -time manner no matter how the vehicle moves, but previous studies can only estimate them by assuming that the vehicle moves 263 at a constant speed and a fixed transverse location . 264 Now that every variable is known except the static axle load F, equation (13) can be used to identify the vehicle weight. This single equation, 265 however, is difficult to solve as the unkn owns are more than one. It is thus necessary to transform this underdetermined problem into a 266 determined or overdetermined one , and the transformation can be realized from two dimensions: time and space. 267 Assuming there are Q strain sensors mounted on the b ridge girder, and P discrete strain data points are recorded when all axles move on 268 the bridge. Here, ‘time dimension ’ means that P sampling points of the qth strain sensor are used to form an overdetermined equation set, such 269 that 270 () ()() () ()() () ()()1 1 1 1 2 2 2 1 1, , ,K q k q k k k K q k q k k k K q P k q k P k P kt F S x t y t t F S x t y t t F S x t y t    = = ==    =      =     (14) where P is the number of strain ’s discrete sampling points when all axles are on the bridge. 271 Correspondingly, ‘space dimension ’ means that data of Q strain sensors at the pth sampling moment are used to form an overdetermined 272 equation set that denote s 273 () ()() () ()() () ()()11 1 22 1 1, , ,K p k k p k p k K p k k p k p k K Q p k Q k p k p kt F S x t y t t F S x t y t t F S x t y t    = = = =    =     =    (15) where Q is the number of strain sensors installed on the bridge. 274 Equation (14) and (15) can be rewrote in the matrix form as 275 () () ()()() ()() ()() ()() ()() ()() ()()()() ()()1 1 1 1 2 1 2 1 1 1 1 2 1 2 1 2 2 2 2 2 2 2 1 1 2 2 1, , , , , ,== , , ,q q q K K q q q q q K K q qPq P P q P P q K P K P PPS x t y t S x t y t S x t y t t t S x t y t S x t y t S x t y t t S x t y t S x t y t S x t y t                                                 ε 1 2 1q KKKF F F   =   SF (16) JIAN ET AL . 12 ()() () ()()()()() ()() ()()()() ()() ()()()() ()()1 1 1 1 2 2 1 1 2 2 1 1 1 2 2 2 1 1 2 2 1, , , , , ,== , , ,p p p p K p K p p p p p p p K p K p p Qp Q p p Q p p Q K p K p QS x t y t S x t y t S x t y t t t S x t y t S x t y t S x t y tt t S x t y t S x t y t S x t y t                                                  ε ()1 2 1p KK QKF Ft F    =  SF (17) Equation (16) and (17) can be furth er combined such that 276 =SFε (18) where the strain vector ε, the axle load vector F, and the influence surface matrix S are denoted as: 277 ()()()()()()()1 1 1 2 1 2 11=T P P Q Q PPQt t t t t t      ε (19)  12 1=T K KF F FF (20) ()() ()() ()() ()()()() ()() ()() ()() ()() ()()()() ()1 1 1 1 1 1 2 1 2 1 1 1 1 1 1 1 1 2 2 1 2 1 1 1 1 2 2 1 2 1 2 1 1 2 1 1 2 2 2 2, , , , , , , , , = , , ,KK P P P P K P K P KK P P P P K P KS x t y t S x t y t S x t y t S x t y t S x t y t S x t y t S x t y t S x t y t S x t y t S x t y t S x t y t S x t y                                                  S () ()() ()() ()() ()()()() ()()()1 1 1 1 2 1 2 1 1 1 1 1 2 2, , , , , ,P Q Q Q K K Q P P Q P P Q K P K PP Q Kt S x t y t S x t y t S x t y t S x t y t S x t y t S x t y t                                        (21) As a result, the axle loads F can be obtained by solving the linear matrix equation (18). Note that equation (18) is an overdetermined 278 equation set where the equations significantly outnumbe r the unknowns, a good identification accuracy can thus be expected when using this 279 equation to solve the axle loads F. The traditional least square method can be employed to solve equation (18), under the constrain t F>0. More 280 details about solving nonnegative linear least -squares problem can be found in [39]. 281 At last, the axle weight AW and gross vehicl e weight GVW can be calculated by 282 kkAW F g= (22) 1K k kGVW AW == (23) where g is the gravitational acceleration known generally as 9.8 m/s2. It is also worth mentioning that, in the In ternational Measurement System, 283 the weight is defined as force (unit: N) instead of mass (unit: kg). However, in this study the vehicle weight is normalized with the acceleration 284 of gravity, and in such case the measurement unit of the vehicle weight is ki logram force (unit: kg). Because this definition of vehicle weight is 285 a widely accepted convention in the field of BWIM. 286 4 Scale model experiment 287 4.1 Model description and instrumentation 288 JIAN ET AL . 13 To investigate the reliability and accuracy of the proposed BWIM method in practice, a scale vehicle -bridge model with high fidelity is 289 utilized. The 1/20-scale bridge model is made of perspex, with static and dynamic similar ity to a real concrete box girder bridge located in 290 Shanghai Urban Expressway, China . In order to measure the normal strain of the bridge for BWIM purpose s, three resistance -type strain sensors 291 are mounted at the bottom of the third mid -span cross -section . For illustration, the configuration and dimension of the bridge model are shown 292 in F IGURE 9. 293 FIGURE 9 Configuration, dimension and instrumentation of the bridge model FIGURE 10 Picture of the truck model In terms of vehicles, remote -control truck models are u sed in the experiments to simulate vehicle loads . As shown in FIGURE 10, the 294 appearance of the three -axle truck model is rather similar to a real dump truck so that the feature -based CV method is applicable. The veh icle is 295 able to forward , back and turn by means of the remote control, at a maximum speed of up to 30 cm/s. The gross weigh t of the truck is 2.64kg, 296 and there are two addition al 1.46 kg weights that can be added onto the vehicle to change the vehicle weigh t. The gross weight of the scale 297 bridge model is 170 kg, so the mass ratio between the vehicle and the bridge is approximately 2%~3%, which is reasonable and realistic. 298 FIGURE 11 shows the on -site situation of the e xperiments. As can be observed, near the scale model two cameras are deployed, among 299 which camera 1 is used to identify the axle configuration of vehicles and camera 2 is used to locate the vehicles. Another reason to use two 300 cameras is to solve the possib le obstruction problem between multiple vehicles , and this problem will be elaborated later . The pavement of the 301 bridge model is simulated by abrasive paper, on which pavement lines are painted to simulate various traffic scenarios. The sampling frequency 302 of the three strain sensors is 200 Hz, and the frame rate of the two cameras is 25 frames per second. It should be noticed that the cameras and 303 strain sensors are strictly time -synchronized in the experiments . Otherwise the data fusion between the strain a nd axle location will be 304 unavailable. 305 JIAN ET AL . 14 306 F IGURE 11 Site situation of the experiment 307 4.2 Results of locating vehicles 308 In the experiments, t asks of locating axles on the bridge are divided into two stages : i) when vehicles start entering the bridge, detecting 309 axles in the image of camera 1, as shown in F IGURE 11(b), and measuring the spacing between each axle. ii) when vehicles fully enter the 310 bridge, detecting the back of vehicles in the image of c amera 2, as shown in FIGURE 11(c), so as to locate every vehicle on the bridge. The 311 purpose of setting two identification stages is to improve the localization accuracy and reliability by avoiding possible obscurati on in images. 312 The theory behind the object detection and length measurement is introduced in the aforementioned visual sensing section. It is worth 313 mentioning that a total of 16 calibration points, of which the location s on the pavement are known, are used in the process of coordinate 314 transformation. Those calibration points are marked on the bridge model in adva nce and can be observed in images of both two cameras , as 315 shown in F IGURE 11(a), (b), and (c). 316 To evaluate the accuracy of locating vehicles, three calibration experiments are carried out where a single vehicle goes straight on the 317 bridge with its left wheels placed on the pavement line so that the paths of vehicles are known as reference s. F IGURE 12 illustrates t he three 318 calibration experiments and the identified vehicle paths , and the localization accuracy can be evaluated by comparing the identified path with 319 the pavement line. As can be seen in the following figures, the pixel coordinate of the detection box ’s bottom midpoint is transferred into the 320 pavement plane coordinate system to locate the rear axle of the vehicle , and the remaining axles are subsequently located by adding the axles 321 spacing identified in the ima ge of camera 2 . The three paths ’ mean lateral distance s to the nearest pavement line are 6.45 cm, 26.21 cm, and 322 46.95 cm, and the variance of the three paths ’ X coordinates are 0.08 cm, 0.41 cm and 0.07 cm, respectively. Considering the half lateral length 323 of the vehicle is 6cm, it can be concluded the localization accuracy and reliability are rather satisfactory. 324 JIAN ET AL . 15 (a) calibration experiment 1 (b) calibration experiment 2 (c) calibration experiment 3 F IGURE 12 Diagram of cal ibration experiments 325 Also worth mentioning is the obstruction problem encountered in this study . For example, when two trucks travel in the same 326 lane, the rear truck may obstruct the view of the front truck in the camera , then using the CV to locate vehicl es may be affected. To 327 solve this problem, this study takes two measures. First, when the front truck is partly o bstructed by the rear truck , a YOLO model 328 that is specially trained for this case can still detect and track the front truck because of its ple asing robustness as shown in FIGURE 329 13 (a). Second, when the front vehicle is completely obstructed by the rear vehicle as shown in F IGURE 13 (b), even a human 330 cannot detect the front truck , let alone the YOLO model. In this case, it is necessary to deploy another camera with a different angle 331 where the front vehicle is not c ompletely obstructed so that the well -trained YOLO can detect the partly obstructed vehicle as shown 332 JIAN ET AL . 16 in FIGURE 13 (c). This is one of the reasons why two cameras are used in this study. In addition to the obstruction, illumination 333 change also affects the YOLO in terms of object detection. Similar to the obstruction p roblem, the YOLO model can still accurately 334 detect all the vehicles regardless of slight illumination change caused by the weather. When there is completely no illuminat ion at 335 night, however, an additional light source or an infrared camera are needed to enable the YOLO model to detect vehicles and axles 336 in the image. 337 (a) Partial obstruction (b) Complete obstruction (c) Another view of (b), part ial obstruction F IGURE 13 Examples of the obstruction between two vehicles 338 4.3 Resul ts of identifying strain influence surface 339 Now that the axle weight, axle spacing, and vehicle path in the three calibration experiments mentioned above have already been known, 340 those experiments are meantime used to identify the SISs belonging to three st rain sensors displayed in FIGURE 9, with the identification 341 method proposed in section 3.2. The identified SISs are visualized in FIGURE 14, which indicates the shape s of these SISs are in good 342 agreement with the influence surface theory despite the fact that they are identified in a model -free approach. Moreover, t he quantitative 343 identification accuracy of the SIS will be discussed in the section of identifying vehicle weights. 344 (a) SIS of the left strain sensor (b) SIS of the middle strain sensor (c) SIS of the right strain sensor F IGURE 14 Identification results of the strain influence surfaces belon ging to three strain sensors 345 4.4 Results of identifying vehicle weight 346 Since the objective of this study is to investigate the accuracy and reliability of the proposed BWIM method in various compl icated traffic 347 scenarios, six basic traffic patterns that can be combined into a variety of traffic scenarios are simulated in the experiments. The six basic patterns 348 are ‘single vehicle – go straight ’, ‘single vehicle – change lane ’, ‘double vehicles – go straight ’, ‘double vehicles – change lane ’, ‘four vehicles 349 – go straight ’, and ‘four vehicles – change lane ’, as illustrated in FIGURE 15. The ‘double vehicles ’ and ‘four vehicles ’ scenarios contain 350 two sub -patterns, which are ‘side-by-side’ pattern shown in F IGURE 15 (a) and ‘one-by-one’ pattern shown in F IGURE 13 (a). 351 JIAN ET AL . 17 (a) single vehicle – go straight (b) single vehicle – change la ne (c) double vehicles – go straight (d) double vehicles – change lane (e) four vehicles – go straight (f) four vehicles – change lane F IGURE 15 Diagram of basic traffic patterns 352 Considering that the weight of each vehicle is different in real ity, three different vehicle weights are set by adding additional weights onto 353 the vehicle model , of which the axle weights (AW) and gross vehicle weights (GVW) are shown in TABLE 2 below. The axle number can 354 be found in FIGURE 10, among which axle 1 and axle 2 are regarded as a group because they are too close to weigh separately. 355 TABLE 2 Weights of vehicles deployed in the experiments 356 No. Weight of axle 1 + axle 2 (kg) Weight of axle 3 ( kg) Gross vehicle weight (kg) 1 1.17 1.47 2.64 2 2.56 1.54 4.10 JIAN ET AL . 18 3 3.97 1.59 5.56 Subsequently, 74 experiments are conducted and a total of 1 16 vehicles are weighed . In each experiment, the path and velocity of vehicles 357 are neither the same nor constant so as to simulate the traffic as real as possible . Experiments in which vehicles stop on the bridge are also 358 conducted to simulate the traffic congestion. The bridge normal bending strains are measured by the three strain sensors with the sampling 359 frequen cy of 200 Hz , and the vehicle location s along with the axle spacing are identified with the CV technique from the videos recorded by 360 two cameras whose frame rates are 25 frames per second. These time-synchronized measurements are then delivered into the pr oposed BWIM 361 method to identify the vehicle weights . 362 The identification results of GVW s are visualized in FIGURE 16, in which each point corresponds to the weighing result of a vehicle. 363 In this figure, the further aw ay the point is from the baseline ±0%, the larger the weighing error is. Four extra reference lines that represent ±5% 364 and ±10% relative error are also plotted in the figure to further evaluate the identification accuracy and reliability. Among the 100 ide ntified 365 GVWs, relative errors of 75 GVW s are less than ±5%, and relative errors of 103 GVW s are less than ±10%. 366 (a) single vehicle – go straight (b) single vehicle – change la ne (c) double vehicles – go straight (d) double vehicles – change lane JIAN ET AL . 19 (e) four vehicles – go straight (f) four vehicles – change lane F IGURE 16 Results of identifying gross vehicle weight 367 For the purpose of quantitative investigation on the identification accuracy and reliability, relative w eighing errors of the identified GVWs 368 are calculated by 369 identified true trueGVW GVW100%GVWerror−= (24) Statistics of the relative weighing errors in the four traffic scenarios are listed in TABLE 3 below. The statistics demonstrate a pleasing 370 accuracy and reliability of the BWIM results regardless of the presence of various complicated traffic scenarios. Besides, an accuracy 371 degradation can be found when the traffic scenario gets more complicated, confirming t he aforementioned limitation of the BWIM technique 372 when it encounters complicate traffic patterns. Apart from those conclusion s about BWIM, the accuracy of weighing vehicles also verifies the 373 successful identification of the bridge structure ’s strain influ ence surface. 374 TABLE 3 Statistics of the relative GVW errors 375 Scenario Number of vehicles Mean (%) Standard Deviation (%) Maximum Absolute (%) single vehicle – go straight 36 -1.63 2.68 8.17 single vehicle – change la ne 10 -2.95 3.56 9.07 single vehicle 46 -1.89 2.88 9.07 double vehicles – go straight 36 -1.91 5.44 13.82 double vehicles – change lane 18 -2.57 6.96 13.06 double vehicles 54 -2.13 5.93 13.82 four vehicles – go straight 8 +4.77 15.00 39.77 four vehicles – change l ane 8 +4.32 29.20 69.32 four vehicles 16 +4.55 22.43 69.32 overall 116 -1.23 9.52 69.32 Similar to FIGURE 16, the identification results of AWs are visualized in FIGURE 17, in which each point corresponds to the 376 weighing result of a single axle or a tandem axle . Among the 232 identified AWs (116 single axles and 1 16 tandem axles) , relative errors of 57 377 AWs are less than ±5%, and relative errors of 108 AWs are less than ±10% . 378 JIAN ET AL . 20 (a) single vehicle – go straight (b) single vehicle – change la ne (c) double vehicles – go straight (d) double vehicles – change lane (e) four vehicles – go straight (f) four vehicles – change lane F IGURE 17 Results of identifying axle weight 379 For the purpose of quantitative investigation on the identification accuracy and reliability, relative weighing errors of the identified AWs 380 are calculated by 381 JIAN ET AL . 21 identified true trueAW AW100%AWerror−= (25) Statistics of the relative weighing errors in the four traffic scenarios are listed in TABLE 4 below. Unsurprisingly, just as the existing 382 BWIM methods behave [47], the identification accuracy and reliability of AWs are signific antly inferior to those of GVWs. Also can be 383 observed is the phenomenon that the more complicated the traffic scenarios are, the larger the errors are , especially when there are more vehicles 384 on the bridge . 385 TABLE 4 Statistics of the relative AW errors 386 Scenario Numb er of axles Mean (%) Standard Deviation (%) Maximum Absolute (%) single vehicle – go straight 72 -2.23 12.51 37.46 single vehicle – change lane 20 -1.00 16.50 43.86 single vehicle 92 -1.96 13.39 43.86 double vehicles – go straight 72 -2.23 19.93 57.27 double vehicles – change lane 36 0.42 26.00 64.43 double vehicles 108 -1.34 22.05 64.43 four vehicles – go straight 16 +1.08 31.87 65.86 four vehicles – change lane 16 +5.87 45.38 119.56 four vehicles 32 +3.48 38.65 119.56 overall 232 -0.93 22.35 119.56 4.5 Error analysis 387 An error analysis is meaningful for further improvement of the identification performance. For the proposed method, there are three main 388 sources of errors . 389 1) Errors in the identification of the SIS. As stated previously, the accuracy of BWIM heavily depend s on the accuracy of SIS identification . 390 Though the proposed method can identify the SILs of bridges with high accuracy, errors are inevitable when transversely interpolating multiple 391 SILs to construct the SIS. That explains why the e rrors of vehicle -change -lane cases are larger than those of vehicle -go-straight cases. However, 392 it is difficult to use the optimization -based method to reduce the error, because the calibration vehicle cannot move on the bridge transversely. 393 Moreover, the transverse width of vehicles is ignored in the process of identifying SILs , which may also lead to errors. For example, the actual 394 trucks to be weighed will likely have different width from the IL calibration truck , and this variation of vehicle width will result in errors . 395 Nonetheless, errors caused by the variation of vehicle width are not supposed to be significant, because the variation range of vehicle width, 396 which is usually from 2.0 m to 2.4 m, is rather tight when compared with the width of the brid ge. Anyhow, it is worthy to address the errors 397 induced by the SIS in the future study. 398 2) Number of vehicles. Many studies on BWIM have shown that the accuracy of existing BWIM systems is strongly affected by the 399 number of vehicles present on the bridge du ring measurement. Because the more the vehicles are on the bridge, the more difficult it is to 400 accurately separate the individual contribution of each axle from the measured overall bridge responses, and the larger the w eighing errors will 401 be. Results of t his study validate this point once again because the identification accuracy of AWs is much lower than that of GVWs , and a 402 significant accuracy degradation can be observed as the number of vehicles increases . Nevertheless, the identified total weight of al l vehicles 403 on the bridge is still accurate since the mean identification errors are small even if there are multiple vehicles. 404 3) Time lag between the strain signals and the videos. The proposed method requires strictly time -synchronized strain signals and videos , 405 otherwise errors will occur due to the mismatching between the vehicle positions and bridge responses . In the experiments, however, inevitable 406 time lags are observed between the strain signals and the videos because the time computers spend on sav ing each frame of videos are longer 407 than that of strain signals. Although most of the time lags can be eliminated by post -processing, this study fails to avoid them completely . 408 At last, concerns may arise over the influence of vehicle velocity on the weigh ing accuracy, but the experimental results manifest that the 409 variation of vehicle velocity has little influence on the identification accuracy. This is because the vehicle velocity can a ffect the weighing process 410 by exciting the bridge structure, but the d ynamic responses of the bridge are eliminated with the LOWESS algorithm mention ed in Section 3.1 411 JIAN ET AL . 22 above. 412 5 Conclusion and future work 413 In this study, a novel BWIM method is pro posed with an extra focus on complicated traffic scenarios. The proposed method is able to 414 identify strain influence surfaces of the bridge structu re and vehicle weights by integra ting the bridge influence surface theory and computer 415 vision techniqu e. A scale vehicle -bridge model with high fidel ity is utiliz ed. E xtensiv e experimental studies are performed to investigate the 416 accuracy and reliability of the proposed method on weighing vehicles in various traffic patterns . The conclusions derived from this study are 417 now summarised as follows: 418 1) A total of 116 vehicles are weighed with the proposed BWIM method in 74 model experiments (40 single -vehicle cases ,30 double -419 vehicles cases , and 4 four -vehicle cases ) where a variety of traffic scenarios , including changing lane and speed , are simulated. Thos e 420 experiments yield statistically satisfactory weighing results . For the identification of gross vehicle weight s, the mean and standard deviation of 421 the overall relative errors are -1.23% and 9.52%, respectively . For the identification of axle weight s, the mean and standard deviation of the 422 overall relative errors are -0.93% and 22.35 %, respectively. These statistics verify the pleasing accuracy and reliability of the proposed method , 423 but an acceptable degradation of accuracy can still be observed as the tr affic scenarios get more and more complicated. 424 2) In addition to BWIM purpose s, the pr oposed method is also able to identify the strain influence surface of bridge structures in a model -425 free and practical way, and BWIM results have verified the identificat ion accuracy . The identified bridge influence surface may be further used 426 for bridge condition assessment or other structural identification tasks . Meanwhile, identifying the strain influence surface of bridge structures 427 in a model -based approach is worth further investigation as well, because a model -based strain influence surface may make results less sensitive 428 to errors and help to perform sensitivity analyses and to make considerations on the uniqueness of the solution. 429 Although the gross vehicle weight can be successfully identified with the proposed method, the accuracy and reliability of identifying 430 individual axle weight remain unsatisfactory. Investigation on improving the identification accuracy of axle weight s is left for future studies. 431 Though th is study has exhausted all the possible complicated scenarios of two trucks, f urther studies should also pay attention to traffic scenarios 432 that are very complicated. For example, it is supposed to be rather tough to identify vehicle weights in the scenari os where four vehicles move 433 on the bridge simultaneously with the possibility of changing lanes. 434 Declaration of competing interest 435 The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the 436 work reported in this paper. 437