# Collapsed Building Detection Using Multiple Object Tracking from Aerial Videos and Analysis of Effective Filming Techniques of Drones

Shono Fujita, Michinori Hatayama

► To cite this version:

Shono Fujita, Michinori Hatayama. Collapsed Building Detection Using Multiple Object Tracking from Aerial Videos and Analysis of Effective Filming Techniques of Drones. 7th International Conference on Information Technology in Disaster Risk Reduction (ITDRR), Oct 2022, Krisitansand, Norway. pp.118-135, 10.1007/978-3-031-34207-3_8. hal-04680542

HAL Id: hal-04680542

https://inria.hal.science/hal-04680542v1

Submitted on 28 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Collapsed Building Detection Using Multiple Object Tracking from Aerial Videos and Analysis of Effective Filming Techniques of Drones

Shono Fujita¹[0000-0002-9610-5039] and Michinori Hatayama²[0000-0002-5257-6373]

¹ Graduate School of Informatics, Kyoto University, 36-1, 606-8501, Yoshidahommachi, Sakyo-Ku, Kyoto-City, Kyoto, Japan
fujita.shono.32x@st.kyoto-u.ac.jp

² Disaster Prevention Research Institute, Kyoto University, 611-0011, Gokasho, Uji-City, Kyoto, Japan
hatayama@dimsis.dpri.kyoto-u.ac.jp

**Abstract.** Earthquake destroyed many buildings, especially wooden ones, in Japan. Collecting information regarding collapsed buildings during the emergency phase (i.e., 72 h after a disaster) is difficult but essential for rescue activities. This study developed an automatic model to detect collapsed buildings using multiple object tracking (MOT) from aerial videos. Roof damage and pancake collapse are destructions unique to traditional Japanese buildings. Previous studies that detected collapsed buildings using the features of debris or damage failed to discriminate between collapsed and held-up buildings when the buildings have the above Japanese feature. Therefore, this study used the deep learning MOT model to classify collapsed and held-up buildings regardless of debris appearance. The recall and precision of each track of collapsed buildings were 29.1% and 36.7%, respectively, based on cross-validation with the drone video of the 2016 Kumamoto Earthquake. Analysis between the recall and other factors indicated that the aspect ratio, speed, and appearance time of the buildings were significant features for the detection. In the relationship between recall and these factors, we deduce that the recall of track increases to 63.9% if the drone operator films aerial videos effectively. Moreover, this study analyzed effective drone filming and flying way to satisfy some conditions for detection. This result provides recommended filming guides to drone operators for future earthquakes.

**Keywords:** Multiple Object Detection Keyword, Collapsed Building, Aerial Video

## 1 Introduction

Previous earthquakes in Japan damaged many buildings, especially wooden buildings. The Great Hanshin-Awaji earthquake damaged about 640,000 buildings [1] and killed over 4,200 people from suffocation or crushing deaths at the hands of collapsed buildings [2]. The Great East Japan Earthquake damaged about 280,000 buildings by seis-
mic motion and tsunami [3]. The 2016 Kumamoto earthquake damaged or collapsed about 200,000 buildings [4]. Because building damage causes human injury, disaster response headquarters and fire brigades need the information on these damaged buildings as quickly as possible. However, the information on damaged or collapsed buildings was challenging to obtain during these earthquakes. In the Great East Japan Earthquake, 25% of fire departments in three affected prefectures could not receive the 119 calls due to communication network interruptions [5]. In the 2016 Kumamoto earthquake, fire brigades could not organize and analyze information of damage reported by each fire station because of a lack of staff [6]. In the 2016 Kumamoto and 2018 Northern Osaka earthquakes, there was an overflow of 119 calls that did not merit the dispatch of firefighters [7, 8, 9], which led to inefficient rescue operations. In the 2016 Kumamoto earthquake, because fire brigades could not obtain accurate information on damaged buildings and victims, they investigated all buildings in the area. However, this operation required abundant manpower, support staff from other local governments, and equipment [6]. We expect that disaster response headquarters and fire brigades cannot gather damage information for future large earthquake disasters that require a large number of human resources and equipment.

Therefore, disaster response headquarters and fire brigades have to actively collect information on damaged buildings. This study develops an automatic model to detect collapsed buildings during earthquakes from aerial videos, mainly drone videos. Drone videos can collect extensive information from a bird's eye view at a low cost in places that cannot be reached by people.

Currently, 52.9% (383/724) of fire departments in Japan have drones that can capture damage situations due to earthquakes, landslides, fire, or toxic gas accidents [10]. Moreover, disaster response organizations, such as local governments or fire brigades, have helicopters to capture disasters in aerial videos [11]. From this, the Japanese disaster response organization recommends using aerial movies, drones, and helicopters to capture damage situations. However, it is difficult for disaster response organization teams to find each damaged or collapsed building in real time from the many buildings shown in aerial videos. Therefore, this study uses a deep learning model to automatically detect collapsed buildings from aerial videos. It also analyzes effective filming and flying techniques for drones to detect collapsed buildings through deep learning.

# 2 Related Study

## 2.1 Detection of Damaged Buildings

Previous studies presented various models to detect damaged or collapsed buildings. Vetrivel et al. developed a damage building detection model through deep learning, 3D point cloud features, and multiple-kernel learning [12]. Cusicanqui et al. [13] and Yamazaki et al. [14] constructed 3D point clouds of buildings from aerial images and videos for effective structure assessments. Although this 3D point cloud analysis effectively assesses individual buildings, it is still difficult to detect many buildings in
the affected areas because of the large calculation volume and small area covered in
the videos.

Our previous study detected roof damage from aerial images through deep learning to ensure more rapid and efficient damage investigation of buildings in Japan [15, 16, 17]. Miura et al. [18] also estimated damaged buildings from the detection of roof damage and roof covered with blue sheets from aerial images through deep learning. Calantropio et al. [19] extracted building regions through deep learning segmentation and detected damaged buildings. Ortho aerial images obtained above can detect roof damage, which can help assess building body damage, in large areas. However, these images cannot detect collapsed buildings without roof damage, where rescuers or injured people are likely located because these images cannot discriminate between vertical displacements of collapsed buildings.

Yamazaki et al. made fragility curves of Japanese buildings based on damage survey data from the 2016 Kumamoto earthquake [20]. These curves represent building damage probability corresponding to the peak ground velocity of earthquakes. The National Research Institute for Earth Science and Disaster Resilience in Japan supplies the estimated number of damaged buildings on the website based on fragility curves during the disaster [21]. The fragility curves estimate the number of individual damaged buildings in 250 m mesh areas. Japanese local governments can use this system to capture a rough damage overview.

Xie et al. [22], Khajwal et al. [23], and Shishido et al [24] proposed a crowdsourcing system to collect information on damaged buildings from aerial images. Volunteer cooperation such as crowdsourcing is effective during restoration and reconstruction. However, it takes more time to complete these tasks in the emergency phase, which lasts 72 h during disasters.

This study uses aerial videos, especially drone videos, to automatically detect col-
lapsed buildings through deep learning. Aerial videos can detect individual collapsed
buildings with vertical displacement and without roof damage from an oblique angle.
Moreover, aerial videos are effective during the emergency phase because of minimal
calculation volume and filming labor.

## 2.2 Detection from Aerial Videos

Qi et al. detected collapsed buildings through image processing from aerial videos in areas affected by the earthquake [25]. They detected debris of collapsed buildings from the disorder of texture in images using the HOG feature, a histogram of gradient direction of luminance. Pi et al. detected flood areas, undamaged roofs, damaged roofs, cars, debris, and vegetation by object detection from aerial videos in areas affected by hurricanes [26]. Object detection is an image recognition task that classifies and localizes object place by bbox, which is a circumscribed quadrangle of an object. Zhu et al. detected damaged buildings using the original model (MSNet) with object detection from aerial videos in areas affected by hurricanes [27]. This MSNet used building bbox and segmentation mask of damage regions, such as debris and roof. Japanese traditional buildings often suffer damaged roofs, especially tile roofs during earthquakes or typhoons. Prior to the amendment in the building standard in 1981,
tile roofs of buildings did not have to be fixed to the roof base. Currently, the number of tile roofs that sustain damage in disasters is decreasing because of revised strict building standards, the use of light raw materials, and a decrease in the number of people using tile roofs. Hence, many old buildings, or buildings without sufficient countermeasures, may still be affected in the future. These studies [25, 26, 27] detected collapsed buildings mainly using debris and damaged parts in videos. Even if buildings did not collapse, these detection systems that focus on debris overly react to roof damage in Japan (Fig. 1). Because fire brigades aim to locate collapsed buildings, where rescuers or injured people are likely located, information on a large number of buildings with light damage limits their rescue activity. Additionally, Japanese buildings, especially wooden ones, may cause pancake collapse, indicating that the collapsed story is crushed flat during earthquakes [28, 29, 30]. In this collapse pattern, the appearance of the building body, such as the wall and roof, may be intact, except for the crushed story. Because many of the crushed story cannot be seen well, these buildings cannot be detected based only on debris or damage regions. Considering the above feature of Japanese buildings, this study classifies collapsed buildings using object detection of deep learning regardless of debris appearance. In other words, the unique point of this study is to focus on not debris or damaged buildings but collapsed buildings.

Moreover, this study uses multiple object tracking (MOT), which is a type of technique applied to object detection. MOT detects an object by drawing a bbox in each frame and associating the same object among different frames in the video to follow the object tracking. Thus, disaster response organizations can grasp the number of collapsed buildings and confirm individual detected buildings after this tracking.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Image visualizing MSNet [27] applied to damaged Japanese buildings in Mashiki city</figcaption></figure>

## 3 System Use Flow

Fig. 2 shows the usage flow of the proposed system. First, MOT detects collapsed buildings from an aerial video filmed by drones or helicopters. Then, the detected collapsed buildings are visualized in the video. Finally, the headquarter staff, such as fire brigades and local governments, watch this video to grasp individual collapsed buildings and rough damage overview in real time. Therefore, object detection is expected to be an effective interface for users to visualize collapsed buildings.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Usage flow of the proposed system</figcaption></figure>

# 4 Dataset

## 4.1 Video Dataset

To develop the detection model, this study uses aerial videos and images filmed by drones in Mashiki city of Kumamoto prefecture after the 2016 Kumamoto earthquake. The 2016 Kumamoto earthquake occurred on April 14 and 16, 2016; the maximum seismic intensity was 7, and the moment magnitude scale was 7.0. The number of deaths reported was 273, and 198,258 buildings were damaged [4]. Mashiki city had large damage to buildings, mainly wooden buildings. The dataset is 6 aerial videos and 659 aerial images of one drone flight (hereinafter referred to as ODF). The durations of the six aerial videos are 2 s (51 frames), 2 s (56 frames), 6 min and 56 s (12,472 frames), 7 min and 37 s (13,683 frames), 7 mins and 40 s (13,776 frames), and 7 min and 52 s (14,138 frames). The image sizes of these videos is 3,840 × 2,160, and the image size of ODF is 4,608 × 3456. The frame rate of these videos is 30 fps, and the average altitude of the drone was about 60 ~ 75 m.

## 4.2 Annotation Dataset

<figure><img src="image_5.png" /><figcaption>Fig. 3. Histogram of the annotation data by bbox areas (left: video data, right: ODF image data)</figcaption></figure>

Bbox of collapsed buildings for MOT was annotated using the annotation tool “Computer Vision Annotation Tools.” We annotated one image out of every ten frames and compensated remain nine frames by linear interpolation. The following are the annotation rules:
* Target objects are buildings that collapses, especially pancake collapse, and those with dropped roofs, where rescuers or injured people are likely located.

* We exclude buildings whose collapse and dropped roofs cannot be recognized, such as those distant and unclear buildings.

* We judge collapse and dropped roofs from only the target frame. We do not use the information of the same building in other frames. Annotation can change in different frames for the same building.

* Bbox is drawn for one continuous building.

* Targets include warehouses.

* The classification class is only collapse.

Fig. 3 shows the histogram of these annotation data by bbox areas (the number of pixel). This study calls the annotated data an “answer” in contrast to estimation data. Fig. 4 shows a captured image of the annotation tool with the video of Mashiki city.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Captured image of the annotation tool with the Mashiki city video</figcaption></figure>

# 5 Training MOT Model

## 5.1 Dataset for Cross-Validation

Each frame image of these 6 videos is divided into eight classes based on four regions (Fig. 5) to make training, validation, and test data for cross-validation. The eight classes are “only A,” “only B,” “only C,” and “only D” (i.e., frames showing only buildings in A, B, C, and D regions) and “A and B,” “B and C,” “C and D,” and “D and A” (i.e., frames showing buildings in both regions A and B, B and C, C and D, and D and A). The images of ODF are also divided into these eight classes. Table 1 shows the cross-validation dataset. We use these data classifications to avoid using the same buildings among training, validation, and test data. Training data update parameters of the model and validation data decide training epoch to prevent overfitting, and test data evaluate the model accuracy after training. Using the data in Table 1, the validation and test data can be accurately evaluated. Test data use only video data to consid-
er the practical usage of this model. “Annotation” in Table1 means each bbox of an object.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Region classification for cross-validation</figcaption></figure>

<table><caption>Table 1. Breakdown of data in cross-validation</caption><thead><tr><th></th><th>Training data</th><th>Validation data</th><th>Test data</th></tr></thead><tbody><tr><td>1st</td><td>Video and ODF of “only C,” “C and D,” “only D”<br>16,004 images, 102,618 annotations, 435 tracks</td><td>Video and ODF of “only B”<br>9,746 images, 12,983 annotations, 104 tracks</td><td>Video of “only A”<br>3,893 images, 14,287 annotations, 71 tracks</td></tr><tr><td>2nd</td><td>Video and ODF of “only D,” “D and A,” “only A”<br>10,428 images, 100,704 annotations, 467 tracks</td><td>Video and ODF of “only C”<br>6,499 images, 36,459 annotations, 203 tracks</td><td>Video of “only B”<br>9,619 images, 12,771 annotations, 52 tracks</td></tr><tr><td>3rd</td><td>Video and ODF of “only A,” “A and B,” “only B”<br>19,185 images, 41,379 annotations, 366 tracks</td><td>Video and ODF of “only D”<br>1,921 images, 24,696 annotations, 137 tracks</td><td>Video of “only C”<br>6,436 images, 35,674 annotations, 88 tracks</td></tr><tr><td>4th</td><td>Video and ODF of “only B,” “B and C,” “only C”<br>17,689 images, 51,650 annotations, 342 tracks</td><td>Video and ODF of “only A”<br>4,014 images, 14,838 annotations, 166 tracks</td><td>Video of “only D”<br>1,921 images, 24,696 annotations, 137 tracks</td></tr></tbody></table>

## 5.2 MOT Model

This study uses ByteTrack [31] as the MOT model, which has high tracking accuracy and speed. ByteTrack uses a detected object with low confidence level, which other MOT models normally discard, when the detected object overlaps a large region with the object predicted by using the Kalman filter from a former track. The model can detect objects with occlusion, which implies hidden in front of other objects, or decrease ID switch, which indicates switching object identity to other objects.
## 5.3 Training Experiment

This study trained Bytetrack by the above dataset in Table 1 for cross-validation four times. The backbone for object detection is YOLOX-m [32], and fine-tuning is executed using a pre-trained backbone model by the COCO dataset. The input image size was 800×1440, batch size was 1, basic learning rate per image was 0.0001/64, max training epoch was 100, and other hyper parameters were the same as the default value in the GitHub code [33]. This experiment used GeForce RTX 2080 SUPER. The training times of the first to fourth train were approximately 36 h, 26 h and a half, 38 h, and 38 h. The upper part of Fig. 6 shows the transition of training data loss. The lower-left- and lower-right-hand sides of the figure show the validated data transition of average precision (AP50-95) and average recall (AR50-95) of the COCO dataset version, which implies an average from 50% to 95% of the IoU threshold. In the test MOT model, we used the trained model with high AP and AR, as shown in Fig. 6. Selected models in the first to fourth train were trained models of 5, 4, 3, and 2 epochs.

<figure><img src="image_8.png" /><figcaption>**Fig. 6.** Upper: training loss transition, lower left: validation AP50-95 transition, lower right: validation AR50-95 transition</figcaption></figure>

# 6 Evaluation of the MOT Model

## 6.1 Result of the Test Data by an Annotation

Table 2 shows the cross-validation result by an object, which implies each bbox in track. These experiments set 0.4 to the threshold of detection confidence, which was lower than the default because preventing overlook of collapsed buildings (FN data) is important for rescue activity. In Table 2, TP is the correct detection to answer data, FP is an incorrect estimation, FN is overlooked answer data, AP50 is the average precision of the COCO dataset version, IDF1 indicates MOT accuracy, and inference
time is measured by one image. This experiment set 0.5 as the threshold IoU of correct detection. The IoU has an overlapped ratio between true bbox and estimated bbox. From this result, because recall was lower than precision, the number of detections of the model tended to be small. Moreover, because the average inference time was 25.0 ms (i.e., an inference speed of 40.0 fps), the MOT model can run with aerial videos of 30 fps in real time.

<table><caption>**Table 2.** Results of the test data by annotation. TP: correct detections of answer data, FP: incorrect estimations, FN: overlooked answer data, AP50: average precision on the COCO dataset, IDF1: an indication of MOT accuracy</caption><thead><tr><th></th><th>TP ↑</th><th>FP ↓</th><th>FN ↓</th><th>Recall ↑</th><th>Precision ↑</th><th>AP50 ↑</th><th>IDF1 ↑</th><th>Inference time (ms) ↓</th></tr></thead><tbody><tr><td>1st</td><td>2080</td><td>2277</td><td>12207</td><td>0.146</td><td>0.477</td><td>0.119</td><td>0.202</td><td>23.0</td></tr><tr><td>2nd</td><td>3047</td><td>7921</td><td>9724</td><td>0.239</td><td>0.278</td><td>0.149</td><td>0.255</td><td>22.4</td></tr><tr><td>3rd</td><td>3197</td><td>5787</td><td>32477</td><td>0.090</td><td>0.356</td><td>0.178</td><td>0.136</td><td>23.7</td></tr><tr><td>4th</td><td>3278</td><td>5192</td><td>21418</td><td>0.133</td><td>0.387</td><td>0.186</td><td>0.193</td><td>30.8</td></tr><tr><td>Average</td><td></td><td></td><td></td><td>0.152</td><td>0.375</td><td>0.158</td><td>0.197</td><td>25.0</td></tr></tbody></table>

## 6.2 Result of Test Data by Track

<table><caption>**Table 3.** Results of the test data by track. MT: an answer track with over 80% of objects detected correctly, PT: an answer track with over 20% and less than 80% of objects detected correctly, OT: an answer track with even one object detected correctly in a track that does not fall into the MT or PT categories, FN: an answer track with no objects detected correctly. CE: an estimation track with even one object estimated correctly, FP: an estimation track with no objects estimated correctly. IDs: the number of times track identity changed in a single track.</caption><thead><tr><th></th><th>MT</th><th>PT</th><th>OT</th><th>FN ↓</th><th>CE ↑</th><th>FP ↓</th><th>IDs ↓</th><th>Recall ↑</th><th>Precision ↑</th></tr></thead><tbody><tr><td>1st</td><td>4</td><td>7</td><td>4</td><td>56</td><td>18</td><td>39</td><td>3</td><td>0.211</td><td>0.316</td></tr><tr><td>2nd</td><td>2</td><td>9</td><td>8</td><td>33</td><td>22</td><td>40</td><td>9</td><td>0.365</td><td>0.355</td></tr><tr><td>3rd</td><td>4</td><td>9</td><td>13</td><td>62</td><td>30</td><td>53</td><td>8</td><td>0.295</td><td>0.361</td></tr><tr><td>4th</td><td>6</td><td>18</td><td>16</td><td>97</td><td>43</td><td>53</td><td>5</td><td>0.292</td><td>0.448</td></tr><tr><td>Average</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>0.291</td><td>0.367</td></tr></tbody></table>

Table 3 shows the result of cross-validation by track. In the table, MT, PT, OT, and FN are the numbers for the answer data. Specifically, MT is an answer track that has over 80% of objects detected correctly in the track, PT is an answer track that has over 20% below 80% of objects detected correctly in the track, OT is an answer track that has even one object detected correctly in the track but MT and PT, and FN is an answer track that has no objects detected correctly in the track. Alternatively, CE and FP are the numbers for estimation data. Specifically, CE is an estimation track that has even one object estimated correctly in the track, and FP is an estimation track with no objects estimated correctly in the track. Meanwhile, IDs are the number of times changing track identity in one track. Recall is the ratio of the sum of MT, PT, and OT to the total answer track, and precision is the ratio of CE to the total estimation track. If users consider the track with even one detected object by the model as a collapsed building, they can detect 29.1% of collapsed buildings with 63.3% of no collapsed building detected incorrectly.
## 6.3 Analysis of Accuracy and Other Factors

This section analyzes the relationship between recall of answer track in Table 3 and other factors. Because recall is calculated from denominator of the answer data, other factors such as bbox size and the appearance time of real collapsed data annotated in the experiment can also be used. However, precision is calculated from the denominator of the data of estimated collapsed buildings. Because other factors obtained from the estimation data are not real values, they cannot be used in the analysis. Hence, we focus on recall in this section. Moreover, recall may also be considered as a more important measure of performance in disaster scenarios because it enables us to focus on reducing overlooked collapsed buildings, which is critical for locating survivors in a timely manner. These other factors are the average bbox area (the number of the pixel) converted to 800×1440, bbox aspect ratio (width/height), displacement pixel of bbox center from the former frame (bbox speed), and appearance time (number of frames) of an object in each answer track. Fig. 7 shows the histogram of the answer track in each rank and recall of the answer track by each rank. If the “correct value” of the detected answer track was set to 1, the “correct value” of the not detected answer track was set to 0. Each correlation coefficient between “correct value” and area, aspect ratio, speed, and appearance time was -0.007, -0.358, -0.204, and 0.251, respectively. From this result and in Fig. 7, the smaller the aspect ratio, the slower speed and longer appearance time, and the higher the model recall. The speed and appearance time results also apply to humans and can be predicted easily. The result of the aspect ratio indicates that an object whose aspect ratio is large (i.e., the object watched from an angle close to the horizon) is difficult for the MOT model to detect. This result may indicate that being able to see collapsed story, mainly the first story, located in the lower part of the building, is important for collapse detection.

<figure><img src="image_9.png" /><figcaption>**Fig. 7.** Histogram and recall of answer track by other factors (upper left: area, upper right: aspect ratio, lower left: speed, lower right: appearance time)</figcaption></figure>
6.4 Qualitative Analysis

<figure><img src="image_10.png" /><figcaption>Fig. 8. Image visualizing bbox of answer (left) and estimation (right)</figcaption></figure>

Fig. 8 shows an image visualizing the bbox of answers and estimation. These estimation images indicate that the MOT model can detect collapsed buildings without large debris or damage, which is a specific feature of Japanese buildings. The image shows the number of estimated collapsed buildings in the upper-left part. This number enables the user to capture a rough damage overview. The color of the bbox corresponds to the detection confidence level to tell collapsed buildings with a high degree of probability of being correctly estimated to users.

Fig. 9 shows an example of the building with the following feature of FN and FP track data. In all 248 FN tracks in Table 3, which is an overlooked track, there were two features in common with some data. First, 60 (24.1%) buildings did not suffer
roof damage. This implies that the MOT model use information about roof damage in addition to the building body as food for thought to detect collapsed buildings. Second, 30 (12.1%) buildings' collapsed story (wall) was not seen because of the angle, other buildings, or obstacles. In the annotation task, these buildings were judged from the roof position to the ground or inclination of the buildings as collapsed. This result matches the fact that the small building aspect ratio is suitable for the detection of the MOT model discussed in Section 6.3.

In all 185 FP tracks in Table 3, which were incorrect estimations, there were three features in common with some data. First, 78 (42.2%) buildings had roof damage. This feature has the same tendency as the above first feature of FN. Second, 20 (10.8%) buildings had the incorrect bbox shape and size against the answer bbox, although they could point to collapsed buildings. However, users can understand collapsed buildings from these incorrect detections because they have only mistaken their shapes or sizes. Third, 19 (10.3%) buildings did not collapse judging from an angle at that time, but collapsed judging from another angle. This difference in annotation labels resulted from the third annotation rule discussed in Section 4.2. This result suggests that the MOT model can possibly extract features of collapsed buildings that humans cannot understand because these buildings actually collapsed.

<figure><img src="image_11.png" /><figcaption>**Fig. 9.** Example of FN and FP features (upper left: building without roof damage, upper right: building whose collapsed story cannot be seen, lower left: building with damage, lower center: building who's detected bbox is bad, lower right: building collapsed in fact)</figcaption></figure>

# 7 Effective Videography and Flight Techniques for Aerial Drones

## 7.1 Necessary Conditions for Video Capture and Flight Simulation

In Subsection 6.3, we determined that the aspect ratio, speed, and appearance time of buildings in videos are important features to identify damaged buildings. When we set tracks with an average aspect ratio of less than 2, a speed of less than 30 pixels per frame, and an appearance time of more than 200 frames (6.66 s) in this experiment,
we found that recall increased to 63.9% (53/83). This implies that 63.9% of collapsed buildings can be detected if drone operators adopt more optimal videography and flight techniques. From this result, we executed a simulation of video capture during a drone flight satisfying the following necessary conditions to determine a set of effective flight and videography techniques for drones.

(1) The aspect ratio of each bbox in the track should be less than 2 so that the average aspect ratio can be less than 2.

(2) The average speed of the buildings in the track should be less than 30 pixels per frame.

(3) The appearance time should be more than 200 frames.

(4) The area of the building bboxes ($h_b \times w_b$ of Fig. 10) in a 3,840 x 2,160 video should be more than 10,000 pixels.

(5) The length of building walls ($h_w$ in Fig. 11) in 3,840 x 2,160 video should be more than 20 pixels.

The necessary condition (4) and (5) were added to avoid the buildings shown in the video from being too small to detect building body and collapsed stories (walls), which is important to detect collapses as described under FN in the second feature of Section 6.4.

## 7.2 Altitude and Camera Angle

Fig. 10 shows videographic and flight techniques used in this simulation. The z-, y- and x-axes respectively indicate the vertical direction, the direction of flight, and the direction perpendicular to the y-z plane. The drone proceeded in a straight line, and the height of the drone $H$ and the camera angle $\theta_D$ to the z-axis in the y-z plane remained constant. This simulation assumed buildings to be rectangular solids, and that their front sides were orthogonal to the direction of flight of the drone. The side shape of the buildings in the x-y plane was a $W_b$ square with a height of $H_b$. The buildings were two stories and the first story was collapsed. Considering the average gross floor area 126.63 m² of Japanese detached houses [34], $W_b$ was set as $\sqrt{126.63/2}$ (m). From the minimum ceiling height of the Japanese Building Standard Law, $H_b$ was set as 2.1 (m). At a certain $H$ and $\theta_D$, the simulation calculated the trapezoid area in Fig. 10, where the building satisfied the necessary conditions (1), (4) and (5), and the rectangle area (ASA) where the building always fell within the area within which the conditions were satisfied even if the drone continued to fly in the same direction. This simulation determined the most effective $H$ and $\theta_D$ to maximize ASA.

In reference to the diagonal viewing angle of 84° and image size of 3840 x 2160 of DJI Mavic 3 drones [35], this simulation utilized a horizontal view angle $\theta_h = 47.6°$ and a vertical view angle $\theta_v = 76.2°$. In calculating ASA, this simulation utilized a building located in the left-right center of the video plane. Fig 11 left and right show the location relation from the perspective of the x- and z-axes, where $h_b$, $h_w$ and $w_b$ are the bbox height, the building wall height and the bbox width, respectively. In this simulation, we examined whether the necessary conditions (1), (4) and (5) were satisfied in all combinations among $\theta_D$ (integer from 0 to 90), $H$ (integer from $H_b$ to 150) and $l$ (an integer in range when the building appeared in the video plane). Then, we
calculated ASA in all of $\theta_D$ and $H$ from the maximum $l$, minimum $l$, and minimum $l_2$ satisfying the above conditions at a given $\theta_D$ and $H$. Fig 12 shows the result, and the maximum ASA (red cross in Fig. 12) was 19,739 m² at $\theta_D$ = 33°, $H$ = 106 m.

<figure><img src="image_12.png" /><figcaption>Fig. 10. Videography methods for drones in the simulation</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 11. Location relation viewed from the x-axis (left) and z-axis (right)</figcaption></figure>

## 7.3 Drone Speed and Area Covered

At the maximum ASA value, the depth of ASA region appearing in video plane was 570 pixels. When the building appeared in the ASA in video plane for 200 frames (necessary condition (3)), the building speed in the video was 2.85 pixels per frame, which satisfies the necessary condition (2). In this case, the drone can have maximum speed 57.78 km/h calculated from an ASA length 107 m. With a battery life of 46 minutes [35], a vehicle could film an area of 8.10 km² in a single flight based on an ASA width of 182 m. These effective flight and videography techniques with optimal $\theta_D$, $H$ and maximum speed can be used to survey an area of 8.10 km² area with 63.9 % recall in a single flight.
<figure><img src="image_14.png" /><figcaption>Fig. 12. (right) 3D graph of ASA with height of drone and camera angle</figcaption></figure>

8 Conclusion

This study developed an automatic MOT model to detect collapsed buildings in earthquakes from aerial videos and analyzed effective filming of drone aerial videos. To focus on collapse and address building damage features, such as roof damage and pancake collapse of Japan, this study used object detection, especially the MOT mod-
el, of deep learning, instead of debris feature detection. From the cross-validation
result using aerial videos of the 2016 Kumamoto earthquake, recall and precision of
the MOT model by track were 29.1% and 36.7%, respectively. The results of the
analysis between recall and other factors indicated that the aspect ratio, speed, and
appearance time of buildings were important features for the detection. However, the
number of detected collapsed buildings was often lower than the actual number. Qual-
itative analysis indicated that the MOT model could detect buildings with pancake
collapse without large debris and damage, specific to Japanese buildings. However,
the detection of the MOT model relied on information about roof damage. Finally,
this study indicated that track recall increases to 63.9% if drone operators effectively
film aerial videos. Then, we analyzed effective drone videography and flight tech-
niques to maintain this recall using the filming simulation. We determined the optimal
altitude and camera angle to be 106 m and 36°, respectively, and that a single flight
can film an area of some 8.10 km².

In the future, another MOT model should be analyzed for improved accuracy, along
with possible hyperparameter configurations and training processes. In addition to
improving accuracy, the allowed recall and precision should be analyzed through a
questionnaire or investigation. Calculation or simulation of more practical flight and
videography techniques for aerial drones is also crucial. In addition to disaster re-
sponse personnel, this system may also be effective to help drone operators determine
where to fly. An analysis of types of residential areas that are difficult for this model
to recognize would enable operators to select areas on which to focus. Moreover, the
practical implementation of this system must be considered, including methods of
sending videos and associated server architectures and communication frameworks.
Finally, the proposed model can also be combined with geospatial information to
produce maps to facilitate rescue activities.

Acknowledgements

This work is supported by JSPS KAKENHI Grant Number JP 22J15895.

References

1. Fire and Disaster Management Agency in Japan. Final report about the Great Hanshin-Awaji Earthquake (in Japanese). Available on https://www.fDMA.go.jp/disaster/info/assets/post1.pdf. Accessed 5 Jul 2022.

2. Hyogo Prefecture Medical Association. Actual state of affairs of human damage by the Great Hanshin-Awaji Earthquake (in Japanese). https://www.hyogo.med.or.jp/jmat-hyogo/day-after/siryo/. Accessed 5 Jul 2022.

3. Disaster response headquarters of Fukushima prefecture. No. 1778 damage report of the 2011 off the Pacific coast of Tohoku Earthquake (in Japanese). https://www.pref.fukushima.lg.jp/uploaded/life/620025_1725169_misc.pdf. Accessed 5 Jul 2022.

4. Disaster response headquarters of Kumamoto prefecture. No. 325 damage report of the 2016 Kumamoto Earthquake (in Japanese). https://www.pref.kumamoto.jp/uploaded/attachment/182677.pdf. Accessed 5 Jul 2022.

5. Fire and Disaster Management Agency in Japan. Report of review meeting about ideal way of effective initial activity of firefighter headquarter in large-scale disasters (in Japanese). https://www.fDMA.go.jp/singi_kento/kento/items/kento004_01_houkoku.pdf. Accessed 5 July 2022.

6. Kumamoto city fire department. Activity record journal of Kumamoto city fire department in the 2016 Kumamoto Earthquake (in Japanese). https://www.city.kumamoto.jp/common/UploadFileDsp.aspx?c_id=5&id=19060&sub_id=1&flid=134936. Accessed 5 Jul 2022.

7. Tv Asahi news site (in Japanese). https://news.tv-asahi.co.jp/news_society/articles/000074837.html. Accessed Jul 2022.

8. Murakami, H.: Study on Rescue and Emergency Activities by Kumamoto City Fire Department after the 2016 Kumamoto Earthquake - Comparison with the 2004 Niigata Chuetsu and the 2005 West off Fukuoka Prefecture Earthquakes - (in Japanese). Report of Tono Research Institute of Earthquake Science 41, 93-98 (2017).

9. The Sankei News (in Japanese). https://www.sankei.com/article/20200115-7M7FDUJP2RLLDJDFXYP23H4J3E/. Accessed 5 Jul 2022.

10. Fire and Disaster Management Agency in Japan. Notice about promotion of drone usage of the fire brigade in disaster response (in Japanese). https://www.fDMA.go.jp/laws/tutatsu/items/040331_drone.pdf. Accessed 5 Jul 2022.

11. Nazarov, E.: Emergency Response management in Japan, Final Research report. ASIAN Disaster Reduction Center, FY2011A Program, (2011)

12. Anand, V., Markus, G., Norman, K., Francesco, N., George, V.: Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high-resolution oblique aerial images, and multiple-kernel-learning. ISPRS Journal of Photogrammetry and Remote Sensing 140, 45–59 (2018).
13. Johnny, C., Norman, K., Francesco, N.: Usability of aerial video footage for 3-d scene re-construction and structural damage assessment. Natural Hazards and Earth System Sciences 18(6), 1583–1598 (2018).

14. Yamazaki, F., Kubo, K., Tanabe, R., Liu, W.: Damage assessment and 3d modeling by UAV flights after the 2016 Kumamoto, Japan earthquake. In: the IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 3182–3185. Fort Worth, TX, USA (2017).

15. Fujita, S., Hatayama, M.: Estimation method for roof-damaged buildings from aero-photo images during earthquakes using deep learning, Information Systems Frontiers 25(1), 351-363 (2021).

16. Fujita, S., Hatayama, M.: Automatic calculation of damage rate of roofs based on image segmentation. In: 6th IFIP WG 5.15 International Conference, ITDRR 2021, pp. 3-22, Morioka, Japan, (2022).

17. Fujita, S., Hatayama, M.: Rapid and accurate detection of building damage investigation using an automatic method to calculate roof damage rate. IDRiM Journal 12(1), 89–111 (2022).

18. Miura, H., Aridome, T., Matsuoka, M.: Deep learning-based identification of collapsed, non-collapsed and blue tarp-covered buildings from post-disaster aerial images. Remote Sensing 12(12), (2020).

19. Calantropio, A., Chiabrando, F., Codastefano, M., Bourke, E.: Deep learning for automatic building damage assessment: application in post-disaster scenarios using UAV data. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences V-1-2021, 113–120 (2021).

20. Yamazaki, F., Suto, T., Liu, W., Matsuoka, M., Horie, K., Kawabe, K., Torisawa, K., Inoguchi M.: Development of fragility curves of Japanese buildings based on the 2016 Kumamoto earthquake. In: the 2019 Pacific Conference on Earthquake Engineering, Auckland, New Zealand, (2019).

21. Kusaka, A., Nakamura, H., Fujiwara, H., Okano, H.: Bayesian updating of damaged building distribution in post-earthquake assessment. In: the 16th World Conference on Earthquake, 16WCEE 2017, Santiago Chile, (2017).

22. Xie, S., Duan, J., Liu, S., Dai, Q., Liu, W., Ma, Y., Guo, R., Ma, C.: Crowdsourcing rapid assessment of collapsed buildings early after the earthquake based on aerial remote sensing image: a case study of Yushu earthquake. Remote Sens. 8(9), (2016).

23. Khajwal, A., B., Noshadravan, A.: An uncertainty-aware framework for reliable disaster damage assessment via crowdsourcing. International Journal of Disaster Risk Reduction 55, (2021).

24. Shishido, H., Kobayashi, K., Kameda, Y., Kitahara, I.: Method to generate building damage maps by combining aerial image processing and crowdsourcing. Journal of Disaster Research 16(5), 827-839 (2021).

25. Qi, J., Song, D., Shang, H., Wang, N., Hua, C., Wu, C., Qi, X., Han, J.: Search and rescue rotary-wing UAV and its application to the Lushan Ms 7.0 earthquake. Journal of Field Robotics 33(3), 290-321 (2016).

26. Pi, Y., Nath, D., N., Behzadan, H., A.: Convolutional neural networks for object detection in aerial imagery for disaster response and recovery. Advanced Engineering Informatics 43, (2020).

27. Zhu, X., Liang, J., Hauptmann, A.: Msnet: A multilevel instance segmentation network for natural disaster damage assessment in aerial videos. In: the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2023-2032 (2021).

28. Scawthorn, C., Yanev, I., P.: Preliminary report 17 January 1995, Hyogo-ken Nambu, Japanese earthquake. Engineering Structures 17(3), 146-157 (1995.)

29. Okada, S., Takai, N.: Classifications of structural types and damage patterns of buildings for earthquake field investigation. In: 12th World Conference on Earthquake Engineering,
Auckland, New Zealand (2000).

30. Scawthorn, C.: Building aspects of the 2004 Niigata Ken Chuetsu, Japan, Earthquake. Earthquake Spectra 22(1), 75-88 (2006)

31. Zhang, Y., Sun, P., Jiang, Y., Yu, D., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: Multi-object tracking by associating every detection box. arXiv preprint arXiv:2110.06864 (2021)

32. Ge, Z., Liu, S., Wang, F., Li, Z., Sun, J.: Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, (2021)

33. GitHub - ifzhang/ByteTrack: [ECCV 2022] ByteTrack: Multi-Object Tracking by Associating Every Detection Box. https://github.com/ifzhang/ByteTrack. Accessed 7 Jul 2022.

34. Ministry of Internal Affairs and Communications: Result summary of basic tabulation about housing and households in housiPng and land survey (in Japanese). https://www.stat.go.jp/data/jyutaku/2018/pdf/kihon_gaiyou.pdf. Accessed 29 Jul 2022.

35. DJI: Mavic 3 specification (in Japanese). https://www.dji.com/jp/mavic-3/specs. Accessed 29 Jul 2022.