## 3 Performance Analysis of SEM-LEWS Based on the Simulation Results of 8-Year Histories  
### Study Framework  
The performance analyses were conducted based on the simulation results of SEM-LEWS operation over a period of eight years from 2009 to 2016. The rainfall input data during the eight-year period were sourced from the 17 KMA observatories that are distributed in and adjacent to the Busan area; the 51 city-owned rain gauges introduced in Section 2.2. were not feasible sources in this study as these gauges were installed and have been in operation since 2017. As shown in Figure 4, the entire area of Busan can be divided into 17 Thiessen polygon areas by black lines based on the locations of the distributed KMA observatories. In this study, each Thiessen area was considered as a unit region that exhibits independent and homogeneous meteorological behaviors (i.e., rainfall characteristics). Therefore, raster dataset layers of rainfall input parameters were generated in such a way that data values are equal within each Thiessen area. By using hourly-recorded rainfall data from the KMA observatories, early warning level maps in the form of a raster dataset were generated with an interval of 1 h for the rainy seasons (June to October) of the study period. On the other hand, historical landslide data with valid quality and sufficient quantity constitute an important basis for testing the performance of landslide early warning systems at a regional scale. Therefore, a GIS historical landslide database was constructed by determining time and location information using several effective methods. Detailed processes are described in detail in the Sections 3.2 and 4.1. The data were collected in two categories depending on its landslide type: slope failure or debris flow. This is important information to test the discriminative capabilities of the SEM-LEWS in terms of debris flows that tend to cause significant damage to surrounding communities. Moreover, the collected historical landslide data were reclassified and analyzed according to the infiltration characteristic zone in which they are located. As shown in Figure 4, the mountainous areas of Busan were divided into eight different zones by considering geographical and geological features. These zones are denoted as infiltration characteristic zones (hereafter referred to as IC zones). It was presumed that unique and homogeneous geo-hydrological features are exhibited throughout each IC zone. Figure 4: Map of eight infiltration characteristic (IC) zones and 17 Korea Meteorological Administration (KMA) rainfall observatories distributed within and near the Busan territory. The SEM-LEWS performance in terms of early warnings was examined according to two categories depending on the focus of early warning performance: analysis of the prediction statistics of historical landslide data and temporal performance analysis. Figure 5 depicts the framework of the performance analysis conducted in this study. Firstly, to analyze the prediction statistics, the relevance and accuracy of the warning level that was issued at each location of the collected historical landslide events were investigated for a particular fixed time (i.e., its occurrence time). In other words, for the simulated historical eight-year period, the prediction statistics were derived to provide answers for the following pragmatic questions: (1) Had appropriate warning levels been assigned when the landslide occurred? (2) How correctly did the warnings issued discriminate between slope failure and debris flow? (3) How many landslides occurred without the issuing of warnings (i.e., in the \"Null\" level)? Secondly, for the temporal performance analysis, several quantitative warning capabilities associated to timing were analyzed for a study region with high landslide-frequency.