# AWMC: Abnormal-Weather Monitoring and Curation Service Based on Dynamic Graph Embedding
## 4 AWMC Description
### Implementation
#### 4.2.3 Data Processing  
The data processing of our system is divided into five steps. The first step is to clean the dataset. In the original dataset, some of the weather data were missing. Therefore, we used the pd.fillna() function in the pandas package to fill in the missing values. The pd.fillna() function fills in missing data and includes six parameters: value, method, axis, inplace, limit, and downcast. We chose fill in the method parameter, which uses previous nonmissing values to fill the missing values. The second step is to build dynamic weather graphs, calculate the graph entropy of each graph, and build dynamic supervised graphs. To build a dynamic weather graph, since the weather data included seven types, we built the graph matrix that transformed the data of every day into a \(7\times 7\) matrix. The rows and columns in the graph matrix represent the seven types of weather data, and the values inside the matrix represent the value of \(PCC\) among the seven types of weather data. Therefore, the values on the diagonal of the graph matrix were all 1. Then, we calculated the graph entropy of each graph by using Equations (3)-(5). Lastly, to build a dynamic supervised graph, we calculated the most similar corresponding graph to each graph based on Equation (6).  
The third step builds a dynamic graph embedding model. We built two autoencoders, one to train the dynamic weather graphs, and another to train training dynamic supervised graphs.  
The fourth step is to apply the local outlier factor (LOF) algorithm [35] to the embedding space to calculate the anomaly scores of each day. LOF is an unsupervised anomaly detection method based on density that calculates the local outlier factor for each point in the space; if the local outlier factor is much greater than 1, the point is judged to be an anomaly, and if the local outlier factor is close to 1, the point is judged to be a normal point.  
Figure 2: Relationship between the tables.  
For calculating local outlier factors, we needed to first calculate the reachability distance. The formula of reachability distance is defined as follows.  
\[reachability-distance_{k}(o,p)=max\{k-distance(p),d(o,p)\} \tag{8}\]  
where \(o\) and \(p\) are data points in the space, \(k-distance(p)\) is the distance of point p to the \(k\)th nearest neighbor. and \(d(o,p)\) is the distance between points \(o\) and \(p\). Next, we calculated the local reachability density. The formula of local reachability density of point \(o\) is defined as follows.  
\[lrd_{k}(o)=1/(\frac{\sum_{p\in N_{k}(o)}reachability-distance_{k}(o,p)}{|N_{k} (o)|}) \tag{9}\]  
where \(N_{k}(o)\) is the set of the \(k\) nearest neighbors of point \(o\). Then, we could calculate the local outlier factor by using local reachability density. The formula of the outlier factor is as follows.  
\[LOF_{k}(p)=\frac{\sum_{o\in N_{k}(p)}}{|N_{k}(p)|}/lrd_{k}(p) \tag{10}\]  
In our system, we set the 10 nearest neighbors, and set the anomaly rate to 0.1 by using LOF function in the sklearn package. Lastly, we could calculate the local outlier factor of each point as the anomaly score for each day.  
In the last step, since the weather data of each day included the data of each hour from 0:00 to 23:00, we calculated the average of each type of weather data for each day to understand the relationship between changes in weather anomalies and weather data.