# Building Survivable Software Systems by Automatically Adapting to Sensor Changes
## 4 Empirical Study
### Evaluation in BRASS Project  
In the evaluation of the BRASS Project [1] Phase 1, we conducted extensive experiments on Weather Underground Data. We organized data into 13 clusters, covering 13 regions in Los Angeles, San Francisco, Austin and Chicago. In each cluster, there were three weather stations, each containing 2 years of weather data. Five individual sensors (temperature, humidity, dew point, wind speed and wind gust) were used in all stations.  
We evaluated our adaptation algorithms over randomly chosen clusters, stations, sensors, and time periods. Once a random cluster was chosen, we randomly picked two stations (A1 and A2). Since the two stations were from the same cluster, their sensor values were relatively similar. We further randomly picked an individual sensor from station A1, and replaced it with the same individual sensor from station A2. To enable adaptation, we used training data from a 2-month time period (without sensor change). We then used 1-month data for the adaptation period (sensor change happened in the beginning), and 1-month data for the evaluation period. The 4-month data were consecutive, as shown in Figure 6. The goal was to learn an adaptation function based on the data in the training and adaptation periods, and then evaluate adaptation performance in the evaluation period.  
To determine whether an adaptation succeeds or not, we introduced a benchmark called the reference error. It defines a domain-specific baseline error bound that our system could tolerate. If adaptation error was less than the reference error, we considered the adaptation to be successful. In our implementation, we estimated the reference error by averaging the errors between every pair of weather stations in a cluster over the evaluation period.  
Table 5 summarizes the adaptation performance on random tests described above. Our evaluation was performed on cases where the error of no adaptation (i.e., direct use of the new sensor) exceeded the reference error. ASC achieved high success rate on temperature, humidity, dew point and wind speed. On wind gust, the success rate was relatively low due to large variance in the sensor values. Despite performance drops on wind speed and wind gust, ASC showed positive improvement over reference error on all individual sensors.  
Figure 7 shows the reconstructed wind gust in one random test. The blue curve represents the wind gust from a target station and a nearby station. The red curve represents  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Sensor** & **Success Rate (\%)** & **Imp.** \\ \hline temperature & 95.4 & 61.6 \\ humidity & 96 & 65.8 \\ dew point & 100 & 71.1 \\ wind speed & 84.6 & 28.7 \\ wind gust & 66.7 & 24.0 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Adaptation performance on random tests in the BRASS project evaluation. Imp. shows the average improvement (in %) over the given reference error.  
Figure 6: Illustration of training, adaptation and evaluation periods in the BRASS project evaluation. We use data from \(K\) sensors (\(S_{1},S_{2},\cdots,S_{K}\)). The training period generates data without sensor change for 2 months. The adaptation period generates data for 1 month, in which \(S_{K}\) is replaced by a different sensor at the beginning. The evaluation period uses data from the next 1 month.  
the wind gust after adaptation, which was much more similar to the original signal in the training period.