# SPOT: Testing Stream Processing Programs with Symbolic Execution and Stream Synthesizing
## 5 Case Studies
### How Efficient in Triggering Failure  
The main goal of our third case study is to show that test data generated by SPOT is more efficient in triggering program failure compared with FlinkCheck and DiffStream. Program failure is one of the incorrect software behavior that necessarily means the existence of some defects. High-quality test data usually can trigger a failure faster than low-quality test data. This subsection adopts the minimum number of test data required to trigger failures as a metric to measure test data quality. We reproduced the experimental results of DiffStream and FlinkCheck in our experiment environment. Those two approaches commonly consist of two main ingredients: generators producing random values as test data and properties defining how the correct behavior looks. We replaced the random strategy of generators with our generation approaches in SPOT and reused the same properties. In particular, we adopted the safety property for the HarassMap program from the Flinkcheck paper, and \"output data preserving some order\" as the property for the TaxiDistance program from the DiffStream paper. The safety (i.e., something incorrect never happens) property states that given a stream of incidents with danger values greater than 1, then \"Safe\" results should never be returned. The \"output data preserving some order\" property defines the orderness of results from a parallel version and a sequential version should match equality for some deterministic DSP programs. In other words, the orderness of TaxiDistance's results should match regardless of whether it is running sequentially or in parallel. We conducted the experiment ten times and recorded the number of test cases executed before any failure was triggered. For HarassMap on FlinkCheck, we obtained the results shown in Figure 9a. To avoid the effects of random window sizes, we set the window size to a fixed value (i.e., four in this experiment) during our experiment. We can see from the results that to trigger the first failure, the number of executed test data varies from 2 to 21, and the mean value is 7, which means test data from FlinkCheck can trigger the failure with 7 test data on average. As a comparison, SPOT will always trigger the failure in the first window, which contains four distinct values corresponding to different paths, and the four data are processed following the increasing order of timestamps. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **\# of Test Data by SPOT** & **\# of Test Data by PBT** & **Difference** \\ \hline HarassMap & 4 & 8.8 & 2.2X \\ \hline SpeedRadar & 8 & 526.8 & \(\sim\)65X \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The minimum test data required to cover 100% JDU path coverage. We also conducted an experiment for the TaxiDistance program on DiffStream and ran it ten times as the experience for HarassMap above. Originally, the TaxiDistance program processes the taxi location information, which basically consists of a taxi identifier, the position data, and timestamps. The program can be divided into two operators, i.e., _project_ (data transformation) and _keyBy_ (partition all data by their ID). This program is required to preserve the ordering of events for each ID. Exchanging these two operators makes no difference when this program executes in sequence mode, but it may disrupt the ordering when it runs in parallel. To demonstrate the temporal features, we extended this program by adding a _AssignerWithPunctuatedWatermarks_, which assigns event time to each stream data according to the timestamp nested in them, and a 10-second _timeWindow_,  
Figure 9: Results of the number of test data to trigger software failure. which splits the stream into groups. Similar to HarassMap, we set the window in a fixed length (i.e., ten in this experiment). Results are shown in Figure 8(b). The number of test data varies from 74 to 2127, and the mean value is 877, which means that test data from DiffStream can trigger a failure with 877 test data on average.