# Aika: A Distributed Edge System for AI Inference
## 4 Experimental Setup and Evaluation
### Failure Experiment  
The purpose of the failure experiment is to gain insight into how the performance is affected when working agents are being regularly shut down. We also investigate to what degree the system manages to remain stable when agents are being shut down and recovered. Figure 14 illustrates the different performance results obtained when stress testing the system while simulating failures. In our failure simulation experiments, a Killer process is instructed to pick one random worker agent or initial agent at 15 s intervals and terminate that process. The Killer runs for the entire duration of the experiment when deployed. Measurements with and without the Killer process are similar both in terms of the series shape and the performance. When the Killer is deployed, the measurements do, however, become slightly worse. Despite the agents being terminated every 15 s, the performance still remains relatively similar in terms of stability. Agents being terminated every 15 s and the number of requests received being measured every 5 s means that the performance should decrease in every third measurement. This could explain the lack of visible dips in performance. Figure 13: Results obtained when measuring the number of requests received where worker agents in the pipeline sleep for one second after processing an item. Measurements are carried out with persistent queues (top) and in-memory queues (bottom). The number of requests received is measured and reset every 5 s. The plot shows number of requests per second over each 5 s interval. Table 2 summarizes the observation for the various experiments. These measurements confirm the overhead created when using persistent queues becomes negligible when the workload is increased. Not only are the results more stable for persistent queues (standard deviation 0.18 versus 0.22), it also performs better overall with an average of 0.88 requests processed per second compared to 0.80 s for the in-memory queue. The performance does decrease slightly, from 5.82 requests per second during the stress test to 5.08 requests per second during the same test with the Killer component deployed. The stability of the system also seems to decrease slightly, as the standard deviation changes from 2.29 to 2.58 requests per second when the Killer component is deployed. It is important to note that despite the performance gain from using in-memory queues, the potential consequence of this is that the system remains unable to properly recover from faults if any component shuts down during the runtime. A local controller may resume an agent that crashes, enabling it to continue working. When using in-memory queues, the data stored will, however, be lost if the agent itself is to crash. This requires a complex recovery routine, where the agent initializing the pipeline has to re-retrieve the specific items that have been lost during the crash and propagate them through the entire pipeline \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Queue Type Used** & **Min Value** & **Mean Value** & **Standard Deviation** & **Median Value** & **Max Value** \\ \hline Persistent Queue (No sleep) & 2.40 & 5.82 & 2.29 & 5.40 & 22.80 \\ \hline In-memory Queue (No sleep) & 551.40 & 636.89 & 68.67 & 612.30 & 878.60 \\ \hline Persistent Queue (Kill every 15 s) & 1.8 & 5.08 & 2.58 & 4.6 & 17.8 \\ \hline Persistent Queue (Sleep 0.9–1.1 s) & 0 & 0.88 & 0.18 & 1 & 1.2 \\ \hline In-memory Queue (Sleep 0.9–1.1 s) & 0 & 0.80 & 0.22 & 0.8 & 1.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: A summary derived from the continuous performance experiments explained previously. The table contains the minimum, maximum, mean, median, and standard deviation of all requests per second measured for each benchmark. Figure 14: Results obtained when measuring the number of requests received during stress-testing over the course 1200 s without (top) and with (bottom) the killer deployed.