# Exploring Distributed Deep Learning Inference Using Raspberry Pi Spark Cluster
## 3 Methodology  
### Evaluation Metrics  
This section outlines and explores the performance metrics that were used for evaluation in the experiments. Most of the performance metrics are chosen from the research paper reviews in Section 2, though interpretations of the metrics may be different depending on the context of the research scope and objective.  
The metrics chosen for the experiments are execution time, frames per second (FPS), and speedup. Their respective equations are shown in Table 2. Execution time is defined as \"a period in which an event is actively operating\" [17]. It is the time taken to complete an execution job. Execution time as a performance metric has been used in many papers involving processing applications, as with [14; 16] reviewed earlier in Section 2. Execution time is used as the primary performance metric in this study because this metric well represents the real-world processing performance for an application or a system, as opposed to the theoretical on-paper performance measures such as GFLOPS. In this paper, this term is synonymous with model inference time.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Study** & **Pi Model** & **DL/ML** & **Clustering** & **Framework Used** & **Findings** \\ \hline \multirow{3}{*}{[13]} & \multirow{3}{*}{4B} & \multirow{3}{*}{Yes} & \multirow{3}{*}{No} & & \multicolumn{3}{c}{Pi 4B performs the worst in inference tests involving} \\  & & & & & & Jetson Nano and Jetson TX2 due to a lack of \\ \cline{4-6}  & & & & & & AI acceleration. \\ \cline{4-6}  & & & & & Pi 3B with AI accelerator performs almost similarly to \\ \cline{4-6}  & & & & & laptop computer in SSD, however it falls behind in SLP and P2P with a vast gap. \\ \cline{4-6}  & & & & & & \\ \cline{4-6}  & & & & & & \\ \cline{4-6}  & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of research coverage from previous works regarding Pi DL inference and distributed Pi clusters.  
The definition of frames per second is perhaps best defined as the number of image frames rendered in one second, which is commonly used for videos and for benchmarking graphics performance in video games. This is the case as videos are essentially a continuous sequence of image frames. However, there are multiple interpretations of this metric. For example, Ref. [18] defined it as the \"measure of fluidness of movement\", as higher FPS relates to smoother and more fluid motion in videos and games alike. Instead of merely focusing on video camera fluidity and smoothness, FPS is also used as a metric for inference speed in object detection and recognition, especially in [19], where the authors benchmarked ZF's in-house ProAI SBC against the NVIDIA Jetson Nano and a Dell laptop with a dedicated NVIDIA Quadro P2000 workstation GPU. This specific metric is used for the experiment where rendering frames that have undergone face detection highly depend on the inference speed of the face detection model, rather than the FPS as captured by the camera. For this study, the former is used since DL model inference related to face detection processes a series of frames that are captured by a source before being returned as another set of frames.  
Speedup is defined as \"the ratio of the performance improvement on different input sizes between two systems processing the same problem\" [20]. Speedup is a metric used to measure the effectiveness of a configuration versus some baseline configuration for a processing job, as found in [15; 20]. Besides being used in clustering versus single node scenarios, as per the case of Pi 4B versus the 2-node Pi 4B Spark cluster, it is also used for comparing the performance difference between different system configurations, processing models, and software settings. This specific metric is used for measuring the effectiveness of distributed inference using Spark TensorFlow Distributor, the effectiveness of the enhancements on the Pi, and the performance gap between systems, which are the scopes of this study.