# Exploring Distributed Deep Learning Inference Using Raspberry Pi Spark Cluster
## 5 Discussion  
### Ineffectiveness of Distributed DL Inference Using Pi  
Based on the results of Configurations 2 and 3 in all experiments, the use of Spark cluster systems for the image processing tasks tested in all three experiments was ineffective, if not limiting. This ineffectiveness was highlighted when the cluster systems failed to complete an inference run with InceptionV3 on both CIFAR10 and ImageNetV2 datasets. Worse, these systems also failed to finish Experiment 2 with all 3 models. Regardless of the architecture of the cluster (Pi or VM), both systems exhibited the same out-of-memory (OOM) crash behaviour when executing inference with non-lightweight models (InceptionV3, RetinaFace) or on larger datasets like WIDER FACE. There were no crashes when testing programs in Experiment 3. It is suspected that the memory size of these complex models and the dataset could be responsible for this behaviour, as the clusters did not immediately crash when beginning testing but only several minutes into testing. It was later confirmed when TensorFlow warned about the lack of free memory allocation via the Spark output log, which subsequently caused the crash minutes into the tests. Aside from the OOM issues, the results did not show any noticeable speedup as seen in Tables 8 and 9, which is far from the expected values from 1.40 up to 2.00, especially as only two nodes were used to minimise the effect of diminishing performance returns associated with higher node counts. As per Figure 7a in [12], which was previously cited in Section 2.2, the authors of [12] used a Pi 4B Spark cluster for Spark MLlib workloads. There was a major reduction in the test set inference time from nearly 500 s to somewhere slightly above 200 s when two executors (nodes) were used instead of just one, which \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Test**} & \multicolumn{2}{c}{**Execution Time (s)**} & \multirow{2}{*}{**Enhancement Speedup**} \\ \cline{2-2} \cline{4-4}  & **Stock** & & **Enhanced** \\ \hline MTCNN & 3156.71 & 2882.82 & 1.10 \\ YOLOface & 14,502.36 & 12,768.22 & 1.14 \\ RetinaFace & 40,590.29 & 35,948.35 & 1.13 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results for Experiment 2 using the enhancements for Pi 4B. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Test**} & \multicolumn{2}{c}{**Frames per Second (FPS)**} & \multirow{2}{*}{**Enhancement Speedup**} \\ \cline{2-2} \cline{4-4}  & **Stock** & & **Enhanced** \\ \hline MTCNN & 2.64 & 2.99 & 1.13 \\ YOLOface & 0.21 & 0.25 & 1.19 \\ RetinaFace & 0.21 & 0.23 & 1.10 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results for Experiment 3 using the enhancements for Pi 4B. represents a speedup of 2.33. The inference results in this paper showed speedup values of less than 1 at 0.99, 0.90, and 0.95, which show slower inference speeds than a single node in some tests, and the cluster could only attain a speedup of 1.22. The slight speedup and the worse performance than running a single Pi completely contradicted the speedup scalability that was initially expected. While all the Pi clustering papers previously cited in Section 2.2 demonstrated speedup scalability across many nodes, the use of Spark TensorFlow Distributor for distributed inference in this paper did not show such scalability. Upon further investigation, it was found that the two executors were running the inference tests on their own, which may also be partly responsible for the OOM issue explored earlier. For context, the experiments utilised distributed versions of TensorFlow through the Spark TensorFlow Distributor, which is a library for Spark to distribute TensorFlow jobs to all executors. While Spark TensorFlow Distributor advertises distributed training, there was no explicit mention of distributed inference. However, support for distributed inference is obliquely suggested as the codebase in Spark TensorFlow Distributor has functions that support such a feature.