## 5 Evaluation
### Evaluation of the Final Architecture Design  
#### 5.3.1 Feasibility  
The following implementation-related lessons were learnt regarding feasibility of the final architecture design:  
* NFS server was tested for storing of model weight files. The integration between Rancher and NFS server was performed successfully with K8S Persistent Volume Claims [48]. However, k3s didn't provide support for a NFS client. Thus, the received weight files were stored locally to the edge-node, and Rancher's Local Path Provisioner [18] was utilised for accessing of the files.
* Initially, Docker images for ARM-devices (Jetson Nano, RPi 4) were built with a laptop, which contained Ubuntu VM in a VirtualBox. Docker images were compiled with Docker's buildx-tool [49], which is an experimental feature in Docker. However, it was discovered that building of the images was too slow as images couldn't be built within a working day. Thus, Docker images for ARM-devices had to be built with Jetson Nano.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Inference** & \multicolumn{3}{c}{**Model Switching**} \\ \cline{2-5}  & **Jetson** & **RPi4** & **Jetson** & **RPi4** \\ \hline
**Avg. latency (s)** & 7.72 & 10.03 & 26.49 & 39.09 \\ \hline
**99 th perc. latency (s)** & 9.05 & 12.85 & 29.17 & 42.29 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Latency in model switching tests.  
Figure 10: Results of model switching latency.  
* Nvidia has published base images including Tensorflow libraries to be used for building of Docker images for ARM-devices based on Nvidia's libraries [50]. The Docker image for Jetson Nano was built based on Nvidia's base image (l4t-tensorflow:r32.4.3-tf1.15-py3), which contained Tensorflow v1.15. However, RPi 4 does not support Nvidia's libraries. The authors could not find official Docker images, which would contain Tensorflow libraries for RPi. Thus, an unofficial Tensorflow image [51] was used as a base image for RPi 4.
* In order to get access to GPU-resources, Docker's default runtime had to be changed to 'nvidia' in Jetson Nano. Additionally, k3s uses contained as the default container runtime, which had to be changed to Docker in order to get access to GPU resources via Nvidia's libraries.  
The SBN-implementation [23] was utilised for energy consumption prediction in the architecture. Model weights of a trained model (trained with the SBN-implementation) were transferred in a file to the edge nodes. Additionally, data had to be available in the edge node for scaling input data similarly, which was performed during training. Model conversion/saving to a file may have been a better option, which would not have required the presence of training data in the edge node(s). However, model conversion was not feasible due to an error, which was encountered during model saving (see Phase 1 in Appendix D).  
Feasibility of the architecture may be evaluated in terms of the RA it is based on, and the applied technological choices. Three architectural views of the RA for edge computing systems [17] were applied as a starting point in the design of the presented implementation architecture (Figures 2-4). The implemented architecture was designed based on the elements of the RA. However, the RA could be evaluated only partly, as all elements of the RA were not covered by the prototype. For example, data extraction was performed in an earlier project [23], and visualisation and interfacing-related elements were not implemented. Similarly, only model development and deployment technological layers of the Archimate-view could be evaluated in this paper. We presented the unimplemented architectural elements, due to their importance for understanding the context of energy consumption prediction in facilities.