# Augmented Reality and Machine Learning Incorporation Using YOLOv3 and ARKit
## 5 Augmented Reality Application Implementation  
Our iOS application was built using Xcode and Swift 5.0 and developed for iOS 14.0 or above. The application was tested on the iPhone X and iPhone XR, but it should also work on other recent iPhone models that support ARKit 4.0 and have the Bionic A11 chip or later built-in. Our iOS application source code is openly available at [[https://drive.google.com/file/d/1](https://drive.google.com/file/d/1) HPnggCnhE2 gqcfh9 Tiuuk2 FxvMEu3 qdw/view?usp=sharing]([https://drive.google.com/file/d/1](https://drive.google.com/file/d/1) HPnggCnhE2 gqcfh9 Tiuuk2 FxvMEu3 qdw/view?usp=sharing), accessed on 24 June 2021. **Importing the pre-trained deep neural network model to XCode. ** The deep neural network we trained (as discussed in Section 4) can be easily imported to XCode by converting to the Apple CoreML format using the Open Neural Network Exchange (ONNX). The ONNX is an open cross-platform deep learning model that helps developers move their trained models into different training frameworks. The converter then takes the pre-trained PyTorch model (.pth) as the input, and instead of running on the actual neural net, it will identify torch.onnx_export as the built-in PyTorch API to export to an ONNX formatted model. Exporting the ONNX format from PyTorch is essentially tracing the \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyper Parameter** & **Value** \\ \hline Activation & Linear, Leaky \\ Backbone & Darknet-53 \\ Batch size & 8 \\ Decay & 0.0005 \\ Epochs & 80 \\ Filter size & 64, 128, 256, 512, 1024 \\ Learning rate (final) & 0.2 \\ Learning rate (initial) & 0.01 \\ Momentum & 0.937 \\ Number of classes & 22 \\ Optimiser & SGD \\ Sub-division & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Training parameters and their values for the deep neural network. network, and the system internally runs the network on 'dummy data' to generate the graph. For this reason, the system needs a blank image with a similar size to the expected input: convert -size 416 \(\times\) 416 xc:white png24:dummy.jpg. The ONNX can then be converted to the CoreML model format by using the built-in convert-onnx-to-coreml package, as shown in the example below (Listing 1):  
```
importsys fromonnximportonnx_pb fromonnx_coremlimport=convert model_in=\"input.onnx\" model_out=\"output.coreml\" model_file=open(model_in,'rb') model_proto=onnx_pb.ModelProto() model_proto.ParseFromString(model_file.read()) coreml_model=convert(model_proto,image_input_names=['0'], image_output_names=['186']) coreml_model.save(model_out)
```  
Listing 1: Python code to convert Pytorch model to CoreML. The CoreML model then can be imported by dragging and dropping the model file into the XCode project. XCode will generate a new Swift class for the new imported model and we can use the model by creating an instance of this class. **iOS application structure**. The class diagram in Figure 14 presents the relationship between different Swift classes and how they fit and work together under the application system. The diagram is divided into three parts: (1) ViewController; (2) Yolo; and (3) AlteredImage. The ViewController class is the main controller of the application that holds the ARSCNView. This class is responsible for rendering the virtual objects and updating the application states. Every instruction is executed on the main UI thread as the system does not capture the camera's input frame in this class. Hence, there is no main thread blocking issue due to the video feedback execution. The Yolo class is where the actual deep neural network predicts the incoming frame from the camera. We set the prediction time interval to 0.03 s, which means that the system will start running the prediction process every 30 milliseconds if no prediction process is currently running. This set-up helps to reduce the system workload by classifying a single frame for every given interval of time instead of for all of the frames. Each predicted frame is converted to MLMultiArray and fed into the deep neural network model.