# Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training
## 3 Design and Implementation of the AR App
### TensorFlow Object Detection API  
TensorFlow (TF) API, developed by Google Brain, is a framework for creating a DL network [13]. It is a powerful tool that can be used to create a robust object detection framework with a set of standard functions, eliminating the need for users to write code from scratch. It also provides a list of pre-trained models, which are useful not only for inference but also for building models with new data. Model Zoo is a collection of models that have been previously trained using the common objects in context (COCO) dataset [49]. A workflow for training a DL model using the TF API is shown in Figure 4 and can be described through the following steps:  
Figure 3: MobileNet SSD deep neural network architecture. 1. _Image Dataset_: The model was given input of 643 images collected from various perspective views and in different lighting settings. Each image is of 4032 x 3024 pixels in size. It is necessary to annotate these images before using them to train the model. A software, LabelImg[50], is used in the annotation process [51] that allows users to draw a rectangle in a specific area of the image. During training, the annotation will help the model precisely locate the object in the image. The outlining will generate and save coordinate points in an XML file. 2. _TensorFlow Dataset_: To make the computation of the DL framework efficient, TF records use a binary file format. Furthermore, TF records enable the dataset to be stored as a sequence of binary strings that improves the model's performance while using less disk space. We converted the XML files generated by LabelImg into TF binary records using a Python script. The last step in configuring the TF dataset is to create a.pktxt file containing all of the categorical label classes that will be stored in a TF record file. 3. _Configuration File_: Multiple pre-trained models based on the common objects in context (COCO) dataset are available in TF. These models can be used to set up DL models prior to training on a new dataset. Table 1 lists several popular architectures with pre-trained models. For instance, _ssd_MobileNet_v1_coco_ is the SSD with a MobileNet v1 configuration, _ssd_inception_v2_coco_ represents an SSD with an Inception v2 configuration, and _faster_rcnn_resnet101_coco_ stands for Faster R-CNN with a Resnet-101 (v1) configuration. All these configurations have been derived for the COCO dataset. From Table 1, it can be observed that _ssd_MobileNet_v1_coco_ reaches the fastest inference speed of 30 ms but with the lowest mean average precision (mAP). In contrast, _faster_rcnn_resnet101_coco_ has the slowest inference speed but the highest mAP of 32. We tested both MobileNet SSD v2 and faster RCNN [52] and concluded that MobileNet SSD v2 performs faster inference in mobile devices than the faster-RCNN model in our study. Using a pre-trained model saves time and computing resources. A configuration file, in addition to the pre-trained model, is also required. It must match the same architecture of the pre-trained model. It is recommended to fine-tune the model to maximize the prediction outcome. The process of fine-tuning is divided into two steps: restoring weights and updating weights. After we completed the requirements, we ran the python code provided for TF API to start the training job. Following training, the API will generate a file serving as a training checkpoint in a specific format named.ckpt. This file is a binary file containing all of the weights, biases, and other variables' values. 4. _Inference_: After training the model, the last step is to put it into production and feed the model with live data to calculate the predicted output. Before testing, we can evaluate the model's accuracy using mAP. In Section 4, the evaluation result is described in detail. We also need a lightweight version of the model to perform inference, so we choose an OpenCV library.