# Benchmarking Deep Learning Models for Instance Segmentation
## 2 Related Work  
The performance of mobile AI accelerators is improving rapidly every year, and thus attempts to utilize deep learning models on mobile hardware continue to increase. [PERSON] et al. [11] benchmarked the performance of mobile chipsets that support hardware acceleration for AI inference for deep learning models. They selected networks that performed vision tasks to present the relative inference performance of mobile SoCs in 16-bit floating-point (fp-16) networks. They also presented the relative inference performance in 32-bit floating-point (fp-32) networks so that the SoCs performance can be compared with that of Intel CPUs and NVIDIA GPUs. Finally, they presented the relative performance of SoCs in 8-bit integer-quantized networks. With the increasing amounts of data available and the rapid improvements in the computing performance of hardware, deep learning models are achieving many successes in various areas. Applying deep learning to mobile or embedded devices is gaining increasing attention, and comparing performances by benchmarking the AI performance of each device has also been adopted as an important challenge. [PERSON] et al. [12] proposed the AloTBench, a benchmark suite for evaluating inference performance on mobile and embedded devices. They also provide data that compare and rank the performance of five mobile devices in representative heavy and light networks. Deep neural networks (DNNs) are achieving remarkable results in many computer vision tasks. The literature shows that researchers have conducted many studies aimed at designing efficient and accurate networks. [PERSON] et al. [13] compared the performance of more than 40 state-of-the-art DNN architectures for image recognition tasks. They provide performance indices such as the recognition accuracy, inference speed, and memory usage of DNNs running on two systems with significant differences in computing power, and analyze the association between these performance indices. With the use of CNNs, many studies have progressed on object detection tasks. Although some state-of-the-art object detectors have been shown to be sufficiently fast to run on mobile devices, it may be difficult for practitioners to determine which architecture is suitable for their purposes. Therefore, [PERSON] et al. [14] conducted experiments and analyzed the major factors affecting the trade-off between the inference speed and detection accuracy of modern object detectors. They presented measurements such as the architecture's inference speed, detection accuracy, and memory usage, which depend on variables such as feature extractors, image resolution, and the number of proposals in a fixed hardware environment. The training of deep learning models is computation-intensive, and new and better hardware and software platforms are being proposed to train and deploy more sophisticated models. Thus, [PERSON] et al. [15] introduced ParaDnn, a parameterized benchmark suite for benchmarking deep learning platforms. They benchmark Google TPU, NVIDIA GPU, and Intel CPU platforms with six DNN models to compare the execution time for training according to various specifications of hardware and software. Many models and algorithms have been proposed in computer vision research, and the importance of datasets and benchmarks has also been demonstrated as a means of quantitatively evaluating and comparing them. [PERSON] et al. [16] presented a new public dataset that focuses on robots performing vision tasks in difficult indoor environments. The dataset contains more than 20,000 RGB-D images and 50,000 object 2D bounding boxes for instances captured in nine unique scenes. [PERSON] et al. [17] presented a new video object segmentation benchmark dataset, called the Densely Annotated Video Segmentation (DAVIS), consisting of high-quality sequences at full HD resolution. They compared state-of-the-art methods with their proposed three metrics using DAVIS, which includes common situations such as motion blur and occlusion that may arise from video object segmentation tasks.