## 4 Proposed Inference Latency Prediction Approaches
### Proposed Inference Latency Prediction between Devices  
This section introduces the first method (Method 1), wherein the client predicts the inference latency of the edge and cloud with a given inference model. To analyze the similarity in inference latency between devices, we compare the inference latencies of  
Figure 5: Inference latency with the RFCN model for the client, edge, and cloud devices: (**a**) gRPC; (**b**) REST. Figure 6: Inference latency with the SSD-MobileNet model for the client, edge, and cloud devices: (**a**) gRPC; (**b**) REST. devices by using the normalized inference latency, which converts the distribution of inference latency to a normal distribution. Let \(X\) be the distribution of the inference latency for each device. Then, the distribution of normalized inference latency \(Z\) can be converted as below:  
\[Z=(X-\mu)/\sigma, \tag{1}\]  
where \(\mu\) and \(\sigma\) are the mean and standard deviation of the inference latency for each device for 5000 images. The notations used in this paper are given in Table 2. From now on, the device information is used as a subscript under the symbols of the mean and standard deviation. For example, \(\mu_{cloud}\) and \(\sigma_{cloud}\) are the mean and standard deviation of the cloud device, respectively. The statistical information is listed in Tables 3 and 4 for the RFCN and SSD-MobileNet models, respectively. devices. Also, the similarity between the client, edge, and cloud devices is more clearly revealed in REST than in gRPC. Therefore, it can be considered that the inference latency of the client can be utilized to predict the inference latency of the edge and cloud. Considering the similarity in normalized inference latency between the client, edge, and cloud devices, we propose a method for predicting the inference latency at the edges and clouds for the clients. The predicted inference latency for the edge and cloud in object detection can be obtained by using the normalized inference latency for the client and statistical information for the edge and cloud, as shown in Figure 9.  
Figure 8: Normalized inference latency with the SSD-MobileNet model for the client, edge, and cloud devices: (**a**) gRPC; (**b**) REST. Figure 7: Normalized inference latency with the RFCN model for the client, edge, and cloud devices: (**a**) gRPC; (**b**) REST. Let \(Z_{client}\) be the distribution of the normalized inference latency for the client device. This distribution can be easily obtained by executing object detection in the client. Then, the distributions of predicted inference latency for the edge and cloud \(\hat{X}_{edge}\) and \(\hat{X}_{cloud}\) can be calculated as below:  
\[\hat{X}_{edge}=\sigma_{edge}\cdot Z_{client}+\mu_{edge}, \tag{2}\]  
\[\hat{X}_{cloud}=\sigma_{cloud}\cdot Z_{client}+\mu_{cloud}. \tag{3}\]  
We assumed that the client was informed of the \(\mu\) and \(\sigma\) of the edge and cloud in advance. To employ the proposed technique, a process of calculating the \(\mu\) and \(\sigma\) for the edge and cloud according to the application and dataset is required. From Equations (2) and (3), the client can predict the inference latency of the edge and cloud. The predicted inference latencies for the edge and cloud are utilized as important parameters in determining whether the client executes the object detection task locally or remotely. In applications such as autonomous driving, which is very sensitive to latency, the communication latency and inference latency should be considered more importantly. Therefore, the proposed inference latency prediction technique is expected to play an important role in various applications. Additionally, the inference latency of the client and cloud is predictable at the edge. Let \(Z_{edge}\) be the distribution of the normalized inference latency for the edge device.