# Hardware-Aware Mobile Building Block Evaluation for Computer Vision
## 4 Hardware-Aware Mobile Building Blocks Evaluation
### Depthwise Separable Convolutions versus Standard Convolutions  
Since its introduction in MobileNet [1], depthwise separable convolutions have been the de facto building block for efficient vision models. A depthwise separable convolution factorizes a standard convolution into a depthwise convolution and a \(1\times 1\) convolution called a pointwise convolution. This factorisation reduces the computations by a factor of \(\frac{1}{C_{out}}+\frac{1}{K^{2}}\), where \(C_{out}\) is the number of output channels of the convolution and \(K\) the kernel size. For the commonly used \(3\times 3\) kernel size, this means a reduction by a factor of eight to nine. While depthwise separable convolutions were initially introduced for small vision models, they became standard in vision models across all complexity ranges, for example, the very large vision model EfficientNet-L2 with 480M parameters, and 585B FLOPS also uses them.  
Although standard convolutions may be more computationally expensive, they can, in certain cases, utilize the hardware resources more effectively. Hardware accelerators for DL commonly use wide single-instruction multiple-data (SIMD) processing units to achieve a high FLOP/S throughput. However high throughput can only be achieved if there is enough data re-use to fully utilize the processing units. In depthwise convolutions, the data re-use is much lower than in standard convolutions as each input feature map is only used in the computation of its corresponding output feature map. During training, resource utilization can be improved by using large batch sizes, obtaining data re-use with the convolutional kernel weights. However, during inference, where, typically, batch sizes of one are used, this places a memory bottleneck on the hardware accelerator. As a result, fused versions of the MobileNetV2 building block (Fused-MBConv), where the first pointwise and depthwise convolution are fused into a standard convolution, have recently been included in some hardware-aware neural architecture searches [24; 29; 30] and been shown to be amongst the best models on certain hardware platforms.  
Figure 5 compares the randomly sampled pareto front of the design space based on the depthwise separable convolution to the design space based on the standard convolution. It can be seen that depthwise separable convolutions outperform standard convolutions when comparing FLOPs. The performance curves for the mobile CPU and mobile GPU have similar trend lines and also favour the depthwise separable convolutions. For the Jetson Nano embedded GPU, however, we see that the theoretical advantage of standard convolutions no longer translates into faster latency. Instead, standard convolutions outperform depthwise convolutions as the complexity grows. On the VPU and server-grade GPU,  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Device** & **Processor** & **Framework** \\ \hline mobile CPU & Pixel4 & CortexA76 CPU & TFLite \\ mobile GPU & Pixel4 & Adreno 640 GPU & TFLite \\ VPU & Intel NCS2 & MyriadX VPU & OpenVINO \\ embedded GPU & Jetson Nano & NVIDIA Maxwell GPU & TensorRT \\ server GPU & Server & NVIDIA RTX 3090 GPU & PyTorch \\ \hline \hline \end{tabular}
\end{table}
Table 1: Evaluated hardware platforms.  
the initial order is reversed and standard convolution executes faster than the depthwise separable convolution.  
To quantify the difference, we selected models with equal FLOPs from both families and compared the latencies on the different hardware platforms. On the mobile CPU, models from both families with equal FLOPs also have about equal latencies, but, on the VPU, the inference time of the depthwise separable convolution models is twice as long as for the standard convolution models. A similar trend was seen on the embedded GPU, where the latencies of those models were a factor of 1.85\(\times\) longer.