# Optimized Deep Neural Networks for Real-Time Object Classification on Embedded GPUs +
## 6 Experiments and Results
### Performance Evaluation of Proposed Scheme on Jetson TX1 (For GPUs having C.C. > 3.5)  
First of all, comparison of different versions of ConvMM layers is performed on the Jetson TX1 embedded board. Table 1 presents the performance of different ConvMM layers for various image sizes. Results show that the GPU-only ConvMM layer is approximately \(2\times\) faster than the heterogeneous version. This gain in performance is due to the elimination of extra memory transfers required by the CPU-based transformation step as discussed previously. In case of slower CPU hardware, this gain can be significantly higher. Figure 11 shows that the when GPU-only ConvMM layer is combined with the proposed memory optimization scheme (Unified), there is a huge performance gain in terms of execution time. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Size of Input Image** & \begin{tabular}{c} **Sequential** \\ **ConvMM (CPU)** \\ \end{tabular} & \begin{tabular}{c} **Hetero ConvMM** \\ **(CPU + GPU)** \\ \end{tabular} & \
\begin{tabular}{c} **GPU-Only** \\ **ConvMM** \\ \end{tabular} &
\begin{tabular}{c} **GPU-Only** \\ **Unified ConvMM** \\ \end{tabular} \\ \hline \hline \multicolumn{5}{c}{**Double**} \\ \hline \multicolumn{5}{c}{**(Milliseconds)**} \\ \hline Cifar (\(32\times 32\times 3\)) & \(11.17\) & \(8.03\) & \(5.12\) & \(2.75\) \\ \hline Imagenet (\(224\times 224\times 3\)) & \(348.67\) & \(89.39\) & \(35.6\) & \(13.13\) \\ \hline VGA (\(480\times 640\times 3\)) & \(2048.63\) & \(375.02\) & \(215.7\) & \(75.03\) \\ \hline SVGA (\(600\times 800\times 3\)) & \(3212.65\) & \(584.13\) & \(369.9\) & \(117.33\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of different versions of ConvMM layers (Single convolutional layer), best results are written in bold. This GPU-only unified memory scheme also outperforms the all other versions for the deeper networks as shown in Table 2. This performance gain is more evident here because of large data transfers required in the deeper networks. After this memory-optimized scheme, the cuBLAS library is used to accelerate the matrix multiplication. cuBLAS accelerated matrix multiplication (CAMM) is employed to compute the solution of matrix multiplication-based convolution algorithm. As the compute capability of Jetson TX1 board is greater than 3.5, cuBLAS library is best choice to access the computational resources of the Nvidia GPU. To use the cuBLAS API, the application must allocate the matrices in the device memory and as transformed input image and filters are already in the GPU memory, they can directly be used by the cuBLAS matrix multiplication subroutine as shown in Figure 9.  
It can be noted that all the trainable parameters imported from the Torch software are of double data type. These parameters and complete CUDA-based implementations discussed previously are reformed to use the single data type. This is done to reduce the size of the trainable parameters and to take benefit from the Single-Precision General Matrix-Matrix Multiply (SGEMM) routine of the cuBLAS library. Table 3 and Figure 12 depict the performance of this accelerated ConvCAMM layer. Results show that the cuBLAS-based ConvCAMM layer gains significant speedup over double and single precision GPU-only ConvMM layers. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Layers**} & \multicolumn{2}{c}{**Sequential**} & \multirow{2}{*}{ \begin{tabular}{c} **Hetero ConvMM** \\ **(CPU + GPU)** \\ \end{tabular}  } & \multicolumn{1}{c}{\ \begin{tabular}{c} **GPU-Only** \\ **ConvMM** \\ \end{tabular}  } & \multicolumn{1}{c}{ \begin{tabular}{c} **GPU-Only** \\ **Unified ConvMM** \\ \end{tabular}  } \\ \hline \hline \multirow{4}{*}{[PERSON]’s CIFAR-10} & \multirow{2}{*}{5} & \multirow{2}{*}{7814.25} & \multirow{2}{*}{129.9} & \multirow{2}{*}{107.03} & \multirow{2}{*}{**49.76**} \\  & & & & & \\ \cline{1-1} \cline{5-5} \cline{7-7}  
Table 4 list the performance of this single-precision ConvCAMM layer over three image classification architectures.