Implementing Deep Convolutional Neural Networks in C without External Libraries | by Milad Abdollahzadeh | Towards Data Science

In this post, we will discuss how to implement the inference of a pre-trained deep Convolutional Neural Network (CNN) in C without any external libraries. This code is developed for the YUV video super-resolution application. However, the functions are modular and this can be applied to other applications and network structures with a little modification. Since this code is written in pure C without any external libraries, it can easily be integrated into other frameworks. In this post, it is assumed that the reader is familiar with C programming and basic topics in CNN. In the following, first, we briefly discuss the YUV video format and the theFSRCNN algorithm for super-resolution. Then the implementation of this deep CNN for YUV video super-resolution is discussed in more detail. The code can be found here.
YUV is a color space like RGB. In a YUV video, each frame has three components. Y is the luminance component which is a weighted average of R, G, and B components:
Y = k1R + k2G + k3B
Color information is represented using U and V components as color difference or chrominance. One primary benefit of luma/chroma systems like YUV is their compatibility with black and white analog television since the Y component holds all the black and white data. However, the key advantage of YUV color space over RGB is that the U and V may be represented with a lower resolution than Y because the Human Visual System (HVS) is less sensitive to color than luminance. This reduces the amount of data required to represent the chrominance components without apparent differences in video quality.
One popular pattern of YUV is 4:2:0 (also called YUV420) which means each of the U and V components has half the horizontal and vertical resolution of Y. For example for a 4×4 grid of pixels, RGB uses 8×16×3 bits, while YUV420 uses 8×(16+4+4). In this way, a YUV420 video uses an average of 12 bits per pixel compared to 24 bits per pixel in RGB color space. This means saving 50% of bandwidth without an obvious effect on the quality.
Video encoders like HEVC or H.264/AVC encode this YUV420 video and the compressed output is used for storage or communication. For playing video, the video player decodes the compressed bitstream. In the cases that the resolution of the video is low, we can increase the resolution of the video artificially before playing it using Super-Resolution (SR) methods. Next, we will discuss a simple yet efficient SR method.
Fast Super-Resolution Convolutional Neural Networks (FSRCNN) is a simple yet efficient algorithm proposed to map low-resolution (LR) input images to high-resolution (HR) images. It includes five main parts, the first four parts are the convolutional layer, and the fifth is deconvolutional (inverse convolution). For simplicity, we use Conv(k, o, i) to show a convolutional layer with kernel size k, i input channels, and o filters. The same notation is used for a deconvolutional layer (inverse convolution) as DeConv(k, o, i).
The first part is the feature extractor which extracts LR features.
The second part is shrinking to reduce the dimension of LR features.
Non-linear mapping consists of 4 convolutional layers which map the LR features to HR features.
Expanding increases the dimension of HR features to make them ready for reconstruction.
Finally, the deconvolution upsamples and aggregates the HR features with a set of deconvolution filters.
Parametric Rectified Linear Unit (PReLU) is used as activation function which is defined as follows:
f(xi) = max(xi,0) + aimin(xi,0)
The parameter ai is the coefficient of the negative part which is learnable. PReLU is used to avoid dead features. Connecting these five parts forms a eight-layer network as: Conv(5,56,1) — PReLU — Conv(1,12,56) — PReLU — 4 × {Conv(3,12,12) — PReLU} — Conv(1,56,12) — PReLU — DeConv(9,1,12). For more information about the design details of FSRCNN, please refer to the original paper.
We apply FSRCNN on the Y component of the YUV420 video. Since the HVS has less sensitivity on U and V, these components could be upsampled using simpler interpolation algorithms like Bicubic. Hence in this post, we focus on implementing CNN on C, for upsampling U and V components, we simply repeat existing elements to fill the unknown location. In order to implement the inference of a pre-trained deep CNN in C we need to perform the following steps:
Read Input Data
Implement Required Operations (Conv, DeConv, PReLU, and etc.)
Read Weights of the pre-trained network
In order to read the YUV file with C, we need to know the dimension of the video (width and height). Then YUV file can be considered as a binary file where for each frame, first, we have the data for the Y component, and then U, and V components. An example for a video with spatial resolution m×n is shown below:
Then we can use the fread function of C to read the binary file. Following code snippet shows how to take the name of the input YUV video from the terminal, and return a pointer indicating the beginning of the video file:
Since we are implementing the CNN in pure C, we need to implement the required functions including the convolution, padding, deconvolution, and PReLU.
Convolution. We will not go into the mathematical details of the convolution operator but will try to recall how it works over images as a matrix. Then change it into a binary data case, where all rows are concatenated and formed a vector. In a 2D convolution, there is a matrix of weights called the kernel. This kernel “slides” over the 2D data, performing an element-wise multiplication with the part of the input it is currently on, and then summing up the results into a single output pixel.
The challenging part of our implementation is that instead of having a matrix of data, we only have a pointer that points to the beginning of a binary file. So, we need to use this pointer properly to mimic the exact situation that occurs in 2D convolution. An example of applying the convolution kernel into a binary file is shown in the figure below for a 4 × 4 input data and a 3 × 3 kernel:
We implement this convolution operation as the following imfilter function:
This function takes the following pointers as input: *img, *kernel, and *img_fltr. For example, *img points to the beginning element of the input image in the memory. Using this pointer we can access each element of the input image by adding an offset value. For example, the third element of the input image (row0, column2 in 2D space) can be accessed by *(img+2). Note that since we are modifying the values using pointers, the imfilter function implicitly applies the changes and it returns nothing (void function).
To start the process, we first pad the input image before convolution. We use the pad_image function for this purpose which simply repeats the border pixel values in padded location. Suppose the kernel_size=k, and padsize=(k-1)/2. Then, for each pixel location in the input image, we determine a k×k region, with that pixel being in the center of this region. Since the raws of each frame are concatenated in the binary file, the offset value in our code for this pixel would be cnt = (i — padsize)*cols + (j — padsize). Next, we perform an element-wise multiplication between the kernel and determined region. To do this, we loop over the locations of the kernel and find the offset value for the neighboring pixels in the same manner: cnt_pad = (i + k1)*cols_pad + j + k2. Then the value of each neighbor pixel can be accessed by *(img_pad+cnt_pad). For accessing the values in the kernel we use a simple incremental counter cnt_krnl.
PReLU. This function is quite straightforward since it performs on each pixel value independently. The following code snippet shows our C implementation of PReLU for a YUV filtered frame:
Deconvolution. The deconvolution layer used in FSRCNN can be considered as the inverse convolution process. For convolution, if the filter is convolved with the image with a stride s, the output would be 1/s times of the input. Then, if we exchange the position of the input and output, the output will be s times of the input. By selecting s as a desired upscaling factor (2 in our implementation), the output would be directly the HR image.
Considering this, we can implement the deconvolution similar to the imfilter function but in a reverse manner. The code for the deconvolution function in C can be found here.
After implementing the required functions, the next step would be to load the weights of the pre-trained network in C. Since the saved file formats of the networks trained in deep learning frameworks like PyTorch or TensorFlow are not supported in C language, we can use a simple trick to load the parameters. We save the parameters of the network (including weights and biases) as a *.txt file and use fread to access them in C. The following code snippet is an example of reading the weights and biases for the first layer:
The important point is that since weights are in 3D tensor format, we should concatenate them by row. The converted weights and biases are available on the GitHub page as txt files.
After implementing the required functions, and finding a way to read the parameters of the network, we implement the different layers of the network by calling the required functions in the main function. As an example, we run this on the foreman_qcif.yuv video. In order to compile and run the code in Linux terminal, we use the following commands:
gcc source.c -o videosr./videosr foreman_qcif_146x144.yuv output_foreman_352x288.yuv
This is an example of how it looks like:
Note that you will need a YUV video player to play these videos!