Machine learning model deployment with C++ | by Babatunde | Towards Data Science

Recently, I have been fascinated with how interesting it would be to build a mathematically inclined application and deploy it at scale without any restriction to model size, platform, or need for api calls. I know that Python has enough of a library for working with prototypes of machine learning projects, however not many are talking about scaling this project, especially when you don’t want to do that over a web api.
I believe true intelligence shouldn’t rely only on calls to an api for a model to be available in scale. This fascination led me to research what it will take to use C++ for machine learning and general intelligence.
My conviction is that both Matlab and Python’s mathematical strengths are based on an underlying c/c++ code. Hence, fundamentally scaling technology to work with mathematical computations involved in machine learning with a blazing fast scenario in mind will likely require that you are able to dig into low-level programming and most especially with c/c++.
Also, I wondered why most computer science schools ensure that there is a c/c++ curriculum in their course. This emphasises the reasoning of using c/c++ for scalable technology.
After learning c++ using an Udemy hands-on course, now the challenge is to integrate a simple face recognition application in an android.
The write-up will include some preliminary approaches of what you need to build a c++ project and deploy in android or any other os environment.
Components:
Set up a c++ project for machine learning with opencv.Learning a PCA to generate eigenfacesSetting up a .so inference library for multiplatform deploymentDeveloping a jni wrapper for the inference library.Using the library in Android or any java code.
Set up a c++ project for machine learning with opencv.
Learning a PCA to generate eigenfaces
Setting up a .so inference library for multiplatform deployment
Developing a jni wrapper for the inference library.
Using the library in Android or any java code.
The first part will be to learn a machine learning algorithm with OpenCV. In this case, we are going to explore the most basic of face recognition algorithm, using principal component analysis for eigenfaces. The machine learning community is very familiar with this in python especially with tools such as sci-kit learn, but when production and most especially offline/on-device production comes to mind, the need to do this from a different dimension is expedient.
OpenCV comes with a very good api for learning principal component analysis and it is quite straight forward to learn once you have your data all set up.
The key part of your CMakeFile.txt is ensuring that there is an OpenCV library in 0 project and available for your library to compile. Ideally, before requesting that Cmake find OpenCV for you, it is important to have the OpenCV library installed on your machine.
find_package(OpenCV REQUIRED)include_directories(${OpenCV_INCLUDE_DIRS})set(LIBS ${OpenCV_LIBS})target_link_libraries(featureCalculation ${LIBS})
PS: I had to set up my cmake differently for training and inference. I will share both
Now that OpenCV is available, learning PCA is quite straight forward. Here is the logic invole:
Read all image data as an array.Using cv::glob from OpenCV, all filenames ending with a .jpg, .png or/and .jpeg can be read with cv::imread, and the data preprocessing of the image data can proceed.
Crop faces.This is important because PCA does much better with the face image than the whole image. I have found Multi-task Cascaded Convolutional Networks (MTCNN) to be the most reliable yet simple and minimal face detection and cropping model out there. There is an implementation in C++ using the original model with a Caffe network in OpenCV.
Convert cropped faces to gray scale.This part is pretty straight forward. Using cv::cvtColor(originalimagemat, grayscaleimagematcontainer, cv::COLOR_BGR2GRAY) we can convert an original BGR image to grayscale in OpenCV.
Other preprocessing — One other preprocessing is to ensure that the data types are right. This is very important because C++ is heavy on the precision of data types. It is quite easy to introduce bugs at this point hence the reason to carefully ensure that your data types are right. Apart from this, it is a good idea to normalize your image data and resize all images into a consistent shape. PCA works only if the data are in the same dimension. Out of the box from OpenCV, we can employ the following functions to take care of the preprocessing:
cv::resize(originalImage, containermatofnewimage, size) for resizing the image and originalmat::convertTo(newmatnormalized, CV_32FC3, 1/255.0) to normalize the pixel values of the image.
Convert all images to a data table — A data table is somewhat like a single table of data where each element is represented as a row, interestingly we can think of each row of our data table as individual images in its flattened format. The essence of PCA is to project the image values to a few columns with a distinct representation of that image. Therefore, the data table will have rows equals to the number of images in the training dataset while the columns will be the normalized grayscale values of each image.
To create the data table, std::vector can be used to hold all the images (with the hope that they fit in memory) which is then copied to every row of the data matrix. Here is an helper function that does exactly that from a vector of images mat.
cv::Mat createdatamatrix(std::vector<cv::Mat> imageArray) {cv::Mat datamatrix(static_cast<int>(imageArray.size()), imageArray[0].rows * imageArray[0].cols, CV_32F);unsigned int i;for(i=0; i < imageArray.size(); i++) {   cv::Mat imageRow = imageArray[i].clone().reshape(1, 1);   cv::Mat rowIData = datamatrix.row(i);   imageRow.copyTo(rowIData);  }  return datamatrix;}
cv::reshape() helps transform mat arrays to different shapes, with (1, 1) it literally means we want the data to exist in a single row.
Learn the actual Pricipal component analysis algorithm.Now that we have created the data table with the preprocessed face images, learning the PCA model is usually smooth. As smooth as passing the data table to an OpenCV pca instance with your expected maximum components like such cv::PCA pca(datatable, cv::Mat(), cv::PCA::DATA_AS_ROW, number_of_components). With this, we have a learned PCA written in C++ which is ready for production use.
To transfer this model for use in any environment, open cv has a FileStorage object that allows you to save a mat as it is. Thus I can save this file and pass its filename over jni for OpenCV to recreate the model instances for inference. Well, it simply as sweet as that to serve the model.
To conclude the inference part of the article, I will simply show how to write the mat object using OpenCV. At the end of the day, the values in the saved model come out as either a YAML file or XML depending on the choice most pleasant to the user.
Save the pca model object for inference on production environment.What exactly needs to be saved in the pca object are the mean and eigenvectors of the trained pca, sometimes it may be a good idea to also save the eigenvalues in case you want to construct your own eigenfaces projection, but OpenCV already implemented a pca->project instance that helps in inference and eigenfaces generation. In any case here is how to save your model:
void Facepca::savemodel(cv::PCA pcaModel, const std::stringfilename){  cv::FileStorage fs(filename,cv::FileStorage::WRITE);  fs << “mean” << pcaModel.mean;  fs << “e_vectors” << pcaModel.eigenvectors;  fs << “e_values” << pcaModel.eigenvalues;  fs.release();}
Having saved the model, the inference layer will involve loading the saved model whose values are stored in the yml file also via OpenCV and developed to form an inference engine with necessary data preprocessing modules which is finally bundled as a .so file for deployment on a linux based environment. For other platforms, the cpp codes can be compiled to produce either a dll or dylib for windows and mac respectively.
Since we are concerned about deploying the model on an android application more, the focus will be building the .so file inference engine from our saved model.
Load a PCA modelFor inference, we let OpenCV load the existing model file from the saved .yml file after which we feed the eigenvalues, eigenvectors and mean to a new PCA object which we can then call a pca->project on to create a new image’s projection.
Here is a sample code that loads a saved OpenCV FileStorage model.
cv::PCA newPcaModel;cv::PCA loadmodel(cv::PCA newPcaModel, const std::string filename){   cv::FileStorage fs(filename,cv::FileStorage::READ);   fs[“mean”] >> newPcaModel.mean ;   fs[“e_vectors”] >> newPcaModel.eigenvectors ;   fs[“e_values”] >> newPcaModel.eigenvalues ;   fs.release();   return newPcaModel;}loadmodel(newPcaModel, “path-to-saved-yml-file”);
Once the model is loaded, the newPcaModel object now contains the saved model from the existing training parameters i.e pca eigenvalues, eigenvectors and mean. Hence when a face projection is done, it is guaranteed that the data returned is relative to the training dataset.
Create new image preprocessing and prediction stage.During inference of a machine learning model, it is important that the incoming image also passes through the same preprocessing as the training dataset.
Several approaches can be used to pass image to the inference engine, it could be that the image is loaded from disk or that the image is passed as a base64 string.
In our case, the likely approach is to use a base64 string since we are also taking into consideration two factors, that a jni will be exposed and also that our final library is for an android application.
With this in mind, we then need to ensure that the library is able to retrieve an image from a base64 string and send it to OpenCV.
Decoding a base64 string in c++ is quite non-trivial, however, a reader can refer to this link on a code snippet that does this.
Once the base64 image string is decoded, we then convert the string to a vector of unsigned character (uchar) which can be thought of as Image values.
OpenCV can decode the vector of uchar into an image using the function call to cv::imdecode(vectorUchar, flag). This process returns a Mat image with which further preprocessing can be done.
The image can then pass through the preprocessing phases of;
Face extraction
Convert cropped faces to grayscale.
Image resize
Image normalization
Create data matrix
Just as described in the training part of this article above.
The last leg of inference on the new image is using the loaded pca object to create a projection of the face in the new image.
The snippet that does that will look like this:
newPcaModel->project(datamatrix.row(0));
The part of recognition or verification happens when you project with a loaded pca object on two faces and compare the projections (eigenfaces) using a distance metrics e.g euclidean distance or cosine similarity.
Package the library with an exposed jni.This part is quite straightforward, once the inference code has been properly structured, most probably within a class, then the jni body can literally call the exposed functions for the model loading, preprocessing and prediction.
However, to create a jni, it is important to understand how the link occurs with java.
The first path is to create a Java class and the functions you want to use in your java class with the input parameters.
This will be what the jni will use as the function name when creating the cpp code. The consistency between the classpath name of the methods in Java with the name in cpp is very important.
Let us assume the method we will be using for our pca feature match is named matchpcafeatures() and our java class can then look like this.
package org.example.codeclass MatchFeatures {static float matchpcafeatures(modelfilename: String, image: String, projectionToCompare: Array[Float])}
With the above java class and method, our jni header file will look something like this.
#include <jni.h>/* Header for class com_example_code_MatchFeatures */#ifndef _Included_com_example_code_MatchFeatures#define _Included_com_example_code_MatchFeatures#ifdef __cplusplusextern “C” {#endifJNIEXPORT jfloat JNICALL Java_com_example_code_MatchFeatures_matchpcafeatures(JNIEnv *, jobject, jstring, jstring, jfloatArray);#ifdef __cplusplus}#endif#endif
You don’t need to bother yet about the details of the extern C the focus is on the name of the method in the jni header file.
Further down, we will look at how to use the java code above to make an inference. Let us first develop the jni bridge for this method.
The last 3 parameters of the jni header are just the same and in the exact position as the parameters in the java method.
Therefore, we will process on those parameters since they are the key requirements from a client to our c++ inference engine.
Below shows how to connect those input parameters and return a value which the java code can take and continue its other processes.
/*** Match features of pca* */JNIEXPORT jfloat JNICALL Java_com_seamfix_qrcode_FaceFeatures_matchpcafeatures(JNIEnv * env, jobject obj, jstring pcafilename, jstring imagestring, jfloatArray projectionToCompare){  const char *pcastring_char;  pcastring_char = env->GetStringUTFChars(pcafilename, 0);  if(pcastring_char == NULL) {    return 0.0f;  }  const char *imagestring_char;  imagestring_char = env->GetStringUTFChars(imagestring, 0);  if(imagestring_char == NULL) {     return 0.0f;  }  //Get file name string as a string for cpp  std::string stdfilename(pcastring_char);  cv::PCA pca;  //Class InferencPca holds the preprocesing and inference methods  InferencePca ef;  std::vector<cv::Mat> imagevec;  //Get image as base64  cv::Mat image = ef.readBase64Image(imagestring_char);  ef.loadmodel(pca, stdfilename);  imagevec.push_back(image);  cv::Mat datamatrix = ef.createdatamatrix(imagevec);  cv::Mat projection = ef.project(datamatrix.row(0));  //Load the existing vector.  std::vector<float> initialProjectionPoints;  //Load existing features -- needed to do comparison of faces  jsize intArrayLen = env->GetArrayLength(existingfeatures);  jfloat *pointvecBody =       env->GetFloatArrayElements(existingfeatures, 0);  for (int i =0; i < intArrayLen; i++) {     initialProjectionPoints.push_back(pointvecBody[i]);  }  std::vector<float> newProjectionpoints =                     ef.matToVector(projection);  float comparisonScores =      ef.compareProjections(newProjectionpoints,                             initialProjectionPoints);  env->ReleaseFloatArrayElements(existingfeatures, pointvecBody, 0);  env->ReleaseStringUTFChars(pcafilename, pcastring_char);  return comparisonScores;}
With this, we are set to build our .so library and release it to java code to use successfully. The protocol to build the library is stated in the CMakeLists.txt.
It looks like below:
find_package(JNI REQUIRED)#Include jni directoriesinclude_directories(${JNI_INCLUDE_DIRS})file (GLOB_RECURSE SOURCE_FILE src/*.h src/*.cpp)set(LIBS ${JNI_LIBRARIES})add_library(matchprojection SHARED ${SOURCE_FILE})target_link_libraries(matchprojection ${LIBS})
Building the project should generate a lib-matchprojection.so file which can be added to your java project.
However, for android, it is a little tricky in the sense that the build tool is different rather than the official cmake build toolchain, Android has its own build tool for native cpp codes. This is called Native development kit (NDK). This is what will be used to build you c++ native codes for the generated .so to be compatible with Android.
Building .so for android using NDK will be a whole tutorial on its own hence I will be skipping that for now.
But generally, once the build is complete using an NDK, you will have the same lib-matchprojection.so which can be used in your android application.
Using the generated library within the Android application is just the same as using it in any java application.
The idea is to load the native library and then make a call with the required parameter to the methods in the initially created Classes that correspond to the native jni method.
To load the library in any java program including android, ensure the .so library is in the classpath of your program, some will put it under a folder called lib or jniLibs. With this I can use a function call to load the library as below:
System.loads(“native-lib”)
Finally, I can make a call to my methods created earlier with the parameters necessary and the native code then can execute for me.
MatchFeatures mf = new MatchFeatures();float matchscores = mf.matchpcafeatures(storedpacfilepath, imagebase64string, anotherimageprojectionarray);
If you notice carefully, the method was declared native and there is no body for the method, this is because the programmes understand that there is a native cpp method that has been defined with this classpath name.
This approach is a fundamental way to build and deploy a project that includes native codes in a java environment. Also, most complex algorithmic problems, including machine learning and core computer vision projects, can easily be reason around in cpp most because it is fast and there are production-ready libraries.
Even TensorFlow has an api for loading deep learning models in c++ as well as using its tflite models in c++.
Hence, I see this approach of building an inference engine with native code as a way to build robust production-ready engines that will leverage high precision mathematics and most especially the deployment of machine learning models to various environment and in particular android environment in an offline environment.
I work as a data scientist and machine learning engineer at Seamfix Nigeria Ltd., where I focus on ensuring data science features ends up in production.