Creating a TensorFlow CNN in C++ (Part 2) | by Benny Friedman | Towards Data Science

In the last two years, Google’s TensorFlow has been gaining popularity. It is by far the most popular deep learning framework and together with Keras it is the most dominant framework.
Now with version 2, TensorFlow includes Keras built it.
However, when it comes to the C++ API, you can’t really find much information about using it. Most of the code samples and documentation are in Python. As I had mentioned in my previous posts, I want to allow C++ users, such as myself, to use the TensorFlow C++ API, which is a low-level API, which practically means that you have to put in more work to implement it. My post is here to help you.
As I wrote in the past, I like C++. I have nothing against Python, but I would like to be able to build my models in C++ and also have the ecosystem of tools to help me debug and optimize it.
I read the documentation, and looked for code samples that will show me how to create a convolutional neural network. There are a couple of them, here and here which inspired me. That’s all!
I had to reverse engineer some of the APIs and went through a lot of trial and errors until it worked.
I hope this article will serve others who would like to start using the C++ API.
I’m using XCode on MacOS, but I tried to build the code in a way that it will also work on Windows or Linux. I did not test that, so if you have any suggestions for a change, let me know.
I am assuming you already have TensorFlow installed, and you have an XCode project ready to use it. If you need help with that, please read and follow my instructions here.
As I wrote in part 1, the goal is to implement the AlexNet scaled-down version from part 4 of the famous article series by Arden Dertat.
I’m also assuming you know the terminology of convolutional networks, if not, I suggest you read Arden’s articles.
To get the data for training, you will need to download the images from Kaggle.
Download the small version from here. There are enough images there for our purpose.
Extract the files and put them in a data folder under the project folder.
It should be organized so you have three sub-folders for train, validation and test.
Code for this article can be found here.
There’s a main.cpp which drives the execution and two class files CatDogCNN.cpp and CatDogCNN.h which are the implementation of the network itself.
mainV1.cpp is the previous article code.
In general, when writing code for TensorFlow, it is wise to separate the creation of the graph from its actual running. This is due to the simple fact that you usually create the graph once and run it many times with different inputs. Even if you do not change the variables of the graph (like weights) there is no need to recreate the graph each time you want to run it, unless it is extremely simple, and the effort of separation is useless.
CreateGraphForImage is a method that creates a graph similar to the one I showed in part 1.
It accepts a Boolean whether to unstack the image. Call it with false when you want to load only one image and true when you want to load a batch. This is since an additional dimension will be added when stacking a batch. But when you want to run the CNN with only one image, you need to have all 4 dimensions.
Note that when changing from unstack to stack you have to re-create the graph.
ReadTensorFromImageFile is in charge of running the graph created by the previous method. You feed it with a full path name of a file and get a 3 or 4 dimensional Tensor.
Code for both these methods is almost identical to the code in mainV1.cpp (article part 1).
ReadFileTensors deals with folders and the files. It accepts a base folder string and a vector of pairs of [sub-folder, label value].
If you downloaded the images from Kaggle, you may have noticed that under train for example, there are two sub-folders cats and dogs. Label each one of them with a number and pass these pairs as input.
The return is a vector of pairs, each one is a Tensor (of an image) and a label.
Here is one way of calling it:
base_folder = "/Users/bennyfriedman/Code/TF2example/TF2example/data/cats_and_dogs_small/train";vector<pair<Tensor, float>> all_files_tensors;model.ReadFileTensors(base_folder, {make_pair("cats", 0), make_pair("dogs", 1)}, all_files_tensors);
We need to open a directory, read its files and go over them one by one.
To join two path strings use io::JoinPath (include tensorflow/core/lib/io/path.h)
string folder_name = io::JoinPath(base_folder_name, “cats”);
To interact with the file system use “Env”. This is a utility class (not really documented) which gives you similar facilitation like C++17 std::filesystem.
Env* penv = Env::Default();TF_RETURN_IF_ERROR(penv->IsDirectory(folder_name));vector<string> file_names;TF_RETURN_IF_ERROR(penv->GetChildren(folder_name, &file_names));for(string file: file_names){...}
ReadFileTensors also shuffles the images in the vector, this is so we can feed different images while training (you do not want to feed all cats and then all dogs).
ReadBatches encapsulates all the logic main needs. You give it the base folder, the vectors of pair of sub-folders and labels and the size of a batch. In return you get two vectors of Tensors, one for images and one for labels. Each Tensor is a batch of images / labels according to the size you passed.
First it reads the folder and its sub-folders’ content into a vector. Next it does some calculations on how to split the batches. Then it breaks the pairs of Tensors and labels to create two Input vectors where each n-th element in the Tensor vector matches the n-th element in the labels vector.
To stack the two vectors, we create a tiny graph on the spot and run it for each iteration.
The results are added to the output vectors.
Note how Stack operation needs to get an InputList of image tensors to stack. The easiest way is to create a vector (or another container) and instantiate the InputList with it.
vector<Input> one_batch_image;one_batch_image.push_back(Input(tensor));//Add more tensorsInputList one_batch_inputs(one_batch_image);Scope root = Scope::NewRootScope();auto stacked_images = Stack(root, one_batch_inputs);ClientSession session(root);vector<Tensor> out_tensors;TF_CHECK_OK(session.Run({}, {stacked_images}, &out_tensors));//use out_tensors[0]
As written above, the tensors go in as 3 dimensional tensors and the batches that are created are 4 dimensional.
Now comes the interesting part.
We would like to create a model similar to AlexNet.
First we want to define the input to the graph: we already know we need to feed a batch of images:
input_batch_var = Placeholder(t_root.WithOpName("input"), DT_FLOAT);
A placeholder is like a function parameter. You specify its name and element type, but its shape is unknown at this time.
The variable getting the result from placeholder is a member of the CatDogCNN class type Output.
First we use it in the graph as input to the first layer and later we will need to specify it in the ClientSession Run command together with its value (the batch Tensor).
Next there are a couple of placeholders which I will explain later.
Unlike in the usual Python Keras networks, the C++ layers are a composition of Operations (Google calls them low level API). To know what layer each operation belongs to, we create a sub-scope per layer.
Scope scope_conv1 = t_root.NewSubScope("Conv1_layer");
The operations will be created with this scope object.
The first layer is a Convolution layer, so we need to define 4 parameters: filter height and width and in and out channels. In addition, we have the activation function (Relu) and a MaxPool operation that follows.
We have 4 Conv layers, so I created a function to create a generic one that includes these operations:
Input CatDogCNN::AddConvLayer(string idx, Scope scope, int in_channels, int out_channels, int filter_side, Input input){TensorShape sp({filter_side, filter_side, in_channels, out_channels});m_vars["W"+idx] = Variable(scope.WithOpName("W"), sp, DT_FLOAT);m_shapes["W"+idx] = sp;m_assigns["W"+idx+"_assign"] = Assign(scope.WithOpName("W_assign"), m_vars["W"+idx], XavierInit(scope, in_channels, out_channels, filter_side));sp = {out_channels};m_vars["B"+idx] = Variable(scope.WithOpName("B"), sp, DT_FLOAT);m_shapes["B"+idx] = sp;m_assigns["B"+idx+"_assign"] = Assign(scope.WithOpName("B_assign"), m_vars["B"+idx], Input::Initializer(0.f, sp));auto conv = Conv2D(scope.WithOpName("Conv"), input, m_vars["W"+idx], {1, 1, 1, 1}, "SAME");auto bias = BiasAdd(scope.WithOpName("Bias"), conv, m_vars["B"+idx]);auto relu = Relu(scope.WithOpName("Relu"), bias);return MaxPool(scope.WithOpName("Pool"), relu, {1, 2, 2, 1}, {1, 2, 2, 1}, "SAME");}
Let’s review the code:
AddConvLayer gets an index string (to distinguish between layers), the sub-scope, in and out channels, a filter size (we assume same height and width) and Input input.
For the first layer in channels is the number of colors the image has (3 colors) and the input is the image Tensor.
The Conv2D operation needs a Tensor variable which holds the different filters (32 in the first layer). This variable will be changed within each step when the network in training.
For that reason, there is a member map m_vars that holds these variables as well as the biases. In addition, we need to store the shapes of these variables (m_shapes map) and we also need to initialize these variable with values (similar to Python variable initializer) — these operations are stored in m_assigns. Xavier initialization is popular these days, so I’ve implemented XavierInit to do that for Conv and Dense.
After the Conv2D there is a call to BiasAdd to add the bias (which is initialized to 0).
Next is a call to Relu followed by a call to MaxPool.
Note that for MaxPool in the Python Keras version, you only need to specify the window size with its two dimensions (e.g. 2, 2). Here we need to supply two shapes of 4 dimensions (window size and strides). To halve the size of the image in each layer, you must put 1, 2, 2, 1 in both window size and strides.
Also note that in the CreateGraphForCNN function I track the sizes of the in and out channels since they have to match between the layers.
Also, since there is no way of getting the shape of the tensors coming out of operations (in is only known in runtime) we have to calculate the image size in each step.
Flattening means we will have only two dimensions, the batch and the flat data.
To do that, we need to reshape the Tensor. Since I don’t want to hard code the batch size (I do not know whether it will be batched or not), I have to know the size of the flat data. That’s why I have these calculations done through the layers until we get flat_len.
auto flat = Reshape(flatten, pool4, {-1, flat_len});
In Reshape you are allowed to pass -1 in one of the dimensions, so it will be automatically calculated. Reshape does not add data, it only changes the way it is organized in.
This layer is in charge of dropping random neurons so the network will not overfit.
We create a new tensor with same shape and fill it with either 0 or 1 randomly with a ratio according to the drop rate and multiply our data by it.
However, when you want to validate or predict, you do not want to drop out neurons.
The logical way to achieve that would be to put a condition in the graph. There are operations that support that in a very cumbersome way (Switch and Merge) but they do not support backpropagation in C++ so we can’t use them.
Instead I introduced two more input parameters to the network drop_rate and skip_drop so for example if you pass 0.4 and 0 respectively, you get 40% drop. If you pass 1 and 1 you get no drops.
Scope dropout = t_root.NewSubScope("Dropout_layer");auto rand = RandomUniform(dropout, Shape(dropout, flat), DT_FLOAT);//binary = floor(rand + (1 - drop_rate) + skip_drop)auto binary = Floor(dropout, Add(dropout, rand, Add(dropout, Sub(dropout, 1.f, drop_rate_var), skip_drop_var)));auto after_drop = Multiply(dropout.WithOpName("dropout"), Div(dropout, flat, drop_rate_var), binary)
When dropping, we increase the values of the remaining neurons by dividing by the drop rate.
Note that when skip_drop is 1, the floor will turn “binary” into all ones.
Next are three dense layers.
Dense is based on multiplying by weights and adding bias. We keep the variables in the map and do the same for shapes and assigns.
We also initialize the weights with Xavier init and the biases with 0.
Note that in the last dense layer, we need to skip the Relu activation (for that we have bActivation).
The last dense layer reduced the size to 1 (over batch size). This is called the logits.
To make logits a 0 (for cat) or 1 (for dog) we call Sigmoid and return this result out.
That’s how the network looks like when turning on TensorBoard (see part 1 for details):
I opened one of the sub-scopes so you can see the low-level operations in the fourth convolution layer.
This is encapsulated in the CreateOptimizationGraph function. It is not a separate graph, but additional nodes in the same graph (same t_root main scope).
We start with a placeholder for the labels. If the image tensors were the x, this is the y.
Next we calculate the loss. This is a classification problem, so we should use cross entropy however I’m not sure how to use the SoftmaxCrossEntropyWithLogits operation and could not find any meaningful information on it, so I had to use the regular mean square diff method.
input_labels_var = Placeholder(t_root.WithOpName("inputL"), DT_FLOAT);Scope scope_loss = t_root.NewSubScope("Loss_scope");out_loss_var = Mean(scope_loss.WithOpName("Loss"), SquaredDifference(scope_loss, out_classification, input_labels_var), {0});TF_CHECK_OK(scope_loss.status());
Next I collect all the weights and biases and send them together with the loss to the magical API call AddSymbolicGradients. This marvelous function is not an operation. It takes a graph (by its associated scope), adds all the relevant backpropagating operations and returns a vector of gradient with the same size as the number of weights and biases.
Then we use the ApplyAdam on each variable from the weights and biases with its respective gradient.
auto adam = ApplyAdam(t_root, i.second, m_var, v_var, 0.f, 0.f, learning_rate, 0.9f, 0.999f, 0.00000001f, {grad_outputs[index]});v_out_grads.push_back(adam.operation);
For some reason, m_var and v_var have to be variables (they cannot be constant).
The most important variable you pass to ApplyAdam is the learning rate. I can’t stress enough how important this value is. I started with 0.001 (from the python version) and could not stop puzzling over why the loss did not converge. Eventually I realized the optimization function jumps too long in between peaks. Only after a few days when I changed it to 0.0001 it started working. You can try different values to see how it works in your case.
You can change the other values ApplyAdam is getting as well, though I’m not sure what’s the impact of that.
Now the TensorBoard image looks “a bit” different:
I have already mentioned the need to initialize all the variables with the Assign operation.
We collect all the m_assigns elements (also the ones we added for ApplyAdam) into a vector and run them in the created session.
vector<Output> ops_to_run;for(pair<string, Output> i: m_assigns)ops_to_run.push_back(i.second);t_session = unique_ptr<ClientSession>(new ClientSession(t_root));TF_CHECK_OK(t_session->Run(ops_to_run, nullptr));
Leave the comments around the summary file writer code and use it only when you want to generate a new graph for TensorBoard visualization.
Let’s go back to main: we constructed an instance of the CatDogCNN class, created the graphs, loaded the images and ran initialization.
It is time for the main epoch / step loops.
An epoch is a run through all the training data. A step is a single run of a batch of images.
We define the number of epochs, loop through them, and then loop through the batches.
In each iteration, we call TrainCNN, we call ValidateCNN and we do some calculation to check the performance of the network.
TrainCNN gets a batch of images, a batch of labels (corresponding these images) and returns results and loss.
The function first runs the batch:
vector<Tensor> out_tensors;//Inputs: batch of images, labels, drop rate and do not skip drop.//Extract: Loss and result. Run also: Apply Adam commandsTF_CHECK_OK(t_session->Run({{input_batch_var, image_batch}, {input_labels_var, label_batch}, {drop_rate_var, 0.5f}, {skip_drop_var, 0.f}}, {out_loss_var, out_classification}, v_out_grads, &out_tensors));
Then it calculates accuracy. Notice the usage of Tensor methods to extract the data: out_tensors[0].scalar<float>()(0) to get a scalar and out_tensors[1].matrix<float>() to get a matrix (which is almost like a vector of vectors).
Validation is important due to the fact we run through the network images that network never saw. If in each epoch we get better performance we are not over fitting.
vector<Tensor> out_tensors;//Inputs: batch of images, drop rate 1 and skip drop.TF_CHECK_OK(t_session->Run({{input_batch_var, image_batch}, {drop_rate_var, 1.f}, {skip_drop_var, 1.f}}, {out_classification}, &out_tensors));
Note the differences: we skip the dropout (see above for the explanation), we do not evaluate loss and we do not run the gradients.
The output console looks like this:
Epoch 1/20:....................................................................................................Validation:..................................................Time: 74 seconds Loss: 0.243231 Results accuracy: 0.561 Validation accuracy: 0.642Epoch 2/20:....................................................................................................Validation:..................................................Time: 72 seconds Loss: 0.203312 Results accuracy: 0.6875 Validation accuracy: 0.692***Epoch 20/20:....................................................................................................Validation:..................................................Time: 73 seconds Loss: 0.041021 Results accuracy: 0.956 Validation accuracy: 0.742
The loss is constantly going down, the results accuracy is going us to 95% and the validation accuracy is going up to 75%.
Let’s test the trained network.
For that we need to recreate the load image graph so it will not unstack.
Then we call ReadFileTensors to load the images in the test sub-folder. Lastly we run a loop of a few of them to see the performance of the network.
Predict is similar to validation, though it runs a single image.
The output console looks like this:
Test number: 1 predicted: 0 actual is: 0Test number: 2 predicted: 1 actual is: 1***Test number: 20 predicted: 0 actual is: 0total successes: 16 out of 20
80% accuracy. There is some room for improvement, but we got better results than Arden got on his network.
We saw how a CNN can be implemented, trained and tested using only C++ and the TensorFlow API.
We saw how to prepare data (images for training, validation and test) and how to batch them so that we can feed these batches to the TensorFlow low level API.
We implemented gradients, and other optimization methods.
We got good results, but there is more work to be done.
Next I’m planning to do some improvements.
As you can see in the console output, I’m running on my local MacBook Pro CPU and it takes a long time to run...
I’m planning to implement freeze model function and GPU usage to speed things up.
Also, I’ll see if I can do some image augmentation to create more data for the training.