CIFAR-10 Image Classification in TensorFlow
By: Park Chansung
Tags: Deep Learning, Convolutional Network, Cifar 10, TensorFlow, Image Classification

Code 8 below shows how the model can be built in TensorFlow. Let’s look into the convolutional layer first. A convolutional layer can be created with either tf.nn.conv2d or tf.layers.conv2d . The latter one is more handy because it comes with a lot more optional arguments. The former choice creates the most basic convolutional layer, and you may need to add more before or after the tf.nn.conv2d . For example, activation function can be specified directly as an argument in tf.layers.conv2d , but you have to add it manually when using tf.nn.conv2d .

When building a convolutional layer, there are three things to consider. That is the stride, padding, and filter. The stride determines how much the window of filter should be moved for every convolving steps, and it is a 1-D tensor of length 4. However, technically, the official document says ‘Must have strides[0] = strides[3] = 1’. So you can only control the values of strides[1] and strides[2], but is it very common to set them equal values. [1, 1, 1, 1] and [1, 2, 2, 1] are the most common use cases. I am going to use [1, 1, 1, 1] because I want to convolve over a pixel by pixel.

When a whole convolving operation is done, the output size of the image gets smaller than the input. However, you can force it to remain the same by applying additional 0 value pixels around the images. When the padding is set as ‘SAME’, the output size of the image will remain the same as the input image. On the other hand, it will be smaller when the padding is set as ‘VALID’. I will use ‘SAME’ padding style because it is easier to manage the sizes of images in every convolutional layers.

filter can be defined with tf.Variable since it is just bunch of weight values and changes while training the network over time. The filter should be a 4-D tensor of shape [filter_height, filter_width, in_channels, out_channels]. in_channels means the number of channels the current convolving operation is applied to, and out_channels is the number of channels the current convolving operation is going to produce.

Code 8. CNN model

As mentioned tf.nn.conv2d doesn’t have an option to take activation function as an argument (while tf.layers.conv2d does), tf.nn.relu is explicitly added right after the tf.nn.conv2d operation. Then max poolings are applied by making use of tf.nn.max_pool function. The max pooling operation can be treated a special kind of conv2d operation except it doesn’t have weights. The purpose is to shrink the image by letting the strongest value survived. ksize=[1,2,2,1] and strides=[1,2,2,1] means to shrink the image into half size.

tf.contrib.layers.flatten , tf.contrib.layers.fully_connected , and tf.nn.dropout functions are intuitively understandable, and they are very ease to use. Only one important thing to remember is you don’t specify activation function at the end of the list of fully connected layers. This is going to be specified later when you define a cost function.

Hyperparameters

The hyper parameters are chosen by a dozen time of experiment. One thing to note is that learning_rate has to be defined before defining the optimizer because that is where you need to put learning rate as an constructor argument.

Code 9. hyper parameters

Cost Function & Optimizer

Finally, you’ll define cost, optimizer, and accuracy. The tf.reduce_mean takes an input tensor to reduce, and the input tensor is the results of certain loss functions between predicted results and ground truths. Because CIFAR-10 has to measure loss over 10 classes, tf.nn.softmax_cross_entropy_with_logis function is used. When training the network, what you want is minimize the cost by applying a algorithm of your choice. It could be SGD, AdamOptimizer, AdagradOptimizer, or something. You have to study how each algorithm works to choose what to use, but AdamOptimizer works find for most cases in general.

Code 10. cost function & optimizer & accuracy

Train the Neural Network

You have defined cost, optimizer and accuracy, and what they really are is..

cost

- reduce_mean => The reduced Tensor

reduce_mean => The reduced optimizer

- AdamOptimizer => An Operation that applies the specified gradients.

AdamOptimizer => An that applies the specified gradients. accuracy

- reduce_mean => The reduced Tensor

tf.Session.run method in the official document explains it runs one “step” of TensorFlow computation, by running the necessary graph fragment to execute every Operation and evaluate every Tensor in fetches, substituting the values in feed_dict for the corresponding input values. The fetches argument may be a single graph element, or an arbitrarily nested list, tuple, etc.

Here what graph element really is tf.Tensor or tf.Operation . Cost, Optimizer, and Accuracy are one of those types. It means they can be specified as part of the fetches argument. Then, you can feed some variables along the way. This is kind of handy feature of TensorFlow. Once you have constructed the graph, all you need to do is feeding data into that graph and specifying what results to retrieve.

Single Optimization

train_neural_network function runs an optimization task on the given batch of data. It will be used inside a loop over a number of epochs and batches later. In a nutshell, session.run takes care of the job. It takes the first argument as what to run and the second argument as a list of data to feed the network for retrieving results from the first argument. As mentioned previously, you want to minimize the cost by running optimizer so that has to be the first argument.

Code 11. single optimization task

Showing Stats

print_stats shows the cost and accuracy in the current training step. Similar process to train_neural_network function is applied here too. Instead of delivering optimizer to the session.run function, cost and accuracy are given. Please note that keep_prob is set to 1. Dropout rate has to be applied on training phase, or it has to be set to 1 otherwise according to the paper.

Code 12. showing stats

Train the Model over Batches

train_neural_network function runs optimization task on a given batch. Because CIFAR-10 dataset comes with 5 separate batches, and each batch contains different image data, train_neural_network should be run over every batches. This can be done with simple codes just like shown in Code 13. Code 13 runs the training over 10 epochs for every batches, and Fig 10 shows the training results.

Code 13. train over batches

Fig 10. loss and accuracy

Test the Model