Title: Fine-tuning ResNet with Keras, TensorFlow, and Deep Learning - PyImageSearch

URL Source: https://pyimagesearch.com/2020/04/27/fine-tuning-resnet-with-keras-tensorflow-and-deep-learning/

Published Time: 2020-04-27T14:00:00+00:00

Markdown Content:
![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20500'%3E%3C/svg%3E)

In this tutorial, you will learn how to fine-tune ResNet using Keras, TensorFlow, and Deep Learning.

A couple of months ago, I posted on Twitter asking my followers for help creating a dataset of **camouflage vs. noncamouflage clothes:**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20525%20464'%3E%3C/svg%3E)

**Figure 1:** My request for a camouflage image dataset to use in my fine-tuning ResNet with Keras, TensorFlow, and deep learning blog post.

This dataset was to be used on a special project that Victor Gevers, an esteemed ethical hacker from the GDI.Foundation, and I were working on (more on that in two weeks, when I’ll reveal the details on what we’ve built).

Two PyImageSearch readers, Julia Riede and Nitin Rai, not only stepped up to the plate to help out **_but hit a home run!_**

Both of them spent a couple of days downloading images for each class, organizing the files, and then uploading them so Victor and I could train a model on them — **thank you so much, Julia and Nitin; we couldn’t have done it without you!**

A few days after I started working with the camouflage vs. noncamouflage dataset, I received an email from PyImageSearch reader Lucas:

> Hi Adrian, I’m big fan of the PyImageSearch blog. It’s helped me tremendously with my undergrad project.
> 
> I have a question for you:
> 
> **Do you have any tutorials on how to fine-tune ResNet?**
> 
> I’ve been going through your archives and it seems like you’ve covered fine-tuning other architectures (ex. VGGNet) but I couldn’t find anything on ResNet. I’ve been trying to fine-tune ResNet with Keras/TensorFlow for the past few days and I just keep running into errors.
> 
> If you can help me out I would appreciate it.

I was already planning on fine-tuning a model on top of the camouflage vs. noncamouflage clothes dataset, so helping Lucas seemed like a natural fit.

**Inside the remainder of this this tutorial you will:**

1.  Discover the seminal ResNet architecture
2.  Learn how to fine-tune it using Keras and TensorFlow
3.  Fine-tune ResNet for camouflage vs. noncamouflage clothes detection

And in two weeks, I’ll show you the practical, real-world use case that Victor and I applied camouflage detection to — it’s a great story, and you won’t want to miss it!

**To learn how to fine-tune ResNet with Keras and TensorFlow, _just keep reading!_**

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this tutorial, you will learn about the ResNet architecture, including how we can fine-tune ResNet using Keras and TensorFlow.

From there, we’ll discuss our camouflage clothing vs. normal clothing image dataset in detail.

We’ll then review our project directory structure and proceed to:

1.  Implement our configuration file
2.  Create a Python script to build/organize our image dataset
3.  Implement a second Python script used to fine-tune ResNet with Keras and TensorFlow
4.  Execute the training script and fine-tune ResNet on our dataset

Let’s get started!

### What is ResNet?

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20289'%3E%3C/svg%3E)

**Figure 2:** Variations of He et al.’s residual module in their 2016 research led to a new variation of ResNet. In this blog post we fine-tune ResNet with Keras, TensorFlow, and deep learning to build a camouflage clothing classifier. (image source: Figure 4 of [He et al. 2016](https://arxiv.org/pdf/1603.05027.pdf))

ResNet was first introduced by He et al. in their seminal 2015 paper, _[Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)_ — that paper has been cited an astonishing 43,064 times!

A follow-up paper in 2016, _[Identity Mappings in Deep Residual Networks](https://arxiv.org/abs/1603.05027)_, performed a series of ablation experiments, playing with the inclusion, removal, and ordering of various components in the residual module, ultimately resulting in a variation of ResNet that:

1.  Is easier to train
2.  Is more tolerant of hyperparameters, including regularization and initial learning rate
3.  Generalizes better

ResNet is arguably _the most important_ network architecture since:

*   AlexNet — which reignited researcher interest in deep neural networks back in 2012
*   VGGNet — which demonstrated how deeper neural networks could be trained successfully using only _3×3_ convolutions (2014)
*   GoogLeNet — which introduced the inception module/micro-architecture (2014)

In fact, the techniques that ResNet employs have been successfully applied to noncomputer vision tasks, including audio classification and Natural Language Processing (NLP)!

### How does ResNet work?

_**Note:** The following section was adapted from Chapter 12 of my book, [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/) (Practitioner Bundle)._

The original residual module introduced by He et al. relies on the concept of _identify mappings_, the process of taking the _original input_ to the module and adding it to the _output_ of a series of operations:

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20450%20439'%3E%3C/svg%3E)

**Figure 3:** ResNet is based on a “residual module” as pictured. In this deep learning blog post, we fine-tune ResNet with Keras and TensorFlow.

At the _top_ of the module, we accept an _input_ to the module (i.e., the previous layer in the network). The _right_ branch is a “linear shortcut” — it connects the input to an addition operation at the bottom of the model. Then, on the _left_ branch of the residual module, we apply a series of convolutions (both of which are _3×3_), activations, and batch normalizations. This is a standard pattern to follow when constructing Convolutional Neural Networks.

But what makes ResNet interesting is that He et al. suggested _adding_ the _original input_ to the _output_ of the CONV, RELU, and BN layers.

**We call this addition an _identity mapping_ since the input (the identity) is added to the output of a series of operations.**

It’s also way the term **residual** is used — the “residual” input is added to the output of a series of layer operations. The connection between the input and addition node is called the **shortcut.**

While traditional neural networks can be seen as learning a function _y = f(x)_, a residual layer attempts to approximate _y_ via _f(x) + id(x) = f(x) + x_ where _id(x)_ is the identity function.

These residual layers _start_ at the identity function and _evolve_ to become more complex as the network learns. This type of residual learning framework allows us to train networks that are _substantially deeper_ than previously proposed architectures.

**Furthermore, since the input is included in every residual module, it turns out the network can learn _faster_ and with _larger learning rates._**

In the original 2015 paper, He et al. also included an extension to the original residual module called **bottlenecks:**

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20450%20398'%3E%3C/svg%3E)

**Figure 4:** He et al.’s “bottlenecks” extension to the residual module. We use TensorFlow and Keras to build a deep learning camouflage classifier based on ResNet in this tutorial.

Here we can see that the same identity mapping is taking place, only now the CONV layers in the left branch of the residual module have been updated:

1.  We are utilizing three CONV layers rather than just two
2.  The first and last CONV layers are _1×1_ convolutions
3.  The number of filters learned in the first two CONV layers are 1/4 the number of filters learned in the final CONV

This variation of the residual module serves as a form of _dimensionality reduction_, thereby reducing the total number of parameters in the network (and doing so without sacrificing accuracy). This form of dimensionality reduction is called the **bottleneck.**

He et al.’s 2016 publication on _Identity Mappings in Deep Residual Networks_ performed a series of ablation studies, playing with the inclusion, removal, and ordering of various components in the residual module, ultimately resulting in the concept of **pre-activation:**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20525%20587'%3E%3C/svg%3E)

**Figure 5:** Comparing the ResNet residual module with bottleneck vs. a pre-activation residual module. Be sure to read this tutorial to learn how to apply fine-tuning with deep learning and ResNet using TensorFlow/Keras.

Without going into too much detail, the pre-activation residual module rearranges the order in which convolution, batch normalization, and activation are performed.

The original residual module (with bottleneck) accepts an input (i.e., a RELU activation map) and then applies a series of `(CONV => BN => RELU) * 2 => CONV => BN` before adding this output to the original input and applying a final RELU activation.

Their 2016 study demonstrated that instead, applying a series of `(BN => RELU => CONV) * 3` led to higher accuracy models that were easier to train.

We call this method of layer ordering _pre-activation_ as our RELUs and batch normalizations are placed _before_ the convolutions, which is in contrast to the typical approach of applying RELUs and batch normalizations _after_ the convolutions.

For a more complete review of ResNet, including how to implement it from scratch using Keras/TensorFlow, be sure to refer to my book, _[Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)_.

### How can we fine-tune it with Keras and TensorFlow?

In order to fine-tune ResNet with Keras and TensorFlow, we need to load ResNet from disk using the _pre-trained_ ImageNet weights but _leaving off_ the fully-connected layer head.

We can do so using the following code:

\>>> baseModel = ResNet50(weights="imagenet", include\_top=False,
	input\_tensor=Input(shape=(224, 224, 3)))

Inspecting the `` `baseModel.summary()` ``, you’ll see the following:

...
conv5\_block3\_3\_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5\_block3\_2\_relu\[0\]\[0\]        
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv5\_block3\_3\_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5\_block3\_3\_conv\[0\]\[0\]        
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv5\_block3\_add (Add)          (None, 7, 7, 2048)   0           conv5\_block2\_out\[0\]\[0\]           
                                                                 conv5\_block3\_3\_bn\[0\]\[0\]          
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv5\_block3\_out (Activation)   (None, 7, 7, 2048)   0           conv5\_block3\_add\[0\]\[0\]           
==================================================================================================

Here, we can observe that the final layer in the ResNet architecture (again, without the fully-connected layer head) is an Activation layer that is _7 x 7 x 2048._

We can construct a _new, freshly initialized_ layer head by accepting the `baseModel.output` and then applying a _7×7_ average pooling, followed by our fully-connected layers:

headModel = baseModel.output
headModel = AveragePooling2D(pool\_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(256, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(len(config.CLASSES), activation="softmax")(headModel)

With the `headModel` constructed, we simply need to append it to the body of the ResNet model:

model = Model(inputs=baseModel.input, outputs=headModel)

Now, if we take a look at the `model.summary()`, we can conclude that we have successfully added a new fully-connected layer head to ResNet, making the architecture suitable for fine-tuning:

conv5\_block3\_3\_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5\_block3\_2\_relu\[0\]\[0\]        
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv5\_block3\_3\_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5\_block3\_3\_conv\[0\]\[0\]        
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv5\_block3\_add (Add)          (None, 7, 7, 2048)   0           conv5\_block2\_out\[0\]\[0\]           
                                                                 conv5\_block3\_3\_bn\[0\]\[0\]          
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
conv5\_block3\_out (Activation)   (None, 7, 7, 2048)   0           conv5\_block3\_add\[0\]\[0\]           
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
average\_pooling2d (AveragePooli (None, 1, 1, 2048)   0           conv5\_block3\_out\[0\]\[0\]           
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
flatten (Flatten)               (None, 2048)         0           average\_pooling2d\[0\]\[0\]          
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
dense (Dense)                   (None, 256)          524544      flatten\[0\]\[0\]                    
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
dropout (Dropout)               (None, 256)          0           dense\[0\]\[0\]                      
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
dense\_1 (Dense)                 (None, 2)            514         dropout\[0\]\[0\]                    
==================================================================================================

In the remainder of this tutorial, I will provide you with a fully working example of fine-tuning ResNet using Keras and TensorFlow.

### Our camouflage vs. normal clothing dataset

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20381'%3E%3C/svg%3E)

**Figure 6:** A camouflage clothing dataset will help us to build a camo vs. normal clothes detector. We’ll fine-tune a ResNet50 CNN using Keras and TensorFlow to build a camouflage clothing classifier in today’s tutorial.

In this tutorial, we will be training a **camouflage clothes vs. normal clothes** detector.

I’ll be discussing exactly _why_ we’re building a camouflage clothes detector in two weeks, but for the time being, let this serve as a standalone example of how to fine-tune ResNet with Keras and TensorFlow.

The dataset we’re using here was curated by PyImageSearch readers, [Julia Riede](https://github.com/jriede/ml-data/tree/master/camouflage-clothing) and [Nitin Rai](https://twitter.com/imneonizer/status/1225749289491554305).

The dataset consists of two classes, each with an equal number of images:

*   `camouflage_clothes`: 7,949 images
*   `normal_clothes`: 7,949 images

A sample of the images for each class can be seen in **Figure 6**.

In the remainder of this tutorial, you’ll learn how to fine-tune ResNet to predict both of these classes — the knowledge that you gain will enable you to fine-tune ResNet on your own datasets as well.

### Downloading our camouflage vs. normal clothing dataset

![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20489'%3E%3C/svg%3E)

**Figure 7:** We will download a normal vs. camouflage clothing dataset from Kaggle. We’ll then fine-tune ResNet on the deep learning dataset using Keras and TensorFlow.

The camouflage clothes vs. normal clothes dataset can be downloaded directly from Kaggle:

[https://www.kaggle.com/imneonizer/normal-vs-camouflage-clothes](https://www.kaggle.com/imneonizer/normal-vs-camouflage-clothes)

Simply click the _“Download”_ button (**Figure 7**) to download a `.zip` archive of the dataset.

### Project structure

Be sure to grab and unzip the code from the **_“Downloads”_** section of this blog post. Let’s take a moment to inspect the organizational structure of our project:

$ tree --dirsfirst --filelimit 10
.
├── 8k\_normal\_vs\_camouflage\_clothes\_images
│   ├── camouflage\_clothes \[7949 entries\]
│   └── normal\_clothes \[7949 entries\]
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   └── config.py
├── build\_dataset.py
├── camo\_detector.model
├── normal-vs-camouflage-clothes.zip
├── plot.png
└── train\_camo\_detector.py

4 directories, 7 files

As you can see, I’ve placed the dataset (`normal-vs-camouflage-clothes.zip`) in the root directory of our project and extracted the files. The images therein now reside in the `8k_normal_vs_camouflage_clothes_images` directory.

Today’s `pyimagesearch` module comes with a single Python configuration file (`config.py`) that houses our important paths and variables. We’ll review this file in the next section.

Our Python driver scripts consist of:

*   `build_dataset.py`: Splits our data into training, testing, and validation subdirectories
*   `train_camo_detector.py`: Trains a camouflage classifier with Python, TensorFlow/Keras, and fine-tuning

### Our configuration file

Before we can (1) build our camouflage vs. noncamouflage image dataset and (2) fine-tune ResNet on our image dataset, let’s first create a simple configuration file to store all our important image paths and variables.

Open up the `config.py` file in your project, and insert the following code:

\# import the necessary packages
import os

# initialize the path to the \*original\* input directory of images
ORIG\_INPUT\_DATASET = "8k\_normal\_vs\_camouflage\_clothes\_images"

# initialize the base path to the \*new\* directory that will contain
# our images after computing the training and testing split
BASE\_PATH = "camo\_not\_camo"

# derive the training, validation, and testing directories
TRAIN\_PATH = os.path.sep.join(\[BASE\_PATH, "training"\])
VAL\_PATH = os.path.sep.join(\[BASE\_PATH, "validation"\])
TEST\_PATH = os.path.sep.join(\[BASE\_PATH, "testing"\])

The `os` module import allows us to build dynamic paths directly in our configuration file.

Our existing input dataset path should be placed on **Line 5** (the Kaggle dataset you should have downloaded by this point).

The path to our new dataset directory that will contain our training, testing, and validation splits is shown on **Line 9**. This path will be _created_ by the `build_dataset.py` script.

Three subdirectories per class (we have two classes) will also be created (**Lines 12-14**) — the paths to our training, validation, and testing dataset splits. Each will be populated with a subset of the images from our dataset.

Next, we’ll define our split percentages and classes:

\# define the amount of data that will be used training
TRAIN\_SPLIT = 0.75

# the amount of validation data will be a percentage of the
# \*training\* data
VAL\_SPLIT = 0.1

# define the names of the classes
CLASSES = \["camouflage\_clothes", "normal\_clothes"\]

Training data will be represented by 75% of all the data available (**Line 17**), 10% of which will be marked for validation (**Line 21**).

Our camouflage and normal clothes classes are defined on **Line 24**.

We’ll wrap up with a few hyperparameters and our output model path:

\# initialize the initial learning rate, batch size, and number of
# epochs to train for
INIT\_LR = 1e-4
BS = 32
NUM\_EPOCHS = 20

# define the path to the serialized output model after training
MODEL\_PATH = "camo\_detector.model"

The initial learning rate, batch size, and number of epochs to train for are set on **Lines 28-30**.

The path to the output serialized ResNet-based camouflage classification model after fine-tuning will be stored at the path defined on **Line 33**.

### Implementing our camouflage dataset builder script

With our configuration file implemented, let’s move on to creating our dataset builder, which will:

1.  Split our dataset into training, validation, and testing sets, respectively
2.  Organize our images on disk so we can use Keras’ `ImageDataGenerator` class and associated `flow_from_directory` function to easily fine-tune ResNet

Open up `build_dataset.py`, and let’s get started:

\# import the necessary packages
from pyimagesearch import config
from imutils import paths
import random
import shutil
import os

We begin by importing our `config` from the previous section along with the `paths` module, which will help us to find the image files on disk. Three modules built into Python will be used for shuffling paths and creating directories/subdirectories.

Let’s go ahead and grab the paths to all original images in our dataset:

\# grab the paths to all input images in the original input directory
# and shuffle them
imagePaths = list(paths.list\_images(config.ORIG\_INPUT\_DATASET))
random.seed(42)
random.shuffle(imagePaths)

# compute the training and testing split
i = int(len(imagePaths) \* config.TRAIN\_SPLIT)
trainPaths = imagePaths\[:i\]
testPaths = imagePaths\[i:\]

# we'll be using part of the training data for validation
i = int(len(trainPaths) \* config.VAL\_SPLIT)
valPaths = trainPaths\[:i\]
trainPaths = trainPaths\[i:\]

# define the datasets that we'll be building
datasets = \[
	("training", trainPaths, config.TRAIN\_PATH),
	("validation", valPaths, config.VAL\_PATH),
	("testing", testPaths, config.TEST\_PATH)
\]

We grab our `imagePaths` and randomly shuffle them with a `seed` for reproducibility (**Line 15-17**).

From there, we calculate the list index for our training/testing split (currently set to 75% by in our configuration file) via **Line 15**. The list index, `i`, is used to form our `trainPaths` and `testPaths`.

The next split index is calculated from the number of `trainPaths` — 10% of the paths are marked as `valPaths` for validation (**Lines 20-22**).

**Lines 25-29** define the dataset splits we’ll be building in the remainder of this script. Let’s proceed:

\# loop over the datasets
for (dType, imagePaths, baseOutput) in datasets:
	# show which data split we are creating
	print("\[INFO\] building '{}' split".format(dType))

	# if the output base output directory does not exist, create it
	if not os.path.exists(baseOutput):
		print("\[INFO\] 'creating {}' directory".format(baseOutput))
		os.makedirs(baseOutput)

	# loop over the input image paths
	for inputPath in imagePaths:
		# extract the filename of the input image along with its
		# corresponding class label
		filename = inputPath.split(os.path.sep)\[-1\]
		label = inputPath.split(os.path.sep)\[-2\]

		# build the path to the label directory
		labelPath = os.path.sep.join(\[baseOutput, label\])

		# if the label output directory does not exist, create it
		if not os.path.exists(labelPath):
			print("\[INFO\] 'creating {}' directory".format(labelPath))
			os.makedirs(labelPath)

		# construct the path to the destination image and then copy
		# the image itself
		p = os.path.sep.join(\[labelPath, filename\])
		shutil.copy2(inputPath, p)

This last block of code handles copying images from their original location into their destination path; directories and subdirectories are created in the process. Let’s review in more detail:

*   We loop over each of the `datasets`, creating the directory if it doesn’t exist (**Lines 32-39**)
*   For each of our `imagePaths`, we proceed to:
    *   Extract the `filename` and class `label` (**Lines 45 and 46**)
    *   Build the path to the label directory (**Line 49**) and create the subdirectory, if required (**Lines 52-54**)
    *   Copy the image from the source directory into its destination (**Lines 58 and 59**)

In the next section, we’ll build our dataset accordingly.

### Building the camouflage image dataset

Let’s now build and organize our image camouflage dataset.

Make sure you have:

1.  Used the **_“Downloads”_** section of this tutorial to download the source code
2.  Followed the “_Downloading our camouflage vs. normal clothing dataset_” section above to download the dataset

From there, open a terminal, and execute the following command:

$ python build\_dataset.py
\[INFO\] building 'training' split
\[INFO\] 'creating camo\_not\_camo/training' directory
\[INFO\] 'creating camo\_not\_camo/training/normal\_clothes' directory
\[INFO\] 'creating camo\_not\_camo/training/camouflage\_clothes' directory
\[INFO\] building 'validation' split
\[INFO\] 'creating camo\_not\_camo/validation' directory
\[INFO\] 'creating camo\_not\_camo/validation/camouflage\_clothes' directory
\[INFO\] 'creating camo\_not\_camo/validation/normal\_clothes' directory
\[INFO\] building 'testing' split
\[INFO\] 'creating camo\_not\_camo/testing' directory
\[INFO\] 'creating camo\_not\_camo/testing/normal\_clothes' directory
\[INFO\] 'creating camo\_not\_camo/testing/camouflage\_clothes' directory

You can then use the `tree` command to inspect `camo_not_camo` directory to validate that each of the training, testing, and validation splits was created:

$ tree camo\_not\_camo --filelimit 20
camo\_not\_camo
├── testing
│   ├── camouflage\_clothes \[2007 entries\]
│   └── normal\_clothes \[1968 entries\]
├── training
│   ├── camouflage\_clothes \[5339 entries\]
│   └── normal\_clothes \[5392 entries\]
└── validation
    ├── camouflage\_clothes \[603 entries\]
    └── normal\_clothes \[589 entries\]

9 directories, 0 files

### Implementing our ResNet fine-tuning script with Keras and TensorFlow

With our dataset created and properly organized on disk, let’s learn how we can fine-tune ResNet using Keras and TensorFlow.

Open the `train_camo_detector.py` file, and insert the following code:

\# set the matplotlib backend so figures can be saved in the background
import matplotlib
matplotlib.use("Agg")

# import the necessary packages
from pyimagesearch import config
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import AveragePooling2D
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.applications import ResNet50
from sklearn.metrics import classification\_report
from imutils import paths
import matplotlib.pyplot as plt
import numpy as np
import argparse

Our most notable imports include the `ResNet50` CNN architecture and Keras `layers` for building the head of our model for fine-tuning. Settings for the entire script are housed in the `config`.

Additionally, we’ll use the `ImageDataGenerator` class for [data augmentation](https://pyimagesearch.com/2019/07/08/keras-imagedatagenerator-and-data-augmentation/) and scikit-learn’s `classification_report` to print statistics in our terminal. We also need `matplotlib` for plotting and `paths` which assists with finding image files on disk.

With our imports ready to go, let’s go ahead and parse command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-p", "--plot", type=str, default="plot.png",
	help="path to output loss/accuracy plot")
args = vars(ap.parse\_args())

# determine the total number of image paths in training, validation,
# and testing directories
totalTrain = len(list(paths.list\_images(config.TRAIN\_PATH)))
totalVal = len(list(paths.list\_images(config.VAL\_PATH)))
totalTest = len(list(paths.list\_images(config.TEST\_PATH)))

We have a single command line argument `--plot`, the path to an image file that will have our accuracy/loss training curves. Our other configurations are in the Python configuration file we reviewed previously.

**Lines 30-32** determine the total number of training, validation, and testing images, respectively.

Next, we’ll prepare for [data augmentation](https://pyimagesearch.com/2019/07/08/keras-imagedatagenerator-and-data-augmentation/):

\# initialize the training training data augmentation object
trainAug = ImageDataGenerator(
	rotation\_range=25,
	zoom\_range=0.1,
	width\_shift\_range=0.1,
	height\_shift\_range=0.1,
	shear\_range=0.2,
	horizontal\_flip=True,
	fill\_mode="nearest")

# initialize the validation/testing data augmentation object (which
# we'll be adding mean subtraction to)
valAug = ImageDataGenerator()

# define the ImageNet mean subtraction (in RGB order) and set the
# the mean subtraction value for each of the data augmentation
# objects
mean = np.array(\[123.68, 116.779, 103.939\], dtype="float32")
trainAug.mean = mean
valAug.mean = mean

Data augmentation allows for training time mutations of our images including random rotations, zooms, shifts, shears, flips, and mean subtraction. **Lines 35-42** initialize our training data augmentation object with a selection of these parameters. Similarly, **Line 46** initializes the validation/testing data augmentation object (it will only be used for mean subtraction).

Both of our data augmentation objects are set up to perform mean subtraction on-the-fly (**Lines 51-53**).

We’ll now instantiate three Python generators from our data augmentation objects:

\# initialize the training generator
trainGen = trainAug.flow\_from\_directory(
	config.TRAIN\_PATH,
	class\_mode="categorical",
	target\_size=(224, 224),
	color\_mode="rgb",
	shuffle=True,
	batch\_size=config.BS)

# initialize the validation generator
valGen = valAug.flow\_from\_directory(
	config.VAL\_PATH,
	class\_mode="categorical",
	target\_size=(224, 224),
	color\_mode="rgb",
	shuffle=False,
	batch\_size=config.BS)

# initialize the testing generator
testGen = valAug.flow\_from\_directory(
	config.TEST\_PATH,
	class\_mode="categorical",
	target\_size=(224, 224),
	color\_mode="rgb",
	shuffle=False,
	batch\_size=config.BS)

Here, we’ve initialized training, validation, and testing image data generators. Notice that both the `valGen` and `testGen` are derived from the same `valAug` object, which performs mean subtraction.

Let’s load our ResNet50 classification model and prepare it for [fine-tuning](https://pyimagesearch.com/2019/06/03/fine-tuning-with-keras-and-deep-learning/):

\# load the ResNet-50 network, ensuring the head FC layer sets are left
# off
print("\[INFO\] preparing model...")
baseModel = ResNet50(weights="imagenet", include\_top=False,
	input\_tensor=Input(shape=(224, 224, 3)))

# construct the head of the model that will be placed on top of the
# the base model
headModel = baseModel.output
headModel = AveragePooling2D(pool\_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(256, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(len(config.CLASSES), activation="softmax")(headModel)

# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)

# loop over all layers in the base model and freeze them so they will
# \*not\* be updated during the training process
for layer in baseModel.layers:
	layer.trainable = False

The process of fine-tuning allows us to reuse the filters learned during a previous training exercise. In our case, we load `ResNet50` pre-trained on the [ImageNet dataset](http://www.image-net.org/), leaving off the fully-connected (FC) head (**Lines 85 and 86**).

We then construct a new FC `headModel` (**Lines 90-95**) and append it to the `baseModel` (**Line 99**).

The final step for fine-tuning is to ensure that the weights of the base of our CNN are frozen (**Lines 103 and 104**) — we only want to train (i.e., fine-tune) the head of the network.

If you need to brush up on the concept of fine-tuning, please refer to my [fine-tuning articles](https://pyimagesearch.com/tag/fine-tuning/), in particular _[Fine-tuning with Keras and Deep Learning](https://pyimagesearch.com/2019/06/03/fine-tuning-with-keras-and-deep-learning/)._

We’re now ready to fine-tune our ResNet-based camouflage detector with TensorFlow, Keras, and deep learning:

\# compile the model
opt = Adam(lr=config.INIT\_LR, decay=config.INIT\_LR / config.NUM\_EPOCHS)
model.compile(loss="binary\_crossentropy", optimizer=opt,
	metrics=\["accuracy"\])

# train the model
print("\[INFO\] training model...")
H = model.fit\_generator(
	trainGen,
	steps\_per\_epoch=totalTrain // config.BS,
	validation\_data=valGen,
	validation\_steps=totalVal // config.BS,
	epochs=config.NUM\_EPOCHS)

First, we compile our model with [learning rate decay](https://pyimagesearch.com/2019/07/22/keras-learning-rate-schedules-and-decay/) and the [Adam optimizer](https://pyimagesearch.com/2019/10/07/is-rectified-adam-actually-better-than-adam/) using `"binary_crossentropy"` loss, since this is a two-class problem (**Lines 107-109**). If you are training with more than two classes of data, be sure to set your `loss` to `"categorical_crossentropy"`.

**Lines 113-118** then train our model using our training and validation data generators.

Upon the completion of training, we’ll evaluate our model on the testing set:

\# reset the testing generator and then use our trained model to
# make predictions on the data
print("\[INFO\] evaluating network...")
testGen.reset()
predIdxs = model.predict\_generator(testGen,
	steps=(totalTest // config.BS) + 1)

# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)

# show a nicely formatted classification report
print(classification\_report(testGen.classes, predIdxs,
	target\_names=testGen.class\_indices.keys()))

# serialize the model to disk
print("\[INFO\] saving model...")
model.save(config.MODEL\_PATH, save\_format="h5")

**Lines 123-133** make predictions on the testing set and generate and `print` a classification report in your terminal for inspection.

Then, we serialize our TensorFlow/Keras camouflage classifier to disk (**Line 137**).

Finally, plot the training accuracy/loss history via `matplotlib`:

\# plot the training loss and accuracy
N = config.NUM\_EPOCHS
plt.style.use("ggplot")
plt.figure()
plt.plot(np.arange(0, N), H.history\["loss"\], label="train\_loss")
plt.plot(np.arange(0, N), H.history\["val\_loss"\], label="val\_loss")
plt.plot(np.arange(0, N), H.history\["accuracy"\], label="train\_acc")
plt.plot(np.arange(0, N), H.history\["val\_accuracy"\], label="val\_acc")
plt.title("Training Loss and Accuracy on Dataset")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend(loc="lower left")
plt.savefig(args\["plot"\])

Once the plot is generated, **Line 151** saves it to disk in the location specified by our `--plot` command line argument.

### Fine-tuning ResNet with Keras and TensorFlow results

We are now ready to fine-tune ResNet with Keras and TensorFlow.

Make sure you have:

*   Used the **_“Downloads”_** section of this tutorial to download the source code
*   Followed the “_Downloading our camouflage vs. normal clothing dataset”_ section above to download the dataset
*   Executed the `build_dataset.py` script to organize the dataset into the project directory structure for training

From there, open up a terminal, and run the `train_camo_detector.py` script:

$ python train\_camo\_detector.py
Found 10731 images belonging to 2 classes.
Found 1192 images belonging to 2 classes.
Found 3975 images belonging to 2 classes.
\[INFO\] preparing model...
\[INFO\] training model...
Epoch 1/20
335/335 \[==============================\] - 311s 929ms/step - loss: 0.1736 - accuracy: 0.9326 - val\_loss: 0.1050 - val\_accuracy: 0.9671
Epoch 2/20
335/335 \[==============================\] - 305s 912ms/step - loss: 0.0997 - accuracy: 0.9632 - val\_loss: 0.1028 - val\_accuracy: 0.9586
Epoch 3/20
335/335 \[==============================\] - 305s 910ms/step - loss: 0.0729 - accuracy: 0.9753 - val\_loss: 0.0951 - val\_accuracy: 0.9730
...
Epoch 18/20
335/335 \[==============================\] - 298s 890ms/step - loss: 0.0336 - accuracy: 0.9878 - val\_loss: 0.0854 - val\_accuracy: 0.9696
Epoch 19/20
335/335 \[==============================\] - 298s 891ms/step - loss: 0.0296 - accuracy: 0.9896 - val\_loss: 0.0850 - val\_accuracy: 0.9679
Epoch 20/20
335/335 \[==============================\] - 299s 894ms/step - loss: 0.0275 - accuracy: 0.9905 - val\_loss: 0.0955 - val\_accuracy: 0.9679
\[INFO\] evaluating network...
                   precision    recall  f1-score   support

    normal\_clothes       0.95      0.99      0.97      2007
camouflage\_clothes       0.99      0.95      0.97      1968

          accuracy                           0.97      3975
         macro avg       0.97      0.97      0.97      3975
      weighted avg       0.97      0.97      0.97      3975

\[INFO\] saving model...

Here, you can see that we are obtaining **~97% accuracy** on our normal clothes vs. camouflage clothes detector.

Our training plot is shown below:

![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20640%20480'%3E%3C/svg%3E)

**Figure 8:** Training plot of our accuracy/loss curves when fine-tuning ResNet on a camouflage deep learning dataset using Keras and TensorFlow.

Our training loss decreases at a much sharper rate than our validation loss; furthermore, it appears that validation loss may be rising toward the end of training, indicating that the model _may_ be overfitting.

Future experiments should look into applying additional regularization to the model as well as gathering additional training data.

**In two weeks, I’ll show you how to take this fine-tuned ResNet model and use it in a practical, real-world application!**

Stay tuned for the post; you won’t want to miss it!

### Credits

This tutorial would not be possible without:

*   Victor Gevers of the [GDI.Foundation](http://gdi.foundation/), who brought this project to my attention
*   [Nitin Rai](https://twitter.com/imneonizer/status/1225749289491554305) who curated the normal clothes vs. camouflage clothes and posted the dataset on Kaggle
*   [Julia Riede](https://github.com/jriede) who curated a variation of the dataset

Additionally, I’d like to credit [Han et al.](https://www.researchgate.net/publication/322621180_Deep_neural_networks_show_an_equivalent_and_often_superior_performance_to_dermatologists_in_onychomycosis_diagnosis_Automatic_construction_of_onychomycosis_datasets_by_region-based_convolutional_deep_) for the ResNet-152 visualization used in the header of this image.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In this tutorial you learned how to fine-tune ResNet with Keras and TensorFlow.

Fine-tuning is the process of:

1.  Taking a pre-trained deep neural network (in this case, ResNet)
2.  Removing the fully-connected layer head from the network
3.  Placing a new, freshly initialized layer head on top of the body of the network
4.  Optionally freezing the weights for the layers in the body
5.  Training the model, using the pre-trained weights as a starting point to help the model learn faster

Using fine-tune we can obtain a higher accuracy model, typically with much less effort, data, and training time.

As a practical application, we fine-tuned ResNet on a dataset of camouflage vs. noncamouflage clothes images.

This dataset was curated and put together for us by PyImageSearch readers, [Julia Riede](https://github.com/jriede/ml-data/tree/master/camouflage-clothing) and [Nitin Rai](https://twitter.com/imneonizer/status/1225749289491554305) — without them, this tutorial, as well as the project Victor Gevers and I were working on, would not have been possible! **Please thank both Julia and Nitin if you see them online.**

In two weeks, I’ll go into the details of the project that Victor Gevers and I have been working on, which wraps a nice a little bow on the following topics that we’ve recently covered on PyImageSearch:

*   Face detection
*   Age detection
*   Removing duplicates from a deep learning dataset
*   Fine-tuning a model for camouflage clothes vs. noncamouflage clothes detection

It’s a great post with _very real_ applications to make the world a better place with computer vision and deep learning — you won’t want to miss it!

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!