Title: Transfer Learning with Keras and Deep Learning - PyImageSearch

URL Source: https://pyimagesearch.com/2019/05/20/transfer-learning-with-keras-and-deep-learning/

Published Time: 2019-05-20T14:00:07+00:00

Markdown Content:
![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%200'%3E%3C/svg%3E)

In this tutorial, you will learn how to perform transfer learning with Keras, Deep Learning, and Python on your own custom datasets.

Imagine this:

You’re just hired by Yelp to work in their computer vision department.

Yelp has just launched a new feature on its website that allows reviewers to take photos of their food/dishes and then associate them with particular items on a restaurant’s menu.

It’s a neat feature…

_**…but they are getting a lot of unwanted image spam.**_

Certain nefarious users aren’t taking photos of their dishes…instead, they are taking photos of… (well, you can probably guess).

Your task?

Figure out how to create an automated computer vision application that can distinguish between “food” and “not food”, thereby allowing Yelp to continue with their new feature launch and provide value to their users.

So, how are you going to build such an application?

**The answer lies in transfer learning via deep learning.**

Today marks the start of a brand new set of tutorials on **_transfer learning_ using Keras**. Transfer learning is the process of:

1.  Taking a network _pre-trained_ on a dataset
2.  And utilizing it to recognize image/object categories it was not trained on

Essentially, we can utilize the robust, discriminative filters learned by state-of-the-art networks on challenging datasets (such as ImageNet or COCO), and then apply these networks to recognize objects the model was _never trained_ on.

In general, there are two types of transfer learning in the context of deep learning:

1.  Transfer learning via **feature extraction**
2.  Transfer learning via **fine-tuning**

When performing _feature extraction_, we treat the pre-trained network as an arbitrary feature extractor, allowing the input image to propagate forward, stopping at pre-specified layer, and taking the _outputs_ of that layer as our features.

_Fine-tuning, on the other hand,_ requires that we update the model architecture itself by removing the previous fully-connected layer heads, providing new, freshly initialized ones, and then training the new FC layers to predict our input classes.

We’ll be covering _both_ techniques in this series here on the PyImageSearch blog, **but today we are going to focus on _feature extraction_.**

**To learn how to perform transfer learning via feature extraction with Keras, _just keep reading!_**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

_**2020-05-13 Update:** This blog post is now TensorFlow 2+ compatible!_

_**Note:** Many of the transfer learning concepts I’ll be covering in this series tutorials also appear in my book, [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/). **Inside the book, I go into much more detail (and include more of my tips, suggestions, and best practices).** If you would like more detail on transfer learning after going through this guide, definitely [take a look at my book](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)._

In the first part of this tutorial, we will review two methods of transfer learning: feature extraction and fine-tuning.

I’ll then provide a detailed discussion of how to perform transfer learning via _feature extraction_ (the primary focus of this tutorial).

From there, we’ll review Food-5k dataset, a dataset containing 5,000 images falling into two classes: “food” and “not-food”.

We’ll utilize transfer learning via feature extraction to recognize both of these classes in this tutorial.

Once we have a good handle on the dataset, we’ll start coding.

We’ll have a number of Python files to review, each accomplishing a specific step, including:

1.  Creating a configuration file.
2.  Building our dataset (i.e., putting the images in the proper directory structure).
3.  Extracting features from our input images using Keras and pre-trained CNNs.
4.  Training a Logistic Regression model on top of the extracted features.

**Parts of the code we’ll be reviewing here today will also be utilized in the rest of the transfer learning series** — if you intend on following along with the tutorials, _take the time now to ensure you understand the code_.

### Two types of transfer learning: _feature extraction_ and _fine-tuning_

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20350'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/05/transfer_learning_keras_animals.jpg)

**Figure 1:** Via “transfer learning”, we can utilize a pre-existing model such as one trained to classify dogs vs. cats. Using that pre-trained model we can break open the CNN and then apply “transfer learning” to another, completely different dataset (such as bears). We’ll learn how to apply transfer learning with Keras and deep learning in the rest of this blog post.

_**Note:** The following section has been adapted from my book, [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/). For the full set of chapters on transfer learning, please refer to the text._

Consider a traditional machine learning scenario where we are given two classification challenges.

In the first challenge, our goal is to train a [Convolutional Neural Network](https://pyimagesearch.com/2018/04/16/keras-and-convolutional-neural-networks-cnns/) to recognize dogs vs. cats in an image.

Then, in the second project, we are tasked with recognizing three separate species of bears: _grizzly bears_, _polar bears_, and _giant pandas_.

Using standard practices in machine learning/deep learning, we could treat these challenges as _two separate problems:_

*   First, we would gather a sufficient labeled dataset of dogs and cats, followed by training a model on the dataset
*   We would then repeat the process a second time, only this time, gathering images of our bear breeds, and then training a model on top of the labeled dataset.

**Transfer learning proposes a different paradigm** — _what if we could utilize an **existing** pre-trained classifier as a starting point for a new classification, object detection, or instance segmentation task?_

Using transfer learning in the context of the proposed challenges above, we would:

*   First train a Convolutional Neural Network to recognize dogs versus cats
*   Then, use the _same CNN trained on the dog and cat data_ and use it to distinguish between the bear classes, even though no bear data was mixed with the dog and cat data during the initial training

**Does this sound too good to be true?**

It’s actually not.

Deep neural networks trained on large-scale datasets such as ImageNet and COCO have proven to be _excellent_ at the task of transfer learning.

These networks learn a set of rich, discriminative features capable of recognizing 100s to 1,000s of object classes — it only makes sense that these filters can be reused for tasks other than what the CNN was originally trained on.

In general, there are two types of transfer learning when applied to deep learning for computer vision:

1.  Treating networks as arbitrary feature extractors.
2.  Removing the fully-connected layers of an existing network, placing a new set of FC layers on top of the CNN, and then _fine-tuning_ these weights (and optionally previous layers) to recognize the new object classes.

In this blog post, we’ll focus primarily on the first method of transfer learning, treating networks as feature extractors.

We’ll discuss fine-tuning networks later in this series on transfer learning with deep learning.

### Transfer learning via feature extraction

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20794'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/05/transfer_learning_keras_feature_extract.png)

**Figure 2:** _Left_: The original VGG16 network architecture that outputs probabilities for each of the 1,000 ImageNet class labels. _Right_: Removing the FC layers from VGG16 and instead of returning the final POOL layer. This output will serve as our extracted features.

_**Note:** The following section has been adapted from my book, [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/). For the full set of chapters on feature extraction, please refer to the text._

Typically, you’ll treat a Convolutional Neural Network as an end-to-end image classifier:

1.  We input an image to the network.
2.  The image forward propagates through the network.
3.  We obtain our final classification probabilities at the end of the network.

However, there is no “rule” that says we _must_ allow the image to forward propagate through the _entire_ network.

Instead, we can:

1.  Stop propagation at an arbitrary, but pre-specified layer (such as an activation or pooling layer).
2.  Extract the values from the specified layer.
3.  Treat the values as a feature vector.

For example, let’s consider the VGG16 network by Simonyan and Zisserman in **Figure 2** (_left_) at the top of this section.

Along with the layers in the network, I have also included the input and output volume shapes for each layer.

**When treating networks a feature extractor, we essentially “chop off” the network at our pre-specified layer** (typically prior to the fully-connected layers, but it really depends on your particular dataset).

If we were to stop propagation before the fully-connected layers in VGG16, the last layer in the network would become the max-pooling layer (**Figure 2**, _right_), which will have an output shape of _7 x 7 x 512_. Flattening, this volume into a feature vector we would obtain a list of _7 x 7 x 512 = 25,088_ values — **this list of numbers serves as our feature vector used to quantify the input image.**

We can then repeat the process for our entire dataset of images.

Given a total of _N_ images in our network, our dataset would now be represented as a list of _N_ vectors, each of 25,088-dim.

Once we have our feature vectors, we can train off-the-shelf machine learning models such as Linear SVM, Logistic Regression, Decision Trees, or Random Forests on top of these features to obtain a classifier that can recognize new classes of images.

**That said, the two most common machine learning models you’ll see for transfer learning via feature extraction are:**

1.  Logistic Regression
2.  Linear SVM

Why those two models?

**First, keep in mind our feature extractor is a CNN.**

CNN’s are non-linear models capable of learning non-linear features — we are assuming that the features learned by the CNN are _already_ robust and discriminative.

**The second, and perhaps arguably more important reason, is that our feature vectors tend to be very large and have high dimensionality.**

We, therefore, need a fast model that can be trained on top of the features — _**linear models tend to be very fast to train.**_

For example, our dataset of 5,000 images, each represented by a feature vector of 25,088-dim, can be trained in a few seconds using a Logistic Regression model.

To wrap up this section, **I want you to keep in mind that the CNN _itself_ is _not_ capable of recognizing these new classes.**

Instead, we are using the CNN as an _intermediary feature extractor_.

The downstream machine learning classifier will take care of learning the underlying patterns of the features extracted from the CNN.

### The Foods-5K dataset

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20264'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/05/transfer_learning_keras_food5k_dataset.jpg)

**Figure 3:** We will apply transfer learning to the [Foods-5K dataset](https://mmspg.epfl.ch/downloads/food-image-datasets/) using Python, Keras, and Deep Learning.

The dataset we’ll be using here today is the [**Food-5K dataset**](https://mmspg.epfl.ch/downloads/food-image-datasets/), curated by the [Multimedia Signal Processing Group (MSPG)](https://mmspg.epfl.ch/) of the Swiss Federal Institute of Technology.

The dataset, as the name suggests, consists of 5,000 images, belonging to two classes:

1.  Food
2.  Non-food

Our goal of is to train a classifier such that we can distinguish between these two classes.

MSPG has provided us with pre-split training, validation, and testing splits. We’ll be using these splits both in this guide on transfer learning via extraction as well as the rest of our tutorials on feature extraction.

#### Downloading the Food-5K dataset

Due to the _**unreliable**_ Food-5K dataset download method originally posted at the following FTP site with password:

*   **Host:** tremplin.epfl.ch
*   **Username:** FoodImage@grebvm2.epfl.ch
*   **Password:** Cahc1moo

… I’ve now **included the dataset directly right here:**

[Download the Food-5K dataset](https://s3-us-west-2.amazonaws.com/static.pyimagesearch.com/food-datasets/Food-5K.zip)

### Project structure

Before moving on, go ahead and:

1.  Find the _**“Downloads”**_ section of this blog post and grab the code.
2.  Download the **Food-5K dataset** using the link directly above.

You’ll then have _two_ .zip files. First, extract `transfer-learning-keras.zip`. Inside, you’ll find an empty folder named `Food-5K/`. Second, place the `Food-5K.zip` file inside that folder and extract it.

Once you’ve completed those steps, you’ll be presented with the following directory structure:

$ tree --dirsfirst --filelimit 10
.
├── Food-5K
│   ├── evaluation \[1000 entries\]
│   ├── training \[3000 entries\]
│   ├── validation \[1000 entries\]
│   └── Food-5K.zip
├── dataset
├── output
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   └── config.py
├── build\_dataset.py
├── extract\_features.py
└── train.py

7 directories, 6 files

As you can see, the `Food-5K/` contains `evaluation/` , `training/` , and `validation/` sub-directories (these will appear _**after**_ you extract `Food-5K.zip`). Each sub-directory contains 1,000 `.jpg` image files.

Our `dataset/` directory, while empty now, will soon contain the Food-5K images in a more organized form (to be discussed in the section, _“Building our dataset for feature extraction”_).

Upon successfully executing today’s Python scripts, the `output/` directory will house our extracted features (stored in three separate `.csv` files) as well as our label encoder and model (both of which are in `.cpickle` format). **These files are intentionally not included in the .zip; you must follow this tutorial to create them.**

Our Python scripts include:

*   `pyimagesearch/config.py` : Our custom configuration file will help us manage our dataset, class names, and paths. It is written in Python directly so that we can use `os.path` to build OS-specific formatted file paths directly in the script.
*   `build_dataset.py` : Using the configuration, this script will create an organized dataset on disk, making it easy to extract features from.
*   `extract_features.py` : The transfer learning magic begins here. This Python script will use a pre-trained CNN to extract raw features, storing the results in a `.csv` file. The label encoder `.cpickle` file will also be output via this script.
*   `train.py` : Our training script will train a Logistic Regression model on top of the previously computed features. We will evaluate and save the resulting model as a `.cpickle` .

The `config.py` and `build_dataset.py` scripts will be re-used in the rest of the series on transfer learning so make sure you pay close attention to them!

### Our configuration file

Let’s get started by reviewing our configuration file.

Open up `config.py` in the `pyimagesearch` submodule and insert the following code:

\# import the necessary packages
import os

# initialize the path to the \*original\* input directory of images
ORIG\_INPUT\_DATASET = "Food-5K"

# initialize the base path to the \*new\* directory that will contain
# our images after computing the training and testing split
BASE\_PATH = "dataset"

We begin with a single import. We’ll use the `os` module (**Line 2**) in this config to concatenate paths properly.

The `ORIG_INPUT_DATASET` is the path to the _original_ input dataset (i.e., where you downloaded and unarchived the Food-5K dataset).

The next path, `BASE_PATH` , will be where our dataset is organized (the result of executing `build_dataset.py` .

_**Note:** The directory structure is not especially useful for this particular post, but it **will** be later in the series once we get to fine-tuning. Again, I consider organizing datasets in this manner a “best practice” for reasons you’ll see in this series._

Let’s specify more dataset configs as well as our class labels and batch size:

\# define the names of the training, testing, and validation
# directories
TRAIN = "training"
TEST = "evaluation"
VAL = "validation"

# initialize the list of class label names
CLASSES = \["non\_food", "food"\]

# set the batch size
BATCH\_SIZE = 32

The path to output training, evaluation, and validation directories is specified on **Lines 13-15**.

The `CLASSES` are specified in list form on **Line 18**. As previously mentioned, we’ll be working with `"food"` and `"non_food"` images.

When extracting features, we’ll break our data into bite-sized chunks called batches. The `BATCH_SIZE` is specified on **Line 21**.

Finally, we can build the rest of our paths:

\# initialize the label encoder file path and the output directory to
# where the extracted features (in CSV file format) will be stored
LE\_PATH = os.path.sep.join(\["output", "le.cpickle"\])
BASE\_CSV\_PATH = "output"

# set the path to the serialized model after training
MODEL\_PATH = os.path.sep.join(\["output", "model.cpickle"\])

Our label encoder path is concatenated on **Line 25** where the result of joining the paths is `output/le.cpickle` on Linux/Mac or `output\le.cpickle` on Windows.

The extracted features will live in a CSV file in the path specified in `BASE_CSV_PATH` .

Lastly, we assemble the path to our exported model file in `MODEL_PATH` .

### Building our dataset for feature extraction

Before we can extract features from our set of input images, let’s take the time to organize our images on disk.

I prefer to have my dataset on disk organized in the format of:

`dataset_name/class_label/example_of_class_label.jpg`

Maintaining this directory structure:

*   Not only keeps our dataset organized on disk…
*   **…but _also_ enables us to utilize Keras’ `flow_from_directory` function when we get to fine-tuning later in this series of tutorials.**

Since the Food-5K dataset also provides _pre-supplied data splits,_ our final directory structure will have the form:

`dataset_name/split_name/class_label/example_of_class_label.jpg`

Let’s go ahead and build our dataset + directory structure now.

Open up the `build_dataset.py` file and insert the following code:

\# import the necessary packages
from pyimagesearch import config
from imutils import paths
import shutil
import os

# loop over the data splits
for split in (config.TRAIN, config.TEST, config.VAL):
	# grab all image paths in the current split
	print("\[INFO\] processing '{} split'...".format(split))
	p = os.path.sep.join(\[config.ORIG\_INPUT\_DATASET, split\])
	imagePaths = list(paths.list\_images(p))

Our packages are imported on **Lines 2-5**. We’ll use our `config` (**Line 2**) throughout this script to recall our settings. The other three imports — `paths` , `shutil` , and `os` — will allow us to traverse directories, create folders, and copy files.

On **Line 8** we begin looping over our training, testing, and validation splits.

**Lines 11 and 12** create a list of all `imagePaths` in the split.

From there we’ll go ahead and loop over the `imagePaths` :

	# loop over the image paths
	for imagePath in imagePaths:
		# extract class label from the filename
		filename = imagePath.split(os.path.sep)\[-1\]
		label = config.CLASSES\[int(filename.split("\_")\[0\])\]

		# construct the path to the output directory
		dirPath = os.path.sep.join(\[config.BASE\_PATH, split, label\])

		# if the output directory does not exist, create it
		if not os.path.exists(dirPath):
			os.makedirs(dirPath)

		# construct the path to the output image file and copy it
		p = os.path.sep.join(\[dirPath, filename\])
		shutil.copy2(imagePath, p)

For each `imagePath` in the split, we proceed to:

*   Extract the class `label` from the filename (**Lines 17 and 18**).
*   Construct the path to the output directory based on the `BASE_PATH` , `split` , and `label` (**Line 21**).
*   Create `dirPath` (if necessary) via **Lines 24 and 25**.
*   Copy the image into the destination path (**Lines 28 and 29**).

* * *

Now that `build_dataset.py` has been coded, use the **_“Downloads”_** section of the tutorial to download an archive of the source code.

You can then execute `build_dataset.py` using the following command:

$ python build\_dataset.py
\[INFO\] processing 'training split'...
\[INFO\] processing 'evaluation split'...
\[INFO\] processing 'validation split'...

Here you can see that our script executed successfully.

To verify your directory structure on disk, use the `ls` command:

$ ls dataset/
evaluation  training  validation

Inside the dataset directory, we have our training, evaluation, and validation splits.

And inside each of those directories, we have class labels directories:

$ ls dataset/training/
food  non\_food

### Extracting features from our dataset using Keras and pre-trained CNNs

Let’s move on to the actual feature extraction component of transfer learning.

All code used for feature extraction using a pre-trained CNN will live inside `extract_features.py` — open up that file and insert the following code:

\# import the necessary packages
from sklearn.preprocessing import LabelEncoder
from tensorflow.keras.applications import VGG16
from tensorflow.keras.applications.vgg16 import preprocess\_input
from tensorflow.keras.preprocessing.image import img\_to\_array
from tensorflow.keras.preprocessing.image import load\_img
from pyimagesearch import config
from imutils import paths
import numpy as np
import pickle
import random
import os

# load the VGG16 network and initialize the label encoder
print("\[INFO\] loading network...")
model = VGG16(weights="imagenet", include\_top=False)
le = None

On **Lines 2-12**, all the packages necessary for extracting features are imported. Most notably this includes `VGG16` .

**VGG16** is the convolutional neural network (CNN) we are using for transfer learning (**Line 3**).

On **Line 16**, we load the `model` while specifying two parameters:

*   `weights="imagenet"` : Pre-trained ImageNet weights are loaded for transfer learning.
*   `include_top=False` : We do not include the fully-connected head with the softmax classifier. In other words, we chop off the head of the network.

**With weights dialed in and by loading our model without the head, we are now ready for transfer learning.** We will use the output values of the network _directly,_ storing the results as feature vectors.

Finally, our label encoder is initialized on **Line 17**.

Let’s loop over our data splits:

\# loop over the data splits
for split in (config.TRAIN, config.TEST, config.VAL):
	# grab all image paths in the current split
	print("\[INFO\] processing '{} split'...".format(split))
	p = os.path.sep.join(\[config.BASE\_PATH, split\])
	imagePaths = list(paths.list\_images(p))

	# randomly shuffle the image paths and then extract the class
	# labels from the file paths
	random.shuffle(imagePaths)
	labels = \[p.split(os.path.sep)\[-2\] for p in imagePaths\]

	# if the label encoder is None, create it
	if le is None:
		le = LabelEncoder()
		le.fit(labels)

	# open the output CSV file for writing
	csvPath = os.path.sep.join(\[config.BASE\_CSV\_PATH,
		"{}.csv".format(split)\])
	csv = open(csvPath, "w")

Looping over each `split` (training, testing, and validation) begins on **Line 20**.

First, we grab all `imagePaths` for the `split` (**Lines 23 and 24**).

Paths are randomly shuffled via **Line 28**, and from there, our class `labels` are extracted from the paths themselves (**Line 29**).

If necessary, our label encoder is instantiated and fitted (**Lines 32-34**), ensuring we can convert the string class labels to integers.

Next, we construct the path to output our CSV files (**Lines 37-39**). We will have three CSV files — one for each data split. Each CSV will have _N_ number of rows — one for each of the images in the data split.

The next step is to loop over our `imagePaths` in `BATCH_SIZE` chunks:

	# loop over the images in batches
	for (b, i) in enumerate(range(0, len(imagePaths), config.BATCH\_SIZE)):
		# extract the batch of images and labels, then initialize the
		# list of actual images that will be passed through the network
		# for feature extraction
		print("\[INFO\] processing batch {}/{}".format(b + 1,
			int(np.ceil(len(imagePaths) / float(config.BATCH\_SIZE)))))
		batchPaths = imagePaths\[i:i + config.BATCH\_SIZE\]
		batchLabels = le.transform(labels\[i:i + config.BATCH\_SIZE\])
		batchImages = \[\]

To create our batches of `imagePaths` , we use Python’s `range` function. The function accepts three parameters: `start` , `stop` , and `step` . You can read more about `range` in [this detailed explanation](https://realpython.com/python-range/).

Our batches will step through the entire list of `imagePaths` . The `step` is our batch size (`32` unless you adjust it in the configuration settings).

On **Lines 48 and 49** the current batch of image paths and labels are extracted using array slicing. Our `batchImages` list is then initialized on **Line 50**.

Let’s go ahead and populate our `batchImages` now:

		# loop over the images and labels in the current batch
		for imagePath in batchPaths:
			# load the input image using the Keras helper utility
			# while ensuring the image is resized to 224x224 pixels
			image = load\_img(imagePath, target\_size=(224, 224))
			image = img\_to\_array(image)

			# preprocess the image by (1) expanding the dimensions and
			# (2) subtracting the mean RGB pixel intensity from the
			# ImageNet dataset
			image = np.expand\_dims(image, axis=0)
			image = preprocess\_input(image)

			# add the image to the batch
			batchImages.append(image)

Looping over `batchPaths` (**Line 53**), we will load each `image` , preprocess it, and gather it into `batchImages` .

The `image` itself is loaded on **Line 56**.

Preprocessing includes:

*   Resizing to _224×224_ pixels via the `target_size` parameter on **Line 56**.
*   Converting to array format (**Line 57**).
*   Adding a batch dimension (**Line 62**).
*   Mean subtraction (**Line 63**).

_If these preprocessing steps appear foreign, please refer to [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)._

Finally, the `image` is added to the batch via **Line 66**.

Now we will pass the batch of images through our network to **extract features:**

		# pass the images through the network and use the outputs as
		# our actual features, then reshape the features into a
		# flattened volume
		batchImages = np.vstack(batchImages)
		features = model.predict(batchImages, batch\_size=config.BATCH\_SIZE)
		features = features.reshape((features.shape\[0\], 7 \* 7 \* 512))

Our batch of images is sent through the network via **Lines 71 and 72.**

Keep in mind that we have removed the fully-connected layer head of the network. Instead, the forward propagation _stops_ at the max-pooling layer. We will treat the output of the max-pooling layer as a list of `features` , also known as a “feature vector”.

The output dimension of the max-pooling layer is _(batch\_size, 7 x 7 x 512)_. We can thus `reshape` the `features` into a NumPy array of shape `(batch_size, 7 * 7 * 512)`, _**treating the output of the CNN as a feature vector.**_

Let’s wrap up this script:

		# loop over the class labels and extracted features
		for (label, vec) in zip(batchLabels, features):
			# construct a row that exists of the class label and
			# extracted features
			vec = ",".join(\[str(v) for v in vec\])
			csv.write("{},{}\\n".format(label, vec))

	# close the CSV file
	csv.close()

# serialize the label encoder to disk
f = open(config.LE\_PATH, "wb")
f.write(pickle.dumps(le))
f.close()

Maintaining our batch efficiency, the `features` and associated class labels are written to our CSV file (**Lines 76-80**).

Inside the CSV file, the class `label` is the first field in each row (enabling us to easily extract it from the row during training). The feature `vec` follows.

Each CSV file will be closed via **Line 83**. Recall that upon completion _we will have one CSV file per data split._

Finally, we can dump the label encoder to disk (**Lines 86-88**).

* * *

Let’s go ahead and extract features from our dataset using the VGG16 network pre-trained on ImageNet.

Use the **_“Downloads”_** section of this tutorial to download the source code, and from there, execute the following command:

$ python extract\_features.py
\[INFO\] loading network...
\[INFO\] processing 'training split'...
...
\[INFO\] processing batch 92/94
\[INFO\] processing batch 93/94
\[INFO\] processing batch 94/94
\[INFO\] processing 'evaluation split'...
...
\[INFO\] processing batch 30/32
\[INFO\] processing batch 31/32
\[INFO\] processing batch 32/32
\[INFO\] processing 'validation split'...
...
\[INFO\] processing batch 30/32
\[INFO\] processing batch 31/32
\[INFO\] processing batch 32/32

On an NVIDIA K80 GPU it took 2m55s to extract features from the 5,000 images in the Food-5K dataset.

You can use a CPU instead, but it will take quite a bit longer.

### Implementing our training script

The final step for transfer learning via feature extraction is to implement a Python script that will take our extracted features from the CNN and then train a Logistic Regression model on top of the features.

**Again, keep in mind that our CNN _did not predict anything!_ Instead, the CNN was treated as an arbitrary feature extractor.**

We inputted an image to the network, it was forward propagated, and then we extracted the layer outputs from the max-pooling layer — these outputs serve as our feature vectors.

To see how we can train a model on these feature vectors, open up the `train.py` file and let’s get to work:

\# import the necessary packages
from sklearn.linear\_model import LogisticRegression
from sklearn.metrics import classification\_report
from pyimagesearch import config
import numpy as np
import pickle
import os

def load\_data\_split(splitPath):
	# initialize the data and labels
	data = \[\]
	labels = \[\]

	# loop over the rows in the data split file
	for row in open(splitPath):
		# extract the class label and features from the row
		row = row.strip().split(",")
		label = row\[0\]
		features = np.array(row\[1:\], dtype="float")

		# update the data and label lists
		data.append(features)
		labels.append(label)

	# convert the data and labels to NumPy arrays
	data = np.array(data)
	labels = np.array(labels)

	# return a tuple of the data and labels
	return (data, labels)

On **Lines 2-7** we import our required packages. Notably, we’ll use `LogisticRegression` as our **machine learning classifier**. Fewer imports are required for our training script as compared to extracting features. This is partly because the training script itself is actually simpler.

Let’s define a function named `load_data_split` on **Line 9**. This function is responsible for loading all data and labels given the path of a data split CSV file (the `splitPath` parameter).

Inside the function, we start by initializing, our `data` and `labels` lists (**Lines 11 and 12**).

From there we open the CSV and loop over all rows beginning on **Line 15**. In the loop, we:

*   Load all comma separated values from the `row` into a list (**Line 17**).
*   Grab the class `label` via **Line 18** (it is the first value in the list).
*   Extract all `features` from the row (**Line 19**). These are all values in the list _except_ the class label. The result is our **feature vector**.
*   From there, we append the feature vector and `label` to the `data` and `labels` lists respectively (**Lines 22 and 23**).

Finally, the `data` and `labels` are returned to the calling function (**Line 30**).

With the `load_data_spit` function ready to go, let’s put it to work by loading our data:

\# derive the paths to the training and testing CSV files
trainingPath = os.path.sep.join(\[config.BASE\_CSV\_PATH,
	"{}.csv".format(config.TRAIN)\])
testingPath = os.path.sep.join(\[config.BASE\_CSV\_PATH,
	"{}.csv".format(config.TEST)\])

# load the data from disk
print("\[INFO\] loading data...")
(trainX, trainY) = load\_data\_split(trainingPath)
(testX, testY) = load\_data\_split(testingPath)

# load the label encoder from disk
le = pickle.loads(open(config.LE\_PATH, "rb").read())

**Lines 33-41** load our training and testing feature data from disk. We’re using our function from the previous code block to handle the loading process.

**Line 44** loads our label encoder.

With our data in memory, we’re now ready to **train our machine learning classifier:**

\# train the model
print("\[INFO\] training model...")
model = LogisticRegression(solver="lbfgs", multi\_class="auto",
	max\_iter=150)
model.fit(trainX, trainY)

# evaluate the model
print("\[INFO\] evaluating...")
preds = model.predict(testX)
print(classification\_report(testY, preds, target\_names=le.classes\_))

# serialize the model to disk
print("\[INFO\] saving model...")
f = open(config.MODEL\_PATH, "wb")
f.write(pickle.dumps(model))
f.close()

**Lines 48-50** are responsible for initializing and training our **Logistic Regression** `model` .

_**Note:** To learn more about Logistic Regression and other machine learning algorithms in detail, be sure to refer to [PyImageSearch Gurus](https://pyimagesearch.com/pyimagesearch-gurus/), my flagship computer vision course and community._

**Lines 54 and 55** facilitate evaluating the `model` on the testing set and printing classification statistics in the terminal.

Finally, the `model` is output in Python’s pickle format (**Lines 59-61**).

That’s a wrap for our training script! As you’ve learned, writing code for training a Logistic Regression model on top of feature data is very straightforward. In the next section, we will run the training script.

If you are wondering how we would handle so much feature data that it can’t fit into memory all at once, _**stay tuned for next week’s tutorial.**_

_**Note:** This tutorial is long enough as is, so I haven’t covered how to tune the hyperparameters to the Logistic Regression model, something I_ definitely _recommend doing to ensure to obtain the highest accuracy possible. If you’re interested in learning more about transfer learning, and how to tune hyperparameters, during feature extraction, be sure to refer to [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/) where I cover the techniques in more detail._

### Training a model on the extracted features

At this point, we are ready to perform the final step on transfer learning via feature extraction with Keras.

Let’s briefly review what we have done so far:

1.  Downloaded the Food-5K dataset (5,000 images belonging to two classes, “food” and “non-food”, respectively).
2.  Restructured the original directory structure of the dataset in a format more suitable for transfer learning (in particular, fine-tuning which we’ll be covering later in this series).
3.  Extracted features from the images using VGG16 pre-trained on ImageNet.

**And now, we’re going to train a Logistic Regression model on top of these extracted features.**

Again, keep in mind that VGG16 was not trained to recognize the “food” versus “non-food” classes. Instead, it was trained to recognize 1,000 ImageNet classes.

But, by leveraging:

1.  Feature extraction with VGG16
2.  And applying a Logistic Regression classifier on top of those extracted features

**…we will be able to recognize the new classes, even though VGG16 was never trained to recognize them!**

Go ahead and use the **_“Downloads”_** section of this tutorial to download the source code to this guide.

From there, open up a terminal and execute the following command:

$ python train.py
\[INFO\] loading data...
\[INFO\] training model...
\[INFO\] evaluating...
              precision    recall  f1-score   support

        food       0.99      0.98      0.98       500
    non\_food       0.98      0.99      0.99       500

    accuracy                           0.98      1000
   macro avg       0.99      0.98      0.98      1000
weighted avg       0.99      0.98      0.98      1000

\[INFO\] saving model...

Training on my machine took only **27 seconds**, and as you can see from our output, **we are obtaining 98-99% accuracy on the testing set!**

When should I use transfer learning and feature extraction?
-----------------------------------------------------------

Transfer learning via feature extraction is often one of the _easiest methods_ to obtain a baseline accuracy in your own projects.

Whenever I am confronted with a new deep learning project, I often throw feature extraction with Keras at it just to see what happens:

*   In some cases, the accuracy is sufficient.
*   In others, it requires me to tune the hyperparameters to my Logistic Regression model or try another pre-trained CNN.
*   And in other situations, I need to explore fine-tuning or even training from scratch with a custom CNN architecture.

Regardless, in the _best case_ transfer learning via feature extraction gives me good accuracy and the project can be completed.

And in the _worst case_ I’ll gain a baseline to beat with my future experiments.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

Today marked the start of our series on _transfer learning_ with Keras and Deep Learning.

The two primary forms of feature extraction via deep learning are:

1.  Feature extraction
2.  Fine-tuning

The focus of today’s tutorial was on _feature extraction_, the process of treating a pre-trained network as an arbitrary feature extractor.

The steps to perform transfer learning via feature extraction include:

1.  Starting with a pre-trained network (typically on a dataset such as ImageNet or COCO; large enough for the model to learn discriminative filters).
2.  Allowing an input image to forward propagate to an arbitrary (pre-specified) layer.
3.  Taking the output of that layer and treating it as a f_eature vector._
4.  Training a “standard” machine learning model on the dataset of extracted features.

The benefit of performing transfer learning via feature extraction is that we do not need to train (or re-train) our neural network.

**Instead, the network serves as a black box feature extractor.**

Those extracted features, which are assumed to be non-linear in nature (since they were extracted from a CNN), are then passed into a linear model for classification.

If you’re interested in learning more about transfer learning, feature extraction, and fine-tuning, be sure to refer to my book, [_Deep Learning for Computer Vision with Python_](https://pyimagesearch.com/deep-learning-computer-vision-python-book/) where I cover the topic in more detail.

I hope you enjoyed today’s post! Stay tuned for next week when we discuss how to work with feature extraction when our dataset is too large too fit into memory.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _just enter your email address in the form below!_**

![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!