Title: Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline - PyImageSearch

URL Source: https://pyimagesearch.com/2024/01/15/adversarial-learning-with-keras-and-tensorflow-part-2-implementing-the-neural-structured-learning-nsl-framework-and-building-a-data-pipeline/

Published Time: 2024-01-15T14:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline](#h2-adversarial-learning-with-keras-and-tensorflow-part-2-implementing-the-neural-structured-learning-nsl-framework-and-building-a-data-pipeline)

*   [Adversarial Learning with NSL](#h3-adversarial-learning-with-nsl)
*   [CIFAR-10 Dataset](#h3-cifar-10-dataset)
*   [Configuring Your Development Environment](#h3-configuring-your-development-environment)
*   [Need Help Configuring Your Development Environment?](#h3-need-help-configuring-your-development-environment)
*   [Project Structure](#h3-project-structure)
*   [Configuration File](#h3-configuration-file)
*   [Building a Data Pipeline](#h3-building-a-data-pipeline)
*   [Building Callbacks](#h3-building-callbacks)

*   [Summary](#h2-summary)

*   [Citation Information](#h3-citation-information)

* * *

In this tutorial, you will learn about the TensorFlow NSL (neural structured learning) framework and how it can enhance our training pipeline and allow us to train better and more robust models. Specifically, we will take a deep dive into the salient features of the NSL framework and understand how it can be used to make models robust to adversarial attacks and perturbations with the help of adversarial training.

Furthermore, we will start building our end-to-end adversarial learning application and discuss details about the dataset we will use and implement the data pipeline using Keras and TensorFlow.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20603%20500'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2024/01/adversarial-learning-with-keras-and-tensorflow-part-2-featured.png)

This lesson is the 2nd of a 4-part series on **Adversarial Learning**:

1.  [_Adversarial Learning with Keras and TensorFlow (Part 1): Overview of Adversarial Learning_](https://pyimg.co/h35l2)
2.  [**_Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline_**](https://pyimg.co/bfzma) **(this tutorial)**
3.  _Adversarial Learning with Keras and TensorFlow (Part 3): Exploring Adversarial Attacks Using Neural Structured Learning (NSL)_
4.  _Adversarial Learning with Keras and TensorFlow (Part 4): Enhancing Adversarial Defense and Comparing Models Trained With and Without Neural Structured Learning (NSL)_

**To learn about the TensorFlow NSL framework,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

[**Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline**](#TOC-h2-adversarial-learning-with-keras-and-tensorflow-part-2-implementing-the-neural-structured-learning-nsl-framework-and-building-a-data-pipeline)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The TensorFlow NSL framework allows neural networks to learn with structured data. It provides a simple, easy-to-use API that seamlessly integrates with any computer vision project workflow and enables us to train better and more robust networks. Basically, NSL tries to augment the workflow of a network with additional structured signals, which help to regularize the training and allow us to train better and more reliable models.

Let us try to understand in detail how the NSL framework works.

**Figure 1** (_yellow_) shows a typical neural network workflow for classifying cats and dogs. Given an input dog image, the network computes features and activations and later outputs the predicted class for the input image (i.e., dog here).

Now, let us understand how the NSL framework tries to enhance this workflow.

The NSL framework takes two inputs (**Figure 1**, _red_):

*   Input images or features
*   Structured signals

The structured signal here refers to other images that are similar or related to the current input. Let us try to understand this with the dog vs. cat classification example.

As shown in **Figure 1**, the NSL framework takes the input dog image and a structure. The structure is a graph representing similarities between other images in our dataset and our input image. Notice that the graph in the figure above contains images of other bulldogs that are similar to our input dog image.

Given the input image and the structure (i.e., other related or similar images to the input), NSL tries to jointly optimize the input features and the structured signal to regularize the neural network training. Let us try to understand this with an example.

**Figure 2** shows a typical NSL training pipeline.

Suppose we have a bulldog image (i.e., the input) and other bulldog images in the dataset (i.e., structured signal) depicting bulldogs of different skin tones, sizes, poses, and backgrounds. We call them neighbor samples.

The NSL framework takes both of these and creates a batch with the original input sample and the neighbor samples (**Figure 2**). These are then passed through the neural network to get sample embeddings (corresponding to our input image) and neighbor embeddings (corresponding to our neighbor samples). Both of these embeddings are then used to compute loss with the ground-truth label, and the loss is back-propagated.

Notice that in the above example, we enabled the network to learn that all images of a bulldog with different skin tones, sizes, poses, and backgrounds belong to the same ground-truth class (i.e., bulldog). This regularizes our model and allows it to become robust to different variations of bulldog images, helping it to correctly recognize a bulldog at test time in varied poses, colors, backgrounds, etc.

Thus, the NSL framework makes use of the structured signals to help us train more robust neural network systems.

* * *

### [**Adversarial Learning with NSL**](#TOC-h3-adversarial-learning-with-nsl)

In the above example, we discussed how the NSL framework allows us to use the input image and the structured signals to learn better networks. The structured signal in the above example was a set of bulldog images (similar to the input image) with varying skin tones, sizes, poses, and backgrounds. Notice these structured examples were based on the similarity of class (bulldog) between samples and neighbors.

Naturally, the question arises: Can we have other structured signals as inputs? Of course, yes !!!!!!

The NSL framework allows us to use structures that align with our use case and the final object of the system we are building. Let us try to dig a little deeper and understand this.

In the previous tutorial of this series, we discussed how adversarial learning can make models robust to adversarial examples. Basically, this involved training or fine-tuning models on both original and adversarial examples of a class. Can the NSL framework help to achieve this goal for our use case.

Of course, yes!!!! Let us take the case where the structured signals to our NSL pipeline are adversarial examples. In this case, we have 2 inputs:

*   original input image
*   engineered adversarial example for that image (i.e., structured signal).

Here, the structured signal is related to the input sample by the following relation: It is the corresponding adversarial sample of the input.

Note, as we discussed above in **Figure 2**, this means that we can train our model on both the images and their corresponding adversarial examples. This will enable the network to learn that original images of a class and adversarially perturbed images from that class should be assigned the same label and belong to the same ground-truth class. This regularizes our model and allows it to become robust (e.g., adversarial examples and attacks), which will make its predictions more reliable.

As we have discussed earlier in this tutorial series, we will use this ability of the TensorFlow NSL framework and implement an end-to-end pipeline to show how this framework can be used to make models robust to adversarial attacks.

* * *

### [**CIFAR-1****0 Dataset**](#TOC-h3-cifar-10-dataset)

We will be using the CIFAR-10 dataset (Canadian Institute For Advanced Research) for the purpose of building our adversarial learning application. The CIFAR-10 dataset, containing 60,000 `32×32` color images from 10 classes, include the following: `airplane`, `automobile`, `bird`, `cat`, `deer`, `dog`, `frog`, `horse`, `ship`, and `truck`.

The dataset is balanced and consists of 6,000 images from each class and is divided into 50,000 training images and 10,000 test images.

* * *

### [**Configuring Your Development Environment**](#TOC-h3-configuring-your-development-environment)

To follow this guide, you need to have the TensorFlow and OpenCV library installed on your system.

Luckily, TensorFlow and OpenCV are pip-installable:

$ pip install tensorflow
$ pip install opencv-contrib-python

**If you need help configuring your development environment for OpenCV, we _highly recommend_ that you read our** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in minutes.

* * *

### [**Need Help Configuring Your Development Environment?**](#TOC-h3-need-help-configuring-your-development-environment)

[![Image 3: Need help configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join PyImageSearch University — you’ll be up and running with this tutorial in minutes.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

Need help configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_immediately_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Project Structure**](#TOC-h3-project-structure)

We first need to review our project directory structure.

Start by accessing this tutorial’s **_“Downloads”_** section to retrieve the source code and example images.

From there, take a look at the directory structure:

├── demo.py
├── inference.py
├── output
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   ├── callbacks.py
│   ├── config.py
│   ├── data.py
│   ├── model.py
│   ├── robust.py
│   └── visualization.py
└── train.py

Given above is the directory which we discussed in detail in the first part of this series.

In this tutorial, we will discuss the following files:

*   `config.py`: contains the code for our initial parameter configurations
*   `data.py`: implements the data pipeline
*   `callbacks.py`: implements the callback routine

* * *

### [**Configuration File**](#TOC-h3-configuration-file)

We start by discussing the parameter configurations we will use for this tutorial series.

We open our `config.py` file, which stores the initial parameter configurations.

\# import the necessary packages
import os

# define the data configurations
INPUT\_SHAPE = \[32, 32, 3\]
NUM\_CLASSES = 10
CLASSES = \["airplane", "automobile", "bird", "cat", "deer", "dog",
	"frog", "horse", "ship", "truck"\]

# define the training configurations
LR\_START = 1e-5
LR\_MAX = 1e-3
LR\_RAMPUP\_EPOCHS = 5
LR\_SUSTAIN\_EPOCHS = 0
LR\_STEP\_DECAY = 0.75
BATCH\_SIZE = 32
EPOCHS = 100

# define the adversary configurations
ADV\_MULTIPLIER = 0.60
ADV\_STEP\_SIZE = 0.01
ADV\_GRAD\_NORM = "infinity"
PGD\_ITERATION = 5
PGD\_EPSILON = 2.0 / 255.0
CLIP\_VALUE\_MIN = 0.0
CLIP\_VALUE\_MAX = 1.0

# define the image and label names
IMAGE\_INPUT\_NAME = "image"
LABEL\_INPUT\_NAME = "label"

# define the output paths for models and plots
OUTPUT\_PATH = "output"
DATASET\_PLOT\_PATH = os.path.join(OUTPUT\_PATH, "data\_viz.png")
BASE\_MODEL\_PATH = os.path.join(OUTPUT\_PATH, "base\_model")
FGSM\_MODEL\_PATH = os.path.join(OUTPUT\_PATH, "fgsm\_model")
PGD\_MODEL\_PATH = os.path.join(OUTPUT\_PATH, "pgd\_model")
BASE\_MODEL\_FGSM\_PLOT\_PATH = os.path.join(OUTPUT\_PATH,
	"base\_model\_fgsm\_plot.png")
BASE\_MODEL\_PGD\_PLOT\_PATH = os.path.join(OUTPUT\_PATH,
	"base\_model\_pgd\_plot.png")
FGSM\_MODEL\_FGSM\_PLOT\_PATH = os.path.join(OUTPUT\_PATH,
	"fgsm\_model\_fgsm\_plot.png")
FGSM\_MODEL\_PGD\_PLOT\_PATH = os.path.join(OUTPUT\_PATH,
	"fgsm\_model\_pgd\_plot.png")
PGD\_MODEL\_FGSM\_PLOT\_PATH = os.path.join(OUTPUT\_PATH,
	"pgd\_model\_fgsm\_plot.png")
PGD\_MODEL\_PGD\_PLOT\_PATH = os.path.join(OUTPUT\_PATH,
	"pgd\_model\_pgd\_plot.png")

First, we import the `os` module on **Line 2**, which allows us to use file-handling functionalities, as we will see later in this post.

Then, on **Lines 5-8**, we define the following data configuration parameters:

*   `INPUT_SHAPE`: the dimensions of our input images
*   `NUM_CLASSES`: number of classes in the CIFAR-10 dataset
*   `CLASSES`: list of all class names in the CIFAR-10 dataset

Now, we go ahead and define the following parameters we will use for training our models:

*   learning rate related parameters (**Lines 11-15**)
*   batch size (**Line 16**)
*   total number of epochs (**Line 17**)

Note that we will discuss more about the different learning rate parameters while discussing the training procedure in the upcoming tutorials of this series.

Next, we define the adversary parameters and configurations (**Lines 20-26**), which include the following:

*   `ADV_MULTIPLIER`: multiplier for our adversarial attack
*   `ADV_STEP_SIZE`: step size
*   `ADV_GRAD_NORM`: the type of norm our attack will be based

Furthermore, we define the number of iterations we will use for our projected gradient descent (PGD) attack and the epsilon value (i.e., `PGD_ITERATION` and `PGD_EPSILON`), which we discussed in the previous tutorial. Additionally, we define the minimum and maximum clipping values (i.e., `CLIP_VALUE_MIN` and `CLIP_VALUE_MAX`), as shown on **Lines 25 and 26**.

Now that we have defined the dataset-, training-, and adversary-related parameters, let us go ahead and define parameters for images and labels and construct the paths where the outputs will be stored.

On **Lines 29 and 30**, we define the image and label parameters (i.e., `IMAGE_INPUT_NAME` and `LABEL_INPUT_NAME`).

Finally, we define the output paths and locations where the final outputs, models, and plots will be stored. On **Line 33**, we define `OUTPUT_PATH`, which points to the folder where the outputs will be stored.

Next, on **Lines 34-37**, we construct the following paths that will be stored within the `OUTPUT_PATH` folder:

*   `DATASET_PLOT_PATH`: data visualization
*   `BASE_MODEL_PATH`: base model
*   `FGSM_MODEL_PATH`: fast gradient sign method (FGSM) model
*   `PGD_MODEL_PATH`: PGD model

Additionally, we construct the paths to store the plots corresponding to different models that we will use in this tutorial series (**Lines 38-49**):

*   `BASE_MODEL_FGSM_PLOT_PATH`
*   `BASE_MODEL_PGD_PLOT_PATH`
*   `FGSM_MODEL_FGSM_PLOT_PATH`
*   `FGSM_MODEL_PGD_PLOT_PATH`
*   `PGD_MODEL_FGSM_PLOT_PATH`
*   `PGD_MODEL_PGD_PLOT_PATH`

* * *

### [**Building a Data Pipeline**](#TOC-h3-building-a-data-pipeline)

Now that we have completed defining the initial parameter configurations, it is time to start building our data pipeline.

As we discussed above, we will use the CIFAR-10 dataset for this tutorial series. The TensorFlow API allows us to seamlessly download and prepare the CIFAR-10 dataset so that we can use it in our computer vision projects.

Let us open the `data.py` file and start our discussion.

\# import the necessary packages
from tensorflow.keras.datasets import cifar10
import tensorflow as tf

def get\_cifar\_data(trainVal=True):
	# check if we are looking to load the training and validation data
	if trainVal:
		# get the CIFAR-10 train dataset and split it into train and
		# val
		(xTrain, yTrain), (\_, \_) = cifar10.load\_data()
		(xTrain, yTrain), (xVal, yVal) = (
			(xTrain\[:40000\], yTrain\[:40000\]),
			(xTrain\[40000:\], yTrain\[40000:\]),
		)

		# build the train and val dataset from the NumPy arrays
		trainDs = tf.data.Dataset.from\_tensor\_slices((xTrain, yTrain))
		valDs = tf.data.Dataset.from\_tensor\_slices((xVal, yVal))

		# return the train and val dataset
		return (trainDs, valDs)

	# otherwise, we are looking to load the testing data
	else:
		# get the CIFAR-10 test dataset and build a TensorFlow
		# Dataset from the NumPy arrays
		(\_, \_), (xTest, yTest) = cifar10.load\_data()
		testDs = tf.data.Dataset.from\_tensor\_slices((xTest, yTest))

		# return the test dataset
		return testDs

def preprocess\_image(image, label):
	# cast the image to float32 type and scale the pixel intensity
	# to be in the \[0, 1\] range
	image = tf.cast(image, dtype=tf.float32) / 255.0

	# return the image and the label
	return (image, label)

def build\_data\_pipeline(ds, batchSize, train=True):
	# check if we are building training data input pipeline
	if train:
		# build the train data input pipeline
		ds = (ds
			.map(preprocess\_image)
			.shuffle(batchSize \* 2)
			.batch(batchSize)
			.prefetch(tf.data.AUTOTUNE)
		)

	# otherwise, we are building validation or testing data input
	# pipeline
	else:
		# build the val or test data input pipeline
		ds = (ds
			.map(preprocess\_image)
			.batch(batchSize)
			.prefetch(tf.data.AUTOTUNE)
		)

	# return the data input pipeline
	return ds

We start by importing the `cifar10` dataset from `tensorflow.keras.datasets` and the `tensorflow` library (**Lines 2 and 3**).

On **Lines 5-31**, we define the `get_cifar_data` function, which loads the CIFAR-10 dataset, splits it into train, validation, and test subsets, and returns the data subset as per our requirement. Note that it takes the `trainVal` parameter as the input argument and returns the training and validation sets when this parameter is `True` or the test data subset when this parameter is `False`.

On **Line 7**, we check if the `trainVal` parameter is `True`, and if yes, we load the `cifar10` dataset using the `cifar10.load_data()` function (**Line 10**).

Note that this function returns the train set and the test set for the CIFAR-10 dataset. On **Line 10**, we only unpack the train set and store it as `(xTrain, yTrain)`.

Next, we split our training dataset into train and validation data subsets. For this, we keep the first 40,000 examples as the train subset (i.e., `(xTrain, yTrain)`) and everything else as the val subset (i.e., `(xVal, yVal)`) (**Lines 11-14**).

Now that we have the train and val subsets as NumPy arrays, we are ready to create TensorFlow datasets. On **Lines 16 and 17**, we use the `tf.data.Dataset.from_tensor_slices` function to convert the `(xTrain, yTrain)` NumPy arrays to the training dataset (i.e., `trainDs`) and the `(xVal, yVal)` NumPy arrays to the validation dataset (i.e., `valDs`), respectively.

Finally, on **Line 21**, we return the training and validation datasets.

Next, we implement code for the case where we need the test dataset (i.e., the `trainVal` input argument is `False`). On **Line 27**, we use the `cifar10.load_data()` function as we saw above, but here we unpack the test subset as `(xTest, yTest)`.

Then, on **Line 28**, we use the `tf.data.Dataset.from_tensor_slices` function, as we saw earlier, to convert our test subset to the TensorFlow test dataset (i.e., `testDs`). Finally, we return our test dataset on **Line 31**.

Now that we have discussed our function to load the CIFAR-10 dataset, let us implement the functions which will allow us to preprocess our images and build our data pipeline.

On **Lines 33-39**, we implement the `preprocess_image` function, which takes as an argument the image and corresponding label and processes it to a format that our model expects.

Specifically, we take the input image and cast it to the `float32` data type as shown and normalize the pixel values between `0` and `1` by dividing by `255.0` (**Line 36**).

Finally, we return the preprocessed `image` and `label` tuple (**Line 39**).

Let us now implement our `build_data_pipeline` function (**Lines 41-63**), which will allow us to construct our data pipeline and get data samples for training or testing our models.

This function takes as arguments the dataset (i.e., `ds`), the batch size (i.e., `batchSize`), and a boolean (i.e., `train`), which specifies if we are building the data pipeline for the training or inference phase (**Line 41**).

In case we want to build a data pipeline for the training phase (i.e., the `train` argument is `True`), we will take the input dataset (i.e., `ds`) and use the functionalities provided by TensorFlow to process the data samples and create batches (**Lines 45-50**).

Specifically, we use the following functionalities:

*   `map`: allows us to apply the `preprocess_image` function to our input data samples (**Line 46**)
*   `shuffle`: randomly samples elements from a buffer of elements with buffer size = `batchSize * 2` (**Line 47**)
*   `batch`: takes as input the `batchSize` and allows us to sample batches of data samples (**Line 48**)
*   `prefetch`: directs TensorFlow to prepare later elements while current elements are being processed (**Line 49**)

On the other hand, in case we want to build a data pipeline for the inference phase (i.e., the `train` argument is `False`), we use the `map`, `batch`, and `prefetch` functionalities as we did above to process the test dataset (**Lines 56-60**). Note that we do not use the `shuffle` functionality since we only need to shuffle samples during training, and this is not required during the inference stage.

Finally, we return the processed data (i.e., `ds`), as shown on **Line 63**.

* * *

### [**Building Callbacks**](#TOC-h3-building-callbacks)

During the training process of our adversarial application, we will need to use callbacks, which will allow us to tune the learning rate during the training process and implement an early stopping strategy to avoid overfitting.

Let us open the `callbacks.py` file and implement our training callbacks in TensorFlow.

\# import the necessary packages
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.callbacks import LearningRateScheduler

def get\_lr\_schedule\_callback(lrStart, lrMax, lrRampupEpochs,
	lrSustainEpochs, lrStepDecay):
	# build the learning rate function
	def lrfn(epoch):
		if epoch < lrRampupEpochs:
			lr = (lrMax - lrStart) / lrRampupEpochs \* epoch + lrStart
		elif epoch < lrRampupEpochs + lrSustainEpochs:
			lr = lrMax
		else:
			lr = lrMax \* lrStepDecay\*\*(
				(epoch - lrRampupEpochs - lrSustainEpochs)//10
			)
		return lr
	
	# build the learning rate schedule callback and return it
	lrScheduleCallback = LearningRateScheduler(lrfn, verbose=True)
	return lrScheduleCallback

def get\_early\_stopping\_callback():
	# build the early stopping callback and return it
	earlyStopCallback = EarlyStopping(monitor="val\_loss", patience=5,
		restore\_best\_weights=True)
	return earlyStopCallback

We start by importing the necessary modules, such as the `EarlyStopping` and `LearningRateScheduler` callbacks from `tensorflow.keras.callbacks` (**Lines 2 and 3**).

Next, we start with the `get_lr_schedule_callback` function, which will allow us to build our callback for learning rate scheduling.

This function takes the following as input (**Lines 5 and 6**):

*   `lrStart`: starting learning rate
*   `lrMax`: maximum learning rate we will reach during training
*   `lrRampupEpochs`: number of epochs we want to ramp up the learning rate
*   `lrSustainEpochs`: number of epochs we want to sustain the learning rate
*   `lrStepDecay`: learning rate decay parameter

Now, let us discuss the logic behind the learning rate scheduler that this callback implements. We start defining the `lrfn` function (**Lines 8-17**), which takes as input the particular epoch and returns the learning rate for that particular epoch as per our scheduler scheme.

Specifically, we first check if the current `epoch` is less than the `lrRampupEpochs` (**Line 9**). If that is true, our current learning rate (i.e., `lr`) is given by the expression `lrStart + (lrMax - lrStart) / lrRampupEpochs * epoch` (**Line 10**), which takes the base learning rate as `lrStart` and divides the interval between maximum learning rate (i.e., `lrMax`) and starting learning rate (i.e., `lrStart`) by the total number of ramp-up epochs (i.e., `lrRampupEpochs`) to distribute the rise over `lrRampupEpochs` evenly.

Next, we check if the current epoch is less than the sum of ramp-up and sustain epochs (i.e., `lrRampupEpochs + lrSustainEpochs`), and if that is true, we simply sustain the learning rate (i.e., keep it constant) at `lrMax` (**Lines 11 and 12**).

Otherwise, we start gradually decaying (or reducing) the learning rate starting from `lrMax` and using the decay rate (i.e., `lrStepDecay`), as shown on **Lines 14-16**.

On **Line 17**, we return the learning rate.

Finally, on **Line 20**, we wrap our scheduling logic (i.e., `lrfn` function) using the `LearningRateScheduler` callback and return our final callback (i.e., `lrScheduleCallback`) on **Line 21**.

Now that we have completed implementing the learning rate scheduler callback, we will discuss the early stopping callback. This will enable us to stop training the model based on the validation loss to avoid overfitting.

We discuss the `get_early_stopping_callback` function, which implements the early stopping callback (**Lines 23-27**).

On **Lines 25 and 26**, we use the `EarlyStopping` callback (that we imported from `tensorflow.keras.callbacks`) and direct it to monitor (or keep track of) the validation loss. Note that we use a `patience` of `5` epochs and set the `restore_best_weights=True`, which means that the training will be automatically stopped if the validation loss does not decrease for 5 continuous epochs and the best model weights (i.e., a model with the least validation loss) will be restored.

We return our `earlyStopCallback` on **Line 27**.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC-h2-summary)
------------------------------

In this tutorial, we learned about the TensorFlow NSL framework and how it can help us train better deep learning models.

Specifically, we took an in-depth look at the NSL framework workflow. We discussed how it can be used to regularize our models and make them robust by learning from adversarial examples during training.

Furthermore, we started building our NSL-based adversarial learning application and implemented the data pipeline and callbacks.

* * *

### [**Citation Information**](#TOC-h3-citation-information)

**Chandhok, S.** “Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline,” _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, and R. Raha, eds., 2024, [https://pyimg.co/bfzma](https://pyimg.co/bfzma)

@incollection{Chandhok\_2024\_ALwKTF-pt2,
  author = {Shivam Chandhok},
  title = {Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha},
  year = {2024},
  url = {https://pyimg.co/bfzma},
}

* * *

![Image 4: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!