Title: The four key ingredients when training any neural network - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/06/the-four-key-ingredients-when-training-any-neural-network/

Published Time: 2021-05-06T14:00:00+00:00

Markdown Content:
You might have started to notice a pattern in our Python code examples when training neural networks. There are four main ingredients you need to put together in your own neural network and deep learning algorithm: a _dataset_, a _model/architecture_, a _loss function_, and an _optimization method_. We’ll review each of these ingredients below.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

### **Dataset**

The dataset is the first ingredient in training a neural network — the data itself along with the problem we are trying to solve define our end goals. For example, are we using neural networks to perform a regression analysis to predict the value of homes in a specific suburb in 20 years? Is our goal to perform unsupervised learning, such as dimensionality reduction? Or are we trying to perform classification?

In the context of this book, we’re strictly focusing on _image classification_; however, the combination of your dataset and the problem you are trying to solve influences your choice in loss function, network architecture, and optimization method used to train the model. Usually, we have little choice in our dataset (unless you’re working on a hobby project) — we are given a dataset with some expectation on what the results from our project should be. It is then up to us to train a machine learning model on the dataset to perform well on the given task.

### **Loss Function**

Given our dataset and target goal, we need to define a loss function that aligns with the problem we are trying to solve. In nearly all image classification problems using deep learning, we’ll be using cross-entropy loss. For _\>2_ classes we call this _categorical cross-entropy_. For two class problems, we call the loss _binary cross-entropy_.

### **Model/Architecture**

Your network architecture can be considered the first actual “choice” you have to make as an ingredient. Your dataset is likely chosen for you (or at least you’ve _decided_ that you want to work with a given dataset). And if you’re performing classification, you’ll in all likelihood be using cross-entropy as your loss function.

However, your network architecture can vary dramatically, especially when with which optimization method you choose to train your network. After taking the time to explore your dataset and look at:

1.  How many data points you have.
2.  The number of classes.
3.  How similar/dissimilar the classes are.
4.  The intra-class variance.

You should start to develop a “feel” for a network architecture you are going to use. This takes practice as deep learning is part science, part art.

Keep in mind that the number of layers and nodes in your network architecture (along with any type of regularization) is likely to change as you perform more and more experiments. The more results you gather, the better equipped you are to make informed decisions on which techniques to try next.

### **Optimization Method**

The final ingredient is to define an optimization method. [_Stochastic Gradient Descent_](https://pyimagesearch.com/2016/10/17/stochastic-gradient-descent-sgd-with-python/) is used quite often. Other optimization methods exist, including RMSprop ([Hinton, _Neural Networks for Machine Learning_](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)), Adagrad ([Duchi, Hazan, and Singer, 2011](http://dl.acm.org/citation.cfm?id=1953048.2021068)), Adadelta ([Zeiler, 2012](http://arxiv.org/abs/1212.5701)), and Adam ([Kingma and Ba, 2014](http://arxiv.org/abs/1412.6980)); however, these are more advanced optimization methods.

Even despite all these newer optimization methods, SGD is _still_ the workhorse of deep learning — most neural networks are trained via SGD, including the networks obtaining state-of-the-art accuracy on challenging image datasets such as ImageNet.

When training deep learning networks, especially when you’re first getting started and learning the ropes, _SGD should be your optimizer of choice_. You then need to set a proper learning rate and regularization strength, the total number of epochs the network should be trained for, and whether or not momentum (and if so, which value) or Nesterov acceleration should be used. Take the time to experiment with SGD _as much as you possibly can_ and _become comfortable with tuning the_ _parameters_.

Becoming familiar with a given optimization algorithm is similar to mastering how to drive a car — you drive your own car better than other people’s cars because you’ve spent so much time driving it; you understand your car and its intricacies. Oftentimes, a given optimizer is chosen to train a network on a dataset _not_ because the optimizer itself is better, but because the _driver_ (i.e., deep learning practitioner) is _more familiar with the optimizer_ and understands the “art” behind tuning its respective parameters.

Keep in mind that obtaining a reasonably performing neural network on even a small/medium dataset can take 10’s to 100’s of experiments even for advanced deep learning users — don’t be discouraged when your network isn’t performing extremely well right out of the gate. Becoming proficient in deep learning will require an investment of your time and _many_ experiments — but it will be worth it once you master how these ingredients come together.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20240%20240'%3E%3C/svg%3E)

#### **About the Author**

Hi there, I’m Adrian Rosebrock, PhD. All too often I see developers, students, and researchers wasting their time, studying the wrong things, and generally struggling to get started with Computer Vision, Deep Learning, and OpenCV. I created this website to show you what I believe is the best possible way to get your start.