Title: Auto-Keras and AutoML: A Getting Started Guide - PyImageSearch

URL Source: https://pyimagesearch.com/2019/01/07/auto-keras-and-automl-a-getting-started-guide/

Published Time: 2019-01-07T15:00:23+00:00

Markdown Content:
[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20294'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/01/autokeras_header.jpg)

In this tutorial, you will learn how to use Auto-Keras, an open source alternative to Google’s AutoML, for automated machine learning and deep learning.

When training a neural network on a dataset there are two primary objectives a deep learning practitioner is trying to optimize and balance:

1.  **Defining a neural network architecture** that lends itself to the nature of the dataset
2.  **Tuning a set of hyperparameters over many experiments** that will lead to a model with high accuracy and ability to generalize to data outside the training and testing sets. Typical hyperparameters that need to be tuned include the optimizer algorithm (SGD, Adam, etc.), learning rate and learning rate scheduling, and regularization, to name a few

Depending on the dataset and problem, it can take a deep learning expert upwards of _tens_ to _hundreds_ of experiments to find a balance between neural network architecture and hyperparameters.

These experiments can add up to _hundreds_ to _thousands_ of hours in GPU compute time.

And that’s just for _experts_ — what about non-deep learning experts?

Enter Auto-Keras and AutoML:

The end goal of both Auto-Keras and AutoML is to reduce the barrier to entry to performing machine learning and deep learning through the use of _automated_ Neural Architecture Search (NAS) algorithms.

**Auto-Keras and AutoML enable non-deep learning experts to train their own models with _minimal_ domain knowledge of either deep learning or their actual data.**

Using AutoML and Auto-Keras, a programmer with minimal machine learning expertise can apply these algorithms to achieve state-of-the-art performance with very little effort.

Sound too good to be true?

Well, maybe — but you’ll need to read the rest of the post first to find out why.

**To learn more about AutoML (and how to _automatically_ train and tune a neural network with Auto-Keras), _just keep reading!_**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this blog post, we’ll discuss **Automated Machine Learning (AutoML)** and **Neural Architecture Search (NAS)**, the algorithm that makes AutoML possible when applied to neural networks and deep learning.

We’ll also briefly discuss **Google’s AutoML**, a suite of tools and libraries allowing programmers with limited machine learning expertise to train high accuracy models on their own data.

Of course, Google’s AutoML is a proprietary algorithm (it’s a bit on the expensive side of as well).

An alternative to AutoML is the **open source Auto-Keras**, built around Keras and PyTorch.

I’ll then show you **how to _automatically_ train a network using Auto-Keras** as well as to evaluate it.

### What is Automated Machine Learning (AutoML)?

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20184'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/01/autokeras_automl.png)

**Figure 1:** Auto-Keras is an alternative to [Google’s AutoML](https://cloud.google.com/automl/). These software projects can help you train models automatically with little intervention. They are great options for novice deep learning practitioners or to obtain a baseline to beat later on.

Outside of unsupervised learning (automatically learning patterns from unlabeled data), automated machine learning for non-experts is considered the “holy grail” of machine learning.

Imagine the ability to _automatically_ create a machine learning model via:

1.  Installing a library/using a web interface
2.  Pointing the library/interface to your data
3.  Automatically training a model on the data _without_ having to tune the parameters/requiring an intimate understanding of the algorithms powering it

Some companies are trying to create such solutions — a big one is [Google’s AutoML](https://cloud.google.com/automl/).

Google AutoML enables developers and engineers with very limited machine learning experience to automatically train neural networks on their own datasets.

Under the hood Google’s AutoML algorithms are iterative:

1.  Training a network on a training set
2.  Evaluating the network on a testing set
3.  Modifying the neural network architecture
4.  Tuning hyperparameters
5.  Repeating the process

The programmer or engineer using AutoML doesn’t need to define their own neural network architecture or tune the hyperparameters — AutoML is doing that for them automatically.

### Neural Architecture Search (NAS) makes AutoML possible

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20357'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/01/autokeras_cells.png)

**Figure 2:** Neural Architecture Search (NAS) produced a model summarized by these graphs when searching for the best CNN architecture for CIFAR-10 (source: Figure 4 of [Zoph et al.](https://arxiv.org/pdf/1707.07012.pdf))

Both Google’s AutoML and Auto-Keras are powered by an algorithm called Neural Architecture Search (NAS).

Given your input dataset, a Neural Architecture Search algorithm will _automatically_ search for the most optimal architecture and corresponding parameters.

Neural Architecture Search essentially replaces the deep learning engineer/practitioner with a set of algorithms that automatically tunes the model!

In the context of computer vision and image recognition, a Neural Architecture Search algorithm will:

1.  Accept an input training dataset
2.  Optimize and find _architectural building blocks_ called “cells” — these cells are automatically learned and may look similar to inception, residual, or squeeze/fire micro-architectures
3.  Continually train and search the “NAS search space” for more optimized cells

If the user of the AutoML system is an experienced deep learning practitioner then they may decide to:

1.  Run the NAS on a significantly smaller subset of the training dataset
2.  Find an optimal set of architectural building blocks/cells
3.  Take these cells and manually define a deeper version of the network found during the architecture search
4.  Train the network on the full training set using their own expertise and best practices

Such an approach is a _hybrid_ between a fully automated machine learning solution and one that requires an expert deep learning practitioner — often this approach will lead to better accuracy than what the NAS finds on its own.

I would recommend reading [_Neural Architecture Search with Reinforcement Learning_](https://arxiv.org/abs/1611.01578) (Zoph and Le, 2016) along with [_Learning Transferable Architectures for Scalable Image Recognition_](https://arxiv.org/abs/1707.07012) (Zoph et al., 2017) for more details on how these algorithms work.

### Auto-Keras: An open source alternative to Google’s AutoML

[![Image 5](https://b2633864.smushcdn.com/2633864/wp-content/uploads/2019/01/autokeras_logo.png?lossy=2&strip=1&webp=1)](https://pyimagesearch.com/wp-content/uploads/2019/01/autokeras_logo.png)

**Figure 3:** The [Auto-Keras](https://autokeras.com/) package was developed by the DATA Lab team at Texas A&M University. Auto-Keras is an open source alternative to [Google’s AutoML](https://cloud.google.com/automl/).

The [Auto-Keras](https://autokeras.com/) package, developed by the [DATA Lab team at Texas A&M University](http://people.tamu.edu/~guangzhou92/Data_Lab/), is an alternative to Google’s AutoML.

Auto-Keras also utilizes the Neural Architecture Search but applies “network morphism” (keeping network functionality while changing the architecture) along with Bayesian optimization to guide the network morphism for more efficient neural network search.

You can find the full details of the Auto-Keras framework in Jin et al.’s 2018 publication, [_Auto-Keras: Efficient Neural Architecture Search with Network Morphism_](https://arxiv.org/abs/1806.10282).

### Project structure

Go ahead and grab the zip from the _**“Downloads”**_ section of today’s blog post.

From there you should unzip the file and navigate into it using your terminal.

Let’s inspect today’s project with the `tree` command:

$ tree --dirsfirst
.
├── output
│   ├── 14400.txt
│   ├── 28800.txt
│   ├── 3600.txt
│   ├── 43200.txt
│   ├── 7200.txt
│   └── 86400.txt
└── train\_auto\_keras.py

1 directory, 7 files

Today we’re going to be reviewing a single Python script: `train_auto_keras.py` .

Since there will be a lot of output printed to the screen, I’ve opted to save our classification reports (generated with the help of scikit-learn’s `classification_report` tool) as text files to disk. Inspecting the `output/` folder above you can see a handful of the reports that have been generated. Go ahead and print one to your terminal (`cat output/14400.txt` ) to see what it looks like.

### Installing Auto-Keras

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20400%20267'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/01/autokeras_libraries.png)

**Figure 4:** The Auto-Keras package depends upon Python 3.6, TensorFlow, and Keras.

As the [Auto-Keras GitHub repository states](https://github.com/jhfjhfj1/autokeras), Auto-Keras is in a “pre-release” state — it is _not_ an official release.

**Secondly, Auto-Keras _requires_ Python 3.6 and is _only_ compatible with Python 3.6.**

If you are using **_any other version of Python other than 3.6_** you will _**not** be able to utilize the Auto-Keras package_.

To check your Python version just use the following command:

$ python --version

Provided you have Python 3.6 you can install Auto-Keras using pip:

$ pip install tensorflow # or tensorflow-gpu
$ pip install keras
$ pip install autokeras

If you have any issues installing or utilizing Auto-Keras make sure you post on the official [Auto-Keras GitHub Issues page](https://github.com/jhfjhfj1/autokeras/issues) where the authors will be able to help you.

### Implementing our training script with Auto-Keras

Let’s go ahead and implement our training script using Auto-Keras. Open up the `train_auto_keras.py` file and insert the following code:

\# import the necessary packages
from sklearn.metrics import classification\_report
from keras.datasets import cifar10
import autokeras as ak
import os

def main():
	# initialize the output directory
	OUTPUT\_PATH = "output"

To begin, we import necessary packages on **Lines 2-5**:

*   As previously mentioned, we’ll be using scikit-learn’s `classification_report` to calculate statistics which we’ll save in our output files.
*   We’re going to use the [CIFAR-10 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html), conveniently built into `keras.datasets` .
*   Then comes our most notable import, `autokeras` , which I’ve imported as `ak` for shorthand.
*   The `os` module is required as we’ll accommodate path separators on various operating systems when building output file paths.

Let’s define the `main` function for our script on **Line 7**. We’re required to wrap our code in a `main` function due to how Auto-Keras and TensorFlow handle threading. See [this GitHub issue thread](https://github.com/jhfjhfj1/autokeras/issues/311) for more details.

Our base `OUTPUT_PATH` is defined on **Line 9**.

Now let’s initialize a list of training times for Auto-Keras:

	# initialize the list of training times that we'll allow
	# Auto-Keras to train for
	TRAINING\_TIMES = \[
		60 \* 60,		# 1 hour
		60 \* 60 \* 2,	# 2 hours
		60 \* 60 \* 4,	# 4 hours
		60 \* 60 \* 8,	# 8 hours
		60 \* 60 \* 12,	# 12 hours
		60 \* 60 \* 24,	# 24 hours
	\]

**Lines 13-20** define a set of `TRAINING_TIMES` , including `[1, 2, 4, 8, 12, 24]` hours. We’ll be exploring the effect of longer training times on accuracy using Auto-Keras today.

Let’s load the CIFAR-10 dataset and initialize class names:

	# load the training and testing data, then scale it into the
	# range \[0, 1\]
	print("\[INFO\] loading CIFAR-10 data...")
	((trainX, trainY), (testX, testY)) = cifar10.load\_data()
	trainX = trainX.astype("float") / 255.0
	testX = testX.astype("float") / 255.0

	# initialize the label names for the CIFAR-10 dataset
	labelNames = \["airplane", "automobile", "bird", "cat", "deer",
		"dog", "frog", "horse", "ship", "truck"\]

Our CIFAR-10 data is loaded and stored into training/testing splits on **Line 25**.

Subsequently, we’ll scale this data to the range of _\[0, 1\]_ (**Lines 26 and 27**).

Our class `labelNames` are initialized on **Lines 30 and 31**. These 10 classes are included in CIFAR-10. Take note that order is important here.

And now let’s begin looping over our `TRAINING_TIMES` , each time putting Auto-Keras to use:

	# loop over the number of seconds to allow the current Auto-Keras
	# model to train for
	for seconds in TRAINING\_TIMES:
		# train our Auto-Keras model
		print("\[INFO\] training model for {} seconds max...".format(
			seconds))
		model = ak.ImageClassifier(verbose=True)
		model.fit(trainX, trainY, time\_limit=seconds)
		model.final\_fit(trainX, trainY, testX, testY, retrain=True)

		# evaluate the Auto-Keras model
		score = model.evaluate(testX, testY)
		predictions = model.predict(testX)
		report = classification\_report(testY, predictions,
			target\_names=labelNames)

		# write the report to disk
		p = os.path.sep.join(OUTPUT\_PATH, "{}.txt".format(seconds))
		f = open(p, "w")
		f.write(report)
		f.write("\\nscore: {}".format(score))
		f.close()

The code block above is the heart of today’s script. On **Line 35** we’ve defined a loop over each of our `TRAINING_TIMES` , where we:

*   Initialize our `model` (`ak.ImageClassifier` ) and allow training to start (**Lines 39 and 40**). Notice that we didn’t instantiate an object for a particular CNN class as we have in previous tutorials such as [this one](https://pyimagesearch.com/2018/04/16/keras-and-convolutional-neural-networks-cnns/). Nor do we need to tune hyperparameters as we typically do. Auto-Keras handles all of this for us and provides a report of its findings.
*   Once the time limit has been reached, take the best `model` and parameters Auto-Keras has found + re-train the model (**Line 41**).
*   Evaluate and construct the classification `report` (**Lines 44-47**).
*   Write the classification `report` along with the accuracy `score` to disk so we can evaluate the effect of longer training times (**Lines 50-54**).

We’ll repeat this process for each of our `TRAINING_TIMES` .

Finally, we’ll check for and start the `main` thread of execution:

\# if this is the main thread of execution then start the process (our
# code must be wrapped like this to avoid threading issues with
# TensorFlow)
if \_\_name\_\_ == "\_\_main\_\_":
	main()

Here we’ve checked to ensure that this is the `main` thread of execution and then the `main` function.

Just 60 lines later, we’re done writing our Auto-Keras with CIFAR-10 example script. But we’re not done yet…

### Training a neural network with Auto-Keras

Let’s go ahead and train our neural network using Auto-Keras.

Make sure you use the _**“Downloads”**_ section of this tutorial to download the source code.

From there, open up a terminal, navigate to where you downloaded the source code, and execute the following command:

$ python train\_auto\_keras.py
\[INFO\] training model for 3600 seconds max...   
Preprocessing the images.
Preprocessing finished.

Initializing search.
Initialization finished.


+----------------------------------------------+
|               Training model 0               |
+----------------------------------------------+
Using TensorFlow backend.

No loss decrease after 5 epochs.


Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           0            |   4.816269397735596    |         0.5852         |
+--------------------------------------------------------------------------+


+----------------------------------------------+
|               Training model 1               |
+----------------------------------------------+
Using TensorFlow backend.
Epoch-14, Current Metric - 0.83:  28%|██████▊                 | 110/387 \[01:02<02:46,  1.67 batch/s\]Time is out.
\[INFO\] training model for 86400 seconds max...  
Preprocessing the images.
Preprocessing finished.

Initializing search.
Initialization finished.


+----------------------------------------------+
|               Training model 0               |
+----------------------------------------------+
Using TensorFlow backend.

No loss decrease after 5 epochs.
...
+----------------------------------------------+
|              Training model 21               |
+----------------------------------------------+
Using TensorFlow backend.

No loss decrease after 5 epochs.


+--------------------------------------------------------------------------+
|    Father Model ID     |                 Added Operation                 |
+--------------------------------------------------------------------------+
|                        |             to\_deeper\_model 16 ReLU             |
|           16           |               to\_wider\_model 16 64              |
+--------------------------------------------------------------------------+

Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           21           |   0.8843476831912994   |   0.9316000000000001   |
+--------------------------------------------------------------------------+


+----------------------------------------------+
|              Training model 22               |
+----------------------------------------------+
Using TensorFlow backend.
Epoch-3, Current Metric - 0.9:  80%|████████████████████▊     | 310/387 \[03:50<00:58,  1.31 batch/s\]Time is out.

No loss decrease after 30 epochs.

Here you can see that our script is instructing Auto-Keras to perform six sets of experiments.

The total training time, including the time limits + the time to re-fit the model, was **a little over 3 days** on an NVIDIA K80 GPU.

### Auto-Keras results

[![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20309'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/01/autokeas_results.png)

**Figure 5:** Using Auto-Keras usually is a very time-consuming process. Training with Auto-Keras produces the best models for CIFAR-10 in the 8-12 hour range. Past that, Auto-Keras is not able to optimize further.

In **Figure 5** above you can see the effect of the amount of training time (_x_\-axis) on overall accuracy (_y_\-axis) using Auto-Keras.

Lower training times, namely 1 and 2 hours, lead to ~73% accuracy. Once we train for 4 hours we are able to achieve up to 93% accuracy.

The best accuracy we obtain is in the 8-12 range where we achieve **95% accuracy**.

Training for longer than 8-12 hours does not increase our accuracy, implying that we have reached a saturation point and Auto-Keras is not able to optimize further.

### Are Auto-Keras and AutoML worth it?

[![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20450%20272'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/01/autokeras_worthit.png)

**Figure 6:** Is Auto-Keras (or AutoML) worth it? It is certainly a great step forward in the industry and is especially helpful for those without deep learning domain knowledge. That said, seasoned deep learning experts can craft architectures + train them in significantly less time + achieve equal or greater accuracy.

Outside of unsupervised learning (automatically learning patterns from unlabeled data), automated machine learning for non-experts is considered the “holy grail” of machine learning.

Both Google’s AutoML and the open source Auto-Keras package attempt to bring machine learning to the masses, even without significant technical experience.

While Auto-Keras worked reasonably well for CIFAR-10, **I ran a second set of experiments** using my previous post on [deep learning, medical imagery, and malaria detection](https://pyimagesearch.com/2018/12/03/deep-learning-and-medical-image-analysis-with-keras/).

In that **previous post, I obtained 97.1% accuracy** using a simplified ResNet architecture which took under one hour to train.

**I then let Auto-Keras run for 24 hours on the same dataset — _the result was barely 96% accuracy_, less than my hand-defined architecture.**

Both Google’s AutoML and Auto-Keras are great steps forward; however, automated machine learning is _nowhere near solved_.

Automatic machine learning (currently) does not beat having expertise in deep learning — domain expertise, specifically in the data you are working with, is _absolutely critical_ to obtain a higher accuracy model.

**My suggestion is to [invest in your own knowledge](https://pyimagesearch.com/deep-learning-computer-vision-python-book/), don’t rely on automated algorithms.**

To be a successful deep learning practitioner and engineer you need to bring the right tool to the job. Use AutoML and Auto-Keras for what they are, _tools_, and then continue to fill your own toolbox with additional knowledge.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s blog post, we discussed Auto-Keras and AutoML, a set of tools and libraries to perform automated machine learning and deep learning.

The end goal of both Auto-Keras and AutoML is to reduce the barrier to entry to performing machine learning and deep learning through the use of Neural Architecture Search (NAS) algorithms.

**NAS algorithms, the backbone of Auto-Keras and AutoML, will _automatically_:**

1.  Define and optimize a neural network architecture
2.  Tune the hyperparameters to the model

**The primary benefits include:**

*   Being able to perform machine learning and deep learning with little expertise
*   Obtaining a high accuracy model with the ability to generalize to data outside the training and testing set
*   Getting up and running quickly with either a GUI interface or a simple API
*   A potentially state-of-the-art performance with little effort

**Of course, there is a price to be paid — two prices in fact.**

First, Google’s AutoML is expensive, approximately $20/hour.

To save funds you could go with Auto-Keras, an open source alternative to Google’s AutoML, _but you still need to pay for GPU compute time_.

Replacing an actual deep learning expert with a NAS algorithm will require many hours of computing to search for optimal parameters.

While we achieved a high accuracy model for CIFAR-10 (~96% accuracy), when I applied Auto-Keras to my previous post on [medical deep learning and malaria prediction](https://pyimagesearch.com/2018/12/03/deep-learning-and-medical-image-analysis-with-keras/), **Auto-Keras only achieved 96.1% accuracy, a full percentage point lower than my 97% accuracy (and Auto-Keras required _2,300% more compute time!_)**

While Auto-Keras and AutoML may be a step in the right direction in terms of automated machine learning and deep learning, there is still quite a bit of work to be done in this area.

There is no silver bullet for solving machine learning/deep learning with off-the-shelf algorithms. **Instead, I recommend you [invest in yourself as a deep learning practitioner and engineer](https://pyimagesearch.com/deep-learning-computer-vision-python-book/).**

The skills you learn today and tomorrow will pay off _tremendously_ in the future.

I hope you enjoyed today’s tutorial!

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _just enter your email address in the form below!_**

![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!