Title: Implementing the Perceptron Neural Network with Python - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/06/implementing-the-perceptron-neural-network-with-python/

Published Time: 2021-05-06T14:00:00+00:00

Markdown Content:
First introduced by [Rosenblatt in 1958, _The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain_](https://doi.org/10.1037/h0042519) is arguably the oldest and most simple of the ANN algorithms. Following this publication, Perceptron-based techniques were all the rage in the neural network community. This paper alone is hugely responsible for the popularity and utility of neural networks today.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20250'%3E%3C/svg%3E)

But then, in 1969, an “AI Winter” descended on the machine learning community that almost froze out neural networks for good. [Minsky and Papert published _Perceptrons: an introduction to computational geometry_](https://mitpress.mit.edu/books/perceptrons), a book that effectively stagnated research in neural networks for almost a decade — there is much controversy regarding the book ([Olazaran, 1996](http://www.jstor.org/stable/285702)), but the authors did successfully demonstrate that a _single layer_ Perceptron is unable to separate nonlinear data points.

Given that most real-world datasets are naturally nonlinearly separable, this it seemed that the Perceptron, along with the rest of neural network research, might reach an untimely end.

Between the Minsky and Papert publication and the broken promises of neural networks revolutionizing industry, the interest in neural networks dwindled substantially. It wasn’t until we started exploring deeper networks (sometimes called _multi-layer perceptrons_) along with the backpropagation algorithm ([Werbos](https://www.worldcat.org/title/beyond-regression-new-tools-for-prediction-and-analysis-in-the-behavioral-sciences/oclc/77001455) and [Rumelhart et al.](http://dl.acm.org/citation.cfm?id=65669.104451)) that the “AI Winter” in the 1970s ended and neural network research started to heat up again.

All that said, the Perceptron is still a _very important algorithm_ to understand as it sets the stage for more advanced multi-layer networks. We’ll start this section with a review of the Perceptron architecture and explain the training procedure (called the **delta rule**) used to train the Perceptron. We’ll also look at the _termination criteria_ of the network (i.e., when the Perceptron should stop training). Finally, we’ll implement the Perceptron algorithm in pure Python and use it to study and examine how the network is unable to learn nonlinearly separable datasets.

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

### **AND, OR, and XOR Datasets**

Before we study the Perceptron itself, let’s first discuss “bitwise operations,” including AND, OR, and XOR (exclusive OR). If you’ve taken an introductory level computer science course before you might already be familiar with bitwise functions.

Bitwise operators and associated bitwise datasets accept two input bits and produce a final output bit after applying the operation. Given two input bits, each potentially taking on a value of `0` or `1`, there are four possible combinations of these two bits — **Table 1** provides the possible input and output values for AND, OR, and XOR:

| _**x**_**0** | _**x**_**1** | **_x_0&_x_1** | _**x**_**0** | _**x**_**1** | _**x**_**0**_**|x**_**1** | _**x**_**0** | _**x**_**1** | _**x**_**0****∧**_**x**_**1** |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 1 |
| 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 |
| 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |

**Table 1:** _Left:_ The bitwise AND dataset. Given two inputs, the output is only 1 if both inputs are 1. _Middle:_ The bitwise OR dataset. Given two inputs, the output is 1 if _either_ of the two inputs is 1. _Right:_ The XOR (e(X)clusive OR) dataset. Given two inputs, the output is 1 if and only if one of the inputs is 1, _but not both_.

As we can see on the _left_, a logical AND is true _if and only if_ both input values are `1`. If _either_ of the input values is `0`, the AND returns `0`. Thus, there is only one combination, _x_0 \= 1 and _x_1 \= 1 when the output of AND is true.

In the _middle_, we have the OR operation which is true when _at least one_ of the input values is `1`. Thus, there are three possible combinations of the two bits _x_0 and _x_1 that produce a value of _y_ \= 1.

Finally, the _right_ displays the XOR operation which is true _if and only if_ one if the inputs is `1` _but not both_. While OR had three possible situations where _y_ \= 1, XOR only has two.

We often use these simple “bitwise datasets” to test and debug machine learning algorithms. If we plot and visualize the AND, OR, and XOR values (with red circles being zero outputs and blue stars one outputs) in **Figure 1**, you’ll notice an interesting pattern:

![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20250'%3E%3C/svg%3E)

**Figure 1:** Both the AND and OR bitwise datasets are linearly separable, meaning that we can draw a _single line_ (green) that separates the two classes. However, for XOR it is impossible to draw a single line that separates the two classes — this is therefore a nonlinearly separable dataset.

Both AND and OR are linearly separable — we can clearly draw a line that separates the `0` and `1` classes — the same is not true for XOR. Take the time now to convince yourself that it is _not possible_ to draw a line that cleanly separates the two classes in the XOR problem. XOR is, therefore, an example of a _nonlinearly separable_ dataset.

Ideally, we would like our machine learning algorithms to be able to separate nonlinear classes as most datasets encountered in the real world are nonlinear. Therefore, when constructing, debugging, and evaluating a given machine learning algorithm, we may use the bitwise values _x_0 and _x_1 as our _design matrix_ and then try to predict the corresponding _y_ values.

Unlike our standard procedure of splitting our data into _training_ and _testing_ splits, when using bitwise datasets we simply train and evaluate our network on the same set of data. Our goal here is simply to determine if it’s even _possible_ for our learning algorithm to learn the patterns in the data. As we’ll find out, the Perceptron algorithm can correctly classify the AND and OR functions but fails to classify the XOR data.

### **Perceptron Architecture**

[Rosenblatt (1958)](https://doi.org/10.1037/h0042519) defined a Perceptron as a system that learns using labeled examples (i.e., supervised learning) of feature vectors (or raw pixel intensities), mapping these inputs to their corresponding output class labels.

In its simplest form, a Perceptron contains _N_ input nodes, one for each entry in the _input row_ of the design matrix, followed by _only one layer_ in the network with just a _single node_ in that layer (**Figure 2**).

There exist connections and their corresponding weights _w_1_, w_2_, …, wi_ from the input _xi_’s to the single output node in the network. This node takes the weighted sum of inputs and applies a _step function_ to determine the output class label. The Perceptron outputs either a `0` or a `1` — `0` for class #1 and `1` for class #2; thus, in its original form, the Perceptron is simply a binary, two-class classifier.

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20652%20475'%3E%3C/svg%3E)

**Figure 2:** Architecture of the Perceptron network.

1\. Initialize our weight vector _**w**_ with small random values  
2\. Until Perceptron converges:  
(a) Loop over each feature vector _xj_ and true class label _di_ in our training set _D  
_(b) Take _**x**_ and pass it through the network, calculating the output value: _yj_ \= _f_(**_w_(_t_)** _· **xj**_)  
(c) Update the weights _**w**_: _wi_(_t_ +1) = _wi_(_t_) +_α_(_dj −yj_)_xj,i_ for all features 0 _<_\= _i <_\= _n_

**Figure 3:** The Perceptron algorithm training procedure.

### **Perceptron Training Procedure and the Delta Rule**

Training a Perceptron is a fairly straightforward operation. Our goal is to obtain a set of weights _**w**_ that accurately classifies each instance in our training set. In order to train our Perceptron, we iteratively feed the network with our training data _multiple times_. Each time the network has seen the _full set_ of training data, we say an _epoch_ has passed. It normally takes many epochs until a weight vector _**w**_ can be learned to linearly separate our two classes of data.

The pseudocode for the Perceptron training algorithm can be found below:

The actual “learning” takes place in Steps 2b and 2c. First, we pass the feature vector _xj_ through the network, take the dot product with the weights _**w**_ and obtain the output _yj_. This value is then passed through the step function which will return 1 if _x >_ 0 and 0 otherwise.

Now we need to update our weight vector _**w**_ to step in the direction that is “closer” to the correct classification. This update of the weight vector is handled by the _delta rule_ in Step 2c.

The expression (_d__j_ _−y__j_) determines if the output classification is correct or not. If the classification is _correct_, then this difference will be zero. Otherwise, the difference will be either positive or negative, giving us the direction in which our weights will be updated (ultimately bringing us closer to the correct classification). We then multiply (_d__j_ _− y__j_) by _x__j_, moving us closer to the correct classification.

The value _α_ is our _learning rate_ and controls how large (or small) of a step we take. It’s _critical_ that this value is set correctly. A larger value of _α_ will cause us to take a step in the right direction; however, this step could be _too large_, and we could easily overstep a local/global optimum.

Conversely, a small value of _α_ allows us to take tiny baby steps in the right direction, ensuring we don’t overstep a local/global minimum; however, these tiny baby steps may take an intractable amount of time for our learning to converge.

Finally, we add in the previous weight vector at time _t_, _**w**j_(_t_) which completes the process of “stepping” towards the correct classification. If you find this training procedure a bit confusing, don’t worry.

### **Perceptron Training Termination**

The Perceptron training process is allowed to proceed until all training samples are classified correctly _or_ a preset number of epochs is reached. Termination is ensured if _α_ is sufficiently small _and_ the training data is linearly separable.

So, what happens if our data is not linearly separable or we make a poor choice in _α_? Will training continue infinitely? In this case, no — we normally stop after a set number of epochs has been hit or if the number of misclassifications has not changed in a large number of epochs (indicating that the data is not linearly separable). For more details on the perceptron algorithm, please refer to either [Andrew Ng’s Stanford lecture](https://www.coursera.org/learn/machine-learning) or the introductory chapters of [Mehrota et al. (1997)](https://mitpress.mit.edu/books/elements-artificial-neural-networks).

### **Implementing the Perceptron in Python**

Now that we have studied the Perceptron algorithm, let’s implement the actual algorithm in Python. Create a file named `perceptron.py` in your `pyimagesearch.nn` package — this file will store our actual `Perceptron` implementation:

|--- pyimagesearch
|    |--- \_\_init\_\_.py
|    |--- nn
|    |    |--- \_\_init\_\_.py
|    |    |--- perceptron.py

After you’ve created the file, open it, and insert the following code:

\# import the necessary packages
import numpy as np

class Perceptron:
	def \_\_init\_\_(self, N, alpha=0.1):
		# initialize the weight matrix and store the learning rate
		self.W = np.random.randn(N + 1) / np.sqrt(N)
		self.alpha = alpha

**Line 5** defines the constructor to our `Perceptron` class, which accepts a single required parameter followed by a second optional one:

1.  `N`: The number of columns in our input feature vectors. In the context of our bitwise datasets, we’ll set `N` equal to two since there are two inputs.
2.  `alpha`: Our learning rate for the Perceptron algorithm. We’ll set this value to `0.1` by default. Common choices of learning rates are normally in the range _α_ \= 0_._1_,_ 0_._01_,_ 0_._001.

**Line 7** files our weight matrix `W` with random values sampled from a “normal” (Gaussian) distribution with zero mean and unit variance. The weight matrix will have _N_ +1 entries, one for each of the `N` inputs in the feature vector, plus one for the bias. We divide `W` by the square-root of the number of inputs, a common technique used to scale our weight matrix, leading to faster convergence. We will cover weight initialization techniques later in this chapter.

Next, let’s define the `step` function:

	def step(self, x):
		# apply the step function
		return 1 if x > 0 else 0

This function mimics the behavior of the step equation — if `x` is positive we return `1`, otherwise, we return `0`.

To actually train the Perceptron we’ll define a function named `fit`. If you have any previous experience with machine learning, Python, and the scikit-learn library then you’ll know that it’s common to name your training procedure function `fit`, as in _“fit a model to the data”_:

	def fit(self, X, y, epochs=10):
		# insert a column of 1's as the last entry in the feature
		# matrix -- this little trick allows us to treat the bias
		# as a trainable parameter within the weight matrix
		X = np.c\_\[X, np.ones((X.shape\[0\]))\]

The `fit` method requires two parameters followed by a single optional one:

The `X` value is our actual training data. The _`y`_ variable is our target output class labels (i.e., what our network _should_ be predicting). Finally, we supply `epochs`, the number of epochs our Perceptron will train for.

**Line 18** applies the bias trick by inserting a column of ones into the training data, which allows us to treat the bias as a trainable parameter _directly_ inside the weight matrix.

Next, let’s review the actual training procedure:

		# loop over the desired number of epochs
		for epoch in np.arange(0, epochs):
			# loop over each individual data point
			for (x, target) in zip(X, y):
				# take the dot product between the input features
				# and the weight matrix, then pass this value
				# through the step function to obtain the prediction
				p = self.step(np.dot(x, self.W))

				# only perform a weight update if our prediction
				# does not match the target
				if p != target:
					# determine the error
					error = p - target

					# update the weight matrix
					self.W += -self.alpha \* error \* x

On **Line 21**, we start looping over the desired number of `epochs`. For each epoch, we also loop over each individual data point `x` and output `target` class label (**Line 23**).

**Line 27** takes the dot product between the input features `x` and the weight matrix `W`, then passes the output through the `step` function to obtain the prediction by the Perceptron.

Applying the same training procedure detailed in **Figure 3**, we only perform a weight update if our prediction _does not_ match the target (**Line 31**). If this is the case, we determine the `error` (**Line 33**) by computing the sign (either positive or negative) via the difference operation.

Updating the weight matrix is handled on **Line 36** where we take a step towards the correct classification, scaling this step by our learning rate `alpha`. Over a series of epochs, our Perceptron is able to learn patterns in the underlying data and shift the values of the weight matrix such that we correctly classify our input samples `x`.

The last function we need to define is `predict`, which, as the name suggests, is used to _predict_ the class labels for a given set of input data:

	def predict(self, X, addBias=True):
		# ensure our input is a matrix
		X = np.atleast\_2d(X)

		# check to see if the bias column should be added
		if addBias:
			# insert a column of 1's as the last entry in the feature
			# matrix (bias)
			X = np.c\_\[X, np.ones((X.shape\[0\]))\]

		# take the dot product between the input features and the
		# weight matrix, then pass the value through the step
		# function
		return self.step(np.dot(X, self.W))

Our `predict` method requires a set of input data `X` that needs to be classified. A check on **Line 43** is made to see if a bias column needs to be added.

Obtaining the output predictions for `X` is the same as the training procedure — simply take the dot product between the input features `X` and our weight matrix `W`, and then pass the value through our step function. The output of the step function is returned to the calling function.

Now that we’ve implemented our `Perceptron` class, let’s try to apply it to our bitwise datasets and see how the neural network performs.

### **Evaluating the Perceptron Bitwise Datasets**

To start, let’s create a file named `perceptron_or.py` that attempts to fit a Perceptron model to the bitwise OR dataset:

\# import the necessary packages
from pyimagesearch.nn import Perceptron
import numpy as np

# construct the OR dataset
X = np.array(\[\[0, 0\], \[0, 1\], \[1, 0\], \[1, 1\]\])
y = np.array(\[\[0\], \[1\], \[1\], \[1\]\])

# define our perceptron and train it
print("\[INFO\] training perceptron...")
p = Perceptron(X.shape\[1\], alpha=0.1)
p.fit(X, y, epochs=20)

**Lines 2 and 3** import our required Python packages. We’ll be using our `Perceptron` implementation. **Lines 6 and 7** define the OR dataset based on **Table 1**.

**Lines 11 and 12** train our Perceptron with a learning rate of _α_ \= 0_._1 for a total of 20 epochs.

We can then evaluate our Perceptron on the data to validate that it did, in fact, learn the OR function:

\# now that our perceptron is trained we can evaluate it
print("\[INFO\] testing perceptron...")

# now that our network is trained, loop over the data points
for (x, target) in zip(X, y):
	# make a prediction on the data point and display the result
	# to our console
	pred = p.predict(x)
	print("\[INFO\] data={}, ground-truth={}, pred={}".format(
		x, target\[0\], pred))

On **Line 18**, we loop over each of the data points in the OR dataset. For each of these data points, we pass it through the network and obtain the prediction (**Line 21**).

Finally, **Lines 22 and 23** display the input data point, the ground-truth label, as well as our predicted label to our console.

To see if our Perceptron algorithm is able to learn the OR function, just execute the following command:

$ python perceptron\_or.py 
\[INFO\] training perceptron...
\[INFO\] testing perceptron...
\[INFO\] data=\[0 0\], ground-truth=0, pred=0
\[INFO\] data=\[0 1\], ground-truth=1, pred=1
\[INFO\] data=\[1 0\], ground-truth=1, pred=1
\[INFO\] data=\[1 1\], ground-truth=1, pred=1

Sure enough, our neural network is able to correctly predict that the OR operation for _x_0 \= 0 and _x_1 \= 0 is zero — all other combinations are one.

Now, let’s move on to the AND function — create a new file named `perceptron_and.py` and insert the following code:

\# import the necessary packages
from pyimagesearch.nn import Perceptron
import numpy as np

# construct the AND dataset
X = np.array(\[\[0, 0\], \[0, 1\], \[1, 0\], \[1, 1\]\])
y = np.array(\[\[0\], \[0\], \[0\], \[1\]\])

# define our perceptron and train it
print("\[INFO\] training perceptron...")
p = Perceptron(X.shape\[1\], alpha=0.1)
p.fit(X, y, epochs=20)

# now that our perceptron is trained we can evaluate it
print("\[INFO\] testing perceptron...")

# now that our network is trained, loop over the data points
for (x, target) in zip(X, y):
	# make a prediction on the data point and display the result
	# to our console
	pred = p.predict(x)
	print("\[INFO\] data={}, ground-truth={}, pred={}".format(
		x, target\[0\], pred))

Notice here that the _only_ lines of code that have changed are **Lines 6 and 7** where we define the AND dataset rather than the OR dataset.

Executing the following command, we can evaluate the Perceptron on the AND function:

$ python perceptron\_and.py 
\[INFO\] training perceptron...
\[INFO\] testing perceptron...
\[INFO\] data=\[0 0\], ground-truth=0, pred=0
\[INFO\] data=\[0 1\], ground-truth=0, pred=0
\[INFO\] data=\[1 0\], ground-truth=0, pred=0
\[INFO\] data=\[1 1\], ground-truth=1, pred=1

Again, our Perceptron was able to correctly model the function. The AND function is only true when _both x_0 \= 1 and _x_1 \= 1 – for all other combinations the bitwise AND is zero.

Finally, let’s take a look at the nonlinearly separable XOR function inside `perceptron_xor.py`:

\# import the necessary packages
from pyimagesearch.nn import Perceptron
import numpy as np

# construct the XOR dataset
X = np.array(\[\[0, 0\], \[0, 1\], \[1, 0\], \[1, 1\]\])
y = np.array(\[\[0\], \[1\], \[1\], \[0\]\])

# define our perceptron and train it
print("\[INFO\] training perceptron...")
p = Perceptron(X.shape\[1\], alpha=0.1)
p.fit(X, y, epochs=20)

# now that our perceptron is trained we can evaluate it
print("\[INFO\] testing perceptron...")

# now that our network is trained, loop over the data points
for (x, target) in zip(X, y):
	# make a prediction on the data point and display the result
	# to our console
	pred = p.predict(x)
	print("\[INFO\] data={}, ground-truth={}, pred={}".format(
		x, target\[0\], pred))

Again, the only lines of code that have been changed are **Lines 6 and 7** where we define the XOR data. The XOR operator is true _if and only if_ one (but not both) _x_’s are one.

Executing the following command we can see that the Perceptron _cannot_ learn this nonlinear relationship:

$ python perceptron\_xor.py 
\[INFO\] training perceptron...
\[INFO\] testing perceptron...
\[INFO\] data=\[0 0\], ground-truth=0, pred=1
\[INFO\] data=\[0 1\], ground-truth=1, pred=1
\[INFO\] data=\[1 0\], ground-truth=1, pred=0
\[INFO\] data=\[1 1\], ground-truth=0, pred=0

No matter how many times you run this experiment with varying learning rates or different weight initialization schemes, you will _never_ be able to correctly model the XOR function with a single layer Perceptron. Instead, what we need is _more layers_ with _nonlinear activation functions_ and with that, comes the start of deep learning.

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!