An Introduction to Perceptron Algorithm | by Yang S | Towards Data Science

This blog will cover following questions and topics
1. What is Perceptron?
2. Stochastic Gradient Descent for Perceptron
3. Implementation in Python
1. What is Perceptron?
Perceptron set the foundations for Neural Network models in 1980s. The algorithm was developed by Frank Rosenblatt and was encapsulated in the paper “Principles of Neuro-dynamics: Perceptrons and the Theory of Brain Mechanisms” published in 1962. At that time, Rosenblatt’s work was criticized by Marvin Minksy and Seymour Papert, arguing that neural networks were flawed and could only solve linear separation problem. However, such limitation only occurs in the single layer neural network.
Perceptron can be used to solve two-class classification problem. The generalized form of algorithm can be written as:
Nonlinear activation sign function is:
While logistic regression is targeting on the probability of events happen or not, so the range of target value is [0, 1]. Perceptron uses more convenient target values t=+1 for first class and t=-1 for second class. Therefore, the algorithm does not provide probabilistic outputs, nor does it handle K>2 classification problem. Another limitation arises from the fact that the algorithm can only handle linear combinations of fixed basis function.
2. Stochastic Gradient Descent for Perceptron
According to previous two formulas, if a record is classified correctly, then:
Otherwise,
Therefore, to minimize cost function for Perceptron, we can write:
M means the set of misclassified records.
By taking partial derivative, we can get gradient of cost function:
Unlike logistic regression, which can apply Batch Gradient Descent, Mini-Batch Gradient Descent and Stochastic Gradient Descent to calculate parameters, Perceptron can only use Stochastic Gradient Descent. We need to initialize parameters w and b, and then randomly select one misclassified record and use Stochastic Gradient Descent to iteratively update parameters w and b until all records are classified correctly:
Note that learning rate a ranges from 0 to 1.
For example, we have 3 records, Y1 = (3, 3), Y2 = (4, 3), Y3 = (1, 1). Y1 and Y2 are labeled as +1 and Y3 is labeled as -1. Given that initial parameters are all 0. Therefore, all points will be classified as class 1.
Stochastic Gradient Descent cycles through all training data. In the initial round, by applying first two formulas, Y1 and Y2 can be classified correctly. However, Y3 will be misclassified. Assuming learning rate equals to 1, by applying gradient descent shown above, we can get:
Then linear classifier can be written as:
That is 1 round of gradient descent iteration.
Table above shows the whole procedure of Stochastic Gradient Descent for Perceptron. If a record is classified correctly, then weight vector w and b remain unchanged; otherwise, we add vector x onto current weight vector when y=1 and minus vector x from current weight vector w when y=-1. Note that last 3 columns are predicted value and misclassified records are highlighted in red. If we carry out gradient descent over and over, in round 7, all 3 records are labeled correctly. Then the algorithm will stop. Final formula for linear classifier is:
Note that there is always converge issue with this algorithm. When the data is separable, there are many solutions, and which solution is chosen depends on the starting values. When the data is not separable, the algorithm will not converge. For details, please see corresponding paragraph in reference below.
3. Implementation in Python
from sklearn import datasetsimport numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport matplotlib.lines as mlinesnp.random.seed(10)# Load datairis=datasets.load_iris()X = iris.data[0:99,:2]y = iris.target[0:99]
First, load the Iris data.
# Plot figureplt.plot(X[:50, 0], X[:50, 1], 'bo', color='blue', label='0')plt.plot(X[50:99, 0], X[50:99, 1], 'bo', color='orange', label='1')plt.xlabel('sepal length')plt.ylabel('sepal width')plt.legend()
Then visualize data
# Update y into -1 and 1y=np.array([1 if i==1 else -1 for i in y])
Update y=0 to y=-1
################################## Gradient Descent################################## Initialize parametersw=np.ones((X.shape[1],1));b=1;learning_rate=0.1;Round=0;All_Correct=False;# Start Gradient Descentwhile not All_Correct:    misclassified_count=0    for i in range(X.shape[0]):        XX=X[i,]        yy=y[i]        if yy * (np.dot(w.T,XX.T)+b)<0:            w+=learning_rate * np.dot(XX,yy).reshape(2,1)            b+=learning_rate * yy            misclassified_count +=1    if misclassified_count==0:        All_Correct=True    else:        All_Correct=False    Round += 1    print(Round)print(w)print(b)
After applying Stochastic Gradient Descent, we get w=(7.9, -10.07) and b=-12.39
x_points = np.linspace(4,7,10)y_ = -(w[0]*x_points + b)/w[1]plt.plot(x_points, y_)plt.plot(X[:50, 0], X[:50, 1], 'bo', color='blue', label='0')plt.plot(X[50:99, 0], X[50:99, 1], 'bo', color='orange', label='1')plt.xlabel('sepal length')plt.ylabel('sepal width')plt.legend()
Figure above shows the final result of Perceptron. We can see that the linear classifier (blue line) can classify all training dataset correctly. In this case, the iris dataset only contains 2 dimensions, so the decision boundary is a line. In the case when the dataset contains 3 or more dimensions, the decision boundary will be a hyperplane.
Conclusion
In this blog, I explain the theory and mathematics behind Perceptron, compare this algorithm with logistic regression, and finally implement the algorithm in Python. Hope after reading this blog, you can have a better understanding of this algorithm. If you have interests in other blogs, please click on the following link:
medium.com
Reference
[1] Christopher M. Bishop, (2009), Pattern Recognition and Machine Leaning
[2] Trevor Hastie, Robert Tibshirani, Jerome Friedman, (2008), The Elements of Statistical Learning