Perceptron Algorithms for Linear Classification | by Edwin Tai | Towards Data Science

The basic perceptron algorithm was first introduced by Ref 1 in the late 1950s. It is a binary linear classifier for supervised learning. The idea behind the binary linear classifier can be described as follows.
where x is the feature vector, θ is the weight vector, and θ0 is the bias. The sign function is used to distinguish x as either a positive (+1) or a negative (-1) label. There is the decision boundary to separate the data with different labels, which occurs at
The decision boundary separates the hyperplane into two regions. The data will be labeled as positive in the region that θ⋅ x + θ0 > 0, and be labeled as negative in the region that θ⋅ x + θ0 < 0. If all the instances in a given data are linearly separable, there exists a θ and a θ0 such that y(i ) (θ⋅ x(i ) + θ0) > 0 for every i-th data point, where y(i ) is the label.
Figure 1 illustrates the aforementioned concepts with the 2-D case where the x = [x1 x2]T, θ = [θ1 θ2] and θ0 is a offset scalar. Note that the margin boundaries are related to the regularization to prevent overfitting of the data, which is beyond the scope discussed here.
One way to find the decision boundary is using the perceptron algorithm. The perceptron algorithm updates θ and θ0 only when the decision boundary misclassifies the data points. The pseudocode of the algorithm is described as follows.
# Perceptron Algorithm# initialize θ and θ0 with 0θ = 0  (vector)θ0 = 0 (scalar)# totally T epoches to iteratefor t = 1 .. T do                         # totally m data points        for i = 1 .. m do        # misclassify data points                         if y(i)(θ ⋅ x(i) + θ0) ≦ 0             then            θ  = θ + y(i) ⋅ x(i)            θ0 = θ0 + y(i)return θ, θ0
The perceptron algorithm iterates through all the data points with labels and updating θ and θ0 correspondingly. The intuition behind the updating rule is to push the y(i ) (θ⋅ x(i ) + θ0) closer to a positive value if y(i ) (θ⋅ x(i ) + θ0) ≦ 0 since y(i ) (θ⋅ x(i ) + θ0) > 0 represents classifying the i-th data point correctly.
The factors that constitute the bound on the number of mistakes made by the perceptron algorithm are maximum norm of data points and maximum margin between positive and negative data points. The theorems of the perceptron convergence has been proven in Ref 2. Given a set of data points that are linearly separable through the origin, the initialization of θ does not impact the perceptron algorithm’s ability to eventually converge.
The number of the iteration k has a finite value implies that once the data points are linearly separable through the origin, the perceptron algorithm converges eventually no matter what the initial value of θ is. The concepts also stand for the presence of θ0.
However, this perceptron algorithm may encounter convergence problems once the data points are linearly non-separable. There are two perceptron algorithm variations introduced to deal with the problems. One is the average perceptron algorithm, and the other is the pegasos algorithm.
Similar to the perceptron algorithm, the average perceptron algorithm uses the same rule to update parameters. The final returning values of θ and θ0 however take the average of all the values of θ and θ0 in each iteration. The pseudocode of the algorithm is described as follows.
# Average Perceptron Algorithm# initialize θ, θ0, sum_θ, sum_θ0, and counter with 0θ = 0  (vector)θ0 = 0 (scalar)sum_θ = 0  (vector)sum_θ0 = 0 (scalar)counter = 0# totally T epoches to iteratefor t = 1 .. T do                         # totally m data points        for i = 1 .. m do          # misclassify data points                       if y(i)(θ ⋅ x(i) + θ0) ≦ 0             then            θ  = θ + y(i) ⋅ x(i)            θ0 = θ0 + y(i)        sum_θ = sum_θ + θ        sum_θ0 = sum_θ0 + θ0        counter = counter + 1return (sum_θ/counter), (sum_θ0/counter)
The pegasos algorithm has the hyperparameter λ, giving more flexibility to the model to be adjusted. The θ are updated whether the data points are misclassified or not. The details are discussed in Ref 3. The pseudocode of the algorithm is described as follows.
# Pegasos Algorithm# initialize θ, θ0, and counter with 0θ = 0  (vector)θ0 = 0 (scalar)counter = 0# totally T epoches to iteratefor t = 1 .. T do     # totally m data points                        for i = 1 .. m do                         counter = counter + 1        η = 1/√counter                  if y(i)(θ⋅x(i) + θ0) ≦ 1            then            θ  = (1 - ηλ)θ + ηy(i)⋅x(i)            θ0 = θ0 + ηy(i)        else        then            θ  = (1 - ηλ)θ            θ0 = θ0return θ,  θ0
Figure 2. visualizes the updating of the decision boundary by the different perceptron algorithms. Note that the given data are linearly non-separable so that the decision boundary drawn by the perceptron algorithm diverges. Both the average perceptron algorithm and the pegasos algorithm quickly reach convergence. The λ for the pegasos algorithm uses 0.2 here.
The sample code written in Jupyter notebook for the perceptron algorithms can be found here. You can play with the data and the hyperparameters yourself to see how the different perceptron algorithms perform.
[1]
F. Rosenblatt,” The perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, 1958. doi: 10.1037/h0042519
[2]
M. Mohri, and A. Rostamizadeh,” Perceptron Mistake Bounds,” arxiv, 2013. https://arxiv.org/pdf/1305.0208.pdf
[3]
S. S.-Shwartz, Y. Singer, N. Srebro, and A. Cotter,” Pegasos: primal estimated sub-gradient solver for SVM,” Mathematical Programming, 2010. doi: 10.1007/s10107–010–0420–4