Learning and Generalization — Understanding Efficient BackProp Part 1
By: Vaishnavi Nandakumar
Tags: Machine Learning, Deep Learning, Research, Neural Networks, Pytorch

Although the values aren’t exactly the same it is obvious that the model has now begun to understand the relationship between the two variables and can thus progressively produce more accurate results with a minimized loss.

Now this is a small example demonstrating a simple linear relationship. What happens when the size of the dataset increases and different sets of samples are provided? Or how can we optimize a model that requires a classification algorithm instead of regression? In such cases, merely reducing the cost function is not enough. This is where we introduce a concept called generalization.

Generalization

Generalization is the ability of a model to predict the correct target values for an entirely new set dataset which it has not been trained on. This technique largely tries to optimize learning by minimizing or correcting the errors introduced in the training set. This process is also known as Empirical Risk Minimization. The basic principle behind empirical risk minimization is to convert the problem of training a model into an optimization problem. It generally focuses on two parts, the loss function and a regularizer which is used to penalize certain values of the parameters involved. A hyper parameter is also used to maintain the balance between the two. This is a very brief explanation but you can read more about it here. The article linked to ERM gives a concise explanation about it along with the demonstration of loss functions, regularizers and some special cases as well.

The analysis of most minimization processes are done by decomposing the generalization error into two terms — bias and variance. Bias’s are learnable parameters in the network just like weights. It gives the total measure of difference between the average of all the network outputs across the different samples of datasets and the desired output. Meanwhile, variance refers to deviation from the average of predictions. Minimal error can only be achieved only when both of these values are at their least.

From the graph shown above we can understand that the initial phases of training sees a large bias and a low variance. However, this scenario soon gets reversed as the training progresses because the network now begins to learns better. But does this mean the longer a model is left to train, the more efficient it gets? No. An important point that has to be noted, is that if the network is trained for too long, it will also understand the noise (meaningless information) and adapt to the specific features within the training dataset. This is a problem known as ‘overfitting’. Early stopping and regularization are some of the techniques used to prevent it which will be covered in detail later. This point proves that even if excellent cost minimization strategies are implemented, improper model selection which negatively impacts generalization causes the whole network to become inefficient.

Summary

This article was based on mainly two topics — learning and generalization in machine learning. We were able to go through some fundamental concepts in gradient based learning with a small example to illustrate how it works. The importance of generalization in terms of accuracy and optimization was also discussed. A brief explanation on empirical risk minimization and the factors on which generalization error is analyzed was also explained. I hope this article was able help you understand or give you an idea at the least about the topics I’ve written about. The next article will be focusing on standard backpropagation as well as some practical tricks that include initialing weights, shuffling examples, normalizing input values and so on.

Thank you for reading!

References