The primary reason overfitting happens is because the model learns even the tiniest details present in the data. So after learning all the possible patterns it can find, the model tends to perform extremely well on the training set but fails to produce good results on the test sets. It falls apart when faced with previously unseen data. And this is critical from an accuracy standpoint. One way to prevent overfitting is to reduce the complexity of the model. This is exactly what regularization does. If we set the regularization parameter to a large value, the decay in the weights during gradient descent update will be more. Hence, the weights of most of the hidden units will be close to zero. Since the weights are negligible, the model will not learn much from these units. This will end up making the network simpler and thus reduce overfitting. Let us take another example. Assume we are using a tanh activation function. Now if we set regularization parameter to a large value, the weight of the units will be less. To calculate the z[l], we can use the following: Z[l] = w[l] m[l-1] + n[l] Hence the z-value will be less. If we use the tanh activation function then these low values of z[l] will lie near the origin. The key aspect with this change is that we are only using the linear region of the tanh function. This will make every layer in the network mostly linear. i.e. we will get linear boundaries that separate the data which prevents overfitting.