Model Parameters and Hyperparameters in Machine Learning — What is the difference? | by Benjamin Obi Tayo Ph.D. | Towards Data Science

In a machine learning model, there are 2 types of parameters:
Model Parameters: These are the parameters in the model that must be determined using the training data set. These are the fitted parameters.Hyperparameters: These are adjustable parameters that must be tuned in order to obtain a model with optimal performance.
Model Parameters: These are the parameters in the model that must be determined using the training data set. These are the fitted parameters.
Hyperparameters: These are adjustable parameters that must be tuned in order to obtain a model with optimal performance.
For example, suppose you want to build a simple linear regression model using an m-dimensional training data set. Then your model can be written as:
where X is the predictor matrix, and w are the weights. Here w_0, w_1, w_2, ...,w_m are the model parameters. If the model uses the gradient descent algorithm to minimize the objective function in order to determine the weights w_0, w_1, w_2, ...,w_m, then we can have an optimizer such as GradientDescent(eta, n_iter). Here eta (learning rate) and n_iter (number of iterations) are the hyperparameters that would have to be adjusted in order to obtain the best values for the model parameters w_0, w_1, w_2, ...,w_m. For more information about this, see the following example: Machine Learning: Python Linear Regression Estimator Using Gradient Descent.
Perceptron Classifier
Perceptron Classifier
Perceptron(n_iter=40, eta0=0.1, random_state=0)
Here, n_iter is the number of iterations, eta0 is the learning rate, and random_state is the seed of the pseudo random number generator to use when shuffling the data.
2. Train, Test Split Estimator
train_test_split( X, y, test_size=0.4, random_state=0)
Here, test_size represents the proportion of the dataset to include in the test split, and random_state is the seed used by the random number generator.
3. Logistic Regression Classifier
LogisticRegression(C=1000.0, random_state=0)
Here, C is the inverse of regularization strength, and random_state is the seed of the pseudo random number generator to use when shuffling the data.
4. KNN (k-Nearest Neighbors) Classifier
KNeighborsClassifier(n_neighbors=5, p=2, metric='minkowski')
Here, n_neighbors is the number of neighbors to use, p is the power parameter for the Minkowski metric. When p = 1, this is equivalent to using manhattan_distance, and euclidean_distance for p = 2.
5. Support Vector Machine Classifier
SVC(kernel='linear', C=1.0, random_state=0)
Here, kernel specifies the kernel type to be used in the algorithm, for example kernel = ‘linear’, for linear classification, or kernel = ‘rbf’ for non-linear classification. C is the penalty parameter of the error term, and random_state is the seed of the pseudo random number generator used when shuffling the data for probability estimates.
6. Decision Tree Classifier
DecisionTreeClassifier(criterion='entropy',                        max_depth=3, random_state=0)
Here, criterion is the function to measure the quality of a split, max_depth is the maximum depth of the tree, and random_state is the seed used by the random number generator.
7. Lasso Regression
Lasso(alpha = 0.1)
Here, alpha is the regularization parameter.
8. Principal Component Analysis
PCA(n_components = 4)
Here, n_components is the number of components to keep. If n_components is not set all components are kept.
It is important that during model building, these hyperparameters be fine-tuned in order to obtain the model with the highest quality. A good example of how the predictive power of a model depends on hyperparameters can be found from the figure below (source: Bad and Good Regression Analysis).
From the figure above, we see that the reliability of our model depends on hyperparameter tuning. If we just pick a random value for the learning rate such as eta = 0.1, this would lead to a poor model. Choosing a value for eta too small, such as eta = 0.00001 also produces a bad model. Our analysis shows that the best choice is when eta = 0.0001, as can be seen from the R-square values.
What makes the difference between a good and a bad machine learning model depends on one’s ability to understand all the details of the model including knowledge about different hyperparameters and how these parameters can be tuned in order to obtain the model with the best performance. Using any machine learning model as a black box without fully understanding the intricacies of the model will lead to a falsified model.
“Python Machine Learning”, 2nd Edition, Sebastian Raschka.Machine Learning: Python Linear Regression Estimator Using Gradient Descent.Bad and Good Regression Analysis.
“Python Machine Learning”, 2nd Edition, Sebastian Raschka.
Machine Learning: Python Linear Regression Estimator Using Gradient Descent.
Bad and Good Regression Analysis.