Theoretical Foundations of Data Science— Should I Care or Simply Focus on Hands-on Skills?
By: Benjamin Obi Tayo Ph.D.
Tags: Machine Learning, Data Science, Mathematics, Statistics, Linear Algebra

Data science is a very hands-on and practical field. Data science requires a solid foundation in mathematics and programming. As a data scientist, it is essential that you understand the theoretical and mathematical foundations of data science in order to be able to build reliable models with real-world applications.

In data science and machine learning, mathematical skills are as important as programming skills. There are so many good packages that can be used for building predictive models. Some of the most common packages for descriptive and predictive analytics include

Ggplot2

Matplotlib

Seaborn

Sci-kit learn package

Caret package

Tensorflow

PyTouch Package

Keras Package

It’s important that before using these packages, you master the fundamentals of data science, that way you are not using these packages simply as blackbox tools.

One way to understand the functioning of machine learning models is for you to understand the theoretical and mathematical foundations behind every model. As a data scientist, your ability to build reliable and efficient models that can be applied to real-world problems depends on how good your mathematical skills are.

This article will discuss some of the theoretical and mathematical foundations essential for data science practice.

(I) Statistics and Probability

Statistics and Probability is used for visualization of features, data preprocessing, feature transformation, data imputation, dimensionality reduction, feature engineering, model evaluation, etc. Here are the topics you need to be familiar with:

Mean Median Mode Standard deviation/variance Correlation coefficient and the covariance matrix Probability distributions (Binomial, Poisson, Normal) p-value Baye’s Theorem (Precision, Recall, Positive Predictive Value, Negative Predictive Value, Confusion Matrix, ROC Curve) Central Limit Theorem R_2 score Mean Square Error (MSE) A/B Testing Monte Carlo Simulation

For example, Mean, Median, and Mode are used for displaying summary statistics for a given dataset. They are also used for data imputation (mean imputation, median imputation, and mode imputation).

Correlation coefficients and covariance matrix are used to study relationships between various features in the dataset and can be used also for feature selection and dimensionality reduction.

Probability distributions are used for feature scaling, for example, normalization and standardization of features. Probability distributions and Monte-Carlo simulation are also used for simulating data. For example, if the sample data is distributed according to the normal distribution with known mean and standard deviation, then a population dataset can be generated using the random number generator for the normal distribution.

Baye’s theorem is used for model testing and evaluation, and for calculating accuracy score.

Central Limit Theorem (CLT) is one of the most important theorems in statistics and data science. According to CLT, using a sample dataset with larger number of observations for model building is advantageous because a larger sample is a better approximation of the population. Find out more about CLT from here: Proof of Central Limit Theorem Using Monte-Carlo Simulation.

R_2 score and MSE are used for model evaluation. Here is an article in which R_2 score and MSE are used for model evaluation:

Building a Machine Learning Recommendation Model from Scratch.

(II) Multivariable Calculus

Most machine learning models are built with a dataset having several features or predictors. Hence, familiarity with multivariable calculus is extremely important for building a machine learning model. Here are the topics you need to be familiar with:

Functions of several variables Derivatives and gradients Step function, Sigmoid function, Logit function, ReLU (Rectified Linear Unit) function Cost function Plotting of functions Minimum and Maximum values of a function

For examples of how multivariable calculus is used in the machine learning process, please see the following examples:

Building Your First Machine Learning Model: Linear Regression Estimator

A Basic Perceptron Model Using Least Squares Method

(III) Linear Algebra

Linear algebra is the most important math skill in machine learning. A data set is represented as a matrix. Linear algebra is used in data preprocessing, data transformation, dimensionality reduction, and model evaluation.

Here are the topics you need to be familiar with:

Vectors Norm of a vector Matrices Transpose of a matrix The inverse of a matrix The determinant of a matrix Dot product Eigenvalues Eigenvectors

For example, the covariance matrix is a very useful matrix that displays correlations between features. Using the covariance matrix, one can select what features to use as predictor variables. Here is an example of how the covariance matrix can be used for feature selection and dimensionality reduction: Feature Selection and Dimensionality Reduction Using Covariance Matrix Plot.

Other advanced methods for feature selection and dimensionality reduction are Principal Component Analysis (PCA), and Linear Discriminant Analysis (LDA). To understand how PCA and LDA work, you need to understand linear algebra topics such as transpose a matrix; the inverse of a matrix; the determinant of a matrix; dot product; eigenvalues; and eigenvectors. Here are some implementations of LDA and PCA:

Machine Learning: Dimensionality Reduction via Principal Component Analysis

Machine Learning: Dimensionality Reduction via Linear Discriminant Analysis

(IV) Optimization Methods

Most machine learning algorithms perform predictive modeling by minimizing an objective function, thereby learning the weights that must be applied to the testing data in order to obtain the predicted labels. Here are the topics you need to be familiar with:

Cost function/Objective function Likelihood function Error function Gradient Descent Algorithm and its variants (e.g. Stochastic Gradient Descent Algorithm)

An example of how optimization methods are used in data science and machine learning can be found here: Machine Learning: Python Linear Regression Estimator Using Gradient Descent.

In summary, we’ve discussed the essential math and theoretical skills that are needed in data science and machine learning. There are several free online courses that will teach you the necessary math skills that you need in data science. As a data scientist, it’s important to keep in mind that the theoretical foundations of data science are very crucial for building efficient and reliable models.

References