The Fundamental Mathematics of Machine Learning
Source: tai_blog

Table Of Contents Overview  Brief Overview of the Importance of Math in ML  Importance of Math in Machine Learning  Linear Algebra and Calculus in ML  Vector Norms  Linear Algebra in ML  Basic Concepts: Vectors  Matrices  and Operations  Practical Applications in ML  Calculus in ML  Fundamental Concepts: Derivatives and Integrals  Partial Derivatives and Gradients  Chain Rule and Backpropagation  Practical Applications in ML  Linear Algebra and Calculus in Model Training  Linear Algebra in Model Training  Calculus in Model Training  Examples of Model Optimization Using These Math Concepts  Case Studies and Practical Examples  Step-by-Step Walkthroughs of Specific Applications  Conclusion  References  Appendix  Additional Mathematical Proofs and Detailed Examples  Call to Action   OverviewThis blog explores the core mathematical concepts needed to understand and build machine learning (ML) models. Well dive into linear algebra and calculus  showing how they are used in model training and optimization. By the end  youll have a more precise grasp of these foundations and their practical applications.   Brief Overview of the Importance of Math in MLMathematics is the backbone of machine learning. Understanding the underlying mathematical principles behind algorithms allows you to grasp how models work  why they make specific predictions  and how to improve their performance. Two of the most critical areas of mathematics for machine learning are linear algebra and calculus.   Linear algebra handles large datasets with operations like matrix multiplication and transformations and is fundamental in building and optimizing machine learning models. The distance between vectors allows us to normalize our data or add regularization terms to loss functions or as part of transformation through a layer of a deep neural network.   On the other hand  the learning process via Calculus is essential for understanding the changes and optimizations within these models. For example  computing gradients are necessary for training algorithms (e.g.  gradient descent).   Grasping these mathematical concepts enables you to develop more efficient algorithms  troubleshoot issues  and heighten your ability to solve complex problems. By diving into the mathematics of machine learning  you can move beyond treating models as black boxes and start understanding the intricate mechanics that drive them.   My motivation for covering this topic is simple. I taught Computational Methods for Data Science and Machine Learning at Northeastern University and Tufts University  respectively. From this  I have lots of great content that I have recently started to draft as blogs. I needed subsections describing the math or assuming preliminary knowledge of the reader. Hence  I decided to start where I started the course: the math requirements. For the first half of the semesters material  the probability of mathematics will come later before covering probabilistic modeling. Hence  this is the first of several blogs that will be delivered at a level as the following:   From Basic Gates to Deep Neural Networks: The Definitive Perceptron TutorialDemystifying Mathematics  Binary Classification  and Logic Gatestowardsdatascience.com   Now  lets begin!   Importance of Math in Machine LearningA sound foundation in mathematics is essential for anyone aiming to excel in machine learning. Mathematics is not just theoretical; its a practical tool that underpins every aspect of machine learning algorithms. Heres why its crucial:   Model Understanding and Development: Math lets you comprehend how models work at a fundamental level  enabling you to develop or improve new models.Algorithm Optimization: Optimization techniques  grounded in calculus  are crucial for minimizing error and enhancing model accuracy.Data Manipulation: Linear algebra provides the means to handle and manipulate large datasets efficiently  which is fundamental in preprocessing data and training models.Performance Improvement: Math concepts like regularization help prevent overfitting  thus enhancing the models generalization to new data.Problem-Solving: A solid mathematical foundation equips you with analytical skills to systematically approach and solve complex problems.Linear Algebra and Calculus in MLMathematics is deeply integrated into machine learning. Heres an overview of how linear algebra and calculus are applied in various machine learning algorithms:   Linear Algebra in ML   Vectors and Matrices: ML algorithms often use vectors and matrices to represent data. For instance  the entire dataset can be represented as a matrix  with each row being described as a vector (i.e.  a sample in a dataset). If X is the data matrix  each row x represents a data point. See the Vector Norms and Linear Algebra in ML sections for more details.Matrix Operations: Matrix multiplication transforms data  calculates distances  and performs various linear transformations. For example  in a neural network  the input data X is multiplied by a weight matrix W  producing Z = XW. See Additional Mathematical Proofs and Detailed Examples at the end of the Appendix for more details.Eigenvalues and Eigenvectors: These are used in dimensionality reduction techniques  e.g.  Principal Component Analysis  where the data's covariance matrix C is decomposed into its eigenvalues and eigenvectors to transform to a new coordinate system where the data variances rank the axes.Singular Value Decomposition (SVD): SVD is used in recommendation systems and for solving linear systems. For this  we decompose a matrix A into three matrices:where U and V are orthogonal matrices  and  is a diagonal matrix.   Calculus in ML   Derivatives and Gradients: Derivatives measure the rate of change of a function (i.e.  the slope at any given point). In ML  gradients (vectors of partial derivatives) minimize the loss function. For example  in gradient descent  we update the parameter  as follows:   where J is the loss function  and  is the learning rate.   Chain Rule: This is used for backpropagation to calculate the gradient of the loss function for each weight in a neural network. If a function f is composed of two functions  g and h  such that f(x) = g(h(x))  then the derivative of f is as follows:   Optimization Techniques: Calculus-based techniques (e.g.  gradient descent) are essential for training models. These involve computing gradients to update model parameters iteratively to reduce the loss. For example  the update rule in gradient descent for parameter  is   Vector NormsA function f :    is called a norm if it satisfies the following properties:   f is non-negative: f(x)  0 for all x  .f is definite: f(x) = 0 implies that x = 0.f is homogeneous: f(tx) = U+007CtU+007Cf(x)  for all x   and t  .f satisfies the triangle in equality: f(x + y)  f(x) + f(y)  for all x  y  .We use the notation f (x) = U+007CU+007CxU+007CU+007C  which suggests that a norm is a generalization of the absolute value on . A norm can be considered a measure of the length of a vector x   : if U+007CU+007CU+007CU+007C is a norm  the distance between two vectors (x  y)   can be measured through U+007CU+007Cx - yU+007CU+007C.   Example: The Euclidian or -norm is defined as:   Similarly  the sum-absolute-value or -norm is defined as:   And the Chebyshev or _-norm is defined as:   More generally  the Minkowski or -norm of a vector  for p1  is defined as:   For p = 1 and p = 2  the Minkowski norm is precisely the  and  norm defined above. The Minkowski norm can also be defined for p  (0  1]; however  for p  (0  1]  it is strictly not a norm  as it does not satisfy the triangle inequality. The unit ball for a given norm k  k is the set:   An illustration of the unit ball on  induced by different norms is in the following figure. For p = 2  the unit ball is a circle (or a sphere  for n = 3)  while for p = 1 the ball is a square (or a cube  for n = 3). The figure also illustrates that  as p tends to   the  tends to the  norm.   All of the above norms over Rn are equivalent; that is  for any two norms U+007CU+007CU+007CU+007C  U+007CU+007CU+007CU+007C  there exist positive constants      such that:   This implies that the definitions of convergence  function continuity  etc.  below are not norm-dependent. For example  if a sequence converges to a fixed point with respect to one norm  convergence is indeed implied for all of the above norms.   Linear Algebra in MLLinear algebra is critical in machine learning. It provides the foundation for understanding data representations and transformations  essential for developing and optimizing ML models. Lets delve into the basic concepts and their practical applications.   Basic Concepts: Vectors  Matrices  and OperationsVectors   A vector is a one-dimensional array of numbers. Vectors can represent data points in space. For example  a vector in 3D space is:   Vectors represent a dataset's features  with each element corresponding to a specific feature.   Matrices   A matrix is a two-dimensional array of numbers or a vector of vectors. Matrices can represent datasets  transformations  and more. For example  a matrix with m rows and n columns is denoted as follows:   In ML  matrices represent entire datasets  where rows are data points (i.e.  samples)  and columns are features.   Matrix Operations   Addition: Adding two matrices element-by-element.Multiplication: Dot product of two matrices  where the element at the product's i-th row and j-th column is the dot product of the i-th row of the first matrix and the j-th column of the second matrix.Transpose: Flipping a matrix over its diagonal.Notice the swap in index w.r.t. the row and column compared to matrix A above.   Eigenvalues and Eigenvectors   Eigenvalues and eigenvectors are fundamental to understanding linear transformations. Given a square matrix A  then an eigenvector v and its corresponding eigenvalue  satisfy the following equation:   The equation transforms v by A  yielding a scaled version of vector v. Eigenvalues and eigenvectors are often used in ML algorithms like Principal Component Analysis.   Matrix Factorization and Decomposition   Principal Component Analysis (PCA) is a dimensionality reduction technique that transforms the data into a new coordinate system where the greatest variances lie on the first coordinates (i.e.  the principal components). The algorithm is as follows:   Standardize the data: Subtract the mean and divide by the standard deviation for each feature.2. Compute the covariance matrix:   3. Compute the eigenvalues and eigenvectors of the covariance matrix. The directions of the axes where there is the most variance (most information) are in the eigenvectors  and the amount of (i.e.  magnitude) of variance is in the eigenvalues.   4. Sort the eigenvectors by decreasing eigenvalues and select the top k eigenvectors. Hence  the top k eigenvalues capture the most variance.   5. Transform the data using the selected eigenvectors.   Singular Value Decomposition (SVD) is a matrix factorization technique that decomposes a matrix A into three matrices:   where U and V are orthogonal matrices  and  is a diagonal matrix of singular values. SVD is used in recommendation systems  latent semantic analysis  and more.   Thus  it decomposes any linear transformation into a composition of three geometrical transformations. Specifically  a rotation (or reflection) V  then a coordinate-by-coordinate scaling   and another rotation (or reflection).   Practical Applications in MLPrincipal Component Analysis (PCA) reduces the dimensionality of the dataset while retaining as much variance as possible. This makes it easier to visualize the data and reduces the computational cost.   import numpy as np from sklearn.decomposition import PCA import matplotlib.pyplot as plt # Generate synthetic data np.random.seed(0) data = np.random.randn(100  5) # Standardize the data data -= np.mean(data  axis=0) # Apply PCA pca = PCA(n_components=2) data_pca = pca.fit_transform(data) # Plot the results plt.scatter(data_pca[:  0]  data_pca[:  1]) plt.xlabel('Principal Component 1') plt.ylabel('Principal Component 2') plt.title('PCA on Synthetic Data') plt.show()Singular Value Decomposition (SVD) is used in recommendation systems to predict user preferences.   import numpy as np from scipy.sparse.linalg import svds # Example user-item rating matrix R = np.array([ [5  3  0  1]  [4  0  0  1]  [1  1  0  5]  [1  0  0  4]  [0  1  5  4]  ])*1.0 # Apply SVD U  sigma  Vt = svds(R  k=2) sigma = np.diag(sigma) # Predict ratings predicted_ratings = np.dot(np.dot(U  sigma)  Vt) print(predicted_ratings)OUTPUT:   [[ 5.13406479 1.90612125 -0.72165061 1.5611261 ] [ 3.43308995 1.28075331 -0.45629689 1.08967559] [ 1.54866643 1.0449763 1.78873709 3.96755551] [ 1.17598269 0.80359806 1.40136891 3.08786154] [-0.44866693 0.5443561 3.09799526 5.15263893]]   Calculus in MLCalculus plays a pivotal role in understanding and optimizing machine learning models. It provides the tools for analyzing and improving algorithms  particularly in optimization.   Fundamental Concepts: Derivatives and IntegralsDerivatives measure how a function changes as its input changes. Its a fundamental concept in calculus and essential for understanding optimization in machine learning. The derivative of a function f(x) w.r.t. x is denoted as f(x) or   For example  the derivative of f(x) is as follows:   Integrals measure the area under a curve. In machine learning  integrals are less commonly used than derivatives but can be important in understanding distributions and probabilistic models. The integral of a function f(x) over an interval [a  b] is denoted as:   Partial Derivatives and GradientsPartial Derivatives are used when dealing with functions of multiple variables. They measure how the function changes as one of the input variables changes  holding the others constant. The partial derivative of a function f(x  y) concerning x is denoted as   For example  if   then:   Gradients are a vector of partial derivatives and point toward the steepest increase of the function. For a function f(x  y)  the gradient is denoted as f and is given by:   More generally  the gradient for an arbitrary vector-valued function is as follows.   In ML  gradients are used in optimization to minimize the loss function by updating the model parameters in the negative gradient direction.   Example:   Show that the gradient of a quadratic function f(x)= (1/2)xQx + bx +c is F(x) = Qx + b  .   The Taylor expansion of f at a point x is given by:   Hence  the affine function   above approximates the function f near x. Setting z = f (x)  (1) can be written as the following vector inner product:   In other words  Eq. (1) defines a hyperplane of points   that passes through point   and whose normal is given by   This is illustrated in the following figure.   Let us gain further intuition on the physical meaning of the gradient. The gradient at x  perpendicular to the contour defined by   Moreover  f(x) indicates the direction of steepest ascent: following the gradient leads to the largest possible increase of f in the vicinity of x. This is depicted in the following figure.   Chain Rule and BackpropagationChain Rule is a fundamental theorem in calculus used to compute the derivative of a composition of functions. If a function z depends on y  and y depends on x  then the derivative of z w.r.t. x is:   For example  if z = f(y) and y = g(x)  then:   Backpropagation is an algorithm for training neural networks. It uses the chain rule to compute the gradient of the loss function w.r.t. each weight in the network. This allows the weights to be updated to minimize the loss.   The steps of backpropagation are:   Forward pass: Compute the output of the network.Compute loss: Calculate the loss between the predicted and actual values.Backward pass: Compute the gradients of the loss w.r.t. each weight using the chain rule.Update weights: Adjust the weights using the gradients to minimize the loss.Practical Applications in MLGradient Descent is an optimization algorithm that minimizes the loss function by iteratively moving toward the steepest descent  as defined by the negative gradient.   Gradient Descent Algorithm:   Initialize the parameters (weights) randomly.Compute the gradient of the loss function w.r.t. the parameters.Update the parameters in the opposite direction of the gradient by a step size (learning rate).Repeat steps 2 and 3 until convergence.Mathematically  the parameter update rule for gradient descent is:   Where:   Practical Example: Gradient Descent   import numpy as np # Example data X = np.array([1  2  3  4  5]) y = np.array([1  3  2  3  5]) # Parameters m = 0 b = 0 learning_rate = 0.01 epochs = 1000 # Gradient descent for _ in range(epochs): y_pred = m * X + b dm = -2 * np.sum((y - y_pred) * X) / len(X) db = -2 * np.sum(y - y_pred) / len(X) m -= learning_rate * dm b -= learning_rate * db print(fOptimized parameters: m = {m}  b = {b})OUTPUT:   Optimized parameters: m = 0.8015522329369132  b = 0.3943959465768995   Lets try it again with 1 000 epochs.   OUTPUT:   Optimized parameters: m = 0.8000000000000033  b = 0.39999999999998903   Note: the solution approaches its approximate numerical result with a y-intercept of 0.4 and a slope m of 0.8. The following plot depicts this approximation.   Linear Algebra and Calculus in Model TrainingLinear algebra and calculus are indispensable tools in training machine learning models. These mathematical concepts underpin the operations that allow models to learn from data  optimize parameters  and make predictions.   Linear Algebra in Model TrainingData Representation: As mentioned  datasets are often represented as matrices  where each row represents a data point (i.e.  sample)  and each column represents a feature. For example  a dataset with m data points and n features is described as an mn matrix X.Linear Transformations: In many models (e.g.  linear regression)  the prediction is a linear combination of the input features. This can be represented as a matrix multiplication: y=Xw where y is the vector of predictions  X is the input matrix  and w is the vector of weights.Matrix Decompositions: Techniques like PCA use eigenvalues and eigenvectors to reduce the dimensionality of data  making it easier to train models and visualize data. SVD is used in recommendation systems  decomposing the user-item interaction matrix into latent factors.Calculus in Model TrainingOptimization: Calculus  specifically derivatives  minimizes the loss function. The gradient of the loss function w.r.t. the model parameters indicates how the parameters should be updated to reduce errors.Gradient Descent: This is an iterative optimization algorithm used to minimize the loss function. It relies on computing the gradient (partial derivatives) of the loss function w.r.t. the parameters and updating them in the direction of the negative gradient.Backpropagation: In neural networks  backpropagation uses the chain rule to compute the gradients of the loss function w.r.t. each weight. This allows for efficient computation of the necessary updates to minimize the loss.Examples of Model Optimization Using These Math ConceptsExample 1: Linear Regression Using Gradient Descent   Linear regression aims to find the best-fitting line through the data points. The goal is to minimize the mean squared error (MSE) between the predicted and actual values.   Mathematical Formulation:   Hypothesis:Loss Function:Using gradient descent  we update the weights w and bias b:   Weight Update:Bias Update:import numpy as np # Example data X = np.array([1  2  3  4  5]).reshape(-1  1) y = np.array([1  3  2  3  5]) # Add a column of ones to include the bias term in the weight vector X_b = np.c_[np.ones((X.shape[0]  1))  X] # Parameters learning_rate = 0.01 n_iterations = 1000 m = len(y) # Initialize weights theta = np.random.randn(2  1) # Gradient Descent for iteration in range(n_iterations): gradients = 2/m * X_b.T.dot(X_b.dot(theta) - y.reshape(-1  1)) theta -= learning_rate * gradients print(fOptimized parameters: {theta.ravel()})OUTPUT:   Optimized parameters: [0.38853948 0.80317438]   Notice that flipping X above flips the approximation of m and b compared to the gradient descent example demonstrated earlier.   Example 2: Neural Network Training Using Backpropagation   Neural networks are trained using the backpropagation algorithm  which relies on the chain rule to compute gradients efficiently.   Mathematical Formulation:   Forward Pass: Compute the output of the network.Compute Loss: Calculate the loss (e.g.  cross-entropy loss for classification).Backward Pass: Use the chain rule to compute gradients of the loss w.r.t. each weight.Update Weights: Update weights using the calculated gradients.import torch import torch.nn as nn import torch.optim as optim # Define a simple neural network class SimpleNN(nn.Module): def __init__(self): super(SimpleNN  self).__init__() self.fc1 = nn.Linear(2  3) self.fc2 = nn.Linear(3  1) def forward(self  x): x = torch.relu(self.fc1(x)) x = torch.sigmoid(self.fc2(x)) return x # Example data X = torch.tensor([[0.0  0.0]  [0.0  1.0]  [1.0  0.0]  [1.0  1.0]]) y = torch.tensor([[0.0]  [1.0]  [1.0]  [0.0]]) # Initialize model  loss function  and optimizer model = SimpleNN() criterion = nn.BCELoss() optimizer = optim.SGD(model.parameters()  lr=0.1) # Training loop for epoch in range(10000): optimizer.zero_grad() output = model(X) loss = criterion(output  y) loss.backward() optimizer.step() print(Finished Training) print(output)OUTPUT:   Finished Training tensor([[0.0259]  [0.8772]  [0.8772]  [0.1237]]  grad_fn=<SigmoidBackward0>)   Notice the output of X is approaching the true values of y.   Case Studies and Practical ExamplesReal-world Examples Where Linear Algebra and Calculus Have Been Crucial in Developing and Optimizing Machine Learning Models   Linear algebra and calculus are fundamental to many successful ML applications. Lets explore real-world examples of how these mathematical tools have been crucial in model development and optimization.   Image Recognition with Convolutional Neural Networks (CNNs)   Linear algebra is used extensively in the convolution operations that underpin CNNs. Calculus  through backpropagation  optimizes the network by updating the weights to minimize the loss function.   Natural Language Processing (NLP) with Word Embeddings   Techniques like word2vec use matrix factorization to capture the relationships between words. Calculus-based optimization algorithms such as gradient descent are used to train these models.   Recommender Systems   Singular Value Decomposition (SVD) factorizes the user-item interaction matrix. This decomposition allows the system to predict user preferences for items they havent rated yet.   Autonomous Vehicles   Machine learning models for object detection and path planning in autonomous vehicles rely heavily on linear algebra for data representation and transformations. Calculus is used for optimization and control algorithms.   Step-by-Step Walkthroughs of Specific ApplicationsLets dive into two detailed examples: building a recommender system using SVD and training a CNN for image classification.   Example 1: Recommender System Using SVD   Let us break down the example above into digestible steps.   Step 1: Data Preparation   We start with a user-item rating matrix  where rows represent users and columns represent items. The entries in the matrix are the ratings given by users to items.   import numpy as np # Example user-item rating matrix R = np.array([ [5  3  0  1]  [4  0  0  1]  [1  1  0  5]  [1  0  0  4]  [0  1  5  4]  ])*1.0Step 2: Apply SVD   We decompose the rating matrix R into three matrices:   from scipy.sparse.linalg import svds # Apply SVD U  sigma  Vt = svds(R  k=2) sigma = np.diag(sigma) print(U matrix:\n  U) print(Sigma matrix:\n  sigma) print(V^T matrix:\n  Vt)OUTPUT:   U matrix: [[-0.66924125 -0.43689593] [-0.44308727 -0.29717498] [ 0.13631518 -0.51589728] [ 0.11077382 -0.39999635] [ 0.5700326 -0.54282768]]   Sigma matrix: [[6.22925557 0. ] [0. 9.03171974]]   V^T matrix: [[-0.78203025 -0.20891356 0.45754472 0.36801718] [-0.47488998 -0.26234348 -0.3005118 -0.78444124]]   Step 3: Reconstruct the Matrix   We reconstruct the original matrix R using the decomposed matrices to predict missing ratings.   # Reconstruct the matrix R_pred = np.dot(np.dot(U  sigma)  Vt) print(Predicted ratings:\n  R_pred)OUTPUT:   Predicted ratings: [[ 5.13406479 1.90612125 -0.72165061 1.5611261 ] [ 3.43308995 1.28075331 -0.45629689 1.08967559] [ 1.54866643 1.0449763 1.78873709 3.96755551] [ 1.17598269 0.80359806 1.40136891 3.08786154] [-0.44866693 0.5443561 3.09799526 5.15263893]]   Step 4: Evaluate the Model   We can evaluate the model by comparing the predicted and actual ratings for the known values.   from sklearn.metrics import mean_squared_error # Known ratings known_ratings = R[R.nonzero()] predicted_ratings = R_pred[R.nonzero()] # Calculate Mean Squared Error mse = mean_squared_error(known_ratings  predicted_ratings) print(Mean Squared Error:  mse)OUTPUT:   Mean Squared Error: 0.7111239245689356   Example 2: Image Classification with CNNs   Step 1: Load and Preprocess Data   We will use the CIFAR-10 dataset  a popular dataset for image classification.   import torch import torchvision import torchvision.transforms as transforms # Define transformations transform = transforms.Compose([ transforms.ToTensor()  transforms.Normalize((0.5  0.5  0.5)  (0.5  0.5  0.5))  ]) # Load datasets trainset = torchvision.datasets.CIFAR10(root='./data'  train=True  download=True  transform=transform) trainloader = torch.utils.data.DataLoader(trainset  batch_size=100  shuffle=True) testset = torchvision.datasets.CIFAR10(root='./data'  train=False  download=True  transform=transform) testloader = torch.utils.data.DataLoader(testset  batch_size=100  shuffle=False)Step 2: Define the CNN Model   We define a simple CNN with convolutional  pooling  and fully connected layers.   import torch.nn as nn import torch.nn.functional as F class SimpleCNN(nn.Module): def __init__(self): super(SimpleCNN  self).__init__() self.conv1 = nn.Conv2d(3  6  5) self.pool = nn.MaxPool2d(2  2) self.conv2 = nn.Conv2d(6  16  5) self.fc1 = nn.Linear(16 * 5 * 5  120) self.fc2 = nn.Linear(120  84) self.fc3 = nn.Linear(84  10) def forward(self  x): x = self.pool(F.relu(self.conv1(x))) x = self.pool(F.relu(self.conv2(x))) x = x.view(-1  16 * 5 * 5) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x net = SimpleCNN()Step 3: Define the Loss Function and Optimizer   We use cross-entropy loss and stochastic gradient descent (SGD) for optimization.   import torch.optim as optim criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(net.parameters()  lr=0.001  momentum=0.9)Step 4: Train the Model   We train the CNN by passing the data through the network  computing the loss  and updating the weights using backpropagation.   for epoch in range(6): # Loop over the dataset multiple times running_loss = 0.0 for i  data in enumerate(trainloader  0): inputs  labels = data optimizer.zero_grad() # Zero the parameter gradients outputs = net(inputs) loss = criterion(outputs  labels) loss.backward() optimizer.step() running_loss += loss.item() if i % 100 == 99: # Print every 100 mini-batches print(f[{epoch + 1}  {i + 1}] loss: {running_loss / 100:.3f}) running_loss = 0.0 print(Finished Training)OUTPUT:   [1  100] loss: 2.304 [1  200] loss: 2.303 [1  300] loss: 2.304 [1  400] loss: 2.302 [1  500] loss: 2.301 [2  100] loss: 2.299 [2  200] loss: 2.297 [2  300] loss: 2.295 [2  400] loss: 2.293 [2  500] loss: 2.288 Finished Training   Step 5: Evaluate the Model   We evaluate the trained CNN on the test dataset to measure its performance.   correct = 0 total = 0 with torch.no_grad(): for data in testloader: images  labels = data outputs = net(images) _  predicted = torch.max(outputs.data  1) total += labels.size(0) correct += (predicted == labels).sum().item() print(fAccuracy of the network on the 10000 test images: {100 * correct / total:.2f}%)OUTPUT:   Accuracy of the network on the 10000 test images: 18.06%   Random would be 10%  as there are ten classes. Still  can we not do better than 18.06%: certainly  lets train more. Here is the output for continuing to train the same model for six additional epochs.   OUTPUT:   [1  100] loss: 2.281 [1  200] loss: 2.267 [1  300] loss: 2.242 [1  400] loss: 2.199 [1  500] loss: 2.132 [2  100] loss: 2.085 [2  200] loss: 2.017 [2  300] loss: 1.993 [2  400] loss: 1.956 [2  500] loss: 1.923 [3  100] loss: 1.898 [3  200] loss: 1.863 [3  300] loss: 1.841 [3  400] loss: 1.810 [3  500] loss: 1.767 [4  100] loss: 1.753 [4  200] loss: 1.729 [4  300] loss: 1.693 [4  400] loss: 1.664 [4  500] loss: 1.663 [5  100] loss: 1.644 [5  200] loss: 1.635 [5  300] loss: 1.603 [5  400] loss: 1.621 [5  500] loss: 1.590 [6  100] loss: 1.590 [6  200] loss: 1.572 [6  300] loss: 1.570 [6  400] loss: 1.556 [6  500] loss: 1.553 Finished Training   Accuracy of the network on the 10000 test images: 43.06%   And six more epochs:   OUTPUT:   Accuracy of the network on the 10000 test images: 51.91%   Many other methods to improve multi-layer perceptions are outside the scope of this piece and will be the subject of future blogs.   ConclusionIn this blog  weve explored the foundational mathematical concepts that underpin machine learning.   Understanding the mathematical foundations of machine learning is not just a theoretical exercise; its a practical necessity for anyone serious about mastering this field. Heres why:   Model Development: A solid grasp of mathematics allows you to develop and improve new algorithms. You can move beyond using machine learning models as black boxes and start customizing them to suit your needs betterOptimization: Many machine learning algorithms rely on optimization techniques derived from calculus. Understanding these principles lets you tune your models more effectively and improve their performance.Data Handling: Linear algebra is essential for handling and manipulating data efficiently. Operations involving matrices and vectors are ubiquitous in machine learning  from data preprocessing to model evaluation.Troubleshooting: A deep understanding of the math behind machine learning helps you diagnose and fix issues more effectively. You can identify why a model is underperforming and make informed adjustments.Due to the limited ability to typeset mathematical expressions on Medium  various bits were omitted. I encourage readers to check out the PDF much of this material was derived from (i.e.  the write-up created by my students in past semesters). Access PDF here: https://www.dropbox.com/scl/fi/5jr8atjpcb2ekyg3ebh4v/Math_Background-1.pdf?rlkey=3f8cy793s6veqa7yuadv5mm95&st=nvif67ea&dl=0   ReferencesGoodfellow  I.  Bengio  Y.  & Courville  A. (2016). Deep Learning. MIT Press.Strang  G. (2016). Introduction to Linear Algebra. Wellesley-Cambridge Press.Bishop  C. M. (2006). Pattern Recognition and Machine Learning. Springer.Press  W. H.  Teukolsky  S. A.  Vetterling  W. T.  & Flannery  B. P. (2007). Numerical Recipes: The Art of Scientific Computing. Cambridge University Press.Rumelhart  D. E.  Hinton  G. E.  & Williams  R. J. (1986). Learning representations by back-propagating errors. Nature  323(6088)  533536.Koren  Y.  Bell  R.  & Volinsky  C. (2009). Matrix factorization techniques for recommender systems. Computer  42(8)  3037.Krizhevsky  A.  Sutskever  I.  & Hinton  G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 10971105).AppendixAdditional Mathematical Proofs and Detailed ExamplesProof of Gradient Descent Convergence   Gradient descent aims to find the minimum of a function by iteratively moving in the direction of the steepest descent  defined by the negative gradient. Consider a simple convex function f(x) with its gradient f(x).   Theorem: If f(x) is convex and differentiable  and the learning rate  is sufficiently small  gradient descent converges to a local minimum.   Proof:   Initialization: Start with an initial guess x.Iterative Update: Update the parameter x usingAssume f(x) is L-Lipschitz continuous  which means:   Descent Property: Using the Taylor series expansion for f(x):Substitution:   Simplifying:   Convergence: For convergence  the termmust be positive  implying   With a suitable learning rate   gradient descent ensures   leading to convergence to a local minimum.   Detailed Example: Matrix Multiplication in Neural Networks   Consider a neural network layer with an input vector x and weight matrix W. The output y is given by:   For example:   The output   Call to ActionWe hope you found this blog insightful and helpful in understanding the importance of mathematics in machine learning.   If you enjoyed this content  please subscribe to our newsletter to stay updated on future posts. I value your feedback and encourage you to comment with your thoughts  questions  and suggestions. Dont forget to share this blog with your peers and colleagues who might find it helpful.   https://jvision.medium.com/   Follow Dr. Robinson on LinkedIn and Facebook. Visit my homepage for papers  blogs  email signups  and more!   AI Research Engineer and Entrepreneur U+007C Joseph P. RobinsonResearcher & Entrepreneur Greetings! As a researcher  Dr. Robinson proposed and employed advanced AI to understandwww.jrobs-vision.com.   Thank you for reading  and happy learning!