Math for Machine Learning: A Motivation
By: Geraldi Dzakwan
Tags: Deep Learning, Machine Learning, Math, Data Science, Statistics

Which Math Exactly?

I think this is the big question, but believe me, the answer is fairly simple. Although the proper answer should be something more thorough, I would like to mention only these two for simplicity:

1. Probability and Statistics (ProbStat)

This is, I guess, what most of you would expect. Of course, you have to master all those distribution stuff (random variable, expectation, plug-in principle, MLE, bias, etc.), including the legendary Bayesian inference (prior distribution, posterior distribution, Bayes estimate, conjugate prior, etc.). However, I want to emphasize that some of you (in regards to your career goal) might not need to actually study this topic in-depth, high-level understanding should suffice. The people I know that need to have a decent understanding of this topic are those who work as a (proper) Data Scientist or as a Quant Engineer.

Data Scientist, of course, needs to know ProbStat in-depth as they, day to day, define the right metric for the problem, design and execute experiments/simulations (this includes defining the right sample) and ultimately define which action to take based on the result of some testing scheme, e.g. A/B testing. Quant Engineers, for those of you who are not familiar with, are those who develop and implement complex mathematical models that financial firms use to make decisions about risk management, investment, and pricing. In Indonesia, this kind of job might not be as popular. But, I believe there are actually some of them, although they might go under different names. In fact, I think actuarial jobs could be classified as Quant jobs. I have an acquaintance who is an actuary and he just simply masters this field of math. Every time I have a question/problem set regarding ProbStat, he could always solve it in minutes, which I find very amusing.

2. Linear Algebra and Calculus (LinAlg & Calc)

This is, I guess, what fewer of you would expect. But, believe me, if you come from a Comp Sci background like me (and not Math), you will probably fancy this topic more than ProbStat. I’ll touch that in a bit.

Since this topic is very broad, I guess you are questioning which LinAlg & Calc exactly is relevant for ML. Hence, below I present you with some of the most important topics that you need to comprehend (or at least know about it in a nutshell), sorted from what I reckon the most important:

Vector And Matrix Operation/Manipulation

Eigenvalue and Eigenvector

Singular Value Decomposition and Spectral Theorem

Function Convexity

Partial Derivative and Chain Rules

Optimization Algorithm (e.g. Gradient Descent families)

Kernel Trick

Matrix Approximation

If the above topics don’t sound familiar to you, it’s perfectly fine. When I was first starting my journey at Columbia, I struggled a lot to meet the prerequisites for the Machine Learning course (COMS W 4771). The Prof was Daniel Hsu, which was an amazing lecturer and theoretical ML researcher, go look him up (https://www.datascience.columbia.edu/daniel-j-hsu)! It probably took me around a month to really be comfortable following the lecture as I needed to catch up with the prerequisites while concurrently being in the course.

That being said, the above topics play very important roles in how you see Machine Learning. Machine Learning is often said to be “glorified Statistics”. Not that I disagree, in fact, I somewhat agree. But, I feel like LinAlg & Calc are less emphasized for ML compared to ProbStat. I would even argue that LinAlg & Calc contribute more to the advancement of Machine Learning (or Deep Learning specifically) nowadays. An example would be the Optimization Algorithm which leads to the resurgence of Deep Learning. Optimization Algorithm (such as SGD) makes computation far lighter while still trying to be accurate up until a certain degree (i.e. trying to get the best local minimum if you recall the concept). Thus, I prefer to say that Machine Learning is “glorified Maths” if that makes any sense to you.

In relation to practical works, I also want to point out that LinAlg & Calc is miles more important, particularly to make computation as efficient as possible while preserving correctness to some degree. Let’s not talk about ML and instead talk about mere matrix multiplication. Noobs like me would certainly do it in cubic time. For folks who are familiar with algorithms, they might know and implement some more efficient methods such as the Strassen algorithm which does it a little bit faster than cubic time (n to the power of 2.8 to be exact, n being the size of the input).

From that simple example, I believe you could follow where I’m heading. There are just so many problems regarding numerical computations out there. Part (or most I should say) of them is solved by Numpy, a Python package built to optimize numerical operations. Numpy is then used by the bigger guys like Scikit-Learn and Tensorflow. This is just mere numerical computation which is probably on the lowest level of the Machine Learning hierarchy (this, I made up myself, so please don’t quote me on this one). Problems such as :

How we could efficiently compute Eigenvectors (e.g. for PCA) for large sparse matrix or rectangle matrix

How we could efficiently find the simplest linear solution when the matrix is not full rank (i.e. the solution is infinitely many) or when the matrix we deal with is not invertible

How we could design an effective optimization algorithm that enables us to deal with a huge load of data while being low on resources (see Batch Optimization for starters)

How we could come up with the best initialization parameter given a certain architecture of a neural network

How we should design a neural network (particularly in regards to its activation function) so that the gradient doesn’t vanish greatly (see LSTM and GRU architecture for starters)

are on the higher hierarchy (again, I just made this up). There are just so many things to look into and the solution usually goes around LinAlg & Calc.

If you aspire to become a (proper) ML Engineer, which I also aspire to be, where you build an efficient ML-related code for your products (either by carefully choosing the right package and algorithm or even when you go hardcore and write the algorithm from scratch, yikes), then you should consider learning about LinAlg & Calc topics that I mention above. I guess an intermediate level of understanding should suffice, measured by these factors (again, I made this up):

You’re able to identify the right solution (preprocessing, feature representation, model, postprocessing) for a certain problem, taking into account correctness (meaning that your solution is optimal to a degree that your institution/company desires) and efficiency (meaning that your solution could infer within a certain time limit)

You’re able to detect a mathematical bottleneck in a certain ML product

When needed, you could tweak certain ML frameworks (e.g. Tensorflow) a wee bit so that it matches your needs, for example using Tensorflow Subclassing API

If you have a lot of free time and want to dive even deeper until you have an advanced level of understanding, you should go for it. At that level, you should be able to write proofs or derive solutions by hand for convoluted problems (e.g. proving that a certain Optimization Algorithm could give a solution for a particular problem within a certain number of steps with an error no more than a certain epsilon). But, that is on a-whole-nother level and I don’t see myself to be able to be somewhere near that in the near future. Usually, folks who have a PhD in Machine Learning or relevant fields (Maths, Stats, Physics) are the ones you could always rely on to do jobs on that level.

I hope the explanation above is clear and easy to follow for folks who have Comp Sci and/or Math background. My apologies for folks outside those backgrounds because I need to use many technical terms, otherwise I think the depth of this article would be too shallow.

Closing Words

I think that would be all for now. Over the course of this last 3–4 months of 2020, I hope I could create more posts, preferably some more technical ones, should I have the time and resources to do so. Kindly let me know in the comment section below what topics would you like to see regarding Math for Machine Learning!

I imagine I could even go to talk about some ML problem sets from the coursework I’ve taken at Columbia, which should be interesting to share for you who are interested to see the examples of math often involved in ML (and how rigorous they are for f***’s sake) or for those of you who are planning to take some graduate-level courses around ML in the future. In the meantime, stay safe, and do care for others!

Cheers,

Geral