A Commentary on Google’s Machine Learning Crash Course
By: Rajanarayanan Thottuvaikkatumana
Tags: TensorFlow, Google, Artificial Intelligence, Google Cloud Platform, Machine Learning

Machine Learning (ML) is a branch of Artificial Intelligence (AI) like Algebra is a branch of Mathematics. In this story, I am not planning to delve into defining what is Machine Learning or Artificial Intelligence. Instead, I am going to present some of the materials already prepared by the best in the industry and give my commentary on them. In essence, this story doesn’t give you any original ideas you can learn ML from, rather use this as a guide to go through some of the already available ML contents.

Machine Learning Crash Course

Google has published a Machine Learning Crash Course and it is one of the first places you can go and start your ML journey from. In this story, if I use the word course, it is about the ML Crash Course delivered by the Google team. This course is divided into multiple parts and each part has multiple sections. This course gives a really good foundation of ML and the tooling available from Google. It consists of very short videos, notes of the content mentioned in the video, and some tests to check your understanding. The course starts with a very short introductory video and then moves into the following topics.

Figure 1 — Screenshot taken from https://developers.google.com/machine-learning/crash-course/

It is important to note that the contents in the following sections are written based on the information available in the course at the time of this writing. I have also tried to provide hyperlinks to other sources such as Wikipedia if you are interested in reading additional material on key concepts and terms.

ML Concepts

The following sections in this part of the course explain the basics of ML and introduce the vocabulary. Going through these sections of this part multiple times and doing further reading outside the course materials will reinforce the learning.

Framing

If you have a task to do, you can write computer programs to do that. But how can you frame that as an ML problem? Or can you frame it as an ML problem at all? The framing section of the course scratches the surface of framing an ML problem. In addition to that, this section introduces some of the key vocabulary terms in the ML estate. This section introduces the concepts of Labels, Features, Examples a.k.a. Training Data, and Models in the ML parlance. It also gives a brief introduction about two important and popular ML models 1) Regression Model and 2) Classification Model. Mathematically speaking, an ML model is also known as a Prediction Function.

Descending into ML

Linear Regression fits a set of points to a straight line. This is one of the easiest ML models that you can start with. Anyone who understands plotting a graph using X and Y axes can understand this model. That is what precisely happens in this section. It talks about the Linear Regression ML model, training the model, and calculating the training loss. It is imperative to understand these concepts well. There are some excellent notes with illustrations available in this section that helps you understand these foundational concepts really well.

Reducing Loss

In any ML model, it impossible to get a perfect prediction for the data that is unseen by the ML model. In other words, there will be some differences between the value predicted by the model and the actual value in the test data, and this is called the ML model’s Loss. Obviously, you can’t make the loss zero but if you can build an ML model with minimal loss. There are many techniques that are used to reduce the loss in an ML model. This section talks about that and just that. Reducing training loss is mostly an iterative approach. The loss calculation method using Gradient Descent is being introduced here. A decent understanding of the concepts like Straight-line Equations, Slope, Differentiation, Local Minima, and Global Minima in Mathematics will come in handy here. The concepts of Learning Rate and Step Size in reducing the loss of an ML model is introduced here. A cameo description of ML Algorithms is given here. There are parameters that programmers can play within these algorithms and they are called Hyper Parameters. This section goes on to describe optimization problems in the context of reducing the training loss by optimizing the learning rate. The Stochastic Gradient Decent (SGD) method of reducing the loss is introduced here. Brushing up the concepts of Scalar, Vector, Direction, and Magnitude components of Vector in Mathematics is going to be a good idea as well.

First Steps with TF

TensorFlow is the heart of ML in Google’s AI offerings. It is open source and can be used for general-purpose scientific computing as well. Introduction to TensorFlow materials and programming exercises are covered in this section. Familiarity with Python programming, NumPy, Pandas, and Keras is required for mastering TensorFlow.

Generalization

When an ML model perfectly works fine with the training data but behaves badly with new data during prediction, it is an Overfitting problem with the ML model. This section talks about the perils of overfitting and how can you Generalize an ML model. It has alluded here that either the ML model is too complex or it is too tied with the training data.

Training and Test Sets

You need a totally different set of data for training and testing your ML models. It is critically important to randomize your labeled data first and divide them into two sets where you use the bigger data set for training your ML model and the remaining for testing your ML model. It is also very important that you are not training your model with your test data. Accidents can happen and watch out for ML models that give zero training loss on test data.

Validation Set

The previous section described the need to separate the labeled data into training and test data sets. Since the training, validation, and testing of the ML model is an iterative process, if you have only two sets of data, and if you tune the hyperparameters too much, there is a chance that the peculiarities of the test data will get into the ML model resulting in overfitting the ML model. To alleviate this problem, validation data is also prepared. Now the ML model can be trained with training data, validate the model with validation data, complete the iteration, and make sure that the results are good with test data. In essence, it is super critical that there must be three district data sets for making sure that an ML model is ready to give good results.

Representation

The data comes in a myriad of formats and making sure that they are ready to be consumed by the ML systems is the art of Feature Engineering. This section gives a very high-level introduction to the techniques of data transformation to support ML modeling. Then this section goes into the qualities of good features in data or how to represent your data to be used by ML systems. Mathematical concepts like Scalars, Vectors, Matrices, and Matrix Operations are nice to have to know from here. That is not enough, it is time to touch upon Statistics as well here mainly to play with the numbers. Statistical concepts such as Scaling, Binning, Scrubbing, and Summary Statistics are mentioned in this section.

Feature Crosses

Many times the individual features in the data on its own may not be sufficient to make make an ML model work. At times, combining multiple features and producing synthetic features make the model simple and effective. These synthetic features are called feature crosses created by multiplying or by creating cross products. This is a very important concept in ML. This section gives an introduction to Feature Crosses and some practical examples.

Regularization: Simplicity

Simple ML models are easy to maintain. It is a common practice to calculate both training loss and validation loss doing the iterative ML model design. In a previous section, the concept of overfitting was introduced. How do we make sure that the ML model complexity is reduced and overfitting is avoided? Regularization concepts come in handy here and this section talks about that. L2 Regularization technique is explained but there is also another one available namely L1 Regularization.

Logistic Regression

Modeling probability of certain classes or events uses Logistic Regression. This is not to be confused with Linear Regression. Linear Regression predicts continuous values and Logistic Regression predicts continuous values between 0 and 1.0 — both inclusive. Discrete classification such as true/false, boy/girl, benign/malignant can be derived out of the output values from Logistic Regression. This section of the course gives an introduction to Logistic Regression. Understanding the basics of Probability Theory, Sigmoid Function, and Baye’s Theorem is essential to understand Logistic Regression. Logistic Regression brings more challenges related to training loss. Regularization techniques used in conjunction with Logistic Regression are also equally challenging. This ML model uses totally different set of loss functions and they can be seen from Loss Functions for Classification in Wikipedia.

Classification

Logistic Regression is widely used for classification problems in ML. The probability value returned by Logistic Regression can be used as it is or with the appropriate Classification Threshold to convert to a binary value such as true/false, boy/girl, benign/malignant. This section talks about classification problems in ML, and various techniques to check the accuracy of the prediction. The concept of the Confusion Matrix is introduced here. The metrics calculated out of the confusion matrix are used in a wide variety of contexts not only in ML but also in many other fields.

Regularization: Sparsity

This is the second section in this part talking about regularization and here the L1 Regularization is introduced. How the lack of doing regularization affecting the computing requirements and the need to have a huge amount of memory is discussed here. This section also gives a really good comparison of L1 and L2 regularization techniques and where each one of them can be applied.

Neural Networks

Many of the classification models need not be linear in nature. A typical liner model can have multiple inputs and one output. Neural networks use multiple layers between inputs and outputs. Whatever you do, if you don’t apply transformations that convert a linear model to a nonlinear model, neural networks will still be a linear model. This section talks about linear ML models. It discusses the concepts of multiple hidden layers and talks about Activation Functions that introduce nonlinearity in the hidden layers of neural networks. Typically used activation functions such as Sigmoid, and ReLU is also described.

Training Neural Nets

Training various ML models were discussed in earlier sections of this course. Training Neural Networks is a totally different ballgame and it is done using the algorithm Backpropagation. Gradient Descent was introduced earlier and Backpropagation uses it very effectively and TensorFlow is designed from the ground up to do this automatically. More than talking about the happy path use cases, this section talks about the failure use cases of the Backpropagation algorithm. The need to regularize is a requirement in neural networks as well. The regularization technique used here is to randomly drop out units in a network. There is only a minimal amount of content in this section, but if you unpack the concepts introduced in this section, you will have loads of questions and it is advisable to refer to outside materials to get a full grasp of the concepts.

Multi-Class Neural Nets

Logistic Regression predicts a binary outcome. In the real world, many classification problems are not binary in nature and you will have to classify things into multiple categories. This can be accomplished by the One vs. All method and this is very effective when the number of categories is small in number. When the number of categories is high, Deep Neural Networks come in very handy. Here the number of features goes through a graph of a huge number of deep layers and produces a given number of output nodes also called Logits. Each of the output nodes represents the desired class in the multi-class classification problem. The Softmax function is also introduced here that calculates probabilities for each class and summing up to 1.

Embeddings

This section delves into a totally different area which is Recommendation Systems. Everyone starts with a movie recommendation system wherein there are many movies and many users and the problems can be very large in number. These two simple dimensions of data if you represent it in a matrix, it is going to be very sparse. If you want to preserve the relationship between these sparse dimensions, converting them to a lower-dimensional space can help a lot and this is called Embedding. This conversion process is explained well in this section and a little bit of extra reading will help.

ML Engineering

This part of the course is dedicated to the engineering aspects of ML whether it is building ML pipelines to serve production ML systems or doing Feature Engineering to make sure that the ML systems are getting the best possible data. This part contains the following short sections.

ML training algorithms are available in abundance. The herculean task on hand is to make the entire pipeline work in production to deliver business value. In the whole workflow, there is only a tiny bit in the ML space. There is a research paper titled Hidden Technical Debt in Machine Learning Systems authored by a few Google scientists that talk about it in detail.

There are multiple approaches to training ML models. The most common ones are 1) Static Training and 2) Dynamic Training. This section talks about both of them and their pros and cons.

Making predictions or inferences out of the trained ML models can be done in multiple ways as well and they are 1) Online and 2) Offline. This section talks about both of them and their pros and cons.

ML systems depend heavily on data. Data could come from upstream systems, goes through transformations, goes through feature engineering processes, and finally becomes part of the training, validation, or test data sets. The less you have a dependency on other systems, the better. Aspects like the reliability of the data and versioning are also important. This section talks in general about what are the good practices that you can follow when dealing with data to be used in ML systems.

Data influences predictions coming out of the ML systems. Biases in training data will reflect the predictions as well. This section deals with designing ML systems for fairness. Many good pointers to avoid data bias and improving fairness is discussed in this section.

ML Systems in the Real World

This is the last part of this course with a few sections. A few real-world ML systems are given in this concluding section. When it comes to the ML model design, data pipeline preparation, logging and monitoring using the right metrics, maintaining almost everything as code, and finally the importance of keeping an up to date documentation of the experiments are covered in a concise fashion.

Conclusion

To be honest, this ML crash course from Google is deceptively simple. No doubt that this is an absolutely right place to start. But don’t expect that this will give you a jump start and you will be able to design ML systems right away. You need to unpack all the foundational elements covered in this course and go out and read as much as possible. If you don’t have high school level training in Mathematics and Statistics, brush up on the concepts and it is critical to understand the vocabulary. Like any subject, ML is a vast field and everyone is not cut for everything. So explore and find yourself a place somewhere in the ML ecosystem that you enjoy and good at.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬