# Understanding Machine Learning (I)

[Understanding Machine Learning](https://www.amazon.com/Understanding-Machine-Learning-Theory-Algorithms/dp/1107057132) is another book on [Miri's research guide](https://intelligence.org/research-guide/). I've previously written about [Topology](https://www.lesswrong.com/posts/rYumRQK3G9cqvaxQ7/insights-from-munkres-topology) and [Linear Algebra Done Right](https://www.lesswrong.com/posts/BgEfvxBHPfHdaQxLH/insights-from-linear-algebra-done-right). With this one, my write-up ended up looking a lot more like a self-contained introduction into the field and a lot less like a review. I've decided to go with that, and make this a sort of shortened version of the book.

Throughout this series, I will be using **_bold and italics_** to mean "this is the first official mention of a new Machine Learning term", whereas normal italics means regular emphasis.

The Learning Model
==================

The book begins with a sort of mission statement. Given a **_domain set_**  X and a **_target set_** Y, we wish to _find a function_ h:X→Y, which we interchangeably call a **_hypothesis_** or a **_predictor_** or a **_classifier_**. In the example used in the first chapter, X represents the set of all [papayas](https://en.wikipedia.org/wiki/Papaya), and Y represents a two-element set where one element is the label "tasty" and the other the label "not-tasty". To compress this into something manageable, we represent papayas by a set of _**features**,_ in this case _color_ and _softness_, which we both map onto the interval \[0,1\], and we represent the two labels by 1 and 0. I.e. X=\[0,1\]2 and Y={0,1}. Now the idea is that, if we learn such a predictor, we can apply it to any new papaya we encounter, say on the market, and it will tell us whether or not it is tasty before we've spent any money on it.

This model seems quite general. It covers chess, for example: one could let X be the set of all possible states of the chessboard and Y be the set of all moves; then we want h to map each x∈X onto the best possible move. In fact, it is hard to see which solvable problem could not be modeled this way. Any time we think we can in principle solve some problem, it would consist of us looking at some stuff and then presenting the solution in some way. Now we just let the set of all possible instances of such stuff be X and the set of all possible solutions be Y, and we have an instance of this model.

One would have to put in a lot more work to fully formalize this problem – and this is exactly what the book does. Recall that X is called the _domain set_. If Y is finite as it is for the papayas example, we also call it the _**label set**_.

We model the _**environment**,_ which creates our labeled domain points (x,y) by a probability distribution D over X×Y as well as the _i.i.d. assumption_ (not bolded since it's not ML-specific), which states that all elements are _independently_ and _identically_ distributed according to D. In other words, whenever a papaya is plucked in the real world, the environment throws some coins and chooses its color and softness, as well as tastiness, according to D. Of course, we generally assume that the color and softness have a major impact on the tastiness of the papaya, but it is not reasonable to assume that they _fully_ determine its tastiness – surely it is possible that two papayas that are equally colored and equally soft taste differently. Thus, given some particular x, we might reasonably hope that either (x,0) or (x,1) is going to be a lot more probable than the other, but both are still going to be possible. On the other hand, in the chess example, one could argue that the domain set fully determines the value of the target set, i.e. the best possible move (although then the idea of having a probability distribution over X becomes problematic). In such a case, one can alternatively model the environment as having a "true" function f:X→Y. This approach is strictly less general than the former (it corresponds to a probability distribution where for every x, either (x,0) or (x,1) has probability 0), but in both cases, the joint distribution / true function is not known, and it is exactly what we are trying to figure out.

We say that the algorithm A which outputs a predictor h:X→Y is the _**learner**._ As its input (assuming we are doing what is called _**supervised learning**_ here), the learner has access to a _**training sequence**_ S. This is a sequence of labeled domain points, i.e. S=((x1,y1),...,(xn,yn)) which _have been generated by the joint distribution_ D. For this reason, one can consider the training sequence to be a glimpse into how the environment works, and the learner's job to be to _generalize_ from that sequence to all of D, i.e. to output a predictor h which does a good job on not just the training data but also on new elements that are generated by D.

We also define a **_loss function_** ℓ to formalize what we mean by "good job". The loss function takes a hypothesis and outputs a number in \[0,1\] that is meant to be a rating of how good the hypothesis is, where small values are good and large values are bad. For the case of **_binary classification_** (where |Y|=2), we will set ℓ(h):=D{(x,y)∈X×Y|h(x)≠y}, i.e. the probability that our predictor gets the label on a freshly sampled point wrong (although in practice, one might also be interested in loss functions that penalize false positives more strongly than false negatives, or vice versa). A perfect predictor will have loss 0, but in the setting of a joint distribution D, this will generally be impossible. How good the best possible predictor will perform depends on how well the feature vector describes the domain points. With a poor choice of features, the learner is doomed to fail from the beginning (like if you want it to recognize spam emails, but you only show it aspects of the emails such that there is at best a statistically weak correlation between them and whether the email is spam or not-spam). Of course, we cannot evaluate our loss function since we don't know D, but it is nonetheless possible to make statements about the expected loss of the predictor which a learner outputs.

Finally, we define an **_empirical loss function_** ℓS which describes the error of a predictor _on the training sequence_, i.e. ℓS(h):=|{(x,y)∈S:h(x)≠y)}||S|. Unlike the regular loss function, the empirical loss function ℓS is something we can evaluate. We use the term **_true error_** to refer to the output of ℓ, and **_empirical error_** to refer to the output of or ℓS.

This describes (almost) all objects of the learning model. Now the question is how to implement the learner A. Formally, A is described as simply an algorithm, so we have complete freedom to do whatever we want.

Generalization / Prior Knowledge / Overfitting
==============================================

The learner A takes a training sequence, does whatever it wants with it, and outputs a predictor. How would we go about defining A? A reasonable-sounding starting point is to choose a predictor which performs well on the training sequence S. Given that S is representative of the environment (recall that it was generated by D), the hope is that such a predictor would also perform well on the environment.

This alone does not answer the question, though. Suppose we are given the following training sequence (or -set as we will ignore order for now):

![](https://i.ibb.co/2g6XM6k/01.png)

(Black points ↔ tasty instances, red points ↔ not-tasty instances.) There are many ways to construct a predictor with zero error on this training set. Here is one (namely the predictor that labels points as tasty iff they are in the blue area):

![](https://i.ibb.co/PM1MYwy/02.png)

This does not seem like a good predictor. For example, the green point I added (which is meant to be a new instance not part of the training sequence) would almost certainly correspond to a tasty papaya, yet our hypothesis predicts that it is not tasty. Nonetheless, this predictor has zero error on the training sequence (by the definition of ℓS). An even more extreme example of this is Amemorize, which simply remembers all domain points and outputs a hypothesis that only labels those instances as positive that have occurred in the training sequence with a positive label. The process of outputting such a hypothesis is often called **_overfitting_**. It's said that the hypothesis fits the data "too well,_"_ as the book puts it. But I think this is not quite the right way to look at it. Consider this alternative hypothesis:

![](https://i.ibb.co/P63BS9W/03.png)

This hypothesis seems better, but not because it is a worse fit for the data – both hypotheses have zero empirical error. There is nothing wrong with fitting the data, there is only something wrong with fitting the data _at the cost of_ _outputting an "unreasonable looking" predictor._ I think the way to arrive at a more accurate description is by looking at it from a Bayesian perspective. We have a prior on how a proper predictor will look like, and this prior says that the first predictor is worse (as illustrated by how it classifies the green point). However, suppose we had a new example right next to the green point, and it was not-tasty. We would probably still dislike that predictor – the one not-tasty papaya could be an outlier. On the other hand, if we had a million sample points right next to the green one, all of which not-tasty, we would eventually admit that something super weird is really going on here and a very particular softness-color combination really does lead to papayas being not-tasty.

I think the correct framing of the overfit/underfit dilemma is that it is about _how far to update away from priors based on evidence,_ or alternatively, _how to weigh priors vs evidence_. It's also worth noting that if we had a training sequence that included _all_ possible instances (assuming there is a true labeling function for the sake of this example), we could disregard priors altogether – in that case, Amemorize would work work perfectly. Analogously, there is always some amount of evidence that should overturn [any prior](https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities). Of course, in our formal model, there may be infinitely many possible feature vectors, but in the real world, every instance description is bounded so the space of all instances must be finite.

One crude way of deciding how much to weigh priors vs evidence is by committing to a **_hypothesis class_** ahead of time – i.e. to a subset H⊂{f:X→Y} – and then finding a predictor that performs maximally well on the training sequence, only taking into considerations predictors in the class we committed to (i.e., a predictor in the set argminh∈H\[ℓS(h)\]). It is crude because it only allows the prior to give a binary "yes" or "no" to a hypothesis, rather than a "probably not, but maybe with enough evidence". Suppose (for the papaya example), we pre-commit to the class of _axis-aligned rectangles_, as the book suggests (i.e. rectangles with two horizontal and two vertical sides). Consider the following training sequence:

![](https://i.ibb.co/R4Z7jmy/04.png)

This does not seem like that unreasonable of a glimpse into reality – perhaps it is really the case that if green papayas are too soft, they become gross, but if they're more orange or whatever, then it's fine if they're soft. You might not expect that to be the case, but you'd probably assign it some reasonable probability. This is not allowed by the class of axis-aligned rectangles: cording to this class, the only possible predictor is one where both a certain softness area and a certain color area have to be hit, and one factor does not influence the other. Therefore, if we applied a learning algorithm to this training sequence and restricted it to that class, it would have to fit an axis-aligned rectangle in there as best as possible – which would perform much better than chance, but still seems like a suboptimal choice.

A more sophisticated way of answering the "how strongly to weigh evidence vs priors" question is to choose a _family of several hypothesis classes_, as well as a _weighting function_ that privileges how much we like each class. Then, perhaps the class of axis-aligned rectangles would have the highest initial weighting, but the class of all rectangles would still be rated decently, so that if it performs substantially better – as is the case here (although it would still have nonzero empirical error) – the learning algorithm would end up preferring a non-axis-aligned rectangle. We can also look at this as having uncertainty about the right model in addition to uncertainty about which predictor within the model to choose. More about this later.

PAC learning
============

Even though I've criticized the pre-commitment approach, it is a good place to begin a more formal analysis. In particular, we can now talk about a hypothesis class being "learnable". Here, being learnable means that, with sufficiently many training instances, one can prove that, a predictor that performs optimally on the training sequence will _probably_ perform _almost_ as well on the real world _as the hypothesis which has the lowest real error in the class._ Crucially, this holds for every joint probability distribution D.

So this definition is not about how well the hypothesis we end up with performs on the real world in _absolute_ terms – that's not something which is possible to bound, because the hypothesis class may just not contain any good predictor at all (as is the case in the example above if we use the class of axis-aligned rectangles, and of course other classes could be much worse). Rather, it is about how good of a choice the lowest-empirical-error hypothesis is _given that_ _we are stuck with this class_ – i.e. how large is the difference between the real error of \[the hypothesis with minimal empirical error\] and the real error of the best hypothesis, i.e. \[the hypothesis with minimal real error\]. Such a bound is important because, since we can't measure the real error, we cannot output the best hypothesis directly.

There are also two caveats built into the definition. One is the "almost" part; one can never guarantee that the hypothesis with minimal empirical error is exactly as good as the hypothesis with minimal real error because there is randomness in the creation of the training sequence. The second is the "probably" part, one can never upper bound the difference with full certainty because there is always some chance that the training sequence just happens to be unrepresentative. For example, there is always some chance that the training sequence doesn't contain any positively labeled points, in which case A has no way of reliably outputting anything reasonable. However, as the training sequence gets larger, this becomes increasingly unlikely.

The complete formal definition can be stated thus (where we write A(S) to denote the predictor which A outputs upon receiving the training sequence S):

> A hypothesis class H is called _**PAC learnable**_ iff there exists a learner A and a function m∗:(0,1)2→N such that, for any probability distribution D over X×Y, any "gap" ϵ∈(0,1), and any failure probability δ∈(0,1), if the training sequence S consists of at least m∗(ϵ,δ) i.i.d. examples sampled via D, then with probability at least 1−δ over the choice of S, it holds that ℓ(A(S))≤ℓ(h∗)+ϵ for all h∗∈argminh∈H\[ℓ(h)\].

While this defintion is complicated, it is not unnecessarily complicated – the complexity is just an inherent feature of the thing we are trying to define. This kind of learnability is called PAC learnability because the definition states that A(S) is _probably_ (hence the δ) _approximately_ (hence the ϵ) _correct_ (hence the "≤").

One can prove that, whenever a class is PAC-learnable at all, then the algorithm AERM which chooses its predictor such that AERM(S)∈argminh∈H\[ℓS(h)\] is a successful learner. In other words, we can always just choose the predictor that performs best _on the training sequenc_e (or one of them if several are tied). The abbreviation ERM stands for _**empirical risk minimization.**_

One can also prove that _every finite hypothesis class is PAC learnable._ In particular, the _**sample complexity**_ (this is a name for the output of the function m∗ which upper-bounds the number of training points needed to be probability approximately correct) is upper bounded by the term ln(|H|/δ)ϵ.

Let's reflect on this result a bit. If we use AERM as our learner, then learnability implies that the difference between the true error of AERM(S) and the true error of the best hypothesis in H is probably very small. In particular, if H consists of 10000 hypotheses, and we want a 99% assurance that the difference between AERM(S) and the best predictor in H is at most 0.1, then ln(10000/0.01)/0.1≈138.15 so we if we have a training sequence with 139 labeled points, we're good. The only remaining error is then the error of the best hypothesis in H. This is also called the **_approximation error_**, while the error we just bounded is called the **_estimation error_**.

Now consider declaring the class H to be the set of all predictors such that there exists a C program, whose compiled code is at most 10000 bits long, that implements the predictor. It's a safe bet that this class has a very low approximation error for almost all practically relevant cases (or if not then we can allow longer programs). Given that the dependence on the size of H – in this case, 210000 – is only logarithmic, it increases the sample complexity by a factor smaller than 10000. If we then demand an estimation error of at most 0.01 and some very high confidence (note that the dependence on the confidence is also logarithmic, so even demanding a 0.9999 certainty doesn't change much), we would still end up with a sample complexity of only around a million. Gathering a million examples is probably doable in many cases (although [as George points out](https://www.lesswrong.com/posts/Z29etNPef6yvrPteK/understanding-machine-learning-i#3drn3c2B75SAmAieg), assuming that the i.i.d. assumption holds may not be realistic in practice). Thus, we have just derived a general algorithm which, provided there is a way to obtain sufficiently many i.i.d. data points, solves every binary classification task, namely

> 1\. Perform an exhaustive search over all C programs of compiled length in bits at most 10000

> 2\. Throw out all programs that don't implement a predictor

> 3\. Compute the empirical risk for each one that does

> 4\. Output the one with the lowest empircal risk.

For the output of this algorithm, both the _approximation error_ (i.e. the error of the best C program) and the _estimation error_ (i.e. the difference between the error of the hypothesis AERM outputs and that of the best C program) are going to be very low!

Back in the real world, step 1 does of course mean that this algorithm is utterly useless in practice, but _only_ due to its runtime_._ With unbounded computing power, this approach would work wonderfully. In principle, everything (well, at least every binary classification task) is learnable from a relatively small set of examples, certainly fewer examples than what is often available in practice. It doesn't seem completely unreasonable to consider this some (small) amount of evidence on the importance of talent vs data, and perhaps as decent evidence on the inference power of a really smart AI?

* * *

The next post will discuss some theoretical limits of learning and prove the No-Free-Lunch theorem.