The Two Schools of Probability Theory
By: Graeme Keith
Tags: Frequentists, Bayesian, Axioms, Probability, Bayes Theorem

Photo by Nik Albert on Unsplash

The Two Schools of Probability Theory

A layman’s look at the foundations of Frequentism and Bayesianism and how you can have the best of both. Graeme Keith Follow Mar 24 · 13 min read

What is probability? We often hear that there are two schools of thought, Bayesian and Frequentist; that Frequentists believe in an objective measure of probability, which we can only access through large numbers of repeated experiments; and that Bayesians, on the other hand, hold that there is no objective measure of probability, but rather that probability is an inherently subjective measure of belief, which we can only refine and modify with data.

As I discuss in my article “Why probability is hard”, to the pupil of probability, this is all rather unsatisfactory. What about when you can’t repeat an experiment even once, let alone a large number of times? And can you just make probabilities up if you can’t figure out what they should be?

There are indeed two schools of probability theory, broadly speaking due to Andrey Kolmogorov and Richard Cox, but the distinction is much more subtle than frequentist versus Bayesian. Kolmogorov’s system leads very naturally to frequentism but does not define probability in that way. Cox’s theorem is built on a notion of belief or credence and naturally leads to Bayes theorem, but it in no way rules out objectively correct probabilities.

By tracing the two approaches from their axiomatic origins, we can identify the point at which their developments begin to diverge. I argue that both dogmatic frequentism and subjective Bayesianism are based on misconceptions, the understanding of which allows us to develop an eclectic approach to probability theory that has all the astonishing versatility of Bayesianism without compromising on either the objective rigor or the conceptual fertility that lies at the foundation of frequentism — the axioms of Kolmogorov.

Kolmogorov

Open almost any advanced textbook on probability theory and somewhere in the first couple of chapters, you’ll come across Kolmogorov’s axioms of probability.

Probability space

Kolmogorov’s axioms are applied to a mathematical structure called a probability space. The probability space consists of

a set of possible outcomes a collection of ways you might want to combine these outcomes into sets of outcomes called events, whose probability you might want to know. a probability measure, which is a rule by which we assign a kind of size to these sets of outcomes called events

If we imagine an experiment or an incident with an uncertain outcome, we say the event occurs if the eventual outcome is one of the outcomes in the set of outcomes that make up the event. For example, if we roll a die, the event “the roll is even” is the set consisting of outcomes 2, 4, and 6. If the roll was even then the outcome was one of 2, 4, and 6.

The set of all possible outcomes is also an event. It is an event that will always be the case because the eventual outcome will always be one of the possible outcomes.

In Kolmogorov’s system, the probability of an event is in fact just a scaled measure of the “size” of the set of outcomes that define the event. This size can be thought of as something you would count for discrete outcomes or a kind of volume you would measure for a continuum of outcomes.

The axioms

The axioms set the scale for this probability measure of events and give the simplest possible rules for how to manipulate probabilities, from which all the other more complex rules are later derived.

The axioms, broadly, are

Every event should be assigned a probability greater than or equal to 0 and less than or equal to 1. The probability of the event defined by the set of all possible outcomes (the event, mentioned above, that always occurs ) is 1. If two events (sets of outcomes) have no outcomes in common, then the probability of the event made up of all the outcomes in both events is the sum of the probabilities of the two individual events. This means you can add probabilities (sizes) of mutually exclusive events. You can add probabilities for infinite collections of mutually exclusive events.

The greater the probability of an event, the more space the event occupies in the full outcome space, and the greater the proportion of all the possible outcomes it contains. Probability is measured relative to the probability of a certain outcome, which is one. So the probability of an event is the proportion of the space the event takes up in the full outcome space.

So, that’s probability then?

The pioneers of probability theory — Cardano, Fermat, Pascal, Bernoulli, Laplace — developed their theory with coins, cards, dice (especially dice), and lottery tombola. These are all problems where the probability space is prescribed and the probabilities are known in principle (usually an indifference principle, based on the equal likelihood of certain outcomes).

Kolmogorov provides a rigorous, but highly flexible framework for these problems where there is consensus on how to ascribe probability, and in doing so generalizes that consensus to a much wider class of problems. The result is a rich and incredibly fertile system in which to build uncertainty models.

Probability in Kolmogorov’s framework is defined with excruciating exactitude, but only relative to the probability space. The problem is that for the wider class of problems to which Kolmogorov’s framework extends the consensus on probability, the relation of the probability space to the quantities and variables in those problems isn’t clearly defined at all.

Probability is defined through measures ascribed to events, but this just moves the problem of defining what we mean by probability to the problem of assigning measures.

The only way to access probability is by sampling — running the experiment — sufficiently many times, so that the number of times you are observed to land in the event part of the outcome space becomes a reasonable proxy for the measure of relative size to an acceptable level of uncertainty.

Thus Kolmogorov’s axioms lead us directly into the frequentist “interpretation” of probability. Except that we now see that in this framework probability isn’t so much defined by repeated experiment as elicited by repeated experiment.

Probability as logic

The alternative approach to the problem of probability is much more direct. Richard Cox is usually credited with the approach, but the theory goes back to the brilliant but tragically short-lived Cambridge philosopher, mathematician, and economist Frank Ramsey and reaches its apotheosis in the work of Edwin Jaynes, whose exposition we will mostly follow here.

Plausibility

Now we reach much further back in history than the pioneering protagonists of probability theory, right back to the syllogistic logic of Aristotle. A syllogism is a logical construct of the form

All sharks are fish Jaws is a shark Therefore Jaws is a fish

The challenge is to make sense of what we get from statements like

“Kenny is a fish”

Even if we restrict ourselves to sea-life, we can not, of course, conclude from this that Kenny is a shark (he might be a cod), but we can’t help feeling that this sentence is giving us some information. It tells us Kenny is not an octopus or a porpoise or any other kind of non-fish, so he can only be a shark or some other kind of fish.

We want to capture the way the information “Kenny is a fish” increases the plausibility that Kenny is a shark without recourse to a Kolmogorovian probability space, which in this case would be a roll call of all the aquatic life in the sea.

In fact, we would like to do better; we’d like to be able to deal with vaguer statements, which themselves only trade in plausibility, such as

Most, but not all sharks have big, sharp teeth Kenny has big, sharp teeth

Jaynes’ axioms

Jaynes defines probability as a quantitative measure of the plausibility of propositions such as “Kenny is a shark” and sets about deriving a new syllogism to deal with propositions like “Kenny has big, sharp teeth”.

He is able to do this from an astonishing simple set of assumptions regarding the nature of this quantitative measure of plausibility.

Plausibility is represented by a real number. The greater the plausibility, the greater the number. This means that propositions can be ordered by their plausibility. Movements in plausibility are aligned with common sense.

Specifically, say the plausibility of a proposition A (Kenny is a shark, for example) is increased by a proposition B (Kenny has big, sharp teeth). And say that the plausibility that proposition C (Kenny is a boy, for example) has nothing to do with proposition B. Then common sense dictates that the plausibility of A and C together (Kenny is a boy shark) is also increased by proposition B. Hardly controversial. The plausibility of A and B together (Kenny is a shark with big, sharp teeth) shouldn’t depend on how you calculate it.

That’s it.

Jaynes, in a dazzling display of mathematical dexterity, turns these axioms into a set of (fairly nasty, it has to be said) equations and solves them to derive the following two fundamental rules of probability theory

The multiplication rule: The probability of A and B together (the probability Kenny is a shark with big, sharp teeth) is the probability of A given that B is true (the probability that Kenny is a shark when we have the information that Kenny has big, sharp teeth) multiplied by the probability of B (the probability that Kenny has big sharp teeth). Or P(A & B) = P(A | B) P(B) The sum rule: The probability that A is true + The probability that A is not true is one. Or P(A) + P(not A) = 1

(Notational aside: The notation P(A | B) is read “The probability of A given B” and means the probability that A is true given that, i.e. in the case that, we know that B is true. So when A is the proposition “Kenny is a shark” and B is the proposition “Kenny has big, sharp teeth” then P(A | B) is the probability that Kenny is a shark given that Kenny has big, sharp teeth.)

These are just the rules we use when we draw probability trees — we multiply down branches and the probabilities of event outcomes add up to 1. Everything else, including axiom 2 of Kolmogorov’s theorem, can be derived from these two rules, which in turn are derived from the axioms.

Bayes’ theorem

Our goal was to understand how the observation that Kenny has big, sharp teeth (B) changes the probability that Kenny is a shark (A), i.e. we want to calculate the probability Kenny is a shark when we know he has big, sharp teeth (A | B).

By construction (due to the 3rd axiom), the multiplication rule is symmetric in A and B, so

P(A & B) = P(A | B) P(B) = P(B | A) P(A)

Rearranging the rightmost equality gives us Bayes’ rule

P(A | B) = P(B | A) P(A) / P(B)

And we’re done! Sort of. We have a formula for the probability that Kenny is a shark when we have the information that Kenny has big, sharp teeth P(A|B). And this is given in terms of the probability that Kenny is a shark when we don’t know anything else P(A). This was the goal.

But in order to use this formula, we have to know the probability that Kenny has big, sharp teeth in the case that we know he’s a shark. We also have to know the probability that Kenny is a shark when we don’t know anything about his teeth, so again Jaynes hasn’t so much solved the problem of how to ascribe probabilities as just moved it a step backward down the information chain.

Kolmogorov versus Cox: What have we learnt?

Cox (via Jaynes) gives us a much more intuitive and direct definition of probability that doesn’t require the laborious construction of ontologically redundant mathematical structures — however practically and conceptually useful they prove to be. Moreover, we have a set of axioms that not only provides sufficient structure on which to build the theoretical framework but that — in contrast to Kolmogorov’s — are also truly self-evident.

But beyond this little philosophical warm fuzzy, and from a practical point of view, there is very little difference between the two schools. Indeed, the Kolmogorov axioms can be derived in Jaynes’ framework, so there is a strong equivalence. And we’re still stuck with the problem of assigning probabilities in the lack of any information. What is the probability Kenny is a shark? What is the probability that Kenny has big, sharp teeth when we know he’s a shark?

It is in tackling these issues that the two schools of probability begin to diverge along their separate tracks leading to dogmatic frequentism and subjective Bayesianism.

Subjective versus Objective

Jaynes is absolutely explicit. Yes, the probability is a quantitative measure of plausibility, but

…probability assignments [are] completely objective in the sense that they are independent of the personality of the user. They are a means of describing (or, what is the same thing, of encoding) the information given in the statement of a problem, independently of whatever personal feelings (hopes, fears, value judgements, etc.) you or I might have about the propositions involved. It is ‘objectivity’ in this sense that is needed for a scientifically respectable theory of inference.

Doesn’t get more explicit than that.

For Jaynes, different individuals’ assessments will be different according to differences in the information they have, but

Anyone who has the same information, but comes to a different conclusion is necessarily violating one of [the axioms]. While nobody has the authority to forbid such violations, it appears to us that a rational person […] would wish to revise [their] thinking.

Nonetheless, it seems only a minority of Bayesians got that memo. Most, apparently, exactly believe that probability is such a subjective measure of belief.

Subjective Bayesianism

In my view, subjectivism is based on the common error of confounding descriptive and normative theories of practice, in this case going back to Frank Ramsey.

Ramsey described experiments by which it is possible to infer individuals’ levels of belief about the outcomes of events — essentially, asking how much they would be willing to wager on those outcomes and converting those proffered stakes into probabilities.

Ramsey is persuasive that individuals indeed possess a subjective sense of a quantitative measure of the plausibility of these outcomes. This is a splendid description of individual subjective belief, as well as an ingenious method of eliciting it.

But it is a very long way from describing that individuals effectively hold subjective quantitative measures of belief to believing that individuals’ quantitative measures of belief ought to be subjective, or that it is rational to define them as such. Ramsey is telling us how people behave, not how they ought to behave.

Frequentism

In much the same that subjective Bayesians confound the elicitation of subjective belief with the definition of probability, so frequentists confound the elicitation of measure theoretical probability with the definition of probability and in doing so proscribe for themselves an abundance of immensely useful practice on the basis that there aren’t enough data.

In truth, frequentism has no doctrinal aversion to using the Bayes theorem. Bayes theorem is just a rearrangement of the multiplication rule, which is as valid in a Kolmogorovian context as a Coxean. Where they’re not (very reasonably) addressing the subjectivism associated with Bayesianism, frequentist objections to Bayesian methods are more in the nature of a conservative disinclination to the kind of rife speculation that Bayesianism so wantonly encourages. Where Bayesians rush in, frequentists fear to tread.

Strategies for becoming an Objective Bayesian

Subjective Bayesianism and Frequentism can be seen both as cases of categorical confounding — mixing up the elicitation with the definition of probability — and as attempts to get at the problem of assigning probabilities with scarce data. In the case of subjective Bayesianism, by making them up and in the case of frequentism by simply refusing to do so until there is data enough.

Where data are abundant, there is little to argue about, but the practical problems of probability don’t go away because there aren’t abundant data. How then do we deal with a dearth of data?

Interpret what it means to have no information

This is a key component of Jaynes’ approach to Objective Bayesianism. Recognizing that Bayes’ theorem just pushes the problem of assigning probabilities down the chain of information, Jaynes develops a theory, based on the concept of information entropy, that determines the forms of probability distributions that represent the absolute minimum information (maximum entropy) within a certain category of uncertainty (e.g. binary outcomes, finite number of discrete outcomes, infinitely many discrete outcomes, etc.).

This gives the information chain a point of attachment to a theoretically justifiable quantity. No need for subjective belief.

Quantify the uncertainty entailed by the lack of data

This is another arrow in Jaynes’ Objective Bayesian quiver. By using the maximum entropy priors described above and by allowing for uncertainty in the probabilities themselves, we are able to articulate exactly the extent to which the quantity of available data constrain our knowledge of the key probabilities we are trying to determine. These uncertainties can be built into the decision processes built around the elicitation of those probabilities.

I describe this approach in some detail in my article on Transforming Scores into Probabilities.

Guess

Subjective Bayesianism basically advocates educated guesswork in the absence of sufficiently abundant data with which to do better. There’s nothing fundamentally wrong with a well-informed estimate (which is a polite way of saying guess) when it is impossible to do better, subject to two conditions.

First, it is important to recognize your guess as such — an estimate, an approximation to an objectively correct value. Treat it with the caution and skepticism it deserves, and analyze your decisions relative to the substantial uncertainty that it entails. The latter is particularly fruitful; if your decision outcome is insensitive to the particular value of a probability then there’s no point exhausting resources chasing it down.

Secondly, just as estimates of deterministic parameters can be steadily improved over time by comparing estimates with outcomes and improving the estimates accordingly, probabilistic assessments can be steadily improved by auditing subjective assessments.

I have described a method for doing this in a companion piece that also further discusses routes to objective Bayesianism called Probability for Pragmatists.

Conclusion

It’s only fair the hero of this story, Edwin Jaynes, gets the last word