Foundations of Probability. Sigma Algebra, Measure Theory, and... | by Sadrach Pierre, Ph.D. | Towards Data Science

Sigma algebra is considered part of the axiomatic foundations of probability theory. The topic is briefly covered in Casella & Berger’s Statistical Inference. The need for sigma algebras arises out of the technical difficulties associated with defining probabilities. So what exactly are sigma algebras?
Simply put, sigma on X is a collection of subsets of X including the empty set and X itself. In other words, sigma is the power set of X. The sigma algebra is also referred to as the Borel field. It is formally defined as follows:
The first property states that the empty set is always in a sigma algebra. Additionally, since the complement of the empty set is also in the sample space S, the first and second statement implies that the sample space is always in the Borel field (or part of the sigma algebra). The last two statements are conditions of countable intersections and unions.
The Borel space is a basic object of measure theory. It consists of a set and it’s corresponding sigma algebra. Specifically:
Let’s walk through a small example. Consider the following set:
Given X, the Borel Field is is a collection of 23 = 8 sets, also called the power set:
The power set is closely related to the binomial theorem which describes the algebraic expansion of powers of a binomial like the following:
We can expand this using the binomial formula:
where the binomial coefficient is:
The number of subsets in the power set of X is given by the number of binomial coeffients C(n,k):
The power set of any set is the set containing the empty set and the combinations of elements within that set from 1 to the size of the original set:
Now we will discuss how to generate the power set using an implementation in python. To begin, let’s import combinations and chain from itertools:
from itertools import combinations, chain
Next, let’s initialize a tuple containing an empty tuple:
powerset = ((),)
Now, let’s define a function that takes a parameter input set and uses the chain and combinations method to generate out power set:
def powerset(input_set):size = len(input_set)    combs = (combinations(input_set, k) for k in range(1, size+1))        return chain(empty_powerset, *combs)
If we call our function with the set we specified earlier:
print(tuple(powerset({10, 20, 30})))
It turns out, unlike the example we showed above, for uncountable sets sigma algebra is necessary. Sigma algebra is necessary in order for us to be able to consider subsets of the real numbers of actual events. In other words, the sets need to be well defined, under the conditions of countable unions and countable intersections, for it to have probabilities assigned to it.
Given our basic understanding of Borel spaces, let’s proceed by defining probability functions.
Given a sample space S and an associated sigma algebra B, a probability function is a function P with domain B that satisfies the following:
Any function P that satisfies the statements above is a candidate probability function. Since there can be many probability functions defined that satisfy the Axioms of Probability, probability theory is concerned with understanding which functions reflect what is likely to be observed in a particular experiment.
One last interesting thing to consider:
The Banach — Tarski Paradox illustrates the contradiction that arises in the absence of sigma algebra. It states the following:
Given two solid 3D balls, one small and one large, either ball can be reassembled into the other. This is also called the pea and sun Paradox, where it’s stated that a pea can be reassembled into the sun and vice versa. This means that if you are working with real numbers in 3 dimensions (ratio of volumes, for example), you will have difficulty defining probabilities of events because you can rearrange sets of your space to change volumes. If your probability depends on volume, by changing the volume of the set you will also change probabilities. This means that no event can have a single probability assigned to it. This necessitates sigma algebra, which allows us to define measurable sets and probabilities.