Critical Introduction to Probability and Statistics: Fundamental Concepts & Learning Resources
By: Dave Amiana
Tags: Science, Statistics, Mathematics, Philosophy

From Probability Theory to Statistics

Statistics was antedated by the theory of probability. In fact, any serious study of statistics must of necessity be preceded by a study of probability theory — since the theory of statistics grounds its foundation. While the theoretical ends of statistics ought to agree that (at least to serve as a common feature) it depends on probability; the question as to what probability is and how it is connected with statistics have experienced certain forms of disagreement [8].

Whereas there are arrays of varying statistical procedures that are still relevant today, most of them rely on the use of modern measure-theoretic probability theory (Kolmogorov) while others express near relative as a means to interpret hypotheses and relate them to data.

Probability is the most important concept in modern science, especially as nobody has the slightest notion what it means (Russell, 1929).

What does probability mean? The mathematical notion of probability does not provide an answer to this. Hence, the formal axiomatization of probability does not guarantee that it be held meaningful for all possible worlds [11].

Interpretations of Probability Theory [10–11]

Since the notion of probability is deemed one of the foremost concepts in scientific investigation and spans its relevance to the philosophy of science in the analysis and interpretation of theories, epistemology, and philosophy of the mind, the foundations of probability [and its interpretations] which is held of utmost relevance in honing our understanding in statistics, bear, at least indirectly, and sometimes directly, upon scientific and philosophical concerns.

The probability function — a particular kind of function that is used to express the measure of a set (Billingsley, 1995) — may be interpreted as either physical or epistemic. In addition, the American Philosopher, Wesley C. Salmon (1966) provides a set of criteria for coming up with an adequate interpretation of probability which is briefly reviewed as follows [11]:

Admissibility — if the meanings assigned to the primitive terms in the interpretation transform the formal axioms, and consequently, all the theorems, into true statements.

— if the meanings assigned to the primitive terms in the interpretation transform the formal axioms, and consequently, all the theorems, into true statements. Ascertainability — This criterion requires that there be some method by which, in principle at least, we can ascertain values of probabilities.

— This criterion requires that there be some method by which, in principle at least, we can ascertain values of probabilities. Applicability — The interpretation of probability should serve as a guide relative to the domain of discourse (or field of interest).

According to Salmon (as cited in Hájek, 2019), most of the work will be done by the applicability criterion. That is to say, more or less, that our decision for interpreting probability should cast the world which we are interested in. For example, Bayesian methods are more appropriately used when we know the prior distribution of our event space — for instances like rolling a dice where there is a geometric symmetry that follows a natural pattern of distribution. For most, experiments, however, Bayesian methods would require the researcher’s guess for setting some prior distribution over their hypotheses. This is where other interpretations may seem more appropriate.

Because we are more concerned with honing our deep understanding with statistics, I limit this article to the most relevant set of interpretations which can be classified into physical and epistemic class. For a more detailed interpretation of probability, the reader is invited to consult the entry from Stanford Encyclopedia of Philosophy on Interpretations of Probability [11].

Physical where the frequency or propensity of the occurrence of a state of affairs often referred to as the chance.

where the frequency or propensity of the occurrence of a state of affairs often referred to as the chance. Epistemic where the degree of belief in the occurrence of the state of affairs, the willingness to act on its assumption, a degree of support or confirmation, or similar.

According to the University of Groningen Philosophy Professor Jan-Willem Romejin (2017), the distinction should not be confused with that between objective and subjective probability. Both physical and epistemic probability can be given an objective and subjective character, in the sense that both can be taken as dependent or independent of a knowing subject and her conceptual apparatus.

Meanwhile, the longheld debate between two different interpretations of probability namely being based on objective evidence and on subjective degrees of belief has caused mathematicians such as Carl Friedrich Gauss and Pierre-Simon Laplace to search for alternatives for more than 200 years ago. As a result, two competing schools of statistics were developed: Bayesian theory and Frequentist theory.

Photo by Ellicia on Unsplash

Note that some authors may define the classical interpretation of probability as Bayesian while classical statistics is frequentist. To avoid this confusion, I will dismiss the term classical to refer to the Frequentist theory.

In the following subsections, I will briefly define the key concepts between Bayesian Theory and Frequentist Theory of Statistics which I got from [14].

1.0 Bayesian Theory

The controversial key concept of Bayesian School of thought was their assumption for prior probabilities — which relies solely on the researcher’s naive guess or confidence towards their hypotheses. But there are also good reasons for using Bayesian methods over the Frequentist approach. The following highlights the key ideas as to why you should or should not use Bayesian methods for your analysis.

Bayesian inference depends on one’s degree of confidence in the chosen prior. Bayesian inference uses probabilities for both hypotheses and data; it depends on the prior and likelihood of observed data [14].

Criticism [14]

Subjective nature of selecting priors. There is no systematic method for selecting priors.

Assigning subjective priors does not constitute outcomes of repeatable experiments.

Reasons for using Bayesian Methods [14–15]

Using Bayesian methods are logically rigorous because once we have a prior distribution, all our calculations are carved with the certainty of deductive logic [14]. Philosophers of science usually come down strongly on the Bayesian side [15].

[15]. The simplicity of the Bayesian approach is especially appealing in a dynamic context, where data arrives sequentially, and where updating one’s beliefs is a natural practice[15].

By trying different priors, we can ascertain how sensitive our results are to the choice of priors [14].

It is relatively easier to communicate a result framed in terms of probabilities of hypotheses.

2.0 Frequentist Theory

While Bayesian methods rely on its priors, Frequentism focuses on behavior. The frequentist approach uses conditional distributions of data given specific hypotheses. Frequentist approach does not depend on a subjective prior that may vary from different researchers. However, there are some objections that one has to keep in mind when deciding to use a frequentist approach.

Criticism [14]

Struggles to balance behavior over a family of possible distributions.

It is highly experimental; it does not carry the template of deductive logic. P-values depends on the exact experimental set-up (p-values are the threshold of which inference is drawn).

P-values and Significance level (both are forms of a threshold for inferential decision) are notoriously prone to misinterpretation [14].

Reasons for using Frequentist Methods [14]

The frequentist approach dominated in the 20th century, and we have achieved tremendous scientific progress [14].

The Frequentist experimental design demands a careful description of the experiment and methods of analysis before staring — it helps control for the experimenter’s bias [14].

Comparison

The figure above is a comparative analysis made by Little (2005).

The difference is that Bayesians aim for the best possible performance versus a single (presumably correct) prior distribution, while frequentists hope to due reasonably well no matter what the correct prior might be [15].

Important notes

The competing notions between Frequentist’s approach to statistical analysis and Bayesian methods have been around for over 250 years. Both schools of thought have been challenged by one another. The Bayesian method had been greatly criticized for its subjective nature while the Frequentist’s method had been put into question for its justification of probability threshold of which it draws an inference (p-values, and significance value). It is worth noting that, albeit the Frequentist’s approach to Statistic prevailed in 20th-century science, the resurgence of Bayesian method has been greatly values in 21st-century Statistics.

For more detailed discussion for this matter, the reader is invited to consult [15].