Using Permutation Tests to proof the Climate Change
By: Kaya Kupferschmidt
Tags: Climate Change, Data Science, Statistics, Weather

Now the interesting question is: Do the maximum and minimum values behave like one would expect if there was no change of the underlying temperature distribution. More specifically we will ask: Assuming no change of the temperature distribution over all the years, how many distinct maximum and minimum values would we expect and what would be the probability of the number of maximum/minimum values that we are seeing here?

One interesting aspect of this type of statistical question is that the actual temperature values themselves are not used at all. Only the relative ordering of the values is relevant. This property of the statistical test proves to be very useful in many other cases where you cannot really assign meaningful values to increasingly severe events, but you still want to be able to test for randomness. Of course the result then is of a purely qualitative nature (i.e. the temperature is increasing at a statistical significant level) but the test will not tell you by how much the temperature is increasing.

2. A Little Bit of Theory

In order to approach the question whether the number of new maximum/minimum values for the yearly average temperature behaves as one would expect, we need to formalize the problem a little bit.

For the following examination, we generalize the problem to a vector of independent, identically distributed (IID) continuous random variables X(1)…X(n). All of the three properties are important here: We assume that every random variable X(i) in this vector has the same probability distribution and we assume that all variables are independent from each other. Finally we also assume that all variables are continues, which implies that the probability of two random variables having the same value is zero.

2.1 Record Statistics

We are now interested in what I call a running record: We call position i a record whenever the value of the random variable X(i) is larger than all previous variables X(j) with j < i. With the weather data, we are interested in the number of records of the vector of average temperatures.

Given a sequence of values X(1)…X(n) we can now define a simple statistics as the number of observed records within that sequence and then we can ask if the value of that statistics looks plausible if we assume that all X(i) are IID. This assumption corresponds to the null hypothesis of “no climate change” in the temperature series.

Translating the question into a mathematical language, we would like to calculate (or at least estimate) the probability of having k records in a sequence of n IID random variables. This question immediately brings us to random permutations.

2.2 Random Permutations

Instead of answering the original question let’s transform it to an equivalent problem. To do so, we first assign the rank of the value to each random variable, i.e. the smallest random variable is assigned a rank of 1, the second smallest variable a rank of 2 and so on until the largest random variable is assigned the rank n. Let’s denote the rank of X(i) with R(i).

The following example gives us the ranks R(i) for the small sequence X(i) with 7 elements.

value X(i): 0.2 2.3 1.1 0.7 2.4 1.8 2.1

rank R(i): 1 6 3 2 7 4 5

Note that the ranks R(i) still have the same relative order as the original values X(i), which in turn implies that the number of records in the ranks vector is the same as the number of records in the original values.

The set of all ranks R(1)…R(n) contains all the number from 1 until n, but in a random order. Assuming that the original values X(i) are IID, the vector of ranks R(i) is a random permutation. Therefore the probability distribution of the number of records in a sequence of IID random variables is the same as of the number of records in a random permutation.

2.3 Monte Carlo Sampling

Unfortunately, it turns out that the distribution we are looking for (the number of records in a random permutation) is not exactly simple to calculate. There are some formulas (see On the outstanding elements of permutations by Herbert Wilf), but these build on the unsigned Stirling numbers of the first kind which are not easy to calculate.

Instead of trying to evaluate some mathematical formula for the probability distribution we chose a different route: We simply let the computer conduct a reasonable huge number of random experiments by sampling random permutations, and in each experiment we calculate the statistics we are interested in. The histogram (i.e. the relative frequency of each possible outcome) then gives us an estimation of the true probability. This approach is called Monte Carlo Sampling and by the law of large numbers, the results will always converge to the real values.

3. Implementation

Now we have gathered all the required theory for implementing the statistical test if the number of running maximums/minimums in the temperature chart below is plausible under the null hypothesis that the climate doesn’t change (i.e. the probability distribution of the average temperature doesn’t change over time).