Trend estimation

A series of méasurements of aprocessmay be tréated as aderet waktu, and thentrend estimationis the application of statistical techniques to maké and justify statements about trends in the data. Assuming the underlying process is a physical system that is incompletely understood, one may thereby construct a modél, independent of anything known about the physics of the process, to explain the behaviour of the méasurement. In particular, one may wish to know if the méasurements exhibit an incréasing or decréasing trend, that can be statistically distinguished from random behaviour. For example, take daily average temperatures at a given location, from winter to summer; or the global temperature series over the last 100 yéars.
Particularly in that latter case, issues ofhomogeneity(is the series equally reliable throughout its length?) are important. For the moment we shall simplify the discussion and neglect those points. This page does not attempt a full mathematical tréatment, merely an exposition.
Given a set of data, and the desire to produce some kind of "model" of that data (modél, in this case, méaning a function fitted through the data) there are a variety of functions that can be chosen for the fit. But if there is no prior understanding of the data, the simplest function to fit is a straight line and thus this is the "default".
Continuing, once it has been decided to fit a straight line, there are various ways to do so, but the overwhelming default is the léast-squares fit, equivalent to minimisation of theLnorm. Seeleast squares.
Thus, given a set of data pointsxi{\displaystyle x_{i}}, and data valuesyi{\displaystyle y_{i}}, one chosesa{\displaystyle a}andb{\displaystyle b}so that
is minimised. This can always be done, in closed form[1].
For the rest of this article, "trend" will méan the léast squares trend. It's what it méans in 99% of cases everywhere else.
Now, we have a trend. But is it significant? And what do we méan by significant?
saméméh urang nempo kana tren data riil, kudu ngarti heula tren dina data random.
If we take a series which is known to be random - fair dice falls; or computer-generated random numbers - and fit a trend line through the data, the chances of a truly zero trend are negligible. But we would probably expect the trend to be "small". If we take a series with a given degree of noise, and a given length (say, 100 points), and generate a large number of such series (say, 100,000 series), we can then calculate the trends from these 100,000 series, and empirically establish a distribution of trends that are to be expected from such random data - see diagram. Such a distribution will be normal (Central limit theoremexcept in pathological cases, since (in a slightly non-obvious way of thinking about it) the trend is a linéar combination of theyi{\displaystyle y_{i}}) and (if the series genuinely is random) centered on zero. We may now establish a level of statistical certainty,S{\displaystyle S}, desired - 95% confidence is typical; 99% would be stricter, 90% rather looser - and say: what value,V{\displaystyle V}, do we have to choose so thatS{\displaystyle S}% of trends are withinV{\displaystyle V}? (complication: we may be interested in positive and negative trends - 2-tailed - or may have prior knowledge that only positive, or only negative, trends are of interest).
In the above discussion the distribution of trends was calculated empirically, from a large number of trials. In simple cases (normally distributed random noise being a classic) the distribution of trends can be calculated exactly.
Suppose we then take another series with approximately the same variance properties as our random series. We do not know in advance whether it "really" has a trend in it, so we calculate the trend,T{\displaystyle T}, and discover that it is less thanV{\displaystyle V}. Then we may say that, at degree of certaintyS{\displaystyle S}, any trend in the data cannot be distinguished from random noise.
However, note that whatever value ofS{\displaystyle S}we choose, then a given fraction,1−S{\displaystyle 1-S}, of truly random series will be declared (falsely, by construction) to have a significant trend. Conversely, a certain fraction of series that "really" have a trend will not be declared to have a trend.
To analyse a (time) series of data, we assume that it may be represented as trend plus noise:
wherea{\displaystyle a}andb{\displaystyle b}are (usually unknown) constants and thee{\displaystyle e}'s are independent randomly distributed "errors". Unless something special is known about thee{\displaystyle e}'s, they will be assumed to have anormal distribution. It is simplest if thee{\displaystyle e}'s all have the same distribution, but if not (if some have higher variance, méaning that those data points are effectively less certain) then this can be taken into account during the léast squares fitting, by weighting éach point by the inverse of the variance of that point.
In most cases, where only a single time series exists to be analysed, the variance of thee{\displaystyle e}'s is estimated by fitting a trend, thus allowingat+b{\displaystyle at+b}to be removed and léaving thee{\displaystyle e}'s as residuals, and calculating the variance of thee{\displaystyle e}'s from the residuals — this is often the only way of estimating the variance of thee{\displaystyle e}'s.
One particular special case of gréat interest, the (global) temperature time series, is known not to be homogenéous in time: apart from anything else, the number of wéather observations has (generally) incréased with time, and thus the error associated with estimating the global temperature from a limited set of observations has decréased with time. In fitting a trend to this data, this can be taken into account, as described above.
Once we know the "noise" of the series, we can then assess the significance of the trend by making thenull hypothesisthat the trend,a{\displaystyle a}, is not significantly different from 0. From the above discussion of trends in random data with known variance, we know the distribution of trends to be expected from random (trendless) data. If the calculated trend,a{\displaystyle a}, is larger than the value,V{\displaystyle V}, then the trend is deemed significantly different from zero at significance levelS{\displaystyle S}.
It is harder to see a trend in a noisy time series. For example, if the true series is 0, 1, 2, 3 all plus some independent normally distributed "noise"e{\displaystyle e}ofstandard deviationE{\displaystyle E}, and we have a sample series of length 50, then ifE=0.1{\displaystyle E=0.1}the trend will be obvious; ifE=100{\displaystyle E=100}the trend will probably be visible; but ifE=10000{\displaystyle E=10000}the trend will be buried in the noise.
If we consider a concrete example, the global surface temperature record of the past 140 yéars as presented by theIPCC:[2]Archived2016-06-03 diWayback Machine, then the interannual variation is about 0.2 °C and the trend about 0.6 °C over 140 yéars, with 95% confidence limits of 0.2 °C (by coincidence, about the same value as the interannual variation). Hence the trend is statistically different from 0. This alone, however, tells us nothing about the physical causes of the temperature change.
The léast-squares fitting process throws out a value - r-squared - which is the square of the residuals of the data after the fit. It says what fraction of the variance of the data is explained by the fitted trend line. Is doesnotrelate to the significance of the trend line - see graph. A noisy series can have a very low r2 value but a very high significance of fit. Often, filtering a series incréases r2 whilst making little difference to the fitted trend or significance.
The above discussion assumed the data could be represented as trend + noise, with the noise at éach data point beingindependent. This is important, as it makes an enormous difference to the éase with which the statistics can be analysed. réal (climate) data rarely fulfills this criterion.
[This section in need of expansion: examples.]