Rewriting History, Time and Time Again

Posted on April 8, 2008 by Anthony Watts 

A guest post by John Goetz

In February I wrote a post asking How much Estimation is too much Estimation? I pointed out that a large 

number of station records contained estimates for the annual average. Furthermore, the number of stations used 

to calculate the annual average had been dropping precipitously for the past 20 years. One was left to wonder just 

how accurate the reported global average really was and how meaningful rankings of the warmest years had 

become.

One question that popped into my mind back then was whether or not – with all of the estimation going on – the 

historical record was static. One could reasonably expect that the record is static. After all, once an estimate for a 

given year is calculated there is no reason to change it, correct? That would be true if your estimate did not rely on 

new data added to the record, in particular temperatures collected at a future date. But in the case of GISStemp, 

this is exactly what is done.

Last September I noted that an estimate of a seasonal or quarterly temperature when one month is missing from 

the record depends heavily on averages for all three months in that quarter. This can be expressed by the following 

equation, where {m}_{a}, {m}_{b}, 

{m}_{c} are the months in the quarter (in no particular order) and one of the three 

months {m}_{a} is missing:

{T}_{q,n} = \frac{1}{3}{\overline{T}}_{{m}_{a},N} + \frac{1}{2}\left({T}_{{m}_{b},n} + {T}_{{m}_{c},n}\right) - 

In the above, T is temperature, q is the given quarter, n is the given year, and N is all years of the record.

One can readily see that as new temperatures are added to the record, the average monthly temperatures will 

change. Because those average monthly temperatures change, the estimated quarterly temperatures will change, 

as will the estimated annual averages.

Interestingly, application of the “bias method” used to combine a station’s scribal records can have a ripple effect 

all the way back to the beginning of a station’s history. This is because the first annual average in every scribal 

record is estimated, and the bias method relies on the overlap between all years of record, estimated or not. Recall 

that annual averages are calculated from December of the prior year through November of the current year. 

However, all scribal records begin in January (well, I have not found one that does not begin in January), so that 

first winter average is estimated due to the missing December value. Thus, with the bias method, at least one of 

the two records contains estimated annual values.

Of course, it is fair to ask whether or not this ultimately has any effect on the global annual averages reported by 

GISS. One does not have to look very hard to find out that the answer is “yes”.

Watts Up With That?

Rewriting History, Time and Time Again | Watts Up With That? Page 1 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
On March 29 I downloaded the GLB.Ts.txt file from GISS and compared it to a copy I had from late August 2007. 

I was surprised to find several hundred differences in monthly temperature. Intrigued, I decided to take a trip 

back in time via the “Way Back Machine”.

Here I found 32 versions of GLB.Ts.txt going back to September 24, 2005. I was a bit disappointed the record did 

not go back further, but was later surprised at how many historical changes can occur in a brief 2 1/2 years.The 

first thing I did was eliminate versions where no changes to the data were made. I then compared the number of 

monthly differences between the remaining sequential records and built the following table. Here I show the 

“Prior” record compared to the next sequential record (referred to as “Current”). The number of changes made to 

the monthly record between Prior and Current is shown in the “Updates” column (this column does not count 

additions to the record – only changes to existing data are counted). The number of valid months contained in the 

Prior record is in the “Months” column. “Change” is simply the percent Updates made to Months.

gbl_table.gif

On average 20% of the historical record was modified 16 times in the last 2 1/2 years. The largest single jump was 

0.27 C. This occurred between the Oct 13, 2006 and Jan 15, 2007 records when Aug 2006 changed from an 

anomoly of +0.43C to +0.70C, a change of nearly 68%.

Wow.

The next question I had was “how often are the months within specific years modified?” As can be seen in the next 

chart, a surprising number of the earliest monthly averages are modified time and again.

gbl_yearly_changes.gif

Click the image to see it in full 

I was surprised at how much of the pre-Y2K temperature record changed! My personal favorite change was

between the August 16, 2007 file and the March 29, 2008 file. Suddenly, in the later file, the J-D annual 

temperature for 1880 could now be calculated. In all previous versions the temperature could not be determined.

But some will want to know only how this process affects the rankings for the top 10 warmest years. Because the 

history goes back to the middle of 2005, I explored this question only for the years before 2005. While the overall 

ranking from top to bottom does change from one record to the other, the top 10 prior to 2005 does not change 

much. However, the top two do exchange position frequently, as can be seen from the following table:

gbl_top10.gif

I will note that the overall trend in changes between now and Sep. 24, 2005 is very close to zero. If one compares 

the latest file with the one from Sep 24, 2005, it can be seen that the earliest and latest years are adjusted lower 

today than in 2005, while the middle years are adjusted higher. However, this is purely coincidence. If one 

compares the file from Aug. 2007 with the latest file, it appears the earliest temperatures have been adjusted 

downward, leading to an overall upward trend. Surely other comparisons will yield a downward tend. It is by pure 

chance that we have selected two endpoint datasets that appear to have no effect on the tend.

In the meantime, will the real historical record please stand up?

Rewriting History, Time and Time Again | Watts Up With That? Page 2 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
This entry was posted in Uncategorized. Bookmark the permalink. 

35 Responses to Rewriting History, Time and Time Again

PeakPerformanceGolfSwing.com Ads by Google

Mike Bryant says:

April 8, 2008 at 1:08 pm

George Orwell would be proud. He’d also be proud of this: 

BBC changes global warming article title...

mark wagner says:

April 8, 2008 at 1:32 pm

Hansenize. Verb. To make a correction to historical records in such a way that the correction has the effect of increasing the 

error that the correction purports to correct. 

Serious question: At what point does this become fraud?

Retired Engineer says:

April 8, 2008 at 2:01 pm

At what point do we say we really don’t know what’s going on? When does the error exceed the claimed trend? The station 

survey raises questions about their observed trends. “Urbanization causes local temperatures to rise.” How useful is that?

Until now, this was mostly an academic debate. When the government imposes restrictions with substantial economic 

impact, it takes on a far greater meaning.

Worse, the mandated shift to biofuels, particularly ethanol, has caused food prices to rise, and a shortage of hops.

Which raises the price of beer. Now that’s serious.

edaniel says:

April 8, 2008 at 2:12 pm

Rewriting History, Time and Time Again | Watts Up With That? Page 3 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
I prefer to call these changes to measured data modifications. Corrections and adjustments have meanings that imply some 

sort of valid reference is employed in the process.

MattN says:

April 8, 2008 at 2:15 pm

Actually I think “Hansenize” would be a transitive verb of “to hansen”.

“He hansenized the data so that his boss would think his project was on track”.

The hansenalization of climate continues....

Robert Coté says:

April 8, 2008 at 2:29 pm

Retired Engineer (14:01:58) : 

At what point do we say we really don’t know what’s going on? When does the error exceed the claimed trend? 

Tough question. The speed of light, supposedly a constant has been measured for centuries. Over that time the measured 

value has greatly exceeded the maximum cumulative error for the various methods. Just like Einstein’s incorrect claim that 

gravity fields and acceleration fields were identical perhaps temperature readings in the past are not as permanent as we 

would like.

That said, our genial host has made a clear case that there exist asymmetrical biases in forward calculations of empty fields 

(missing data).

A shortage of hops? And here i was previously unable to put in words my general unease with the dangers of actually 

addressing the theory of AGW. This is serious.

Ric Werme says:

April 8, 2008 at 2:40 pm

Oh come on. What you’re saying is that we won’t know what the past temperatures were until we have all the future ones. 

That means all the papers on past temperatures may now be wrong or may become wrong in the future. Or may become right 

again. I think there’s a word for this, and it’s not Science.

You note there was a cooling trend in the the 1998 & 2002 temps. Well, of course. The data recorded since the hottest years 

on record will be cooler, so the bias adjustment will have to make things cooler. Note also that given the warming trend over 

the last few decades, most old temperatures should be getting adjusted upward. So things have “actually” gotten hotter faster 

than we thought. Panic now before it’s too late! Oh no – it’s too late already. What comes after panic?

Pet peeve time. (1) There are several references to the only science that is good is that which has been published after peer 

review. This argument has been used to denigrate the substantial efforts of professional scientists, amateur scientists, and I 

suppose even non-scientists. (2) One thing I can’t stand about sports talk radio are people who vehmently complain complain 

about flubbed plays in various games. I figure I can’t do better than the athletes involved, I’m not going to complain about 

them.(*) (1+2) I’m a software engineer, I’m not a scientist. I could be a scientist, and I could be a good one. And if I used this 

sort of estimation I’d hope my software engineer persona would excoriate me because he could do a lot better.

Rewriting History, Time and Time Again | Watts Up With That? Page 4 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
(*) There are a couple exceptions, one person who picked a _very_ bad time to let a ground ball roll between his legs threw 

out the first pitch at the Red Sox home opener. That may have been the only play I could have handled better. However, he 

should have been forgiven long ago. Well, at least by late 2004. See http://wermenh.com/runnings.html

Tom Johnson says:

April 8, 2008 at 3:08 pm

I very much wish you could get into contact with statistician William M. Briggs (wmbriggs.com/blog/). 

He has written a remarkable paper which attempts to quantify the uncertainty involved in doing weather analysis and 

prediction. The paper is “Quantifying Uncertainty in AGW”. 

The abstract follows: 

A month does not go by without some new study appearing in a peer-reviewed journal which purports to demonstrate some 

ill effect that will be caused by global warming. The effects are conditional on global warming being true, which is itself not 

certain, and which must be categorized and bounded. Evidence for global warming is in two parts: observations and 

explanations of those observations, both of which must be faithful, accurate, and useful in predicting new observations. To be 

such, the observations have to be of the right kind, the locations and timing where and when they were taken should be ideal, 

and the measurement error should be negligible. The physics of our explanations, both of motion and e.g. heat, must be 

accurate, the algorithms used to solve and approximate the physics inside software must be good, chaos on the time scale of 

predictions must be unimportant, and there must be no experimenter effect. None of these categories is certain. As an 

exercise, bounds are estimated for their certainty and for the unconditional certainty in ill effects. Doing so shows that we are 

more certain than we should be.

REPLY: I have his email, I’ll give him a shout.

Doug says:

April 8, 2008 at 4:01 pm

well the enviroevangelists have an agenda – truth should not get in their way

great article in our news today;

climate change is based on “over-certainty in the absence of convincing argument and data” and “over-reliance on computer 

models”.

http://www.theaustralian.news.com.au/story/0,25197,23509775-2702,00.html

Philip_B says:

April 8, 2008 at 4:18 pm

The very considerable irony in these estimates (of missing data) is that were Global Warming really global, they wouldn’t be 

needed.

Let me explain.

If the warming was occuring in most places by about the same amount, then any sufficiently large set of measurements over 

time would clearly show the amount of warming (over that period of time). Whether they were taken at one location or 

another would be irrelevant. I.e. the trend should show up in a random sample of temperatures over time (assuming no daily 

or seasonal bias).

Rewriting History, Time and Time Again | Watts Up With That? Page 5 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
Of course, Hansen knows that local and regional effects dominate temperatures, so it is necessary to keep measuring at the 

same locations. As anyone who has followed Anthony’s Surface Stations project will know, even a small change at a location 

can affect temperatures by as much or more than the claimed global warming.

Even then, missing data wouldn’t matter if you were interested in measuring a global effect, because the global effect should 

show up across the average of all sites and even large numbers of gaps in the records at individual sites shouldn’t matter 

statistically.

It is precisely because the temperature record at small numbers of individual sites can significantly affect the global 

temperature record that missing data matters.

So there you have it. The estimates for the missing data are needed in order to show the amount of global warming, because 

the warming isn’t global.

Mike Bryant says:

April 8, 2008 at 4:19 pm

I have a message for NASA:

STOP... PLEASE STOP... How many times do you have to keep changing the data? 

Please stop NASA, and then put it back the way it was.

Thanks for listening. 

Mike

Evan Jones says:

April 8, 2008 at 4:51 pm

Serious question: At what point does this become fraud?

Pish-Posh!

Not until policy costing trillions of dollars and seriously adversely affecting billions of lives . . . hang on . . .

Evan Jones says:

April 8, 2008 at 5:13 pm

Hullo, John. Great post!

Fernando Mafili says:

April 8, 2008 at 5:42 pm

Anthony :

When this story end. ....If....... 

we must have missed the turnoff.

Fantasy

Rewriting History, Time and Time Again | Watts Up With That? Page 6 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
jeez says:

April 8, 2008 at 5:58 pm

Hey, not to pat myself on the back or anything, but you could update that post to reflect that you now have access to data 

going back to early 2001 thanks to some random guesswork on my part. One of my only talents, I’m good at making random 

guesses.

Maybe I should use that talent and be contributing to data adjustment algorithms?

REPLY: Jeez, I’m unclear as to what you are referring to, but maybe John Goetz can chime in. Whatever you did, thanks!

John Goetz says:

April 8, 2008 at 5:59 pm

First, let me say that I personally don’t think fraud of any sort is involved. I think this is just an unexpected consequence of 

the mathematics involved. Hansen has made an honest attempt to fill in missing data, and reading his papers I probably 

would have come up with a similar method initially. I don’t think anyone reading this forum can come up with a method for 

estimating missing temperatures that could not legitimately be picked apart by others. However, now that we know and can 

see the flaws, the question is can we find a better method?

To expand, I posted the following comment on CA:

Obviously no method is perfect. Initially I thought a method like Hu’s would suffice, but it assumes climate behaves 

normally. We know it does not.

Example: Where I live, November of 2006 was normal, December 2006 was way above average, and January 2007 was 

slightly below normal. If I were missing the December 2006 data point, just about any scheme I come up with will grossly 

under-estimate that temperature.

Step forward one year later. November was slightly above normal, December was slightly below normal, and January was 

even colder than the previous year. If I were missing December of 2007 but had both adjacent months as well as all three 

corresponding months from the previous year, I would over-estimate the December 2007 temperature.

Note however that the method GISS uses does not use adjacent months – it uses seasonal months. The autumn season is Sep

-Oct-Nov. If N is missing it is derived from Sep-Oct. You and I might think it more reasonable to derive it from Oct-Dec. 

Likewise for missing Sep. Only Oct would be derived from adjacent months. In fact, only four of the missing seasons would 

be derived from missing months. If an entire season is missing, the relationship gets really strange.

Estimation of values during the intermediate steps of determining an average global temperature should be avoided. This is 

particularly true of the earliest steps, because the uncertainty of that estimation only grows as more and more results are 

derived from it. 

In this case, estimation begins very early in the process. One of the first steps is to take each scribal record and estimate 

seasonal averages when months are missing. Then missing annual averages are estimated if any seasonal averages are 

estimated or missing. These scribal records are then combined with other scribal records to form a station record, and the 

consequence of the estimation ripples throughout the resulting record. GISS then applies a homogeneity adjustment to this 

data containing significant numbers of estimates. After that GISS goes through gridcell calculations and then the final 

calculation of the annual global temperature.

The only purpose of the estimation seems to be to calculate an annual average for each scribal record prior to combining, 

adjusting, griding and averaging. I propose that annual averages are not important as intermediate steps. Instead of 

calculating an annual global average temperature, calculate a monthly average temperature. If a data point is missing, then it 

Rewriting History, Time and Time Again | Watts Up With That? Page 7 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
is not adjusted, gridded or averaged. It simply is not included. There should be enough other data points from around the 

world to calculate an average temperature with some degree of certainty (of course, it is desirable that this degree of certainty 

be published).

Mike Bryant says:

April 8, 2008 at 6:08 pm

Historical records are called historical for a reason. Leave them alone.

Philip_B says:

April 8, 2008 at 6:31 pm

If a data point is missing, then it is not adjusted, gridded or averaged. It simply is not included. There should be enough 

other data points from around the world to calculate an average temperature with some degree of certainty 

John G, that was essentially my point. 

The $64K question is, does the estimated data effect the global temperature average and trend?

I await the average and trend calculated with and without the estimated data with considerable interest.

John Goetz says:

April 8, 2008 at 6:52 pm

Philip_B, I don’t know with certainty if the estimated data effect the global temperature average and trend. I suspect they 

do, because I doubt the answer would be identical to the one we have now. But the result may be a greater trend upward 

rather than a lesser trend. We just don’t know.

pdm says:

April 8, 2008 at 6:59 pm

in the end, doesn’t GISS match the other temperature records? It may be tortuous, but it seems to work.

Evan Jones says:

April 8, 2008 at 7:44 pm

http://botd.wordpress.com/top-posts/

Check it out, John. You made top 100 WordPress (#60).

Mike Bryant says:

April 8, 2008 at 8:31 pm

#57 even

Rewriting History, Time and Time Again | Watts Up With That? Page 8 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
Evan Jones says:

April 8, 2008 at 9:34 pm

Movin’ up!

John Goetz says:

April 9, 2008 at 4:18 am

pdm, what do you mean by “match”? The data are not identical.

DNorris says:

April 9, 2008 at 4:56 am

#32 now

AJ Abrams says:

April 9, 2008 at 6:15 am

PDM.

Look at March numbers for GISS. No it most certainly does NOT work and does NOT match other records. Time for them to 

fiddle with the numbers again.

John Goetz says:

April 9, 2008 at 8:08 am

jeez, Anthony: Yes, jeez did find data going back to 1999 and posted links on CA. I’ve already downloaded all of it but have 

not had a chance to crawl through it. I’m not sure it merits much more investigation because I don’t think we will learn any 

more than we know, which is the algorithm modifies the past when temperatures are added to the present. That should be 

enough to cause a double-take.

Thanks jeez!

Mike Bryant says:

April 9, 2008 at 8:34 am

#29 !

Francois says:

April 9, 2008 at 10:03 am

A bit off topic, but also about rewriting history. Recently I was looking for a nice picture of Antarctica’s temperature trend, 

from NASA, that I had seen somewhere. I found it here , but then I also found it... here !... One shows the trend from 1982-

Rewriting History, Time and Time Again | Watts Up With That? Page 9 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
2004, the other between 1981-2007. I can’t figure out how we could have gone from cooling to warming by adding 4 years to 

the data...

Earle Williams says:

April 9, 2008 at 10:38 am

pdm,

It’s not the end that is of concern, it is the beginning. GISS matches the satellite record to some degree, true. But it reqrites 

history by applying adjustments to past temperatures. Look at the adjustments applied in the 100 years prior to the satellite 

record and you will see a remarkable similarity to the alleged increase in golbal temperature since the start of the industrial 

revolution.

Gary Gulrud says:

April 9, 2008 at 2:10 pm

“First, let me say that I personally don’t think fraud of any sort is involved.”

There is, after all, no controlling authority overseeing the ‘Science’.

pdm says:

April 9, 2008 at 2:16 pm

John: 

Match is the wrong word. Looking back over the years of data it seems that GISS tells the same temperature story that RSS 

and the others do. Is that not the case? I say story because they don’t use the same baseline. 

Having said that:

Earle: 

I hear what you are saying regarding pre-satellite data. It is rather suspicious, to put it kindly. There isn’t anything to 

compare against.

Would you agree that in the satellite period the data tell the same story? If, in the future, the satellite history diverges then I 

would agree that the books are being cooked.

John Cross says:

April 10, 2008 at 10:12 am

OK, I had some spare time over lunch and used the Wayback link (thanks for providing the link) to get the data for september 

24, 2005 and then used the current data (with the March value). Using Excel I subtracted the anomalies from the two data 

sets to look at the changes. 

Most of the monthly data change were on the order of + / – 1 or 0.01C. I also did the same with the annual data and again 

most of the annual changes were + / – 1 with a couple being +/- 2. So I agree that there are a significant number of changes, 

but I do not think the changes are large enough to be relevant. I seem to recall that the accuracy of the GISS is +/- 0.1C and 

more the further back you go – I am sure some of the people here will be able to supply a more accurate figure. I also did a 

Rewriting History, Time and Time Again | Watts Up With That? Page 10 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010
Watts Up With That? 

comparison between the monthly anomalies and the annual ones to see if the annual changes were being caused by the 

monthly ones and – to within round off error – they seemed to be.

A question further up caused me to look at the overall trends for the annual record (D-N). From the two cases above I plotted 

the annual and using the trend function on Excel (yeah, I know its for wimps, but I was in a hurry) I found the trend for one 

to be +0.5695 and for the other to be +0.5742. The R2 value was almost identical (0.6323 and 0.6341).

As a final check I plotted to annual values against each other. I figure this would give me a good idea of how close they were. 

A perfect match would have a slope of 1. The slope in this case was 1.006. 

So I don’t think there is a “gotch-ya” hiding there. However I do tend to agree with the people who say it is a strange way to 

calculate a missing value. If the data series was fairly flat (no trend) then it might make sense. But I don’t see any rationale 

for doing it this way.

John

John Goetz says:

April 10, 2008 at 11:44 am

John Cross:

Correct, the magnitude of the fluctuations is not significant. It is significant enough, however, to cause changes in the ever-so

-important temperature rankings we have all come to know and love. I found that surprising – it was not something I 

expected to see.

Something else I did not expect to see was how much a specific year’s average can change over time. Take the “hottest year on 

record” – 1998. On Oct. 13, 1999 the anomaly was 65. On Jan. 26, 2001 it had risen to 66. On Feb. 11, 2002 it was 67. By Apr. 

19, 2003 it was 70, which is where it stands today after a brief interlude back at 69 in mid-2006. That’s an almost 10% drift 

upward, due solely to the fact (as it appears) to more recent temperatures being added to the record. One would think an 

agency as august as NASA would be able to develop a more stable methodology. It just looks sloppy. 

So if there is a “gotcha” it is that we have found their process to add additional uncertainty beyond that already present in the 

data.

Alan Wilkinson says:

July 11, 2008 at 1:24 am

This whole process seems bizarre and rather worse than sloppy. 

Surely you should not estimate missing data in order to evaluate a trend? The missing data increases the uncertainty of your 

evaluation and cannot be replaced by estimates without loss of both data integrity and scientific integrity.

Blog at Wordpress.com. 

Rewriting History, Time and Time Again | Watts Up With That? Page 11 of 11

http://wattsupwiththat.com/2008/04/08/rewriting-history-time-and-time-again/ 7/19/2010