## Rewriting History, Time and Time Again Posted on April 8, 2008 by Anthony Watts 

### A guest post by John Goetz In February I wrote a post asking How much Estimation is too much Estimation? I pointed out that a large number of station records contained estimates for t he annual average. Furthermore, the number of stati ons used to calculate the annual average had been dropping p recipitously for the past 20 years. One was left to wonder just how accurate the reported global average really was and how meaningful rankings of the warmest years h ad become. One question that popped into my mind back then was whether or not – with all of the estimation going on – the historical record was static. One could reasonably expect that the record is static. After all, once a n estimate for a given year is calculated there is no reason to chan ge it, correct? That would be true if your estimate did not rely on new data added to the record, in particular tempera tures collected at a future date. But in the case o f GISStemp , this is exactly what is done. Last September I noted that an estimate of a season al or quarterly temperature when one month is missi ng from the record depends heavily on averages for all thre e months in that quarter. This can be expressed by the following equation, where {m}_{a}, {m}_{b}, {m}_{c} 

### are the months in the quarter (in no particular or der) and one of the three months {m}_{a} 

### is missing:

{T}_{q,n} = \frac{1}{3}{\overline{T}}_{{m}_{a},N} + \frac{1}{2}\left({T}_{{m}_{b},n} + {T}_{{m}_{c},n}\right) - 

### In the above, T is temperature, q is the given quar ter, n is the given year, and N is all years of the record. One can readily see that as new temperatures are ad ded to the record, the average monthly temperatures will change. Because those average monthly temperatures change, the estimated quarterly temperatures will c hange, as will the estimated annual averages. Interestingly, application of the “bias method” use d to combine a station’s scribal records can have a ripple effect all the way back to the beginning of a station’s hi story. This is because the first annual average in every scribal record is estimated, and the bias method relies on the overlap between all years of record, estimated or not. Recall that annual averages are calculated from December o f the prior year through November of the current ye ar. However, all scribal records begin in January (well , I have not found one that does not begin in Janua ry), so that first winter average is estimated due to the missin g December value. Thus, with the bias method, at le ast one of the two records contains estimated annual values. Of course, it is fair to ask whether or not this ul timately has any effect on the global annual averag es reported by GISS. One does not have to look very hard to find o ut that the answer is “yes”. Watts Up With That? 

## Page 1 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/

### On March 29 I downloaded the GLB.Ts.txt file from GISS and compared it to a copy I had from late August 2 007.

I was surprised to find several hundred differences in monthly temperature. Intrigued, I d ecided to take a trip back in time via the “Way Back Machine”
. Here I found 32 versions of GLB.Ts.txt going back t o September 24, 2005. I was a bit disappointed the record did not go back further, but was later surprised at how many historical changes can occur in a brief 2 1/2 years.The first thing I did was eliminate versions where no c hanges to the data were made. I then compared the n umber of monthly differences between the remaining sequentia l records and built the following table. Here I sho w the “Prior” record compared to the next sequential reco rd (referred to as “Current”). The number of changes made to the monthly record between Prior and Current is sho wn in the “Updates” column (this column does not cou nt additions to the record – only changes to existing data are counted). The number of valid months contai ned in the Prior record is in the “Months” column. “Change” is simply the percent Updates made to Months.

gbl_table.gif 

### On average 20% of the historical record was modifie d 16 times in the last 2 1/2 years. The largest sin gle jump was 0.27 C. This occurred between the Oct 13, 2006 and Jan 15, 2007 records when Aug 2006 changed from an anomoly of +0.43C to +0.70C, a change of nearly 68%
. Wow. The next question I had was “how often are the mont hs within specific years modified?” As can be seen in the next chart, a surprising number of the earliest monthly averages are modified time and again.

gbl_yearly_changes.gif 

### Click the image to see it in full I was surprised at how much of the pre-Y2K temperat ure record changed! My personal favorite change was between the August 16, 2007 file and the March 29, 2008 file. Suddenly, in the later file, the J-D ann ual temperature for 1880 could now be calculated. In al l previous versions the temperature could not be de termined. But some will want to know only how this process af fects the rankings for the top 10 warmest years. Be cause the history goes back to the middle of 2005, I explored this question only for the years before 2005. Whil e the overall ranking from top to bottom does change from one rec ord to the other, the top 10 prior to 2005 does not change much. However, the top two do exchange position fre quently, as can be seen from the following table:

gbl_top10.gif 

### I will note that the overall trend in changes between now and Sep. 24, 2005 is very c lose to zero. If one compares the latest file with the one from Sep 24, 2005, it can be seen that the earliest and latest years are adjusted lower today than in 2005, while the middle years are adju sted higher. However, this is purely coincidence. I f one compares the file from Aug. 2007 with the latest fi le, it appears the earliest temperatures have been adjusted downward, leading to an overall upward trend. Surel y other comparisons will yield a downward tend. It is by pure chance that we have selected two endpoint datasets that appear to have no effect on the tend. In the meantime, will the real historical record pl ease stand up? 

## Page 2 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/This entry was posted in Uncategorized. Bookmark th e permalink.



### 35 Responses to Rewriting History, Time and Time Again PeakPerformanceGolfSwing.com Ads by Google

Mike Bryant says: April 8, 2008 at 1:08 pm George Orwell would be proud. He’d also be proud of this: BBC changes global warming article title...

mark wagner says: April 8, 2008 at 1:32 pm Hansenize. Verb. To make a correction to historical records in such a way that the correction has the effect of increasing the error that the correction purports to correct. Serious question: At what point does this become fr aud?

Retired Engineer says: April 8, 2008 at 2:01 pm At what point do we say we really don’t know what’s going on? When does the error exceed the claimed t rend? The station survey raises questions about their observed trends . “Urbanization causes local temperatures to rise.” How useful is that? Until now, this was mostly an academic debate. When the government imposes restrictions with substanti al economic impact, it takes on a far greater meaning. Worse, the mandated shift to biofuels, particularly ethanol, has caused food prices to rise, and a sho rtage of hops. Which raises the price of beer. Now that’s serious.

edaniel says: April 8, 2008 at 2:12 pm 

## Page 3 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/I prefer to call these changes to measured data modifications
. Corrections and adjustments have meanings that imply some sort of valid reference is employed in the process.

MattN says: April 8, 2008 at 2:15 pm Actually I think “Hansenize” would be a transitive verb of “to hansen”. “He hansenized the data so that his boss would thin k his project was on track”. The hansenalization of climate continues....

Robert Coté says: April 8, 2008 at 2:29 pm Retired Engineer (14:01:58) : At what point do we say we really don’t know what’s going on? When does the error exceed the claimed t rend? Tough question. The speed of light, supposedly a co nstant has been measured for centuries. Over that t ime the measured value has greatly exceeded the maximum cumulative e rror for the various methods. Just like Einstein’s incorrect claim that gravity fields and acceleration fields were identic al perhaps temperature readings in the past are not as permanent as we would like. That said, our genial host has made a clear case th at there exist asymmetrical biases in forward calcu lations of empty fields (missing data). A shortage of hops? And here i was previously unabl e to put in words my general unease with the danger s of actually addressing the theory of AGW. This is serious
.

Ric Werme says: April 8, 2008 at 2:40 pm Oh come on. What you’re saying is that we won’t kno w what the past temperatures were until we have all the future ones. That means all the papers on past temperatures may now be wrong or may become wrong in the future. Or may become right again. I think there’s a word for this, and it’s no t Science. You note there was a cooling trend in the the 1998 & 2002 temps. Well, of course. The data recorded si nce the hottest years on record will be cooler, so the bias adjustment wi ll have to make things cooler. Note also that given the warming trend over the last few decades, most old temperatures should be getting adjusted upward. So things have “actuall y” gotten hotter faster than we thought. Panic now before it’s too late! Oh no – it’s too late already. What comes after panic ? Pet peeve time. (1) There are several references to the only science that is good is that which has be en published after peer review. This argument has been used to denigrate th e substantial efforts of professional scientists, a mateur scientists, and I suppose even non-scientists. (2) One thing I can’t stand about sports talk radio are people who vehmen tly complain complain about flubbed plays in various games. I figure I ca n’t do better than the athletes involved, I’m not g oing to complain about them.(*) (1+2) I’m a software engineer, I’m not a s cientist. I could be a scientist, and I could be a good one. And if I used this sort of estimation I’d hope my software engineer pe rsona would excoriate me because he could do a lot better. 

## Page 4 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/(*) There are a couple exceptions, one person who p icked a _very_ bad time to let a ground ball roll b etween his legs threw out the first pitch at the Red Sox home opener. Tha t may have been the only play I could have handled better. However, he should have been forgiven long ago. Well, at least by late 2004. See http://wermenh.com/runnings.html

Tom Johnson says: April 8, 2008 at 3:08 pm I very much wish you could get into contact with st atistician William M. Briggs (wmbriggs.com/blog/). He has written a remarkable paper which attempts to quantify the uncertainty involved in doing weather analysis and prediction. The paper is “Quantifying Uncertainty i n AGW”. The abstract follows: A month does not go by without some new study appea ring in a peer-reviewed journal which purports to d emonstrate some ill effect that will be caused by global warming. T he effects are conditional on global warming being true, which is itself not certain, and which must be categorized and bounded. Evidence for global warming is in two parts: obser vations and explanations of those observations, both of which m ust be faithful, accurate, and useful in predicting new observations. To be such, the observations have to be of the right kind , the locations and timing where and when they were taken should be ideal, and the measurement error should be negligible. The physics of our explanations, both of motion and e. g. heat, must be accurate, the algorithms used to solve and approxim ate the physics inside software must be good, chaos on the time scale of predictions must be unimportant, and there must be no experimenter effect. None of these categories is certain. As an exercise, bounds are estimated for their certainty and for the unconditional certainty in ill effects. Doing so shows that we are more certain than we should be.
REPLY: I have his email, I’ll give him a shout.

Doug says: April 8, 2008 at 4:01 pm well the enviroevangelists have an agenda – truth s hould not get in their way great article in our news today; climate change is based on “over-certainty in the a bsence of convincing argument and data” and “over-r eliance on computer models”. http://www.theaustralian.news.com.au/story/0,25197, 23509775-2702,00.html

Philip_B says: April 8, 2008 at 4:18 pm The very considerable irony in these estimates (of missing data) is that were Global Warming really gl obal, they wouldn’t be needed. Let me explain. If the warming was occuring in most places by about the same amount, then any sufficiently large set o f measurements over time would clearly show the amount of warming (over that period of time). Whether they were taken at o ne location or another would be irrelevant. I.e. the trend should show up in a random sample of temperatures over tim e (assuming no daily or seasonal bias). 

## Page 5 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/Of course, Hansen knows that local and regional eff ects dominate temperatures, so it is necessary to k eep measuring at the same locations. As anyone who has followed Anthony’ s Surface Stations project will know, even a small change at a location can affect temperatures by as much or more than the claimed global warming. Even then, missing data wouldn’t matter if you were interested in measuring a global effect, because t he global effect should show up across the average of all sites and even la rge numbers of gaps in the records at individual si tes shouldn’t matter statistically. It is precisely because the temperature record at s mall numbers of individual sites can significantly affect the global temperature record that missing data matters. So there you have it. The estimates for the missing data are needed in order to show the amount of glo bal warming, because the warming isn’t global.

Mike Bryant says: April 8, 2008 at 4:19 pm I have a message for NASA: STOP... PLEASE STOP... How many times do you have to ke ep changing the data? Please stop NASA, and then put it back the way it w as. Thanks for listening. Mike

Evan Jones says: April 8, 2008 at 4:51 pm Serious question: At what point does this become fr aud? Pish-Posh! Not until policy costing trillions of dollars and s eriously adversely affecting billions of lives . . . hang on . . .

Evan Jones says: April 8, 2008 at 5:13 pm Hullo, John. Great post!

Fernando Mafili says: April 8, 2008 at 5:42 pm Anthony : When this story end. ....If....... we must have missed the turnoff. Fantasy 

## Page 6 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/jeez says: April 8, 2008 at 5:58 pm Hey, not to pat myself on the back or anything, but you could update that post to reflect that you now have access to data going back to early 2001 thanks to some random gues swork on my part. One of my only talents, I’m good at making random guesses. Maybe I should use that talent and be contributing to data adjustment algorithms?
REPLY: Jeez, I’m unclear as to what you are referring to, but maybe John Goetz can chime in. Whatever you di d, thanks!

John Goetz says: April 8, 2008 at 5:59 pm First, let me say that I personally don’t think fra ud of any sort is involved. I think this is just an unexpected consequence of the mathematics involved. Hansen has made an honest attempt to fill in missing data, and reading his p apers I probably would have come up with a similar method initially. I don’t think anyone reading this forum can come u p with a method for estimating missing temperatures that could not legi timately be picked apart by others. However, now th at we know and can see the flaws, the question is can we find a better method? To expand, I posted the following comment on CA: Obviously no method is perfect. Initially I thought a method like Hu’s would suffice, but it assumes c limate behaves normally. We know it does not. Example: Where I live, November of 2006 was normal, December 2006 was way above average, and January 2007 was slightly below normal. If I were missing the Decemb er 2006 data point, just about any scheme I come up with will grossly under-estimate that temperature. Step forward one year later. November was slightly above normal, December was slightly below normal, a nd January was even colder than the previous year. If I were missi ng December of 2007 but had both adjacent months as well as all three corresponding months from the previous year, I woul d over-estimate the December 2007 temperature. Note however that the method GISS uses does not use adjacent months – it uses seasonal months. The autumn season is Sep -Oct-Nov. If N is missing it is derived from Sep-Oc t. You and I might think it more reasonable to deri ve it from Oct-Dec. Likewise for missing Sep. Only Oct would be derived from adjacent months. In fact, only four of the mi ssing seasons would be derived from missing months. If an entire season is missing, the relationship gets really strange. Estimation of values during the intermediate steps of determining an average global temperature should be avoided. This is particularly true of the earliest steps, because th e uncertainty of that estimation only grows as more and more results are derived from it. In this case, estimation begins very early in the p rocess. One of the first steps is to take each scri bal record and estimate seasonal averages when months are missing. Then mis sing annual averages are estimated if any seasonal averages are estimated or missing. These scribal records are the n combined with other scribal records to form a sta tion record, and the consequence of the estimation ripples throughout th e resulting record. GISS then applies a homogeneity adjustment to this data containing significant numbers of estimates. A fter that GISS goes through gridcell calculations a nd then the final calculation of the annual global temperature. The only purpose of the estimation seems to be to c alculate an annual average for each scribal record prior to combining, adjusting, griding and averaging. I propose that an nual averages are not important as intermediate ste ps. Instead of calculating an annual global average temperature, c alculate a monthly average temperature. If a data p oint is missing, then it 

## Page 7 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/is not adjusted, gridded or averaged. It simply is not included. There should be enough other data poi nts from around the world to calculate an average temperature with some degree of certainty (of course, it is desirable th at this degree of certainty be published).

Mike Bryant says: April 8, 2008 at 6:08 pm Historical records are called historical for a reas on. Leave them alone.

Philip_B says: April 8, 2008 at 6:31 pm If a data point is missing, then it is not adjusted , gridded or averaged. It simply is not included. T here should be enough other data points from around the world to calculat e an average temperature with some degree of certai nty John G, that was essentially my point. The $64K question is, does the estimated data effec t the global temperature average and trend? I await the average and trend calculated with and w ithout the estimated data with considerable interes t.

John Goetz says: April 8, 2008 at 6:52 pm Philip_B, I don’t know with certainty if the estimated data effect the global temperature average and trend . I suspect they do, because I doubt the answer would be identical t o the one we have now. But the result may be a grea ter trend upward rather than a lesser trend. We just don’t know.

pdm says: April 8, 2008 at 6:59 pm in the end, doesn’t GISS match the other temperatur e records? It may be tortuous, but it seems to work
.

Evan Jones says: April 8, 2008 at 7:44 pm http://botd.wordpress.com/top-posts/ Check it out, John. You made top 100 WordPress (#60 ).

Mike Bryant says: April 8, 2008 at 8:31 pm #57 even 

## Page 8 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/Evan Jones says: April 8, 2008 at 9:34 pm Movin’ up!

John Goetz says: April 9, 2008 at 4:18 am pdm, what do you mean by “match”? The data are not identical.

DNorris says: April 9, 2008 at 4:56 am #32 now

AJ Abrams says: April 9, 2008 at 6:15 am PDM. Look at March numbers for GISS. No it most certainl y does NOT work and does NOT match other records. T ime for them to fiddle with the numbers again.

John Goetz says: April 9, 2008 at 8:08 am jeez, Anthony: Yes, jeez did find data going back t o 1999 and posted links on CA. I’ve already downloa ded all of it but have not had a chance to crawl through it. I’m not sure it merits much more investigation because I don’t t hink we will learn any more than we know, which is the algorithm modifies the past when temperatures are added to the present . That should be enough to cause a double-take. Thanks jeez!

Mike Bryant says: April 9, 2008 at 8:34 am #29 !

Francois says: April 9, 2008 at 10:03 am A bit off topic, but also about rewriting history. Recently I was looking for a nice picture of Antarc tica’s temperature trend, from NASA, that I had seen somewhere. I found it here , but then I also found it... here !... One shows the trend from 1982- 

## Page 9 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/2004, the other between 1981-2007. I can’t figure o ut how we could have gone from cooling to warming b y adding 4 years to the data...

Earle Williams says: April 9, 2008 at 10:38 am pdm, It’s not the end that is of concern, it is the begi nning. GISS matches the satellite record to some de gree, true. But it reqrites history by applying adjustments to past temperature s. Look at the adjustments applied in the 100 years prior to the satellite record and you will see a remarkable similarity to the alleged increase in golbal temperature since th e start of the industrial revolution.

Gary Gulrud says: April 9, 2008 at 2:10 pm “First, let me say that I personally don’t think fr aud of any sort is involved.” There is, after all, no controlling authority overs eeing the ‘Science’.

pdm says: April 9, 2008 at 2:16 pm John: Match is the wrong word. Looking back over the year s of data it seems that GISS tells the same tempera ture story that RSS and the others do. Is that not the case? I say stor y because they don’t use the same baseline. Having said that: Earle: I hear what you are saying regarding pre-satellite data. It is rather suspicious, to put it kindly. Th ere isn’t anything to compare against. Would you agree that in the satellite period the da ta tell the same story? If, in the future, the sate llite history diverges then I would agree that the books are being cooked.

John Cross says: April 10, 2008 at 10:12 am OK, I had some spare time over lunch and used the W ayback link (thanks for providing the link) to get the data for september 24, 2005 and then used the current data (with the M arch value). Using Excel I subtracted the anomalies from the two data sets to look at the changes. Most of the monthly data change were on the order o f + / – 1 or 0.01C. I also did the same with the an nual data and again most of the annual changes were + / – 1 with a coup le being +/- 2. So I agree that there are a signifi cant number of changes, but I do not think the changes are large enough to be relevant. I seem to recall that the accuracy of the GISS is +/- 0.1C and more the further back you go – I am sure some of th e people here will be able to supply a more accurat e figure. I also did a 

## Page 10 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/Watts Up With That? comparison between the monthly anomalies and the an nual ones to see if the annual changes were being c aused by the monthly ones and – to within round off error – they seemed to be. A question further up caused me to look at the over all trends for the annual record (D-N). From the tw o cases above I plotted the annual and using the trend function on Excel (y eah, I know its for wimps, but I was in a hurry) I found the trend for one to be +0.5695 and for the other to be +0.5742. The R2 value was almost identical (0.6323 and 0.6341). As a final check I plotted to annual values against each other. I figure this would give me a good ide a of how close they were. A perfect match would have a slope of 1. The slope in this case was 1.006. So I don’t think there is a “gotch-ya” hiding there . However I do tend to agree with the people who sa y it is a strange way to calculate a missing value. If the data series was f airly flat (no trend) then it might make sense. But I don’t see any rationale for doing it this way. John

John Goetz says: April 10, 2008 at 11:44 am John Cross: Correct, the magnitude of the fluctuations is not s ignificant. It is significant enough, however, to c ause changes in the ever-so -important temperature rankings we have all come to know and love. I found that surprising – it was no t something I expected to see. Something else I did not expect to see was how much a specific year’s average can change over time. Ta ke the “hottest year on record” – 1998. On Oct. 13, 1999 the anomaly was 65 . On Jan. 26, 2001 it had risen to 66. On Feb. 11, 2002 it was 67. By Apr. 19, 2003 it was 70, which is where it stands today after a brief interlude back at 69 in mid-2006. Tha t’s an almost 10% drift upward, due solely to the fact (as it appears) to m ore recent temperatures being added to the record. One would think an agency as august as NASA would be able to develop a more stable methodology. It just looks sloppy. So if there is a “gotcha” it is that we have found their process to add additional uncertainty beyond that already present in the data.

Alan Wilkinson says: July 11, 2008 at 1:24 am This whole process seems bizarre and rather worse t han sloppy. Surely you should not estimate missing data in orde r to evaluate a trend? The missing data increases t he uncertainty of your evaluation and cannot be replaced by estimates with out loss of both data integrity and scientific inte grity.

Blog at Wordpress.com. 

## Page 11 of 11 Rewriting History, Time and Time Again | Watts Up With That?

7/19/2010 http://wattsupwiththat.com/2008/04/08/rewriting

- history

- time

- and

- time

- again/