Package: ccc-gistemp
CLEAR CLIMATE CODE GISTEMP README FOR RELEASE 0.6.1Nick Barnes, Climate Code FoundationDavid Jones, Climate Code Foundation$Date: 2011-06-01 16:08:26 -0400 (Wed, 01 Jun 2011) $CONTENTS1. Introduction2. Dependencies3. Installation4. Input Data5. Running6. Results7. Regression TestingA. ReferencesB. Document historyC. Copyright and license1. INTRODUCTIONThis is release 0.6.1 of the Clear Climate Code GISTEMP project(ccc-gistemp).Clear Climate Code have reimplemented GISTEMP (the GISS surfacetemperature analysis system), to make it clearer.  Work continuestowards making it more clear and more accessible.ccc-gistemp release 0.6.1 is a release of ccc-gistemp version 0.6.The purpose of version 0.6 is to make ccc-gistemp more useful by:- allowing more flexible choices of input data.  For example: GHCN v3;USHCN only; ocean only.- allowing a land mask to be used in the step where land and oceandata are combined.- making use on NumPy array.- adding a setup.pyIn addition there are various bug fixes and improvements to clarity.Changes since earlier releases are described in more detail inrelease-notes.txt.URLs for further information:http://clearclimatecode.org/ Clear Climate Code website and blog.http://ccc-gistemp.googlecode.com/ ccc-gistemp code repository.2. DEPENDENCIESYou need Python and a machine that can run it, and a networkconnection; there are no explicit operating system or CPU architecturedependencies so "any" operating system or CPU should be okay.Python comes in several versions.  We recommend Python 2.6 or Python2.7, but ccc-gistemp should work on any version of Python from the2.x branch (since 2.4).  It will not work with Python 3.x (whichpython.org calls "shiny new thing").  Support for Python 2.4 (andto some extent 2.5) remains fragile, and it has caused some problemsin the past.The code should run on OS X, FreeBSD, Windows, and probably a variety ofother Unix-like operating systems.A network connection is required to download the input files (whichneed only be done once), and to display an optional graph from theresults.  If you use a proxy to access the internet then Python requiresthat the "http_proxy" environment variable is set.  The proxy will needto handle both HTTP and FTP requests (this seems to cause some trouble,see "INPUT DATA" below for downloading data by hand).Python may already be installed on your machine (for example, it comespre-installed on OS X), it may be possible to install it using youroperating system's package manager; for Windows you can download aninstaller from http://www.python.org/download/ .  We recommend you use astable production release from the Python 2.x series (Python 3.x willnot work).3. INSTALLATIONUnpack ccc-gistemp-0.6.1.tar.gz.4. INPUT DATAccc-gistemp uses input data in the subdirectory input/.  Thisinput data includes large files (a few megabytes to a few dozenmegabytes) of temperature records from GHCN, USHCN, and sea surfacedata, and small files of additional temperature records and stationtables from GISS.  ccc-gistemp includes code (tool/preflight.py)to fetch this data from the originating organisations over theinternet.  It will not download a file if it is already present inthe input/ directory, so if you wish to run ccc-gistemp with updatedinput data, you can delete the input/ directory before you start.Downloading the input data is a common causes of problems.  Maintainingthe part of the code that does this (which has nothing to do with thecore GISTEMP algorithm) is a significant cost.  If the toolswe provide do not seem to download the input data correctly, you candownload the data "by hand" and install it in the input/ directory.  Seedoc/input.txt for more details.5. RUNNINGTo run ccc-gistemp:python tool/run.pyThat command runs steps 0 through 5.  To run only a single step or a shortersequence of steps, use the -s argument.  For instance:python tool/run.py -s 3         # Runs just step 3python tool/run.py -s 0-3,5     # Runs steps 0,1,2,3,5 (omitting 4)We use this directory structure:ccc-gistemp-x.x.x/code/     Source code for the GISTEMP algorithm only/config/   Configuration files/doc/      Internal developer documentation/input/    Input data files/log/      Log files/tool/     Tools - sources other than the GISTEMP algorithm/work/     Intermediate data files/result/   Final result filesRunning the code should write to the input/ directory when fetchinginput data, but subsequently only write to the work/ log/ and result/directories.  Before running tool/run.py, these directories can all bedeleted (if you wish, for example, to have a clean run).In 2011 a complete run takes about 35 minutes on an Amazon EC2 "High-CPUMedium Instance" which is a bit faster than the sorts of not-very-impressivemachines that the developers use.  If you want this to go muchfaster we recommend that you run using PyPy (an alternate implementationof Python http://codespeak.net/pypy/dist/pypy/doc/ ).  See this notefrom Paul Ollis:http://groups.google.com/group/ccc-gistemp-discuss/browse_thread/thread/cb0409e7708206806. RESULTSAfter running run.py, the GISTEMP result files are all in the result/directory.  A simple graphical chart is made using the Google ChartAPI; this file:result/google-chart.urlcontains the URL of a chart showing the global annual mean surfacetemperature anomaly.If you have the results of two separate runs in two differentdirectories, old-result/ and new-result/ , then an HTML report comparingthe two can be generated with this command:python tool/compare_results.py --labela=old --labela=new old-result new-resultThis will produce a file called index.html in the current directory,including various statistical comparisons of the two result files.7. REGRESSION TESTINGTo test ccc-gistemp against GISTEMP:python tool/regression.pyThis will fetch a tarball fromhttp://ccc-gistemp.googlecode.com/files/ccc-gistemp-test-2009-12-28.tar.gzand uncompress it to a local directory ccc-gistemp-test-2009-12-28/.This contains input files and result data kindly provided to theccc-gistemp project by Dr Reto Ruedy of NASA GISS, from an actual runof GISTEMP at GISS on 2009-12-28.  Once the tarball is fetched andunpacked, the local ccc-gistemp code will be run on it and the resultscompared, generating a report in index.html.Note that there are indeed some changes between the results of thereference run and 0.4.1 (and subsequent versions), mainly causedby a change to the GISTEMP algorithm, for rural/urban stationdistinction, made at GISS since the reference run.  We have replicatedthat change in ccc-gistemp (seehttp://ccc-gistemp.googlecode.com/issues/detail?id=54).  To testccc-gistemp running the same algorithm as the reference GISTEMPrun, edit code/parameters.py to set use_global_brightness = Falsebefore running tool/regression.py.A. REFERENCESNone.B. DOCUMENT HISTORYMost recent changes first:2010-10-29 DRJ Updated to prepare for 0.6.1.2010-10-22 DRJ Updated to prepare for 0.6.0.2010-07-21 DRJ Updated to prepare for 0.5.1.2010-07-19 DRJ Updated to prepare for 0.5.0.2010-07-13 DRJ Added note about PyPy.2010-03-11 DRJ Updated to prepare for 0.4.1.2010-03-09 DRJ Updated to prepare for 0.4.0.2010-01-26 NB  Updated to prepare for 0.3.0.2010-01-25 DRJ Removed PNG result.2010-01-22 NB  Updated to reflect some code moving to tool/.2010-01-11 NB  Updated to describe preflight better.2010-01-06 DRJ Updated for our all-Python status.2009-12-03 NB  Updated for transfer to GoogleCode project.2008-09-19 DRJ Added PNG result.2008-09-13 NB  Updated for CCC 0.1.0.2008-09-12 NB  Updated for CCC 0.0.3.2008-09-12 NB  Updated for CCC 0.0.2.2008-09-11 NB  Updated for CCC 0.0.1.2008-09-08 NB  Created.C. COPYRIGHT AND LICENSEThis document is copyright (C) 2009, 2010 Ravenbrook Limited; and (C)2010 Climate Code Foundation.  All rights reserved.Redistribution and use of this document in any form, with or withoutmodification, is permitted provided that redistributions of thisdocument retain the above copyright notice, this condition and thefollowing disclaimer.THIS DOCUMENT IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "ASIS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITEDTO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR APARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTHOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITEDTO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, ORPROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OFLIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDINGNEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THISDOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.$URL: https://ccc-gistemp.googlecode.com/svn/branches/2011-05-18/gsoc/readme.txt $$Rev: 772 $