Package: cdx.client
ContentsInstallationQuick InstructionsFull InstructionsExecutablesInstalling EasyInstallInstalling Without EasyInstallIssues and QuestionsUsing CDX ClientCommandsUse from Shell ScriptsCDX LibraryThecdxModuleThecdx.clientModuleThecdx.client.cdxfileModuleCDXDirectory ObjectsCDXFileObjectsChangelog1.3.1 - 10/19/111.3.0 - 10/17/111.2.0 - 06/27/111.1.0 - 11/13/20101.0.0 - 09/10/100.0.9 - 03/24/20100.0.8 - Inclusion of improvements to cdxcd, virtual roots and new tools0.0.7 - Add Resource Files0.0.6 - Inclusion of cdxsubset and other tools, and some minor bug fixes0.0.5 - Repaired Unit Tests0.0.4 - Bugfix to 0.0.3 release0.0.3 - Directory caching0.0.2 - Bug fix for cdxls0.0.1 - URL specification0.0.0 - InitialCDX Client provides an API library and command-line tools for
accessing CDX data.  CDX is the Climate Data Exchange, an effort of the Jet
Propulsion Laboratory to create a virtual environment for the sharing of
climate data.InstallationThis document tells you how to install cdx.client.Quick InstructionsAs a user with administrative privileges, run:easy_install cdx.clientThat’s it.Full Instructionscdx.client requires thePythonprogramming language.  We recommend version 2.4
or later.  As of this writing, 2.6 is the latest stable version.  If Python is
not yet installed on your system, you can find binary and and source
distributions from the Python website.To test if a correct version of Python is available on your system, run:python -VYou should see output similar to:Python 2.6indicating the version of Python installed.  cdx.client also requiresAgile
OODT.OODTis Object Oriented Data Technology, a framework for metadata
and data grids.  Agile OODT is a Python version of OODT that supports higher
performance and easier integration than theJavaversion.By far the easiest, recommended, and encouraged way to install cdx.client is
to useEasyInstall.  If your Python installation has EasyInstall available to
it, then this one command is all you need to run in order to download, build,
install, and generate command-line tools all in one go for all users on your
system:easy_install cdx.clientBe sure to run that command as an administrative user.  For example, on Mac OS
X and other Unix systems, you might need to run:sudo easy_install cdx.clientThat will also download and install all dependencies, including Agile OODT.ExecutablesThe commandscdxlsandcdxgetwill be generated and placed with your
standard installation directory for Python commands.  Usually, this is the
same location as thepythonexecutable itself.  For example, on Mac OS X
10.5, the directory is:/Library/Frameworks/Python.framework/Versions/Current/binYou may want to add that directory to your shell’s PATH variable, as well as
forcing your shell to re-scan the PATH variable for new executables.Installing EasyInstallIf you happen to be on a system where your Python installation lacks easy
install, fret not.  Upgrading your system to gain EasyInstall’s abilities is
quite simple.  Follow these instructions:Downloadhttp://peak.telecommunity.com/dist/ez_setup.pyAs an administrative user, run the freshly-downloaded ez_setup.py file
using your system’s Python.EasyInstall and its necessary libraries will be downloaded, built, and
installed for you, and theeasy_installexecutable generated.  The
location of theeasy_installexecutable is as described above.Installing Without EasyInstallIf EasyInstall is not available on your system, you can still make a proper
installation of cdx.client. Follow these instructions:Download the Agile OODT source distribution fromhttp://oodt.jpl.nasa.gov/dist/agile-oodt/oodt-0.0.1.tar.gz.
Substitute version numbers as appropriate.Download the cdx.client source distribution fromhttp://cdx.jpl.nasa.gov/software/dist/cdx.client-0.0.3.tar.gz.
Substitute version numbers as appropriate.Unpack each archive.Change the current working directory to each newly-created subdirectory,oodt-0.0.1andcdx.client-0.0.2, again substituting version
numbers as appropriate.As an administrative user, run:python setup.py installin each
subdirectory.Issues and QuestionsTo report any problems with or ask for help about cdx.client, visit ourcontactweb page.Using CDX ClientInstalling the CDX Client package makes available three things on your
computer:cdxlscommandThecdxlscommand lets you list the contents of a CDX server from your
terminal prompt or a shell script.cdxgetcommandThecdxgetcommand lets you retrieve data from CDX from either your
terminal prompt or a shell script.CDX LibraryThe CDX Library is a Python-based API for using CDX servers.This document describes how to use the above three items, with special
attention to the CDX Library.CommandsAfter installing the CDX Client package, two new command are made available on
your system,cdxlsandcdxget.  These commands enable you to list the
contents of the data on a CDX server and retrieve selected files from the
server.To use these commands from your interactive prompt, you just need to make sure
your shell’s PATH environment variable includes the directory where the
commands are installed.  On most systems, these two commands are installed in:/usr/local/binHowever, on Mac OS X, the installation location may be:/Library/Frameworks/Python.framework/Versions/Current/binAnd on Windows, it may be:c:\Program Files\PythonNote also that some interactive shells create a cache of commands in order to
execute your requests more quickly.  You may need to force your shell to
re-build that cache.  The csh and tcsh shells are two such examples; you can
make these shells rebuild their caches by running therehashcommand.Use from Shell ScriptsThecdxlsandcdxgetcommands may be used from shell scripts as well.  The
only requirement for making these commands available to shell scripts is the
same as for interactive sessions:  the shell’s PATH environment variable must
include the directory that contains thecdxlsandcdxgetcommands.Here is a sample shell script that retrieves the MLS Aura L2GP data files (and
metadata) files for HO2 and HOCl from day 325 in 2008:#!/bin/sh
PATH=/usr/local/bin:/usr/bin:/bin; export PATH
CDX_SERVER=http://mlscdx.jpl.nasa.gov:8080/cdx/prod; export CDX_SERVER

for kind in HO2 HOCl; do
    for extension in he5 he5.met; do
        cdxget 2008/325/MLS-Aura_L2GP-${kind}_v02-23-c01_2008d325.${extension}
    done
doneThe above shell script assumes thatcdxgetwill be found in/usr/local/bin,/usr/bin, or/bin.  It also sets theCDX_SERVERenvironment variable to set what CDX server to talk to.  It then loops through
two kinds of data (HO2andHOCl), and loops through two kinds of file
extensions (he5andhe5.met).  The results is it retrieves four files to
the current working directory, specifically:2008/325/MLS-Aura_L2GP-HO2_v02-23-c01_2008d325.he52008/325/MLS-Aura_L2GP-HO2_v02-23-c01_2008d325.he5.met2008/325/MLS-Aura_L2GP-HOCl_v02-23-c01_2008d325.he52008/325/MLS-Aura_L2GP-HOCl_v02-23-c01_2008d325.he5.metThecdxsubsetcommand may also be used from a shell script.  It is
configured by two environment variables:CDX_SUBSET_MODE - if set then local data wrapper mode will be used (remote
is assumed as default)CDX_SERVER - set to the product server to talk to for subsettingSome example working commands are:Subset spatial bounding box from NCAR CCSM model output:cdxsubset -b /esg/data18/commit/atm/da/hfls/ncar_ccsm3_0/run1/hfls_A2.Commit_1.CCSM.atmd.2000-01-01_cat_2039-12-31.ncSubset time range from NCAR CCSM model output:cdxsubset -t /esg/data18/commit/atm/da/hfls/ncar_ccsm3_0/run1/hfls_A2.Commit_1.CCSM.atmd.2000-01-01_cat_2039-12-31.ncGet time array variable data from the MLS L2 granule:cdxsubset -p Time /mls/2005/100/MLS-Aura_L2GP-BrO_v01-51-c01_2005d100.he5Get spatial bounding box from AIRS level 2 granule:cdxsubset -b /airs/data/s4pa/Aqua_AIRS_Level2/AIRX2RET.003/2007/005/AIRS.2007.01.05.240.L2.RetStd.v4.0.9.0.G07007180718.hdfSubset by lat lon and variable for an AIRS level 2 granule:cdxsubset -p TAirStd --latitude-range=67.35:78.40 -longitude-range=172.226:176.10 /airs/2009/01/01/airx2ret/AIRS.2009.01.01.001.L2.RetStd.v5.2.2.0.G09002135510.hdfCDX LibraryThe CDX Library is a Python-based application programming interface (API) for
communicating with CDX servers.  In fact, the two commandscdxlsandcdxgetare implemented using the CDX Library.  If shell-script programming
is not to your taste, and you know Python, then using the CDX Library may be
right for you.The CDX Library uses an object-oriented approach to model the contents of a
CDX server.  Objects represent CDX files and directories, and you call methods
on those objects to determine file attributes, directory contents, or retrieve
a file’s contents.The remainder of this document describes the modules, classes, and functions
that comprise the CDX Library.  If you don’t know Python, you may wish to skip
the rest.ThecdxModuleThecdxmodule is a namespace module.  It provides no classes or functions.
Rather, it contains a single, nested module calledclient.Thecdx.clientModuleThecdx.clientmodule contains nested modules that provide the CDX Library.
It also contains implementations of thecdxlsandcdxgetcommands.Thecdx.client.cdxfileModuleThecdx.client.cdxfilemodule is where all the action is.  It contains
classes and functions for communicating with and modeling the contents of CDX
servers. It contains the following items:CDXDirectoryObjects of this class represent directories on a CDX server.  You can use
Python’s iterator, length, and containment protocols to examine the
contents of the directory. They can also be sorted.CDXFileObjects of this class represent files on a CDX server.  While you can
instantiate objects of this class, you’d typically instantiate a
CDXDirectory and examine its contents which will include CDXFile objects
for files in the directory and nested CDXDirectory objects for
subdirectories.  A CDXFile object also provides a method to let you
retrieve its data.findFileThefindFilefunction is a utility function that, given a starting
CDXDirectory and a path name, yields the matching CDXDirectory or CDXFile
on a CDX server.CDXDirectory ObjectsCDXDirectoryobjects represent directories in a CDX server.  You can
create these objects directly or you can use thefindFilemethod in thecdx.client.cdxfilemodule.class CDXDirectory(path,cdxURL= None)Create aCDXDirectoryobject with the givenpath.  You can also
specify the URL to a CDX server to use by passing in a string forcdxURL.sort(cmp= cmp,reverse= False)Return the contents of the directory, sorted, using the a comparison
functioncmp, defaulting to Python’s built-incmp.  Ifreverseis
True, reverse the order of the sort.  Comparison withcmponCDXFileandCDXDirectoryobjects is by CDX server URL and by name.  You can
pass in your owncmpthat, for example, sorts by file size.isFile()Always returns False.pathThe path name of the directory.nameThe name of the directory; this is the last element of the path.sizeBy convention sizes for directories are always zero.CDXDirectoryobjects obey Python’s protocols for hashing, comparison,
containment testing, iteration, indexing, and length query.  Containment
testing with directories with withCDXDirectoryobjects,CDXFileobjects, or plain strings:>>> from cdx.client.cdxfile import CDXDirectory
>>> root = CDXDirectory('/', 'http://localhost:8192/cdx/prod')
>>> len(root)
3
>>> subdir = root['2005']
>>> subdir
CDXDirectory(path=/2005)
>>> subdir in root
True
>>> '2005' in root
True
>>> subdir < root
False
>>> subdir > root
True
>>> for i in root:
...     print i
...
/2008
/2007
/2005
>>> root.sort()
[CDXDirectory(path=/2005), CDXDirectory(path=/2007), CDXDirectory(path=/2008)]CDXFileObjectsTBD.Changelog1.3.1 - 10/19/11This release includes an updated version of the datawrappers
package (0.0.8) as specified in CDX-122.For the issue tracker, seehttps://oodt.jpl.nasa.gov/jira/browse/CDX1.3.0 - 10/17/11This release fixes an important issue in the CDX regrid service
and computes an average of the running sum of data points indexed
by cube cell. See CDX-118 for more information.For the issue tracker, seehttps://oodt.jpl.nasa.gov/jira/browse/CDX1.2.0 - 06/27/11This release provides tight integration with the ESG, and plugs
into its security infrastructure and adds a bunch of virtual
roots for use in cdxregrid and cdxsubset for ESG data. See CDX-110
and CDX-111 for more information.For the issue tracker, seehttp://oodt.jpl.nasa.gov/jira/browse/CDX.1.1.0 - 11/13/2010This release incorporates version 0.0.6 of the cdx.datawrappers package
which includes CDX-103, which implements GetVariable by lat, lon, and
time. In turn this release also provides CDX-102, which incoprorates
this functionality into the basic cdxregrid functionality. At this point,
cdxregrid is pretty much fully functional.For the issue tracker, seehttp://oodt.jpl.nasa.gov/jira/browse/CDX.1.0.0 - 09/10/10This release made a minor improvement to the public cdxls API, exposing
the set of found and lost files from the listFiles function, taking away
its function-local orchestration and exposing the lists of found and notfound
files to the user. See CDX-93 for more information. This release additionally
exposes the CDX MODIS product server via cdxls. See CDX-98 for more details.
Finally, this release includes updates to fix CloudSat as a cdxsubset source,
as described in CDX-99.For the issue tracker, seehttp://oodt.jpl.nasa.gov/jira/browse/CDX.0.0.9 - 03/24/2010This release includes improvements to cdxsubset, specifically the ability
to print out the full numpy array returned from a DataWrapper. See
CDX-82 for specific details. Additionally cdxsubset has been updated to expose
the subset by LatLon functionality per CDX-84 and CDX-85. Subset by range query
allowing constraints to be specified was also included in this release (see
CDX-86 for more information).For the issue tracker, seehttp://oodt.jpl.nasa.gov/jira/browse/CDX.0.0.8 - Inclusion of improvements to cdxcd, virtual roots and new toolsThis release includes improvements to cdxcd to make it work nicely
with cdx virtual roots, and includes integration with the other cdx client
toolkit including cdxls, cdxsubset and cdxget. See CDX-70 and CDX-71 for
further details.For the issue tracker, seehttp://oodt.jpl.nasa.gov/jira/browse/CDX.0.0.7 - Add Resource FilesRelease 0.0.6 was mis-configured and didn’t include some important resource
files.  This emergency release includes them!0.0.6 - Inclusion of cdxsubset and other tools, and some minor bug fixesThis release includes the cdxsubset tool, as described in CDX-56.
This release also includes the cdxcd tool, as described in CDX-69.
This release also includes minor aestetic bug fixes that address pathing
issues in cdxls, e.g., CDX-29.For the issue tracker, seehttp://oodt.jpl.nasa.gov/jira/browse/CDX.0.0.5 - Repaired Unit TestsThis release updates the unit tests and test data based on the changes in
0.0.4 and the new behavior of actual product servers. In addition, it
fixes some documentation problems (incorrect package name cdx-client
which should’ve been cdx.client) in the INSTALL.txt file.The sole bug report addressed in this release is CDX-45, “Unit tests in
cdx-client failing”. For the issue tracker, seehttp://oodt.jpl.nasa.gov/jira/browse/CDX.0.0.4 - Bugfix to 0.0.3 releaseThis is a bugfix release to 0.0.3, which includes some error checking to
deal with some data format inconsistencies on the OODT OFSN product server
end.JIRA issues addressed (seehttp://oodt.jpl.nasa.gov/jira/browse/CDX):CDX-43 Directory structure shouldn’t be preserved if cdxget is called
without the -r parameterCDX-42 cdxget -r fails to retrieve MLS dataCDX-41 cdxls -R chokes if dir size not provided0.0.3 - Directory cachingThe major feature of this release is thecdx.client.dircachemodule which
enables local-disk caching of a subset of a remote CDX product server’s
contents.  It also introduces the concept of acdx:scheme URL.  Such a
URL has this form:cdx://hostname[:port]/endpoint/prod/path/to/a/directorywhere hostname is the name or IP address of a CDX product server, port is an
optional port number on which the server is listening, endpoint is theWebGridservice identifier (typically just the stringcdx), prod is the
fixed keywordprod, andpath/to/a/directoryis an absolute path to a
directory within that product server.Such caching is intended to support theCCMValDiagsoftware.0.0.2 - Bug fix for cdxlsThis release repairs a bug in cdxls that caused directories with only one item
in them to not be listed properly.0.0.1 - URL specificationThis release provides support for a (-u url, –url=url) pair of command-line
options that enable specification of a specific URL to use, falling back to
the URL specified in the CDX_SERVER environment variable (and, if that’s
unset, thenhttp://mlscdx.jpl.nasa.gov:8080/cdx/prod).  This supports two
ideas suggested in CDX-16 (the first two, not the third with a cdx:  style
URL).0.0.0 - InitialThis is an initial release of cdx-client supporting minimalcdxlsandcdxgetfunction.