# Presentation: 024044

## Configuring NVODS Datasets in LAS 

**Joseph McLean**
**TMAP/PMEL/NOAA and JISAO/UW**
**OAR WebShop 2002**

## Part 1 -Getting a Dataset "into" LAS- 

**Best Case - COARDS Compliant NetCDF File**
-       The metadata ingestion process.
-       The relational database and user interface.
-       Ready to access and process.
**The LAS Data Configuration XML File **
**      **operations/institution information
-       data set attributes
-       variable attributes
-       back-end properties
-       lon/lat/depth/time axis definitions

**Notes:**
Best Case - COARDS Compliant NetCDF File 
    Metadata is incorporated into LAS through the metadata ingestion process. A Perl program takes as an argument a DODS URL to read use metadata from .dds and .das files to generate portions of the LAS data configuration XML file. In the case of NetCDF files with complete COARDS compliant metadata, the LAS administrator, after some minor additions to the XML file, may then use a second Perl program which serializes the XML data to a relational database, and generates Javascript for the user interface. Thus with two Perl programs the metadata from a DODS dataset may be added to the LAS database and the dataset is ready to access and process via the LAS user interface. 
The LAS Data Configuration XML File 
   Typically the data configuration file includes a collection of dataset names, access URL's, and documentation URL's; variable names, units, and lat/lon/depth/time axis range and step values. Along with operations and institution information, it may also contain elements which associate back-end application visualization hints or user interface customizations with a specific dataset or variable.

## -Data Configuration XML File- 

- <?xml version='1.0' ?> <!DOCTYPE spec SYSTEM "spec.dtd" [<!ENTITY StdOperations SYSTEM "operations.xml">]>
-        <lasdata>
-           <institution name="Pacific Marine Environmental Lab“
-             url="http://www.pmel.noaa.gov"/>
-             <operations url="http://foo.server.com/las-bin/LASserver.pl">         &StdOperations;    </operations><!-- **Define datasets/variables** -->
-          <datasets>
-            <coads_climatology_cdf name="COADS Climatology"
-             url="http://ferret.pmel.noaa.gov/cgi-bin/dods/nph-dods/data/PMEL/coads_climatology.ncl"
-             doc="doc/coads_climatology.html">
-             <variables>
-               <airt name="Air Temperature" units="DEG C">
-                <link match="/lasdata/grids/coads_climatology_cdf_grid"/>
-                <!-- **Define properties** -->
-                <properties>
-                 <ferret>
-                  <palette>light_centered</palette>
-                  <init_script>COADS_clim_init</init_script>
-                 <ferret>
-                <properties>
-               </airt>
-             </variables>
-            </datasets>
-            <!-- **Define grids** -->
-            <grids>
-             <coads_climatology_cdf_grid>
-              <link match="/lasdata/axes/coads_climatology_cdf_COADSX"/>
-              <link match="/lasdata/axes/coads_climatology_cdf_COADSY"/>
-              <link match="/lasdata/axes/coads_climatology_cdf_TIME"/>
-             </coads_climatology_cdf_grid>
-            </grids>
-           <!-- **Define axes** -->
-           <axes>
-            <coads_climatology_cdf_COADSX type="x" units="degrees_east">
-             <arange start="21" step="2" size="180"/>
-            </coads_climatology_cdf_COADSX>
-            <coads_climatology_cdf_COADSY type="y" units="degrees_north">
-             <arange start="-89" step="2" size="90"/>
-           </coads_climatology_cdf_COADSY>
-           <coads_climatology_cdf_TIME type="t" units="month">
-            <arange start="1-1-16" step="1" size="12"/>
-           </coads_climatology_cdf_TIME>
-          </axes>
-         </lasdata>

- LAS attributes

- Data set attributes

- variable attributes

- back-end properties

- Axis definitions

## Part 1 -Getting a Dataset "into" LAS-

**Configuring the XML File**
-       when needed
-       custom palettes or contour levels
-       variable name conflicts
-       virtual variables
-       incomplete/incorrect metadata
**Worst Case - No Metadata**
-        frequently missing or inadequate
-        supply metadata in XML file by hand
-        need adequate documentation
-        explore the dataset
-        custom initialization scripts

**Notes:**
Configuring the XML File 
    It is the responsibility of the LAS administrator to configure the XML file by hand when needed. Instances when this is the case include the need for custom palettes or contour levels in the output graphic; when two datasets share the same name, or two variables in the administrator- designed dataset share the same name; defining new variables from existing variables when the back-end application supports this; and of course, when the metadata provided by the remotely accessed dataset are incomplete or incorrect.
Worst Case - No Metadata 
    Unfortunately, metadata in netCDF files are frequently missing or inadequate. As long as you are using only COARDS compliant netCDF files, you can still use the automatic XML generation technique described above to generate a template configuration file, which you can then edit. If you are using other types of datasets, you have to manually insert the metadata for the dataset. To do this, you need adequate documentation and first must explore the dataset via an application such as Ferret. Often, after this exploration, custom initialization scripts must be written and incorporated into the LAS structure. 

## Part 1 -Getting a Dataset "into" LAS-

**Back-end Scripts**
-      multiple back-ends
-       "NCL","CDAT",GrADS, Matlab,IDL
-      Ferret is free and distributed with LAS.
-      NVODS LAS(via Ferret) uses the NetCDF API
-      to access DODS datasets.
**Custom User Interfaces**
**     **high resolution maps
-      data organization

**Notes:**
The LAS design accommodates multiple back-ends (NCAR has "NCL", LLNL has "CDAT", COLA is working on GrADS, prototypes have been built using Matlab and IDL). Ferret is free and distributed with LAS and largely pre- configured. It is what we are using for the NVODS LAS. For the most part, LAS (via Ferret) uses the NetCDF API to access DODS datasets. There is an exception for sequences (currently)where in some cases LAS uses Perl to create a temporary file for Ferret to read. The Live Access Server allows you to correct inadequate metadata on the fly without requiring you to recreate the data that you are serving. This is typically done by specifying in the XML configuration file that a particular initialization script other than the default should be used. The default merely opens the dataset in the back-end application and regrids variables for comparison if needed. A custom script may define axes, change the attributes of an axis (making a climatological axis modulo, or setting the depth axis upside-down), set the bad data flag, or define virtual variables. 

## Part 2 -Examples of Dataset Problems- 

**Missing Attributes**
-       bad dataflag/missing_value attribute
-       Usually this is straightforward.
**Dimensional Metadata**
-       COARDS Coordinate Variables
-       Fixing this can be time consuming.
-       need documentation.
-       custom scripts
**Time axes **
-       some time units not supported
-       "modulo" attribute
**Wrong Metadata**
-       Freeform file format descriptor
-       Ancillary DAS Files

**Notes:**
Missing Attributes 
A common attribute missing from the metadata is the bad data flag/missing_value attribute. In order to correct this for the LAS, the dataset must be explored in an application such as Ferret, and a custom initialization script must be written and incorporated into the data configuration XML file. Usually this is straightforward. 
Dimensional Metadata 
     COARDS conventions state that use metadata should include Coordinate Variables: 1-dimensional netCDF variables whose dimension names are identical to their variable names are regarded as "coordinate variables" (axes of the underlying grid structure of other variables defined on this dimension). When these are not found, documentation must be available describing the spatial and temporal characteristics of the data, testing must be done in Ferret to define correct axes for the data, then a custom initialization script must be written which defines a grid for the dataset. This can be time consuming. In an example of one dataset, the coordinates of all axes were unitless. I used Ferret to compare the geo-shape of the data with a known land mask, and continued shifting the grid until it fit the data. The metadata for the time axis was found in the human-readable documentation. Once the testing was completed, a back-end script needed to be written which defined all of the axes, and this information was incorporated into the data configuration XML file. 
Time axes 
   The Ferret application does not support time units such as "Julian Day" or "01-Jan". Other non-COARDS compliant units are often found. Climatological time axes are sometimes lacking a "modulo" attribute. When these situations arise, again, a custom back-end initialization script must be written and referenced in the configuration XML file. 
Wrong Metadata 
     Last September, I was trying to use DODS data from a DODS Freeform server. When I was testing the data, the shade plot along the lat and lon axes which ferret generated looked like a train wreck. I could not understand the problem, and the documentation was of no help. It turned out that the format descriptor for the file at the provider's site was mistakenly written to define the lon axis as having 180 points and the lat axis 360. This was corrected, but the plot still looked horrible. Another look at the freeform file showed that the axes should have had the respective dimensions of 171 and 61. This was corrected, but it turns out that the freeform files had no way of telling the DODS server what the axes' attributes were. A temporary solution was arrived at by using Ancillary DAS Files to augment the existing DAS files for the 5 or so datasets with this problem. This was a temporary fix in that there was no way of knowing if the data provider would be willing to write and store these Ancillary DAS Files. It would also take quite an effort to write individual Ancillary DAS Files for 100 or 10,000 files of this sort. 

## -DODS .das File with Missing Attributes- 

- Attributes {
-     time {
-     }
-     level {
-     }
-     mean_freq {
-     }
-     NC_GLOBAL {
-       String dtg "2002010700“
-       String model_name "WAM“
-       String dset_name "fcst_ops“
-       String geom_name "adriatic“
-       Float32 southern_latitude 40.;
-       Float32 northern_latitude 46.;
-       Float32 western_longitude 12.;
-       Float32 eastern_longitude 20.;
-       Float32 delta_x 0.08333332837;
-       Float32 delta_y 0.08333332837;
-    }
-    DODS_EXTRA {
-      String Unlimited_Dimension "time"
-    }

## Part 3 -NVODS/DODS Dataset Statistics 

**How does Metadata affect Performance? **
-     A summary of Tests on a sample of the DODS Dataset List
-     Dataset Ready to Use as accessed - 30%
-     Dataset Usable after Ferret repairs - 35%
-     Unable to use Dataset - 35%
- Please Note that not all of the problems with opening and using datasets were related with unavailable use metadata. Some of the reasons for not being able to use a Dataset included password needs, bad links, and incompatible file types. With the use of back-end scripts to enhance and correct use metadata, we were able to successfully access 65% of the DODS datasets tested.

## - Conclusion -

**How can the Ancillary Information Service Help?**
-     The NVODS LAS is Not Unique
-     Fix the Dataset, Not the Application
-     Allow a User to Determine the Fix

**Notes:**
How can the Ancillary Information Service Help? 

The NVODS LAS is Not Unique 
      Other DODS clients have similar issues. Unavailable or incomplete metadata can affect the the performance or even incapacitate DODS clients. Back-end applications other than Ferret running under LAS have similar issues. 
Fix the Dataset, Not the Application 
      The metadata problems different applications run into may be similar, but the solutions are varied. The Ancillary Information Service is a kind of one stop mechanism which enables many different applications and DODS clients to use the same dataset. 
Allow a User to Determine the Fix 
      The AIS framework is needed to allow users to configure metadata to the specific needs of the application they are using to access DODS datasets. DODS has made choices to get data online "despite" metadata problems. It has been successful so far in allowing a high quantity of data to be accessed. The metadata problem in DODS must be solved, but it can be approached incrementally, and the burden need not be put on the same people who are serving the data.