Package: zarr-eosdis-store
zarr-eosdis-storeThe zarr-eosdis-store library allows NASA EOSDIS Collections to be accessed efficiently
by theZarr Python library <https://zarr.readthedocs.io/en/stable/index.html>_, provided they
have a sidecar DMR++ metadata file generated.InstallationThis module requires Python 3.8 or greater::$ python --version
Python 3.8.2Install from PyPI::$ pip install zarr-eosdis-storeTo install the latest development version::$ pip install pip install git+https://github.com/nasa/zarr-eosdis-store.git@main#egg=zarr-eosdis-storeEarthdata LoginTo access EOSDIS data, you need to sign in with a free NASA Earthdata Login account, which you can obtain at<https://urs.earthdata.nasa.gov/>_.Once you have an account, you will need to add your credentials to your~/.netrcfile::machine urs.earthdata.nasa.gov login YOUR_USERNAME password YOUR_PASSWORDIf you are accessing test data, you will need to use an account from the Earthdata Login test system at<https://uat.urs.earthdata.nasa.gov/>_ instead, adding a corresponding line to your~/.netrcfile::machine uat.urs.earthdata.nasa.gov login YOUR_USERNAME password YOUR_PASSWORDUsageTo use the library, simply instantiateeosdis_store.EosdisStorewith the URL to the data file you would
like to access, pass it to the Zarr library as you would with any other store, and use the Zarr API as with any
other read-only Zarr file.  Note: the URL to the data file will typically end with an HDF5 or NetCDF4 extension,
not .zarr... code-block:: pythonfrom eosdis_store import EosdisStore
import zarrAssumes you have set up .netrc with your Earthdata Login informationf = zarr.open(EosdisStore('https://example.com/your/data/file.nc4'))Read metadata and data from f using the Zarr APIprint(f['parameter_name'][0:0:0])If the data has _FillValue (to flag nodata), scale_factor, or add_offset set (defined in metadata using CF-conventions)
they can be retrieved from the parameter attributes... code-block:: pythonimport numpy as npscale_factor = f['parameter_name].scale_factor
add_offset = f['parameter_name].add_offset
nodata = f['parameter_name]._FillValuearr = f['parameter_name'][] * scale_factor + add_offsetnodata_locs = np.where(arr == nodata)A better way to handle these is to use XArray. Rather than reading the data immediately when a slice is requested, XArray
defers the read until the data is actually accessed. With the Zarr backend to XArray, the scale and offset can be set so that
when the data is accessed it will apply those values. This is more efficient if the data is going to be used in other operations.The scale_factor and get_offset will be used if specified in the NetCDF/HDF5 file... code-block:: pythonimport xarraystore = EosdisStore('https://example.com/your/data/file.nc4')f = xarray.open_zarr(store)the data is not read yetxa = f['parameter_name'][]convert to numpy array, data is readarr = xa.valuesThe resulting array will have had scale and offset applied, and any element that is equal to the _FillValue attribute will be
set to numpynan. To use XArray without apply the scale and offset or setting the nodata tonan, supply themask_and_scalekeyword to xarray.open_zarr to False:.. code-block:: pythonstore = EosdisStore('https://example.com/your/data/file.nc4')f = xarray.open_zarr(store, mask_and_scale=False)Technical SummaryWe make use of a technique to read NetCDF4 and some HDF5 files that was prototyped by The HDF Group and USGS, describedhere <https://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314)>_.To allow the technique to work with EOSDIS data, we have extended it and optimized access in the following key ways:TheEosdisStorereads a DMR++ file generated by OPeNDAP to present its metadata and determine byte offsets to the
Zarr library. By reusing these, we avoid needing to generate new metadata sidecar files to support new data.The store uses HTTPS and authenticates with a.netrcentry, rather than the S3 API, making it compatible with
EOSDIS access patterns and requirementsThe store caches redirect URLs for a period of time set by the Cache-Control header.  Doing this avoids the overhead
of repeated redirects when accessing parts of files.The store uses a parallel API that allows it to make more efficient access optimizations:When the Zarr library accesses data that requires reading multiple near-sequential bytes in the file, the store combines
these smaller requests into a single larger request.After an initial request to cache any authentication and redirect information, the store runs subsequent requests in
parallel.DevelopmentClone the repository, thenpip installits dependencies::pip install -r requirements.txt
pip install -r requirements-dev.txtTo check code coverage and run tests::coverage run -m pytestTo check coding style::flake8 . --count --select=E9,F63,F7,F82 --show-source --statisticsTo build documentation, generated atdocs/_build/html/index.html::cd docs && make html