Package: dem-stitcher
dem-stitcherThis tool provides a raster of a Digital Elevation Model (DEM) over an area of interest utilizing global or continental, publicly available tile sets such as theGlobal Copernicus Digital Elevation Model at 30 meter resolution. See theDatasetssection below for all the tiles supported and their shortnames. This tool also performs some standard transformations for processing such as:the conversion of the vertical datum from a reference geoid to the WGS84 ellipsoidalthe accounting of a coordinate reference system centered at either the upper-left corner (Areatag) or center of the pixel (Pointtag).We rely on the GIS formats fromrasterio. The API can be summarized asfrom dem_stitcher import stitch_dem

# as xmin, ymin, xmax, ymax in epsg:4326
bounds = [-119.085, 33.402, -118.984, 35.435]

X, p = stitch_dem(bounds,
                  dem_name='glo_30',  # Global Copernicus 30 meter resolution DEM
                  dst_ellipsoidal_height=False,
                  dst_area_or_point='Point')
# X is an m x n numpy array
# p is a dictionary (or a rasterio profile) including relevant GIS metadata; CRS is epsg:4326Then, to save the DEM raster to disk:import rasterio

with rasterio.open('dem.tif', 'w', **p) as ds:
   ds.write(X, 1)
   ds.update_tags(AREA_OR_POINT='Point')The rasters are returned in the global lat/lon projectionepsg:4326and the API assumes that bounds are supplied in this format.InstallationIn order to easily manage dependencies, we recommend using dedicated project environments
viaAnaconda/Miniconda.
orPython virtual environments.dem_stitchercan be installed into a conda environment withconda install -c conda-forge dem_stitcheror into a virtual environment withpython -m pip install dem_stitcherCurrently, python 3.9+ is supported.With ISCE2 or gdalAlthough the thrust of using this package is for staging DEMs for InSAR (particularly ISCE2), testing and maintaining suitable environments to use with InSAR processors is beyond the scope of what we are attempting to accomplish here. We provide an example notebookherethat demonstrates how to stage a DEM for ISCE2, which requires additional packages than required for the package on its own. For the notebook, we use the environment found inenvironment.ymlof the Dockerized TopsApprepository, used to generate interferograms (GUNWs) in the cloud.About the raster metadataThe creation metadata unrelated to georeferencing (e.g. thecompresskey or various other optionshere) returned in the dictionaryprofilefrom thestitch_demAPI is copied directly from the source tiles being used if they are GeoTiff formatted (such asglo_30) else the creation metadata are copied from the GeoTiff Default Profile inrasterio(seehereexcludingnodataanddtype). Such metadata creation options are beyond the scope of this library.CredentialsThe accessing of NASADEM and SRTM require earthdata login credentials to be put into the~/.netrcfile. If these are not present, the stitcher will
fail withBadZipFile Erroras we userequeststo obtain zipped data and load the data usingrasterio. An entry in the.netrcwill look like:machine urs.earthdata.nasa.gov
    login <username>
    password <password>NotebooksWe have notebooks to demonstrate common usage:Basic DemoComparing DEMsGenerating a VRT from source DEM tilesStaging a DEM for ISCE2- this notebook requires the installation of a few extra libraries including ISCE2 viaconda-forgeWe also demonstrate how the tiles used to organize the urls for the DEMs were generated for this tool were generated in thisnotebook.DEMs SupportedTheDEMsthat are currently supported are:In [1]: from dem_stitcher.datasets import DATASETS; DATASETS
Out[1]: ['srtm_v3', 'nasadem', 'glo_90_missing', 'glo_30', '3dep', 'glo_90']The shortnames aboves are the strings required to usestitch_dem. Below, we expound upon these DEM shortnames and link to their respective data repositories.glo_30/glo_90: Copernicus GLO-30/GLO-90 DEM. The tile sets are the 30 and 90 meter resolution, respectively [link].The USGS DEM3dep: 3Dep 1/3 arc-second over North America - we are storing the ~10 meter resolution dataset. There are many more as notedhere. The files for these DEMs areheresrtm_v3: SRTM v3 [link]nasadem: Nasadem [link]glo_90_missing: these are tiles that are inglo_90but not inglo_30. They are over the countries Armenia and Azerbaijan. Used internally to help fill in gaps in coverage ofglo_30.All the tiles are given in lat/lon CRS (i.e.epsg:4326for global tiles orepsg:4269for USGS tiles in North America). A notable omission to the tile sets is the Artic DEMhere, which is suitable for DEMs merged at the north pole of the globe due to lat/lon distortion.If there are issues with obtaining dem tiles from urls embedded within the geojson tiles (e.g. a404error ashere), please see theDevelopmentsection below and/or open an issue ticket.DEM TransformationsWherever possible, we do not resample the original DEMs unless specified by the user to do so. When extents are specified, we obtain the the minimum  pixel extent within the merged tile DEMs that contain that extent. Any required resampling (e.g. updating the CRS or updating the resolution because the tiles have non-square resolution at high latitudes) is done after these required translations. We importantly note that order in which these transformations are done is crucial, i.e. first translating the DEM (to either pixel- or area-center coordinates) and then resampling is different than first resampling and then translation (as affine transformations are not commutative). Here are some notes/discussions:All DEMs are resampled toepsg:4326. Most DEMs are already in this CRS except the USGS DEMs over North America, which are inepsg:4269, whosexyprojection is alsolon/latbut has different vertical data. For our purposes, these two CRSs are almost identical. The nuanced differences between these CRS's is notedhere.All DEM outputs will have origin and pixel spacing aligning with the original DEM tiles unless a resolution for the final product is specified, which will alter the pixel spacing.Georeferenced rasters can be tied to map coordintaes using either (a) upper-left corners of pixels or (b) the pixel centers i.e.PointandAreatags ingdal, respectively, and seen as{'AREA_OR_POINT: 'Point'}. Note that tying a pixel to the upper-left cortner (i.e.Areatag) is thedefaultpixel reference forgdalas indicatedhere. Some helpful resources in reference to DEMs about this book-keeping are below.SRTM v3, NASADEM, and TDX arePixel-centered, i.e.{'AREA_OR_POINT: 'Point'}.The USGS DEMs arenot, i.e.{'AREA_OR_POINT: 'Area'}.Transform geoid heights to WGS84 Ellipsoidal height. This is done using the rastershere. We:Adjust the geoid to pixel/area coordinatesresample the geoids into the DEM reference frameAdjust the vertical datumAll DEMs are converted tofloat32and have nodatanp.nan. Although this can increase data size of certain rasters (SRTM is distributed in which pixels are recorded as integers), this ensures (a) easy comparison across DEMs and (b) no side-effects of the stitcher due to dtypes and/or nodata values. There is one caveat: the user can ensure that DEM nodata pixels are set to0usingmerge_nodata_valueinstitch_dem, in which case0is filled in wherenp.nanwas. We note specifying this "fill value" viamerge_nodata_valuedoesnotchange the nodata value of output DEM dataset (i.e.nodatain the rasterio profile will remainnp.nan). When transforming to ellipsoidal heights and setting0asmerge_nodata_value, the geoid values are filled in the DEMs nodata areas; if the geoid has nodata in the bounding box, this will be the source of subsequent no data.  For reference, this datatype and nodata is specified inmerge_tile_datasetsinmerge.py. Other nodata values can be specified outside the stitcher for the application of choice (e.g. ISCE2 requires nodata to be filled as0).There are somenotebooksthat illustrate how tiles are merged by comparing the output of our stitcher with the original tiles.As a performance note, when merging DEM tiles, we merge the needed tiles within the extent in memory and this process has an associated overhead. An alternative approach would be to download the tiles to disk and use virtual warping or utilize VRTs more effectively. Ultimately, the accuracy of the final DEM is our prime focus and these minor performance tradeoffs are sidelined.Dateline supportWe assume that the supplied bounds overlap the standard lat/lon CRS grid i.e. longitudes between -/+ 180 longitude and are within -/+ 90 latitude. If there is a single dateline crossing by the supplied bounds, then the tiles are wrapped the dateline and individually translated to a particular hemisphere dicated by the bounds provided to generate a continuous raster over the area provided. We assume a maximum of one dateline crossing in the bounds you specified (if you have multiple dateline crossings, thenstitch_demwill run out of memory). Similar wrapping tiles around the North and South poles (i.e. at -/+ 90 latitude) isnotsupported (a different CRS is what's required) and an exception will be raised.For DevelopmentThis is almost identical to normal installation:Clone this repogit clone https://github.com/ACCESS-Cloud-Based-InSAR/dem-stitcher.gitNavigate with your terminal to the repo.Create a new environment and install requirements usingconda env update --file environment.yml(or usemambato speed the install up)Install the package from cloned repo usingpython -m pip install -e .DEM UrlsIf urls or readers need to be updated (they consistently do) or you want to add a new global or large DEM, then there are two points of contact:The notebooks that format the geojsons used for this library arehereThe readers arehereThe former is the more likely. When re-generating tiles, make sure to run all tests including integration tests (i.e.pytest tests). For example, if regeneratingglotiles,glo-30requires both resolution parameters (30 meters and 90 meters) and an additional notebook for filling in missing 30 meter tiles. These should be clearly spelled out in the notebook linked above.TestingFor the test suite:Installpytestviaconda-forgeRunpytest testsThere are two category of tests: unit tests and integration tests. The former can be run usingpytest tests -m 'not integration'and similarly the latter withpytest tests -m 'integration'. Our unit tests are those marked without theintegrationtag (viapytest) that use synthetic data or data within the library to verify correct outputs of the library (e.g. that a small input raster is modified correctly). Integration tests ensure thedem-stitcherAPI works as expected, downloading the DEM tiles from their respective servers to ensure the stitcher runs to completion - the integration tests only make very basic checks to ensure the format of the ouptut data is correct (e.g. checking the output raster has a particular shape or that nodata isnp.nan). Our integration tests also include tests that run the notebooks that serve as documentation viapapermill(such tests have an additional tagnotebook). Integration tests will require the~/.netrcsetup above and working internet. Our testing workflow via Github actions currently runs the entire test suite except those tagged withnotebook, as these tests take considerably longer to run.ContributingWe welcome contributions to this open-source package. To do so:Create an GitHub issue ticket desrcribing what changes you need (e.g. issue-1)Fork this repoMake your modifications in your own forkMake a pull-request (PR) in this repo with the code in your fork and tag the repo owner or a relevant contributor.We useflake8and associated linting packages to ensure some basic code quality (see theenvironment.yml). These will be checked for each commit in a PR. Try to write tests wherever possible.SupportCreate an GitHub issue ticket desrcribing what changes you would like to see or to report a bug.We will work on solving this issue (hopefully with you).AcknowledgementsThis tool was developed to support cloud SAR processing using ISCE2 and various research projects at JPL. The early work of this repository was done by Charlie Marshak, David Bekaert, Michael Denbina, and Marc Simard. Since the utilization of this package for GUNW generation (see thisrepo), a subset of the ACCESS team, including Joseph (Joe) H. Kennedy, Simran Sangha, Grace Bato, Andrew Johnston, and Charlie Marshak, have improved this repository greatly. In particular, Joe Kennedy has lead the inclusion/development of actions, tests, packaging, distribution (including PyPI andconda-forge) and all the things to make this package more reliable, accessible, readable, etc. Simran Sangha has helped make sure output rasters are compatible with ISCE2 and other important bug-fixes.