Package: usgs-topo-tiler
usgs-topo-tilerPython package to read Web Mercator map tiles from USGS Historical Topographic
Maps, and utilities to create aMosaicJSONcollection from these
maps.A mosaic of USGS historical maps around the Grand Canyon, with added relief
shading.OverviewI stumbled upon a hidden gem: the entire USGS historical topographic map
archive, consisting of 183,112 digitized maps created between 1884 and 2006, is
stored in Cloud-Optimized GeoTIFF (COG) format on AWS S3.The fact that maps are accessible publicly and stored in COG format means that
you can easily and cheaply set up a serverless function on AWS Lambda to serve
map tiles on the fly.TheCOG formatis a backwards-compatible, cloud-native storage format
for raster files that allow selected portions of the file to be read over the
network without needing to download and parse the entire file. This fast random
read access allows for dynamic tiling of map tiles on demand, without needing to
preprocess and store any map data.There are three parts to serving your own tiles:usgs-topo-tiler: a library to extract asingleWeb Mercator tile fromonesource historical map.usgs-topo-tiler's CLI, which helps to construct MosaicJSON files. These files tellusgs-topo-mosaicwhat source files should be combined to create a single Web Mercator tile.usgs-topo-mosaic: a library to use a MosaicJSON file created above to create a seamless mosaic of tiles. This is designed to be deployed with AWS Lambda and AWS API Gateway as a serverless tile endpoint.Generate a Web Mercator tileInstallpipinstallusgs-topo-tiler'rio-tiler>=2.0a6'UsageHere I'll show a quick overview of reading a single mercator tile from a single
USGS historical map. If you're looking for a specific map, the simplest way is
probably to use theNational Map Viewer. Check the box for
"Historical Topographic Maps", make sure the file format is "GeoTIFF". Click
"Find Products", and then right click "Download" to get the HTTPS url to the
GeoTIFF on S3.For this demo, I'll make a mercator tile from an 1897 topographic map of
Yosemite Valley.Read a tile from a file over the internetfromusgs_topo_tilerimporttileurl='https://prd-tnm.s3.amazonaws.com/StagedProducts/Maps/HistoricalTopo/GeoTIFF/CA/CA_Yosemite_299696_1897_125000_geo.tif'# Mercator tilex,y,z=687,1583,12tile,mask=tile(url,x,y,z,tilesize=512)print(tile.shape)# (3, 512, 512)print(mask.shape)# (512, 512)Create image from tileNote that if you're usingrio-tilerv1, you should replacerenderwitharray_to_image.fromrio_tiler.utilsimportrenderbuffer=render(tile,mask,img_format='png')Write image to filewithopen('yose_1897.png','wb')asf:f.write(buffer)You now have a 512x512 png image aligned with the Web Mercator grid, and because
the source is aCloud-optimizedGeoTIFF, the image was made with a minimal
number of requests to the source, and without reading the entire GeoTIF.Create a MosaicJSONThe process described above is for createonetile. But often we want to join
many mercator tiles to make a seamless web map. This is whereMosaicJSONcomes in. It describes how to join sources and where to
place them.This section outlines how to create a MosaicJSON from USGS historical topo
assets. This MosaicJSON can then be used withusgs-topo-mosaicto serve a web
map of tiles with AWS Lambda.InstallWhen you installusgs-topo-tiler, it doesn't include dependencies necessary
for the CLI commands so as to keep the deployment size small when used withusgs-topo-mosaic.To install the additional CLI dependencies, run:pipinstall'usgs-topo-tiler[cli]'Download bulk metadataFirst you need to download metadata including at least the geospatial bounds of
each map, and the URL of each map. It's possible to use theNational Map
APIto retrieve such metadata, however I prefer to download a
CSV of bulk metadata from S3. This file includes metadata on every image in
their collection.mkdir-pdata
wgethttps://prd-tnm.s3-us-west-2.amazonaws.com/StagedProducts/Maps/Metadata/topomaps_all.zip-Pdata/
unziptopomaps_all.zipdata/topomaps_all.csvis the extracted bulk metadata file.data/readme.txthas helpful information about what fields are in the bulk metadata file.Download list of COG files:Occasionally there are some files listed in the metadata that don't exist as
GeoTIFF. I download a list of the.tiffiles on S3 so that I can
cross-reference against the metadata and ensure that only files that exist are
included in the MosaicJSON.This step is optional, but recommended.mkdir-pdata/
usgs-topo-tilerlist-s3\--bucket'prd-tnm'\--prefix'StagedProducts/Maps/HistoricalTopo/GeoTIFF/'\--ext'.tif'\>data/geotiff_files.txt

>wc-ldata/geotiff_files.txt183112data/geotiff_files.txt183112COG files!Create MosaicJSONNow you're ready to start creating mosaics. This isn't entirely straightforward
because maps were created at different times and at different scales in
different regions of the U.S. So it's not usually as simple as creating a mosaic
of all maps of a single scale, unless you're ok with having gaps in the mosaic
spatially, where tiles might return empty data.FootprintsOne of the best way to visually see the results of a filtering query on the
metadata is to generatefootprintsand then display them on a map.Lets say we want to create a mosaic of mid-scale maps. We can use the--filter-onlycli flag to export newline-delimited GeoJSON of the query.usgs-topo-tilermosaic-bulk\--meta-pathdata/topomaps_all.csv\--s3-list-pathdata/geotiff_files.txt\--min-scale63360\--max-scale249000\--filter-only\>mid_scale_footprints.geojsonlThen we can visualize this data, e.g. withmy CLIforkepler.gl.keplermid_scale_footprints.geojsonlThis illustrates the core problem of these historical maps when making a
MosaicJSON. Some areas have been mapped more than others, and some have never
been mapped at this scale range. If you were to create a MosaicJSON from these
parameters, you'd get empty images when requesting data over Northern Montana
and Western Texas.Generate MosaicJSONOnce you know the desired parameters of your query, remove the--filter-onlyflag to generate the MosaicJSON. For example:usgs-topo-tilermosaic-bulk\--meta-pathdata/topomaps_all.csv\--s3-list-pathdata/geotiff_files.txt\--min-scale63360\--max-scale249000\>mid_scale_mosaic.jsonmid_scale_mosaic.jsonis now a MosaicJSON file that can be used withusgs-topo-mosaicto render a web map. Note, however that this uses a custom
asset string format, as described inRemoving Map
Collars, and won't necessarily work with all other
MosaicJSON tools.ExamplesLow zoom, newest availableusgs-topo-tilermosaic-bulk\--meta-pathdata/topomaps_all.csv\--s3-list-pathdata/geotiff_files.txt\--min-scale250000\>mosaic_low.jsonMedium zoom, newest available, filling in with lower-resolution maps where necessaryusgs-topo-tilermosaic-bulk\--meta-pathdata/topomaps_all.csv\--s3-list-pathdata/geotiff_files.txt\--min-scale63360\>mosaic_medium.jsonMedium zoom, oldest available, filling in with lower-resolution maps where necessaryusgs-topo-tilermosaic-bulk\--meta-pathdata/topomaps_all.csv\--s3-list-pathdata/geotiff_files.txt\--min-scale63360\--sort-preferenceoldest\>mosaic_medium_oldest.jsonHigh zoom, newest available, continental U.S. onlyusgs-topo-tilermosaic-bulk\--meta-pathdata/topomaps_all.csv\--s3-list-pathdata/geotiff_files.txt\--min-scale24000\--max-scale63359\`# Lower 48 states only` \--bounds'-161.96,12.85,-55.01,50.53'\>mosaic_high.jsonAPIUsage: usgs-topo-tiler mosaic-bulk [OPTIONS]

  Create MosaicJSON from CSV of bulk metadata

Options:
  --meta-path PATH                Path to csv file of USGS bulk metadata dump
                                  from S3  [required]

  --s3-list-path PATH             Path to txt file of list of s3 GeoTIFF files
  --min-scale FLOAT               Minimum map scale, inclusive
  --max-scale FLOAT               Maximum map scale, inclusive
  --min-year FLOAT                Minimum map year, inclusive
  --max-year FLOAT                Maximum map year, inclusive
  --woodland-tint / --no-woodland-tint
                                  Filter on woodland tint or no woodland tint.
                                  By default no filtering is applied.

  --allow-orthophoto              Allow orthophoto  [default: False]
  --bounds TEXT                   Bounding box for mosaic. Must be of format
                                  "minx,miny,maxx,maxy"

  -z, --minzoom INTEGER           Force mosaic minzoom
  -Z, --maxzoom INTEGER           Force mosaic maxzoom
  --quadkey-zoom INTEGER          Force mosaic quadkey zoom
  --sort-preference [newest|oldest|closest-to-year]
                                  Method for choosing assets within a given
                                  mercator tile at the quadkey zoom.
                                  [default: newest]

  --closest-to-year INTEGER       Year used for comparisons when preference is
                                  closest-to-year.

  --filter-only                   Output filtered GeoJSON features, without
                                  creating the MosaicJSON. Useful for
                                  inspecting the footprints   [default: False]

  --help                          Show this message and exit.AddendumRemoving map collarsAll of the USGS historical maps havecollars, regions of space around the map
where metadata is printed. In order to create continuous map tiles from a
collection of these maps, these collars have to be clipped, so that only the map
is showing.These maps are georeferenced, which means that it's straightforward to remove
the collar when you know the actual bounds contained in the map. However, I've
found that there's no reliable way to determine the bounds on the fly with just
the image and its filename.While building the mosaic ahead of time, you have access to this information,
but with the usual tiling setup, you'd only have access to the URL and image
while tiling.To get around this, I apply a "hack" to the MosaicJSON format. Instead of just
encoding a URL string, I encode the urlandthe bounds of the map as a JSON
string.Summary: when you build a mosaic using the cli in this library, it encodes a
non-standard MosaicJSON that works well with theusgs-mosaic-tilertiler, but
isn't necessarily readable by other MosaicJSON tools