Package: rasterioxyz
RasterioXYZIntroductionDesignMemory efficiencyFlexibilityExamplesCloud StorageBenchmarksRoadmapHighMediumLowTestingContributionsRasterioXYZPackage for tiling georeferenced Rasterio DatasetReader objects according to the XYZ tiles standard.IntroductionRasterioXYZ is a lightweight XYZ tile generator depending on the following packages:rasterionumpyWritten in Python 3.10.11 with Rasterio 1.3.9 and NumPy 1.26.2.For greater functionality in navigating the tile-tree, seeMercantile.DesignMemory efficiencyWhile reprojecting the entire source image at the maximum resolution required, thereof dictated by the maximum zoom level specified, would result in faster tiling, this represents a considerable potential source of memory issues. Such an approach would preclude the tiling of large images, be they large due to spatial resolution, data type, number of bands, and/or area covered. By lazily reading and, if needed, reprojecting windows of the source dataset at a given tile's resolution, memory use is kept low.FlexibilitySome basic design decisions for flexibility:Imagery of all data types and PROJ-recognised projections can be tiled with no alterations made to the original datasetTiles need not be generated for all zoom levels in a rangeTiles can be saved in PNG or JPEG format, though the XYZ standard dictates the former (as well as lossless and seemingly better colour balancing)Use of any Rasterio recognised resampling techniqueCreation of standard (256 px) or high (512 px) resolution tilesExamplesimg = rasterio.open("georeferenced_image.tif")
tiled = rasterioxyz.Tiles(image=img, zooms=range(10), pixels=256, resampling="bilinear")
tiles.write("out_directory")Several measurements can be taken to improve the speed of tiling:Employ source imagery already projected in EPSG:3857Specify a maximum zoom level at which tile spatial resolution does not greatly exceed that of the source imageryUse the default tile dimensions of 256 pixelsUse the default nearest-neighbour resamplingTiles saved locally can be served to desktop GIS platforms. This is useful in testing and evaluating results prior to incurring costs of cloud storage read and write operations.
The following demonstrates this withCommand PromptandQGISonWindows:In a Command Prompt window, change directory to that of the tiles. Alternatively, instantiate Command Prompt directly from the directory in File Explorer with CTRL + L, enter "cmd", then press Enter.Thehttpandsocketservermodules of the standard library may also be used to create an HTTP server with greater control over features like caching.In Command Prompt, enterpython -m http.server [port], where[port]is an unused TCP port (e.g., 8080).Add the connection in QGIS via theData Source ManagerorBrowser Panel:URLshould be set to the connection string, formatted as follows:http://localhost:8888/[directory]/{z}/{x}/{y}.[extension], where[directory]is the path to the folder within which zoom level folders, column folders, and tile images reside and[extension]is the format of tile images.Min. Zoom Levelshould be set to the minimum zoom level of the generated tiles.Max. Zoom Levelshould be set to the maximum zoom level of the generated tiles.Tile Resolutionshould be set to the resolution of the generated tiles, denoted by theTilesclass'pixelsparameter.The XYZ layer should now be available for addition to the Layers Panel and navigable within the map canvas.Once added, assuming the connection is OK, the Command Prompt window will display the tiles requested by QGIS as the map is navigated.Cloud StorageGenerated tiles can be written to cloud storage services like Azure Blob Storage and Amazon S3.
Once relevant functionality has been added, an example will be shown here.
For now, the generated data can be written to cloud storage using the relevant SDK.BenchmarksTimes are measured in seconds for a single 100 KM by 100 KM, 10m resolution, uint8 Sentinel-2 scene. The image was tiled up to and including zoom level 13 (the zoom level before which 256 pixel tiles of the source image require source resolution upscaling), producing 983 tiles. Tiles were written in PNG format by both applications.MethodThreadsSource image CRSTile dimensions (px)Zooms 0-13 time (s)rasterioxyz1EPSG:3261725631.79rasterioxyz1EPSG:3261751281.33rasterioxyz1EPSG: 385725617.01rasterioxyz1EPSG: 385751232.01QGIS16EPSG:326172564.13QGIS16EPSG:326175129.37QGIS16EPSG:38572562.06QGIS16EPSG:38575124.96Clearly, QGIS is - at present - many times faster (between 6.6 and 8.7 times). It's likely that some combination of threads, use of lower-level languages, on-the-fly reprojection (in the case of non-3857 images), caching, and pyramids may contribute to QGIS' speed. Potential adoption of threading across bottlenecks (reprojection, writing) is a high priority.That said, tiles generated by RasterioXYZ appear to exhibit better horizontal accuracy and fewer resampling-derived artefacts than those of QGIS:Source image (EPSG:32617) without (T) and with (B) OSM overlay.QGIS-generated tiles without (T) and with (B) OSM overlay.RasterioXYZ-generated tiles without (T) and with (B) OSM overlay.RoadmapA number of improvements are possible and/or planned. Sorted by priority, these are as follows.HighBetter handling of non-uint8 datasets with extreme outlying valuesFor example, uint16 Sentinel-2 imagery with extreme outliers in the form of snow or cloud may result in poorly contrasted tilesSource data does not display as such in QGIS due to percentile minimum/maximum stretching (i.e., 2nd and 98th percentiles are set as the minimum and maximum values)In-built support for AWS, Azure, and GCP cloud storage writeIdeally via requests to reduce additional dependenciesEnable user selection of no data valueInvestigate threads for data reprojection and writingReduce the write time of tile imagesPIL and OpenCV write to file faster but increase dependenciesStatic method to identify maximum zoom level without data upscalingMediumTile data within a user-provided areaEnable user selection of raster bands/channels to tileAdd command line functionalityEnable tiling of sequences of Rasterio datasetsSupport for colour ramps/tables with singleband imagesLowA resume/continue modeGreater support for different argument values/types (e.g., int/float for resampling)Generate HTML map of tiled dataTestingThe test suite for RasterioXYZ uses Rasterio MemoryFile objects containing randomly generated data of specified dtype projected in a specified CRS.To add additional mock data for testing, do the following inconftest.py:Add the target CRS integer toSUPPORTED_TEST_CRSAdd the target bounding box in said CRS toTEST_BOUNDSIf desired, change the values forheight,width, andcount- these are currently hardcodedThen, intest_tile.py:Add the appropriate arguments to thepytest.mark.parameterize()decorator for the method to test. For example, after adding EPSG:4269 (NAD83) and a corresponding bounding box, the argument for thetest_dataparameter may appear like so:{"crs": 4269, "dtype": "uint8"}Currently, tests cannot be written to validate the correct tiles (rows and columns) are written by thewrite()method due to said method exhausting the generator of objects containing the requisite information.ContributionsFeel free to raise any issues, especially bugs and feature requests!