Package: gps2var
gps2vargps2varis a Python library providing fast loading of geospatial variables from raster files by GPS coordinates and with interpolation.In particular, it allows parallel calls with coordinates specified as large NumPy arrays of arbitrary shapes, and is compatible with PyTorch data loaders.Install with:pip install gps2varSeebenchmark.ipynbfor a performance benchmark.ExamplesReading from a single filePATH="/vsizip/wc2.1_30s_bio.zip/wc2.1_30s_bio_1.tif"# WorldClim annual mean temperaturewithgps2var.RasterValueReader(PATH,interpolation='bilinear')asreader:lat,lon=48.858222,2.2945reader.get(lon,lat)# [11.94036207]Parallel reading from multiple filesMultiRasterValueReaderreads from multiple raster files and concatenates the results.# min and max temperature by month (averages for 1970-2000)PATHS=[f"/vsizip/wc2.1_30s_{var}.zip/wc2.1_30s_{var}_{i:02}.tif"forvarin["tmin","tmax"]foriinrange(1,13)]withgps2var.MultiRasterValueReader(PATHS,num_threads=len(PATHS))asreader:lat,lon=48.858222,2.2945reader.get(lon,lat).reshape(2,12)# [[ 2.3  2.5  4.6  6.3 10.1 13.  15.  14.9 12.   8.8  5.   3.4]#  [ 7.2  8.4 11.9 14.9 19.2 22.  24.7 24.8 20.9 15.9 10.6  8. ]]Setuse_multiprocessing=Trueto create a separate process for each raster. This is likely to be faster than the default (i.e. multithreading), at least if the number of rasters is large.APIRasterValueReaderCan be created with a path to a file that can be read by Rasterio, or an open RasterioDatasetReader. The behavior can be customized with keyword parameters; the most important ones are:crs: The coordinate reference system to use for querying. By default this is EPSG:4326, i.e. longitude and latitude (in this order) as used by GPS.interpolation:"nearest"(default) or"bilinear"(slower).fill_value: The value (scalar) with which to replace missing values. Defaults tonp.nan.feat_dtype: The dtype to which to convert the result. Defaults tonp.float32.feat_center: Center each of the features at the given value. Defaults toNone(no centering).feat_scale: Scale each of the (centered) features by multiplying it by the given value. Defaults toNone(no scaling).block_shape: The shape of the blocks read into memory (and stored in the GDAL block cache).preload_all: Indicates whether the whole dataset should be loaded into memory instead of loading one block at a time. Defaults toFalse.Another option is to wrap all these arguments in aRasterReaderSpec(or simply a dictionary) and pass it as the first argument.RasterValueReaderPoolLikeRasterValueReader, but spawnsnum_workersworker processes that all read from the same file concurrently.
Besidesget()(which blocks until the result is ready), it providesasync_get(), which returns aconcurrent.futures.Future.MultiRasterValueReaderExpects as the first argument a list of file paths,RasterReaderSpecs, ordicts, and reads from each file in a separate thread or process. Additional options to be applied to all items can be passed as keyword arguments. Additionally, the following parameters are accepted:use_multiprocessing: IfTrue, each raster will be processed in a separate process.num_threads: The number of threads to use for parallel reading. By default, this is set to the number of rasters. Set to 0 to read in the main thread.ProcessManagerAmultiprocessing.BaseManager– a context manager that spawns a separate process. It providesRasterValueReader(),MultiRasterValueReader(), andRasterValueReaderPool()methods that create the corresponding reader in that process and return a proxy object that can be used in much the same way as the reader itself. A nice property of a proxy object is that it can be copied between processes without copying the underlying reader, so it works with PyTorchDataLoader.PyTorchDataLoaderand parallelismSimply using aRasterValueReaderwith a PyTorchDataLoaderwithnum_workers > 0and with the"fork"start method(default on Unix)will not work.Here are examples of usage that do work:Usingmultiprocessing.set_start_method("spawn"). This will create a copy of the reader in each worker process.Settingpreload_all=Trueso that the whole raster is loaded into memory.UsingMultiRasterValueReaderas above, but withuse_multiprocessing=True. This way, each raster wil be read in a separate process.UsingProcessManager, e.g.:# in __init__:self.manager=gps2var.ProcessManager()self.manager.start()# start a new processself.reader=manager.RasterValueReader(path)# proxy object# in __getitem__:self.reader.get(lon,lat)In this case, the reader is placed in a separate process and the workers connect to it using the proxy object.Caveats and limitationsgps2varreads the raster in windows, which is efficientif the locations requested in a single call tend to be close together. If this is not the case, better performance can be achieved usingpreload_all=Trueat the expense of loading the whole raster into memory.By default, the window shape used bygps2varis identical to the shape of the blocks in which the dataset is stored. This ensures the windows can be read efficiently, but it might mean reading a lot of data unnecessarily if the block size is large. Adjusting the window shape using theblock_shapeparamater might improve performance.