Package: hydro-opendata
hydro-opendata📜中文文档Methods and paths for obtaining, managing, and utilizing open data for hydrological scientific computations.Free software: MIT licenseDocumentation:https://hydro-opendata.readthedocs.io/en/latest/BackgroundIn the era of artificial intelligence, data-driven hydrological models have been extensively researched and applied. With the advancements in remote sensing technologies and the trend towards open data sharing, accessing data has become more straightforward with a plethora of options. For researchers, questions like what data is required, what data can be accessed, where to download it, how to read it, and how to process it, are crucial. This repository aims to address these concerns.This repository primarily focuses on external open data, categorizing data types, and creating a list. It aims to build a data flow and its tech stack that can seamlessly "download-store-process-read-write-visualize" the data.Overall SolutionMain Data SourcesFrom our current understanding, the external data suitable for hydrological modeling includes but is not limited to:Primary CategorySecondary CategoryUpdate FrequencyData StructureExampleBasic GeographyHydrological ElementsStaticVectorWatershed boundary, siteTerrainStaticRasterDEM, flow direction, land useWeather & MeteorologyReanalysisDynamicRasterERA5Near Real-TimeDynamicRasterGPMForecastRollingRasterGFSImagerySatellite Remote SensingDynamicRasterLandsat, Sentinel, MODISStreet View ImagesStaticMultimediaSurveillance VideosDynamicMultimediaDrone FootageDynamicMultimediaCrowdsourced DataPOIStaticVectorBaidu MapSocial NetworksDynamicMultimediaWeiboHydrological DataRiver Flow DataDynamicTabularGRDCData can be categorized based on their update frequency into static and dynamic data.From a structural perspective, data can be classified into vector, raster, and multimedia (unstructured data).Structure and Functional Frameworkwis-stacData inventory and its metadata. Returns a data list based on AOI.wis-downloaderDownloads data from external sources. Depending on the data source, the download methods may vary, including:Integration with official APIs, e.g.,bmi_era5Retrieving data download links, e.g.,Herbie,MultiEarth,Satpy. Most cloud data platforms like Microsoft, AWS, etc., organize data mostly asstac.wis-processorPreprocesses the data, such as watershed averaging, feature extraction, etc.Useskerchunkto convert different format data tozarrformat and stores it inMinIOserver. This enables cross-file reading and enhances data reading efficiency.wis-s3apiAfter data processing in MinIO, it supports cross-file reading. Just provide data type, time range, and spatial range parameters to fetch the data.For remote sensing imagery, due to the vast amount of data, it's not feasible to download and read each file. One can usestac+stackstacto directly read Sentinel or Landsat data into an xarray dataset.wis-gistoolsIntegrates commonly used GIS tools, such as Kriging interpolation, Thiessen polygons, etc.Kriging interpolation:PyKrigeThiessen polygon:WhiteboxTools.VoronoiDiagram,scipy.spatial.VoronoiWatershed delineation:Rapid Watershed Delineation using an Automatic Outlet Relocation Algorithm,High-performance watershed delineation algorithm for GPU using CUDA and OpenMPWatershed averaging:plotting and creation of masks of spatial regionsVisualizationUseleafmapto display geospatial data within the Jupyter platform.Othershydro-GIS resource directory