Package: async-retriever
PackageDescriptionStatusPyNHDNavigate and subset NHDPlus (MR and HR) using web servicesPy3DEPAccess topographic data through National Map’s 3DEP web servicePyGeoHydroAccess NWIS, NID, WQP, eHydro, NLCD, CAMELS, and SSEBop databasesPyDaymetAccess daily, monthly, and annual climate data via DaymetPyGridMETAccess daily climate data via GridMETPyNLDAS2Access hourly NLDAS-2 data via web servicesHydroSignaturesA collection of tools for computing hydrological signaturesAsyncRetrieverHigh-level API for asynchronous requests with persistent cachingPyGeoOGCSend queries to any ArcGIS RESTful-, WMS-, and WFS-based servicesPyGeoUtilsUtilities for manipulating geospatial, (Geo)JSON, and (Geo)TIFF dataAsyncRetriever: Asynchronous requests with persistent cachingFeaturesAsyncRetriever is a part ofHyRiversoftware stack that
is designed to aid in hydroclimate analysis through web services. This package serves as HyRiver’s
engine for asynchronously sending requests and retrieving responses astext,binary, orjsonobjects. It uses persistent caching usingaiohttp-client-cacheto speed up the retrieval
even further. Moreover, thanks tonest_asyncioyou can use this package in Jupyter notebooks. Although this package is part of the HyRiver
software stack, it can be used for any web calls. There are three functions that you can
use to make web calls:retrieve_text: Get responses astextobjects.retrieve_binary: Get responses asbinaryobjects.retrieve_json: Get responses asjsonobjects.stream_write: Stream responses and write them to disk in chunks.You can also use the general-purposeretrievefunction to get responses as any
of the three types. All responses are returned as a list that has the same order as the
input list of requests. Moreover, there is another function calleddelete_url_cachefor removing all requests from a cache file that contains a given URL.You can control the request/response caching behavior and verbosity of the package
by setting the following environment variables:HYRIVER_CACHE_NAME: Path to the caching SQLite database. It defaults to./cache/aiohttp_cache.sqliteHYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to
one week.HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.HYRIVER_SSL_CERT: Path to a SSL certificate file.For example, in your code before making any requests you can do:importosos.environ["HYRIVER_CACHE_NAME"]="path/to/file.sqlite"os.environ["HYRIVER_CACHE_EXPIRE"]="3600"os.environ["HYRIVER_CACHE_DISABLE"]="true"os.environ["HYRIVER_SSL_CERT"]="path/to/cert.pem"You can find some example notebookshere.You can also try using AsyncRetriever without installing
it on your system by clicking on the binder badge. A Jupyter Lab
instance with the HyRiver stack pre-installed will be launched in your web browser, and you
can start coding!Moreover, requests for additional functionalities can be submitted viaissue tracker.CitationIf you use any of HyRiver packages in your research, we appreciate citations:@article{Chegini_2021,author={Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},doi={10.21105/joss.03175},journal={Journal of Open Source Software},month={10},number={66},pages={1--3},title={{HyRiver: Hydroclimate Data Retriever}},volume={6},year={2021}}InstallationYou can installasync-retrieverusingpip:$pipinstallasync-retrieverAlternatively,async-retrievercan be installed from theconda-forgerepository
usingConda:$condainstall-cconda-forgeasync-retrieverQuick startAsyncRetriever by default creates and/or uses./cache/aiohttp_cache.sqliteas the cache
that you can customize by thecache_nameargument. Also, by default, the cache doesn’t
have any expiration date and thedelete_url_cachefunction should be used if you know
that a database on a server was updated, and you want to retrieve the latest data.
Alternatively, you can use theexpire_afterto set the expiration date for the cache.As an example for retrieving abinaryresponse, let’s use the DAAC server to getNDVI.
The responses can be directly passed toxarray.open_mfdatasetto get the data as
axarrayDataset. We can also disable SSL certificate verification by settingssl=False.importioimportxarrayasxrimportasync_retrieverasarfromdatetimeimportdatetimewest,south,east,north=(-69.77,45.07,-69.31,45.45)base_url="https://thredds.daac.ornl.gov/thredds/ncss/ornldaac/1299"dates_itr=((datetime(y,1,1),datetime(y,1,31))foryinrange(2000,2005))urls,kwds=zip(*[(f"{base_url}/MCD13.A{s.year}.unaccum.nc4",{"params":{"var":"NDVI","north":f"{north}","west":f"{west}","east":f"{east}","south":f"{south}","disableProjSubset":"on","horizStride":"1","time_start":s.strftime("%Y-%m-%dT%H:%M:%SZ"),"time_end":e.strftime("%Y-%m-%dT%H:%M:%SZ"),"timeStride":"1","addLatLon":"true","accept":"netcdf",}},)fors,eindates_itr])resp=ar.retrieve_binary(urls,kwds,max_workers=8,ssl=False)data=xr.open_mfdataset(io.BytesIO(r)forrinresp)We can remove these requests and their responses from the cache like so:ar.delete_url_cache(base_url)For ajsonresponse example, let’s get water level recordings of an NOAA’s water level station,
8534720 (Atlantic City, NJ), during 2012, using CO-OPS API. Note that this CO-OPS product has a
31-day limit for a single request, so we have to break the request down accordingly.importpandasaspdstation_id="8534720"start=pd.to_datetime("2012-01-01")end=pd.to_datetime("2012-12-31")s=startdates=[]foreinpd.date_range(start,end,freq="m"):dates.append((s.date(),e.date()))s=e+pd.offsets.MonthBegin()url="https://api.tidesandcurrents.noaa.gov/api/prod/datagetter"urls,kwds=zip(*[(url,{"params":{"product":"water_level","application":"web_services","begin_date":f'{s.strftime("%Y%m%d")}',"end_date":f'{e.strftime("%Y%m%d")}',"datum":"MSL","station":f"{station_id}","time_zone":"GMT","units":"metric","format":"json",}},)fors,eindates])resp=ar.retrieve_json(urls,kwds)wl_list=[]forrjsoninresp:wl=pd.DataFrame.from_dict(rjson["data"])wl["t"]=pd.to_datetime(wl.t)wl=wl.set_index(wl.t).drop(columns="t")wl["v"]=pd.to_numeric(wl.v,errors="coerce")wl_list.append(wl)water_level=pd.concat(wl_list).sort_index()water_level.attrs=rjson["metadata"]Now, let’s see an example without any payload or headers. Here’s how we can retrieve
harmonic constituents of several NOAA stations from CO-OPS:stations=["8410140","8411060","8413320","8418150","8419317","8419870","8443970","8447386",]base_url="https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations"urls=[f"{base_url}/{i}/harcon.json?units=metric"foriinstations]resp=ar.retrieve_json(urls)amp_list=[]phs_list=[]forrjsoninresp:sid=rjson["self"].rsplit("/",2)[1]const=pd.DataFrame.from_dict(rjson["HarmonicConstituents"]).set_index("name")amp=const.rename(columns={"amplitude":sid})[sid]phase=const.rename(columns={"phase_GMT":sid})[sid]amp_list.append(amp)phs_list.append(phase)amp=pd.concat(amp_list,axis=1)phs=pd.concat(phs_list,axis=1)ContributingContributions are appreciated and very welcomed. Please readCONTRIBUTING.rstfor instructions.