Package: geoprepare
geoprepareA Python package to prepare (download, extract, process input data) for GEOCIF and related modelsFree software: MIT licenseDocumentation:https://ritviksahajpal.github.io/geoprepareInstallationInstall AnacondaWe recommend that you use the conda package manager to install thegeopreparelibrary and all its
dependencies. If you do not have it installed already, you can get it from theAnaconda distributionUsing the CDS APIIf you intend to download AgERA5 data, you will need to install the CDS API.
You can do this by following the instructionshereCreate a new conda environment (optional but highly recommended)geopreparerequires multiple Python GIS packages includinggdalandrasterio. These packages are not always easy
to install, especially on Windows. To make the process easier, you can optionally create a new environment using the
following commands, specify the python version you have on your machine (python >= 3.9 is recommended). we use thepygislibrary
to install multiple Python GIS packages includinggdalandrasterio.condacreate--name<name_of_environment>python=3.xcondaactivate<name_of_environment>condainstall-cconda-forgemambamambainstall-cconda-forgegdalmambainstall-cconda-forgerasteriomambainstall-cconda-forgexarraymambainstall-cconda-forgerioxarraymambainstall-cconda-forgepyresamplemambainstall-cconda-forgecdsapimambainstall-cconda-forgepygispipinstallwgetpipinstallpyl4cInstall the octvi package to download MODIS datapipinstallgit+https://github.com/ritviksahajpal/octvi.gitDownloading from the NASA distributed archives (DAACs) requires a personal app key. Users must
configure the module using a new console script,octviconfig. After installation, runoctviconfigin your command prompt to prompt the input of your personal app key. Information on obtaining app keys
can be foundhereUsing PyPi (default)pipinstall--upgradegeoprepareUsing Github repository (for development)pipinstall--upgrade--no-deps--force-reinstallgit+https://github.com/ritviksahajpal/geoprepare.gitLocal installationNavigate to the directory containingsetup.pyand run the following command:pipinstall.UsageExecute the following code to download the datafromgeoprepareimportgeoprepare# Provide full path to the configuration files# Download and preprocess datageoprepare.run([r"PATH_TO_geoprepare.txt"])Execute the following code to extract crop masks and EO datafromgeoprepareimportgeoextract# Extract crop masks and EO variablesgeoextract.run([r"PATH_TO_geoprepare.txt",r"PATH_TO_geoextract.txt"])Execute the following code to prepare the data for the crop yield ML model and AgMet graphicsfromgeoprepareimportgeomerge# Merge EO files into one, this is needed to create AgMet graphics and to run the crop yield modelgeomerge.run([r"PATH_TO_geoprepare.txt",r"PATH_TO_geoextract.txt"])Before running the code above, we need to specify the two configuration files:geoprepare.txtcontains configuration settings for downloading and processing the input data.geoextract.txtcontains configuration settings for extracting crop masks and EO variables.Configuration filesgeoprepare.txtdatasets: Specify which datasets need to be downloaded and processeddir_base: Path where to store the downloaded and processed filesstart_year,end_year: Specify time-period for which data should be downloaded and processedlogfile: What directory name to use for the log fileslevel: Which level to use forloggingparallel_process: Whether to use multiple CPUsfraction_cpus: What fraction of available CPUs to use[DATASETS]datasets=['CPC','SOIL-MOISTURE','LST','CPC','AVHRR','AGERA5','CHIRPS','CHIRPS-GEFS'][PATHS]dir_base=/home/servir/GEOCIFdir_input=${dir_base}/inputdir_log=${dir_base}/logdir_interim=${dir_input}/interimdir_download=${dir_input}/downloaddir_output=${dir_base}/outputdir_global_datasets=${dir_input}/global_datasetsdir_masks=${dir_global_datasets}/masksdir_regions=${dir_global_datasets}/regionsdir_regions_shp=${dir_regions}/shpsdir_crop_masks=${dir_input}/crop_masksdir_models=${dir_input}/models[AGERA5]start_year=2022[AVHRR]data_dir=https://www.ncei.noaa.gov/data/avhrr-land-normalized-difference-vegetation-index/access[CHIRPS]fill_value=-2147483648prelim=/pub/org/chc/products/CHIRPS-2.0/prelim/global_daily/tifs/p05/final=/pub/org/chc/products/CHIRPS-2.0/global_daily/tifs/p05/start_year=2022[CHIRPS-GEFS]fill_value=-2147483648data_dir=/pub/org/chc/products/EWX/data/forecasts/CHIRPS-GEFS_precip_v12/15day/precip_mean/[CPC]data_dir=ftp://ftp.cdc.noaa.gov/Datasets[ESI]data_dir=https://gis1.servirglobal.net//data//esi//[FLDAS][LST]num_update_days=7[NDVI]product=MOD09CMGvi=ndviscale_glam=Falsescale_mark=Trueprint_missing=False[SOIL-MOISTURE]data_dir=https://gimms.gsfc.nasa.gov/SMOS/SMAP/L03/[LOGGING]level=ERROR[DEFAULT]logfile=logparallel_process=Falsefraction_cpus=0.5start_year=2022end_year=2022geoextract.txtcountries: List of countries to processforecast_seasons: List of seasons to processmask: Name of file to use as a mask for cropland/croptyperedo: Redo the processing for all days (True) or only days with new data (False)threshold: Use athresholdvalue (True) or apercentile(False) on the cropland/croptype maskfloor: Value below which to set the mask to 0ceil: Value above which to set the mask to 1eo_model: List of datasets to extract from[kenya]category=EWCMscales=['admin_1'];canbeadmin_1(statelevel)oradmin_2(countylevel)growing_seasons=[1];1isprimary/longseason,2issecondary/shortseasoncrops=['mz','sr','ml','rc','ww','tf']use_cropland_mask=True[rwanda]category=EWCMscales=['admin_1'];canbeadmin_1(statelevel)oradmin_2(countylevel)growing_seasons=[1];1isprimary/longseason,2issecondary/shortseasoncrops=['mz','sr','ml','rc','ww','tf']use_cropland_mask=True[malawi]category=EWCMscales=['admin_1'];canbeadmin_1(statelevel)oradmin_2(countylevel)growing_seasons=[1];1isprimary/longseason,2issecondary/shortseasoncrops=['mz','sr','ml','rc','ww','tf']use_cropland_mask=True[zambia]category=EWCMscales=['admin_1'];canbeadmin_1(statelevel)oradmin_2(countylevel)growing_seasons=[1];1isprimary/longseason,2issecondary/shortseasoncrops=['mz','sr','ml','rc','ww','tf']use_cropland_mask=True[united_republic_of_tanzania]category=EWCMscales=['admin_1'];canbeadmin_1(statelevel)oradmin_2(countylevel)growing_seasons=[1];1isprimary/longseason,2issecondary/shortseasoncrops=['mz','sr','ml','rc','ww','tf']use_cropland_mask=True[ww]mask=cropland_v9.tif[mz]mask=cropland_v9.tif[sb]mask=cropland_v9.tif[rc]mask=cropland_v9.tif[tf]mask=cropland_v9.tif[sr]mask=cropland_v9.tif[ml]mask=cropland_v9.tif[EWCM]calendar_file=EWCM_2021-6-17.xlsx[AMIS]calendar_file=AMISCM_2021-6-17.xlsx[DEFAULT]redo=Falsethreshold=Truefloor=20ceil=90scales=['admin_1']growing_seasons=[1]countries=['kenya']forecast_seasons=[2022]mask=cropland_v9.tifshp_boundary=EWCM_Level_1.shpstatistics_file=statistics.csvzone_file=countries.csvcalendar_file=crop_calendar.csveo_model=['ndvi','cpc_tmax','cpc_tmin','chirps','chirps_gefs','esi_4wk','soil_moisture_as1','soil_moisture_as2']Accessing EO data using the earthaccess libraryimportgeopandasasgpdfromtqdmimporttqdmfrompathlibimportPathfromgeoprepare.eoaccessimporteoaccessdg=gpd.read_file(PATH_TO_SHAPEFILE,engine="pyogrio")# Convert to CRS 4326 if not alreadyifdg.crs!="EPSG:4326":dg=dg.to_crs("EPSG:4326")# Iterate over each row of the shapefileforindex,rowintqdm(dg.iterrows(),desc="Iterating over shapefile",total=len(dg)):# Get bbox from geometry of the rowbbox=row.geometry.boundsobj=eoaccess.NASAEarthAccess(dataset=["HLSL30","HLSS30"],bbox=bbox,temporal=(f"{row['year']}-01-01",f"{row['year']}-12-31"),output_dir=".",)obj.search_data()ifobj.results:obj.download_parallel()obj=eoaccess.EarthAccessProcessor(dataset=["HLSL30","HLSS30"],input_dir=".",shapefile=Path(PATH_TO_SHAPEFILE),)obj.mosaic()Upload package to pypiUpdate requirements.txtUpdate version="A.B.C" in setup.pyNavigate to the directory containingsetup.pyand run the following command:pipfreeze>requirements.txtpythonsetup.pysdisttwineuploaddist/geoprepare-A.B.C.tar.gzCreditsThis package was created withCookiecutterand thegiswqs/pypackageproject template.