Package: mlossp
Satellite Data PreprocessorA package built for preprocessing netCDF, netCDF4, GeoJSON, oratomicCSV files using spatial joins and concatenation to create time-series numpy and csv datasets.Keyword ArgumentsKwargDescriptionextra_configPath to the configuration file used for loading nc files. (Only provide this if you are loading netCDF or CSV files.record_outA function to generate a label from a file name.region_outA function to generate a label from a region file name.suffix_outA function that returns a string to be appended to the output file.Usage:Define a directory structure and create the required configuration files. (see Configuration Files)Create an instance of Preprocessor and provide the path to a config json file along with keyword arguments.Call Preprocessor.preprocess to preprocess data according to the provided config.To define configurations for different data sources, use Preprocessor.preprocess_multi. Config_list[i] and Kwargs_list[i] will apply to the ith data source.importmlosspasspconfig_path="config.json"extra_config_path="nc_config.json"f=lambda:"_1"preprocessor=sp.Preprocessor(config_path,extra_config=extra_config_path,suffix_out=f)preprocessor.preprocess()config_list=["config_1.json","../configs/config_2.json"]kwargs_list=[{"extra_config":extra_config_path,"out_out":f},{}]preprocessor.preprocess_multi(config_list,kwargs_list)Configuration FilesEvery category of data, i.e., data within a directory should have a json configuration file. The following keys should be defined within the configuration file:KeyDescriptionOptionalregions_dirThe path to the directory where the geographical data is storedNodata_dirThe path to the directory where the satellite data is storedNoout_dirThe path to the directory where the output files will be storedNocrsA geopandas supported coordinate reference systemYesregions_file_mapA mapping of region name to their file pathNoselected_regionsThe list of regions to extract from the satellite data. Must be in regions_file_map.Nojoin_onThe feature that duplicate data entries in a timestamp will be aggregated onNojoinsThe aggregate metrics that will be stored from join_on's aggregation.Nofile_extensionThe format of all data being layered. Can be one of *.pkl, *.json, *.geojson, *.nc4, or *.ncNocompress_toThe path to the directory where visualization files will be stored.YeschunkThis will aggregate everychunkfiles together. E.g. 365 chunked into 7 will result in 53 chunks.YesGiven the following directory structure:- Data
    - Visualizations
    - EO
        - data1.nc
        - data2.nc
    - EO_OUT
    - Countries
        - gadm41_LKA_1.json
        - gadm41_USA_1.json
- preprocesser.pyAnd region and satellite data with the following features:Region: NAME, NAME_1, NAME_2, POINT, ...
Satellite Data: longitude, latitude, precipitationCal, ...A sample configuration would be defined as follows:{"compress_to":"Data/Visualizations","crs":"EPSG:4326","regions_dir":"Data/Countries","data_dir":"Data/EO","out_dir":"Data/EO_OUT","regions_file_map":{"Sri Lanka":"gadm41_LKA_1.json","USA":"gadm41_USA_1.json"},"selected_regions":["Sri Lanka"],"join_on":"NAME_1","joins":{"precipitationCal":["mean","min","max"]},"file_extension":"*.nc","chunk":5}Preprocessing netCDF dataTo use this package with netCDF data, a separate configuration must be provided for each data source. A sample configuration is shown below.{"latVar":"latitude","lonVar":"longitude","is360":false,"extraVars":["NDVI"]}Configuration Keys:KeyDescriptionlatVarThe name of the latitude columnlonVarThe name of the longitude columnis360Set to true if the netCDF contains data within 0 - 360 degreesextraVarsDesired features to be retrieved from the netcdf file.Preprocessing CSV dataTo use this package to preprocess CSV data, a configuration file must be defined for each data source.
CSV data provided will be aggregated across the given time period; Information across higher temporal resolutions will belost.The number of csv files provided will be the final number of timestamps.
If you would like to atomicize your csv files, see [Formatting CSVs](#Formatting CSVs).{"latVar":"Lattitude","lonVar":"Longitude","is360":false,"filter":[["Disease Name","Dengue Fever","e"],["Location Name","SRILANKA","n"],["Location Name","Kalmune","n"]],"extraVars":["Cases","Location Name"]}Configuration Keys:KeyDescriptionlatVarThe name of the latitude columnlonVarThe name of the longitude columnis360Set to true if the longitude and latitude are measured in 360 degreesfilterA list of length 3 arrays of [Column Name, Target Value, Equals (e) or Not Equals (n)]. All rows whose column name has a value that is either equal or not equal to the target will be excluded.extraVarsFeatures to keep in the output.Formatting CSVsCSVs are inherently designed to support tabular, 2D data. Hence, storing higher-dimensional data such as tensors and ndarrays as CSVs results in inconsistent formats.We offermlossp.formattersas a solution to parse a (or a list of) 3D CSV files into a 3D numpy tensor, which is formatted then written into a .npy file and a .csv file that is supported byalign_csvandalign_npy.CitationsThis package incorporates modified code fromhttps://github.com/podaac/netcdf_to_geojson_vectors, which is licensed under Apache 2.0. You may obtain a copy of the license at:http://www.apache.org/licenses/LICENSE-2.0.