Package: satip
SatipSatip is a library forsatelliteimageprocessing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT dataInstallationTo install thesatiplibrary please run:pipinstallsatipOr if you're working in the development environment you can run the following from the directory root:pipinstall-e.CondaOr, if you want to usecondafrom a cloned Satip repository:condaenvcreate-fenvironment.yml
condaactivatesatip
pipinstall-e.If you plan to work on the development of Satip then also consider installing these development tools:condainstallpytestflake8jedimypyblackpre-commit
pre-commitinstallDevelopment EnvironmentIn order to contribute:it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.OperationGetting your own API keyIn order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:Go tohttps://eoportal.eumetsat.intand register an account.You can log in and go tohttps://data.eumetsat.int/to check available data services. From there go to your profile and choose the option "API key" or go tohttps://api.eumetsat.int/api-key/directly.Please make sure that you added the key and secret to your user's environment variables.Downloading EUMETSAT DataThe following command will download the last 2 hours of RSS imagery into NetCDF files at the specified locationpythonsatip/app.py--api-key=<EUMETSATAPIKey>--api-secret=<EUMETSATAPISecret>--save-dir="/path/to/saving/files/"--history="2 hours"To download more historical data, the command below will download the native files, compress with bz2, and save into a subdirectory.pythonsatip/get_raw_eumetsat_data.py--user-key=<EUMETSATAPIKey>--user-secret=<EUMETSATAPISecret>Converting Native files to Zarrscripts/convert_native_to_zarr.pyconverts EUMETSAT.natfiles to Zarr datasets, using very mild lossyJPEG-XLcompression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalentbz2compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range[0, 1]. NaNs are encoded as the value0.025. All "real" values are in the range[0.075, 1]. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (seethis comment for more details). Usesatip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNsto decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range[0, 1].Running in ProductionThe live service usesapp.pyas the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.--api-keyorAPI_KEYis the EUMETSAT API key--api-secretorAPI_SECRETis the EUMETSAT API secret--save-dirorSAVE_DIRis the top level directory to save the output files, alatestsubfolder will be added to that directory to contain the latest data--historyorHISTORYis the amount of history timesteps to use in thelatest.zarrfiles--db-urlorDB_URLis the URL to the database to save to when a run has finished--use-rescalerorUSE_RESCALERtells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.TestingTo run tests, simply runpytest .from the root of the repository. To generate the test plots, runpython scripts/generate_test_plots.py.Environmental VariablesSome tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:EUMETSAT_USER_KEY: the EUMETSAT API keyEUMETSAT_USER_SECRET: the EUMETSAT API secretThese can be added using theexportcommand in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).Contributors ✨Thanks goes to these wonderful people (emoji key):Jacob Bieker💻Jack Kelly💻Ayrton Bourn💻Laurence Watson💻Notger Heinz📖Peter Dudfield📖This project follows theall-contributorsspecification. Contributions of any kind welcome!