Package: sdap-os
SDAP experiment with object storage and other technologies (ray.io, xarray...)FeaturesApplies simple algorithms on earth observation datasetsRead Cloud Optimized GeoTiff on S3 bucketManage S3 bucket with public access or with access keysManage datasets in any CRS (WGS84, UTM, ...)Manage multi-band datasetsAlgorithms managed:Spatial Mean: on a x,y,t range averages on x,y dimensionsEVI: linear combinations of multi-band spectraPrerequisitespython 3.9datasets on s3Configure application and testThe collection configuration should follow the template found onsrc/sdap/test/data_access/collection-config.yamlIf some of the data you need to access on S3 requires authentication, use a secrets files as the one provided in examplesrc/sdap/test/data_access/secrets.yaml.You have one secret configuration (e.g. AWS credential) per collection.Install and run tests:pip install -e '.[dev]'
python setup.py test &> out.log &RunAs a command line (suitable for serverless deployments)Use command linesdap-cmd --helpFor example withEVIoperator:sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --collection hls \
         --x-range -71.24 -71.23 \
         --y-range 42.40 42.42 \
         --time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
         --operator-name EVI \
         --operator-args '[[0,0,-2.5,2.5,0,0,0],[0,0,2.4,1,0,0,1]]' \
         --plotWithSpatialMeanoperator:sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --secrets my_secrets.yaml \
         --collection hls \
         --bbox -71.24 42.40 -71.23 42.42 \
         --time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
         --operator-name SpatialMean

    sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --collection maiac \
         --bbox -71.25 42.39 -71.23 42.42 \
         --time-range 2019-01-01T00:00:00.000000+00:00 2019-02-01T00:00:00.000000+00:00 \
         --operator-name SpatialMeanAs a serviceStarts a web server with an API:sdap-serv  --conf src/sdap/test/data_access/collection-config.yaml --secrets my_secrets.yamlTest the service, you can use the jupyter notebook(s) provided onsrc/sdap/notebookYou can run it locally by running:pip install -e '.[client]'
jupyter-labThen select the notebook in the jupyuter web console and execute it, step by step.Resource allocationThe resources needed to run the application can be estimated as follows:maximum size of 1 tile for each worker = T = 64 * x * y * t * obs dim, for example 643660366016 = 5Gbmaximum size of the user result returned to the user = R = 64xytobs dimsnumber of parallels workers: nTotal RAM = T*n + RNumber of CPU = nFor developerspip install -e '.[dev]'Do you work, aad unit testsValidatepython setup.py testPackageUpgrade version in `src/sdap/VERSION.txt`

python setup.py sdist bdist_wheelCreate and publish the docker imagedocker build . -t tloubrieu/sdap-os:0.1.1
docker push tloubrieu/sdap-os:0.1.1Create the kubernetes namespace:kubectl create namespace sdap-osDeploy the ray.io operator:(See instructions onhttps://docs.ray.io/en/latest/cluster/kubernetes.html#ray-helm)git clone https://github.com/ray-project/ray.git
cd ray/deploy/charts 
helm -n ray install sdap-cluster --create-namespace ./rayDeploy the helm charthelm install sdap-os ./helm/sdap-os --dependency-update -n sdap-osStop the service:helm delete sdap-os -n sdap-osOn a local deployment with single node, the node might be tainted to prevent pods from being deployed on it. In this case you can run something like:kubectl taint node docker-desktop node.kubernetes.io/disk-pressure:NoSchedule-(seehttps://stackoverflow.com/questions/62991596/1-nodes-had-taints-that-the-pod-didnt-tolerate-in-kubernetes-clusterfor more details)Use with local minioSet minio server (not used in the development yet)Standalone minio:Server install:brew install minio/stable/minioStart the serverexport MINIO_ROOT_USER=myminio
export MINIO_ROOT_PASSWORD=minio123AB

minio server /Users/loubrieu/Documents/sdap/minio --address localhost:9000 --console-address localhost:19000Clientbrew install minio/stable/mc
mc --helpCreate server alias:mc alias set minio http://localhost:9000 myminio minio123ABCreate a bucket:mc mb hlsCopy test datasetsCreate a bucket:mc mb hlsCopy some files:mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019263_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019260_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019253_cog.tif minio/hls