Package: datacube-alchemist
Datacube Alchemist - ODC Dataset to Dataset ConverterPURPOSEDatacube Alchemist is a command line application for performing Dataset to Dataset transformations in the context
of an Open Data Cube system.It uses a configuration file which specifies an inputProductorProducts, aTransformationto perform, and
output parameters and destination.FeaturesWrites output to Cloud Optimised GeoTIFFsEasily runs within a Docker ContainerParallelism using AWS SQS queues and KubernetesWrite output data to S3 or a file systemGenerateseo3format dataset metadata, along with processing informationGenerates STAC 1.0.0.beta2 dataset metadataConfigurable thumbnail generationPass any command line options as Environment VariablesINSTALLATIONYou can build the docker image locally with Docker or Docker Compose. The commands aredocker build --tag opendatacube/datacube-alchemist .ordocker-compose build.There's a Python setup file, so you can dopip3 install .in the root folder. You will
need to ensure that the Open Data Cube and all its dependencies happily install though.USAGEDevelopment environmentTo run some example processes you can use the Docker Compose file to create a local workspace.
To start the workspace and run an example, you can do the following:Export the environment variablesODC_ACCESS_KEYandODC_SECRET_KEYwith valid AWS credentialsRunmake upordocker-compose upto start the postgres and datacube-alchemist Docker containersmake initdbto initialise the ODC database (or see the Makefile for the specific command)make metadatawill add the metadata that the Landsat example product needsmake productwill add the Landsat product definitionsmake indexwill index a range of Landsat scenes to test processing withmake wofs-oneormake fc-onewill process a single Fractional Cover or Water
Observations from Space scene and output the results to the ./examples folder in this project directoryCommandsNote that the--config-filecan be a local path or a URI.datacube-alchemist run-oneNote that--dryrunis optional, and will run a 1/10 scale load and will not
write output to the final destination.datacube-alchemistrun-one\--config-file./examples/c3_config_wo.yaml\--uuid7b9553d4-3367-43fe-8e6f-b45999c5ada6\--dryrun\datacube-alchemist run-manyNote that the final argument is a datacubeexpression, seeDatacube Search documentation.datacube-alchemistrun-many\--config-file./examples/c3_config_wo.yaml\--limit=2\--dryrun\timein2020-01datacube-alchemist run-from-queueNotes on queues. To run jobs from an SQS queue, good practice is to create a deadletter queue
as well as a main queue. Jobs (messages) get picked up off the main queue, and if they're successful,
then they're deleted. If they aren't successful, they're not deleted, and they go back on the
main queue after a defined amount of time. If this happens more than the defined number of times
then the message is moved to the deadletter queue. In this way, you can track work completion.datacube-alchemistrun-from-queue\--config-file./examples/c3_config_wo.yaml\--queueexample-queue-name\--limit=1\--queue-timeout=600\--dryrundatacube-alchemist add-to-queueThe--limitis the total number of datasets to limit to, whereas the--product-limitis
the number of datasets per product, in the case that you have multiple input products.datacube-alchemistadd-to-queue\--config-file./examples/c3_config_wo.yaml\--queueexample-queue-name\--limit=300\--product-limit=100datacube-alchemist redrive-to-queueThis will get items from a deadletter queue and push them to an
alive queue. Be careful, because it doesn't know what queue is what.
You need to know that!datacube-alchemistredrive-to-queue\--queueexample-from-queue\--to-queueexample-to-queueLicenseApache License 2.0Copyright© 2021, Open Data Cube Community