Package: cirrus-lib
Cirrus LibraryA common set of functions used by the Cirrus geospatial pipeline. This library are convenience functions for interacting with the various Cirrus AWS resources (queues, Lambdas, Batch, DynamoDB, etc.) and should be preferred over writing separate code where possible.InstallationCirrus-lib requiresboto3-utils, a library of convenience functions built uponboto3. Cirrus-lib is published to PyPi which is the preferred method of installing:$pipinstallcirrus-lib# install specific version$pipinstallcirrus-lib==0.1.0# in a requirements.txt file - the tilda (~) will install the highest compatible versioncirrus-lib~=0.1.0ModulesModuleDescriptionprocess_payloadProcessPayload and ProcessPayloads classes representing a Cirrus input payloads and a set of ProcessPayloadserrorsCustom Error classesstatedbThe StateDB class for interacting with the Cirrus state database (DynamoDB)transferConvenience functions for uploading and download assets in STAC ItemsutilsMiscellaneous utility functionsprocess payloaderrorsTheerrorsmodule contains a single class:InvalidInput, which can be thrown from a task if there is a problem with the input to the task.  The Item will be marked as INVALID in the state database rather than FAILED to avoid running it again.statedbTheStateDBclass from thestatedbmodule is used to interact with the Cirrus state database (DynamoDB): creating new items in the database, querying, and updating statuses. Each item in the state database represents one input payload into Cirrus. TheStateDBclass is abstracted from what is actually stored in the DynamoDB, which is optimized for queries. The actual entry in the database can be retrieved with theStateDB.get_dbitem()andStateDB.get_dbitems()functions which are used internally, but clients should generally opt to useStateDB.get_items(), which translates the dbitem into:Field NameTypeDescriptionpayload_idstringThe ID of the Cirrus input payloadcollectionsstringA list of all the collections used by the input itemsworkflowstringThe name of the workflow used to process the input dataitemsstringA list of all the input Item IDspayloadstringThe s3 URL of the original input payload JSONstatestringOne of: 'PROCESSING', 'COMPLETED', 'FAILED', 'INVALID'createdstringThe datetime that this item was created in the databaseupdatedstringThe datetime stamp that this item's state was last updatedexecutions[string]A list of http URLs to the State Machine executions, in chronological orderoutputs[string]Ifstateis COMPLETED, a list of s3 (if private payload) or http (if public) URLs of all STAC Items generated by this process.last_errorstringThis is the error message from the last FAILED or INVALID execution, if any. It is not cleared if state is PROCESSING or COMPLETEDpayload_id: The payload ID is formed by a combination of all the input collections, output collections, and the name of the workflow.transferThetransfermodule contains convenience functions for transferring data.FunctionDescriptionget_s3_sessionGet a boto3-utils s3 object for transfer (used by the other functions) to/from s3download_from_httpDownload a file from an HTTP urldownload_item_assetsFor a given STAC Item download 1 or more assets to a local pathupload_item_assetsFor a given STAC Item upload 1 or more assets to s3utilsThis module contains miscellaneous utility functions.FunctionDescriptionsubmit_batch_jobStart a Cirrus Batch job with a given payloadAboutCirrus is an open-source pipeline for processing geospatial data in AWS. Cirrus was developed byElement 84originally under a [NASA Access project].