Package: cirrus-geo
CirrusCirrus is aSTAC-based geospatial processing pipeline platform,
implemented using a scalable architecture deployed on AWS. Cirrus provides the generic
infrastructure for processing, allowing a user to focus on implementing the specific
processing logic for their data.As input, Cirrus takes a STAC ItemCollection, with a process definition block.
That input is called a CirrusProcessPayload(CPP).An input is run through a workflow that generates one or more output STAC Items.
These output Items are added to the Cirrus static STAC catalog in S3,
and are also broadcast via an SNS topic. Subscrptions to that topic can
triggering additional workflows or external processes, such as indexing into a
STAC API catalog (e.g.,stac-server).Cirrus workflows range from the simple publishing of unmodified input items to
the complex transformation of input Items and generation of wholly-new output
Items. The current state of CPP processing is tracked in a state database
to prevent duplicate processing and allow for a user to follow the state of any
input through the pipeline.As shown in this high-level overview of Cirrus, users input data to Cirrus
through the use offeeders. Feeders are simply programs that get/generate
some type of STAC metadata, combine it with processing parameters, and pass it
into Cirrus as a CPP.Because Cirrus output is published via SNS, a Feeder can be configured to
subscribe to that SNS and thus workflows can be chained, such that the output
of one workflow becomes the input to another workflow, createing multiple levels
of products all with published STAC metadata and clear links showing data
provenance.Cirrus DevelopmentIf developing new code for cirrus-geo, checkout theContributing Guide.Cirrus QuickstartA Cirrus project is managed via thecirruscli tool.
Here's everything required to create, modify, and deploy a new project:# Make a new directory for a project
❯ mkdir cirrus-project; cd cirrus-project

# Create a python virtual environment for isolation
❯ python -m venv .venv

# Activate our venv
❯ . .venv/bin/activate

# Install cirrus-geo
❯ pip install cirrus-geo
...

# Now we should have cirrus on our path
❯ cirrus
Usage: cirrus [OPTIONS] COMMAND [ARGS]...

  cli for cirrus, a severless STAC-based processing pipeline

Options:
  --cirrus-dir DIRECTORY
  -v, --verbose           Increase logging level. Can be specified multiple
                          times.  [x>=0]
  --help                  Show this message and exit.

Commands:
  build             Build the cirrus configuration into a serverless.yml.
  clean             Remove all files from the cirrus build directory.
  create            Create a new component in the project.
  init              Initialize a cirrus project in DIRECTORY.
  serverless (sls)  Run serverless within the cirrus build directory.
  show              Multifunction command to list/show components,
                    component...

# Fantastic!
# We can init our new project and see what all was created
❯ cirrus init
Succesfully initialized project in '/Users/jkeifer/cirrus-project'.

❯ ls
.venv/	cirrus.yml  /cloudformation  feeders/  functions/  package.json  tasks/  workflows/

# The cirrus.yml is almost good to go for a minimal install,
# but it does require a few parameters either set in the
# config or as environment variables:
#
#   custom:
#     batch:
#       SecurityGroupIds:
#         - ${env:SECURITY_GROUP_1}
#       Subnets:
#         - ${env:SUBNET_1}
#         - ${env:SUBNET_2}
#         - ${env:SUBNET_3}
#         - ${env:SUBNET_4}
#
# Use your favorite editor to set these values approriately
# based on your existing AWS resources.

# As we do have node.js dependencies from serverless,
# let's install those with the generated configuration
❯ npm install
...

# We can see all the built in feeders, tasks, and workflows (among others)
❯ cirrus show feeders
feed-rerun (built-in): Rerun items in the database

❯ cirrus show tasks
post-batch (built-in, lambda): Post process batch job by copying input from S3
pre-batch (built-in, lambda): Pre process batch job by copying input to S3
publish (built-in, lambda): Publish resulting STAC Collections and Items to catalog, and optionally SNS

❯ cirrus show workflows
publish-only (built-in): Simple example that just published input Collections and Items

# To create a new task, for example, we can do this
❯ cirrus create task -t lambda a_task "A task that doesn't do much yet"
task a_task created

❯ cirrus show tasks
post-batch (built-in, lambda): Post process batch job by copying input from S3
pre-batch (built-in, lambda): Pre process batch job by copying input to S3
publish (built-in, lambda): Publish resulting STAC Collections and Items to catalog, and optionally SNS
a_task (lambda): A task that doesn't do much yet

# We can see that created a task and its
# associated config inside the tasks directory
❯ tree tasks
tasks
└── a_task
    ├── README.md
    ├── definition.yml
    └── lambda_function.py

# To build our configuration in to something
# compatible with serverless, we use the build command
❯ cirrus build

# The output of build is in the .cirrus directory
❯ ls .cirrus
lambdas/  serverless.yml

# To deploy with serverless, we can simply do the following
# (optionally set the stage with `--stage <stage_name>`)
❯ cirrus serverless deployCirrus Project StructureA Cirrus project, most basically, is a directory containing a cirrus.yml
configuration file. However, several subfolders are used to organize
additional object definitions for custom implementations.FolderPurposecloudformationRaw cloudformation templates to include in the projectfeedersFeeder Lambda functions used to add data to CirrusfunctionsMisc Lambda functions required by a projecttasksTask Lambda function used within workflowsworkflowsAWS Step Function definitions describing data processing workflowsCirrus RepositoriesCirrus is divided up into several repositories, all under thecirrus-geoorganization on GitHub,
with this repository (cirrus-geo) the main one of interest to users.RepositoryPurposecirrus-geoMain Cirrus repo implementing thecirruscli tool for managing Cirrus projects. Also provides the base set of lambda functions and workflows.cirrus-libA Python library of convenience functions to interact with Cirrus. Lambda functions are kept lightweightcirrus-task-imagesDockerfiles and code for publishing Cirrus Docker images to Docker Hub that are used in Cirrus Batch tasksThecirruscli utilitiy is what is used to create, manage, and deploy
Cirrus projects, and is pip-installable. The pip-installable python
librarycirrus-libis used from all Cirrus Lambdas and tasks and is
available to developers for writing their own tasks.DocumentationDocumentation for deploying, using, and customizing Cirrus is contained
within thedocsdirectory:Learn how toget startedUnderstand thearchitectureof Cirrus and key conceptsUseCirrus to process input data and publish resulting STAC ItemsCirrus features severalcomponent typesthat each represent a specific role within the Cirrus architectureAboutCirrus is an Open-Source pipeline for processing geospatial data in AWS.
Cirrus was developed byElement 84originally
under aNASA ACCESS projectcalledCommunity Tools for Analysis of NASA Earth Observation System Data in the Cloud.