Package: scalable-pypeline
PypeLineQuickstartAddpypelineas a dependency to your Python applicationInstall extras depending on what you are building:flask- Convenient interface for Flask applicationsweb- Some standard web server dependencies we likeworkers- InstallsCeleryandnetworkx, which are required if using pipelines.OverviewPypeLines is a fork of [Sermos] (https://gitlab.com/sermos/sermos). PypeLines diverges from Sermos as a SAAS platform and is intented as a suite for job management in conjuction with or indepent from a Flask Web App. Common job management workflow's include running pipelines, scheduled tasks, and other various types of jobs. Pypelines is designed to make these systems faster and more intuitive to create for Python developers.Under the hood we are simply extending various Celery capabilities like their existing complex workflows and make them suitable for large scale pipelines that can be run in production. To do this PypeLines uses a custom Celery configuration and a library known asCelery-Dyrygentto help orchestrate thousands of tasks at once.PypelineCelery ConfigurationPipelinesCronJobsAPIsUtilitiesYour ApplicationThis is where all of your code lives and only has a fewrequirements:It is a base application written in Python.Scheduled tasks and Pipeline nodes must be Python Methods that accept
at least one positional argument:eventAsermos.yamlfile, which is a configuration file for running scheduled tasks and pipelines.CeleryPypelines provides sensical default configurations for the use ofCelery. The default deployment uses RabbitMQ,
and is recommended. This library can be implemented in any other workflow
(e.g. Kafka) as desired.There are two core aspects of Celery that pypeline handles and differ from a
standard Celery deployment.ChainedTaskIncelery.pywhen imported it will configure Celery and also runGenerateCeleryTasks().generate(), which will use thesermos.yamlconfig
to turn customer methods into decorated Celery tasks.Part of this process includes addingChainedTaskas thebasefor all of
these dynamically generated tasks.ChainedTaskis a CeleryTaskthat injectstoolsandeventinto the
signature of all dynamically generated tasks.SermosSchedulerWe allow users to set new scheduled / recurring tasks on-the-fly. Celery's
defaultbeat_schedulerdoes not support this behavior and would require the
Beat process be killed/restarted upon every change. Instead, we set our
customsermos.celery_beat:SermosScheduleras thebeat_scheduler,
which takes care of watching the database for new/modified entries and reloads
dynamically.Workers / Tasks / Pipeline NodesPypeLine handles decorating the tasks, generating the correct Celery
chains, etc.Customer code has one requirement: write a python method that accepts one
positional argument:evente.g.def demo_pipeline_node_a(event):
    logger.info(f"RUNNING demo_pipeline_node_a: {event}")
    returnGeneratorsTODO: This needs to be updated both in code and documentation. Leaving here
because it's valuable to update in the future.A common task associated with processing batches of documents is generating
the list of files to process.pypeline.generatorscontains two helpful
classes to generate lists of files from S3 and from a local file system.S3KeyGeneratorwill produce a list of object keys in S3. Example:gen = S3KeyGenerator('access-key', 'secret-key')
files = gen.list_files(
    'bucket-name',
    'folder/path/',
    offset=0,
    limit=4,
    return_full_path=False
)LocalKeyGeneratorwill produce a list of file names on a local file system.
Example:gen = LocalKeyGenerator()
files = gen.list_files('/path/to/list/')TestingIf you are developing pypeline and want to test this package,
install the test dependencies:$ pip install -e .[test]Now, run the tests:$ tox