Package: django-cog
Django-CogAdjango-celery-beatextension to build pipelines of chronological stages and parallel tasks.AboutUsing the Djano admin, this library allows you to create pipelines of multi-staged tasks.  Each pipeline is launched at a specific time, utilizing thedjango-celery-beatCrontabScheduleobject to define the time of launch.  Once launched, the pipeline looks for the first stage(s) of parallel tasks.  Each task is submitted to a celery worker for completion.  Once all tasks of a stage complete, the stage is considered complete and any proceeding stages will launch (assuming all previous stages required are completed).PipelinesA pipeline is a collection of stages.  It has an optional launch schedule tied to aCronSchedulethat defines when the pipeline should be launched.  By default, a pipeline will not launch if it detects that a previous launch has not yet completed.You can also leave the launch schedule blank should you not want a particular pipeline to run on its own.  Instead, clicking theLaunch Nowbutton on that pipeline's Django admin page will begin its execution.Note:If a pipeline was previously scheduled using aCronScheduleobject but needs to be taken off the schedule, simply set theSchedulefield of the pipeline to the first option of the drop-down menu (---------).  This will remove the generatedPeriodicTaskwithin the database scheduler, preventing further auto-execution of that pipeline.StagesA stage is a collection of tasks that can all be ran independently (and in parallel) of each other.  It can be dependent on any number of previous stages to be complete before launching, but will be launched upon the completion of all prerequisite stages.If a stage has no prerequisite stages, it will be launched at the start of the Pipeline's launch.  You can have multiple stages run at the same time.Upon launching a stage, each task that belongs to it will be sent to thecelerybroker for execution.Tasks and CogsCogs:ACogis a registered python function that can be used in a task.  To register a function, use the@cogfunction decorator:fromdjango_cogimportcog@cogdefmy_task():# do something in a celery workerpassNOTE: These functionsmustbe imported in your application's__init__.pyfile.  Otherwise the auto-discovery process will not find them.Once Django starts, an auto-discovery process will find all functions with this decorator and createCogrecords for them in the database.  This allows them to be reference in the Django admin.Tasks:Once you have cogs registered, you can create a task.  Tasks are specific execution definitions for a cog, and are tied to a stage.  This allows you to run the same function through multiple stages, if needed.ParametersIf your function has parameters needed, you can set these in the Task creation.  See below for an example:fromdjango_cogimportcog@cogdefadd(a,b):# add the two numbers togetherreturna+bThen in the Task Django admin page, set these variables in theArguments as JSON:field:{"a":1,"b":2}InstallationIMPORTANT: It is required that the library is installed and migrations ran PRIOR to registering functions ascogs.
First install the library with:pipinstalldjango-cogThen add it to your Django application'sINSTALLED_APPSinside yoursettings.pyfile:INSTALLED_APPS=[...# Django-Cog:'django_cog.apps.DjangoCogConfig',# Required for nested inline child views in the Django Admin:'nested_inline',# Optional (recommended):'django_celery_beat',]Lastly, run migrations:pythonmanage.pymigratedjango_cogCog RegistrationOnce migrations complete, it is safe to register your functions using thecogdecorator:fromdjango_cogimportcog@cogdefmy_task():# do something in a celery workerpassManual RegistrationIf the library doesn't pick up your registered cogs and doesn't createCogrecords in the admin automatically, you can call theCogrecord creation function manually in the Django shell:fromdjango_cog.appsimportcreate_cog_recordscreate_cog_records()Note: This does require the functions to be registered cogs.  You can list the functionsdjango_coghas discovered with:fromdjango_cogimportcogprint(cog.all)This will output a dictionary where the key is the function name it discovered, and the value is the actual function itself.If your function is not listed here:this means the registration was not called.  This is likely due to the function not being imported in your applications__init__.py.  Double check to make sure you are importing the function somewhere that gets called on Django's startup (__init__.pyis the recommended place for this).Docker-ComposeBelow is a sample docker-compose.yml segment to add the required services for Celery workers, Celery-Beat, and Redis:version: '3'

# 3 name volumes are named here.
volumes:
    volume_postgresdata:        # Store the postgres database data. Only linked with postgres.
    volume_django_media:        # Store the Django media files. Volume is shared between djangoweb and nginx.
    volume_django_static:       # Store the Django static files. Volume is shared between djangoweb and nginx.

services:
    # Postgresql database settings.
    postgresdb:
        image: postgres:9.6-alpine
        environment:
            - POSTGRES_DB
            - POSTGRES_USER
            - POSTGRES_PASSWORD
        restart: unless-stopped
        volumes:
            - volume_postgresdata:/var/lib/postgresql/data
        ports:
            - "127.0.0.1:5432:5432"
        networks:
            - backend

    # Django settings.
    djangoweb:
        build:
            context: .
            args:
                - DJANGO_ENVIRONMENT=${DJANGO_ENVIRONMENT:-production}
        image: djangoapp:latest
        networks:
            - backend
            - celery
            - frontend
        volumes:
            - .:/app/
            - volume_django_static:/var/staticfiles
            - volume_django_media:/var/mediafiles
        ports: # IMPORTANT: Make sure to use 127.0.0.1 to keep it local. Otherwise, this will be broadcast to the web.
            - 127.0.0.1:8000:8000
        depends_on:
            - postgresdb
            - mailhog
        environment:
            - POSTGRES_DB
            - POSTGRES_USER
            - POSTGRES_PASSWORD
            - POSTGRES_HOST=postgresdb # Name of the postgresql service.
            - POSTGRES_PORT
            - DJANGO_SETTINGS_MODULE
            - FORCE_SCRIPT_NAME
            - DJANGO_ENVIRONMENT
            - SECRET_KEY
            - DJANGO_ALLOWED_HOSTS
        links:
            - "postgresdb"

    # add redis as a message broker
    redis:
        image: "redis:alpine"
        networks:
            - celery

    # celery worker process -- launches child celery processes equal to the number of available cores
    celery:
        build:
            context: .
            dockerfile: Dockerfile.celery
            args:
                - DJANGO_ENVIRONMENT=${DJANGO_ENVIRONMENT:-production}
        command: celery -A django_cog worker -l info
        image: djangoapp_celery:latest
        volumes:
            - .:/app/
        environment:
            - POSTGRES_DB
            - POSTGRES_USER
            - POSTGRES_PASSWORD
            - POSTGRES_HOST=postgresdb # Name of the postgresql service.
            - POSTGRES_PORT
            - FORCE_SCRIPT_NAME
            - DJANGO_ENVIRONMENT
            - DJANGO_SETTINGS_MODULE
            - SECRET_KEY
            - DJANGO_ALLOWED_HOSTS
        depends_on:
            - postgresdb
            - redis
        networks:
            - backend
            - celery
        links:
            - "postgresdb"

    # celery worker process -- launches child celery processes equal to the number of available cores
    celerybeat:
        build:
            context: .
            dockerfile: Dockerfile.celery
            args:
                - DJANGO_ENVIRONMENT=${DJANGO_ENVIRONMENT:-production}
        command: celery -A django_cog beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler
        image: djangoapp_celerybeat:latest
        volumes:
            - .:/app/
        environment:
            - POSTGRES_DB
            - POSTGRES_USER
            - POSTGRES_PASSWORD
            - POSTGRES_HOST=postgresdb # Name of the postgresql service.
            - POSTGRES_PORT
            - FORCE_SCRIPT_NAME
            - DJANGO_ENVIRONMENT
            - DJANGO_SETTINGS_MODULE
            - SECRET_KEY
            - DJANGO_ALLOWED_HOSTS
        depends_on:
            - celery
        networks:
            - backend
            - celery

networks:
    frontend:
        name: djangocog_frontend
    backend:
        name: djangocog_backend
    celery:
        name: djangocog_celeryAnd the matchingDockerfile.celery(of which theceleryandcelerybeatservices will build from):FROMpython:3.8ENVPYTHONUNBUFFERED1ARGDJANGO_ENVIRONMENT# Make the static/media folder.RUNmkdir/var/staticfilesRUNmkdir/var/mediafiles# Make a location for all of our stuff to go intoRUNmkdir/app# Set the working directory to this new locationWORKDIR/app# Add our Django codeADD./app/RUNpipinstall--upgradepip# Install requirements for DjangoRUNpipinstall-rrequirements/base.txtRUNpipinstall-rrequirements/${DJANGO_ENVIRONMENT}.txtRUNpipinstall-rrequirements/custom.txt# No need for an entry point as they are defined in the docker-compose.yml servicesQueuesYou can create specific queues within celery, and assign tasks to these queues.  Within theTaskDjango admin, you'll find aqueuefield that defaults tocelery.  This is the default queue that celery works on.  This queue option is created automatically and is used as the default when you run the django-cog migrations.Adding QueuesIf you want to have a separate collection of workers dedicated for certain tasks, create a newCeleryQueuerecord in the Django admin and set your tasks to this queue.Additionally, you'll need workers to work on this queue.  You can do this by adding a-Q queue_nameparameter to thecommandcall in thedocker-compose.ymlfile'sceleryservice:services:
    celery:
        command: celery -A django_cog worker -l info -Q queue_nameTask Optimization and WeightsIn order to achieve better runtimes, tasks are queued in order of greatest "weight" first.  This works best in scenarios where you have somennumber of tasks that have a longer run time, but have extra tasks whose summation runtimes do not exceed that of the longer tasks.  The longer tasks will be launched first, allowing the runtime of each stage to be in direct correlation of the longest running individual task, rather than an arbitrary coefficient of shorter tasks that happened to be queued up before it.The weight of a task is updated at the end of each of its Pipeline run completions.  Theweightfield of aTaskis literally the average number of total seconds it took to complete.  (NOTE: This field is limited to 11 decimal places with a precision of 5 points, which means the maximum runtime supported for a single task is roughly 11.5 days).The sample size taken defaults to the past 10 runtimes of that task, but this can be adjusted by adding the following to your Djangosettings.py:DJANGO_COG_TASK_WEIGHT_SAMPLE_SIZE=10If you do not want to have the weight auto-calculated, you can disable this calculation by adding the following to your Djangosettings.py:DJANGO_COG_AUTO_WEIGHT=FalseNote: Tasks will still be ordered by weight (descending) when queued, but they all default to a weight of1prior to any runs.  You can also manually adjust this weight through theTaskDjango admin, which enables you to determine the order of queuing yourself (which of course requires the disabling of auto-weight calculation, as described above).Error HandlingEach task can be assigned asCritical(defaultTrue).  This will cause the pipeline to halt should any critical task fail to execute.NOTE: Currently running tasks will be allowed to complete, but no new tasks of the same stage will launch, and no further stages of that pipeline will begin.This leaves the Pipeline in aFailedstate, so if the pipeline has been marked as "Prevent overlapping runs": the pipeline cannot be launched again until the incomplete pipeline run object has been deleted.  (See below about failed pipelines.)  Once the last task of a failed pipeline completes, the task will update the Pipeline'sCompleted Ontimestamp to reflect when that last task completed.Error HandlersEach task call is wrapped in a generictry/exceptclause.  Within theexceptclause, an error handler function is called.  By default, this is thedefaultCogErrorHandler, which records the error in theCogErrorstable in the Django admin.  This stores information on the error type, the task run that failed, the date/time of the error, and the traceback of the error.Custom Error HandlersEach task can be assigned an error handler (if none are assigned, thedefaultCogErrorHandleris used).  You can write your own error handler function and register it like so:fromdjango_cogimportcog_error_handler@cog_error_handlerdefmyErrorHandler(error):"""Do something with the error (Excemption type)"""print(error)Any custom error handler needs to take inat leastone positional parameter, that is: the Excemption error.OPTIONAL:You can also include a keyword argumenttask_runto have the failed task run object passed in:fromdjango_cogimportcog_error_handler@cog_error_handlerdefmyErrorHandler(error,task_run):"""Do something with the error (Excemption type)and know where it came from."""print("The following error came from the task:",task_run.task.name)print(error)Failed PipelinesShould a task markedcriticalfail, the task, its stage, and pipeline will fall into aFailedstate.  By default, new pipeline runs cannot be called if the last run failed.  This is done intentionally so the same error does not keep repeating itself.However, youcanoverride this safety feature by settingDJANGO_COG_OVERLAP_FAILED = Truein your Django settings.  Doing this will allow a pipeline to launch even if the last run of that pipeline failed.Canceling Pipeline RunsWhen a pipeline is currently running, you can visit thePipeline RunsDjango admin page to see which ones are running (displayed at the top of the list).  Clicking on the details of a running pipeline will show a redCancel Runbutton at the top.  Clicking this will send the pipeline into aCanceledstate.  Any currently running tasks will be allowed to complete.  As soon as the last one of the currently running tasks completes, it will update the stage'sCompleted Onfield to properly reflect the end execution timestamp of the last task.  The stage will then also fall into aCanceledstate, and no further stages or tasks will be launched.