Package: servicer
ServicerServicer is a configuration-based CI automation framework that runs inside popular CI platforms, such as Bitbucket Pipelines, Jenkins, and CircleCI.
Servicer may be the right tool to use if you find yourself in any these situations:Writing a lot of Bash to augment your CI/CD jobs.Have dependent services building from the same repository (common in mono-repos).Want to speed up your CI job times by building services in parallel, or avoiding building services that have not changed.Core Features:Run and debug your CI jobs locally (very helpful for isolating or testing build issues before you deploy).CI provider agnostic configuration (make your build process portable between multiple CI providers).Map your branches and tags to independent service environments, allowing you to use an identical process to deploy production, testing, and development environments.Interpolate dynamic values that change from environment to environment, allowing you to re-use the same configuration for your environments.Configure intelligent Change Detection, to only build the services and dependencies you need for the changes made (great for mono-repos!).Servicer executes a set of service-steps you define in.servicer/services.yaml. A servicer CI job will roughly do the following:Normalize the current CI environment (by standardizing environment variables).Determine the current Service Environment, if it exists.Construct the complete servicer configuration (interpolating dynamic value tokens).Determine which services can be ignored by Change Detection.Build a dependency graph of remaining service-steps.Execute each service-step in the order determined by the dependency graph, if it is appropriate for the current Service Environment.Contributing
Servicer is an open-source project. Feel free to fork and open a PR at any time. Each PR should have an associated Issue linked for reference.Separation of ConcernsServicer is separated into three main components: adapters, services, and the project environment.Adaptersare interfaces that allow Servicer to use or interact with external systems, like Google Cloud Platform, GitHub, Jenkins, Docker, etc. Three kinds of adapters exist: CI adapters (for CI providers), auth adapters (to manage credentials), and service adapters (for everything else). These are written as reusable python modules, and are located inservicer/builtin.Configurationis the primary building block of your project. Services are defined in the.servicer/services.yamlfile in your project, and these definitions include the configuration and commands needed to build, test and deploy each service. Additionally, dependencies may be defined to automate the order service-steps are executed in.Yourproject environmentshould contain environment variables for holding credentials and other secrets that should not be committed to source control. These can be configured on your CI tool, or, if you are running Servicer locally, from a.servicer/*.env.yamlfile.Using ServicerFor the simplest use case, use the command below. With it, Servicer will execute every step for every service you've defined in.servicer/services.yaml.servicerIf you want Servicer to execute all steps for one or more specific services, like my_docker_image and my_python_package, specify those services in the command.servicer --service=my_docker_image,my_python_packageSteps, like build and unit_test, can also be specified.servicer --step=build,unit_testWhen combined, you can execute a single service-step:servicer --service=my_docker_image --step=buildThese commands can be run locally or in the context of a CI job.By default, all service-step dependencies will also be executed. If you will like to disable this behavior, use the--ignore_dependenciesflag.If you would like to execute a test run of servicer without actually executing any service-steps, you can add the--dryflag.To set the desired logging level (debug, info, warn, error), use the--log_levelflag.For a complete list of flags and options that can be provided to theservicercommand, please seeservicer --help.ConfigurationBy default, Servicer will look for.servicer/services.yamlat the root of your project.servicer/builtin/defaults.yaml(https://github.com/wmgroot/servicer/blob/master/servicer/builtin/defaults.yaml) contains default values and descriptions of the values you can provide in.servicer/services.yaml.Within this section of the README, consider each of these subsections to compose theservices.yamlon one project.EnvironmentTheenvironmentsection ofservices.yamlwill let you define environment variables that you're ok with existing in source control. Additionally, you can control the branch -> service_environment settings, allowing you to change the defaults. Consider the example below.environment:
  # In this example, SERVICE_ENVIRONMENT will be `production` if the branch is `master`.
  # SERVICE_ENVIRONMENT will be `develop` if the branch is `develop`.
  # Otherwise, if the branch matches `env-*`, SERVICE_ENVIRONMENT will be the sanitized branch name.
  # If the CI provider supports tag based builds, servicer will also set the SERVICE_ENVIRONMENT if a version tag is found.
  mappings:
    - tag: '*.*.*'             # match a version tag -> 1.2.3
    - branch: master
      environment: production  # master -> production
    - branch: develop          # develop -> develop
    - branch: env-*            # env-my-branch -> env-my-branch

  # Define environment variables here.
  variables:
    FOO: BAR
    PROJECT_NAME: ${PRE_DEFINED_ENV_VAR}-plus-some
...Servicer requires several environment variables to be configured which control the parameters of each service. Some of these environment variables will be automatically pulled from your CI environment and standardized by the CI Adapter it is using. Check the relevant adapter file inservicer/builtin/ci_adapters. The remaining variables may be specific to each service adapter, and will need to be configured in your CI environment, or defined in theenvironmentsection ofservices.yaml, or be provided locally in your.servicer/.env.yamlfile.CI AdaptersIn order to standardize the environment Servicer runs in, a CI Adapter is used. Each CIAdapter is a python class that populates an instance variable calledenv_map, which contains key value pairs mapping environment variable names to their standardized Servicer equivalents.Servicer provides default CI Adapters for your convenience. Please explore theservicer/builtin/ci_adaptersdirectory to see what's available for each CI provider. In order to customize servicer to your needs, you can create your own CI Adapters in your project. By default, servicer will prefer CI Adapters found in the.servicer/ci_adaptersdirectory of your project. See the Advanced Servicer Use Cases for more information on adapter overrides.StepsSteps are sets of actions that share some configuration between services. For example, the 'deploy' step may only be executed when a Service Environment is present. Steps are declared project-wide inservices.yaml. In the example below, several steps are defined....
steps:
  - name: build
  - name: test
  - name: deploy
    config:
      service_environment: '*'              # will only be executed within a service environment (based on environment mappings)
  - name: cleanup
    config:
      auxiliary: true                       # can only be explicitly executed (using the --step argument)
...Steps are available for any service to implement. Given the above definition, during the first step,build, Servicer will look at every service and include itsbuildstep as a service-step in the complete dependency graph, if it exists. These service-steps will later be executed in the correct order.Steps can be configured to be disregarded unless they are being run within an environment that matches one of the mappings defined inenvironmentabove. Given the environment defined above, anydeployservice-steps will be disregarded unless Servicer is running in a matching Service Environment (any Service Environment will match the'*'). This syntax accepts glob and regex notations, and multiple matchers can be provided as a list. If there are multiple matchers, any match will allow execution of that particular service-step.ServicesServicesare abstract containers for pieces of your project. They can be a set of commands, or utilize a python module that wraps an integration with a 3rd party tool. Builtin service adapters automatically available through servicer are defined inservicer/builtin/service_adapters. You can also add your own service adapter within your project by adding it to your.servicer/service_adaptersdirectory. However, you may decide to use Servicer as a command orchestrator, without leveraging any service adapters, and that is perfectly fine.The service below builds a Docker image, and pushes/deploys it to the Google Container Registry....
services:
  my_docker_image:                     # User chosen name to uniquely identify this service
    providers:
      - gcloud                         # Initializes the given provider entry before running steps, useful for re-usable Auth
    service_type: gcloud/docker_image  # Optional, maps to the corresponding service adapter defined in servicer/builtin/service_adapters/
    steps:                             # Steps defined here must match a step defined in the `steps` list explained above
      build:
        config:
          # All config defined here is passed to the __init__ function of the corresponding service adapter.
          steps:
            - type: build
              args:
                image: demo-image
                dockerfile: Dockerfile
      deploy:
        commands:   # Commands are executed before the chosen service adapter is executed
          - docker login -u _json_key -p "$(cat /path/to/keyfile.json)" https://us.gcr.io/my-gcp-project
        config:
          # All config defined here is passed to the __init__ function of the corresponding service adapter.
          registry_path: my-gcp-project/this-project
          steps:
            - type: push
              args:
                image: demo-image
                tags:
                  - latest
...Theproviderskey is an optional key that allows you to specify one or more provider dependencies for the service. These entries should match the keys listed in theproviderssection of yourservices.yaml. For each entry here, servicer will ensure that the correct provider has been initialized before the steps for the service are executed.Here is a service that builds a python package, and deploys it to Artifactory....
  my_python_package:
    service_type: package/pypi

    steps:
      build:
        commands:
          - rm -rf python_example/dist
          - python setup.py sdist
        config:
          package_info:
            package_file_path: setup.py
          # service adapter tasks map to functions available within the service adapter
          tasks:
            - type: read_package_info
      deploy:
        depends_on:
          - pypi-credentials:credentials

        config:
          package_info:
            package_file_path: setup.py
          tasks:
            - type: upload
              args:
                server: artifactory
                path: dist/*.tar.gz
...Usingdepends_ongives you control over how Servicer's dependency graph is built. In the above example, Servicer will ensure that themy_python_package:deployservice-step is executed after thepypi-credentials:credentialsservice-step. Themy_python_packge:deployservice-step also has an implicit dependency on themy_python_packge:buildservice-step, because the two service-steps belong to the same service, and thebuildstep is defined before thedeploystep in the steps list above.A wildcard may also be provided in the place of a service name. For example,*:testmeans that the given service or service-step depends on the completion of all othertestservice-steps.CommandsAny service_type is able to execute arbitrary commands via the shell by using thecommandskey. You can avoid providing aservice_typeif all you want is a service wrapper for a set of commands. Thecommandskey will execute commands before a service adapter config, while thepost_commandskey will execute commands after.Thecommandslist optionally accepts a raw string, or map containing a context providing more customization.services:
  my-service:
    steps:
      build:
        commands:
          - docker build -f Dockerfile -t my-image .
      test:
        commands:
          - context:
              # interpolate all commands in this context using the given template
              template: '/bin/sh -c "%s"'
              # run all commands in this context in the provided docker context
              docker:
                image: my-image
                name: # defaults to a random string
                command: /bin/sh -c 'while sleep 5; do :; done'   # command to start the container with, sleep forever
                options:
                  env:
                    - MY_ENV_VAR
                  volume:
                    - $HOME/.ssh:/root/.ssh/
            commands:
              - echo 'Hi I'm running in a container!'
              - echo 'Me too!'

# final commands:
# docker run -d --env=MY_ENV_VAR --volume=$HOME/.ssh:/root/.ssh --name=H6GHGiXYjMwbAWEw my-image /bin/sh -c 'while sleep 5; do :; done'
# docker exec H6GHGiXYjMwbAWEw /bin/sh -c "echo 'Hi I'm running in a container!'"
# docker exec H6GHGiXYjMwbAWEw /bin/sh -c "echo 'Me too!'"
# docker stop H6GHGiXYjMwbAWEw
# docker rm H6GHGiXYjMwbAWEwConfig File ManagementUltimately, Servicer runs with one compiled configuration file (this can be printed by runningservicer --show_config). The heart of this isservices.yaml. For convenience,extendsandincludesare two tools that can be used to merge other config files intoservices.yaml.Theextendskey allows you to inherit from another YAML file. The structure of each file will be deep-merged, with values in the extending file overwriting values in the extended file. Lists are not merged, and will be completely overwritten.In the example below, a file calledbase.yamlis essentially providing the top half of theservices.yamlfile.# base.yaml

ci:
  providers:
    - bitbucket
  image: python:3.6.4

providers:
  aws:
    libraries:
      - awscli==1.15.49
      - boto3==1.7.48
  gcloud:
    auth_script: auth/gcloud.sh
    libraries:
      - google-api-python-client==1.6.5# services.yaml

extends: base.yaml

services:
  postgres:
    provider: aws
    service_type: rds_instance

  api-django:
    docker: true
    depends_on:
      - postgres
    provider: gcloud
    service_type: kube_clusterThe includes key can also inherit from a different YAML file. However, unlike extends, values from the included file will overwrite values in the including file if there are key conflicts.In the example below, kube-service.yaml includes configuration values for a service, and those values are being inherited by the api-django service.# kube-service.yaml

docker: true
depends_on:
  - postgres
provider: gcloud
service_type: kube_cluster# services.yaml

extends: base.yaml

services:
  api-django:
    includes: kube-service.yamlAn optionalparamskey may be used for interpolation, allowing for re-usable, generic config files. The example below is equivalent to the example above.# kube-service.yaml

docker: true
depends_on:
  - ${db}
provider: ${provider}
service_type: kube_cluster# services.yaml

extends: base.yaml

services:
  api-django:
    includes:
      path: kube-service.yaml
      params:
        db: postres
        provider: gcloudVariable InterpolationAll configuration files support variable interpolation using the${MY_VAR}format.
To pull from an environment variable, use${MY_ENV_VAR}.
To fall back to a default value if the chosen environment variable does not exist, use${MY_ENV_VAR:"my-default-value"}Testing the JobTo make it easy to test deployments in your local environment, servicer will automatically read from an.env.yamlfile in your servicer configuration directory, if it exists. It is recommended to add this file to your.gitignore, as it is only intended to be used for local development and manual environment creation.Example:# .servicer/.env.yaml
AWS_DEFAULT_REGION: us-east-2
BRANCH: env-test
BUILD_NUMBER: '0001'
DATABASE_NAME: my_db
DATABASE_USERNAME: postgres
DATABASE_PASSWORD: xxxxxxxx
GCLOUD_KEY_FILE_PATH: /Users/me/my_gcloud_credentials.json
GCLOUD_REGION: us-central1
GCLOUD_ZONE: us-central1-f
PROJECT_NAME: my-project
SERVICE_ENVIRONMENT: test.env.yamlcan also be defined at~/.servicer/.env.yamlto support multi-project defaults. Values in the project level.env.yamlwill take precedence over values in your user folder.Git IntegrationServicer provides a few handy Git integrations to optimize jobs. By default, git integration is disabled. To enable it, setgit: enabled: true.Change DetectionServicer can use agit diffto determine which services in your project are different between the current commit and either the commit associated with its last successful job, or one passed to Servicer by an environment variable. This second commit is Servicer'sreference point, and the list presented bygit diffprovides Servicer its change detection.By default, Servicer will maintain its own set of git tags that identify commits it should use as reference points. The format of these tags isservicer-<BRANCH>-<BUILD_DATE>-<BUILD_NUMBER>. Servicer relies upon the CI framework it is working on (like Jenkins) to provide theBRANCHandBUILD_NUMBERvariables. Alternatively, theGIT_DIFF_REFenvironment variable can be set manually.Servicer will use the first reference point it can find, and attempts to find it in this order:TheGIT_DIFF_REFenvironment variable, if it’s set and is a valid commit.The latest servicer git tag matching the current branch, ifgit: diff-tagging-enabled: true.The latest servicer git tag matching any branch, ifgit: diff-defaults-to-latest-tag: true.Servicer will also attempt to remove any stale tags automatically. Tags present only on deleted branches, or tags that are no longer the latest tag for a branch will be removed.Skipping Steps for Unchanged ServicesWhen using Git integration, Servicer can skip steps for the services that haven't changed (unless those steps are explicit dependencies). Here is an example of configuring this.git:
  enabled: true
  config:
    user.name: servicer   # The name Servicer will use when committing to the repo.
  ignore-unchanged: true  # Determine whether to skip unchanged services or not.
  ignore_paths:           # When running git diff, Servicer will ignore files that are matched by
    - *README.md          # values in this array.

services:
  my_python_package:      # Name shared by this service and the relevant directory within the repo.
    provider: pypi
    service_type: pypi

    git:
      watch_paths:              # When running git diff, Servicer will consider this service changed if
        - my_python_package/*   # any non-ignored paths returned are matched by any value in this array.
    steps:
      build:
        commands:
          - echo 'I must have changed!'In this case, Servicer will only executemy_python_package's steps (onlybuildis present) if any file within the my_python_package directory, except README.md, was returned by Servicer'sgit diff.Supported formats forwatch_pathsandignore_pathsare simple text with*as a wildcard, and full regexes in the format/.*/.Automatic VersioningWarning:This is an experimental feature. Automatic versioning requires Servicer to make automated commits back to your repository with updated version numbers for package services. This can have unintended side effects with your CI solution unless "you know what you're doing".As part of git integration, Servicer can automatically increase the version number of the packages it manages and commit the new version numbers to the repository. When Servicer is being run on a CI pipeline,ignore-servicer-commitscan be set to true to have servicer inspect the commit authors, and self-terminate if the job was initiated in response to a commit made by Servicer.Below is an example of a PyPi service that inherits from the Package Service Adapter and implements autoversioning.steps:
  build:
    config:
      package_file_path: ${package_directory}/setup.py
      steps:
        - type: set_auto_version
    post_commands:
      - rm -rf ${package_directory}/dist
      - python3 ${package_directory}/setup.py sdist --dist-dir=${package_directory}/dist
  deploy:
    config:
      package_file_path: ${package_directory}/setup.py
      steps:
        - type: upload
          args:
            server: artifactory
            path: ${package_directory}/dist/*.tar.gz
        - config:
          steps:
            - type: commit_and_push_changesEnvironment DestructionBecause every environment setup is unique, environment destruction is not automated. However, by configuring thedestroystep for services, you can condense the complete destruction of a service environment into a single command:SERVICE_ENVIRONMENT=my-env servicer --destroyExample kubernetes destroy configuration, assuming your services exist within a kubernetes namespace that matches your service_environment name:services:
  kubernetes:
    destroy:
      commands:
        - kubectl delete ns ${SERVICE_ENVIRONMENT}Thedestroystep is a magic step that does not need to be defined in your.servicer/services.yaml.Advanced Servicer Use CasesServicer's core interface with external CI's, cloud provider APIs, etc, exists as a collection of python modules called Adapters. These Adapters can be found in thebuiltindirectory of the project, and may be overridden or extended if they do not meet your needs.To override an Adapter, simply mirror the directory structure ofbuiltindirectly inside your.servicerfolder..servicer/
  service_adapters/
    aws/
      rds_instance.py  # this overrides servicer's builtin rds_instance adapter
      custom_task_service.py
  services.yamlcustom_task_service.pycan also inherit from servicer'sbuiltinadapter classes, and selectively override just the functionality that isn't working for you.#custom_adapter.py

from servicer.builtin.service_adapters.task_service import Service as BaseService

class Service(BaseService):
    # overrides task_service.Service's up() method, but still calls it
    def up(self):
        super().up()

        print('my custom logic here!')

    def foo(self):
        print('bar')Then you can utilize your custom service adapters like you normally would with builtin adapters:services:
  my-service:
    service_type: aws/custom_task_service