Package: bsl-appcli
BSL Application CLI LibraryA library for adding CLI interfaces to applications in the brightSPARK Labs style.OverviewThis library can be leveraged to add a standardised CLI capability to applications to:Handle system lifecycle events for services (service [start|shutdown]).Allow running arbitrary short-lived tasks (task run).Manage configuration (configure).Upgrade to a newer version of the application (upgrade|migrate).And more.The CLI is designed to run within a Docker container and launch other Docker containers (i.e.
Docker-in-Docker). This is generally managed via adocker-compose.ymlfile.The library exposes the following environment variables to thedocker-compose.ymlfile:APP_VERSION- the version of containers to launch.<APP_NAME>_CONFIG_DIR- the directory containing configuration files.<APP_NAME>_DATA_DIR- the directory containing data produced/consumed by the system.<APP_NAME>_GENERATED_CONFIG_DIR- the directory containing configuration files generated from
the templates in<APP_NAME>_CONFIG_DIR.<APP_NAME>_BACKUP_DIR- the directory to use for system backups.<APP_NAME>_ENVIRONMENT- the deployment environment the system is running in. For exampleproductionorstaging. This allows multiple instances of the application to run on the same
Docker daemon. Defaults toproduction.NOTE:
TheAPP_NAMEvariable is derived from theapp_namepassed in to theConfigurationobject in
the main python entrypoint to the application. In order for the application to work, theapp_nameis forced to conform with the shell variable name standard:[a-zA-Z_][a-zA-Z_0-9]*. Any characters
that do not fit this regex will be replaced with_. Seehereorherefor details.Thedocker-compose.ymlcan be templated by renaming todocker-compose.yml.j2, and setting
variables within thesettings.ymlfile as described in the Installation section.Stack variables can be set within thestack-settings.ymlfile as described in theBuild configuration template directoriessection.Quick StartRefer to thequick start guideto get a basic application running.Otherwise refer to the Installation section below to see all options.InstallationAdd the library to your python CLI applicationpip install git+https://github.com/brightsparklabs/appcli.git@<VERSION>Define the CLI for your applicationmyappNote for appcli version 1.1.3 and below: Import paths to access to appcli
internal classes and methods is now by a full path, rather than being exposed
at the root. This was done to allow access to all methods and classes using
python3 implicit namespaced packages.# filename: myapp.py#!/usr/bin/env python3# # -*- coding: utf-8 -*-# standard librariesfrompathlibimportPath# vendor librariesfromappcli.cli_builderimportcreate_clifromappcli.models.configurationimportConfigurationfromappcli.orchestratorsimportDockerComposeOrchestrator# ------------------------------------------------------------------------------# CONSTANTS# ------------------------------------------------------------------------------# directory containing this scriptBASE_DIR=Path(__file__).parent# ------------------------------------------------------------------------------# PRIVATE METHODS# ------------------------------------------------------------------------------defmain():configuration=Configuration(app_name='myapp',docker_image='brightsparklabs/myapp',seed_app_configuration_file=BASE_DIR/'resources/settings.yml',application_context_files_dir=BASE_DIR/'resources/templates/appcli/context',stack_configuration_file=BASE_DIR/'resources/stack-settings.yml',baseline_templates_dir=BASE_DIR/'resources/templates/baseline',configurable_templates_dir=BASE_DIR/'resources/templates/configurable',orchestrator=DockerComposeOrchestrator(# NOTE: These paths are relative to 'resources/templates/baseline'.docker_compose_file=Path('docker-compose.yml'),docker_compose_override_directory=Path('docker-compose.override.d/'),docker_compose_task_file=Path('docker-compose.tasks.yml'),docker_compose_task_override_directory=Path('docker-compose.tasks.override.d/'),),mandatory_additional_data_dirs=['EXTRA_DATA',],mandatory_additional_env_variables=['ENV_VAR_2',],)cli=create_cli(configuration)cli()# ------------------------------------------------------------------------------# ENTRYPOINT# ------------------------------------------------------------------------------if__name__=='__main__':main()A lot of the fields in the appcli constructor can be defaulted, resulting in less code.configuration=Configuration(app_name='myapp',docker_image='brightsparklabs/myapp',)cli=create_cli(configuration)cli()NullOrchestratorFor applications with no services to orchestrate, theNullOrchestratorcan be used. This is useful
for appcli applications which consist only of the launcher container containing various additional CLI
command groups. TheNullOrchestratordisables commands related to managing services.orchestrator=NullOrchestrator()Custom CommandsYou can specify some custom top-level commands by adding click commands or command groups to the
configuration object.  Assuming 'web' is the name of the service in the docker-compose.yml file
which you wish to exec against, we can create three custom commands in the following example:myapp ls-rootwhich lists the contents of the root directory within thewebservice container
and prints it out.myapp ls-root-to-filewhich lists the contents of the root directory within thewebservice
container and dumps to file within the container.myapp tee-filewhich takes some text andtees it into another file thewebservice
container.defget_ls_root_command(orchestrator:DockerComposeOrchestrator):@click.command(help="List files in the root directory",)@click.pass_contextdefls_root(ctx:click.Context):# Equivalent command within the container:# `ls -alh`cli_context:CliContext=ctx.objoutput:CompletedProcess=orchestrator.exec(cli_context,"web",["ls","-alh","/"])print(output.stdout.decode())returnls_rootdefget_tee_file_command(orchestrator:DockerComposeOrchestrator):@click.command(help="Tee some text into a file",)@click.pass_contextdeftee_file(ctx:click.Context):# Equivalent command within the container:# `echo "Some data to tee into the custom file" | tee /ls-root.txt`cli_context:CliContext=ctx.objoutput:CompletedProcess=orchestrator.exec(cli_context,"web",["tee","/my_custom_file.txt"],stdin_input="Some data to tee into the custom file")returntee_filedefget_ls_root_to_file_command(orchestrator:DockerComposeOrchestrator):@click.command(help="List files in the root directory and tee to file",)@click.pass_contextdefls_root_to_file(ctx:click.Context):# Equivalent command within the container:# `ls -alh | tee /ls-root.txt`cli_context:CliContext=ctx.objoutput:CompletedProcess=orchestrator.exec(cli_context,"web",["ls","-alh","/"])data=output.stdout.decode()orchestrator.exec(cli_context,"web",["tee","/ls-root.txt"],stdin_input=data)returnls_root_to_filedefmain():orchestrator=DockerComposeOrchestrator(Path("docker-compose.yml"))configuration=Configuration(app_name="appcli_nginx",docker_image="thomas-anderson-bsl/appcli-nginx",seed_app_configuration_file=Path(BASE_DIR,"resources/settings.yml"),stack_configuration_file=Path(BASE_DIR,"resources/stack-settings.yml"),baseline_templates_dir=Path(BASE_DIR,"resources/templates/baseline"),configurable_templates_dir=Path(BASE_DIR,"resources/templates/configurable"),orchestrator=orchestrator,custom_commands={get_tee_file_command(orchestrator),get_ls_root_command(orchestrator),get_ls_root_to_file_command(orchestrator)})cli=create_cli(configuration)cli()Build configuration template directoriesStore any Jinja2 variable definitions you wish to use in your configuration
template files inresources/settings.yml.Store any application context files inresources/templates/appcli/context/Store any appcli stack specific keys inresources/stack-settings.yml.Store yourdocker-compose.yml/docker-compose.yml.j2file inresources/templates/baseline/.Configuration files (Jinja2 compatible templates or otherwise) can be stored in one
of two locations:resources/templates/baseline- for templates which the end useris notexpected to modify.resources/templates/configurable- for templates which the end user is expected to modify.Schema validationConfiguration files will be automatically validated against provided schema files wheneverconfigure applyis run.
Validation is done withjsonschemaand is only available foryaml/ymlandjson/jsnfiles.
The JSON schema file must match the name of the file to validate with a suffix of.schema.json..
It must be placed in the same directory as the file to validate,
Thesettings.yml,stack_settings.ymlfile, and any files in theresource/templatesorresources/overridesdirectory can be validated.# resources/templates/configurable/my-config.ymlfoobar:5#resources/templates/configurable/my-config.yml.schema.json{"type":"object","properties":{"foobar":{"type":"number"}}}Application context filesTemplate files are templated with Jinja2. The 'data' passed into the templating engine
is a combination of thesettings.ymland all application context files
(stored inresources/templates/appcli/context, and referenced in theConfigurationobject asapplication_context_files_dir). Application context files that have the
extension.j2are templated using the settings fromsettings.yml.These are combined to make the data for templating as follows:{"settings":{...allsettingsfrom`settings.yml`},"application":{<app_context_file_1>:{...settingsfrom`app_context_file_1.yml`,optionallyjinja2templatedusingsettingsfrom`settings.yml`},...additionalapp_context_files}}As a minimal example with the following YAML files:# ./settings.ymlmain_settings:abc:123# ./resources/templates/appcli/context/app_constants.ymlother_settings:hello:world# ./resources/templates/appcli/context/app_variables.yml.j2variables:main_abc_setting:{{settings.main_settings.abc}}The data for Jinja2 templating engine will be:{"settings":{"main_settings":{"abc":123}},"application":{"app_constants":{"other_settings":{"hello":"world"}},"app_variables":{"variables":{"main_abc_setting":123}}}}Configure application backupAppcli'sbackupcommand creates backups of configuration and data of an application, stored
locally in the backup directory. The settings for backups are configured through entries in abackupsblock instack-settings.yml.The available keys for entries in thebackupsblock are:keyDescriptionnameThe name of the backup. Must be unique between backup definitions and usekebab-case.backup_limitThe number of local backups to keep. Set to0to disable rolling deletion.file_filterThe file_filter contains lists of glob patterns used to specify what files to include or exclude from the backup.frequencyThe cron-like frequency at which backups will execute.remote_backupsThe list of remote backup strategies.# filename: stack-settings.yml

backups:
  - name: "full"
    backup_limit: 0
    file_filter:
      data_dir:
        include_list:
        exclude_list:
      conf_dir:
        include_list:
        exclude_list:
    frequency: "* * *"
    remote_backups:Backup nameThe backupnameis a short descriptive name for the backup definition.
To avoid problems, wehighlyrecommendnamebe:unique between items in thebackupslistusekebab-caseExamples of good names:fullconf-onlyaudit-logsWithout a uniquename, backups from different items inbackupswill
overwrite each other without warning.Usingkebab-caseis necessary to avoid some issues withclickand filesystem
naming issues.When using thebackupcommand, you are able to supply the name
of the backup to run. If you have a backupnamewith a space in it, theclicklibrary cannot interpret the name as a whole string (even with quotes), so you
will be unable to run the backup individually.If the backupnamedoesn't usekebab-case, it may use some characters that
are incompatible with file and directory naming conventions. Appcli will
automatically slugify the name to something compatible, but this may cause
collisions in the folder names of backups to be taken which will lead to backups
being overwritten. e.g.s3#1ands3&1will both translate internally tos3-1.Backup limitA rolling deletion strategy is used to remove local backups, in order to keepbackup_limitnumber
of backups.If more thanbackup_limitnumber of backups exist in the backup directory, the oldest backups will
be deleted.Set this value to0to keep all backups.File filterThefile_filterblock enables filtering of files to backup fromconfanddatadirectories. For
more details including examples, seehere.# filename: stack-settings.yml
# Includes all log files from data dir only

backups:
  - name: "full"
    backup_limit: 0
    file_filter:
      data_dir:
        include_list:
          - "**/*.log"
        exclude_list:
        conf_dir:
          include_list:
          exclude_list:
            - "**/*"
    frequency: "* * *"
    remote_backups:FreqencyAppcli supports limiting individual backups to run on only specific days using a cron-like frequency
filter.When thebackupcommand is run, each backup strategy will check if thefrequencypattern matches
today's date. Only strategies whosefrequencypattern match today's date will execute.The input patternpatternis prefixed with"* * "and is used as a standard cron expression to
check for a match. i.e."* * $pattern".Examples:"* * *"(cron equivalent"* * * * *") will always run."* * 0"(cron equivalent"* * * * 0") will only run on Sunday."1 */3 *"(cron equivalent"* * 1 */3 *") will only run on the first day-of-month of every 3rd
month.Remote backupAppcli supports pushing local backups to remote storage. The list of strategies for pushing to
remote storage are defined within theremote_backupsblock.The available keys for every remote backup strategy are:keyDescriptionnameA short name or description used to describe this backup.strategy_typeThe type of this backup, must match an implemented remote backup strategy.frequencyThe cron-like frequency at which remote backups will execute. Behaves the same as local backupfrequency.configurationCustom configuration block that is specific to each remote backup strategy.N.B. remote backups will only run for a local backup that has run. Therefore thefrequencyof the
local backup will apply first, followed by thefrequencyof the remote backup. This means that
it's possible to write a remote backup frequency that will never execute. e.g. Local* * 0and
remote* * 1.StrategiesAWS S3 remote strategyTo use S3 remote backup, setstrategy_typetoS3.
The available configuration keys for an S3 backup are:keyDescriptionbucket_nameThe name of the bucket to upload to.access_keyThe AWS Access key ID for the account to upload with.secret_keyThe AWS Secret access key for the account to upload with. The valuemustbe encrypted using the appcliencryptcommand.bucket_pathThe path in the S3 bucket to upload to. Set this to an empty string to upload to the root of the bucket.tagsKey value pairs of tags to set on the backup object.# filename: stack-settings.yml

backups:
  - name: "full_backup"
    backup_limit: 0
    remote_backups:
    - name: "weekly_S3"
      strategy_type: "S3"
      frequency: "* * 0"
      configuration:
        bucket_name: "aws.s3.bucket"
        access_key: "aws_access_key"
        secret_key: "enc:id=1:encrypted_text:end"
        bucket_path: "bucket/path"
        tags:
          frequency: "weekly"
          type: "data"Restoring a remote backupTo restore from a remote backup:Acquire the remote backup (.tgzfile) that you wish to restore. For S3 this can be done by
downloading the backup from the specified bucket.Place the backupmyapp_date.tgzfile in the backup directory. By default this will be/opt/brightsparklabs/${APP_NAME}/production/backup/Confirm that appcli can access the backup by running theview-backupscommandRun the restore command./myapp restore BACKUP_FILE.tgze.g../myapp restore APP_2021-02-02T10:55:48+00:00.tgz. The restore process will trigger a backup.Define a container for your CLI application# filename: Dockerfile

FROM brightsparklabs/appcli

ENTRYPOINT ["./myapp.py"]
WORKDIR /app

# install compose if using it as the orchestrator
RUN pip install docker-compose

COPY requirements.txt .
RUN pip install --requirement requirements.txt
COPY src .

ARG APP_VERSION=latest
ENV APP_VERSION=${APP_VERSION}Build the container# sh
docker build -t brightsparklabs/myapp --build-arg APP_VERSION=latest .(Optional) Login to private Docker registries and pass through credentialsIt is possible to login to private Docker registries on the host, and pass through credentials to
the CLI container run by the launcher script. This enables pulling and running Docker images from
private Docker registries.Login using:docker login ${REGISTRY_URL}The credentials file path can be passed as an option via--docker-credentials-fileor-pto themyappcontainer.View the installer script# sh
docker run --rm brightsparklabs/myapp:<version> install

# or if using a private registry for images
docker run --rm brightsparklabs/myapp:<version> \
  --docker-credentials-file ~/.docker/config.json \
  installWhile it is not mandatory to view the script before running, it is highly recommended.Run the installer script# sh
docker run --rm brightsparklabs/myapp:<version> install | sudo bashThe above will use the following defaults:environment=>production.install-dir=>/opt/brightsparklabs/${APP_NAME}/production/.configuration-dir=>/opt/brightsparklabs/${APP_NAME}/production/conf/.data-dir=>/opt/brightsparklabs/${APP_NAME}/production/data/.backup-dir=>/opt/brightsparklabs/${APP_NAME}/production/backup/.You can modify any of the above if desired. E.g.# sh
docker run --rm brightsparklabs/myapp:<version> \
    --environment "uat" \
    --configuration-dir /etc/myapp \
    --data-dir /mnt/data/myapp \
    install --install-dir ${HOME}/apps/myapp \
| sudo bashWhere:--environmentdefines the environment name for the deployment. This allows multiple instances of
the application to be present on the same host.
Defaults toproduction.--install-dirdefines the base path for launcher and the default locations for the configuration
and data directories if they are not overrideen (see below).
Defaults to/opt/brightsparklabs/${APP_NAME}/${ENVIRONMENT}/(where${ENVIRONMENT}is defined
by--environmentabove).--configuration-dirdefines the path to the configuration directory.
Defaults to${INSTALL_DIR}/conf/(${INSTALL_DIR}is defined by--install-dirabove).--data-dirdefines the path to the data directory.
Defaults to${INSTALL_DIR}/data/(${INSTALL_DIR}is defined by--install-dirabove).The installation script will generate a launcher script for controlling the application. The script
location will be printed out when running the install script. This script should now be used as the
main entrypoint to all appcli functions for managing your application.Migration from appcli version <=1.3.6 to version >1.3.6As a result of supporting application context files, all references to
settings in template files have moved.All settings insettings.ymlused in templating are now namespaced undersettings. All templates will need to change their references to use this new
namespacing scheme. For example, in templates that refer to settings, change the
references like so:my_app.server.hostname->settings.my_app.server.hostnamemy_app.server.http.port->settings.my_app.server.http.portUsageThis section details what commands and options are available.Top-level CommandsTo be used in conjunction with your application./myapp <command>e.g../myapp configure initCommandDescriptionbackupCreate a backup of application data and configuration.configureConfigures the application.encryptEncrypts the specified string.initInitialises the application.launcherOutputs an appropriate launcher bash script.migrateMigrates the configuration of the application to a newer version.orchestratorPerform docker orchestrationrestoreRestore a backup of application data and configuration.serviceLifecycle management commands for application services.taskCommands for application tasks.versionFetches the version of the app being managed with appcli.view-backupsView a list of locally-available backups.OptionsOptionDescription--debugEnables debug level logging.-c, --configuration-dir PATHDirectory containing configuration files. [This is required unless subcommand is one of:install.-d, --data-dir PATHDirectory containing data produced/consumed by the system. This is required unless subcommand is one of:install.-t, --environment TEXTDeployment environment the system is running in. Defaults toproduction.-p, --docker-credentials-file PATHPath to the Docker credentials file (config.json) on the host for connecting to private Docker registries.-a, --additional-data-dir TEXTAdditional data directory to expose to launcher container. Can be specified multiple times.-e, --additional-env-var TEXTAdditional environment variables to expose to launcher container. Can be specified multiple times.--helpShow the help message and exit.Command:backupCreates a backup.tgzfile in the backup directory that contains files from the configuration and
data directory, as configured instack-settings.yml. After the backup is taken, remote backup
strategies will be executed (if applicable).usage:./myapp backup [OPTIONS] [ARGS]OptionDescription--pre-stop-services/--no-pre-stop-servicesWhether to stop services before performing backup.--post-start-services/--no-post-start-servicesWhether to start services after performing backup.--helpShow the help message and exit.Thebackupcommand optionally takes an argument corresponding to thenameof the backup to run.
If nonameis provided, all backups will attempt to run.Command Group:configureConfigures the application.usage:./myapp configure [OPTIONS] COMMAND [ARGS]CommandDescriptionapplyApplies the settings from the configuration.diffGet the differences between current and default configuration settings.getReads a setting from the configuration.get-secureReads a setting from the configuration, decrypting if it is encrypted. This will prompt for the setting key.initInitialises the configuration directory.setSaves a setting to the configuration. Allows setting the type of value with option--type, and defaults to string type.templateConfigures the baseline templates.editOpen the settings file for editing with vim-tiny.OptionDescription--helpShow the help message and exit.Command:encryptEncrypts the specified string.usage:./myapp encrypt [OPTIONS] TEXTOptionDescription--helpShow the help message and exit.Command Group:initInitialises the application.usage:./myapp init [OPTIONS] COMMAND [ARGS]CommandDescriptionkeycloakInitialises a Keycloak instance with BSL-specific initial configuration.OptionDescription--helpShow the help message and exit.Command:launcherOutputs an appropriate launcher bash script to stdout.usage:./myapp launcher [OPTIONS]OptionDescription--helpShow the help message and exit.Command:migrateMigrates the application configuration to work with the current application version.usage:./myapp migrate [OPTIONS]OptionDescription--helpShow the help message and exit.Command Group:orchestratorPerform tasks defined by the orchestrator.usage:./myapp orchestrator [OPTIONS] COMMAND [ARGS]All commands are defined within the orchestrators themselves. Run./myapp orchestratorto list
available commands.For example, the following commands are available to docker-compose:CommandDescriptionpsList containers for the appcli project, with current status and exposed ports.composeRun a docker compose command. Seedocker compose.OptionDescription--helpShow the help message and exitCommand:restoreRestores a specified backup.tgzfile from the configured backup folder.usage:./myapp restore BACKUP_FILEOptionDescription--helpShow the help message and exit.Command Group:serviceRuns application services. These are the long-running services which should only exit on command.usage:./myapp service [OPTIONS] COMMAND [ARGS]CommandDescriptionlogsPrints logs from all services.shutdownShuts down the system. If one or more service names are provided, shuts down the specified service(s) only.startStarts the system. If one or more service names are provided, starts the specified service(s) only.restartRestarts service(s) (shutdownfollowed bystart). Optionally run aconfigure applyduring the restart with the--applyflag. If one or more service names are provided, restarts the specified service(s) only.statusLists all containers for the appcli project, with current status and exposed ports. If one or more service names are provided, lists the status andexposed ports of the specified service(s) only.OptionDescription--helpShow the help message and exit.Command Group:taskRuns application tasks. These are short-lived services which should exit when the task is complete.usage:./myapp task [OPTIONS] COMMAND [ARGS]CommandDescriptionrunRuns a specified application task. Optionally run in the background with-d/--detachflag.OptionDescription--helpShow the help message and exit.Command:versionFetches the version of the app being managed with appcli.usage:./myapp versionCommand:view-backupsView a list of all backups in the configured backup folder.usage:./myapp view-backupsOptionDescription--helpShow the help message and exit.Usage within scripts and cronBy default, the generatedappclilauncher script will run the CLI container with a virtual
terminal session (tty).  This may interfere with crontab entries or scripts that use the appcli
launcher.To disable tty when running the launcher script, setNO_TTYenvironment variable totrue.NO_TTY=true ./myapp [...]orexport NO_TTY=true
./myapp [...]If required, you can also disable interactive mode with theNO_INTERACTIVEenvironment variable.NO_INTERACTIVE=true ./myapp [...]orexport NO_INTERACTIVE=true
./myapp [...]DevelopmentThis section details how to build/test/run/debug the system in a development environment.PrerequisitesThe following must be installed and in thePATH:makepython 3.7+virtualenvgitBuildmake allInstallpip install -e .Running unit testsmake testUsage while developing your CLI applicationWhile developing, it may be preferable to run your python script directly rather than having to
rebuild a container each time you update it.Ensure docker is installed (more specifically a docker socket at/var/run/docker.sock).Set the environment variables which the CLI usually sets for you:export \
      MYAPP_DATA_DIR=/tmp/myapp/data \
      MYAPP_CONFIG_DIR=/tmp/myapp/config \
      MYAPP_GENERATED_CONFIG_DIR=/tmp/myapp/config/.generated \
      MYAPP_BACKUP_DIR=/tmp/myapp/backup \
      MYAPP_ENVIRONMENT=devRun your CLI application:./src/myapp.py \
        --debug \
        --configuration-dir "${MYAPP_CONFIG_DIR}" \
        --data-dir "${MYAPP_DATA_DIR}" \
        --backup-dir "${MYAPP_BACKUP_DIR}" \
        --environment "${MYAPP_ENVIRONMENT}ContributingWhen committing code, callmake allto automatically run code formatting/ linting/testing.Appcli uses the python code formatterblackwith default
settings. This ensures that PR diffs are minimal and focussed on the code change rather than
stylistic coding decisions.Install withpip install black. This can be run through VSCode or via the CLI. See theblackdocumentation for details.LicensesRefer to theLICENSEfile for details.This project makes use of several libraries and frameworks. Refer to theLICENSESfolder for
details.