Package: dcos-deploy
dcos-deploy is a command line tool that helps you to deploy and manage groups of services and apps onDC/OS. It acts as an orchestration engine on top of the existing DC/OS tools and APIs to install, configure, update and remove frameworks, marathon apps and metronome jobs.It is based on a yaml-configuration file which describes a goal state of the services that you want. dcos-deploy will read this file and execute any changes necessary so that your DC/OS cluster reflects your desired configuration / goal-state. dcos-deploy does not keep its own state. Instead it uses the dcos cli to determine the current state of the cluster and applies any change required to reach the goal state defined in your yaml configuration file.For example: To deploy a complete elasticsearch stack on your cluster you would typically need to install the elasticsearch framework from the DC/OS universe, a kibana app, expose both to your loadbalancer and add some regular jobs for backups and cleanup. Additionally if you run elasticsearch with x-pack or searchguard installed you also need to create a public-private keypair, service-account and secret for your framework. This amounts to quite a number of steps. With dcos-deploy you just describe your entire goal state in one simple yaml file and let dcos-deploy do the rest. See the examples folder for more.(!) This tool is under heavy development. Use in production environments at your own risk.FeaturesHandles the following "entities":DC/OS packagesMarathon appsMetronome jobssecretsserviceaccountspublic-private-keypairs (for use in secrets)Edge-LBS3filesFor DC/OS packages it supports version updates and configuration changes.Handles install and update dependencies between entities (e.g. a framework is only installed after its serviceaccount is created, an app is restarted if an attached secret changes).Parameterise your configuration using variables (e.g. to support different instances of a service).Stateless / no backend: There is no local or remote state. Everything needed is pulled directly from the cluster: dcos-deploy is a client-only tool that directly uses the existing DC/OS APIs to access your DC/OS cluster and does not need its own backend service.Uses your installed dcos-cli to figure out how to talk to your cluster. No extra configuration needed.dry-run mode: Check which changes would be done.partial deployment: Choose the entities to be deployed.Can be extended with extra modules (still in development).LimitationsDeleting packages/apps/jobs is not supported: Since dcos-deploy does not keep a state it cannot detect if you remove a service/app/job from its configuration. Therefore you are responsible to delete any no longer wanted entities yourself.Frameworks/packages with more complicated update procedures (like Edge-LB) are at the moment not fully supported.RequirementsA DC/OS cluster with version >= 1.11 (for features like secrets an EE cluster is needed)dcos-cli installed  and connected to your cluster (to verify it works rundcos nodeand it should display a list of nodes in your cluster)Python >= 3.5 (if you do not use a binary)InstallationThere are several ways to install dcos-deploy:Install from pypi (pip install dcos-deploy) (Recommended)Or if you want the newest features directly from master:pip install git+https://github.com/MaibornWolff/dcos-deployBinaryDownload the binary for your system from the Releases pageMake the file executable and copy it into a folder inside your pathNote: The binaries have not been specifically tested and might not support all features. Only use them if you do not have python on your systemUsageCreate yourdcos.yml(start from scratch or use one of the examples). You should separate your stack into groups and create adcos.ymlfor each of them (e.g. one for all hdfs related services, one for all elastic related and so on) to keep the complexity manageable.Rundcos-deploy apply.Seedcos-deploy apply --helpfor all options.Credentialsdcos-deploy has several ways to retrieve connection credentials for your DC/OS cluster:dcos-cli: By default dcos-deploy uses the authentication information from the dcos-cli, so make sure you are logged in (verify by runningdcos node, if it works you should see a list of nodes in your cluster).Static token via environment variables:DCOS_BASE_URL(set this to the public URL of your master) andDCOS_AUTH_TOKEN(set this to a valid auth token). This way is handy for automation situations where you do not want to expose the admin username and password by adding them to your automation tool (e.g. Gitlab CI secrets). Instead you can provide the automation job with short-lived credentials.DC/OS serviceaccount secret: If you are running dcos-deploy from inside your DC/OS cluster (e.g. in a metronome job) you can also provide the credentials via a serviceaccount secret. To do that create a serviceaccount with superuser rights and expose its credentials secret to your service/job via the environment variableDCOS_SERVICE_ACCOUNT_CREDENTIAL. Also provide the variableDCOS_BASE_URLand set it to the internal URL of your master (should behttps://leader.mesosin most cases).Username and Password via environment variables:DCOS_BASE_URL(set this to the public URL of your master),DCOS_USERNAME(username of a DC/OS admin user) andDCOS_PASSWORD(password of a DC/OS admin user).Config file syntaxThe config file is written as a yaml file. The root level consists of key/value-pairs (a dictionary). Each key represents the unique name for one entity, the value is again a dictionary with all the options for that entity.Meta fieldsThere are some meta fields for further configuation:variables: Define variables to be used in the rest of the file and in app definitions and package options. See theVariablessection for more info.includes: Structure your config further by separating parts into different files and including them. Provide a list of filenames. The include files must be structured the same way as the main file. Each entity name must be unique over the base file and all included files.modules: Extend the features of dcos-deploy using external modules (still in development, not documented yet).VariablesTo be more generic with your services you can use variables in certain places. All variables must be defined in thevariablessection of your config file.variables:var1:required:True# Required variables must be provided when calling dcos-deploy using the -e option (e.g. -e var1=foo)var2:required:Falsedefault:foobar# If a variable is not provided by the user the default value is used. Variables without a default are treated as empty valuesvar3:values:# You can restrict the allowed values for a variable-foo-bar-bazdefault:barvar4:env:MY_VAR4# Variables can also be read from the environmentrequired:Truevar5:file:myvalue.txt# The value of the variable will be the content of the file. Can be useful to provide longer valuesencode:base64# This will encode the value using base64. Some DC/OS frameworks have config options that require base64-encoded values. Using this option the value can be kept in clear-text and will be automatically encoded by dcos-deploy when rendering the options file for the framework.Variables can be used viaMustachetemplates in most options and in marathon app definitions and package options. Seeexamples/elasticfor usage examples. You can not use variables in entity names or outside option strings (ground rule: the yaml file must be syntactically correct without mustache template rendering).Some variables are automatically provided bydcos-deploybased on your cluster. These are:_cluster_version: The DC/OS version of your cluster, for example1.12.2_cluster_variant: The DC/OS variant of your cluster, for exampleenterprise_cluster_name: The name of your cluster_num_masters: The number of masters in your cluster_num_private_agents: The number of private agents in your cluster (useful if you want to tailor the size of an app or framework node to the cluster size)_num_public_agents: The number of public agents in your cluster_num_all_agents: The number of public and private agents in your clusterDuring rendering of entity configs some entity specific variables are available. These are:_entity_name: The name of the entity (the key of the yaml definition)_pre_apply_script_hash,_pre_delete_script_hash,_post_apply_script_hashand_post_delete_script_hash: md5-hashes of pre and post scripts if defined (seePre and post script)Global configTo specifiy an attribute for all entities of a specific type you can define it in the global config. The config for a specific entity will be merged with the global config for the corresponding type.Example:global:s3file:server:endpoint:"{{s3_endpoint}}"access_key:"{{s3_access_key}}"secret_key:"{{s3_secret_key}}"Encryptiondcos-deploy supports encrypting files so that sensitive information is not stored unencrypted. Files are symmetricly encrypted usingFernet(AES-128) from the pythoncryptographylibrary.
In most places where you provide a filename (at the moment not possible withs3file) you can use encrypted files by using the following syntax as filename:vault:<encryption-key>:<filename-to-encrypted-file>.
You should use a variable for the encryption key.Example:variables:encryption_key:env:ENCRYPTION_KEYrequired:Trueservicepasswords:type:secretpath:/myservice/passwordsfile:"vault:{{encryption_key}}:servicepasswords.encrypted"You can also specify the encryption key in the global config to avoid writing it for every occurence:variables:encryption_key:env:ENCRYPTION_KEYrequired:Trueglobal:vault:key:"{{encryption_key}}"servicepasswords:type:secretpath:/myservice/passwordsfile:vault::servicepasswords.encrypteddcos-deploy has several util commands that help you in creating encrypted files:dcos-deploy vault generate-key: Generates a new key that you can use for encrypting filesdcos-deploy vault encrypt: Encrypts a file using a given keydcos-deploy vault decrypt: Decrypts a file using a given keydcos-deploy vault view: Show the decrypted content of an encrypted filedcos-deploy vault edit: Edit an encrypted file in-placeExample usage:$echo"supersecret">servicepasswords
$dcos-deployvaultgenerate-key
Yournewkey:6htLemxXcEXnahVl1aZI6Aa3TXIHmbE8abtibC2iO6c=Keepthissafe.
$exportENCRYPTION_KEY="6htLemxXcEXnahVl1aZI6Aa3TXIHmbE8abtibC2iO6c="$dcos-deployvaultencrypt-iservicepasswords-oservicepasswords.encrypted-eENCRYPTION_KEY
$catservicepasswords.encrypted
gAAAAABb9rkSdEqT1xn0w5G5ipjPH6Vd5TsUfDLAWnN7I8FXTOPK6t1tMUstm8nA_yiA6SV5B1Blxj1h-xqCl8VqqbH7D7RF9Q==$dcos-deployvaultdecrypt-iservicepasswords.encrypted-oservicepasswords.clear-eENCRYPTION_KEY
$catservicepasswords.clear
supersecretTo use encrypted files with includes make sure that the variable for the encryption key is either defined in the file itsself or in an included file that is read before the encrypted one.EntityEach entity has the same structure and some common options:entityname:type:fooonly:var1:fooexcept:var2:barstate:removeddependencies:-otherentityloop:var3:-a-btypedefines what kind of entity this is. Currently implemented areappframeworkjobaccountsecretcertrepositoryedgelbs3filetaskexeciam_groupiam_usermarathon_groupSee their respective sections below for details.onlyandexcepttake key/value-pairs of variable names and values. These are evaluated when reading the config file based on all provided and default variables. The entity is excluded if one of the variables in theonlysection does not have the value specified or if one of the variables in theexceptsection has the specified value. In the example above the entity is only included ifvar1 == fooandvar2 != bar. If a variable in theonlysection is not defined (no default value) the condition is treated as false and the entity is ignored.statecan be used to define a specific state for an entity. Currently only none (default, option is ignored) andremovedare supported. By specifyingstate: removedthe entity will be deleted if it exists. This can be useful in several ways. For example in air-gapped clusters the normally configured universe repository is not reachable and must be removed before other frameworks can be installed from local universes / package registries. This can be accomplished by defining the universe repo as an entity withstate: removed.dependenciestakes a list of entity names that this entity depends on. Optionally the dependency type can be provided. Currently supported arecreate(default) andupdate. They can be defined by adding a colon after the entity name and then the type (e.g.otherentity:create). Acreatedependency is only honored during creation time of an entity and means that the entity will only be created after all its dependencies have been successfully created (e.g. a service account for a framework). An update dependency extends thecreatedependency and is currently only honored by themarathonmodule. If during an apply-operation a dependency of a marathon app is changed (e.g. a secret) and the app has no changes it will be restarted.loopis very useful for describing multiple entities that are very similar. The values of the variables defined underloopwill be extended into a cross product and for each combination an entity with these extra variables will be created. These extra variables can be used like normal variables to parametrize the entity. By default the entity name will be created by concatenating the given name with the values of all loop variables (in the example above these would beentityname-aandentityname-b). This can be overridden by using an entityname that has template parameters (for example{{var3}}-myentity).Pre and post scriptIn case you have need of more complex orchestrations for entities you can usepre_scriptandpost_script. With these you can specify python scripts for theapplyand thedeleteoperations that will be executed before or respectively after the entitiy has been created/updated or deleted. They can be specified as follows:variables:post_script:file:path:my_script.pyrender:Truefoobar:pre_script:apply:|print("Hello pre apply")delete:|print("Hello pre delete")post_script:apply:"{{{post_script}}}"delete:|print("Hello post delete")If it is just a short snippet you can specify it inline, otherwise you can put the script into a file, read the file into a variable and just reference the variable. You can even use moustache templating in the scripts and dcos-deploy will render the template before executing the script.The scripts are executed in the context of dcos-deploy, so you can import any dcosdeploy library functions that you need. Additonally during script execution the following globals are provided:entity: the entity config for the entity in questionentity_variables: A dict of any entity_specific variablesvariables: An object that allows access to all defined variables (usevariables.get(name)to get the value for a variable orvariables.render(text)to render a moustache template)Important: As the script is executed in the context of dcos-deploy it has access to the entire runtime state, including DC/OS authentication and all (encryped) variables. So only use scripts that you wrote yourself or from completely trusted sources.If an entity has pre or post scripts defined, hash values for those scripts are provided as entity variables (_pre_apply_script_hash,_pre_delete_script_hash,_post_apply_script_hash,_post_delete_script_hash). These can be used to force an update with script execution of the entity whenever the script changes. In marathon apps you can e.g. put the variable in an environment variable or in a framework you can put it into an extra unused config option.Marathon apptype: appdefines a marathon app. It has the following specific options:path: id of the app. If not specified theidfield of the marathon app definition is used. Variables can be used.marathon: path to the marathon app definition json file. Required. Variables can be used in the path and in the json file itsself.extra_vars: key/value-pairs of of extra variables and values to be used when rendering the app definition file.If you want to start several apps from the same basic app definition, there is a meta option to allow this:mymultiapp:type:app_template:marathon.json_vars:mymultiapp.ymlThis option andloopcan not be used at the same time.Themarathon.jsonfile is your normal app definition json file, paramerised with mustache variables. Themymultiapp.ymlfile has the following structure:defaults:# optional, key/value-pairs of variables, can get overwritten in the instances sectionvar1:foovar2:barinstances:instance1:var1:bazvar3:helloinstance2:var3:worldFor each key underinstancesdcos-deploy will create a marathon app from the template file specialized with the variables provided.In both theextra_varsand the variables in theinstancesyou can define dependent variables. The variable name must be in the form<existing variable name>:<value to compare>, and the value must be key-value pairs of variables. If<existing variable name>has value<value to compare>the variables defined under that key will be added to the variables for that entity, otherwise they will be discarded. A simple example:variables:env:required:Truefoo:somethingmyapp:type:appmarathon:myapp.jsonextra_vars:env:test:foo:barenv:int:foo:bazIfenvhas valuetest, the variablefoowill have valuebar, ifenvhas valueint, the variablefoowill have valuebaz, in neither of these casesfoowill have valuesomething.If not defined marathon will add a number of default fields to an app definition. dcos-deploy tries to detect these defaults and exclude them when checking for changes between the local definition and the one known to marathon. This is rather complex as these default values partly depend on several modes (network, container type, etc.). If you find a case where dcos-deploy falesly reports a change please open an issue at the github project and attach your app definition and the definition reported by marathon (viadcos marathon app show <app-id>).Frameworktype: frameworkdefines a DC/OS framework. It has the following specific options:path: name of the framework. If not specified theservice.namefield of the package options is used. Variables can be used.package:name: name of the package in the DC/OS universe. This is the same as used when installing a package via the cli (dcos package install <packagename>). Required. Variables can be used.version: version of the package. This is the same as used when installing a package via the cli (dcos package install <packagename> --package-version=<version>). Required. Variables can be used.options: Path to the json options file to configure the package. Required. Variables can be used in the path and in the json file itsself.wait_on_update: By default dcos-deploy assumes that frameworks handle updates in a rolling and high available manner (like elastic or kafka) and therefore does not wait for plans to complete during updates. If this option is set to true than dcos-deploy will instead wait for theupdate(if it exists) ordeployplan to complete before moving on to the next entity. Optional. Defaults to false.Most of these options correspond to the parameters provided when installing a package via the dcos-cli:dcos package install <packagename> --package-version=<version> --options=<options.json>.During installation of a package dcos-deploy will wait until the framework is installed. If the framework exposes the standard plan API (like all frameworks based on theMesosphere SDK, e.g. elastic, hdfs or kafka) dcos-deploy will also wait (with a timeout of 10 minutes) until the deploy-plan is complete.
Any change in the options file or in the package version will trigger an update (the same as doingdcos <framework> update start --package-version=<version> --options=<options.json>).Metronome jobtype: jobdefines a metronome job. It has the following specific options:path: id of the job. If not specified theidfield of the job definition is used. Variables can be used.definition: path to the metronome job definition json file. Required. Variables can be used in the path and in the json file itsself.run: Run the job directly after creating/updating iton_create: If set to true job will be run after it has been created. Optional. Defaults to false.on_update: If set to true job will be run after it has been updated. The run is also triggered if an update dependency has been changed. Optional. Defaults to false.wait_for_success: If set to true dcos-deploy will wait until the job run has completed. If the job run fails the deployment will fail. Optional. Defaults to true.timeout: Time in seconds to wait for job run completion. After that time the deployment will fail. Optional. Defaults to 10 minutes. Variables can be used.The job definition is expected to be in the format used starting with DC/S 1.13. It is described in theDC/OS 1.13 release notes.Secrettype: secretdefines a secret. This can only be used on EE clusters. It has the following specific options:path: path for the secret. Required. Variables can be used.value: Value for the secret. Variables can be used. Either this orfileis required.file: Path to a file. The content of the file will be used as value for the secret. Either this orvalueis required.render: Wether to render the file content with mustache. Use if your file contains variables. Boolean. Defaults to False.Serviceaccounttype: serviceaccountdefines a serviceaccount. This can only be used on EE clusters. It has the following specific options:name: Name of the serviceaccount. Required. Variables can be used.secret: Path to use for the secret that contains the private key associated with the account. Required. Variables can be used.groups: List of groups the account should be added to. Optional. Variables can be used.permissions: Permissions to give to the account. Dictionary. Key is the name of the permission (rid), value is a list of actions to allow. If a permission does not yet exist, it will be created. Variables can be used.This entity equates to the steps required to create a serviceaccount for a framework as described in theDC/OS documentation.
Any groups or permissions not specified in the config are removed from the account during the update process.Example:type:serviceaccountname:hdfs-service-accountsecret:hdfs/serviceaccount-secretpermissions:"dcos:mesos:master:framework:role:hdfs-role":-create"dcos:mesos:master:reservation:role:hdfs-role":-create-deleteX.509 certificatetype: certis a special type that uses theDC/OS CA APIto create a public-private keypair, sign it with the internal DC/OS CA and provide the key and certificate as secrets. This can only be used on EE clusters. It has the following specific options:cert_secret: secret path to store the certificate. Required. Variables can be used.key_secret: secret path to store the key. Required. Variables can be used.dn: distinguished name to be used for the certificate. Requied. Variables can be used.hostnames: List of hostnames to put in the certificate. Optional.encoding: Encoding to use for the private key (PEM, DER), default is PEM. Optional.format: Format to use for the private key (PKCS1, PKCS8), default is PKCS1. Optional.algorithm: Algorithm to use for the private key (RSA, ECDSA), default is RSA. Optional.key_size: Key size in bits to use for the private key. Default is 2048 for RSA and 256 for ECDSA. Optional.You can use this feature if you want to secure communication between services inside DC/OS. For each service you define a correspondingcertentity. The keys and certificates are securely kept inside the cluster and using thepath restrictionsfor secrets can only be accessed by the intended services.All names from thehostnamesoption will be added to the SAN part of the certificate. If your service requires theCNto be set you must add it to thednoption, otherwise the certificate will not have one.Once the certificate has been created any changes in any of the options will not be detected and will not trigger the creation of a new certificate unless the secrets for key and cert are deleted beforehand. Also be aware that the created certificates are valid for about 10 years and there is no way to revoke certificates. As such please only use them inside your DC/OS cluster.Package repositorytype: repositorydefines a package repository. It has the following specific options:name: name of the repository. Required. Variables can be used.uri: uri for the repository. Required. Variables can be used.index: at what index to place the repository in the repository list. 0 for beginning. Do not set for end of list.With this type you can add additional package repositories to DC/OS. You can for example use it to add the Edge-LB repositories to your EE cluster. Set the repository as a dependency for any frameworks/packages installed from it.Edge-LB pooltype: edgelbdefines an edgelb pool. This can only be used on EE clusters. It has the following specific options:api_server: path to the edgelb-api server if not installed at default location. Variables can be used. Optional. E.g. if you installed Edge-LB atinfra/edgelbthen use this forapi_server.name: name of the pool. Taken from pool config if not present. Variables can be used.pool: filename to the yaml file for configuring the pool. Required. The filename itsself and the yaml file can contain variables.template: filename to an haproxy template file that should be used for the pool (seecustomizing-templates). Optional. The filename can contain variables, the template cannot.For configuring a pool see theEdge-LB pool configuration reference.For this to work you must have the edgelb package installed via the repository URLs provided by Mesosphere (use therepositoryandpackagetypes, there is acomplete exampleavailable).
At the moment dcos-deploy can not safely detect if the pool config was changed so it will always apply it. Be aware that changing certain options in the pool config (like ports or secrets) will result in a restart of the haproxy instances. Make sure you have a HA setup so that there is no downtime.Only Edge-LB 1.5 is supported. If you need support for an older version please open an issue in github.S3 Filetype: s3filedefines a file to be uploaded to S3-compatible storage. If you are not running on AWS you can useMinioor any other storage service with an s3-compatible interface. This entitiy is designed to provide files for apps / services (via the fetch mechanism) that are either too big or otherwise not suited for storage as secrets (for example plugins for a service). It has the following specific options:server:endpoint: endpoint of your s3-compatible storage. If you use AWS S3 set this to your region endpoint (e.g.s3.eu-central-1.amazonaws.com). Required. Variables can be used.access_key: your access key. Required. Variables can be used. For security reasons you should provide the value via environment variable. Make sure the iam user associated with these credentials has all rights for retrieving and uploading objects (s3:Get*ands3:Put*).secret_key: your secret access key. Required. Variables can be used. For security reasons you should provide the value via environment variable.ssl_verify: Set to false to disable ssl verifcation for s3 connection. Defaults to true. Optional.secure: Set to false to use insecure http connections for s3 connection. Defaults to true. Optional.wait_for_endpoint: if you use minio and deploy it in the same deployment as s3files then there can be situations where minio is deployed but not yet reachable via loadbalancer. Setting this option to true will make dcos-deploy wait until minio is reachable before continuing with the deployment. Defaults to false. Optional.source: path to your file or folder to be uploaded to s3. Should be relative to thedcos.ymlfile. Required. Variables can be used.destination:bucket: name of the bucket to use. Required. Variables can be used.key: key to store the file(s) under. Required. Variables can be used.create_bucket: Set to true to create bucket if it does not exist. Defaults to false. Optional.bucket_policy: S3 bucket policy content. When bucket will be created because ofcreate_bucket: truethe policy will be applied. Optional.compress: type of compressed archive to combine the files in. Currently only supported iszip. Optional. Variables can be used.This entity has several modes of operation:Upload a single file:sourcepoints to a file,compressis not set. The file is uploaded anddestination.keyis used as name.Upload a folder:sourcepoints to a folder,compressis not set. All files and folders undersourceare recursively uploaded.destination.keyis used as prefix.Upload a folder as a zip file:sourcepoints to a folder,compress: zip. The files and folders undersourceare compressed into a zip file and uploaded.destiation.keyis used as name of the zip file.Changes to the uploaded files are detected by comparing md5 hashsums, which are stored as metadata with the object in S3. Only changed files will be uploaded. If you try to manage files that were already uploaded some other way, on the first run they will be detected as changed (due to the missing hash value metadata) and reuploaded. For comparing hashsums for files to be uploaded in compressed form the tool will temporarily create the zip file to calculate the hashsum.For multi-file upload: Ifsourcedoes not end in a slash, the last part of the name will be used as base folder: Forsource: files/fooanddestination.key: bar, all files will under foo be uploaded asbar/foo/<filename>. In contrast ifsource: files/foo/, all files will be uploaded asbar/<filename>.To use your S3 bucket to serve files to apps and services you must configure it for anonymous read access. For security reasons you should restrict that access to the IP range of your cluster. See theAWS S3 documentationfor details.
Creating and configuring the bucket is outside the scope of this tool.To make sure a marathon app that uses s3 files is made aware of changes to the uploaded files, you should set thes3fileentity as anupdatedependency to theappentity. dcos-deploy will restart the app whenever the uploaded file changes.If you runapplywith--debugdcosdeploy will download already existing files from s3 and print the differences between the local and remote version in the unified diff format. So only use--debugfor textual files.
If a configuration key ends with base64, b64 or base_64 its value will be decoded for the diff.Task exectype: taskexecallows to execute commands inside tasks. This is primarily meant to trigger configuration reloads on services that can do some sort of hot-reload as to avoid restarting a service. It has the following specific options:task: task identifier to uniquely identify the task. Can be part of a task name. Required. Variables can be used.command: command to execute in the task. Required. Variables can be used.print: boolean. Wether to print the output of the executed command. Optional. Defaults to false.This has the same effect as runningdcos task exec <task> <command>.As dcos-deploy has no state it cannnot detect if this command has been run before. As such it will run this command every timeapplyis called. You must either make the command idempotent so that running it multiple times is possible without changing the result, or (recommended) add dependencies (with:update) to it and addwhen: dependencies-changed. That way the command will only be executed when one of its dependencies has been changed.Example for this is: Uploading configuration files to S3 and triggering a redownload of the files into the service by a command. Seeexamplesfor details.HttpCalltype: httpcallallows to make HTTP calls as part of deployments. This is similar totaskexecand can be used to upload configuration to services as part of deployments. It has the following specific options:url: HTTP(s) URL to call. Required. Variables can be used.method: HTTP method to use for the call, defaults toGET.ignore_errors: If set to true a non-200 result status code of the call will not be treated as a failure. Defaults to false.body: Allows to send a request body along. Optional. Has the following sub options:content: Specifies the content of the request body.file: Contents of this file will be used as request body. Is mutually exclusive withcontent. Variables can be used in the filename.render: If set to true the request body fromfileorcontentwill be rendered using mustache.headers: Allows to specify extra headers to send as key-value pairs (e.g.Content-Type). Optional. Variables can be used for both keys and values.If the URL is neither a http nor a https url it wil be treated as a path behind the DC/OS adminrouter. When executing the call dcos-deploy will use its DC/OS credentials to authenticate against the adminrouter. So for example if a DC/OS cluster is reachable underhttps://dcos.myclusterandurlis defined as/service/myservice/reloadthen dcos-deploy will call the URLhttps://dcos.mycluster/service/myservice/reload.The same restrictions fromtaskexecin regards to state apply. As such this entity will run every timeapplyis called unlesswhenand dependencies are used (see description fortaskexecabove for details).IAM Grouptype: iam_groupdefines a group in the DC/OS IAM system. It has the following options:name: Name of the group. Required. Variables can be used.description: Description of the group. Required. Variables can be used.permissions: Permissions to give to the group. Dictionary. Key is the name of the permission (rid), value is a list of actions to allow. If a permission does not yet exist, it will be created. Variables can be used. SeeServiceaccountfor an example.IAM Usertype: iam_userdefines a user in the DC/OS IAM system. It is similar to theserviceaccountbut represents a normal user with username and password. It has the following options:name: Name of the user. Required. Variables can be used.description: Description of the user. Is represented asFull namein the DC/OS Admin UI. Required. Variables can be used.update_password: Cleartext password to set for the user. Required. Variables can be used.update_password: If set to true will always overwrite the current user password with the one specified here. Defaults to true.groups: List of groups the user should be added to. Optional. Variables can be used.permissions: Permissions to give to the user. Dictionary. Key is the name of the permission (rid), value is a list of actions to allow. If a permission does not yet exist, it will be created. Variables can be used. SeeServiceaccountfor an example.Marathon Grouptype: marathon_groupallows to manage groups in marathon. For top-level groups in DC/OS>=2.0it can also manage group quotas.name: Name of the group. Required. Variables can be sued.enforce_role: Whether to enforce the group quota. Takes effect only for top-level groups. Defaults to False. In the DC/OS Admin UI values are seen asUse Group Role(True) andUse Legacy Role(False). Optional.quota: Optionalcpus: Maximum number of CPUs the role can use. Optional. Defaults to 0.mem: Maximum memory the role can use. In MB. Optional. Defaults to 0.disk: Maximum disk the role can use. In MB. Optional. Defaults to 0.gpus: Maximum number of GPUs the role can use. Optional. Defaults to 0.Restrictions:Removing quotas is currently not supported.Quotas are only supported for DC/OS>=2.0.Deployment processWhen running theapplycommand dcos-deploy will first check all entities if they have changed. To do this it will first render all options and files using the provided variables, retrieve the currently running configurations from the DC/OS cluster using the specific APIs (e.g. get the app definition from marathon) and compare them. It will print a list of changes and ask for confirmation (unless--yesis used). If an entity needs to be created it will first recursively create any dependencies.
There is no guaranteed order of execution. Only that any defined dependencies will be created before the entity itsself is created.The deployment process has some specific restrictions:Names/paths/ids may not be changed.Options files for frameworks, apps and jobs are not verified for structural correctness. If the new file is not accepted by the API an error will be thrown.For frameworks dcos-deploy does not verify beforehand if a version update is supported by the framework. If the framework does not accept the new version an error will be thrown.An already createdcertentity will not be changed if thednorhostnamesfields change.Dependencies must be explicitly defined in thedcos.yml. Implicit dependencies (like a secret referenced in a marathon app) that are not explicitly stated are not honored by dcos-deploy.Deleting entitiesdcos-deploy has support for deleting entities. You can use it to delete one or all entities defined (for example to clean up after tests). Do so use the commanddcos-deploy delete. It will delete all entities defined in your configuration, honoring the dependencies (e.g. deleting a service before deleting the secret associated with it). If you only want to delete a specific entity use--only <entity-name>. All entities that have this entity as a dependency will also be deleted (e.g. if you delete a secret a marathon app depending on it will also be deleted). Check the dry-run output to make sure you don't unintentionally delete the wrong entity. The command is idempotent, so deleting an already deleted entity has no effect.
The delete command will not modify your configuration files. So to make sure that the deleted entity will not be recreated during the nextapply-run, remove the entity definition from your yaml files.RoadmapSupport for creating and configuring kubernetes clustersAdd as package to the Mesosphere universe for easy installation as a module for the dcos-cliBetter and documented plugin supportProvide some sort of verification for configurationsDevelopmentIf you want to develop dcos-deploy you need to be able to run it directly from source. This can be done in two ways:Clone this github repository to your system and runpip install -eManually install all dependencies fromsetup.pyand create a symlink ofdcos-deployto a folder in your exectuable path (e.g.ln -s $(pwd)/dcos-deploy ~/usr/bin/dcos-deploy)This project contains unittests. For convenience they can be run usingmake test.Release processCheck if any of the dependencies insetup.pyneed to be updatedSet the correct version indcosdeploy/__init__.pyPush changes and download the built binaries from the github actions pipelinePush the new release to pypi by runningmake release-pypiYou need the dependencies fromrequirements-dev.txtto run this commandYou need to have credentials configured for pypi (under linux at~/.pypirc)Create a new release with a tag corresponding to the version in the formatvX.Y.ZCollect notable new features and write a release textAttach binariesChange the version indcosdeploy/__init__.pyto<next-version>-devand pushContributingIf you found a bug or have a feature request, please open an issue in Github.