neulab/tldr huggingface dataset:
The rendered output is the result of the Jinja parsing and is a good way to view the result of referencing Jinja variables. If the result is empty then Jinja produced an empty result and the Reactor will ignore it.
Passing Event Data to Minions or Orchestration as Pillar An interesting trick to pass data from the Reactor SLS file to state.apply is to pass it as inline Pillar data since both functions take a keyword argument named pillar.
The following example uses Salt's Reactor to listen for the event that is fired when the key for a new minion is accepted on the master using salt-key.
/etc/salt/master.d/reactor.conf:
reactor: - 'salt/key': - /srv/salt/haproxy/react_new_minion.sls
The Reactor then fires a :state.apply command targeted to the HAProxy servers and passes the ID of the new minion from the event to the state file via inline Pillar.
/srv/salt/haproxy/react_new_minion.sls:
{% if data['act'] == 'accept' and data['id'].startswith('web') %} add_new_minion_to_pool: local.state.apply: - tgt: 'haproxy*' - args: - mods: haproxy.refresh_pool - pillar: new_minion: {{ data['id'] }} {% endif %}
The above command is equivalent to the following command at the CLI:
salt 'haproxy*' state.apply haproxy.refresh_pool pillar='{new_minion: minionid}'
This works with Orchestrate files as well:
call_some_orchestrate_file: runner.state.orchestrate: - args: - mods: orchestrate.some_orchestrate_file - pillar: stuff: things
Which is equivalent to the following command at the CLI:
salt-run state.orchestrate orchestrate.some_orchestrate_file pillar='{stuff: things}'
Finally, that data is available in the state file using the normal Pillar lookup syntax. The following example is grabbing web server names and IP addresses from Salt Mine. If this state is invoked from the Reactor then the custom Pillar value from above will be available and the new minion will be added to the pool but with the disabled flag so that HAProxy won't yet direct traffic to it.
/srv/salt/haproxy/refresh_pool.sls:
{% set new_minion = salt['pillar.get']('new_minion') %}
listen web *:80 balance source {% for server,ip in salt['mine.get']('web*', 'network.interfaces', ['eth0']).items() %} {% if server == new_minion %} server {{ server }} {{ ip }}:80 disabled {% else %} server {{ server }} {{ ip }}:80 check {% endif %} {% endfor %}
A Complete Example In this example, we're going to assume that we have a group of servers that will come online at random and need to have keys automatically accepted. We'll also add that we don't want all servers being automatically accepted. For this example, we'll assume that all hosts that have an id that starts with 'ink' will be automatically accepted and have state.apply executed. On top of this, we're going to add that a host coming up that was replaced (meaning a new key) will also be accepted.
Our master configuration will be rather simple. All minions that attempte to authenticate will match the tag of salt/auth. When it comes to the minion key being accepted, we get a more refined tag that includes the minion id, which we can use for matching.
reactor: - 'salt/auth': - /srv/reactor/auth-pending.sls - 'salt/minion/ink*/start': - /srv/reactor/auth-complete.sls
In this SLS file, we say that if the key was rejected we will delete the key on the master and then also tell the master to ssh in to the minion and tell it to restart the minion, since a minion process will die if the key is rejected.
We also say that if the key is pending and the id starts with ink we will accept the key. A minion that is waiting on a pending key will retry authentication every ten seconds by default.
/srv/reactor/auth-pending.sls:
{# Ink server failed to authenticate -- remove accepted key #} {% if not data['result'] and data['id'].startswith('ink') %} minion_remove: wheel.key.delete: - args: - match: {{ data['id'] }} minion_rejoin: local.cmd.run: - tgt: salt-master.domain.tld - args: - cmd: ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no "{{ data['id'] }}" 'sleep 10 && /etc/init.d/salt-minion restart' {% endif %}
{# Ink server is sending new key -- accept this key #} {% if 'act' in data and data['act'] == 'pend' and data['id'].startswith('ink') %} minion_add: wheel.key.accept: - args: - match: {{ data['id'] }} {% endif %}
No if statements are needed here because we already limited this action to just Ink servers in the master configuration.
/srv/reactor/auth-complete.sls:
{# When an Ink server connects, run state.apply. #} highstate_run: local.state.apply: - tgt: {{ data['id'] }} - ret: smtp
The above will also return the highstate result data using the smtp_return returner (use virtualname like when using from the command line with --return).  The returner needs to be configured on the minion for this to work.  See salt.returners.smtp_return documentation for that.
Syncing Custom Types on Minion Start Salt will sync all custom types (by running a saltutil.sync_all) on every highstate. However, there is a chicken-and-egg issue where, on the initial highstate, a minion will not yet have these custom types synced when the top file is first compiled. This can be worked around with a simple reactor which watches for salt/minion/*/start events, which each minion fires when it first starts up and connects to the master.
On the master, create /srv/reactor/sync_grains.sls with the following contents:
sync_grains: local.saltutil.sync_grains: - tgt: {{ data['id'] }}
And in the master config file, add the following reactor configuration:
reactor: - 'salt/minion/*/start': - /srv/reactor/sync_grains.sls
This will cause the master to instruct each minion to sync its custom grains when it starts, making these grains available when the initial highstate is executed.
Other types can be synced by replacing local.saltutil.sync_grains with local.saltutil.sync_modules, local.saltutil.sync_all, or whatever else suits the intended use case.
Also, if it is not desirable that every minion syncs on startup, the * can be replaced with a different glob to narrow down the set of minions which will match that reactor (e.g. salt/minion/appsrv*/start, which would only match minion IDs beginning with appsrv).
Reactor Tuning for Large-Scale Installations The reactor uses a thread pool implementation that's contained inside salt.utils.process.ThreadPool. It uses Python's stdlib Queue to enqueue jobs which are picked up by standard Python threads. If the queue is full, False is simply returned by the firing method on the thread pool.
As such, there are a few things to say about the selection of proper values for the reactor.
For situations where it is expected that many long-running jobs might be executed by the reactor, reactor_worker_hwm should be increased or even set to 0 to bound it only by available memory. If set to zero, a close eye should be kept on memory consumption.
If many long-running jobs are expected and execution concurrency and performance are a concern, you may also increase the value for reactor_worker_threads. This will control the number of concurrent threads which are pulling jobs from the queue and executing them. Obviously, this bears a relationship to the speed at which the queue itself will fill up.  The price to pay for this value is that each thread will contain a copy of Salt code needed to perform the requested action.
Orchestrate Runner Executing states or highstate on a minion is perfect when you want to ensure that minion configured and running the way you want. Sometimes however you want to configure a set of minions all at once.
For example, if you want to set up a load balancer in front of a cluster of web servers you can ensure the load balancer is set up first, and then the same matching configuration is applied consistently across the whole cluster.
Orchestration is the way to do this.
The Orchestrate Runner New in version 0.17.0.
NOTE: Orchestrate Deprecates OverState
The Orchestrate Runner (originally called the state.sls runner) offers all the functionality of the OverState, but with some advantages:
• All requisites available in states can be used.
• The states/functions will also work on salt-ssh minions.
The Orchestrate Runner replaced the OverState system in Salt 2015.8.0.
The orchestrate runner generalizes the Salt state system to a Salt master context.  Whereas the state.sls, state.highstate, et al. functions are concurrently and independently executed on each Salt minion, the state.orchestrate runner is executed on the master, giving it a master-level view and control over requisites, such as state ordering and conditionals.  This allows for inter minion requisites, like ordering the application of states on different minions that must not happen simultaneously, or for halting the state run on all minions if a minion fails one of its states.
The state.sls, state.highstate, et al. functions allow you to statefully manage each minion and the state.orchestrate runner allows you to statefully manage your entire infrastructure.
Writing SLS Files Orchestrate SLS files are stored in the same location as State SLS files. This means that both file_roots and gitfs_remotes impact what SLS files are available to the reactor and orchestrator.
It is recommended to keep reactor and orchestrator SLS files in their own uniquely named subdirectories such as _orch/, orch/, _orchestrate/, react/, _reactor/, etc. This will avoid duplicate naming and will help prevent confusion.
Executing the Orchestrate Runner The Orchestrate Runner command format is the same as for the state.sls function, except that since it is a runner, it is executed with salt-run rather than salt.  Assuming you have a state.sls file called /srv/salt/orch/webserver.sls the following command, run on the master, will apply the states defined in that file.
salt-run state.orchestrate orch.webserver
NOTE: state.orch is a synonym for state.orchestrate
Changed in version 2014.1.1: The runner function was renamed to state.orchestrate to avoid confusion with the state.sls execution function. In versions 0.17.0 through 2014.1.0, state.sls must be used.
Masterless Orchestration New in version 2016.11.0.
To support salt orchestration on masterless minions, the Orchestrate Runner is available as an execution module. The syntax for masterless orchestration is exactly the same, but it uses the salt-call command and the minion configuration must contain the file_mode: local option. Alternatively, use salt-call --local on the command line.
salt-call --local state.orchestrate orch.webserver
NOTE: Masterless orchestration supports only the salt.state command in an sls file; it does not (currently) support the salt.function command.
Examples Function To execute a function, use salt.function:
# /srv/salt/orch/cleanfoo.sls cmd.run: salt.function: - tgt: '*' - arg: - rm -rf /tmp/foo
salt-run state.orchestrate orch.cleanfoo
If you omit the "name" argument, the ID of the state will be the default name, or in the case of salt.function, the execution module function to run. You can specify the "name" argument to avoid conflicting IDs:
copy_some_file: salt.function: - name: file.copy - tgt: '*' - arg: - /path/to/file - /tmp/copy_of_file - kwarg: remove_existing: true
Fail Functions When running a remote execution function in orchestration, certain return values for those functions may indicate failure, while the function itself doesn't set a return code. For those circumstances, using a "fail function" allows for a more flexible means of assessing success or failure.
A fail function can be written as part of a custom execution module. The function should accept one argument, and return a boolean result. For example:
def check_func_result(retval): if some_condition: return True else: return False
The function can then be referenced in orchestration SLS like so:
do_stuff: salt.function: - name: modname.funcname - tgt: '*' - fail_function: mymod.check_func_result
IMPORTANT: Fail functions run on the master, so they must be synced using salt-run saltutil.sync_modules.
State To execute a state, use salt.state.
# /srv/salt/orch/webserver.sls install_nginx: salt.state: - tgt: 'web*' - sls: - nginx
Highstate To run a highstate, set highstate: True in your state config:
# /srv/salt/orch/web_setup.sls webserver_setup: salt.state: - tgt: 'web*' - highstate: True
salt-run state.orchestrate orch.web_setup
Runner To execute another runner, use salt.runner.  For example to use the cloud.profile runner in your orchestration state additional options to replace values in the configured profile, use this:
# /srv/salt/orch/deploy.sls create_instance: salt.runner: - name: cloud.profile - prof: cloud-centos - provider: cloud - instances: - server1 - opts: minion: master: master1
To get a more dynamic state, use jinja variables together with inline pillar data.  Using the same example but passing on pillar data, the state would be like this.
# /srv/salt/orch/deploy.sls {% set servers = salt['pillar.get']('servers', 'test') %} {% set master = salt['pillar.get']('master', 'salt') %} create_instance: salt.runner: - name: cloud.profile - prof: cloud-centos - provider: cloud - instances: - {{ servers }} - opts: minion: master: {{ master }}
To execute with pillar data.
salt-run state.orch orch.deploy pillar='{"servers": "newsystem1", "master": "mymaster"}'
Return Codes in Runner/Wheel Jobs New in version 2018.3.0.
State (salt.state) jobs are able to report failure via the state return dictionary. Remote execution (salt.function) jobs are able to report failure by setting a retcode key in the __context__ dictionary. However, runner (salt.runner) and wheel (salt.wheel) jobs would only report a False result when the runner/wheel function raised an exception. As of the 2018.3.0 release, it is now possible to set a retcode in runner and wheel functions just as you can do in remote execution functions. Here is some example pseudocode:
def myrunner(): ... # do stuff ... if some_error_condition: __context__["retcode"] = 1 return result
This allows a custom runner/wheel function to report its failure so that requisites can accurately tell that a job has failed.
More Complex Orchestration Many states/functions can be configured in a single file, which when combined with the full suite of requisites, can be used to easily configure complex orchestration tasks. Additionally, the states/functions will be executed in the order in which they are defined, unless prevented from doing so by any requisites, as is the default in SLS files since 0.17.0.
bootstrap_servers: salt.function: - name: cmd.run - tgt: 10.0.0.0/24 - tgt_type: ipcidr - arg: - bootstrap
storage_setup: salt.state: - tgt: 'role:storage' - tgt_type: grain - sls: ceph - require: - salt: webserver_setup
webserver_setup: salt.state: - tgt: 'web*' - highstate: True
Given the above setup, the orchestration will be carried out as follows:
1. The shell command bootstrap will be executed on all minions in the 10.0.0.0/24 subnet.
2. A Highstate will be run on all minions whose ID starts with "web", since the storage_setup state requires it.
3. Finally, the ceph SLS target will be executed on all minions which have a grain called role with a value of storage.
NOTE: Remember, salt-run is always executed on the master.
Parsing Results Programmatically Orchestration jobs return output in a specific data structure. That data structure is represented differently depending on the outputter used. With the default outputter for orchestration, you get a nice human-readable output.  Assume the following orchestration SLS:
good_state: salt.state: - tgt: myminion - sls: - succeed_with_changes
bad_state: salt.state: - tgt: myminion - sls: - fail_with_changes
mymod.myfunc: salt.function: - tgt: myminion
mymod.myfunc_false_result: salt.function: - tgt: myminion
Running this using the default outputter would produce output which looks like this:
fa5944a73aa8_master: ---------- ID: good_state Function: salt.state Result: True Comment: States ran successfully. Updating myminion. Started: 21:08:02.681604 Duration: 265.565 ms Changes: myminion: ---------- ID: test succeed with changes Function: test.succeed_with_changes Result: True Comment: Success! Started: 21:08:02.835893 Duration: 0.375 ms Changes: ---------- testing: ---------- new: Something pretended to change old: Unchanged
Summary for myminion ------------ Succeeded: 1 (changed=1) Failed:    0 ------------ Total states run:     1 Total run time:   0.375 ms ---------- ID: bad_state Function: salt.state Result: False Comment: Run failed on minions: myminion Started: 21:08:02.947702 Duration: 177.01 ms Changes: myminion: ---------- ID: test fail with changes Function: test.fail_with_changes Result: False Comment: Failure! Started: 21:08:03.116634 Duration: 0.502 ms Changes: ---------- testing: ---------- new: Something pretended to change old: Unchanged
Summary for myminion ------------ Succeeded: 0 (changed=1) Failed:    1 ------------ Total states run:     1 Total run time:   0.502 ms ---------- ID: mymod.myfunc Function: salt.function Result: True Comment: Function ran successfully. Function mymod.myfunc ran on myminion. Started: 21:08:03.125011 Duration: 159.488 ms Changes: myminion: True ---------- ID: mymod.myfunc_false_result Function: salt.function Result: False Comment: Running function mymod.myfunc_false_result failed on minions: myminion. Function mymod.myfunc_false_result ran on myminion. Started: 21:08:03.285148 Duration: 176.787 ms Changes: myminion: False
Summary for fa5944a73aa8_master ------------ Succeeded: 2 (changed=4) Failed:    2 ------------ Total states run:     4 Total run time: 778.850 ms
However, using the json outputter, you can get the output in an easily loadable and parsable format:
salt-run state.orchestrate test --out=json
{ "outputter": "highstate", "data": { "fa5944a73aa8_master": { "salt_|-good_state_|-good_state_|-state": { "comment": "States ran successfully. Updating myminion.", "name": "good_state", "start_time": "21:35:16.868345", "result": true, "duration": 267.299, "__run_num__": 0, "__jid__": "20171130213516897392", "__sls__": "test", "changes": { "ret": { "myminion": { "test_|-test succeed with changes_|-test succeed with changes_|-succeed_with_changes": { "comment": "Success!", "name": "test succeed with changes", "start_time": "21:35:17.022592", "result": true, "duration": 0.362, "__run_num__": 0, "__sls__": "succeed_with_changes", "changes": { "testing": { "new": "Something pretended to change", "old": "Unchanged" } }, "__id__": "test succeed with changes" } } }, "out": "highstate" }, "__id__": "good_state" }, "salt_|-bad_state_|-bad_state_|-state": { "comment": "Run failed on minions: test", "name": "bad_state", "start_time": "21:35:17.136511", "result": false, "duration": 197.635, "__run_num__": 1, "__jid__": "20171130213517202203", "__sls__": "test", "changes": { "ret": { "myminion": { "test_|-test fail with changes_|-test fail with changes_|-fail_with_changes": { "comment": "Failure!", "name": "test fail with changes", "start_time": "21:35:17.326268", "result": false, "duration": 0.509, "__run_num__": 0, "__sls__": "fail_with_changes", "changes": { "testing": { "new": "Something pretended to change", "old": "Unchanged" } }, "__id__": "test fail with changes" } } }, "out": "highstate" }, "__id__": "bad_state" }, "salt_|-mymod.myfunc_|-mymod.myfunc_|-function": { "comment": "Function ran successfully. Function mymod.myfunc ran on myminion.", "name": "mymod.myfunc", "start_time": "21:35:17.334373", "result": true, "duration": 151.716, "__run_num__": 2, "__jid__": "20171130213517361706", "__sls__": "test", "changes": { "ret": { "myminion": true }, "out": "highstate" }, "__id__": "mymod.myfunc" }, "salt_|-mymod.myfunc_false_result-mymod.myfunc_false_result-function": { "comment": "Running function mymod.myfunc_false_result failed on minions: myminion. Function mymod.myfunc_false_result ran on myminion.", "name": "mymod.myfunc_false_result", "start_time": "21:35:17.486625", "result": false, "duration": 174.241, "__run_num__": 3, "__jid__": "20171130213517536270", "__sls__": "test", "changes": { "ret": { "myminion": false }, "out": "highstate" }, "__id__": "mymod.myfunc_false_result" } } }, "retcode": 1 }
The 2018.3.0 release includes a couple fixes to make parsing this data easier and more accurate. The first is the ability to set a return code in a custom runner or wheel function, as noted above. The second is a change to how failures are included in the return data. Prior to the 2018.3.0 release, minions that failed a salt.state orchestration job would show up in the comment field of the return data, in a human-readable string that was not easily parsed. They are now included in the changes dictionary alongside the minions that succeeded. In addition, salt.function jobs which failed because the fail function returned False used to handle their failures in the same way salt.state jobs did, and this has likewise been corrected.
Running States on the Master without a Minion The orchestrate runner can be used to execute states on the master without using a minion. For example, assume that salt://foo.sls contains the following SLS:
/etc/foo.conf: file.managed: - source: salt://files/foo.conf - mode: 0600
In this case, running salt-run state.orchestrate foo would be the equivalent of running a state.sls foo, but it would execute on the master only, and would not require a minion daemon to be running on the master.
This is not technically orchestration, but it can be useful in certain use cases.
Limitations Only one SLS target can be run at a time using this method, while using state.sls allows for multiple SLS files to be passed in a comma-separated list.
This section contains details on Solaris specific quirks and workarounds.
NOTE: Solaris refers to both Solaris 10 compatible platforms like Solaris 10, illumos, SmartOS, OmniOS, OpenIndiana,... and Oracle Solaris 11 platforms.
Solaris-specific Behaviour Salt is capable of managing Solaris systems, however due to various differences between the operating systems, there are some things you need to keep in mind.
This document will contain any quirks that apply across Salt or limitations in some modules.
FQDN/UQDN On Solaris platforms the FQDN will not always be properly detected.  If an IPv6 address is configured pythons `socket.getfqdn()` fails to return a FQDN and returns the nodename instead. For a full breakdown see the following issue on github: #37027
Grains Not all grains are available or some have empty or 0 as value. Mostly grains that are depenend on hardware discovery like: - num_gpus - gpus
Also some resolver related grains like: - domain - dns:options - dns:sortlist
Getting Started Salt SSH is very easy to use, simply set up a basic roster file of the systems to connect to and run salt-ssh commands in a similar way as standard salt commands.
• Salt ssh is considered production ready in version 2014.7.0
• Python is required on the remote system (unless using the -r option to send raw ssh commands).  The python version requirement is the same as that for a standard Salt installation.
• On many systems, the salt-ssh executable will be in its own package, usually named salt-ssh
• The Salt SSH system does not supersede the standard Salt communication systems, it simply offers an SSH-based alternative that does not require ZeroMQ and a remote agent. Be aware that since all communication with Salt SSH is executed via SSH it is substantially slower than standard Salt with ZeroMQ.
• At the moment fileserver operations must be wrapped to ensure that the relevant files are delivered with the salt-ssh commands.  The state module is an exception, which compiles the state run on the master, and in the process finds all the references to salt:// paths and copies those files down in the same tarball as the state run.  However, needed fileserver wrappers are still under development.
Salt SSH Roster The roster system in Salt allows for remote minions to be easily defined.
NOTE: See the SSH roster docs for more details.
Simply create the roster file, the default location is /etc/salt/roster:
web1: 192.168.42.1
This is a very basic roster file where a Salt ID is being assigned to an IP address. A more elaborate roster can be created:
web1: host: 192.168.42.1 # The IP addr or DNS hostname user: fred         # Remote executions will be executed as user fred passwd: foobarbaz  # The password to use for login, if omitted, keys are used sudo: True         # Whether to sudo to root, not enabled by default web2: host: 192.168.42.2
NOTE: sudo works only if NOPASSWD is set for user in /etc/sudoers: fred ALL=(ALL) NOPASSWD: ALL
Deploy ssh key for salt-ssh By default, salt-ssh will generate key pairs for ssh, the default path will be /etc/salt/pki/master/ssh/salt-ssh.rsa. The key generation happens when you run salt-ssh for the first time.
You can use ssh-copy-id, (the OpenSSH key deployment tool) to deploy keys to your servers.
ssh-copy-id -i /etc/salt/pki/master/ssh/salt-ssh.rsa.pub user@server.demo.com
One could also create a simple shell script, named salt-ssh-copy-id.sh as follows:
#!/bin/bash if [ -z $1 ]; then echo $0 user@host.com exit 0 fi ssh-copy-id -i /etc/salt/pki/master/ssh/salt-ssh.rsa.pub $1
NOTE: Be certain to chmod +x salt-ssh-copy-id.sh.
./salt-ssh-copy-id.sh user@server1.host.com ./salt-ssh-copy-id.sh user@server2.host.com
Once keys are successfully deployed, salt-ssh can be used to control them.
Alternatively ssh agent forwarding can be used by setting the priv to agent-forwarding.
Calling Salt SSH NOTE: salt-ssh on target hosts without Python 3
The salt-ssh command requires at least python 3, which is not installed by default on some target hosts.  An easy workaround in this situation is to use the -r option to run a raw shell command that installs python26:
salt-ssh centos-5-minion -r 'yum -y install epel-release ; yum -y install python26'
NOTE: salt-ssh on systems with Python 3.x
Salt, before the 2017.7.0 release, does not support Python 3.x which is the default on for example the popular 16.04 LTS release of Ubuntu. An easy workaround for this scenario is to use the -r option similar to the example above:
salt-ssh ubuntu-1604-minion -r 'apt update ; apt install -y python-minimal'
The salt-ssh command can be easily executed in the same way as a salt command:
salt-ssh '*' test.version
Commands with salt-ssh follow the same syntax as the salt command.
The standard salt functions are available! The output is the same as salt and many of the same flags are available. Please see Salt SSH reference for all of the available options.
Raw Shell Calls By default salt-ssh runs Salt execution modules on the remote system, but salt-ssh can also execute raw shell commands:
salt-ssh '*' -r 'ifconfig'
States Via Salt SSH The Salt State system can also be used with salt-ssh. The state system abstracts the same interface to the user in salt-ssh as it does when using standard salt. The intent is that Salt Formulas defined for standard salt will work seamlessly with salt-ssh and vice-versa.
The standard Salt States walkthroughs function by simply replacing salt commands with salt-ssh.
Targeting with Salt SSH Due to the fact that the targeting approach differs in salt-ssh, only glob and regex targets are supported as of this writing, the remaining target systems still need to be implemented.
NOTE: By default, Grains are settable through salt-ssh. By default, these grains will not be persisted across reboots.
See the "thin_dir" setting in Roster documentation for more details.
Configuring Salt SSH Salt SSH takes its configuration from a master configuration file. Normally, this file is in /etc/salt/master. If one wishes to use a customized configuration file, the -c option to Salt SSH facilitates passing in a directory to look inside for a configuration file named master.
Minion Config New in version 2015.5.1.
Minion config options can be defined globally using the master configuration option ssh_minion_opts. It can also be defined on a per-minion basis with the minion_opts entry in the roster.
Running Salt SSH as non-root user By default, Salt read all the configuration from /etc/salt/. If you are running Salt SSH with a regular user you have to modify some paths or you will get "Permission denied" messages. You have to modify two parameters: pki_dir and cachedir. Those should point to a full path writable for the user.
It's recommended not to modify /etc/salt for this purpose. Create a private copy of /etc/salt for the user and run the command with -c /new/config/path.
Define CLI Options with Saltfile If you are commonly passing in CLI options to salt-ssh, you can create a Saltfile to automatically use these options. This is common if you're managing several different salt projects on the same server.
So you can cd into a directory that has a Saltfile with the following YAML contents:
salt-ssh: config_dir: path/to/config/dir ssh_log_file: salt-ssh.log ssh_max_procs: 30 ssh_wipe: True
Instead of having to call salt-ssh --config-dir=path/to/config/dir --max-procs=30 --wipe \* test.version you can call salt-ssh \* test.version.
Boolean-style options should be specified in their YAML representation.
NOTE: The option keys specified must match the destination attributes for the options specified in the parser salt.utils.parsers.SaltSSHOptionParser.  For example, in the case of the --wipe command line option, its dest is configured to be ssh_wipe and thus this is what should be configured in the Saltfile.  Using the names of flags for this option, being wipe: True or w: True, will not work.
NOTE: For the Saltfile to be automatically detected it needs to be named Saltfile with a capital S and be readable by the user running salt-ssh.
At last you can create ~/.salt/Saltfile and salt-ssh will automatically load it by default.
Advanced options with salt-ssh Salt's ability to allow users to have custom grains and custom modules is also applicable to using salt-ssh. This is done through first packing the custom grains into the thin tarball before it is deployed on the system.
For this to happen, the config file must be explicit enough to indicate where the custom grains are located on the machine like so:
file_client: local file_roots: base: - /home/user/.salt - /home/user/.salt/_states - /home/user/.salt/_grains module_dirs: - /home/user/.salt pillar_roots: base: - /home/user/.salt/_pillar root_dir: /tmp/.salt-root
It's better to be explicit rather than implicit in this situation. This will allow urls all under salt:// to be resolved such as salt://_grains/custom_grain.py.
One can confirm this action by executing a properly setup salt-ssh minion with salt-ssh minion grains.items. During this process, a saltutil.sync_all is ran to discover the thin tarball and then consumed. Output similar to this indicates a successful sync with custom grains.
local: ---------- ... executors: grains: - grains.custom_grain log_handlers: ...
This is especially important when using a custom file_roots that differ from /etc/salt/.
NOTE: Please see https://docs.saltproject.io/en/latest/topics/grains/ for more information on grains and custom grains.
Debugging salt-ssh One common approach for debugging salt-ssh is to simply use the tarball that salt ships to the remote machine and call salt-call directly.
To determine the location of salt-call, simply run salt-ssh with the -ltrace flag and look for a line containing the string, SALT_ARGV. This contains the salt-call command that salt-ssh attempted to execute.
It is recommended that one modify this command a bit by removing the -l quiet, --metadata and --output json to get a better idea of what's going on the target system.
Salt Rosters Salt rosters are pluggable systems added in Salt 0.17.0 to facilitate the salt-ssh system.  The roster system was created because salt-ssh needs a means to identify which systems need to be targeted for execution.
SEE ALSO: all-salt.roster
NOTE: The Roster System is not needed or used in standard Salt because the master does not need to be initially aware of target systems, since the Salt Minion checks itself into the master.
Since the roster system is pluggable, it can be easily augmented to attach to any existing systems to gather information about what servers are presently available and should be attached to by salt-ssh. By default the roster file is located at /etc/salt/roster.
How Rosters Work The roster system compiles a data structure internally referred to as targets. The targets is a list of target systems and attributes about how to connect to said systems. The only requirement for a roster module in Salt is to return the targets data structure.
Targets Data The information which can be stored in a roster target is the following:
<Salt ID>:       # The id to reference the target system with host:        # The IP address or DNS name of the remote host user:        # The user to log in as passwd:      # The password to log in with
# Optional parameters port:        # The target system's ssh port number sudo:        # Boolean to run command via sudo sudo_user:   # Str: Set this to execute Salt as a sudo user other than root. # This user must be in the same system group as the remote user # that is used to login and is specified above. Alternatively, # the user must be a super-user. tty:         # Boolean: Set this option to True if sudo is also set to # True and requiretty is also set on the target system priv:        # File path to ssh private key, defaults to salt-ssh.rsa # The priv can also be set to agent-forwarding to not specify # a key, but use ssh agent forwarding priv_passwd: # Passphrase for ssh private key timeout:     # Number of seconds to wait for response when establishing # an SSH connection minion_opts: # Dictionary of minion opts thin_dir:    # The target system's storage directory for Salt # components. Defaults to /tmp/salt-<hash>. cmd_umask:   # umask to enforce for the salt-call command. Should be in # octal (so for 0o077 in YAML you would do 0077, or 63) ssh_pre_flight: # Path to a script that will run before all other salt-ssh # commands. Will only run the first time when the thin dir # does not exist, unless --pre-flight is passed to salt-ssh # command or ssh_run_pre_flight is set to true in the config # Added in 3001 Release. set_path:    # Set the path environment variable, to ensure the expected python # binary is in the salt-ssh path, when running the command. # Example: '$PATH:/usr/local/bin/'. Added in 3001 Release. ssh_options: # List of options (as 'option=argument') to pass to ssh.
ssh_pre_flight A Salt-SSH roster option ssh_pre_flight was added in the 3001 release. This enables you to run a script before Salt-SSH tries to run any commands. You can set this option in the roster for a specific minion or use the roster_defaults to set it for all minions.  This script will only run if the thin dir is not currently on the minion. This means it will only run on the first run of salt-ssh or if you have recently wiped out your thin dir. If you want to intentionally run the script again you have a couple of options:
• Wipe out your thin dir by using the -w salt-ssh arg.
• Set ssh_run_pre_flight to True in the config
• Run salt-ssh with the --pre-flight arg.
Target Defaults The roster_defaults dictionary in the master config is used to set the default login variables for minions in the roster so that the same arguments do not need to be passed with commandline arguments.
thin_dir Salt needs to upload a standalone environment to the target system, and this defaults to /tmp/salt-<hash>. This directory will be cleaned up per normal systems operation.
If you need a persistent Salt environment, for instance to set persistent grains, this value will need to be changed.
SSH Ext Alternatives In the 2019.2.0 release the ssh_ext_alternatives feature was added.  This allows salt-ssh to work across different supported python versions. You will need to ensure you have the following:
• Salt is installed, with all required dependnecies for the Python version.
• Everything needs to be importable from the respective Python environment.
To enable using this feature you will need to edit the master configuration similar to below:
ssh_ext_alternatives: 2019.2:                     # Namespace, can be anything. py-version: [2, 7]      # Constraint to specific interpreter version path: /opt/2019.2/salt  # Main Salt installation directory. dependencies:           # List of dependencies and their installation paths jinja2: /opt/jinja2 yaml: /opt/yaml tornado: /opt/tornado msgpack: /opt/msgpack certifi: /opt/certifi singledispatch: /opt/singledispatch.py singledispatch_helpers: /opt/singledispatch_helpers.py markupsafe: /opt/markupsafe backports_abc: /opt/backports_abc.py
WARNING: When using Salt versions >= 3001 and Python 2 is your py-version you need to use an older version of Salt that supports Python 2.  For example, if using Salt-SSH version 3001 and you do not want to install Python 3 on your target host you can use ssh_ext_alternatives's path option. This option needs to point to a 2019.2.3 Salt installation directory on your Salt-SSH host, which still supports Python 2.
auto_detect In the 3001 release the auto_detect feature was added for ssh_ext_alternatives.  This allows salt-ssh to automatically detect the path to all of your dependencies and does not require you to define them under dependencies.
ssh_ext_alternatives: 2019.2:                     # Namespace, can be anything. py-version: [2, 7]      # Constraint to specific interpreter version path: /opt/2019.2/salt  # Main Salt installation directory. auto_detect: True       # Auto detect dependencies py_bin: /usr/bin/python2.7 # Python binary path used to auto detect dependencies
If py_bin is not set alongside auto_detect, it will attempt to auto detect the dependnecies using the major version set in py-version. For example if you have [2, 7] set as your py-version, it will attempt to use the binary python2.
You can also use auto_detect and dependencies together.
ssh_ext_alternatives: 2019.2:                     # Namespace, can be anything. py-version: [2, 7]      # Constraint to specific interpreter version path: /opt/2019.2/salt  # Main Salt installation directory. auto_detect: True       # Auto detect dependencies py_bin: /usr/bin/python2.7 # Python binary path to auto detect dependencies dependencies:           # List of dependencies and their installation paths jinja2: /opt/jinja2
If a dependency is defined in the dependecies list ssh_ext_alternatives will use this dependency, instead of the path that auto_detect finds. For example, if you define /opt/jinja2 under your dependencies for jinja2, it will not try to autodetect the file path to the jinja2 module, and will favor /opt/jinja2.
Different Python Versions The 3001 release removed python 2 support in Salt. Even though this python 2 support is being dropped we have provided multiple ways to work around this with Salt-SSH. You can use the following options:
• ssh_pre_flight
• Using the Salt-SSH raw shell calls to install Python3.
• Use an older version of Salt on the target host that still supports Python 2 using the feature SSH ext alternatives
The original Salt Reactor is based on the idea of listening for a specific event and then reacting to it. This model comes with many logical limitations, for instance it is very difficult (and hacky) to fire a reaction based on aggregate data or based on multiple events.
The Thorium reactor is intended to alleviate this problem in a very elegant way.  Instead of using extensive jinja routines or complex python sls files the aggregation of data and the determination of what should run becomes isolated to the sls data logic, makes the definitions much cleaner.
Starting the Thorium Engine To enable the thorium engine add the following configuration to the engines section of your Salt Master or Minion configuration file and restart the daemon:
engines: - thorium: {}
Thorium Modules Because of its specialized nature, Thorium uses its own set of modules. However, many of these modules are designed to wrap the more commonly-used Salt subsystems. These modules are:
• local: Execution modules
• runner: Runner modules
• wheel: Wheel modules
There are other modules that ship with Thorium as well. Some of these will be highlighted later in this document.
Writing Thorium Formulas Like some other Salt subsystems, Thorium uses its own directory structure. The default location for this structure is /srv/thorium/, but it can be changed using the thorium_roots setting in the master configuration file.
This would explicitly set the roots to the default:
thorium_roots: base: - /srv/thorium
Example thorium_roots configuration:
thorium_roots: base: - /etc/salt/thorium
It is also possible to use gitfs with Thorium, using the thoriumenv or thorium_top settings.
Example using thorium_top:
thorium_top: salt://thorium/top.sls gitfs_provider: pygit2
gitfs_remotes: - git@github.com:user/repo.git: - name: salt-backend - root: salt - base: master - git@github.com:user/repo.git: - name: thorium-backend - root: thorium - base: master - mountpoint: salt://thorium
NOTE: When using this method don't forget to prepend the mountpoint to files served by this repo, for example top.sls:
base: '*': - thorium.key_clean
Example using thoriumenv:
thoriumenv: thorium gitfs_provider: pygit2
gitfs_remotes: - git@github.com:user/repo.git: - name: salt-backend - root: salt - base: master - git@github.com:user/repo.git: - name: thorium-backend - root: thorium - saltenv: - thorium: - ref: master
NOTE: When using this method all state will run under the defined environment, for example top.sls:
thorium: '*': - key_clean
The Thorium top.sls File Thorium uses its own top.sls file, which follows the same convention as is found in /srv/salt/:
<srv>: <target>: - <formula 1> - <formula 2> - <etc...>
For instance, a top.sls using a standard base environment and a single Thorium formula called key_clean, would look like:
base: '*': - key_clean
Take note that the target in a Thorium top.sls is not used; it only exists to follow the same convention as other top.sls files. Leave this set to '*' in your own Thorium top.sls.
Thorium Formula Files Thorium SLS files are processed by the same state compiler that processes Salt state files. This means that features like requisites, templates, and so on are available.
Let's take a look at an example, and then discuss each component of it. This formula uses Thorium to detect when a minion has disappeared and then deletes the key from the master when the minion has been gone for 60 seconds:
statreg: status.reg
keydel: key.timeout: - delete: 60 - require: - status: statreg
There are two stanzas in this formula, whose IDs are statreg and keydel. The first stanza, statreg, tells Thorium to keep track of minion status beacons in its register. We'll talk more about the register in a moment.
The second stanza, keydel, is the one that does the real work. It uses the key module to apply an expiration (using the timeout function) to a minion. Because delete is set to 60, this is a 60 second expiration. If a minion does not check in at least once every 60 seconds, its key will be deleted from the master. This particular function also allows you to use reject instead of delete, allowing for a minion to be rejected instead of deleted if it does not check in within the specified time period.
There is also a require requisite in this stanza. It states that the key.timeout function will not be called unless the status.reg function in the statreg codeblock has been successfully called first.
Thorium Links to Beacons The above example was added in the 2016.11.0 release of Salt and makes use of the status beacon also added in the 2016.11.0 release. For the above Thorium state to function properly you will also need to enable the status beacon in the minion configuration file:
beacons: status: - interval: 10
This will cause the minion to use the status beacon to check in with the master every 10 seconds.
The Thorium Register In order to keep track of information, Thorium uses an in-memory register (or rather, collection of registers) on the master. These registers are only populated when told to by a formula, and they normally will be erased when the master is restarted. It is possible to persist the registers to disk, but we'll get to that in a moment.
The example above uses status.reg to populate a register for you, which is automatically used by the key.timeout function. However, you can set your own register values as well, using the reg module.
Because Thorium watches the event bus, the reg module is designed to look for user-specified tags, and then extract data from the payload of events that match those tags. For instance, the following stanza will look for an event with a tag of my/custom/event:
foo: reg.list: - add: bar - match: my/custom/event
When such an event is found, the data found in the payload dictionary key of bar will be stored in a register called foo. This register will store that data in a list. You may also use reg.set to add data to a set() instead.
If you would like to see a copy of the register as it is stored in memory, you can use the file.save function:
myreg: file.save
In this case, each time the register is updated, a copy will be saved in JSON format at /var/cache/salt/master/thorium/saves/myreg. If you would like to see when particular events are added to a list-type register, you may add a stamp option to reg.list (but not reg.set). With the above two stanzas put together, this would look like:
foo: reg.list: - add: bar - match: my/custom/event - stamp: True
If you would like to only keep a certain number of the most recent register entries, you may also add a prune option to reg.list (but not reg.set):
foo: reg.list: - add: bar - match: my/custom/event - stamp: True - prune: 50
This example will only keep the 50 most recent entries in the foo register.
Using Register Data Putting data in a register is useless if you don't do anything with it. The check module is designed to examine register data and determine whether it matches the given parameters. For instance, the check.contains function will return True if the given value is contained in the specified register:
foo: reg.list: - add: bar - match: my/custom/event - stamp: True - prune: 50 check.contains: - value: somedata
Used with a require requisite, we can call one of the wrapper modules and perform an operation. For example:
shell_test: local.cmd: - tgt: dufresne - func: cmd.run - arg: - echo 'thorium success' > /tmp/thorium.txt - require: - check: foo
This stanza will only run if the check.contains function under the foo ID returns true (meaning the match was found).
There are a number of other functions in the check module which use different means of comparing values:
• gt: Check whether the register entry is greater than the given value
• gte: Check whether the register entry is greater than or equal to the given value
• lt: Check whether the register entry is less than the given value
• lte: Check whether the register entry is less than or equal to the given value
• eq: Check whether the register entry is equal to the given value
• ne: Check whether the register entry is not equal to the given value
There is also a function called check.event which does not examine the register. Instead, it looks directly at an event as it is coming in on the event bus, and returns True if that event's tag matches. For example:
salt/foo/*/bar: check.event
run_remote_ex: local.cmd: - tgt: '*' - func: test.version - require: - check: salt/foo/*/bar
This formula will look for an event whose tag is salt/foo/<anything>/bar and if it comes in, issue a test.version to all minions.
Register Persistence It is possible to persist the register data to disk when a master is stopped gracefully, and reload it from disk when the master starts up again. This functionality is provided by the returner subsystem, and is enabled whenever any returner containing a load_reg and a save_reg function is used.
Configuration Salt Cloud provides a powerful interface to interact with cloud hosts. This interface is tightly integrated with Salt, and new virtual machines are automatically connected to your Salt master after creation.
Since Salt Cloud is designed to be an automated system, most configuration is done using the following YAML configuration files:
• /etc/salt/cloud: The main configuration file, contains global settings that apply to all cloud hosts. See Salt Cloud Configuration.
• /etc/salt/cloud.providers.d/*.conf: Contains settings that configure a specific cloud host, such as credentials, region settings, and so on. Since configuration varies significantly between each cloud host, a separate file should be created for each cloud host. In Salt Cloud, a provider is synonymous with a cloud host (Amazon EC2, Google Compute Engine, Rackspace, and so on).  See Provider Specifics.