neulab/tldr huggingface dataset:
What Ports does the Minion Need Open? No ports need to be opened on the minion, as it makes outbound connections to the master. If you've put both your Salt master and minion in debug mode and don't see an acknowledgment that your minion has connected, it could very well be a firewall interfering with the connection. See our firewall configuration page for help opening the firewall on various platforms.
If you have netcat installed, you can check port connectivity from the minion with the nc command:
$ nc -v -z salt.master.ip.addr 4505 Connection to salt.master.ip.addr 4505 port [tcp/unknown] succeeded! $ nc -v -z salt.master.ip.addr 4506 Connection to salt.master.ip.addr 4506 port [tcp/unknown] succeeded!
The Nmap utility can also be used to check if these ports are open:
# nmap -sS -q -p 4505-4506 salt.master.ip.addr
Starting Nmap 6.40 ( http://nmap.org ) at 2013-12-29 19:44 CST Nmap scan report for salt.master.ip.addr (10.0.0.10) Host is up (0.0026s latency). PORT     STATE  SERVICE 4505/tcp open   unknown 4506/tcp open   unknown MAC Address: 00:11:22:AA:BB:CC (Intel)
Nmap done: 1 IP address (1 host up) scanned in 1.64 seconds
If you've opened the correct TCP ports and still aren't seeing connections, check that no additional access control system such as SELinux or AppArmor is blocking Salt. Tools like tcptraceroute can also be used to determine if an intermediate device or firewall is blocking the needed TCP ports.
Using salt-call The salt-call command was originally developed for aiding in the development of new Salt modules. Since then, many applications have been developed for running any Salt module locally on a minion. These range from the original intent of salt-call (development assistance), to gathering more verbose output from calls like state.apply.
When initially creating your state tree, it is generally recommended to invoke highstates by running state.apply directly from the minion with salt-call, rather than remotely from the master. This displays far more information about the execution than calling it remotely. For even more verbosity, increase the loglevel using the -l argument:
# salt-call -l debug state.apply
The main difference between using salt and using salt-call is that salt-call is run from the minion, and it only runs the selected function on that minion. By contrast, salt is run from the master, and requires you to specify the minions on which to run the command using salt's targeting system.
Live Python Debug Output If the minion seems to be unresponsive, a SIGUSR1 can be passed to the process to display what piece of code is executing. This debug information can be invaluable in tracking down bugs.
To pass a SIGUSR1 to the minion, first make sure the minion is running in the foreground. Stop the service if it is running as a daemon, and start it in the foreground like so:
Then pass the signal to the minion when it seems to be unresponsive:
# killall -SIGUSR1 salt-minion
Multiprocessing in Execution Modules As is outlined in github issue #6300, Salt cannot use python's multiprocessing pipes and queues from execution modules. Multiprocessing from the execution modules is perfectly viable, it is just necessary to use Salt's event system to communicate back with the process.
The reason for this difficulty is that python attempts to pickle all objects in memory when communicating, and it cannot pickle function objects. Since the Salt loader system creates and manages function objects this causes the pickle operation to fail.
Salt Minion Doesn't Return Anything While Running Jobs Locally When a command being run via Salt takes a very long time to return (package installations, certain scripts, etc.) the minion may drop you back to the shell. In most situations the job is still running but Salt has exceeded the set timeout before returning. Querying the job queue will provide the data of the job but is inconvenient. This can be resolved by either manually using the -t option to set a longer timeout when running commands (by default it is 5 seconds) or by modifying the minion configuration file: /etc/salt/minion and setting the timeout value to change the default timeout for all commands, and then restarting the salt-minion service.
NOTE: Modifying the minion timeout value is not required when running commands from a Salt Master. It is only required when running commands locally on the minion.
Running in the Foreground A great deal of information is available via the debug logging system, if you are having issues with minions connecting or not starting run the minion and/or master in the foreground:
salt-master -l debug salt-minion -l debug
What Ports do the Master and Minion Need Open? No ports need to be opened up on each minion. For the master, TCP ports 4505 and 4506 need to be open. If you've put both your Salt master and minion in debug mode and don't see an acknowledgment that your minion has connected, it could very well be a firewall.
You can check port connectivity from the minion with the nc command:
nc -v -z salt.master.ip 4505 nc -v -z salt.master.ip 4506
There is also a firewall configuration document that might help as well.
If you've enabled the right TCP ports on your operating system or Linux distribution's firewall and still aren't seeing connections, check that no additional access control system such as SELinux or AppArmor is blocking Salt.
Using salt-call The salt-call command was originally developed for aiding in the development of new Salt modules. Since then, many applications have been developed for running any Salt module locally on a minion. These range from the original intent of salt-call, development assistance, to gathering more verbose output from calls like state.apply.
When initially creating your state tree, it is generally recommended to invoke state.apply directly from the minion with salt-call, rather than remotely from the master. This displays far more information about the execution than calling it remotely. For even more verbosity, increase the loglevel using the -l argument:
salt-call -l debug state.apply
And modify that value to be at least equal to the number of minions x 2. This setting can be changed in limits.conf as the nofile value(s), and activated upon new a login of the specified user.
So, an environment with 1800 minions, would need 1800 x 2 = 3600 as a minimum.
Salt and SELinux Currently there are no SELinux policies for Salt. For the most part Salt runs without issue when SELinux is running in Enforcing mode. This is because when the minion executes as a daemon the type context is changed to initrc_t.  The problem with SELinux arises when using salt-call or running the minion in the foreground, since the type context stays unconfined_t.
This problem is generally manifest in the rpm install scripts when using the pkg module. Until a full SELinux Policy is available for Salt the solution to this issue is to set the execution context of salt-call and salt-minion to rpm_exec_t:
# CentOS 5 and RHEL 5: chcon -t system_u:system_r:rpm_exec_t:s0 /usr/bin/salt-minion chcon -t system_u:system_r:rpm_exec_t:s0 /usr/bin/salt-call
# CentOS 6 and RHEL 6: chcon system_u:object_r:rpm_exec_t:s0 /usr/bin/salt-minion chcon system_u:object_r:rpm_exec_t:s0 /usr/bin/salt-call
This works well, because the rpm_exec_t context has very broad control over other types.
Red Hat Enterprise Linux 5 Salt requires Python 2.6 or 2.7. Red Hat Enterprise Linux 5 and its variants come with Python 2.4 installed by default. When installing on RHEL 5 from the EPEL repository this is handled for you. But, if you run Salt from git, be advised that its dependencies need to be installed from EPEL and that Salt needs to be run with the python26 executable.
Common YAML Gotchas An extensive list of YAML idiosyncrasies has been compiled:
YAML Idiosyncrasies One of Salt's strengths, the use of existing serialization systems for representing SLS data, can also backfire. YAML is a general purpose system and there are a number of things that would seem to make sense in an sls file that cause YAML issues. It is wise to be aware of these issues. While reports or running into them are generally rare they can still crop up at unexpected times.
Spaces vs Tabs YAML uses spaces, period. Do not use tabs in your SLS files! If strange errors are coming up in rendering SLS files, make sure to check that no tabs have crept in! In Vim, after enabling search highlighting with: :set hlsearch,  you can check with the following key sequence in normal mode(you can hit ESC twice to be sure): /, Ctrl-v, Tab, then hit Enter. Also, you can convert tabs to 2 spaces by these commands in Vim: :set tabstop=2 expandtab and then :retab.
Indentation The suggested syntax for YAML files is to use 2 spaces for indentation, but YAML will follow whatever indentation system that the individual file uses. Indentation of two spaces works very well for SLS files given the fact that the data is uniform and not deeply nested.
Nested Dictionaries When dictionaries are nested within other data structures (particularly lists), the indentation logic sometimes changes.  Examples of where this might happen include context and default options from the file.managed state:
/etc/http/conf/http.conf: file: - managed - source: salt://apache/http.conf - user: root - group: root - mode: 644 - template: jinja - context: custom_var: "override" - defaults: custom_var: "default value" other_var: 123
Notice that while the indentation is two spaces per level, for the values under the context and defaults options there is a four-space indent. If only two spaces are used to indent, then those keys will be considered part of the same dictionary that contains the context key, and so the data will not be loaded correctly. If using a double indent is not desirable, then a deeply-nested dict can be declared with curly braces:
/etc/http/conf/http.conf: file: - managed - source: salt://apache/http.conf - user: root - group: root - mode: 644 - template: jinja - context: { custom_var: "override" } - defaults: { custom_var: "default value", other_var: 123 }
Here is a more concrete example of how YAML actually handles these indentations, using the Python interpreter on the command line:
>>> import yaml >>> yaml.safe_load( ...     """mystate: ...   file.managed: ...     - context: ...         some: var""" ... ) {'mystate': {'file.managed': [{'context': {'some': 'var'}}]}} >>> yaml.safe_load( ...     """mystate: ...   file.managed: ...     - context: ...       some: var""" ... ) {'mystate': {'file.managed': [{'some': 'var', 'context': None}]}}
Note that in the second example, some is added as another key in the same dictionary, whereas in the first example, it's the start of a new dictionary.  That's the distinction. context is a common example because it is a keyword arg for many functions, and should contain a dictionary.
Multi-line Strings Similarly, when a multi-line string is nested within a list item (such as when using the contents argument for a file.managed state), the indentation must be doubled. Take for example the following state:
/tmp/foo.txt: file.managed: - contents: | foo bar baz
This is invalid YAML, and will result in a rather cryptic error when you try to run the state:
myminion: Data failed to compile: ---------- Rendering SLS 'base:test' failed: could not find expected ':'; line 5
--- /tmp/foo.txt: file.managed: - contents: | foo bar    <====================== baz
The correct indentation would be as follows:
True/False, Yes/No, On/Off PyYAML will load these values as boolean True or False. Un-capitalized versions will also be loaded as booleans (true, false, yes, no, on, and off). This can be especially problematic when constructing Pillar data. Make sure that your Pillars which need to use the string versions of these values are enclosed in quotes. Pillars will be parsed twice by salt, so you'll need to wrap your values in multiple quotes, including double quotation marks (" ") and single quotation marks (' '). Note that spaces are included in the quotation type examples for clarity.
Multiple quoting examples looks like this:
- '"false"' - "'True'" - "'YES'" - '"No"'
NOTE: When using multiple quotes in this manner, they must be different. Using "" "" or '' '' won't work in this case (spaces are included in examples for clarity).
The '%' Sign The % symbol has a special meaning in YAML, it needs to be passed as a string literal:
cheese: ssh_auth.present: - user: tbortels - source: salt://ssh_keys/chease.pub - config: '%h/.ssh/authorized_keys'
Time Expressions PyYAML will load a time expression as the integer value of that, assuming HH:MM. So for example, 12:00 is loaded by PyYAML as 720. An excellent explanation for why can be found here.
To keep time expressions like this from being loaded as integers, always quote them.
NOTE: When using a jinja load_yaml map, items must be quoted twice. For example:
{% load_yaml as wsus_schedule %}
FRI_10: time: '"23:00"' day: 6 - Every Friday SAT_10: time: '"06:00"' day: 7 - Every Saturday SAT_20: time: '"14:00"' day: 7 - Every Saturday SAT_30: time: '"22:00"' day: 7 - Every Saturday SUN_10: time: '"06:00"' day: 1 - Every Sunday {% endload %}
YAML does not like Double Short Decs If I can find a way to make YAML accept "Double Short Decs" then I will, since I think that double short decs would be awesome. So what is a "Double Short Dec"? It is when you declare a multiple short decs in one ID. Here is a standard short dec, it works great:
The short dec means that there are no arguments to pass, so it is not required to add any arguments, and it can save space.
YAML though, gets upset when declaring multiple short decs, for the record...
THIS DOES NOT WORK:
vim: pkg.installed user.present
Similarly declaring a short dec in the same ID dec as a standard dec does not work either...
ALSO DOES NOT WORK:
fred: user.present ssh_auth.present: - name: AAAAB3NzaC... - user: fred - enc: ssh-dss - require: - user: fred
The correct way is to define them like this:
vim: pkg.installed: [] user.present: []
fred: user.present: [] ssh_auth.present: - name: AAAAB3NzaC... - user: fred - enc: ssh-dss - require: - user: fred
Alternatively,  they can be defined the "old way",  or with multiple "full decs":
vim: pkg: - installed user: - present
fred: user: - present ssh_auth: - present - name: AAAAB3NzaC... - user: fred - enc: ssh-dss - require: - user: fred
YAML supports only plain ASCII According to YAML specification, only ASCII characters can be used.
Within double-quotes, special characters may be represented with C-style escape sequences starting with a backslash ( \ ).
- micro: "\u00b5" - copyright: "\u00A9" - A: "\x41" - alpha: "\u0251" - Alef: "\u05d0"
List of usable Unicode characters  will help you to identify correct numbers.
Python can also be used to discover the Unicode number for a character:
repr(u"Text with wrong characters i need to figure out")
This shell command can find wrong characters in your SLS files:
find . -name '*.sls'  -exec  grep --color='auto' -P -n '[^\x00-\x7F]' \{} \;
Alternatively you can toggle the yaml_utf8 setting in your master configuration file. This is still an experimental setting but it should manage the right encoding conversion in salt after yaml states compilations.
Underscores stripped in Integer Definitions If a definition only includes numbers and underscores, it is parsed by YAML as an integer and all underscores are stripped.  To ensure the object becomes a string, it should be surrounded by quotes.  More information here.
Here's an example:
>>> import yaml >>> yaml.safe_load("2013_05_10") 20130510 >>> yaml.safe_load('"2013_05_10"') '2013_05_10'
Automatic datetime conversion If there is a value in a YAML file formatted 2014-01-20 14:23:23 or similar, YAML will automatically convert this to a Python datetime object.  These objects are not msgpack serializable, and so may break core salt functionality.  If values such as these are needed in a salt YAML file (specifically a configuration file), they should be formatted with surrounding strings to force YAML to serialize them as strings:
>>> import yaml >>> yaml.safe_load("2014-01-20 14:23:23") datetime.datetime(2014, 1, 20, 14, 23, 23) >>> yaml.safe_load('"2014-01-20 14:23:23"') '2014-01-20 14:23:23'
Additionally, numbers formatted like XXXX-XX-XX will also be converted (or YAML will attempt to convert them, and error out if it doesn't think the date is a real one).  Thus, for example, if a minion were to have an ID of 4017-16-20 the minion would not start because YAML would complain that the date was out of range.  The workaround is the same, surround the offending string with quotes:
>>> import yaml >>> yaml.safe_load("4017-16-20") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load return load(stream, SafeLoader) File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load return loader.get_single_data() File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 39, in get_single_data return self.construct_document(node) File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 43, in construct_document data = self.construct_object(node) File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 88, in construct_object data = constructor(self, node) File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 312, in construct_yaml_timestamp return datetime.date(year, month, day) ValueError: month must be in 1..12 >>> yaml.safe_load('"4017-16-20"') '4017-16-20'
Keys Limited to 1024 Characters Simple keys are limited by the YAML Spec to a single line, and cannot be longer that 1024 characters. PyYAML enforces these limitations (see here), and therefore anything parsed as YAML in Salt is subject to them.
Live Python Debug Output If the minion or master seems to be unresponsive, a SIGUSR1 can be passed to the processes to display where in the code they are running. If encountering a situation like this, this debug information can be invaluable. First make sure the master of minion are running in the foreground:
Then pass the signal to the master or minion when it seems to be unresponsive:
killall -SIGUSR1 salt-master killall -SIGUSR1 salt-minion
Also under BSD and macOS in addition to SIGUSR1 signal, debug subroutine set up for SIGINFO which has an advantage of being sent by Ctrl+T shortcut.
When filing an issue or sending questions to the mailing list for a problem with an unresponsive daemon this information can be invaluable.
Salt 0.16.x minions cannot communicate with a 0.17.x master As of release 0.17.1 you can no longer run different versions of Salt on your Master and Minion servers. This is due to a protocol change for security purposes. The Salt team will continue to attempt to ensure versions are as backwards compatible as possible.
Debugging the Master and Minion A list of common master and minion troubleshooting steps provide a starting point for resolving issues you may encounter.
Frequently Asked Questions FAQ • Frequently Asked Questions
• Is Salt open-core?
• I think I found a bug! What should I do?
• What ports should I open on my firewall?
• I'm seeing weird behavior (including but not limited to packages not installing their users properly)
• My script runs every time I run a state.apply. Why?
• When I run test.ping, why don't the Minions that aren't responding return anything? Returning False would be helpful.
• How does Salt determine the Minion's id?
• I'm trying to manage packages/services but I get an error saying that the state is not available. Why?
• Why aren't my custom modules/states/etc. available on my Minions?
• Module X isn't available, even though the shell command it uses is installed. Why?
• Can I run different versions of Salt on my Master and Minion?
• Does Salt support backing up managed files?
• Is it possible to deploy a file to a specific minion, without other minions having access to it?
• What is the best way to restart a Salt Minion daemon using Salt after upgrade?
• Upgrade without automatic restart
• Restart using states
• Restart using remote executions
• Waiting for minions to come back online
• Salting the Salt Master
• Is Targeting using Grain Data Secure?
• Why Did the Value for a Grain Change on Its Own?
Is Salt open-core? No. Salt is 100% committed to being open-source, including all of our APIs. It is developed under the Apache 2.0 license, allowing it to be used in both open and proprietary projects.
To expand on this a little:
There is much argument over the actual definition of "open core".  From our standpoint, Salt is open source because
1. It is a standalone product that anyone is free to use.
2. It is developed in the open with contributions accepted from the community for the good of the project.
3. There are no features of Salt itself that are restricted to separate proprietary products distributed by SaltStack, Inc.
4. Because of our Apache 2.0 license, Salt can be used as the foundation for a project or even a proprietary tool.
5. Our APIs are open and documented (any lack of documentation is an oversight as opposed to an intentional decision by SaltStack the company) and available for use by anyone.
SaltStack the company does make proprietary products which use Salt and its libraries, like company is free to do, but we do so via the APIs, NOT by forking Salt and creating a different, closed-source version of it for paying customers.
I think I found a bug! What should I do? The salt-users mailing list as well as the salt IRC channel can both be helpful resources to confirm if others are seeing the issue and to assist with immediate debugging.
To report a bug to the Salt project, please follow the instructions in reporting a bug.
What ports should I open on my firewall? Minions need to be able to connect to the Master on TCP ports 4505 and 4506.  Minions do not need any inbound ports open. More detailed information on firewall settings can be found here.
I'm seeing weird behavior (including but not limited to packages not installing their users properly) This is often caused by SELinux.  Try disabling SELinux or putting it in permissive mode and see if the weird behavior goes away.
My script runs every time I run a state.apply. Why? You are probably using cmd.run rather than cmd.wait. A cmd.wait state will only run when there has been a change in a state that it is watching.
A cmd.run state will run the corresponding command every time (unless it is prevented from running by the unless or onlyif arguments).
More details can be found in the documentation for the cmd states.
When I run test.ping, why don't the Minions that aren't responding return anything? Returning False would be helpful. When you run test.ping the Master tells Minions to run commands/functions, and listens for the return data, printing it to the screen when it is received.  If it doesn't receive anything back, it doesn't have anything to display for that Minion.
There are a couple options for getting information on Minions that are not responding. One is to use the verbose (-v) option when you run salt commands, as it will display "Minion did not return" for any Minions which time out.
salt -v '*' pkg.install zsh
Another option is to use the manage.down runner:
salt-run manage.down
Also, if the Master is under heavy load, it is possible that the CLI will exit without displaying return data for all targeted Minions. However, this doesn't mean that the Minions did not return; this only means that the Salt CLI timed out waiting for a response. Minions will still send their return data back to the Master once the job completes. If any expected Minions are missing from the CLI output, the jobs.list_jobs runner can be used to show the job IDs of the jobs that have been run, and the jobs.lookup_jid runner can be used to get the return data for that job.
salt-run jobs.list_jobs salt-run jobs.lookup_jid 20130916125524463507
If you find that you are often missing Minion return data on the CLI, only to find it with the jobs runners, then this may be a sign that the worker_threads value may need to be increased in the master config file. Additionally, running your Salt CLI commands with the -t option will make Salt wait longer for the return data before the CLI command exits. For instance, the below command will wait up to 60 seconds for the Minions to return:
salt -t 60 '*' test.ping
How does Salt determine the Minion's id? If the Minion id is not configured explicitly (using the id parameter), Salt will determine the id based on the hostname. Exactly how this is determined varies a little between operating systems and is described in detail here.
I'm trying to manage packages/services but I get an error saying that the state is not available. Why? Salt detects the Minion's operating system and assigns the correct package or service management module based on what is detected. However, for certain custom spins and OS derivatives this detection fails. In cases like this, an issue should be opened on our tracker, with the following information:
1. The output of the following command:
salt <minion_id> grains.items | grep os
2. The contents of /etc/lsb-release, if present on the Minion.
Why aren't my custom modules/states/etc. available on my Minions? Custom modules are synced to Minions when saltutil.sync_modules, or saltutil.sync_all is run.
Similarly, custom states are synced to Minions when saltutil.sync_states, or saltutil.sync_all is run.
They are both also synced when a highstate is triggered.
As of the 2019.2.0 release, as well as 2017.7.7 and 2018.3.2 in their respective release cycles, the sync argument to state.apply/state.sls can be used to sync custom types when running individual SLS files.
Other custom types (renderers, outputters, etc.) have similar behavior, see the documentation for the saltutil module for more information.
This reactor example can be used to automatically sync custom types when the minion connects to the master, to help with this chicken-and-egg issue.
Module X isn't available, even though the shell command it uses is installed. Why? This is most likely a PATH issue. Did you custom-compile the software which the module requires? RHEL/CentOS/etc. in particular override the root user's path in /etc/init.d/functions, setting it to /sbin:/usr/sbin:/bin:/usr/bin, making software installed into /usr/local/bin unavailable to Salt when the Minion is started using the initscript. In version 2014.1.0, Salt will have a better solution for these sort of PATH-related issues, but recompiling the software to install it into a location within the PATH should resolve the issue in the meantime. Alternatively, you can create a symbolic link within the PATH using a file.symlink state.
/usr/bin/foo: file.symlink: - target: /usr/local/bin/foo
Can I run different versions of Salt on my Master and Minion? This depends on the versions.  In general, it is recommended that Master and Minion versions match.
When upgrading Salt, the master(s) should always be upgraded first. Backwards compatibility for minions running newer versions of salt than their masters is not guaranteed.
Whenever possible, backwards compatibility between new masters and old minions will be preserved.  Generally, the only exception to this policy is in case of a security vulnerability.
Recent examples of backwards compatibility breakage include the 0.17.1 release (where all backwards compatibility was broken due to a security fix), and the 2014.1.0 release (which retained compatibility between 2014.1.0 masters and 0.17 minions, but broke compatibility for 2014.1.0 minions and older masters).
Does Salt support backing up managed files? Yes. Salt provides an easy to use addition to your file.managed states that allow you to back up files via backup_mode, backup_mode can be configured on a per state basis, or in the minion config (note that if set in the minion config this would simply be the default method to use, you still need to specify that the file should be backed up!).
Is it possible to deploy a file to a specific minion, without other minions having access to it? The Salt fileserver does not yet support access control, but it is still possible to do this. As of Salt 2015.5.0, the file_tree external pillar is available, and allows the contents of a file to be loaded as Pillar data. This external pillar is capable of assigning Pillar values both to individual minions, and to nodegroups. See the documentation for details on how to set this up.
Once the external pillar has been set up, the data can be pushed to a minion via a file.managed state, using the contents_pillar argument:
/etc/my_super_secret_file: file.managed: - user: secret - group: secret - mode: 600 - contents_pillar: secret_files:my_super_secret_file
In this example, the source file would be located in a directory called secret_files underneath the file_tree path for the minion. The syntax for specifying the pillar variable is the same one used for pillar.get, with a colon representing a nested dictionary.
WARNING: Deploying binary contents using the file.managed state is only supported in Salt 2015.8.4 and newer.
What is the best way to restart a Salt Minion daemon using Salt after upgrade? Updating the salt-minion package requires a restart of the salt-minion service. But restarting the service while in the middle of a state run interrupts the process of the Minion running states and sending results back to the Master. A common way to workaround that is to schedule restarting the Minion service in the background by issuing a salt-call command calling service.restart function. This prevents the Minion being disconnected from the Master immediately. Otherwise you would get Minion did not return. [Not connected] message as the result of a state run.
Upgrade without automatic restart Doing the Minion upgrade seems to be a simplest state in your SLS file at first. But the operating systems such as Debian GNU/Linux, Ubuntu and their derivatives start the service after the package installation by default.  To prevent this, we need to create policy layer which will prevent the Minion service to restart right after the upgrade:
{%- if grains['os_family'] == 'Debian' %}
Disable starting services: file.managed: - name: /usr/sbin/policy-rc.d - user: root - group: root - mode: 0755 - contents: - '#!/bin/sh' - exit 101 # do not touch if already exists - replace: False - prereq: - pkg: Upgrade Salt Minion
{%- endif %}
Upgrade Salt Minion: pkg.installed: - name: salt-minion - version: 2016.11.3{% if grains['os_family'] == 'Debian' %}+ds-1{% endif %} - order: last
Enable Salt Minion: service.enabled: - name: salt-minion - require: - pkg: Upgrade Salt Minion
Enable starting services: file.absent: - name: /usr/sbin/policy-rc.d - onchanges: - pkg: Upgrade Salt Minion
Restart using states Now we can apply the workaround to restart the Minion in reliable way. The following example works on UNIX-like operating systems:
{%- if grains['os'] != 'Windows' %} Restart Salt Minion: cmd.run: - name: 'salt-call service.restart salt-minion' - bg: True - onchanges: - pkg: Upgrade Salt Minion {%- endif %}
Note that restarting the salt-minion service on Windows operating systems is not always necessary when performing an upgrade. The installer stops the salt-minion service, removes it, deletes the contents of the \salt\bin directory, installs the new code, re-creates the salt-minion service, and starts it (by default). The restart step would be necessary during the upgrade process, however, if the minion config was edited after the upgrade or installation. If a minion restart is necessary, the state above can be edited as follows:
Restart Salt Minion: cmd.run: {%- if grains['kernel'] == 'Windows' %} - name: 'C:\salt\salt-call.bat service.restart salt-minion' {%- else %} - name: 'salt-call service.restart salt-minion' {%- endif %} - bg: True - onchanges: - pkg: Upgrade Salt Minion
However, it requires more advanced tricks to upgrade from legacy version of Salt (before 2016.3.0) on UNIX-like operating systems, where executing commands in the background is not supported. You also may need to schedule restarting the Minion service using masterless mode after all other states have been applied for Salt versions earlier than 2016.11.0. This allows the Minion to keep the connection to the Master alive for being able to report the final results back to the Master, while the service is restarting in the background. This state should run last or watch for the pkg state changes:
Restart Salt Minion: cmd.run: {%- if grains['kernel'] == 'Windows' %} - name: 'start powershell "Restart-Service -Name salt-minion"' {%- else %} # fork and disown the process - name: |- exec 0>&- # close stdin exec 1>&- # close stdout exec 2>&- # close stderr nohup salt-call --local service.restart salt-minion & {%- endif %}
Restart using remote executions Restart the Minion from the command line:
salt -G kernel:Windows cmd.run_bg 'C:\salt\salt-call.bat service.restart salt-minion' salt -C 'not G@kernel:Windows' cmd.run_bg 'salt-call service.restart salt-minion'
Waiting for minions to come back online A common issue in performing automated restarts of a salt minion, for example during an orchestration run, is that it will break the orchestration since the next statement is likely to be attempted before the minion is back online. This can be remedied by inserting a blocking waiting state that only returns when the selected minions are back up (note: this will only work in orchestration states since manage.up needs to run on the master):
Wait for salt minion: loop.until_no_eval: - name: saltutil.runner - expected: - my_minion - args: - manage.up - kwargs: tgt: my_minion - period: 3 - init_wait: 3
This will, after an initial delay of 3 seconds, execute the manage.up-runner targeted specifically for my_minion. It will do this every period seconds until the expected data is returned. The default timeout is 60s but can be configured as well.
Salting the Salt Master In order to configure a master server via states, the Salt master can also be "salted" in order to enforce state on the Salt master as well as the Salt minions. Salting the Salt master requires a Salt minion to be installed on the same machine as the Salt master. Once the Salt minion is installed, the minion configuration file must be pointed to the local Salt master:
master: 127.0.0.1
Once the Salt master has been "salted" with a Salt minion, it can be targeted just like any other minion. If the minion on the salted master is running, the minion can be targeted via any usual salt command. Additionally, the salt-call command can execute operations to enforce state on the salted master without requiring the minion to be running.
More information about salting the Salt master can be found in the salt-formula for salt itself:
https://github.com/saltstack-formulas/salt-formula
Restarting the salt-master service using execution module or application of state could be done the same way as for the Salt minion described above.
Is Targeting using Grain Data Secure? WARNING: Grains can be set by users that have access to the minion configuration files on the local system, making them less secure than other identifiers in Salt. Avoid storing sensitive data, such as passwords or keys, on minions. Instead, make use of pillar and/or sdb.
Because grains can be set by users that have access to the minion configuration files on the local system, grains are considered less secure than other identifiers in Salt. Use caution when targeting sensitive operations or setting pillar values based on grain data.
The only grain which can be safely used is grains['id'] which contains the Minion ID.
When possible, you should target sensitive operations and data using the Minion ID. If the Minion ID of a system changes, the Salt Minion's public key must be re-accepted by an administrator on the Salt Master, making it less vulnerable to impersonation attacks.
Why Did the Value for a Grain Change on Its Own? This is usually the result of an upstream change in an OS distribution that replaces or removes something that Salt was using to detect the grain.  Fortunately, when this occurs, you can use Salt to fix it with a command similar to the following:
salt -G 'grain:ChangedValue' grains.setvals "{'grain': 'OldValue'}"
(Replacing grain, ChangedValue, and OldValue with the grain and values that you want to change / set.)
You should also file an issue describing the change so it can be fixed in Salt.
Salt Best Practices Salt's extreme flexibility leads to many questions concerning the structure of configuration files.
This document exists to clarify these points through examples and code.
IMPORTANT: The guidance here should be taken in combination with hardening-salt.
General rules 1. Modularity and clarity should be emphasized whenever possible.
2. Create clear relations between pillars and states.
3. Use variables when it makes sense but don't overuse them.
4. Store sensitive data in pillar.
5. Don't use grains for matching in your pillar top file for any sensitive pillars.
WARNING: Grains can be set by users that have access to the minion configuration files on the local system, making them less secure than other identifiers in Salt. Avoid storing sensitive data, such as passwords or keys, on minions. Instead, make use of pillar and/or sdb.
Structuring States and Formulas When structuring Salt States and Formulas it is important to begin with the directory structure. A proper directory structure clearly defines the functionality of each state to the user via visual inspection of the state's name.
Reviewing the MySQL Salt Formula it is clear to see the benefits to the end-user when reviewing a sample of the available states:
/srv/salt/mysql/files/ /srv/salt/mysql/client.sls /srv/salt/mysql/map.jinja /srv/salt/mysql/python.sls /srv/salt/mysql/server.sls
This directory structure would lead to these states being referenced in a top file in the following way:
base: 'web*': - mysql.client - mysql.python 'db*': - mysql.server
This clear definition ensures that the user is properly informed of what each state will do.
Another example comes from the vim-formula:
/srv/salt/vim/files/ /srv/salt/vim/absent.sls /srv/salt/vim/init.sls /srv/salt/vim/map.jinja /srv/salt/vim/nerdtree.sls /srv/salt/vim/pyflakes.sls /srv/salt/vim/salt.sls
Once again viewing how this would look in a top file:
base: 'web*': - vim - vim.nerdtree - vim.pyflakes - vim.salt 'db*': - vim.absent
The usage of a clear top-level directory as well as properly named states reduces the overall complexity and leads a user to both understand what will be included at a glance and where it is located.
In addition Formulas should be used as often as possible.
NOTE: Formulas repositories on the saltstack-formulas GitHub organization should not be pointed to directly from systems that automatically fetch new updates such as GitFS or similar tooling. Instead formulas repositories should be forked on GitHub or cloned locally, where unintended, automatic changes will not take place.
Structuring Pillar Files Pillars are used to store secure and insecure data pertaining to minions. When designing the structure of the /srv/pillar directory, the pillars contained within should once again be focused on clear and concise data which users can easily review, modify, and understand.
The /srv/pillar/ directory is primarily controlled by top.sls. It should be noted that the pillar top.sls is not used as a location to declare variables and their values. The top.sls is used as a way to include other pillar files and organize the way they are matched based on environments or grains.
An example top.sls may be as simple as the following:
base: '*': - packages 'web*': - apache - vim
Or an even more complicated example, using a variety of matchers in numerous environments:
base: '*': - apache dev: 'os:Debian': - match: grain - vim test: '* and not G@os: Debian': - match: compound - emacs
It is clear to see through these examples how the top file provides users with power but when used incorrectly it can lead to confusing configurations. This is why it is important to understand that the top file for pillar is not used for variable definitions.
Each SLS file within the /srv/pillar/ directory should correspond to the states which it matches.
This would mean that the apache pillar file should contain data relevant to Apache. Structuring files in this way once again ensures modularity, and creates a consistent understanding throughout our Salt environment. Users can expect that pillar variables found in an Apache state will live inside of an Apache pillar:
/srv/pillar/apache.sls:
apache: lookup: name: httpd config: tmpl: /etc/httpd/httpd.conf
While this pillar file is simple, it shows how a pillar file explicitly relates to the state it is associated with.
Variable Flexibility Salt allows users to define variables in SLS files. When creating a state variables should provide users with as much flexibility as possible. This means that variables should be clearly defined and easy to manipulate, and that sane defaults should exist in the event a variable is not properly defined. Looking at several examples shows how these different items can lead to extensive flexibility.
Although it is possible to set variables locally, this is generally not preferred:
/srv/salt/apache/conf.sls:
{% set name = 'httpd' %} {% set tmpl = 'salt://apache/files/httpd.conf' %}
apache_conf: file.managed: - name: {{ name }} - source: {{ tmpl }} - template: jinja - user: root - watch_in: - service: apache
When generating this information it can be easily transitioned to the pillar where data can be overwritten, modified, and applied to multiple states, or locations within a single state:
apache: lookup: name: httpd config: tmpl: salt://apache/files/httpd.conf
{% from "apache/map.jinja" import apache with context %}
apache_conf: file.managed: - name: {{ salt['pillar.get']('apache:lookup:name') }} - source: {{ salt['pillar.get']('apache:lookup:config:tmpl') }} - template: jinja - user: root - watch_in: - service: apache
This flexibility provides users with a centralized location to modify variables, which is extremely important as an environment grows.
Modularity Within States Ensuring that states are modular is one of the key concepts to understand within Salt. When creating a state a user must consider how many times the state could be re-used, and what it relies on to operate. Below are several examples which will iteratively explain how a user can go from a state which is not very modular to one that is:
httpd: pkg: - installed service.running: - enable: True
/etc/httpd/httpd.conf: file.managed: - source: salt://apache/files/httpd.conf - template: jinja - watch_in: - service: httpd
The example above is probably the worst-case scenario when writing a state.  There is a clear lack of focus by naming both the pkg/service, and managed file directly as the state ID. This would lead to changing multiple requires within this state, as well as others that may depend upon the state.
Imagine if a require was used for the httpd package in another state, and then suddenly it's a custom package. Now changes need to be made in multiple locations which increases the complexity and leads to a more error prone configuration.
There is also the issue of having the configuration file located in the init, as a user would be unable to simply install the service and use the default conf file.
Our second revision begins to address the referencing by using - name, as opposed to direct ID references:
apache: pkg.installed: - name: httpd service.running: - name: httpd - enable: True
apache_conf: file.managed: - name: /etc/httpd/httpd.conf - source: salt://apache/files/httpd.conf - template: jinja - watch_in: - service: apache
The above init file is better than our original, yet it has several issues which lead to a lack of modularity. The first of these problems is the usage of static values for items such as the name of the service, the name of the managed file, and the source of the managed file. When these items are hard coded they become difficult to modify and the opportunity to make mistakes arises. It also leads to multiple edits that need to occur when changing these items (imagine if there were dozens of these occurrences throughout the state!). There is also still the concern of the configuration file data living in the same state as the service and package.
In the next example steps will be taken to begin addressing these issues. Starting with the addition of a map.jinja file (as noted in the Formula documentation), and modification of static values:
/srv/salt/apache/map.jinja:
{% set apache = salt['grains.filter_by']({ 'Debian': { 'server': 'apache2', 'service': 'apache2', 'conf': '/etc/apache2/apache.conf', }, 'RedHat': { 'server': 'httpd', 'service': 'httpd', 'conf': '/etc/httpd/httpd.conf', }, }, merge=salt['pillar.get']('apache:lookup')) %}
apache: lookup: config: tmpl: salt://apache/files/httpd.conf
apache: pkg.installed: - name: {{ apache.server }} service.running: - name: {{ apache.service }} - enable: True
apache_conf: file.managed: - name: {{ apache.conf }} - source: {{ salt['pillar.get']('apache:lookup:config:tmpl') }} - template: jinja - user: root - watch_in: - service: apache
The changes to this state now allow us to easily identify the location of the variables, as well as ensuring they are flexible and easy to modify. While this takes another step in the right direction, it is not yet complete.  Suppose the user did not want to use the provided conf file, or even their own configuration file, but the default apache conf. With the current state setup this is not possible. To attain this level of modularity this state will need to be broken into two states.
This new structure now allows users to choose whether they only wish to install the default Apache, or if they wish, overwrite the default package, service, configuration file location, or the configuration file itself. In addition to this the data has been broken between multiple files allowing for users to identify where they need to change the associated data.
Storing Secure Data Secure data refers to any information that you would not wish to share with anyone accessing a server. This could include data such as passwords, keys, or other information.
As all data within a state is accessible by EVERY server that is connected it is important to store secure data within pillar. This will ensure that only those servers which require this secure data have access to it. In this example a use can go from an insecure configuration to one which is only accessible by the appropriate hosts:
/srv/salt/mysql/testerdb.sls:
testdb: mysql_database.present: - name: testerdb
/srv/salt/mysql/user.sls:
include: - mysql.testerdb
testdb_user: mysql_user.present: - name: frank - password: "test3rdb" - host: localhost - require: - sls: mysql.testerdb
Many users would review this state and see that the password is there in plain text, which is quite problematic. It results in several issues which may not be immediately visible.
The first of these issues is clear to most users -- the password being visible in this state. This  means that any minion will have a copy of this, and therefore the password which is a major security concern as minions may not be locked down as tightly as the master server.
The other issue that can be encountered is access by users on the master. If everyone has access to the states (or their repository), then they are able to review this password. Keeping your password data accessible by only a few users is critical for both security and peace of mind.
There is also the issue of portability. When a state is configured this way it results in multiple changes needing to be made. This was discussed in the sections above but it is a critical idea to drive home. If states are not portable it may result in more work later!
Fixing this issue is relatively simple, the content just needs to be moved to the associated pillar:
/srv/pillar/mysql.sls:
mysql: lookup: name: testerdb password: test3rdb user: frank host: localhost
testdb: mysql_database.present: - name: {{ salt['pillar.get']('mysql:lookup:name') }}
testdb_user: mysql_user.present: - name: {{ salt['pillar.get']('mysql:lookup:user') }} - password: {{ salt['pillar.get']('mysql:lookup:password') }} - host: {{ salt['pillar.get']('mysql:lookup:host') }} - require: - sls: mysql.testerdb
Now that the database details have been moved to the associated pillar file, only machines which are targeted via pillar will have access to these details.  Access to users who should not be able to review these details can also be prevented while ensuring that they are still able to write states which take advantage of this information.