neulab/tldr huggingface dataset:
# salt-call --local grains.items local: ---------- <Snipped for brevity> my_grains: ---------- foo: bar hello: world
However, if you don't prepend the my_custom_grain function with an underscore, the function will be rendered twice by Salt in the items output: once for the my_custom_grain call itself, and again when it is called in the main function:
# salt-call --local grains.items local: ---------- <Snipped for brevity> foo: bar <Snipped for brevity> hello: world <Snipped for brevity> my_grains: ---------- foo: bar hello: world
Precedence Core grains can be overridden by custom grains. As there are several ways of defining custom grains, there is an order of precedence which should be kept in mind when defining them. The order of evaluation is as follows:
1. Core grains.
2. Custom grains in /etc/salt/grains.
3. Custom grains in /etc/salt/minion.
4. Custom grain modules in _grains directory, synced to minions.
Each successive evaluation overrides the previous ones, so any grains defined by custom grains modules synced to minions that have the same name as a core grain will override that core grain. Similarly, grains from /etc/salt/minion override both core grains and custom grain modules, and grains in _grains will override any grains of the same name.
For custom grains, if the function takes an argument grains, then the previously rendered grains will be passed in.  Because the rest of the grains could be rendered in any order, the only grains that can be relied upon to be passed in are core grains. This was added in the 2019.2.0 release.
Examples of Grains The core module in the grains package is where the main grains are loaded by the Salt minion and provides the principal example of how to write grains:
https://github.com/saltstack/salt/blob/master/salt/grains/core.py
Syncing Grains Syncing grains can be done a number of ways. They are automatically synced when state.highstate is called, or (as noted above) the grains can be manually synced and reloaded by calling the saltutil.sync_grains or saltutil.sync_all functions.
NOTE: When the grains_cache is set to False, the grains dictionary is built and stored in memory on the minion. Every time the minion restarts or saltutil.refresh_grains is run, the grain dictionary is rebuilt from scratch.
Storing Static Data in the Pillar Pillar is an interface for Salt designed to offer global values that can be distributed to minions. Pillar data is managed in a similar way as the Salt State Tree.
Pillar was added to Salt in version 0.9.8
NOTE: Storing sensitive data
Pillar data is compiled on the master. Additionally, pillar data for a given minion is only accessible by the minion for which it is targeted in the pillar configuration. This makes pillar useful for storing sensitive data specific to a particular minion.
Declaring the Master Pillar The Salt Master server maintains a pillar_roots setup that matches the structure of the file_roots used in the Salt file server. Like file_roots, the pillar_roots option maps environments to directories. The pillar data is then mapped to minions based on matchers in a top file which is laid out in the same way as the state top file. Salt pillars can use the same matcher types as the standard top file.
conf_master:pillar_roots is configured just like file_roots.  For example:
pillar_roots: base: - /srv/pillar
This example configuration declares that the base environment will be located in the /srv/pillar directory. It must not be in a subdirectory of the state tree.
The top file used matches the name of the top file used for States, and has the same structure:
/srv/pillar/top.sls
base: '*': - packages
In the above top file, it is declared that in the base environment, the glob matching all minions will have the pillar data found in the packages pillar available to it. Assuming the pillar_roots value of /srv/pillar taken from above, the packages pillar would be located at /srv/pillar/packages.sls.
Any number of matchers can be added to the base environment. For example, here is an expanded version of the Pillar top file stated above:
/srv/pillar/top.sls:
base: '*': - packages 'web*': - vim
In this expanded top file, minions that match web* will have access to the /srv/pillar/packages.sls file, as well as the /srv/pillar/vim.sls file.
Another example shows how to use other standard top matching types to deliver specific salt pillar data to minions with different properties.
Here is an example using the grains matcher to target pillars to minions by their os grain:
dev: 'os:Debian': - match: grain - servers
Pillar definitions can also take a keyword argument ignore_missing.  When the value of ignore_missing is True, all errors for missing pillar files are ignored. The default value for ignore_missing is False.
Here is an example using the ignore_missing keyword parameter to ignore errors for missing pillar files:
base: '*': - servers - systems - ignore_missing: True
Assuming that the pillar servers exists in the fileserver backend and the pillar systems doesn't, all pillar data from servers pillar is delivered to minions and no error for the missing pillar systems is noted under the key _errors in the pillar data delivered to minions.
Should the ignore_missing keyword parameter have the value False, an error for the missing pillar systems would produce the value Specified SLS 'servers' in environment 'base' is not available on the salt master under the key _errors in the pillar data delivered to minions.
/srv/pillar/packages.sls
{% if grains['os'] == 'RedHat' %} apache: httpd git: git {% elif grains['os'] == 'Debian' %} apache: apache2 git: git-core {% endif %}
company: Foo Industries
IMPORTANT: See Is Targeting using Grain Data Secure? for important security information.
The above pillar sets two key/value pairs. If a minion is running RedHat, then the apache key is set to httpd and the git key is set to the value of git. If the minion is running Debian, those values are changed to apache2 and git-core respectively. All minions that have this pillar targeting to them via a top file will have the key of company with a value of Foo Industries.
Consequently this data can be used from within modules, renderers, State SLS files, and more via the shared pillar dictionary:
apache: pkg.installed: - name: {{ pillar['apache'] }}
git: pkg.installed: - name: {{ pillar['git'] }}
Finally, the above states can utilize the values provided to them via Pillar.  All pillar values targeted to a minion are available via the 'pillar' dictionary. As seen in the above example, Jinja substitution can then be utilized to access the keys and values in the Pillar dictionary.
Note that you cannot just list key/value-information in top.sls. Instead, target a minion to a pillar file and then list the keys and values in the pillar. Here is an example top file that illustrates this point:
base: '*': - common_pillar
And the actual pillar file at '/srv/pillar/common_pillar.sls':
foo: bar boo: baz
NOTE: When working with multiple pillar environments, assuming that each pillar environment has its own top file, the jinja placeholder {{ saltenv }} can be used in place of the environment name:
{{ saltenv }}: '*': - common_pillar
Yes, this is {{ saltenv }}, and not {{ pillarenv }}. The reason for this is because the Pillar top files are parsed using some of the same code which parses top files when running states, so the pillar environment takes the place of {{ saltenv }} in the jinja context.
Dynamic Pillar Environments If environment __env__ is specified in pillar_roots, all environments that are not explicitly specified in pillar_roots will map to the directories from __env__. This allows one to use dynamic git branch based environments for state/pillar files with the same file-based pillar applying to all environments. For example:
pillar_roots: __env__: - /srv/pillar
ext_pillar: - git: - __env__ https://example.com/git-pillar.git
New in version 2017.7.5,2018.3.1.
Pillar Namespace Flattening The separate pillar SLS files all merge down into a single dictionary of key-value pairs. When the same key is defined in multiple SLS files, this can result in unexpected behavior if care is not taken to how the pillar SLS files are laid out.
For example, given a top.sls containing the following:
base: '*': - packages - services
with packages.sls containing:
bind: bind9
and services.sls containing:
bind: named
Then a request for the bind pillar key will only return named. The bind9 value will be lost, because services.sls was evaluated later.
NOTE: Pillar files are applied in the order they are listed in the top file. Therefore conflicting keys will be overwritten in a 'last one wins' manner!  For example, in the above scenario conflicting key values in services will overwrite those in packages because it's at the bottom of the list.
It can be better to structure your pillar files with more hierarchy. For example the package.sls file could be configured like so:
packages: bind: bind9
This would make the packages pillar key a nested dictionary containing a bind key.
Pillar Dictionary Merging If the same pillar key is defined in multiple pillar SLS files, and the keys in both files refer to nested dictionaries, then the content from these dictionaries will be recursively merged.
For example, keeping the top.sls the same, assume the following modifications to the pillar SLS files:
packages.sls:
bind: package-name: bind9 version: 9.9.5
services.sls:
bind: port: 53 listen-on: any
The resulting pillar dictionary will be:
$ salt-call pillar.get bind local: ---------- listen-on: any package-name: bind9 port: 53 version: 9.9.5
Since both pillar SLS files contained a bind key which contained a nested dictionary, the pillar dictionary's bind key contains the combined contents of both SLS files' bind keys.
Including Other Pillars New in version 0.16.0.
Pillar SLS files may include other pillar files, similar to State files. Two syntaxes are available for this purpose. The simple form simply includes the additional pillar as if it were part of the same file:
include: - users
The full include form allows two additional options -- passing default values to the templating engine for the included pillar file as well as an optional key under which to nest the results of the included pillar:
include: - users: defaults: sudo: ['bob', 'paul'] key: users
With this form, the included file (users.sls) will be nested within the 'users' key of the compiled pillar. Additionally, the 'sudo' value will be available as a template variable to users.sls.
In-Memory Pillar Data vs. On-Demand Pillar Data Since compiling pillar data is computationally expensive, the minion will maintain a copy of the pillar data in memory to avoid needing to ask the master to recompile and send it a copy of the pillar data each time pillar data is requested. This in-memory pillar data is what is returned by the pillar.item, pillar.get, and pillar.raw functions.
Also, for those writing custom execution modules, or contributing to Salt's existing execution modules, the in-memory pillar data is available as the __pillar__ dunder dictionary.
The in-memory pillar data is generated on minion start, and can be refreshed using the saltutil.refresh_pillar function:
salt '*' saltutil.refresh_pillar
This function triggers the minion to asynchronously refresh the in-memory pillar data and will always return None.
In contrast to in-memory pillar data, certain actions trigger pillar data to be compiled to ensure that the most up-to-date pillar data is available. These actions include:
• Running states
• Running pillar.items
Performing these actions will not refresh the in-memory pillar data. So, if pillar data is modified, and then states are run, the states will see the updated pillar data, but pillar.item, pillar.get, and pillar.raw will not see this data unless refreshed using saltutil.refresh_pillar.
How Pillar Environments Are Handled When multiple pillar environments are used, the default behavior is for the pillar data from all environments to be merged together. The pillar dictionary will therefore contain keys from all configured environments.
The pillarenv minion config option can be used to force the minion to only consider pillar configuration from a single environment. This can be useful in cases where one needs to run states with alternate pillar data, either in a testing/QA environment or to test changes to the pillar data before pushing them live.
For example, assume that the following is set in the minion config file:
pillarenv: base
This would cause that minion to ignore all other pillar environments besides base when compiling the in-memory pillar data. Then, when running states, the pillarenv CLI argument can be used to override the minion's pillarenv config value:
salt '*' state.apply mystates pillarenv=testing
The above command will run the states with pillar data sourced exclusively from the testing environment, without modifying the in-memory pillar data.
NOTE: When running states, the pillarenv CLI option does not require a pillarenv option to be set in the minion config file. When pillarenv is left unset, as mentioned above all configured environments will be combined. Running states with pillarenv=testing in this case would still restrict the states' pillar data to just that of the testing pillar environment.
Starting in the 2017.7.0 release, it is possible to pin the pillarenv to the effective saltenv, using the pillarenv_from_saltenv minion config option. When this is set to True, if a specific saltenv is specified when running states, the pillarenv will be the same. This essentially makes the following two commands equivalent:
salt '*' state.apply mystates saltenv=dev salt '*' state.apply mystates saltenv=dev pillarenv=dev
However, if a pillarenv is specified, it will override this behavior. So, the following command will use the qa pillar environment but source the SLS files from the dev saltenv:
salt '*' state.apply mystates saltenv=dev pillarenv=qa
So, if a pillarenv is set in the minion config file, pillarenv_from_saltenv will be ignored, and passing a pillarenv on the CLI will temporarily override pillarenv_from_saltenv.
Viewing Pillar Data To view pillar data, use the pillar execution module. This module includes several functions, each of them with their own use. These functions include:
• pillar.item - Retrieves the value of one or more keys from the in-memory pillar data.
• pillar.items - Compiles a fresh pillar dictionary and returns it, leaving the in-memory pillar data untouched. If pillar keys are passed to this function however, this function acts like pillar.item and returns their values from the in-memory pillar data.
• pillar.raw - Like pillar.items, it returns the entire pillar dictionary, but from the in-memory pillar data instead of compiling fresh pillar data.
• pillar.get - Described in detail below.
The pillar.get Function New in version 0.14.0.
The pillar.get function works much in the same way as the get method in a python dict, but with an enhancement: nested dictonaries can be traversed using a colon as a delimiter.
If a structure like this is in pillar:
foo: bar: baz: qux
Extracting it from the raw pillar in an sls formula or file template is done this way:
{{ pillar['foo']['bar']['baz'] }}
Now, with the new pillar.get function the data can be safely gathered and a default can be set, allowing the template to fall back if the value is not available:
{{ salt['pillar.get']('foo:bar:baz', 'qux') }}
This makes handling nested structures much easier.
NOTE: pillar.get() vs salt['pillar.get']()
It should be noted that within templating, the pillar variable is just a dictionary.  This means that calling pillar.get() inside of a template will just use the default dictionary .get() function which does not include the extra : delimiter functionality.  It must be called using the above syntax (salt['pillar.get']('foo:bar:baz', 'qux')) to get the salt function, instead of the default dictionary behavior.
Setting Pillar Data at the Command Line Pillar data can be set at the command line like the following example:
salt '*' state.apply pillar='{"cheese": "spam"}'
This will add a pillar key of cheese with its value set to spam.
NOTE: Be aware that when sending sensitive data via pillar on the command-line that the publication containing that data will be received by all minions and will not be restricted to the targeted minions. This may represent a security concern in some cases.
Pillar Encryption Salt's renderer system can be used to decrypt pillar data. This allows for pillar items to be stored in an encrypted state, and decrypted during pillar compilation.
Encrypted Pillar SLS New in version 2017.7.0.
Consider the following pillar SLS file:
secrets: vault: foo: | -----BEGIN PGP MESSAGE-----
hQEMAw2B674HRhwSAQgAhTrN8NizwUv/VunVrqa4/X8t6EUulrnhKcSeb8sZS4th W1Qz3K2NjL4lkUHCQHKZVx/VoZY7zsddBIFvvoGGfj8+2wjkEDwFmFjGE4DEsS74 ZLRFIFJC1iB/O0AiQ+oU745skQkU6OEKxqavmKMrKo3rvJ8ZCXDC470+i2/Hqrp7 +KWGmaDOO422JaSKRm5D9bQZr9oX7KqnrPG9I1+UbJyQSJdsdtquPWmeIpamEVHb VMDNQRjSezZ1yKC4kCWm3YQbBF76qTHzG1VlLF5qOzuGI9VkyvlMaLfMibriqY73 zBbPzf6Bkp2+Y9qyzuveYMmwS4sEOuZL/PetqisWe9JGAWD/O+slQ2KRu9hNww06 KMDPJRdyj5bRuBVE4hHkkP23KrYr7SuhW2vpe7O/MvWEJ9uDNegpMLhTWruGngJh iFndxegN9w== =bAuo -----END PGP MESSAGE----- bar: this was unencrypted already baz: | -----BEGIN PGP MESSAGE-----
hQEMAw2B674HRhwSAQf+Ne+IfsP2IcPDrUWct8sTJrga47jQvlPCmO+7zJjOVcqz gLjUKvMajrbI/jorBWxyAbF+5E7WdG9WHHVnuoywsyTB9rbmzuPqYCJCe+ZVyqWf 9qgJ+oUjcvYIFmH3h7H68ldqbxaAUkAOQbTRHdr253wwaTIC91ZeX0SCj64HfTg7 Izwk383CRWonEktXJpientApQFSUWNeLUWagEr/YPNFA3vzpPF5/Ia9X8/z/6oO2 q+D5W5mVsns3i2HHbg2A8Y+pm4TWnH6mTSh/gdxPqssi9qIrzGQ6H1tEoFFOEq1V kJBe0izlfudqMq62XswzuRB4CYT5Iqw1c97T+1RqENJCASG0Wz8AGhinTdlU5iQl JkLKqBxcBz4L70LYWyHhYwYROJWjHgKAywX5T67ftq0wi8APuZl9olnOkwSK+wrY 1OZi =7epf -----END PGP MESSAGE----- qux: - foo - bar - | -----BEGIN PGP MESSAGE-----
hQEMAw2B674HRhwSAQgAg1YCmokrweoOI1c9HO0BLamWBaFPTMblOaTo0WJLZoTS ksbQ3OJAMkrkn3BnnM/djJc5C7vNs86ZfSJ+pvE8Sp1Rhtuxh25EKMqGOn/SBedI gR6N5vGUNiIpG5Tf3DuYAMNFDUqw8uY0MyDJI+ZW3o3xrMUABzTH0ew+Piz85FDA YrVgwZfqyL+9OQuu6T66jOIdwQNRX2NPFZqvon8liZUPus5VzD8E5cAL9OPxQ3sF f7/zE91YIXUTimrv3L7eCgU1dSxKhhfvA2bEUi+AskMWFXFuETYVrIhFJAKnkFmE uZx+O9R9hADW3hM5hWHKH9/CRtb0/cC84I9oCWIQPdI+AaPtICxtsD2N8Q98hhhd 4M7I0sLZhV+4ZJqzpUsOnSpaGyfh1Zy/1d3ijJi99/l+uVHuvmMllsNmgR+ZTj0= =LrCQ -----END PGP MESSAGE-----
When the pillar data is compiled, the results will be decrypted:
# salt myminion pillar.items myminion: ---------- secrets: ---------- vault: ---------- bar: this was unencrypted already baz: rosebud foo: supersecret qux: - foo - bar - baz
Salt must be told what portions of the pillar data to decrypt. This is done using the decrypt_pillar config option:
decrypt_pillar: - 'secrets:vault': gpg
The notation used to specify the pillar item(s) to be decrypted is the same as the one used in pillar.get function.
If a different delimiter is needed, it can be specified using the decrypt_pillar_delimiter config option:
decrypt_pillar: - 'secrets|vault': gpg
decrypt_pillar_delimiter: '|'
The name of the renderer used to decrypt a given pillar item can be omitted, and if so it will fall back to the value specified by the decrypt_pillar_default config option, which defaults to gpg.  So, the first example above could be rewritten as:
decrypt_pillar: - 'secrets:vault'
Encrypted Pillar Data on the CLI New in version 2016.3.0.
The following functions support passing pillar data on the CLI via the pillar argument:
• pillar.items
Triggerring decryption of this CLI pillar data can be done in one of two ways:
1. Using the pillar_enc argument:
# salt myminion pillar.items pillar_enc=gpg pillar='{foo: "-----BEGIN PGP MESSAGE-----\n\nhQEMAw2B674HRhwSAQf+OvPqEdDoA2fk15I5dYUTDoj1yf/pVolAma6iU4v8Zixn\nRDgWsaAnFz99FEiFACsAGDEFdZaVOxG80T0Lj+PnW4pVy0OXmXHnY2KjV9zx8FLS\nQxfvmhRR4t23WSFybozfMm0lsN8r1vfBBjbK+A72l0oxN78d1rybJ6PWNZiXi+aC\nmqIeunIbAKQ21w/OvZHhxH7cnIiGQIHc7N9nQH7ibyoKQzQMSZeilSMGr2abAHun\nmLzscr4wKMb+81Z0/fdBfP6g3bLWMJga3hSzSldU9ovu7KR8rDJI1qOlENj3Wm8C\nwTpDOB33kWIKMqiAjY3JFtb5MCHrafyggwQL7cX1+tI+AbSO6kZpbcDfzetb77LZ\nxc5NWnnGK4pGoqq4MAmZshw98RpecSHKMosto2gtiuWCuo9Zn5cV/FbjZ9CTWrQ=\n=0hO/\n-----END PGP MESSAGE-----"}'
The newlines in this example are specified using a literal \n. Newlines can be replaced with a literal \n using sed:
$ echo -n bar | gpg --armor --trust-model always --encrypt -r user@domain.tld | sed ':a;N;$!ba;s/\n/\\n/g'
NOTE: Using pillar_enc will perform the decryption minion-side, so for this to work it will be necessary to set up the keyring in /etc/salt/gpgkeys on the minion just as one would typically do on the master. The easiest way to do this is to first export the keys from the master:
# gpg --homedir /etc/salt/gpgkeys --export-secret-key -a user@domain.tld >/tmp/keypair.gpg
Then, copy the file to the minion, setup the keyring, and import:
# mkdir -p /etc/salt/gpgkeys # chmod 0700 /etc/salt/gpgkeys # gpg --homedir /etc/salt/gpgkeys --list-keys # gpg --homedir /etc/salt/gpgkeys --import --allow-secret-key-import keypair.gpg
The --list-keys command is run create a keyring in the newly-created directory.
Pillar data which is decrypted minion-side will still be securely transferred to the master, since the data sent between minion and master is encrypted with the master's public key.
2. Use the decrypt_pillar option. This is less flexible in that the pillar key passed on the CLI must be pre-configured on the master, but it doesn't require a keyring to be setup on the minion. One other caveat to this method is that pillar decryption on the master happens at the end of pillar compilation, so if the encrypted pillar data being passed on the CLI needs to be referenced by pillar or ext_pillar during pillar compilation, it must be decrypted minion-side.
Adding New Renderers for Decryption Those looking to add new renderers for decryption should look at the gpg renderer for an example of how to do so. The function that performs the decryption should be recursive and be able to traverse a mutable type such as a dictionary, and modify the values in-place.
Once the renderer has been written, decrypt_pillar_renderers should be modified so that Salt allows it to be used for decryption.
If the renderer is being submitted upstream to the Salt project, the renderer should be added in salt/renderers/. Additionally, the following should be done:
• Both occurrences of decrypt_pillar_renderers in salt/config/__init__.py should be updated to include the name of the new renderer so that it is included in the default value for this config option.
• The documentation for the decrypt_pillar_renderers config option in the master config file and minion config file should be updated to show the correct new default value.
• The commented example for the decrypt_pillar_renderers config option in the master config template should be updated to show the correct new default value.
Binary Data in the Pillar Salt has partial support for binary pillar data.
NOTE: There are some situations (such as salt-ssh) where only text (ASCII or Unicode) is allowed.
The simplest way to embed binary data in your pillar is to make use of YAML's built-in binary data type, which requires base64 encoded data.
salt_pic: !!binary iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAMAAAC67D+PAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAA
Then you can use it as a contents_pillar in a state:
/tmp/salt.png: file.managed: - contents_pillar: salt_pic
It is also possible to add ASCII-armored encrypted data to pillars, as mentioned in the Pillar Encryption section.
Master Config in Pillar For convenience the data stored in the master configuration file can be made available in all minion's pillars. This makes global configuration of services and systems very easy but may not be desired if sensitive data is stored in the master configuration. This option is disabled by default.
To enable the master config from being added to the pillar set pillar_opts to True in the minion config file:
pillar_opts: True
Minion Config in Pillar Minion configuration options can be set on pillars. Any option that you want to modify, should be in the first level of the pillars, in the same way you set the options in the config file. For example, to configure the MySQL root password to be used by MySQL Salt execution module, set the following pillar variable:
mysql.pass: hardtoguesspassword
Master Provided Pillar Error By default if there is an error rendering a pillar, the detailed error is hidden and replaced with:
The error is protected because it's possible to contain templating data which would give that minion information it shouldn't know, like a password!
To have the master provide the detailed error that could potentially carry protected data set pillar_safe_render_error to False:
pillar_safe_render_error: False
Pillar Walkthrough NOTE: This walkthrough assumes that the reader has already completed the initial Salt walkthrough.
Pillars are tree-like structures of data defined on the Salt Master and passed through to minions. They allow confidential, targeted data to be securely sent only to the relevant minion.
NOTE: Grains and Pillar are sometimes confused, just remember that Grains are data about a minion which is stored or generated from the minion. This is why information like the OS and CPU type are found in Grains. Pillar is information about a minion or many minions stored or generated on the Salt Master.
Pillar data is useful for:
Highly Sensitive Data: Information transferred via pillar is guaranteed to only be presented to the minions that are targeted, making Pillar suitable for managing security information, such as cryptographic keys and passwords.
Minion Configuration: Minion modules such as the execution modules, states, and returners can often be configured via data stored in pillar.
Variables: Variables which need to be assigned to specific minions or groups of minions can be defined in pillar and then accessed inside sls formulas and template files.
Arbitrary Data: Pillar can contain any basic data structure in dictionary format, so a key/value store can be defined making it easy to iterate over a group of values in sls formulas.
Pillar is therefore one of the most important systems when using Salt. This walkthrough is designed to get a simple Pillar up and running in a few minutes and then to dive into the capabilities of Pillar and where the data is available.
Setting Up Pillar The pillar is already running in Salt by default. To see the minion's pillar data:
salt '*' pillar.items
NOTE: Prior to version 0.16.2, this function is named pillar.data. This function name is still supported for backwards compatibility.
By default, the contents of the master configuration file are not loaded into pillar for all minions. This default is stored in the pillar_opts setting, which defaults to False.
The contents of the master configuration file can be made available to minion pillar files. This makes global configuration of services and systems very easy, but note that this may not be desired or appropriate if sensitive data is stored in the master's configuration file. To enable the master configuration file to be available to minion as pillar, set pillar_opts: True in the master configuration file, and then for appropriate minions also set pillar_opts: True in the minion(s) configuration file.
Similar to the state tree, the pillar is comprised of sls files and has a top file.  The default location for the pillar is in /srv/pillar.
NOTE: The pillar location can be configured via the pillar_roots option inside the master configuration file. It must not be in a subdirectory of the state tree or file_roots. If the pillar is under file_roots, any pillar targeting can be bypassed by minions.
To start setting up the pillar, the /srv/pillar directory needs to be present:
mkdir /srv/pillar
Now create a simple top file, following the same format as the top file used for states:
base: '*': - data
This top file associates the data.sls file to all minions. Now the /srv/pillar/data.sls file needs to be populated:
/srv/pillar/data.sls:
info: some data
To ensure that the minions have the new pillar data, issue a command to them asking that they fetch their pillars from the master:
Now that the minions have the new pillar, it can be retrieved:
The key info should now appear in the returned pillar data.
More Complex Data Unlike states, pillar files do not need to define formulas.  This example sets up user data with a UID:
/srv/pillar/users/init.sls:
users: thatch: 1000 shouse: 1001 utahdave: 1002 redbeard: 1003
NOTE: The same directory lookups that exist in states exist in pillar, so the file users/init.sls can be referenced with users in the top file.
The top file will need to be updated to include this sls file:
base: '*': - data - users
Now the data will be available to the minions. To use the pillar data in a state, you can use Jinja:
/srv/salt/users/init.sls
{% for user, uid in pillar.get('users', {}).items() %} {{user}}: user.present: - uid: {{uid}} {% endfor %}
This approach allows for users to be safely defined in a pillar and then the user data is applied in an sls file.
Parameterizing States With Pillar Pillar data can be accessed in state files to customise behavior for each minion. All pillar (and grain) data applicable to each minion is substituted into the state files through templating before being run. Typical uses include setting directories appropriate for the minion and skipping states that don't apply.
A simple example is to set up a mapping of package names in pillar for separate Linux distributions:
/srv/pillar/pkg/init.sls:
pkgs: {% if grains['os_family'] == 'RedHat' %} apache: httpd vim: vim-enhanced {% elif grains['os_family'] == 'Debian' %} apache: apache2 vim: vim {% elif grains['os'] == 'Arch' %} apache: apache vim: vim {% endif %}
The new pkg sls needs to be added to the top file:
base: '*': - data - users - pkg
Now the minions will auto map values based on respective operating systems inside of the pillar, so sls files can be safely parameterized:
/srv/salt/apache/init.sls:
apache: pkg.installed: - name: {{ pillar['pkgs']['apache'] }}
Or, if no pillar is available a default can be set as well:
NOTE: The function pillar.get used in this example was added to Salt in version 0.14.0
apache: pkg.installed: - name: {{ salt['pillar.get']('pkgs:apache', 'httpd') }}
In the above example, if the pillar value pillar['pkgs']['apache'] is not set in the minion's pillar, then the default of httpd will be used.
NOTE: Under the hood, pillar is just a Python dict, so Python dict methods such as get and items can be used.
Pillar Makes Simple States Grow Easily One of the design goals of pillar is to make simple sls formulas easily grow into more flexible formulas without refactoring or complicating the states.
A simple formula:
/srv/salt/edit/vim.sls:
vim: pkg.installed: []
/etc/vimrc: file.managed: - source: salt://edit/vimrc - mode: 644 - user: root - group: root - require: - pkg: vim
Can be easily transformed into a powerful, parameterized formula:
vim: pkg.installed: - name: {{ pillar['pkgs']['vim'] }}
/etc/vimrc: file.managed: - source: {{ pillar['vimrc'] }} - mode: 644 - user: root - group: root - require: - pkg: vim
Where the vimrc source location can now be changed via pillar:
/srv/pillar/edit/vim.sls:
{% if grains['id'].startswith('dev') %} vimrc: salt://edit/dev_vimrc {% elif grains['id'].startswith('qa') %} vimrc: salt://edit/qa_vimrc {% else %} vimrc: salt://edit/vimrc {% endif %}
Ensuring that the right vimrc is sent out to the correct minions.
The pillar top file must include a reference to the new sls pillar file:
base: '*': - pkg - edit.vim
Setting Pillar Data on the Command Line Pillar data can be set on the command line when running state.apply <salt.modules.state.apply_() like so:
salt '*' state.apply pillar='{"foo": "bar"}' salt '*' state.apply my_sls_file pillar='{"hello": "world"}'
Nested pillar values can also be set via the command line:
salt '*' state.sls my_sls_file pillar='{"foo": {"bar": "baz"}}'
Lists can be passed via command line pillar data as follows:
salt '*' state.sls my_sls_file pillar='{"some_list": ["foo", "bar", "baz"]}'
NOTE: If a key is passed on the command line that already exists on the minion, the key that is passed in will overwrite the entire value of that key, rather than merging only the specified value set via the command line.
The example below will swap the value for vim with telnet in the previously specified list, notice the nested pillar dict:
salt '*' state.apply edit.vim pillar='{"pkgs": {"vim": "telnet"}}'
This will attempt to install telnet on your minions, feel free to uninstall the package or replace telnet value with anything else.
More On Pillar Pillar data is generated on the Salt master and securely distributed to minions. Salt is not restricted to the pillar sls files when defining the pillar but can retrieve data from external sources. This can be useful when information about an infrastructure is stored in a separate location.
Reference information on pillar and the external pillar interface can be found in the Salt documentation:
Minion Config in Pillar Minion configuration options can be set on pillars. Any option that you want to modify, should be in the first level of the pillars, in the same way you set the options in the config file. For example, to configure the MySQL root password to be used by MySQL Salt execution module:
This is very convenient when you need some dynamic configuration change that you want to be applied on the fly. For example, there is a chicken and the egg problem if you do this:
mysql-admin-passwd: mysql_user.present: - name: root - password: somepasswd
mydb: mysql_db.present
The second state will fail, because you changed the root password and the minion didn't notice it. Setting mysql.pass in the pillar, will help to sort out the issue. But always change the root admin password in the first place.
This is very helpful for any module that needs credentials to apply state changes: mysql, keystone, etc.
Targeting Minions Targeting minions is specifying which minions should run a command or execute a state by matching against hostnames, or system information, or defined groups, or even combinations thereof.
For example the command salt web1 apache.signal restart to restart the Apache httpd server specifies the machine web1 as the target and the command will only be run on that one minion.
Similarly when using States, the following top file specifies that only the web1 minion should execute the contents of webserver.sls:
base: 'web1': - webserver
The simple target specifications, glob, regex, and list will cover many use cases, and for some will cover all use cases, but more powerful options exist.
Targeting with Grains The Grains interface was built into Salt to allow minions to be targeted by system properties. So minions running on a particular operating system can be called to execute a function, or a specific kernel.
Calling via a grain is done by passing the -G option to salt, specifying a grain and a glob expression to match the value of the grain. The syntax for the target is the grain key followed by a glob expression: "os:Arch*".
salt -G 'os:Fedora' test.version
Will return True from all of the minions running Fedora.
To discover what grains are available and what the values are, execute the grains.item salt function:
More info on using targeting with grains can be found here.
Compound Targeting New in version 0.9.5.
Multiple target interfaces can be used in conjunction to determine the command targets. These targets can then be combined using and or or statements.  This is well defined with an example:
salt -C 'G@os:Debian and webser* or E@db.*' test.version
In this example any minion who's id starts with webser and is running Debian, or any minion who's id starts with db will be matched.
The type of matcher defaults to glob, but can be specified with the corresponding letter followed by the @ symbol. In the above example a grain is used with G@ as well as a regular expression with E@. The webser* target does not need to be prefaced with a target type specifier because it is a glob.
More info on using compound targeting can be found here.
Node Group Targeting New in version 0.9.5.
For certain cases, it can be convenient to have a predefined group of minions on which to execute commands. This can be accomplished using what are called nodegroups. Nodegroups allow for predefined compound targets to be declared in the master configuration file, as a sort of shorthand for having to type out complicated compound expressions.
nodegroups: group1: 'L@foo.domain.com,bar.domain.com,baz.domain.com and bl*.domain.com' group2: 'G@os:Debian and foo.domain.com' group3: 'G@os:Debian and N@group1'
Advanced Targeting Methods There are many ways to target individual minions or groups of minions in Salt:
Matching the minion id Each minion needs a unique identifier. By default when a minion starts for the first time it chooses its FQDN as that identifier. The minion id can be overridden via the minion's id configuration setting.
TIP: minion id and minion keys
The minion id is used to generate the minion's public/private keys and if it ever changes the master must then accept the new key as though the minion was a new host.
Globbing The default matching that Salt utilizes is shell-style globbing around the minion id. This also works for states in the top file.
NOTE: You must wrap salt calls that use globbing in single-quotes to prevent the shell from expanding the globs before Salt is invoked.
Match all minions:
Match all minions in the example.net domain or any of the example domains:
salt '*.example.net' test.version salt '*.example.*' test.version
Match all the webN minions in the example.net domain (web1.example.net, web2.example.net … webN.example.net):
salt 'web?.example.net' test.version
Match the web1 through web5 minions:
salt 'web[1-5]' test.version
Match the web1 and web3 minions:
salt 'web[1,3]' test.version
Match the web-x, web-y, and web-z minions:
salt 'web-[x-z]' test.version
NOTE: For additional targeting methods please review the compound matchers documentation.
Regular Expressions Minions can be matched using Perl-compatible regular expressions (which is globbing on steroids and a ton of caffeine).
Match both web1-prod and web1-devel minions:
salt -E 'web1-(prod|devel)' test.version
When using regular expressions in a State's top file, you must specify the matcher as the first option. The following example executes the contents of webserver.sls on the above-mentioned minions.
base: 'web1-(prod|devel)': - match: pcre - webserver
Lists At the most basic level, you can specify a flat list of minion IDs:
salt -L 'web1,web2,web3' test.version
Targeting using Grains Grain data can be used when targeting minions.
For example, the following matches all CentOS minions:
salt -G 'os:CentOS' test.version
Match all minions with 64-bit CPUs, and return number of CPU cores for each matching minion:
salt -G 'cpuarch:x86_64' grains.item num_cpus
Additionally, globs can be used in grain matches, and grains that are nested in a dictionary can be matched by adding a colon for each level that is traversed.  For example, the following will match hosts that have a grain called ec2_tags, which itself is a dictionary with a key named environment, which has a value that contains the word production:
salt -G 'ec2_tags:environment:*production*'
Targeting using Pillar Pillar data can be used when targeting minions. This allows for ultimate control and flexibility when targeting minions.
NOTE: To start using Pillar targeting it is required to make a Pillar data cache on Salt Master for each Minion via following commands: salt '*' saltutil.refresh_pillar or salt '*' saltutil.sync_all.  Also Pillar data cache will be populated during the highstate run. Once Pillar data changes, you must refresh the cache by running above commands for this targeting method to work correctly.
salt -I 'somekey:specialvalue' test.version
Like with Grains, it is possible to use globbing as well as match nested values in Pillar, by adding colons for each level that is being traversed. The below example would match minions with a pillar named foo, which is a dict containing a key bar, with a value beginning with baz:
salt -I 'foo:bar:baz*' test.version
Subnet/IP Address Matching Minions can easily be matched based on IP address, or by subnet (using CIDR notation).
salt -S 192.168.40.20 test.version salt -S 2001:db8::/64 test.version
Ipcidr matching can also be used in compound matches
salt -C 'S@10.0.0.0/24 and G@os:Debian' test.version
It is also possible to use in both pillar and state-matching
'172.16.0.0/12': - match: ipcidr - internal
Compound matchers Compound matchers allow very granular minion targeting using any of Salt's matchers. The default matcher is a glob match, just as with CLI and top file matching. To match using anything other than a glob, prefix the match string with the appropriate letter from the table below, followed by an @ sign.
┌───────┬─────────────────┬──────────────────────────────────────────┬────────────────┐ │Letter │ Match Type      │ Example                                  │ Alt Delimiter? │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │G      │ Grains glob     │ G@os:Ubuntu                              │ Yes            │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │E      │ PCRE Minion ID  │ E@web\d+\.(dev|qa|prod)\.loc             │ No             │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │P      │ Grains PCRE     │ P@os:(RedHat|Fedora|CentOS)              │ Yes            │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │L      │ List of minions │ L@minion1.example.com,minion3.domain.com │ No             │ │       │                 │ or bl*.domain.com                        │                │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │I      │ Pillar glob     │ I@pdata:foobar                           │ Yes            │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │J      │ Pillar PCRE     │ J@pdata:^(foo|bar)$                      │ Yes            │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │S      │ Subnet/IP       │ S@192.168.1.0/24 or S@192.168.1.100      │ No             │ │       │ address         │                                          │                │ ├───────┼─────────────────┼──────────────────────────────────────────┼────────────────┤ │R      │ Range cluster   │ R@%foo.bar                               │ No             │ └───────┴─────────────────┴──────────────────────────────────────────┴────────────────┘
│N      │ Nodegroups      │ N@group1                                 │ No             │ └───────┴─────────────────┴──────────────────────────────────────────┴────────────────┘
Matchers can be joined using boolean and, or, and not operators.
For example, the following string matches all Debian minions with a hostname that begins with webserv, as well as any minions that have a hostname which matches the regular expression web-dc1-srv.*:
salt -C 'webserv* and G@os:Debian or E@web-dc1-srv.*' test.version
That same example expressed in a top file looks like the following:
base: 'webserv* and G@os:Debian or E@web-dc1-srv.*': - match: compound - webserver
Excluding a minion based on its ID is also possible:
salt -C 'not web-dc1-srv' test.version
Versions prior to 2015.8.0 a leading not was not supported in compound matches. Instead, something like the following was required:
salt -C '* and not G@kernel:Darwin' test.version
Excluding a minion based on its ID was also possible:
salt -C '* and not web-dc1-srv' test.version
Precedence Matching Matchers can be grouped together with parentheses to explicitly declare precedence amongst groups.
salt -C '( ms-1 or G@id:ms-3 ) and G@id:ms-3' test.version
NOTE: Be certain to note that spaces are required between the parentheses and targets. Failing to obey this rule may result in incorrect targeting!
Alternate Delimiters New in version 2015.8.0.
Matchers that target based on a key value pair use a colon (:) as a delimiter. Matchers with a Yes in the Alt Delimiters column in the previous table support specifying an alternate delimiter character.
This is done by specifying an alternate delimiter character between the leading matcher character and the @ pattern separator character. This avoids incorrect interpretation of the pattern in the case that : is part of the grain or pillar data structure traversal.
salt -C 'J|@foo|bar|^foo:bar$ or J!@gitrepo!https://github.com:example/project.git' test.ping
Node groups Nodegroups are declared using a compound target specification. The compound target documentation can be found here.
The nodegroups master config file parameter is used to define nodegroups. Here's an example nodegroup configuration within /etc/salt/master:
NOTE: The L within group1 is matching a list of minions, while the G in group2 is matching specific grains. See the compound matchers documentation for more details.
As of the 2017.7.0 release of Salt, group names can also be prepended with a dash. This brings the usage in line with many other areas of Salt. For example:
nodegroups: - group1: 'L@foo.domain.com,bar.domain.com,baz.domain.com or bl*.domain.com'
NOTE: Nodegroups can reference other nodegroups as seen in group3.  Ensure that you do not have circular references.  Circular references will be detected and cause partial expansion with a logged error message.
Compound nodegroups can be either string values or lists of string values.  When the nodegroup is A string value will be tokenized by splitting on whitespace.  This may be a problem if whitespace is necessary as part of a pattern.  When a nodegroup is a list of strings then tokenization will happen for each list element as a whole.
To match a nodegroup on the CLI, use the -N command-line option:
salt -N group1 test.version
NOTE: The N@ classifier historically could not be used in compound matches within the CLI or top file, it was only recognized in the nodegroups master config file parameter. As of the 2019.2.0 release, this limitation no longer exists.
To match a nodegroup in your top file, make sure to put - match: nodegroup on the line directly following the nodegroup name.
base: group1: - match: nodegroup - webserver
NOTE: When adding or modifying nodegroups to a master configuration file, the master must be restarted for those changes to be fully recognized.
A limited amount of functionality, such as targeting with -N from the command-line may be available without a restart.
Defining Nodegroups as Lists of Minion IDs A simple list of minion IDs would traditionally be defined like this:
nodegroups: group1: L@host1,host2,host3
They can now also be defined as a YAML list, like this:
nodegroups: group1: - host1 - host2 - host3
Batch Size The -b (or --batch-size) option allows commands to be executed on only a specified number of minions at a time. Both percentages and finite numbers are supported.
salt '*' -b 10 test.version
salt -G 'os:RedHat' --batch-size 25% apache.signal restart
This will only run test.version on 10 of the targeted minions at a time and then restart apache on 25% of the minions matching os:RedHat at a time and work through them all until the task is complete. This makes jobs like rolling web server restarts behind a load balancer or doing maintenance on BSD firewalls using carp much easier with salt.
The batch system maintains a window of running minions, so, if there are a total of 150 minions targeted and the batch size is 10, then the command is sent to 10 minions, when one minion returns then the command is sent to one additional minion, so that the job is constantly running on 10 minions.
New in version 2016.3.
The --batch-wait argument can be used to specify a number of seconds to wait after a minion returns, before sending the command to a new minion.
SECO Range SECO range is a cluster-based metadata store developed and maintained by Yahoo!
The Range project is hosted here:
https://github.com/ytoolshed/range
Learn more about range here:
https://github.com/ytoolshed/range/wiki/
Prerequisites To utilize range support in Salt, a range server is required. Setting up a range server is outside the scope of this document. Apache modules are included in the range distribution.
With a working range server, cluster files must be defined. These files are written in YAML and define hosts contained inside a cluster. Full documentation on writing YAML range files is here:
https://github.com/ytoolshed/range/wiki/%22yamlfile%22-module-file-spec