neulab/tldr huggingface dataset:
ext_pillar: - http_yaml: url: http://example.com/api/<nodename> with_grains: True
Module Documentation salt.pillar.http_yaml.ext_pillar(minion_id, pillar, url, with_grains=False) Read pillar data from HTTP response.
salt.pillar.libvirt Load up the libvirt keys into Pillar for a given minion if said keys have been generated using the libvirt key runner
depends certtool
salt.pillar.libvirt.ext_pillar(minion_id, pillar, command) Read in the generated libvirt keys
salt.pillar.libvirt.gen_hyper_keys(minion_id, country='US', state='Utah', locality='Salt Lake City', organization='Salted', expiration_days='365') Generate the keys to be used by libvirt hypervisors, this routine gens the keys and applies them to the pillar for the hypervisor minions
salt.pillar.makostack Simple and flexible YAML ext_pillar which can read pillar from within pillar.
This custom saltstack ext_pillar is a direct ripoff of the 'stack' ext_pillar, simply ported to use mako instead of jinja2 for templating.
It supports the following features:
• multiple config files that are mako templates with support for pillar, __grains__, __salt__, __opts__ objects.
• a config file renders as an ordered list of files. Unless absolute, the paths of these files are relative to the current config file - if absolute, they will be treated literally.
• this list of files are read in order as mako templates with support for stack, pillar, __grains__, __salt__, __opts__ objects.
• all these rendered files are then parsed as yaml.
• then all yaml dicts are merged in order, with support for the following.  merging strategies: merge-first, merge-last, remove, and overwrite.
• stack config files can be matched based on pillar, grains, or opts values, which make it possible to support kind of self-contained environments.
Configuration in Salt Like any other external pillar, its configuration takes place through the ext_pillar key in the master config file.
However, you can configure MakoStack in 3 different ways:
Single config file This is the simplest option, you just need to set the path to your single MakoStack config file like below:
ext_pillar: - makostack: /path/to/stack.cfg
List of config files You can also provide a list of config files:
ext_pillar: - makostack: - /path/to/stack1.cfg - /path/to/stack2.cfg
Select config files through grains|pillar|opts matching You can also opt for a much more flexible configuration: MakoStack allows one to select the config files for the current minion based on matching values from either grains, or pillar, or opts objects.
Here is an example of such a configuration, which should speak by itself:
ext_pillar: - makostack: pillar:environment: dev: /path/to/dev/stack.cfg prod: /path/to/prod/stack.cfg grains:custom:grain: value: - /path/to/stack1.cfg - /path/to/stack2.cfg opts:custom:opt: value: /path/to/stack0.cfg
Grafting data from files to arbitrary namespaces An extended syntax for config files permits defining "graft points" on a per-config-file basis.  As an example, if the file foo.cfg would produce the following:
foo: - bar - baz
and you specified the cfg file as /path/to/foo.cfg:yummy:fur, the following would actually end up in pillar after all merging was complete:
yummy: fur: foo: - bar - baz
MakoStack configuration files The config files that are referenced in the above ext_pillar configuration are mako templates which must render as a simple ordered list of yaml files that will then be merged to build pillar data.
Unless an absolute path name is specified, the path of these yaml files is assumed to be relative to the directory containing the MakoStack config file.  If a path begins with '/', however, it will be treated literally and can be anywhere on the filesystem.
The following variables are available in mako templating of makostack configuration files:
• pillar: the pillar data (as passed by Salt to our ext_pillar function)
• minion_id: the minion id ;-)
• __opts__: a dictionary of mostly Salt configuration options
• __grains__: a dictionary of the grains of the minion making this pillar call
• __salt__: a dictionary of Salt module functions, useful so you don't have to duplicate functions that already exist (note: runs on the master)
So you can use all the power of mako to build your list of yaml files that will be merged in pillar data.
For example, you could have a MakoStack config file which looks like:
$ cat /path/to/stack/config.cfg core.yml osarchs/%{ __grains__['osarch'] }}.yml oscodenames/%{ __grains__['oscodename'] }.yml % for role in pillar.get('roles', []): roles/%{ role }.yml % endfor minions/%{ minion_id }.yml
And the whole directory structure could look like:
$ tree /path/to/stack/ /path/to/stack/ ├── config.cfg ├── core.yml ├── osarchs/ │   ├── amd64.yml │   └── armhf.yml ├── oscodenames/ │   ├── wheezy.yml │   └── jessie.yml ├── roles/ │   ├── web.yml │   └── db.yml └── minions/ ├── test-1-dev.yml └── test-2-dev.yml
Overall process In the above MakoStack configuration, given that test-1-dev minion is an amd64 platform running Debian Jessie, and which pillar roles is ["db"], the following yaml files would be merged in order:
• core.yml
• osarchs/amd64.yml
• oscodenames/jessie.yml
• roles/db.yml
• minions/test-1-dev.yml
Before merging, every files above will be preprocessed as mako templates. The following variables are available in mako templating of yaml files:
• stack: the MakoStack pillar data object that has currently been merged (data from previous yaml files in MakoStack configuration)
So you can use all the power of mako to build your pillar data, and even use other pillar values that has already been merged by MakoStack (from previous yaml files in MakoStack configuration) through the stack variable.
Once a yaml file has been preprocessed by mako, we obtain a Python dict - let's call it yml_data - then, MakoStack will merge this yml_data dict in the main stack dict (which contains already merged MakoStack pillar data).  By default, MakoStack will deeply merge yml_data in stack (similarly to the recurse salt pillar_source_merging_strategy), but 3 merging strategies are currently available for you to choose (see next section).
Once every yaml files have been processed, the stack dict will contain your whole own pillar data, merged in order by MakoStack.  So MakoStack ext_pillar returns the stack dict, the contents of which Salt takes care to merge in with all of the other pillars and finally return the whole pillar to the minion.
Merging strategies The way the data from a new yaml_data dict is merged with the existing stack data can be controlled by specifying a merging strategy. Right now this strategy can either be merge-last (the default), merge-first, remove, or overwrite.
Note that scalar values like strings, integers, booleans, etc. are always evaluated using the overwrite strategy (other strategies don't make sense in that case).
The merging strategy can be set by including a dict in the form of:
__: <merging strategy>
as the first item of the dict or list.  This allows fine grained control over the merging process.
merge-last (default) strategy If the merge-last strategy is selected (the default), then content of dict or list variables is merged recursively with previous definitions of this variable (similarly to the recurse salt pillar_source_merging_strategy).  This allows for extending previously defined data.
merge-first strategy If the merge-first strategy is selected, then the content of dict or list variables are swapped between the yaml_data and stack objects before being merged recursively with the merge-last previous strategy.
remove strategy If the remove strategy is selected, then content of dict or list variables in stack are removed only if the corresponding item is present in the yaml_data dict.  This allows for removing items from previously defined data.
overwrite strategy If the overwrite strategy is selected, then the content of dict or list variables in stack is overwritten by the content of yaml_data dict.  So this allows one to overwrite variables from previous definitions.
Merging examples Let's go through small examples that should clarify what's going on when a yaml_data dict is merged in the stack dict.
When you don't specify any strategy, the default merge-last strategy is selected:
┌───────────────────────────┬─────────────────────────────┬─────────────────────────────┐
│stack                      │ yaml_data                   │ stack (after merge)         │
├───────────────────────────┼─────────────────────────────┼─────────────────────────────┤
│                           │                             │                             │
│          users:           │           users:            │           users:            │
│            tom:           │             tom:            │             tom:            │
│              uid: 500     │               uid: 1000     │               uid: 1000     │
│              roles:       │               roles:        │               roles:        │
│                - sysadmin │                 - developer │                 - sysadmin  │
│            root:          │             mat:            │                 - developer │
│              uid: 0       │               uid: 1001     │             mat:            │
│                           │                             │               uid: 1001     │
│                           │                             │             root:           │
│                           │                             │               uid: 0        │
└───────────────────────────┴─────────────────────────────┴─────────────────────────────┘
Then you can select a custom merging strategy using the __ key in a dict:
You can also select a custom merging strategy using a __ object in a list:
┌───────────────────┬───────────────────────────────┬─────────────────────┐ │stack              │ yaml_data                     │ stack (after merge) │ ├───────────────────┼───────────────────────────────┼─────────────────────┤ │                   │                               │                     │ │          users:   │           users:              │           users:    │ │            - tom  │             - __: merge-last  │             - tom   │ │            - root │             - mat             │             - root  │ │                   │                               │             - mat   │ ├───────────────────┼───────────────────────────────┼─────────────────────┤ │                   │                               │                     │ │          users:   │           users:              │           users:    │ │            - tom  │             - __: merge-first │             - mat   │ │            - root │             - mat             │             - tom   │ │                   │                               │             - root  │ ├───────────────────┼───────────────────────────────┼─────────────────────┤ │                   │                               │                     │ │          users:   │           users:              │           users:    │ │            - tom  │             - __: remove      │             - root  │ │            - root │             - mat             │                     │ │                   │             - tom             │                     │ ├───────────────────┼───────────────────────────────┼─────────────────────┤ │                   │                               │                     │ │          users:   │           users:              │           users:    │ │            - tom  │             - __: overwrite   │             - mat   │ │            - root │             - mat             │                     │ └───────────────────┴───────────────────────────────┴─────────────────────┘
salt.pillar.makostack.ext_pillar(minion_id, pillar, *args, **kwargs)
salt.pillar.mongo Read Pillar data from a mongodb collection
depends pymongo (for salt-master)
This module will load a node-specific pillar dictionary from a mongo collection. It uses the node's id for lookups and can load either the whole document, or just a specific field from that document as the pillar dictionary.
Salt Master Mongo Configuration The module shares the same base mongo connection variables as salt.returners.mongo_return. These variables go in your master config file.
• mongo.db - The mongo database to connect to. Defaults to 'salt'.
• mongo.host - The mongo host to connect to. Supports replica sets by specifying all hosts in the set, comma-delimited. Defaults to 'salt'.
• mongo.port - The port that the mongo database is running on. Defaults to 27017.
• mongo.user - The username for connecting to mongo. Only required if you are using mongo authentication. Defaults to ''.
• mongo.password - The password for connecting to mongo. Only required if you are using mongo authentication. Defaults to ''.
Configuring the Mongo ext_pillar The Mongo ext_pillar takes advantage of the fact that the Salt Master configuration file is yaml. It uses a sub-dictionary of values to adjust specific features of the pillar. This is the explicit single-line dictionary notation for yaml. One may be able to get the easier-to-read multi-line dict to work correctly with some experimentation.
ext_pillar: - mongo: {collection: vm, id_field: name, re_pattern: \.example\.com, fields: [customer_id, software, apache_vhosts]}
In the example above, we've decided to use the vm collection in the database to store the data. Minion ids are stored in the name field on documents in that collection. And, since minion ids are FQDNs in most cases, we'll need to trim the domain name in order to find the minion by hostname in the collection. When we find a minion, return only the customer_id, software, and apache_vhosts fields, as that will contain the data we want for a given node. They will be available directly inside the pillar dict in your SLS templates.
Module Documentation salt.pillar.mongo.ext_pillar(minion_id, pillar, collection='pillar', id_field='_id', re_pattern=None, re_replace='', fields=None) Connect to a mongo database and read per-node pillar information.
• collection (*) -- The mongodb collection to read data from. Defaults to 'pillar'.
• id_field (*) -- The field in the collection that represents an individual minion id. Defaults to '_id'.
• re_pattern (*) -- If your naming convention in the collection is shorter than the minion id, you can use this to trim the name.  re_pattern will be used to match the name, and re_replace will be used to replace it. Backrefs are supported as they are in the Python standard library. If None, no mangling of the name will be performed - the collection will be searched with the entire minion id. Defaults to None.
• re_replace (*) -- Use as the replacement value in node ids matched with re_pattern. Defaults to ''. Feel free to use backreferences here.
• fields (*) -- The specific fields in the document to use for the pillar data. If None, will use the entire document. If using the entire document, the _id field will be converted to string. Be careful with other fields in the document as they must be string serializable. Defaults to None.
salt.pillar.mysql Retrieve Pillar data by doing a MySQL query
MariaDB provides Python support through the MySQL Python package. Therefore, you may use this module with both MySQL or MariaDB.
This module is a concrete implementation of the sql_base ext_pillar for MySQL.
Configuring the mysql ext_pillar Use the 'mysql' key under ext_pillar for configuration of queries.
MySQL configuration of the MySQL returner is being used (mysql.db, mysql.user, mysql.pass, mysql.port, mysql.host) for database connection info.
Complete example mysql: user: 'salt' pass: 'super_secret_password' db: 'salt_db' port: 3306 ssl: cert: /etc/mysql/client-cert.pem key: /etc/mysql/client-key.pem
ext_pillar: - mysql: fromdb: query: 'SELECT col1,col2,col3,col4,col5,col6,col7 FROM some_random_table WHERE minion_pattern LIKE %s' depth: 5 as_list: True with_lists: [1,3]
class salt.pillar.mysql.MySQLExtPillar This class receives and processes the database rows from MySQL.
extract_queries(args, kwargs) This function normalizes the config block into a set of queries we can use.  The return is a list of consistently laid out dicts.
salt.pillar.mysql.ext_pillar(minion_id, pillar, *args, **kwargs) Execute queries against MySQL, merge and return as a dict
salt.pillar.nacl Decrypt pillar data through the builtin NACL renderer
nacl.config: keyfile: /root/.nacl
ext_pillar: - stack: /path/to/stack.cfg - nacl: {}
Set nacl.config in your config.
salt.pillar.nacl.ext_pillar(minion_id, pillar, *args, **kwargs)
salt.pillar.netbox A module that adds data to the Pillar structure from a NetBox API.
Configuring the NetBox ext_pillar To use this pillar, you must first create a token in your NetBox instance at http://netbox.example.com/user/api-tokens/ (substituting the hostname of your NetBox instance)
The NetBox api_url and api_token must be set in the master config.
For example /etc/salt/master.d/netbox.conf:
ext_pillar: - netbox: api_url: http://netbox.example.com/api/ api_token: 123abc
The following options are optional, and determine whether or not the module will attempt to configure the proxy pillar data for use with the napalm proxy-minion:
proxy_return: True proxy_username: admin
By default, this module will query the NetBox API for the platform associated with the device, and use the 'NAPALM driver' field to set the napalm proxy-minion driver. (Currently only 'napalm' is supported for drivertype.)
This module currently only supports the napalm proxy minion and assumes you will use SSH keys to authenticate to the network device.  If password authentication is desired, it is recommended to create another proxy key in pillar_roots (or git_pillar) with just the passwd key and use salt.renderers.gpg to encrypt the value.
If you use more than one username for your devices, leave proxy_username unset, and set the username key in your pillar as well. If any additional options for the proxy setup are needed, they should also be configured in pillar_roots.
Other available configuration options:
site_details: True Whether should retrieve details of the site the device belongs to.
site_prefixes: True Whether should retrieve the prefixes of the site the device belongs to.
devices: True New in version 3004.0.
Whether should retrieve physical devices.
virtual_machines: False New in version 3004.0.
Whether should retrieve virtual machines.
interfaces: False New in version 3004.0.
Whether should retrieve the interfaces of the device.
interface_ips: False New in version 3004.0.
Whether should retrieve the IP addresses for interfaces of the device.  (interfaces must be set to True as well)
api_query_result_limit: Use NetBox default New in version 3004.0.
An integer specifying how many results should be returned for each query to the NetBox API. Leaving this unset will use NetBox's default value.
Note that each option you enable can have a detrimental impact on pillar performance, so use them with caution.
After configuring the pillar, you must restart the Salt master for the changes to take effect.
systemctl restart salt-master
To query perform a quick test of the pillar, you should refresh the pillar on the minion with the following:
salt minion1 saltutil.refresh_pillar
And then query the pillar:
salt minion1 pillar.items 'netbox'
salt.pillar.netbox.ext_pillar(minion_id, pillar, *args, **kwargs) Query NetBox API for minion data
salt.pillar.neutron Use Openstack Neutron data as a Pillar source. Will list all networks listed inside of Neutron, to all minions.
• python-neutronclient
A keystone profile must be used for the pillar to work (no generic keystone configuration here). For example:
my openstack_config: keystone.user: 'admin' keystone.password: 'password' keystone.tenant: 'admin' keystone.auth_url: 'http://127.0.0.1:5000/v2.0/' keystone.region_name: 'RegionOne' keystone.service_type: 'network'
After the profile is created, configure the external pillar system to use it.
ext_pillar: - neutron: my_openstack_config
Using these configuration profiles, multiple neutron sources may also be used:
ext_pillar: - neutron: my_openstack_config - neutron: my_other_openstack_config
By default, these networks will be returned as a pillar item called networks. In order to have them returned under a different name, add the name after the Keystone profile name:
• neutron: my_openstack_config neutron_networks
salt.pillar.neutron.ext_pillar(minion_id, pillar, conf) Check neutron for all data
salt.pillar.nodegroups Nodegroups Pillar Introspection: to which nodegroups does my minion belong?  Provides a pillar with the default name of nodegroups which contains a list of nodegroups which match for a given minion.
Command Line salt-call pillar.get nodegroups local: - class_infra - colo_sj - state_active - country_US - type_saltmaster
Configuring Nodegroups Pillar extension_modules: /srv/salt/ext ext_pillar: - nodegroups: pillar_name: 'nodegroups'
salt.pillar.nodegroups.ext_pillar(minion_id, pillar, pillar_name=None) A salt external pillar which provides the list of nodegroups of which the minion is a member.
• minion_id -- used for compound matching nodegroups
• pillar -- provided by salt, but not used by nodegroups ext_pillar
• pillar_name -- optional name to use for the pillar, defaults to 'nodegroups'
Returns a dictionary which is included by the salt master in the pillars returned to the minion
salt.pillar.pepa Pepa Configuration templating for SaltStack using Hierarchical substitution and Jinja.
Configuring Pepa extension_modules: /srv/salt/ext
ext_pillar: - pepa: resource: host                # Name of resource directory and sub-key in pillars sequence:                     # Sequence used for hierarchical substitution - hostname:                 # Name of key name: input             # Alias used for template directory base_only: True         # Only use templates from Base environment, i.e. no staging - default: - environment: - location..region: name: region - location..country: name: country - location..datacenter: name: datacenter - roles: - osfinger: name: os - hostname: name: override base_only: True subkey: True                  # Create a sub-key in pillars, named after the resource in this case [host] subkey_only: True             # Only create a sub-key, and leave the top level untouched
pepa_roots:                         # Base directory for each environment base: /srv/pepa/base              # Path for base environment dev: /srv/pepa/base               # Associate dev with base qa: /srv/pepa/qa prod: /srv/pepa/prod
# Use a different delimiter for nested dictionaries, defaults to '..' since some keys may use '.' in the name #pepa_delimiter: ..
# Supply Grains for Pepa, this should **ONLY** be used for testing or validation #pepa_grains: #  environment: dev
# Supply Pillar for Pepa, this should **ONLY** be used for testing or validation #pepa_pillars: #  saltversion: 0.17.4
# Enable debug for Pepa, and keep Salt on warning #log_level: debug
#log_granular_levels: #  salt: warning #  salt.loaded.ext.pillar.pepa: debug
Pepa can also be used in Master-less SaltStack setup.
Command line usage: pepa.py [-h] [-c CONFIG] [-d] [-g GRAINS] [-p PILLAR] [-n] [-v] hostname
positional arguments: hostname              Hostname
optional arguments: -h, --help            show this help message and exit -c CONFIG, --config CONFIG Configuration file -d, --debug           Print debug info -g GRAINS, --grains GRAINS Input Grains as YAML -p PILLAR, --pillar PILLAR Input Pillar as YAML -n, --no-color        No color output -v, --validate        Validate output
Templates Templates is configuration for a host or software, that can use information from Grains or Pillars. These can then be used for hierarchically substitution.
Example File: host/input/test_example_com.yaml
location..region: emea location..country: nl location..datacenter: foobar environment: dev roles: - salt.master network..gateway: 10.0.0.254 network..interfaces..eth0..hwaddr: 00:20:26:a1:12:12 network..interfaces..eth0..dhcp: False network..interfaces..eth0..ipv4: 10.0.0.3 network..interfaces..eth0..netmask: 255.255.255.0 network..interfaces..eth0..fqdn: {{ hostname }} cobbler..profile: fedora-19-x86_64
As you see in this example you can use Jinja directly inside the template.
Example File: host/region/amer.yaml
network..dns..servers: - 10.0.0.1 - 10.0.0.2 time..ntp..servers: - ntp1.amer.example.com - ntp2.amer.example.com - ntp3.amer.example.com time..timezone: America/Chihuahua yum..mirror: yum.amer.example.com
Each template is named after the value of the key using lowercase and all extended characters are replaced with underscore.
osfinger: Fedora-19
Would become:
fedora_19.yaml
Nested dictionaries In order to create nested dictionaries as output you can use double dot ".." as a delimiter. You can change this using "pepa_delimiter" we choose double dot since single dot is already used by key names in some modules, and using ":" requires quoting in the YAML.
network..dns..servers: - 10.0.0.1 - 10.0.0.2 network..dns..options: - timeout:2 - attempts:1 - ndots:1 network..dns..search: - example.com
network: dns: servers: - 10.0.0.1 - 10.0.0.2 options: - timeout:2 - attempts:1 - ndots:1 search: - example.com
Operators Operators can be used to merge/unset a list/hash or set the key as immutable, so it can't be changed.
┌────────────┬───────────────────────────┐ │Operator    │ Description               │ ├────────────┼───────────────────────────┤ │merge()     │ Merge list or hash        │ ├────────────┼───────────────────────────┤ │unset()     │ Unset key                 │ ├────────────┼───────────────────────────┤ │immutable() │ Set the key as immutable, │ │            │ so it can't be changed    │ ├────────────┼───────────────────────────┤ │imerge()    │ Set immutable and merge   │ ├────────────┼───────────────────────────┤ │iunset()    │ Set immutable and unset   │ └────────────┴───────────────────────────┘
network..dns..search..merge(): - foobar.com - dummy.nl owner..immutable(): Operations host..printers..unset():
Validation Since it's very hard to test Jinja as is, the best approach is to run all the permutations of input and validate the output, i.e. Unit Testing.
To facilitate this in Pepa we use YAML, Jinja and Cerberus <‐ https://github.com/nicolaiarocci/cerberus>.
Schema So this is a validation schema for network configuration, as you see it can be customized with Jinja just as Pepa templates.
This was designed to be run as a build job in Jenkins or similar tool. You can provide Grains/Pillar input using either the config file or command line arguments.
File Example: host/validation/network.yaml
network..dns..search: type: list allowed: - example.com
network..dns..options: type: list allowed: ['timeout:2', 'attempts:1', 'ndots:1']
network..dns..servers: type: list schema: regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$
network..gateway: type: string regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$
{% if network.interfaces is defined %} {% for interface in network.interfaces %}
network..interfaces..{{ interface }}..dhcp: type: boolean
network..interfaces..{{ interface }}..fqdn: type: string regex: ^([a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?\.)+[a-zA-Z]{2,6}$
network..interfaces..{{ interface }}..hwaddr: type: string regex: ^([0-9a-f]{1,2}\:){5}[0-9a-f]{1,2}$
network..interfaces..{{ interface }}..ipv4: type: string regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$
network..interfaces..{{ interface }}..netmask: type: string regex: ^([0-9]{1,3}\.){3}[0-9]{1,3}$
{% endfor %} {% endif %}
Links For more examples and information see <https://github.com/mickep76/pepa>.
salt.pillar.pepa.ext_pillar(minion_id, pillar, resource, sequence, subkey=False, subkey_only=False) Evaluate Pepa templates
salt.pillar.pepa.key_value_to_tree(data) Convert key/value to tree
salt.pillar.pepa.validate(output, resource) Validate Pepa templates
salt.pillar.pillar_ldap Use LDAP data as a Pillar source
This pillar module executes a series of LDAP searches.  Data returned by these searches are aggregated, whereby data returned by later searches override data by previous searches with the same key.
The final result is merged with existing pillar data.
The configuration of this external pillar module is done via an external file which provides the actual configuration for the LDAP searches.
Configuring the LDAP ext_pillar The basic configuration is part of the master configuration.
ext_pillar: - pillar_ldap: /etc/salt/master.d/pillar_ldap.yaml
NOTE: When placing the file in the master.d directory, make sure its name doesn't end in .conf, otherwise the salt-master process will attempt to parse its content.
WARNING: Make sure this file has very restrictive permissions, as it will contain possibly sensitive LDAP credentials!
The only required key in the master configuration is pillar_ldap pointing to a file containing the actual configuration.
Configuring the LDAP searches The file is processed using Salt's Renderers <renderers> which makes it possible to reference grains within the configuration.
WARNING: When using Jinja in this file, make sure to do it in a way which prevents leaking sensitive information. A rogue minion could send arbitrary grains to trick the master into returning secret data.  Use only the 'id' grain which is verified through the minion's key/cert.
Map Mode The it-admins configuration below returns the Pillar it-admins by:
• filtering for: - members of the group it-admins - objects with objectclass=user
• returning the data of users, where each user is a dictionary containing the configured string or list attributes.
Configuration salt-users: server:    ldap.company.tld port:      389 tls:       true dn:        'dc=company,dc=tld' binddn:    'cn=salt-pillars,ou=users,dc=company,dc=tld' bindpw:    bi7ieBai5Ano referrals: false anonymous: false mode:      map dn:        'ou=users,dc=company,dc=tld' filter:    '(&(memberof=cn=it-admins,ou=groups,dc=company,dc=tld)(objectclass=user))' attrs: - cn - displayName - givenName - sn lists: - memberOf
search_order: - salt-users
Result { 'salt-users': [ { 'cn': 'cn=johndoe,ou=users,dc=company,dc=tld', 'displayName': 'John Doe' 'givenName': 'John' 'sn': 'Doe' 'memberOf': [ 'cn=it-admins,ou=groups,dc=company,dc=tld', 'cn=team01,ou=groups,dc=company' ] }, { 'cn': 'cn=janedoe,ou=users,dc=company,dc=tld', 'displayName': 'Jane Doe', 'givenName': 'Jane', 'sn': 'Doe', 'memberOf': [ 'cn=it-admins,ou=groups,dc=company,dc=tld', 'cn=team02,ou=groups,dc=company' ] } ] }
salt.pillar.pillar_ldap.ext_pillar(minion_id, pillar, config_file) Execute LDAP searches and return the aggregated data
salt.pillar.postgres Retrieve Pillar data by doing a postgres query
Complete Example postgres: user: 'salt' pass: 'super_secret_password' db: 'salt_db'
ext_pillar: - postgres: fromdb: query: 'SELECT col1,col2,col3,col4,col5,col6,col7 FROM some_random_table WHERE minion_pattern LIKE %s' depth: 5 as_list: True with_lists: [1,3]
class salt.pillar.postgres.POSTGRESExtPillar This class receives and processes the database rows from POSTGRES.
salt.pillar.postgres.ext_pillar(minion_id, pillar, *args, **kwargs) Execute queries against POSTGRES, merge and return as a dict
salt.pillar.puppet Execute an unmodified puppet_node_classifier and read the output as YAML. The YAML data is then directly overlaid onto the minion's Pillar data.
salt.pillar.puppet.ext_pillar(minion_id, pillar, command) Execute an unmodified puppet_node_classifier and read the output as YAML
salt.pillar.reclass_adapter Use the "reclass" database as a Pillar source
This ext_pillar plugin provides access to the reclass database, such that Pillar data for a specific minion are fetched using reclass.
You can find more information about reclass at http://reclass.pantsfullofunix.net.
To use the plugin, add it to the ext_pillar list in the Salt master config and tell reclass by way of a few options how and where to find the inventory:
ext_pillar: - reclass: storage_type: yaml_fs inventory_base_uri: /srv/salt
This would cause reclass to read the inventory from YAML files in /srv/salt/nodes and /srv/salt/classes.
If you are also using reclass as master_tops plugin, and you want to avoid having to specify the same information for both, use YAML anchors (take note of the differing data types for ext_pillar and master_tops):
reclass: &reclass storage_type: yaml_fs inventory_base_uri: /srv/salt reclass_source_path: ~/code/reclass
ext_pillar: - reclass: *reclass
master_tops: reclass: *reclass
If you want to run reclass from source, rather than installing it, you can either let the master know via the PYTHONPATH environment variable, or by setting the configuration option, like in the example above.
salt.pillar.reclass_adapter.ext_pillar(minion_id, pillar, **kwargs) Obtain the Pillar data from reclass for the given minion_id.
salt.pillar.redismod Read pillar data from a Redis backend New in version 2014.7.0.
• redis Python module (on master)
Salt Master Redis Configuration The module shares the same base Redis connection variables as salt.returners.redis_return. These variables go in your master config file.
• redis.db - The Redis database to use. Defaults to 0.
• redis.host - The Redis host to connect to. Defaults to 'salt'.
• redis.port - The port that the Redis database is listening on. Defaults to 6379.
• redis.password - The password for authenticating with Redis. Only required if you are using master auth. Defaults to None.
Configuring the Redis ext_pillar ext_pillar: - redis: {function: key_value}
salt.pillar.redismod.ext_pillar(minion_id, pillar, function, **kwargs) Grabs external pillar data based on configured function
salt.pillar.redismod.key_json(minion_id, pillar, pillar_key=None) Pulls a string from redis and deserializes it from json. Deserialized dictionary data loaded directly into top level if pillar_key is not set.
pillar_key Pillar key to return data into
salt.pillar.redismod.key_value(minion_id, pillar, pillar_key='redis_pillar') Looks for key in redis matching minion_id, returns a structure based on the data type of the redis key. String for string type, dict for hash type and lists for lists, sets and sorted sets.
salt.pillar.rethinkdb_pillar Provide external pillar data from RethinkDB
depends rethinkdb (on the salt-master)
salt master rethinkdb configuration These variables must be configured in your master configuration file.
• rethinkdb.host - The RethinkDB server. Defaults to 'salt'
• rethinkdb.port - The port the RethinkDB server listens on. Defaults to '28015'
• rethinkdb.database - The database to connect to.  Defaults to 'salt'
• rethinkdb.username - The username for connecting to RethinkDB. Defaults to ''
• rethinkdb.password - The password for connecting to RethinkDB. Defaults to ''
salt-master ext_pillar configuration The ext_pillar function arguments are given in single line dictionary notation.
ext_pillar: - rethinkdb: {table: ext_pillar, id_field: minion_id, field: pillar_root, pillar_key: external_pillar}
In the example above the following happens.
• The salt-master will look for external pillars in the 'ext_pillar' table on the RethinkDB host
• The minion id will be matched against the 'minion_id' field
• Pillars will be retrieved from the nested field 'pillar_root'
• Found pillars will be merged inside a key called 'external_pillar'
Module Documentation salt.pillar.rethinkdb_pillar.ext_pillar(minion_id, pillar, table='pillar', id_field=None, field=None, pillar_key=None) Collect minion external pillars from a RethinkDB database
• table: The RethinkDB table containing external pillar information.  Defaults to 'pillar'
• id_field: Field in document containing the minion id.  If blank then we assume the table index matches minion ids
• field: Specific field in the document used for pillar data, if blank then the entire document will be used
• pillar_key: The salt-master will nest found external pillars under this key before merging into the minion pillars. If blank, external pillars will be merged at top level
salt.pillar.s3 Copy pillar data from a bucket in Amazon S3
The S3 pillar can be configured in the master config file with the following options
ext_pillar: - s3: bucket: my.fancy.pillar.bucket keyid: KASKFJWAKJASJKDAJKSD key: ksladfDLKDALSFKSD93q032sdDasdfasdflsadkf multiple_env: False environment: base prefix: somewhere/overthere verify_ssl: True service_url: s3.amazonaws.com kms_keyid: 01234567-89ab-cdef-0123-4567890abcde s3_cache_expire: 30 s3_sync_on_update: True path_style: False https_enable: True
The bucket parameter specifies the target S3 bucket. It is required.
The keyid parameter specifies the key id to use when access the S3 bucket.  If it is not provided, an attempt to fetch it from EC2 instance meta-data will be made.
The key parameter specifies the key to use when access the S3 bucket. If it is not provided, an attempt to fetch it from EC2 instance meta-data will be made.
The multiple_env defaults to False. It specifies whether the pillar should interpret top level folders as pillar environments (see mode section below).
The environment defaults to 'base'. It specifies which environment the bucket represents when in single environments mode (see mode section below). It is ignored if multiple_env is True.
The prefix defaults to ''. It specifies a key prefix to use when searching for data in the bucket for the pillar. It works when multiple_env is True or False.  Essentially it tells ext_pillar to look for your pillar data in a 'subdirectory' of your S3 bucket
The verify_ssl parameter defaults to True. It specifies whether to check for valid S3 SSL certificates. NOTE If you use bucket names with periods, this must be set to False else an invalid certificate error will be thrown (issue #12200).
The service_url parameter defaults to 's3.amazonaws.com'. It specifies the base url to use for accessing S3.
The kms_keyid parameter is optional. It specifies the ID of the Key Management Service (KMS) master key that was used to encrypt the object.
The s3_cache_expire parameter defaults to 30s. It specifies expiration time of S3 metadata cache file.
The s3_sync_on_update parameter defaults to True. It specifies if cache is synced on update rather than jit.
The path_style parameter defaults to False. It specifies whether to use path style requests or dns style requests
The https_enable parameter defaults to True. It specifies whether to use https protocol or http protocol
This pillar can operate in two modes, single environment per bucket or multiple environments per bucket.
Single environment mode must have this bucket structure:
s3://<bucket name>/<prefix>/<files>
Multiple environment mode must have this bucket structure:
s3://<bucket name>/<prefix>/<environment>/<files>
If you wish to define your pillar data entirely within S3 it's recommended that you use the prefix= parameter and specify one entry in ext_pillar for each environment rather than specifying multiple_env. This is due to issue #22471 (https://github.com/saltstack/salt/issues/22471)
class salt.pillar.s3.S3Credentials(key, keyid, bucket, service_url, verify_ssl=True, kms_keyid=None, location=None, path_style=False, https_enable=True)
salt.pillar.s3.ext_pillar(minion_id, pillar, bucket, key=None, keyid=None, verify_ssl=True, location=None, multiple_env=False, environment='base', prefix='', service_url=None, kms_keyid=None, s3_cache_expire=30, s3_sync_on_update=True, path_style=False, https_enable=True) Execute a command and read the output as YAML
salt.pillar.saltclass SaltClass Pillar Module ext_pillar: - saltclass: - path: /srv/saltclass
For additional configuration instructions, see the saltclass module
salt.pillar.saltclass.ext_pillar(minion_id, pillar, *args, **kwargs) Compile pillar data
salt.pillar.sql_base Retrieve Pillar data by doing a SQL query
This module is not meant to be used directly as an ext_pillar.  It is a place to put code common to PEP 249 compliant SQL database adapters.  It exposes a python ABC that can be subclassed for new database providers.
Theory of sql_base ext_pillar Ok, here's the theory for how this works...
• First, any non-keyword args are processed in order.
• Then, remaining keywords are processed.
We do this so that it's backward compatible with older configs.  Keyword arguments are sorted before being appended, so that they're predictable, but they will always be applied last so overall it's moot.
For each of those items we process, it depends on the object type:
• Strings are executed as is and the pillar depth is determined by the number of fields returned.
• A list has the first entry used as the query, the second as the pillar depth.
• A mapping uses the keys "query" and "depth" as the tuple
You can retrieve as many fields as you like, how they get used depends on the exact settings.
Configuring a sql_base ext_pillar The sql_base ext_pillar cannot be used directly, but shares query configuration with its implementations. These examples use a fake 'sql_base' adapter, which should be replaced with the name of the adapter you are using.
A list of queries can be passed in
ext_pillar: - sql_base: - "SELECT pillar,value FROM pillars WHERE minion_id = %s" - "SELECT pillar,value FROM more_pillars WHERE minion_id = %s"
Or you can pass in a mapping
ext_pillar: - sql_base: main: "SELECT pillar,value FROM pillars WHERE minion_id = %s" extras: "SELECT pillar,value FROM more_pillars WHERE minion_id = %s"
The query can be provided as a string as we have just shown, but they can be provided as lists
ext_pillar: - sql_base: - "SELECT pillar,value FROM pillars WHERE minion_id = %s" 2
Or as a mapping
ext_pillar: - sql_base: - query: "SELECT pillar,value FROM pillars WHERE minion_id = %s" depth: 2
The depth defines how the dicts are constructed.  Essentially if you query for fields a,b,c,d for each row you'll get:
• With depth 1: {a: {"b": b, "c": c, "d": d}}
• With depth 2: {a: {b: {"c": c, "d": d}}}
• With depth 3: {a: {b: {c: d}}}
Depth greater than 3 wouldn't be different from 3 itself.  Depth of 0 translates to the largest depth needed, so 3 in this case.  (max depth == key count - 1)
Then they are merged in a similar way to plain pillar data, in the order returned by the SQL database.
Thus subsequent results overwrite previous ones when they collide.
The ignore_null option can be used to change the overwrite behavior so that only non-NULL values in subsequent results will overwrite.  This can be used to selectively overwrite default values.
ext_pillar: - sql_base: - query: "SELECT pillar,value FROM pillars WHERE minion_id = 'default' and minion_id != %s" depth: 2 - query: "SELECT pillar,value FROM pillars WHERE minion_id = %s" depth: 2 ignore_null: True
If you specify as_list: True in the mapping expression it will convert collisions to lists.
If you specify with_lists: '...' in the mapping expression it will convert the specified depths to list.  The string provided is a sequence numbers that are comma separated.  The string '1,3' will result in:
a,b,c,d,e,1  # field 1 same, field 3 differs a,b,c,f,g,2  # ^^^^ a,z,h,y,j,3  # field 1 same, field 3 same a,z,h,y,k,4  # ^^^^ ^   ^
These columns define list grouping
{a: [ {c: [ {e: 1}, {g: 2} ] }, {h: [ {j: 3, k: 4 } ] } ]}
The range for with_lists is 1 to number_of_fields, inclusive.  Numbers outside this range are ignored.
Finally, if you pass the queries in via a mapping, the key will be the first level name where as passing them in as a list will place them in the root.  This isolates the query results into their own subtrees.  This may be a help or hindrance to your aims and can be used as such.
You can basically use any SELECT query that gets you the information, you could even do joins or subqueries in case your minion_id is stored elsewhere.  It is capable of handling single rows or multiple rows per minion.
Configuration of the connection depends on the adapter in use.
More complete example for MySQL (to also show configuration) mysql: user: 'salt' pass: 'super_secret_password' db: 'salt_db'
class salt.pillar.sql_base.SqlBaseExtPillar This class receives and processes the database rows in a database agnostic way.
as_list = False
depth = 0
enter_root(root) Set self.focus for kwarg queries
fetch(minion_id, pillar, *args, **kwargs) Execute queries, merge and return as a dict.
field_names = None
focus = None
ignore_null = False
num_fields = 0
process_fields(field_names, depth) The primary purpose of this function is to store the sql field list and the depth to which we process.
process_results(rows) This function takes a list of database results and iterates over, merging them into a dict form.
result = None
with_lists = None
salt.pillar.sqlcipher Retrieve Pillar data by running a SQLCipher query
Python SQLCipher support is provided by the pysqlcipher Python package. You need this module installed to query Pillar data from a SQLCipher database.
This module is a concrete implementation of the sql_base ext_pillar for SQLCipher.
depends pysqlcipher (for py2) or pysqlcipher3 (for py3)
Configuring the sqlcipher ext_pillar Use the 'sqlcipher' key under ext_pillar for configuration of queries.
SQLCipher database connection configuration requires the following values configured in the master config:
• sqlcipher.database - The SQLCipher database to connect to.  Defaults to '/var/lib/salt/pillar-sqlcipher.db'.
• sqlcipher.pass - The SQLCipher database decryption password.
• sqlcipher.timeout - The connection timeout in seconds.
sqlcipher: database: /var/lib/salt/pillar-sqlcipher.db pass: strong_pass_phrase timeout: 5.0
Complete Example sqlcipher: database: '/var/lib/salt/pillar-sqlcipher.db' pass: strong_pass_phrase timeout: 5.0
ext_pillar: - sqlcipher: fromdb: query: 'SELECT col1,col2,col3,col4,col5,col6,col7 FROM some_random_table WHERE minion_pattern LIKE ?' depth: 5 as_list: True with_lists: [1,3]
class salt.pillar.sqlcipher.SQLCipherExtPillar This class receives and processes the database rows from SQLCipher.
salt.pillar.sqlcipher.ext_pillar(minion_id, pillar, *args, **kwargs) Execute queries against SQLCipher, merge and return as a dict
salt.pillar.sqlite3 Retrieve Pillar data by doing a SQLite3 query
sqlite3 is included in the stdlib since Python 2.5.
This module is a concrete implementation of the sql_base ext_pillar for SQLite3.
Configuring the sqlite3 ext_pillar Use the 'sqlite3' key under ext_pillar for configuration of queries.
SQLite3 database connection configuration requires the following values configured in the master config:
Note, timeout is in seconds.
sqlite3.database: /var/lib/salt/pillar.db sqlite3.timeout: 5.0
Complete Example sqlite3: database: '/var/lib/salt/pillar.db' timeout: 5.0
ext_pillar: - sqlite3: fromdb: query: 'SELECT col1,col2,col3,col4,col5,col6,col7 FROM some_random_table WHERE minion_pattern LIKE ?' depth: 5 as_list: True with_lists: [1,3]
class salt.pillar.sqlite3.SQLite3ExtPillar This class receives and processes the database rows from SQLite3.
salt.pillar.sqlite3.ext_pillar(minion_id, pillar, *args, **kwargs) Execute queries against SQLite3, merge and return as a dict
salt.pillar.stack Simple and flexible YAML ext_pillar which can read pillar from within pillar.
PillarStack is a custom saltstack ext_pillar which was inspired by varstack but is heavily based on Jinja2 for maximum flexibility.
Any issue should be reported to the upstream project at: https://github.com/bbinet/pillarstack/issues
• multiple config files that are jinja2 templates with support for pillar, __grains__, __salt__, __opts__ objects
• a config file renders as an ordered list of files (paths of these files are relative to the current config file)
• this list of files are read in ordered as jinja2 templates with support for stack, pillar, __grains__, __salt__, __opts__ objects
• all these rendered files are then parsed as yaml
• then all yaml dicts are merged in order with support for the following merging strategies: merge-first, merge-last, remove, and overwrite
• stack config files can be matched based on pillar, grains, or opts values, which make it possible to support kind of self-contained environments
Installation PillarStack is already bundled with Salt since 2016.3.0 version so there is nothing to install from version 2016.3.0.
If you use an older Salt version or you want to override PillarStack with a more recent one, follow the installation procedure below.
Installing the PillarStack ext_pillar is as simple as dropping the stack.py file in the <extension_modules>/pillar directory (no external python module required), given that extension_modules is set in your salt-master configuration, see: https://docs.saltproject.io/en/latest/ref/configuration/master.html#extension-modules
However, you can configure PillarStack in 3 different ways:
Single config file This is the simplest option, you just need to set the path to your single PillarStack config file like below:
ext_pillar: - stack: /path/to/stack.cfg
ext_pillar: - stack: - /path/to/stack1.cfg - /path/to/stack2.cfg