neulab/tldr huggingface dataset:
< {'local': True} > ----------------- \ \ \ ▄▄▄▄▄▄▄ ▀▄▄████▄▄ ▄▄▄█████▄█▄█▄█▄▄▄ ██████▄▄▄█▄▄█████▄▄ ▀▄▀ █████▄▄█▄▄█████ ▄▄▄███████████▄▄▄ ████▄▄▄▄▄▄███▄▄██           ▄▄▄▄▄▄▄ ████▄████▄██▄▄███       ▄▄▄▄██▄▄▄▄▄▄ █▄███▄▄█▄███▄▄██▄▀     ▄▄███████▄▄███▄▄ ▀▄██████████████▄▄    ▄▄█▄▀▀▀▄▄█████▄▄██ ▀▀▀▀▀█████▄█▄█▄▄▄▄▄▄▄█     ▀▄████▄████ ████▄███▄▄▄▄▄▄▄▄▄     ▄▄█████▄███ ▀▄█▄█▄▄▄██▄▄▄▄▄██    ▄▄██▄██████ ▀▄████████████▄▀  ▄▄█▄██████▄▀ ██▄██▄▄▄▄█▄███▄ ███▄▄▄▄▄██▄▀ ██████  ▀▄▄█████ ▀████████ ▄▄▄▄███   ███████ ██████▄█▄▄ ███████   ████████▀▄▀███▄▄█▄▄ ▄██▄▄████   ████████   ▀▄██▀▄▄▀ █▄▄██████   █▄▄██████ █▄▄▄▄█       █▄▄▄▄█
salt '*' foo.bar --out=pony
salt.output.pony.output(data, **kwargs) Mane function
salt.output.pprint_out Python pretty-print (pprint) The python pretty-print system was once the default outputter. It simply passes the return data through to pprint.pformat and prints the results.
salt '*' foo.bar --out=pprint
{'saltmine': {'foo': {'bar': 'baz', 'dictionary': {'abc': 123, 'def': 456}, 'list': ['Hello', 'World']}}}
salt.output.pprint_out.output(data, **kwargs) Print out via pretty print
salt.output.profile Display profiling data in a table format Show profile data for returners that would normally show a highstate output.
salt '*' state.apply something --out=profile
Attempt to output the returns of state.sls and state.highstate as a table of names, modules and durations that looks somewhat like the following:
name                mod.fun                duration (ms) -------------------------------------------------------- I-fail-unless-stmt  other.function               -1.0000 old-minion-config   grains.list_present           1.1200 salt-data           group.present                48.3800 /etc/salt/minion    file.managed                 63.1450
To get the above appearance, use settings something like these:
out.table.separate_rows: False out.table.justify: left out.table.delim: '  ' out.table.prefix: '' out.table.suffix: ''
salt.output.profile.output(data, **kwargs) Display the profiling data in a table format.
salt.output.progress Display return data as a progress bar
salt.output.progress.output(ret, bar, **kwargs) Update the progress bar
salt.output.progress.progress_iter(progress) Initialize and return a progress bar iter
salt.output.raw Display raw output data structure This outputter simply displays the output as a python data structure, by printing a string representation of it. It is similar to the pprint outputter, only the data is not nicely formatted/indented.
This was the original outputter used by Salt before the outputter system was developed.
salt '*' foo.bar --out=raw
salt '*' foo.bar --out=table {'myminion': {'foo': {'list': ['Hello', 'World'], 'bar': 'baz', 'dictionary': {'abc': 123, 'def': 456}}}}
salt.output.raw.output(data, **kwargs) Rather basic....
salt.output.table_out Display output in a table format New in version 2017.7.0.
The table outputter displays a sequence of rows as table.
edge01.bjm01: ---------- comment: ---------- out: ---------- ______________________________________________________________________________ | Active | Interface | Last Move |        Mac        | Moves | Static | Vlan | ______________________________________________________________________________ |  True  |  ae1.900  |    0.0    | 40:A6:77:5A:50:01 |   0   | False  | 111  | ______________________________________________________________________________ |  True  |  ae1.111  |    0.0    | 64:16:8D:32:26:58 |   0   | False  | 111  | ______________________________________________________________________________ |  True  |  ae1.111  |    0.0    | 8C:60:4F:73:2D:57 |   0   | False  | 111  | ______________________________________________________________________________ |  True  |  ae1.111  |    0.0    | 8C:60:4F:73:2D:7C |   0   | False  | 111  | ______________________________________________________________________________ |  True  |  ae1.222  |    0.0    | 8C:60:4F:73:2D:57 |   0   | False  | 222  | ______________________________________________________________________________ |  True  |  ae1.222  |    0.0    | F4:0F:1B:76:9D:97 |   0   | False  | 222  | ______________________________________________________________________________ result: ----------
salt '*' foo.bar --out=table
class salt.output.table_out.TableDisplay(has_header=True, row_delimiter='-', delim=' | ', justify='center', separate_rows=True, prefix='| ', suffix=' |', width=50, wrapfunc=None) Manage the table display content.
display(ret, indent, out, rows_key=None, labels_key=None) Display table(s).
display_rows(rows, labels, indent) Prepares row content and displays.
prepare_rows(rows, indent, has_header) Prepare rows content to be displayed.
ustring(indent, color, msg, prefix='', suffix='', endc=None) Build the unicode string to be displayed.
wrap_onspace(text) When the text inside the column is longer then the width, will split by space and continue on the next line.
salt.output.table_out.output(ret, **kwargs) Display the output as table.
• nested_indent (*) -- integer, specify the left alignment.
• has_header (*) -- boolean specifying if header should be displayed. Default: True.
• row_delimiter (*) -- character to separate rows. Default: _.
• delim (*) -- character to separate columns. Default: " | ".
• justify (*) -- text alignment. Default: center.
• separate_rows (*) -- boolean specifying if row separator will be displayed between consecutive rows. Default: True.
• prefix (*) -- character at the beginning of the row. Default: "| ".
• suffix (*) -- character at the end of the row. Default: " |".
• width (*) -- column max width. Default: 50.
• rows_key (*) -- display the rows under a specific key.
• labels_key (*) -- use the labels under a certain key. Otherwise will try to use the dictionary keys (if any).
• title (*) -- display title when only one table is selected (using the rows_key argument).
salt.output.txt Simple text outputter The txt outputter has been developed to make the output from shell commands on minions appear as they do when the command is executed on the minion.
salt '*' foo.bar --out=txt
salt.output.txt.output(data, **kwargs) Output the data in lines, very nice for running commands
salt.output.virt_query virt.query outputter Used to display the output from the virt.query runner.
salt.output.virt_query.output(data, **kwargs) Display output for the salt-run virt.query function
salt.output.yaml_out Display return data in YAML format This outputter defaults to printing in YAML block mode for better readability.
salt '*' foo.bar --out=yaml
saltmine: foo: bar: baz dictionary: abc: 123 def: 456 list: - Hello - World
salt.output.yaml_out.output(data, **kwargs) Print out YAML using the block mode
│rethinkdb_pillar            │ Provide external pillar     │ │                            │ data from RethinkDB         │ ├────────────────────────────┼─────────────────────────────┤ │s3                          │ Copy pillar data from a     │ │                            │ bucket in Amazon S3         │ ├────────────────────────────┼─────────────────────────────┤ │saltclass                   │ SaltClass Pillar Module     │ ├────────────────────────────┼─────────────────────────────┤ │sql_base                    │ Retrieve Pillar data by     │ │                            │ doing a SQL query           │ ├────────────────────────────┼─────────────────────────────┤ │sqlcipher                   │ Retrieve Pillar data by     │ │                            │ running a SQLCipher query   │ ├────────────────────────────┼─────────────────────────────┤ │sqlite3                     │ Retrieve Pillar data by     │ │                            │ doing a SQLite3 query       │ ├────────────────────────────┼─────────────────────────────┤ │stack                       │ Simple and flexible YAML    │ │                            │ ext_pillar which can read   │ │                            │ pillar from within pillar.  │ ├────────────────────────────┼─────────────────────────────┤ │svn_pillar                  │ Clone a remote SVN          │ │                            │ repository and use the      │ │                            │ filesystem as a Pillar      │ │                            │ source                      │ ├────────────────────────────┼─────────────────────────────┤ │varstack_pillar             │ Use Varstack data as a      │ │                            │ Pillar source               │ ├────────────────────────────┼─────────────────────────────┤ │vault                       │ Vault Pillar Module         │ ├────────────────────────────┼─────────────────────────────┤ │venafi                      │ Venafi Pillar Certificates  │ ├────────────────────────────┼─────────────────────────────┤ │virtkey                     │ Accept a key from a         │ │                            │ hypervisor if the virt      │ │                            │ runner has already          │ │                            │ submitted an authorization  │ │                            │ request                     │ ├────────────────────────────┼─────────────────────────────┤ │vmware_pillar               │ Pillar data from vCenter or │ │                            │ an ESXi host                │ └────────────────────────────┴─────────────────────────────┘
salt.pillar.azureblob Use Azure Blob as a Pillar source.
• azure-storage-blob >= 12.0.0
The Azure Blob ext_pillar can be configured with the following parameters:
ext_pillar: - azureblob: container: 'test_container' connection_string: 'connection_string' multiple_env: False environment: 'base' blob_cache_expire: 30 blob_sync_on_update: True
param container The name of the target Azure Blob Container.
param connection_string The connection string to use to access the specified Azure Blob Container.
param multiple_env Specifies whether the pillar should interpret top level folders as pillar environments.  Defaults to false.
param environment Specifies which environment the container represents when in single environment mode. Defaults to 'base' and is ignored if multiple_env is set as True.
param blob_cache_expire Specifies expiration time of the Azure Blob metadata cache file. Defaults to 30s.
param blob_sync_on_update Specifies if the cache is synced on update. Defaults to True.
salt.pillar.azureblob.ext_pillar(minion_id, pillar, container, connection_string, multiple_env=False, environment='base', blob_cache_expire=30, blob_sync_on_update=True) Execute a command and read the output as YAML.
• container -- The name of the target Azure Blob Container.
• connection_string -- The connection string to use to access the specified Azure Blob Container.
• multiple_env -- Specifies whether the pillar should interpret top level folders as pillar environments. Defaults to false.
• environment -- Specifies which environment the container represents when in single environment mode. Defaults to 'base' and is ignored if multiple_env is set as True.
• blob_cache_expire -- Specifies expiration time of the Azure Blob metadata cache file. Defaults to 30s.
• blob_sync_on_update -- Specifies if the cache is synced on update. Defaults to True.
salt.pillar.cmd_json Execute a command and read the output as JSON. The JSON data is then directly overlaid onto the minion's Pillar data.
salt.pillar.cmd_json.ext_pillar(minion_id, pillar, command) Execute a command and read the output as JSON
salt.pillar.cmd_yaml Execute a command and read the output as YAML. The YAML data is then directly overlaid onto the minion's Pillar data
salt.pillar.cmd_yaml.ext_pillar(minion_id, pillar, command) Execute a command and read the output as YAML
salt.pillar.cmd_yamlex Execute a command and read the output as YAMLEX.
The YAMLEX data is then directly overlaid onto the minion's Pillar data
salt.pillar.cmd_yamlex.ext_pillar(minion_id, pillar, command) Execute a command and read the output as YAMLEX
salt.pillar.cobbler A module to pull data from Cobbler via its API into the Pillar dictionary
Configuring the Cobbler ext_pillar The same cobbler.* parameters are used for both the Cobbler tops and Cobbler pillar modules.
ext_pillar: - cobbler: key: cobbler # Nest results within this key. By default, values are not nested. only: [parameters] # Add only these keys to pillar.
cobbler.url: https://example.com/cobbler_api #default is http://localhost/cobbler_api cobbler.user: username # default is no username cobbler.password: password # default is no password
Module Documentation salt.pillar.cobbler.ext_pillar(minion_id, pillar, key=None, only=()) Read pillar data from Cobbler via its API.
salt.pillar.confidant An external pillar module for getting credentials from confidant.
Configuring the Confidant module The module can be configured via ext_pillar in the minion config:
ext_pillar:
confidant:
profile: # The URL of the confidant web service url: '‐ https://confidant-production.example.com' # The context to use for KMS authentication auth_context: from: example-production-iad to: confidant-production-iad user_type: service # The KMS master key to use for authentication auth_key: "alias/authnz" # Cache file for KMS auth token token_cache_file: /run/confidant/confidant_token # The duration of the validity of a token, in minutes token_duration: 60 # key, keyid and region can be defined in the profile, but it's # generally best to use IAM roles or environment variables for AWS # auth.  keyid: 98nh9h9h908h09kjjk key: jhf908gyeghehe0he0g8h9u0j0n0n09hj09h0 region: us-east-1
depends confidant-common, confidant-client
Module Documentation salt.pillar.confidant.ext_pillar(minion_id, pillar, profile=None) Read pillar data from Confidant via its API.
salt.pillar.consul_pillar Use Consul K/V as a Pillar source with values parsed as YAML
• python-consul
In order to use an consul server, a profile must be created in the master configuration file:
my_consul_config: consul.host: 127.0.0.1 consul.port: 8500 consul.token: b6376760-a8bb-edd5-fcda-33bc13bfc556 consul.scheme: http consul.consistency: default consul.dc: dev consul.verify: True
The consul.token requires python-consul >= 0.4.7.
If you have a multi-datacenter Consul cluster you can map your pillarenv entries to your data centers by providing a dictionary of mappings in consul.dc field:
my_consul_config: consul.dc: dev: us-east-1 prod: us-west-1
In the example above we specifying static mapping between Pillar environments and data centers: the data for dev and prod Pillar environments will be fetched from us-east-1 and us-west-1 datacenter respectively.
In fact when consul.dc is set to dictionary keys are processed as regular expressions (that can capture named parameters) and values are processed as string templates as per PEP 3101.
my_consul_config: consul.dc: ^dev-.*$: dev-datacenter ^(?P<region>.*)-prod$: prod-datacenter-{region}
This example maps all Pillar environments starting with dev- to dev-datacenter whereas Pillar environment like eu-prod will be mapped to prod-datacenter-eu.
Before evaluation patterns are sorted by length in descending order.
If Pillar environment names correspond to data center names a single pattern can be used:
my_consul_config: consul.dc: ^(?P<env>.*)$: '{env}'
After the profile is created, configure the external pillar system to use it.  Optionally, a root may be specified.
ext_pillar: - consul: my_consul_config
ext_pillar: - consul: my_consul_config root=salt
Using these configuration profiles, multiple consul sources may also be used:
ext_pillar: - consul: my_consul_config - consul: my_other_consul_config
Either the minion_id, or the role, or the environment grain  may be used in the root path to expose minion-specific information stored in consul.
ext_pillar: - consul: my_consul_config root=salt/%(minion_id)s - consul: my_consul_config root=salt/%(role)s - consul: my_consul_config root=salt/%(environment)s
Minion-specific values may override shared values when the minion-specific root appears after the shared root:
ext_pillar: - consul: my_consul_config root=salt-shared - consul: my_other_consul_config root=salt-private/%(minion_id)s
If using the role or environment grain in the consul key path, be sure to define it using /etc/salt/grains, or similar:
role: my-minion-role environment: dev
It's possible to lock down where the pillar values are shared through minion targeting. Note that double quotes " are required around the target value and cannot be used inside the matching statement. See the section on Compound Matchers for more examples.
ext_pillar: - consul: my_consul_config root=salt target="L@salt.example.com and G@osarch:x86_64"
The data from Consul can be merged into a nested key in Pillar.
ext_pillar: - consul: my_consul_config pillar_root=consul_data
By default, keys containing YAML data will be deserialized before being merged into Pillar.  This behavior can be disabled by setting expand_keys to false.
ext_pillar: - consul: my_consul_config expand_keys=false
salt.pillar.consul_pillar.consul_fetch(client, path) Query consul for all keys/values within base path
salt.pillar.consul_pillar.ext_pillar(minion_id, pillar, conf) Check consul for all data
salt.pillar.consul_pillar.fetch_tree(client, path, expand_keys) Grab data from consul, trim base path and remove any keys which are folders. Take the remaining data and send it to be formatted in such a way as to be used as pillar data.
salt.pillar.consul_pillar.get_conn(opts, profile) Return a client object for accessing consul
salt.pillar.consul_pillar.pillar_format(ret, keys, value, expand_keys) Perform data formatting to be used as pillar data and merge it with the current pillar data
salt.pillar.csvpillar Store key/value pairs in a CSV file
Example configuration:
ext_pillar: - csv: /path/to/file.csv
ext_pillar: - csv: path: /path/to/file.csv namespace: 'subkey' fieldnames: - col1 - col2 - col2
The first column must be minion IDs and the first row must be dictionary keys.  E.g.:
┌───────┬──────┬───────┐ │id     │ role │ env   │ ├───────┼──────┼───────┤ │jerry  │ web  │ prod  │ ├───────┼──────┼───────┤ │stuart │ web  │ stage │ ├───────┼──────┼───────┤ │dave   │ web  │ qa    │ ├───────┼──────┼───────┤ │phil   │ db   │ prod  │ ├───────┼──────┼───────┤ │kevin  │ db   │ stage │ ├───────┼──────┼───────┤ │mike   │ db   │ qa    │ └───────┴──────┴───────┘
Will produce the following Pillar values for a minion named "jerry":
{ 'role': 'web', 'env': 'prod', }
salt.pillar.csvpillar.ext_pillar(mid, pillar, path, idkey='id', namespace=None, fieldnames=None, restkey=None, restval=None, dialect='excel') Read a CSV into Pillar
• path (str) -- Absolute path to a CSV file.
• idkey (str) -- (Optional) The column name of minion IDs.
• namespace (str) -- (Optional) A pillar key to namespace the values under.
• fieldnames (list) -- (Optional) if the first row of the CSV is not column names they may be specified here instead.
salt.pillar.digicert Digicert Pillar Certificates
This module will only return pillar data if the digicert runner module has already been used to create certificates.
To configure this module, set digicert to True in the ext_pillar section of your master configuration file:
ext_pillar: - digicert: True
salt.pillar.digicert.ext_pillar(minion_id, pillar, conf) Return an existing set of certificates
salt.pillar.django_orm Generate Pillar data from Django models through the Django ORM
maintainer Micah Hausler <micah.hausler@gmail.com>
Configuring the django_orm ext_pillar To use this module, your Django project must be on the salt master server with database access. This assumes you are using virtualenv with all the project's requirements installed.
ext_pillar: - django_orm: pillar_name: my_application project_path: /path/to/project/ settings_module: my_application.settings env_file: /path/to/env/file.sh # Optional: If your project is not using the system python, # add your virtualenv path below. env: /path/to/virtualenv/
django_app:
# Required: the app that is included in INSTALLED_APPS my_application.clients:
# Required: the model name Client:
# Required: model field to use as the key in the rendered # Pillar. Must be unique; must also be included in the # ``fields`` list below. name: shortname
# Optional: # See Django's QuerySet documentation for how to use .filter() filter:  {'kw': 'args'}
# Required: a list of field names # List items will be used as arguments to the .values() method. # See Django's QuerySet documentation for how to use .values() fields: - field_1 - field_2
This would return pillar data that would look like
my_application: my_application.clients: Client: client_1: field_1: data_from_field_1 field_2: data_from_field_2 client_2: field_1: data_from_field_1 field_2: data_from_field_2
As another example, data from multiple database tables can be fetched using Django's regular lookup syntax. Note, using ManyToManyFields will not currently work since the return from values() changes if a ManyToMany is present.
ext_pillar: - django_orm: pillar_name: djangotutorial project_path: /path/to/mysite settings_module: mysite.settings
django_app: mysite.polls: Choices: name: poll__question fields: - poll__question - poll__id - choice_text - votes
Module Documentation salt.pillar.django_orm.ext_pillar(minion_id, pillar, pillar_name, project_path, settings_module, django_app, env=None, env_file=None, *args, **kwargs) Connect to a Django database through the ORM and retrieve model fields
• pillar_name (str) -- The name of the pillar to be returned
• project_path (str) -- The full path to your Django project (the directory manage.py is in)
• settings_module (str) -- The settings module for your project. This can be found in your manage.py file
• django_app (str) -- A dictionary containing your apps, models, and fields
• env (str) -- The full path to the virtualenv for your Django project
• env_file (str) -- An optional bash file that sets up your environment. The file is run in a subprocess and the changed variables are then added
salt.pillar.ec2_pillar Retrieve EC2 instance data for minions for ec2_tags and ec2_tags_list
The minion id must be the AWS instance-id or value in tag_match_key.  For example set tag_match_key to Name to have the minion-id matched against the tag 'Name'. The tag contents must be unique. The value of tag_match_value can be 'uqdn' or 'asis'. if 'uqdn', then the domain will be stripped before comparison.
Additionally, the use_grain option can be set to True. This allows the use of an instance-id grain instead of the minion-id. Since this is a potential security risk, the configuration can be further expanded to include a list of minions that are trusted to only allow the alternate id of the instances to specific hosts. There is no glob matching at this time.
NOTE: If you are using use_grain: True in the configuration for this external pillar module, the minion must have metadata_server_grains enabled in the minion config file (see also here).
It is important to also note that enabling the use_grain option allows the minion to manipulate the pillar data returned, as described above.
The optional tag_list_key indicates which keys should be added to ec2_tags_list and be split by tag_list_sep (by default ;). If a tag key is included in tag_list_key it is removed from ec2_tags. If a tag does not exist it is still included as an empty list.
NOTE: As with any master configuration change, restart the salt-master daemon for changes to take effect.
ext_pillar: - ec2_pillar: tag_match_key: 'Name' tag_match_value: 'asis' tag_list_key: - Role tag_list_sep: ';' use_grain: True minion_ids: - trusted-minion-1 - trusted-minion-2 - trusted-minion-3
This is a very simple pillar configuration that simply retrieves the instance data from AWS. Currently the only portion implemented are EC2 tags, which returns a list of key/value pairs for all of the EC2 tags assigned to the instance.
salt.pillar.ec2_pillar.ext_pillar(minion_id, pillar, use_grain=False, minion_ids=None, tag_match_key=None, tag_match_value='asis', tag_list_key=None, tag_list_sep=';') Execute a command and read the output as YAML
salt.pillar.etcd_pillar Use etcd data as a Pillar source
In order to use an etcd server, a profile must be created in the master configuration file:
my_etcd_config: etcd.host: 127.0.0.1 etcd.port: 4001
ext_pillar: - etcd: my_etcd_config
ext_pillar: - etcd: my_etcd_config root=/salt
Using these configuration profiles, multiple etcd sources may also be used:
ext_pillar: - etcd: my_etcd_config - etcd: my_other_etcd_config
The minion_id may be used in the root path to expose minion-specific information stored in etcd.
ext_pillar: - etcd: my_etcd_config root=/salt/%(minion_id)s
ext_pillar: - etcd: my_etcd_config root=/salt-shared - etcd: my_other_etcd_config root=/salt-private/%(minion_id)s
Using the configuration above, the following commands could be used to share a key with all minions but override its value for a specific minion:
etcdctl set /salt-shared/mykey my_value etcdctl set /salt-private/special_minion_id/mykey my_other_value
salt.pillar.etcd_pillar.ext_pillar(minion_id, pillar, conf) Check etcd for all data
salt.pillar.extra_minion_data_in_pillar Add all extra minion data to the pillar.
codeauthor Alexandru.Bleotu@morganstanley.ms.com
One can filter on the keys to include in the pillar by using the include parameter. For subkeys the ':' notation is supported (i.e. 'key:subkey') The keyword <all> includes all keys.
Complete example in etc/salt/master ext_pillar: - extra_minion_data_in_pillar: include: *
ext_pillar: - extra_minion_data_in_pillar: include: - key1 - key2:subkey2
ext_pillar: - extra_minion_data_in_pillar: include: <all>
salt.pillar.extra_minion_data_in_pillar.ext_pillar(minion_id, pillar, include, extra_minion_data=None)
salt.pillar.file_tree The file_tree external pillar allows values from all files in a directory tree to be imported as Pillar data.
NOTE: This is an external pillar and is subject to the rules and constraints governing external pillars.
In this pillar, data is organized by either Minion ID or Nodegroup name. To setup pillar data for a specific Minion, place it in <root_dir>/hosts/<minion_id>.  To setup pillar data for an entire Nodegroup, place it in <root_dir>/nodegroups/<node_group> where <node_group> is the Nodegroup's name.
Example file_tree Pillar Master Configuration ext_pillar: - file_tree: root_dir: /srv/ext_pillar follow_dir_links: False keep_newline: True
The root_dir parameter is required and points to the directory where files for each host are stored. The follow_dir_links parameter is optional and defaults to False. If follow_dir_links is set to True, this external pillar will follow symbolic links to other directories.
WARNING: Be careful when using follow_dir_links, as a recursive symlink chain will result in unexpected results.
Changed in version 2018.3.0: If root_dir is a relative path, it will be treated as relative to the pillar_roots of the environment specified by pillarenv. If an environment specifies multiple roots, this module will search for files relative to all of them, in order, merging the results.
If keep_newline is set to True, then the pillar values for files ending in newlines will keep that newline. The default behavior is to remove the end-of-file newline. keep_newline should be turned on if the pillar data is intended to be used to deploy a file using contents_pillar with a file.managed state.
Changed in version 2015.8.4: The raw_data parameter has been renamed to keep_newline. In earlier releases, raw_data must be used. Also, this parameter can now be a list of globs, allowing for more granular control over which pillar values keep their end-of-file newline. The globs match paths relative to the directories named for minion IDs and nodegroups underneath the root_dir (see the layout examples in the below sections).
ext_pillar: - file_tree: root_dir: /path/to/root/directory keep_newline: - files/testdir/*
NOTE: In earlier releases, this documentation incorrectly stated that binary files would not affected by the keep_newline configuration.  However, this module does not actually distinguish between binary and text files.
Changed in version 2017.7.0: Templating/rendering has been added. You can now specify a default render pipeline and a black- and whitelist of (dis)allowed renderers.
template must be set to True for templating to happen.
ext_pillar: - file_tree: root_dir: /path/to/root/directory render_default: jinja|yaml renderer_blacklist: - gpg renderer_whitelist: - jinja - yaml template: True
Assigning Pillar Data to Individual Hosts To configure pillar data for each host, this external pillar will recursively iterate over root_dir/hosts/id (where id is a minion ID), and compile pillar data with each subdirectory as a dictionary key and each file as a value.
For example, the following root_dir tree:
./hosts/ ./hosts/test-host/ ./hosts/test-host/files/ ./hosts/test-host/files/testdir/ ./hosts/test-host/files/testdir/file1.txt ./hosts/test-host/files/testdir/file2.txt ./hosts/test-host/files/another-testdir/ ./hosts/test-host/files/another-testdir/symlink-to-file1.txt
will result in the following pillar tree for minion with ID test-host:
test-host: ---------- apache: ---------- config.d: ---------- 00_important.conf: <important_config important_setting="yes" /> 20_bob_extra.conf: <bob_specific_cfg has_freeze_ray="yes" /> corporate_app: ---------- settings: ---------- common_settings: // This is the main settings file for the corporate // internal web app main_setting: probably bob_settings: role: bob
NOTE: The leaf data in the example shown is the contents of the pillar files.
salt.pillar.file_tree.ext_pillar(minion_id, pillar, root_dir=None, follow_dir_links=False, debug=False, keep_newline=False, render_default=None, renderer_blacklist=None, renderer_whitelist=None, template=False) Compile pillar data from the given root_dir specific to Nodegroup names and Minion IDs.
If a Minion's ID is not found at <root_dir>/host/<minion_id> or if it is not included in any Nodegroups named at <root_dir>/nodegroups/<node_group>, no pillar data provided by this pillar module will be available for that Minion.
• minion_id -- The ID of the Minion whose pillar data is to be collected
• pillar -- Unused by the file_tree pillar module
• root_dir --
Filesystem directory used as the root for pillar data (e.g.  /srv/ext_pillar)
• follow_dir_links --
Follow symbolic links to directories while collecting pillar files.  Defaults to False.
WARNING: Care should be exercised when enabling this option as it will follow links that point outside of root_dir.
WARNING: Symbolic links that lead to infinite recursion are not filtered.
• debug -- Enable debug information at log level debug. Defaults to False.  This option may be useful to help debug errors when setting up the file_tree pillar module.
• keep_newline --
Preserve the end-of-file newline in files.  Defaults to False.  This option may either be a boolean or a list of file globs (as defined by the Python fnmatch package) for which end-of-file newlines are to be kept.
keep_newline should be turned on if the pillar data is intended to be used to deploy a file using contents_pillar with a file.managed state.
Changed in version 2015.8.4: The raw_data parameter has been renamed to keep_newline. In earlier releases, raw_data must be used. Also, this parameter can now be a list of globs, allowing for more granular control over which pillar values keep their end-of-file newline. The globs match paths relative to the directories named for Minion IDs and Nodegroup namess underneath the root_dir.
ext_pillar: - file_tree: root_dir: /srv/ext_pillar keep_newline: - apache/config.d/* - corporate_app/settings/*
NOTE: In earlier releases, this documentation incorrectly stated that binary files would not affected by the keep_newline.  However, this module does not actually distinguish between binary and text files.
• render_default --
Override Salt's default global renderer for the file_tree pillar.
render_default: jinja
• renderer_blacklist --
Disallow renderers for pillar files.
renderer_blacklist: - json
• renderer_whitelist --
Allow renderers for pillar files.
renderer_whitelist: - yaml - jinja
• template -- Enable templating of pillar files.  Defaults to False.
salt.pillar.foreman A module to pull data from Foreman via its API into the Pillar dictionary
Configuring the Foreman ext_pillar Set the following Salt config to setup Foreman as external pillar source:
ext_pillar: - foreman: key: foreman # Nest results within this key only: ['hostgroup_name', 'parameters'] # Add only these keys to pillar
foreman.url: https://example.com/foreman_api foreman.user: username # default is admin foreman.password: password # default is changeme
The following options are optional:
foreman.api: apiversion # default is 2 (1 is not supported yet) foreman.verifyssl: False # default is True foreman.certfile: /etc/ssl/certs/mycert.pem # default is None foreman.keyfile: /etc/ssl/private/mykey.pem # default is None foreman.cafile: /etc/ssl/certs/mycert.ca.pem # default is None foreman.lookup_parameters: True # default is True
An alternative would be to use the Foreman modules integrating Salt features in the Smart Proxy and the webinterface.
Further information can be found on GitHub.
Module Documentation salt.pillar.foreman.ext_pillar(minion_id, pillar, key=None, only=()) Read pillar data from Foreman via its API.
salt.pillar.git_pillar Use a git repository as a Pillar source This external pillar allows for a Pillar top file and Pillar SLS files to be sourced from a git repository.
However, since git_pillar does not have an equivalent to the pillar_roots parameter, configuration is slightly different. A Pillar top file is required to be in the git repository and must still contain the relevant environment, like so:
base: '*': - foo
The branch/tag which maps to that environment must then be specified along with the repo's URL. Configuration details can be found below.
IMPORTANT: Each branch/tag used for git_pillar must have its own top file. This is different from how the top file works when configuring States. The reason for this is that each git_pillar branch/tag is processed separately from the rest. Therefore, if the qa branch is to be used for git_pillar, it would need to have its own top file, with the qa environment defined within it, like this:
qa: 'dev-*': - bar
Additionally, while git_pillar allows for the branch/tag to be overridden (see here), keep in mind that the top file must reference the actual environment name. It is common practice to make the environment in a git_pillar top file match the branch/tag name, but when remapping, the environment of course no longer matches the branch/tag, and the top file needs to be adjusted accordingly.  When expected Pillar values configured in git_pillar are missing, this is a common misconfiguration that may be to blame, and is a good first step in troubleshooting.
Configuring git_pillar for Salt Beginning with Salt version 2015.8.0, pygit2 is now supported in addition to GitPython. The requirements for GitPython and pygit2 are the same as for GitFS, as described here.
IMPORTANT: git_pillar has its own set of global configuration parameters. While it may seem intuitive to use the global gitfs configuration parameters (gitfs_base, etc.) to manage git_pillar, this will not work.  The main difference for this is the fact that the different components which use Salt's git backend code do not all function identically. For instance, in git_pillar it is necessary to specify which branch/tag to be used for git_pillar remotes. This is the reverse behavior from gitfs, where branches/tags make up your environments.
See here for documentation on the git_pillar configuration options and their usage.
Here is an example git_pillar configuration:
ext_pillar: - git: # Use 'prod' instead of the branch name 'production' as the environment - production https://gitserver/git-pillar.git: - env: prod # Use 'dev' instead of the branch name 'develop' as the environment - develop https://gitserver/git-pillar.git: - env: dev # No per-remote config parameters (and no trailing colon), 'qa' will # be used as the environment - qa https://gitserver/git-pillar.git # SSH key authentication - master git@other-git-server:pillardata-ssh.git: # Pillar SLS files will be read from the 'pillar' subdirectory in # this repository - root: pillar - privkey: /path/to/key - pubkey: /path/to/key.pub - passphrase: CorrectHorseBatteryStaple # HTTPS authentication - master https://other-git-server/pillardata-https.git: - user: git - password: CorrectHorseBatteryStaple
The main difference between this and the old way of configuring git_pillar is that multiple remotes can be configured under one git section under ext_pillar. More than one git section can be used, but it is not necessary. Remotes will be evaluated sequentially.
Per-remote configuration parameters are supported (similar to gitfs), and global versions of the git_pillar configuration parameters can also be set.
To remap a specific branch to a specific Pillar environment, use the env per-remote parameter:
ext_pillar: - git: - production https://gitserver/git-pillar.git: - env: prod
If __env__ is specified as the branch name, then git_pillar will decide which branch to use based on the following criteria:
• If the minion has a pillarenv configured, it will use that pillar environment. (2016.11.2 and later)
• Otherwise, if the minion has an environment configured, it will use that environment.
• Otherwise, the master's git_pillar_base will be used.
NOTE: The use of environment to choose the pillar environment dates from a time before the pillarenv parameter was added.  In a future release, it will be ignored and either the minion's pillarenv or the master's git_pillar_base will be used.
Here's an example of using __env__ as the git_pillar environment:
ext_pillar: - git: - __env__ https://gitserver/git-pillar.git: - root: pillar
The corresponding Pillar top file would look like this:
"{{saltenv}}": '*': - bar
With the addition of pygit2 support, git_pillar can now interact with authenticated remotes. Authentication works just like in gitfs (as outlined in the Git Fileserver Backend Walkthrough), only with the global authentication parameter names prefixed with git_pillar instead of gitfs (e.g. git_pillar_pubkey, git_pillar_privkey, git_pillar_passphrase, etc.).
NOTE: The name parameter can be used to further differentiate between two remotes with the same URL and branch. When using two remotes with the same URL, the name option is required.
How Multiple Remotes Are Handled As noted above, multiple remotes can be included in the same git ext_pillar configuration. Consider the following:
ext_pillar: - etcd: my_etcd_config - git: - master https://mydomain.tld/foo.git: - root: pillar - master https://mydomain.tld/bar.git - master https://mydomain.tld/baz.git - dev https://mydomain.tld/qux.git - git: - master https://mydomain.tld/abc.git - dev https://mydomain.tld/123.git
To understand how pillar data from these repos will be compiled, it's important to know how Salt will process them. The following points should be kept in mind:
1. Each ext_pillar is called separately from the others. So, in the above example, the etcd ext_pillar will be evaluated first, with the first group of git_pillar remotes evaluated next (and merged into the etcd pillar data). Lastly, the second group of git_pillar remotes will be evaluated, and then merged into the ext_pillar data evaluated before it.
2. Within a single group of git_pillar remotes, each remote will be evaluated in order, with results merged together as each remote is evaluated.
NOTE: Prior to the 2017.7.0 release, remotes would be evaluated in a non-deterministic order.
3. By default, when a repo is evaluated, other remotes' which share its pillar environment will have their files made available to the remote being processed.
The first point should be straightforward enough, but the second and third could use some additional clarification.
First, point #2. In the first group of git_pillar remotes, the top file and pillar SLS files in the foo remote will be evaluated first. The bar remote will be evaluated next, and its results will be merged into the pillar data compiled when the foo remote was evaluated. As the subsequent remotes are evaluated, their data will be merged in the same fashion.
But wait, don't these repositories belong to more than one pillar environments?  Well, yes. The default method of generating pillar data compiles pillar data from all environments. This behavior can be overridden using a pillarenv.  Setting a pillarenv in the minion config file will make that minion tell the master to ignore any pillar data from environments which don't match that pillarenv. A pillarenv can also be specified for a given minion or set of minions when running states, by using the pillarenv argument. The CLI pillarenv will override one set in the minion config file. So, assuming that a pillarenv of base was set for a minion, it would not get any of the pillar variables configured in the qux remote, since that remote is assigned to the dev environment. The only way to get its pillar data would be to specify a pillarenv of dev, which would mean that it would then ignore any items from the base pillarenv. A more detailed explanation of pillar environments can be found here.
Moving on to point #3, and looking at the example ext_pillar configuration, as the foo remote is evaluated, it will also have access to the files from the bar and baz remotes, since all three are assigned to the base pillar environment. So, if an SLS file referenced by the foo remotes's top file does not exist in the foo remote, it will be searched for in the bar remote, followed by the baz remote. When it comes time to evaluate the bar remote, SLS files referenced by the bar remote's top file will first be looked for in the bar remote, followed by foo, and baz, and when the baz remote is processed, SLS files will be looked for in baz, followed by foo and bar. This "failover" logic is called a directory overlay, and it is also used by file_roots and :conf_minion`pillar_roots`. The ordering of which remote is checked for SLS files is determined by the order they are listed.  First the remote being processed is checked, then the others that share the same environment are checked. However, before the 2017.7.0 release, since evaluation was unordered, the remote being processed would be checked, followed in no specific order by the other repos which share the same environment.
Beginning with the 2017.7.0 release, this behavior of git_pillar remotes having access to files in other repos which share the same environment can be disabled by setting git_pillar_includes to False. If this is done, then all git_pillar remotes will only have access to their own SLS files. Another way of ensuring that a git_pillar remote will not have access to SLS files from other git_pillar remotes which share the same pillar environment is to put them in a separate git section under ext_pillar. Look again at the example configuration above. In the second group of git_pillar remotes, the abc remote would not have access to the SLS files from the foo, bar, and baz remotes, and vice-versa.
Mountpoints New in version 2017.7.0.
Assume the following pillar top file:
base: 'web*': - common - web.server.nginx - web.server.appdata
Now, assume that you would like to configure the web.server.nginx and web.server.appdata SLS files in separate repos. This could be done using the following ext_pillar configuration (assuming that git_pillar_includes has not been set to False):
ext_pillar: - git: - master https://mydomain.tld/pillar-common.git - master https://mydomain.tld/pillar-nginx.git - master https://mydomain.tld/pillar-appdata.git
However, in order to get the files in the second and third git_pillar remotes to work, you would need to first create the directory structure underneath it (i.e. place them underneath web/server/ in the repository). This also makes it tedious to reorganize the configuration, as changing web.server.nginx to web.nginx in the top file would require you to also move the SLS files in the pillar-nginx up a directory level.
For these reasons, much like gitfs, git_pillar now supports a "mountpoint" feature. Using the following ext_pillar configuration, the SLS files in the second and third git_pillar remotes can be placed in the root of the git repository:
ext_pillar: - git: - master https://mydomain.tld/pillar-common.git - master https://mydomain.tld/pillar-nginx.git: - mountpoint: web/server/ - master https://mydomain.tld/pillar-appdata.git: - mountpoint: web/server/
Now, if the top file changed the SLS target from web.server.nginx, instead of reorganizing the git repository, you would just need to adjust the mountpoint to web/ (and restart the salt-master daemon).
• Leading and trailing slashes on the mountpoints are optional.
• Use of the mountpoint feature requires that git_pillar_includes is not disabled.
• Content from mounted git_pillar repos can only be referenced by a top file in the same pillar environment.
• Salt versions prior to 2018.3.4 ignore the root parameter when mountpoint is set.
all_saltenvs New in version 2018.3.4.
When __env__ is specified as the branch name, all_saltenvs per-remote configuration parameter overrides the logic Salt uses to map branches/tags to pillar environments (i.e. pillarenvs). This allows a single branch/tag to appear in all saltenvs. Example:
ext_pillar: - git: - __env__ https://mydomain.tld/top.git - all_saltenvs: master - __env__ https://mydomain.tld/pillar-nginx.git: - mountpoint: web/server/ - __env__ https://mydomain.tld/pillar-appdata.git: - mountpoint: web/server/
fallback New in version 3001.
Setting fallback per-remote or global configuration parameter will map non-existing environments to a default branch. Example:
ext_pillar: - git: - __env__ https://mydomain.tld/top.git - all_saltenvs: master - __env__ https://mydomain.tld/pillar-nginx.git: - mountpoint: web/server/ - fallback: master - __env__ https://mydomain.tld/pillar-appdata.git: - mountpoint: web/server/ - fallback: master
salt.pillar.git_pillar.ext_pillar(minion_id, pillar, *repos) Checkout the ext_pillar sources and compile the resulting pillar SLS
salt.pillar.gpg Decrypt pillar data through the builtin GPG renderer
In most cases, you'll want to make this the last external pillar used. For example, to pair with the builtin stack pillar you could do something like this:
ext_pillar: - stack: /path/to/stack.cfg - gpg: {}
Set gpg_keydir in your config to adjust the homedir the renderer uses.
salt.pillar.gpg.ext_pillar(minion_id, pillar, *args, **kwargs)
salt.pillar.hg_pillar Use remote Mercurial repository as a Pillar source.
The module depends on the hglib python module being available.  This is the same requirement as for hgfs_ so should not pose any extra hurdles.
This external Pillar source can be configured in the master config file as such:
ext_pillar: - hg: ssh://hg@example.co/user/repo
class salt.pillar.hg_pillar.Repo(repo_uri) Deal with remote hg (mercurial) repository for Pillar
close() Cleanup mercurial command server
pull()
update(branch='default') Ensure we are using the latest revision in the hg repository
salt.pillar.hg_pillar.ext_pillar(minion_id, pillar, repo, branch='default', root=None) Extract pillar from an hg repository
salt.pillar.hg_pillar.update(repo_uri) Execute an hg pull on all the repos
salt.pillar.hiera Use hiera data as a Pillar source
salt.pillar.hiera.ext_pillar(minion_id, pillar, conf) Execute hiera and return the data
salt.pillar.http_json A module that adds data to the Pillar structure retrieved by an http request
Configuring the HTTP_JSON ext_pillar Set the following Salt config to setup http json result as external pillar source:
ext_pillar: - http_json: url: http://example.com/api/minion_id ::TODO:: username: username password: password
If the with_grains parameter is set, grain keys wrapped in can be provided (wrapped in <> brackets) in the url in order to populate pillar data based on the grain value.
ext_pillar: - http_json: url: http://example.com/api/<nodename> with_grains: True
Changed in version 2018.3.0: If %s is present in the url, it will be automatically replaced by the minion_id:
ext_pillar: - http_json: url: http://example.com/api/%s
Module Documentation salt.pillar.http_json.ext_pillar(minion_id, pillar, url, with_grains=False) Read pillar data from HTTP response.
• url (str) -- Url to request.
• with_grains (bool) -- Whether to substitute strings in the url with their grain values.
Returns A dictionary of the pillar data to add.
salt.pillar.http_yaml A module that adds data to the Pillar structure retrieved by an http request
Configuring the HTTP_YAML ext_pillar Set the following Salt config to setup an http endpoint as the external pillar source:
ext_pillar: - http_yaml: url: http://example.com/api/minion_id ::TODO:: username: username password: password