neulab/tldr huggingface dataset:
external_auth: <eauth backend>: <user or group%>: <@runner or @wheel>: - <regex to match function>
By user, by runner+wheel module:
external_auth: <eauth backend>: <user or group%>: <@module_name>: - <regex to match function without module_name>
Groups To apply permissions to a group of users in an external authentication system, append a % to the ID:
external_auth: pam: admins%: - '*': - 'pkg.*'
Limiting by function arguments Positional arguments or keyword arguments to functions can also be whitelisted.
New in version 2016.3.0.
external_auth: pam: my_user: - '*': - 'my_mod.*': args: - 'a.*' - 'b.*' kwargs: 'kwa': 'kwa.*' 'kwb': 'kwb' - '@runner': - 'runner_mod.*': args: - 'a.*' - 'b.*' kwargs: 'kwa': 'kwa.*' 'kwb': 'kwb'
The rules:
1. The arguments values are matched as regexp.
2. If arguments restrictions are specified the only matched are allowed.
3. If an argument isn't specified any value is allowed.
4. To skip an arg use "everything" regexp .*. I.e. if arg0 and arg2 should be limited but arg1 and other arguments could have any value use:
args: - 'value0' - '.*' - 'value2'
Usage The external authentication system can then be used from the command-line by any user on the same system as the master with the -a option:
$ salt -a pam web\* test.version
The system will ask the user for the credentials required by the authentication system and then publish the command.
Tokens With external authentication alone, the authentication credentials will be required with every call to Salt. This can be alleviated with Salt tokens.
Tokens are short term authorizations and can be easily created by just adding a -T option when authenticating:
$ salt -T -a pam web\* test.version
Now a token will be created that has an expiration of 12 hours (by default).  This token is stored in a file named salt_token in the active user's home directory.
Once the token is created, it is sent with all subsequent communications. User authentication does not need to be entered again until the token expires.
Token expiration time can be set in the Salt master config file.
LDAP and Active Directory NOTE: LDAP usage requires that you have installed python-ldap.
Salt supports both user and group authentication for LDAP (and Active Directory accessed via its LDAP interface)
OpenLDAP and similar systems LDAP configuration happens in the Salt master configuration file.
Server configuration values and their defaults:
# Server to auth against auth.ldap.server: localhost
# Port to connect via auth.ldap.port: 389
# Use TLS when connecting auth.ldap.tls: False
# Use STARTTLS when connecting auth.ldap.starttls: False
# LDAP scope level, almost always 2 auth.ldap.scope: 2
# Server specified in URI format auth.ldap.uri: ''    # Overrides .ldap.server, .ldap.port, .ldap.tls above
# Verify server's TLS certificate auth.ldap.no_verify: False
# Bind to LDAP anonymously to determine group membership # Active Directory does not allow anonymous binds without special configuration # In addition, if auth.ldap.anonymous is True, empty bind passwords are not permitted. auth.ldap.anonymous: False
# FOR TESTING ONLY, this is a VERY insecure setting. # If this is True, the LDAP bind password will be ignored and # access will be determined by group membership alone with # the group memberships being retrieved via anonymous bind auth.ldap.auth_by_group_membership_only: False
# Require authenticating user to be part of this Organizational Unit # This can be blank if your LDAP schema does not use this kind of OU auth.ldap.groupou: 'Groups'
# Object Class for groups.  An LDAP search will be done to find all groups of this # class to which the authenticating user belongs. auth.ldap.groupclass: 'posixGroup'
# Unique ID attribute name for the user auth.ldap.accountattributename: 'memberUid'
# These are only for Active Directory auth.ldap.activedirectory: False auth.ldap.persontype: 'person'
auth.ldap.minion_stripdomains: []
# Redhat Identity Policy Audit auth.ldap.freeipa: False
Authenticating to the LDAP Server There are two phases to LDAP authentication.  First, Salt authenticates to search for a users' Distinguished Name and group membership.  The user it authenticates as in this phase is often a special LDAP system user with read-only access to the LDAP directory.  After Salt searches the directory to determine the actual user's DN and groups, it re-authenticates as the user running the Salt commands.
If you are already aware of the structure of your DNs and permissions in your LDAP store are set such that users can look up their own group memberships, then the first and second users can be the same.  To tell Salt this is the case, omit the auth.ldap.bindpw parameter.  Note this is not the same thing as using an anonymous bind.  Most LDAP servers will not permit anonymous bind, and as mentioned above, if auth.ldap.anonymous is False you cannot use an empty password.
You can template the binddn like this:
auth.ldap.basedn: dc=saltstack,dc=com auth.ldap.binddn: uid={{ username }},cn=users,cn=accounts,dc=saltstack,dc=com
Salt will use the password entered on the salt command line in place of the bindpw.
To use two separate users, specify the LDAP lookup user in the binddn directive, and include a bindpw like so
auth.ldap.binddn: uid=ldaplookup,cn=sysaccounts,cn=etc,dc=saltstack,dc=com auth.ldap.bindpw: mypassword
As mentioned before, Salt uses a filter to find the DN associated with a user. Salt substitutes the {{ username }} value for the username when querying LDAP
auth.ldap.filter: uid={{ username }}
Determining Group Memberships (OpenLDAP / non-Active Directory) For OpenLDAP, to determine group membership, one can specify an OU that contains group data. This is prepended to the basedn to create a search path.  Then the results are filtered against auth.ldap.groupclass, default posixGroup, and the account's 'name' attribute, memberUid by default.
auth.ldap.groupou: Groups
Note that as of 2017.7, auth.ldap.groupclass can refer to either a groupclass or an objectClass.  For some LDAP servers (notably OpenLDAP without the memberOf overlay enabled) to determine group membership we need to know both the objectClass and the memberUid attributes.  Usually for these servers you will want a auth.ldap.groupclass of posixGroup and an auth.ldap.groupattribute of memberUid.
LDAP servers with the memberOf overlay will have entries similar to auth.ldap.groupclass: person and auth.ldap.groupattribute: memberOf.
When using the ldap('DC=domain,DC=com') eauth operator, sometimes the records returned from LDAP or Active Directory have fully-qualified domain names attached, while minion IDs instead are simple hostnames. The parameter below allows the administrator to strip off a certain set of domain names so the hostnames looked up in the directory service can match the minion IDs.
auth.ldap.minion_stripdomains: ['.external.bigcorp.com', '.internal.bigcorp.com']
Determining Group Memberships (Active Directory) Active Directory handles group membership differently, and does not utilize the groupou configuration variable.  AD needs the following options in the master config:
auth.ldap.activedirectory: True auth.ldap.filter: sAMAccountName={{username}} auth.ldap.accountattributename: sAMAccountName auth.ldap.groupclass: group auth.ldap.persontype: person
To determine group membership in AD, the username and password that is entered when LDAP is requested as the eAuth mechanism on the command line is used to bind to AD's LDAP interface. If this fails, then it doesn't matter what groups the user belongs to, he or she is denied access. Next, the distinguishedName of the user is looked up with the following LDAP search:
(&(<value of auth.ldap.accountattributename>={{username}}) (objectClass=<value of auth.ldap.persontype>) )
This should return a distinguishedName that we can use to filter for group membership.  Then the following LDAP query is executed:
(&(member=<distinguishedName from search above>) (objectClass=<value of auth.ldap.groupclass>) )
external_auth: ldap: test_ldap_user: - '*': - test.ping
To configure a LDAP group, append a % to the ID:
external_auth: ldap: test_ldap_group%: - '*': - test.echo
In addition, if there are a set of computers in the directory service that should be part of the eAuth definition, they can be specified like this:
external_auth: ldap: test_ldap_group%: - ldap('DC=corp,DC=example,DC=com'): - test.echo
The string inside ldap() above is any valid LDAP/AD tree limiter.  OU= in particular is permitted as long as it would return a list of computer objects.
Peer Communication Salt 0.9.0 introduced the capability for Salt minions to publish commands. The intent of this feature is not for Salt minions to act as independent brokers one with another, but to allow Salt minions to pass commands to each other.
In Salt 0.10.0 the ability to execute runners from the master was added. This allows for the master to return collective data from runners back to the minions via the peer interface.
The peer interface is configured through two options in the master configuration file. For minions to send commands from the master the peer configuration is used. To allow for minions to execute runners from the master the peer_run configuration is used.
Since this presents a viable security risk by allowing minions access to the master publisher the capability is turned off by default. The minions can be allowed access to the master publisher on a per minion basis based on regular expressions. Minions with specific ids can be allowed access to certain Salt modules and functions.
Peer Configuration The configuration is done under the peer setting in the Salt master configuration file, here are a number of configuration possibilities.
The simplest approach is to enable all communication for all minions, this is only recommended for very secure environments.
This configuration will allow minions with IDs ending in example.com access to the test, ps, and pkg module functions.
peer: .*example.com: - test.* - ps.* - pkg.*
The configuration logic is simple, a regular expression is passed for matching minion ids, and then a list of expressions matching minion functions is associated with the named minion. For instance, this configuration will also allow minions ending with foo.org access to the publisher.
peer: .*example.com: - test.* - ps.* - pkg.* .*foo.org: - test.* - ps.* - pkg.*
NOTE: Functions are matched using regular expressions.
Peer Runner Communication Configuration to allow minions to execute runners from the master is done via the peer_run option on the master. The peer_run configuration follows the same logic as the peer option. The only difference is that access is granted to runner modules.
To open up access to all minions to all runners:
peer_run: .*: - .*
This configuration will allow minions with IDs ending in example.com access to the manage and jobs runner functions.
peer_run: .*example.com: - manage.* - jobs.*
Using Peer Communication The publish module was created to manage peer communication. The publish module comes with a number of functions to execute peer communication in different ways. Currently there are three functions in the publish module. These examples will show how to test the peer system via the salt-call command.
To execute test.version on all minions:
# salt-call publish.publish \* test.version
To execute the manage.up runner:
# salt-call publish.runner manage.up
To match minions using other matchers, use tgt_type:
# salt-call publish.publish 'webserv* and not G@os:Ubuntu' test.version tgt_type='compound'
NOTE: In pre-2017.7.0 releases, use expr_form instead of tgt_type.
When to Use Each Authentication System publisher_acl is useful for allowing local system users to run Salt commands without giving them root access. If you can log into the Salt master directly, then publisher_acl allows you to use Salt without root privileges. If the local system is configured to authenticate against a remote system, like LDAP or Active Directory, then publisher_acl will interact with the remote system transparently.
Examples The access controls are manifested using matchers in these configurations:
publisher_acl: fred: - web\*: - pkg.list_pkgs - test.* - apache.*
In the above example, fred is able to send commands only to minions which match the specified glob target. This can be expanded to include other functions for other minions based on standard targets (all matchers are supported except the compound one).
external_auth: pam: dave: - test.version - mongo\*: - network.* - log\*: - network.* - pkg.* - 'G@os:RedHat': - kmod.* steve: - .*
The above allows for all minions to be hit by test.version by dave, and adds a few functions that dave can execute on other minions. It also allows steve unrestricted access to salt commands.
Job Management New in version 0.9.7.
Since Salt executes jobs running on many systems, Salt needs to be able to manage jobs running on many systems.
The Minion proc System Salt Minions maintain a proc directory in the Salt cachedir. The proc directory maintains files named after the executed job ID. These files contain the information about the current running jobs on the minion and allow for jobs to be looked up. This is located in the proc directory under the cachedir, with a default configuration it is under /var/cache/salt/{master|minion}/proc.
Functions in the saltutil Module Salt 0.9.7 introduced a few new functions to the saltutil module for managing jobs. These functions are:
1. running Returns the data of all running jobs that are found in the proc directory.
2. find_job Returns specific data about a certain job based on job id.
3. signal_job Allows for a given jid to be sent a signal.
4. term_job Sends a termination signal (SIGTERM, 15) to the process controlling the specified job.
5. kill_job Sends a kill signal (SIGKILL, 9) to the process controlling the specified job.
These functions make up the core of the back end used to manage jobs at the minion level.
The jobs Runner A convenience runner front end and reporting system has been added as well.  The jobs runner contains functions to make viewing data easier and cleaner.
The jobs runner contains a number of functions...
active The active function runs saltutil.running on all minions and formats the return data about all running jobs in a much more usable and compact format.  The active function will also compare jobs that have returned and jobs that are still running, making it easier to see what systems have completed a job and what systems are still being waited on.
# salt-run jobs.active
lookup_jid When jobs are executed the return data is sent back to the master and cached.  By default it is cached for 24 hours, but this can be configured via the keep_jobs option in the master configuration.  Using the lookup_jid runner will display the same return data that the initial job invocation with the salt command would display.
# salt-run jobs.lookup_jid <job id number>
list_jobs Before finding a historic job, it may be required to find the job id. list_jobs will parse the cached execution data and display all of the job data for jobs that have already, or partially returned.
# salt-run jobs.list_jobs
Scheduling Jobs Salt's scheduling system allows incremental executions on minions or the master. The schedule system exposes the execution of any execution function on minions or any runner on the master.
Scheduling can be enabled by multiple methods:
• schedule option in either the master or minion config files.  These require the master or minion application to be restarted in order for the schedule to be implemented.
• Minion pillar data.  Schedule is implemented by refreshing the minion's pillar data, for example by using saltutil.refresh_pillar.
• The schedule state or schedule module
NOTE: The scheduler executes different functions on the master and minions. When running on the master the functions reference runner functions, when running on the minion the functions specify execution functions.
A scheduled run has no output on the minion unless the config is set to info level or higher. Refer to minion-logging-settings.
States are executed on the minion, as all states are. You can pass positional arguments and provide a YAML dict of named arguments.
schedule: job1: function: state.sls seconds: 3600 args: - httpd kwargs: test: True
This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour).
schedule: job1: function: state.sls seconds: 3600 args: - httpd kwargs: test: True splay: 15
This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour) splaying the time between 0 and 15 seconds.
schedule: job1: function: state.sls seconds: 3600 args: - httpd kwargs: test: True splay: start: 10 end: 15
This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour) splaying the time between 10 and 15 seconds.
Schedule by Date and Time New in version 2014.7.0.
Frequency of jobs can also be specified using date strings supported by the Python dateutil library. This requires the Python dateutil library to be installed.
schedule: job1: function: state.sls args: - httpd kwargs: test: True when: 5:00pm
This will schedule the command: state.sls httpd test=True at 5:00 PM minion localtime.
schedule: job1: function: state.sls args: - httpd kwargs: test: True when: - Monday 5:00pm - Tuesday 3:00pm - Wednesday 5:00pm - Thursday 3:00pm - Friday 5:00pm
This will schedule the command: state.sls httpd test=True at 5:00 PM on Monday, Wednesday and Friday, and 3:00 PM on Tuesday and Thursday.
schedule: job1: function: state.sls args: - httpd kwargs: test: True when: - 'tea time'
whens: tea time: 1:40pm deployment time: Friday 5:00pm
The Salt scheduler also allows custom phrases to be used for the when parameter.  These whens can be stored as either pillar values or grain values.
schedule: job1: function: state.sls seconds: 3600 args: - httpd kwargs: test: True range: start: 8:00am end: 5:00pm
This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour) between the hours of 8:00 AM and 5:00 PM. The range parameter must be a dictionary with the date strings using the dateutil format.
schedule: job1: function: state.sls seconds: 3600 args: - httpd kwargs: test: True range: invert: True start: 8:00am end: 5:00pm
Using the invert option for range, this will schedule the command state.sls httpd test=True every 3600 seconds (every hour) until the current time is between the hours of 8:00 AM and 5:00 PM. The range parameter must be a dictionary with the date strings using the dateutil format.
schedule: job1: function: pkg.install kwargs: pkgs: [{'bar': '>1.2.3'}] refresh: true once: '2016-01-07T14:30:00'
This will schedule the function pkg.install to be executed once at the specified time. The schedule entry job1 will not be removed after the job completes, therefore use schedule.delete to manually remove it afterwards.
The default date format is ISO 8601 but can be overridden by also specifying the once_fmt option, like this:
schedule: job1: function: test.ping once: 2015-04-22T20:21:00 once_fmt: '%Y-%m-%dT%H:%M:%S'
Maximum Parallel Jobs Running New in version 2014.7.0.
The scheduler also supports ensuring that there are no more than N copies of a particular routine running. Use this for jobs that may be long-running and could step on each other or pile up in case of infrastructure outage.
The default for maxrunning is 1.
schedule: long_running_job: function: big_file_transfer jid_include: True maxrunning: 1
Cron-like Schedule New in version 2014.7.0.
schedule: job1: function: state.sls cron: '*/15 * * * *' args: - httpd kwargs: test: True
The scheduler also supports scheduling jobs using a cron like format. This requires the Python croniter library.
Job Data Return New in version 2015.5.0.
By default, data about jobs runs from the Salt scheduler is returned to the master. Setting the return_job parameter to False will prevent the data from being sent back to the Salt master.
schedule: job1: function: scheduled_job_function return_job: False
Job Metadata New in version 2015.5.0.
It can be useful to include specific data to differentiate a job from other jobs. Using the metadata parameter special values can be associated with a scheduled job. These values are not used in the execution of the job, but can be used to search for specific jobs later if combined with the return_job parameter. The metadata parameter must be specified as a dictionary, othewise it will be ignored.
schedule: job1: function: scheduled_job_function metadata: foo: bar
Run on Start New in version 2015.5.0.
By default, any job scheduled based on the startup time of the minion will run the scheduled job when the minion starts up. Sometimes this is not the desired situation. Using the run_on_start parameter set to False will cause the scheduler to skip this first run and wait until the next scheduled run:
schedule: job1: function: state.sls seconds: 3600 run_on_start: False args: - httpd kwargs: test: True
Until and After New in version 2015.8.0.
schedule: job1: function: state.sls seconds: 15 until: '12/31/2015 11:59pm' args: - httpd kwargs: test: True
Using the until argument, the Salt scheduler allows you to specify an end time for a scheduled job. If this argument is specified, jobs will not run once the specified time has passed. Time should be specified in a format supported by the dateutil library.  This requires the Python dateutil library to be installed.
schedule: job1: function: state.sls seconds: 15 after: '12/31/2015 11:59pm' args: - httpd kwargs: test: True
Using the after argument, the Salt scheduler allows you to specify an start time for a scheduled job.  If this argument is specified, jobs will not run until the specified time has passed. Time should be specified in a format supported by the dateutil library.  This requires the Python dateutil library to be installed.
Scheduling States schedule: log-loadavg: function: cmd.run seconds: 3660 args: - 'logger -t salt < /proc/loadavg' kwargs: stateful: False shell: /bin/sh
Scheduling Highstates To set up a highstate to run on a minion every 60 minutes set this in the minion config or pillar:
schedule: highstate: function: state.highstate minutes: 60
Time intervals can be specified as seconds, minutes, hours, or days.
Scheduling Runners Runner executions can also be specified on the master within the master configuration file:
schedule: run_my_orch: function: state.orchestrate hours: 6 splay: 600 args: - orchestration.my_orch
The above configuration is analogous to running salt-run state.orch orchestration.my_orch every 6 hours.
Scheduler With Returner The scheduler is also useful for tasks like gathering monitoring data about a minion, this schedule option will gather status data and send it to a MySQL returner database:
schedule: uptime: function: status.uptime seconds: 60 returner: mysql meminfo: function: status.meminfo minutes: 5 returner: mysql
Since specifying the returner repeatedly can be tiresome, the schedule_returner option is available to specify one or a list of global returners to be used by the minions when scheduling.
Managing the Job Cache The Salt Master maintains a job cache of all job executions which can be queried via the jobs runner. This job cache is called the Default Job Cache.
Default Job Cache A number of options are available when configuring the job cache. The default caching system uses local storage on the Salt Master and can be found in the job cache directory (on Linux systems this is typically /var/cache/salt/master/jobs). The default caching system is suitable for most deployments as it does not typically require any further configuration or management.
The default job cache is a temporary cache and jobs will be stored for 24 hours. If the default cache needs to store jobs for a different period the time can be easily adjusted by changing the keep_jobs parameter in the Salt Master configuration file. The value passed in is measured via hours:
Reducing the Size of the Default Job Cache The Default Job Cache can sometimes be a burden on larger deployments (over 5000 minions). Disabling the job cache will make previously executed jobs unavailable to the jobs system and is not generally recommended. Normally it is wise to make sure the master has access to a faster IO system or a tmpfs is mounted to the jobs dir.
However, you can disable the job_cache by setting it to False in the Salt Master configuration file. Setting this value to False means that the Salt Master will no longer cache minion returns, but a JID directory and jid file for each job will still be created. This JID directory is necessary for checking for and preventing JID collisions.
The default location for the job cache is in the /var/cache/salt/master/jobs/ directory.
Setting the job_cache to False in addition to setting the keep_jobs option to a smaller value, such as 1, in the Salt Master configuration file will reduce the size of the Default Job Cache, and thus the burden on the Salt Master.
NOTE: Changing the keep_jobs option sets the number of hours to keep old job information and defaults to 24 hours. Do not set this value to 0 when trying to make the cache cleaner run more frequently, as this means the cache cleaner will never run.
Additional Job Cache Options Many deployments may wish to use an external database to maintain a long term register of executed jobs. Salt comes with two main mechanisms to do this, the master job cache and the external job cache.
See Storing Job Results in an External System.
Storing Job Results in an External System After a job executes, job results are returned to the Salt Master by each Salt Minion. These results are stored in the Default Job Cache.
In addition to the Default Job Cache, Salt provides two additional mechanisms to send job results to other systems (databases, local syslog, and others):
• External Job Cache
• Master Job Cache
The major difference between these two mechanism is from where results are returned (from the Salt Master or Salt Minion). Configuring either of these options will also make the Jobs Runner functions to automatically query the remote stores for information.
External Job Cache - Minion-Side Returner When an External Job Cache is configured, data is returned to the Default Job Cache on the Salt Master like usual, and then results are also sent to an External Job Cache using a Salt returner module running on the Salt Minion.  [image]
• Advantages: Data is stored without placing additional load on the Salt Master.
• Disadvantages: Each Salt Minion connects to the external job cache, which can result in a large number of connections.  Also requires additional configuration to get returner module settings on all Salt Minions.
Master Job Cache - Master-Side Returner New in version 2014.7.0.
Instead of configuring an External Job Cache on each Salt Minion, you can configure the Master Job Cache to send job results from the Salt Master instead. In this configuration, Salt Minions send data to the Default Job Cache as usual, and then the Salt Master sends the data to the external system using a Salt returner module running on the Salt Master.  [image]
• Advantages: A single connection is required to the external system. This is preferred for databases and similar systems.
• Disadvantages: Places additional load on your Salt Master.
Configure an External or Master Job Cache Step 1: Understand Salt Returners Before you configure a job cache, it is essential to understand Salt returner modules ("returners"). Returners are pluggable Salt Modules that take the data returned by jobs, and then perform any necessary steps to send the data to an external system. For example, a returner might establish a connection, authenticate, and then format and transfer data.
The Salt Returner system provides the core functionality used by the External and Master Job Cache systems, and the same returners are used by both systems.
Salt currently provides many different returners that let you connect to a wide variety of systems. A complete list is available at all Salt returners.  Each returner is configured differently, so make sure you read and follow the instructions linked from that page.
For example, the MySQL returner requires:
• A database created using provided schema (structure is available at MySQL returner)
• A user created with privileges to the database
• Optional SSL configuration
A simpler returner, such as Slack or HipChat, requires:
• An API key/version
• The target channel/room
• The username that should be used to send the message
Step 2: Configure the Returner After you understand the configuration and have the external system ready, the configuration requirements must be declared.
External Job Cache The returner configuration settings can be declared in the Salt Minion configuration file, the Minion's pillar data, or the Minion's grains.
If external_job_cache configuration settings are specified in more than one place, the options are retrieved in the following order. The first configuration location that is found is the one that will be used.
• Minion configuration file
• Minion's grains
• Minion's pillar data
Master Job Cache The returner configuration settings for the Master Job Cache should be declared in the Salt Master's configuration file.
Configuration File Examples MySQL requires:
mysql.host: 'salt' mysql.user: 'salt' mysql.pass: 'salt' mysql.db: 'salt' mysql.port: 3306
Slack requires:
slack.channel: 'channel' slack.api_key: 'key' slack.from_name: 'name'
After you have configured the returner and added settings to the configuration file, you can enable the External or Master Job Cache.
Step 3: Enable the External or Master Job Cache Configuration is a single line that specifies an already-configured returner to use to send all job data to an external system.
External Job Cache To enable a returner as the External Job Cache (Minion-side), add the following line to the Salt Master configuration file:
ext_job_cache: <returner>
ext_job_cache: mysql
NOTE: When configuring an External Job Cache (Minion-side), the returner settings are added to the Minion configuration file, but the External Job Cache setting is configured in the Master configuration file.
Master Job Cache To enable a returner as a Master Job Cache (Master-side), add the following line to the Salt Master configuration file:
master_job_cache: <returner>
master_job_cache: mysql
Verify that the returner configuration settings are in the Master configuration file, and be sure to restart the salt-master service after you make configuration changes. (service salt-master restart).
Logging The salt project tries to get the logging to work for you and help us solve any issues you might find along the way.
If you want to get some more information on the nitty-gritty of salt's logging system, please head over to the logging development document, if all you're after is salt's logging configurations, please continue reading.
Log Levels The log levels are ordered numerically such that setting the log level to a specific level will record all log statements at that level and higher. For example, setting log_level: error will log statements at error, critical, and quiet levels, although nothing should be logged at quiet level.
Most of the logging levels are defined by default in Python's logging library and can be found in the official Python documentation.  Salt uses some more levels in addition to the standard levels.  All levels available in salt are shown in the table below.
NOTE: Python dependencies used by salt may define and use additional logging levels.  For example, the Python 2 version of the multiprocessing standard Python library uses the levels subwarning, 25 and subdebug, 5.
┌─────────┬───────────────┬──────────────────────┐ │Level    │ Numeric value │ Description          │ ├─────────┼───────────────┼──────────────────────┤ │quiet    │ 1000          │ Nothing should be    │ │         │               │ logged at this level │ ├─────────┼───────────────┼──────────────────────┤ │critical │ 50            │ Critical errors      │ ├─────────┼───────────────┼──────────────────────┤ │error    │ 40            │ Errors               │ ├─────────┼───────────────┼──────────────────────┤ │warning  │ 30            │ Warnings             │ ├─────────┼───────────────┼──────────────────────┤ │info     │ 20            │ Normal log           │ │         │               │ information          │ ├─────────┼───────────────┼──────────────────────┤ │profile  │ 15            │ Profiling            │ │         │               │ information on salt  │ │         │               │ performance          │ ├─────────┼───────────────┼──────────────────────┤ │debug    │ 10            │ Information useful   │ │         │               │ for debugging both   │ │         │               │ salt implementations │ │         │               │ and salt code        │ ├─────────┼───────────────┼──────────────────────┤ │trace    │ 5             │ More detailed code   │ │         │               │ debugging            │ │         │               │ information          │ ├─────────┼───────────────┼──────────────────────┤ │garbage  │ 1             │ Even more debugging  │ │         │               │ information          │ ├─────────┼───────────────┼──────────────────────┤ │all      │ 0             │ Everything           │ └─────────┴───────────────┴──────────────────────┘
Available Configuration Settings log_file The log records can be sent to a regular file, local path name, or network location.  Remote logging works best when configured to use rsyslogd(8) (e.g.: file:///dev/log), with rsyslogd(8) configured for network logging.  The format for remote addresses is:
<file|udp|tcp>://<host|socketpath>:<port-if-required>/<log-facility>
Where log-facility is the symbolic name of a syslog facility as defined in the SysLogHandler documentation. It defaults to LOG_USER.
Default: Dependent of the binary being executed, for example, for salt-master, /var/log/salt/master.
log_file: file:///dev/log/LOG_DAEMON
The level of log record messages to send to the console. One of all, garbage, trace, debug, profile, info, warning, error, critical, quiet.
NOTE: Add log_level: quiet in salt configuration file to completely disable logging. In case of running salt in command line use --log-level=quiet instead.
log_level_logfile Default: info
The level of messages to send to the log file. One of all, garbage, trace, debug, profile, info, warning, error, critical, quiet.
The date and time format used in console log messages. Allowed date/time formatting matches those used in time.strftime().
The date and time format used in log file messages. Allowed date/time formatting matches those used in time.strftime().
The format of the console logging messages. All standard python logging LogRecord attributes can be used. Salt also provides these custom LogRecord attributes to colorize console log output:
"%(colorlevel)s"  # log level name colorized by level "%(colorname)s"  # colorized module name "%(colorprocess)s"  # colorized process number "%(colormsg)s"  # log message colorized by level
NOTE: The %(colorlevel)s, %(colorname)s, and %(colorprocess) LogRecord attributes also include padding and enclosing brackets, [ and ] to match the default values of their collateral non-colorized LogRecord attributes.
log_fmt_console: '[%(levelname)-8s] %(message)s'
The format of the log file logging messages. All standard python logging LogRecord attributes can be used.  Salt also provides these custom LogRecord attributes that include padding and enclosing brackets [ and ]:
"%(bracketlevel)s"  # equivalent to [%(levelname)-8s] "%(bracketname)s"  # equivalent to [%(name)-17s] "%(bracketprocess)s"  # equivalent to [%(process)5s]
This can be used to control logging levels more specifically, based on log call name.  The example sets the main salt library at the 'warning' level, sets salt.modules to log at the debug level, and sets a custom module to the all level:
log_granular_levels: 'salt': 'warning' 'salt.modules': 'debug' 'salt.loader.saltmaster.ext.module.custom_module': 'all'
log_fmt_jid Default: [JID: %(jid)s]
The format of the JID when added to logging messages.
log_fmt_jid: '[JID: %(jid)s]'
External Logging Handlers Besides the internal logging handlers used by salt, there are some external which can be used, see the external logging handlers document.
External Logging Handlers ┌──────────────┬───────────────────────────┐ │fluent_mod    │ Fluent Logging Handler    │ ├──────────────┼───────────────────────────┤ │log4mongo_mod │ Log4Mongo Logging Handler │ ├──────────────┼───────────────────────────┤ │logstash_mod  │ Logstash Logging Handler  │ ├──────────────┼───────────────────────────┤ │sentry_mod    │ Sentry Logging Handler    │ └──────────────┴───────────────────────────┘
salt.log.handlers.fluent_mod Fluent Logging Handler New in version 2015.8.0.
This module provides some fluentd logging handlers.
Fluent Logging Handler In the fluent configuration file:
<source> type forward bind localhost port 24224 </source>
Then, to send logs via fluent in Logstash format, add the following to the salt (master and/or minion) configuration file:
fluent_handler: host: localhost port: 24224
To send logs via fluent in the Graylog raw json format, add the following to the salt (master and/or minion) configuration file:
fluent_handler: host: localhost port: 24224 payload_type: graylog tags: - salt_master.SALT
The above also illustrates the tags option, which allows one to set descriptive (or useful) tags on records being sent.  If not provided, this defaults to the single tag: 'salt'.  Also note that, via Graylog "magic", the 'facility' of the logged message is set to 'SALT' (the portion of the tag after the first period), while the tag itself will be set to simply 'salt_master'.  This is a feature, not a bug :)
Note: There is a third emitter, for the GELF format, but it is largely untested, and I don't currently have a setup supporting this config, so while it runs cleanly and outputs what LOOKS to be valid GELF, any real-world feedback on its usefulness, and correctness, will be appreciated.
Log Level The fluent_handler configuration section accepts an additional setting log_level. If not set, the logging level used will be the one defined for log_level in the global configuration file section.
This work was inspired in fluent-logger-python
salt.log.handlers.log4mongo_mod Log4Mongo Logging Handler This module provides a logging handler for sending salt logs to MongoDB
Configuration In the salt configuration file (e.g. /etc/salt/{master,minion}):
log4mongo_handler: host: mongodb_host port: 27017 database_name: logs collection: salt_logs username: logging password: reindeerflotilla write_concern: 0 log_level: warning
Log Level If not set, the log_level will be set to the level defined in the global configuration file setting.
This work was inspired by the Salt logging handlers for LogStash and Sentry and by the log4mongo Python implementation.
salt.log.handlers.logstash_mod Logstash Logging Handler New in version 0.17.0.
This module provides some Logstash logging handlers.
UDP Logging Handler For versions of Logstash before 1.2.0:
In the salt configuration file:
logstash_udp_handler: host: 127.0.0.1 port: 9999 version: 0 msg_type: logstash
In the Logstash configuration file:
input { udp { type => "udp-type" format => "json_event" } }
For version 1.2.0 of Logstash and newer:
logstash_udp_handler: host: 127.0.0.1 port: 9999 version: 1 msg_type: logstash
input { udp { port => 9999 codec => json } }
Please read the UDP input configuration page for additional information.
ZeroMQ Logging Handler For versions of Logstash before 1.2.0:
logstash_zmq_handler: address: tcp://127.0.0.1:2021 version: 0
input { zeromq { type => "zeromq-type" mode => "server" topology => "pubsub" address => "tcp://0.0.0.0:2021" charset => "UTF-8" format => "json_event" } }
logstash_zmq_handler: address: tcp://127.0.0.1:2021 version: 1
input { zeromq { topology => "pubsub" address => "tcp://0.0.0.0:2021" codec => json } }
Please read the ZeroMQ input configuration page for additional information.
Important Logstash Setting
One of the most important settings that you should not forget on your Logstash configuration file regarding these logging handlers is format.  Both the UDP and ZeroMQ inputs need to have format as json_event which is what we send over the wire.
Log Level Both the logstash_udp_handler and the logstash_zmq_handler configuration sections accept an additional setting log_level. If not set, the logging level used will be the one defined for log_level in the global configuration file section.
HWM The high water mark for the ZMQ socket setting. Only applicable for the logstash_zmq_handler.
This work was inspired in pylogstash, python-logstash, canary and the PyZMQ logging handler.
salt.log.handlers.sentry_mod Sentry Logging Handler New in version 0.17.0.
This module provides a Sentry logging handler. Sentry is an open source error tracking platform that provides deep context about exceptions that happen in production. Details about stack traces along with the context variables available at the time of the exception are easily browsable and filterable from the online interface. For more details please see Sentry.
The Raven library needs to be installed on the system for this logging handler to be available.
Configuring the python Sentry client, Raven, should be done under the sentry_handler configuration key. Additional context may be provided for corresponding grain item(s).  At the bare minimum, you need to define the DSN. As an example:
sentry_handler: dsn: https://pub-key:secret-key@app.getsentry.com/app-id
More complex configurations can be achieved, for example:
sentry_handler: servers: - https://sentry.example.com - http://192.168.1.1 project: app-id public_key: deadbeefdeadbeefdeadbeefdeadbeef secret_key: beefdeadbeefdeadbeefdeadbeefdead context: - os - master - saltversion - cpuarch - ec2.tags.environment
The public_key and secret_key variables are not supported with Sentry > 3.0. The DSN key should be used instead.
All the client configuration keys are supported, please see the Raven client documentation.
The default logging level for the sentry handler is ERROR. If you wish to define a different one, define log_level under the sentry_handler configuration key:
sentry_handler: dsn: https://pub-key:secret-key@app.getsentry.com/app-id log_level: warning
The available log levels are those also available for the salt cli tools and configuration; salt --help should give you the required information.
Threaded Transports Raven's documents rightly suggest using its threaded transport for critical applications. However, don't forget that if you start having troubles with Salt after enabling the threaded transport, please try switching to a non-threaded transport to see if that fixes your problem.
Salt File Server Salt comes with a simple file server suitable for distributing files to the Salt minions. The file server is a stateless ZeroMQ server that is built into the Salt master.
The main intent of the Salt file server is to present files for use in the Salt state system. With this said, the Salt file server can be used for any general file transfer from the master to the minions.
File Server Backends In Salt 0.12.0, the modular fileserver was introduced. This feature added the ability for the Salt Master to integrate different file server backends. File server backends allow the Salt file server to act as a transparent bridge to external resources. A good example of this is the git backend, which allows Salt to serve files sourced from one or more git repositories, but there are several others as well. Click here for a full list of Salt's fileserver backends.
Enabling a Fileserver Backend Fileserver backends can be enabled with the fileserver_backend option.
fileserver_backend: - git
See the documentation for each backend to find the correct value to add to fileserver_backend in order to enable them.
Using Multiple Backends If fileserver_backend is not defined in the Master config file, Salt will use the roots backend, but the fileserver_backend option supports multiple backends. When more than one backend is in use, the files from the enabled backends are merged into a single virtual filesystem. When a file is requested, the backends will be searched in order for that file, and the first backend to match will be the one which returns the file.
fileserver_backend: - roots - git
With this configuration, the environments and files defined in the file_roots parameter will be searched first, and if the file is not found then the git repositories defined in gitfs_remotes will be searched.
Defining Environments Just as the order of the values in fileserver_backend matters, so too does the order in which different sources are defined within a fileserver environment. For example, given the below file_roots configuration, if both /srv/salt/dev/foo.txt and /srv/salt/prod/foo.txt exist on the Master, then salt://foo.txt would point to /srv/salt/dev/foo.txt in the dev environment, but it would point to /srv/salt/prod/foo.txt in the base environment.
file_roots: base: - /srv/salt/prod qa: - /srv/salt/qa - /srv/salt/prod dev: - /srv/salt/dev - /srv/salt/qa - /srv/salt/prod
Similarly, when using the git backend, if both repositories defined below have a hotfix23 branch/tag, and both of them also contain the file bar.txt in the root of the repository at that branch/tag, then salt://bar.txt in the hotfix23 environment would be served from the first repository.
gitfs_remotes: - https://mydomain.tld/repos/first.git - https://mydomain.tld/repos/second.git
NOTE: Environments map differently based on the fileserver backend. For instance, the mappings are explicitly defined in roots backend, while in the VCS backends (git, hg, svn) the environments are created from branches/tags/bookmarks/etc. For the minion backend, the files are all in a single environment, which is specified by the minionfs_env option.
See the documentation for each backend for a more detailed explanation of how environments are mapped.
Requesting Files from Specific Environments The Salt fileserver supports multiple environments, allowing for SLS files and other files to be isolated for better organization.
For the default backend (called roots), environments are defined using the roots option.  Other backends (such as gitfs) define environments in their own ways. For a list of available fileserver backends, see here.
Querystring Syntax Any salt:// file URL can specify its fileserver environment using a querystring syntax, like so:
salt://path/to/file?saltenv=foo
In Reactor configurations, this method must be used to pull files from an environment other than base.
In States Minions can be instructed which environment to use both globally, and for a single state, and multiple methods for each are available:
Globally A minion can be pinned to an environment using the environment option in the minion config file.
Additionally, the environment can be set for a single call to the following functions:
• state.apply
• state.highstate
• state.sls
• state.top
NOTE: When the saltenv parameter is used to trigger a highstate using either state.apply or state.highstate, only states from that environment will be applied.
On a Per-State Basis Within an individual state, there are two ways of specifying the environment.  The first is to add a saltenv argument to the state. This example will pull the file from the config environment:
/etc/foo/bar.conf: file.managed: - source: salt://foo/bar.conf - user: foo - mode: 600 - saltenv: config
Another way of doing the same thing is to use the querystring syntax described above:
/etc/foo/bar.conf: file.managed: - source: salt://foo/bar.conf?saltenv=config - user: foo - mode: 600
NOTE: Specifying the environment using either of the above methods is only necessary in cases where a state from one environment needs to access files from another environment. If the SLS file containing this state was in the config environment, then it would look in that environment by default.
File Server Configuration The Salt file server is a high performance file server written in ZeroMQ. It manages large files quickly and with little overhead, and has been optimized to handle small files in an extremely efficient manner.
The Salt file server is an environment aware file server. This means that files can be allocated within many root directories and accessed by specifying both the file path and the environment to search. The individual environments can span across multiple directory roots to create overlays and to allow for files to be organized in many flexible ways.