neulab/tldr huggingface dataset:
Running pre-defined or arbitrary commands on remote hosts, also known as remote execution, is the core function of Salt. The following links explore modules and returners, which are two key elements of remote execution.
Salt Execution Modules
Salt execution modules are called by the remote execution system to perform a wide variety of tasks. These modules provide functionality such as installing packages, restarting a service, running a remote command, transferring files, and so on.
Full list of execution modules Contains: a list of core modules that ship with Salt.
Writing execution modules Contains: a guide on how to write Salt modules.
Running Commands on Salt Minions Salt can be controlled by a command line client by the root user on the Salt master. The Salt command line client uses the Salt client API to communicate with the Salt master server. The Salt client is straightforward and simple to use.
Using the Salt client commands can be easily sent to the minions.
Each of these commands accepts an explicit --config option to point to either the master or minion configuration file.  If this option is not provided and the default configuration file does not exist then Salt falls back to use the environment variables SALT_MASTER_CONFIG and SALT_MINION_CONFIG.
SEE ALSO: Configuration
Using the Salt Command The Salt command needs a few components to send information to the Salt minions. The target minions need to be defined, the function to call and any arguments the function requires.
Defining the Target Minions The first argument passed to salt, defines the target minions, the target minions are accessed via their hostname. The default target type is a bash glob:
salt '*foo.com' sys.doc
Salt can also define the target minions with regular expressions:
salt -E '.*' cmd.run 'ls -l | grep foo'
Or to explicitly list hosts, salt can take a list:
salt -L foo.bar.baz,quo.qux cmd.run 'ps aux | grep foo'
More Powerful Targets See Targeting.
Calling the Function The function to call on the specified target is placed after the target specification.
New in version 0.9.8.
Functions may also accept arguments, space-delimited:
Arguments are formatted as YAML:
salt '*' cmd.run 'echo "Hello: $FIRST_NAME"' env='{FIRST_NAME: "Joe"}'
Note: dictionaries must have curly braces around them (like the env keyword argument above).  This was changed in 0.15.1: in the above example, the first argument used to be parsed as the dictionary {'echo "Hello': '$FIRST_NAME"'}. This was generally not the expected behavior.
If you want to test what parameters are actually passed to a module, use the test.arg_repr command:
salt '*' test.arg_repr 'echo "Hello: $FIRST_NAME"' env='{FIRST_NAME: "Joe"}'
Finding available minion functions The Salt functions are self documenting, all of the function documentation can be retried from the minions via the sys.doc() function:
Compound Command Execution If a series of commands needs to be sent to a single target specification then the commands can be sent in a single publish. This can make gathering groups of information faster, and lowers the stress on the network for repeated commands.
Compound command execution works by sending a list of functions and arguments instead of sending a single function and argument. The functions are executed on the minion in the order they are defined on the command line, and then the data from all of the commands are returned in a dictionary. This means that the set of commands are called in a predictable way, and the returned data can be easily interpreted.
Executing compound commands if done by passing a comma delimited list of functions, followed by a comma delimited list of arguments:
salt '*' cmd.run,test.ping,test.echo 'cat /proc/cpuinfo',,foo
The trick to look out for here, is that if a function is being passed no arguments, then there needs to be a placeholder for the absent arguments. This is why in the above example, there are two commas right next to each other.  test.ping takes no arguments, so we need to add another comma, otherwise Salt would attempt to pass "foo" to test.ping.
If you need to pass arguments that include commas, then make sure you add spaces around the commas that separate arguments. For example:
salt '*' cmd.run,test.ping,test.echo 'echo "1,2,3"' , , foo
You may change the arguments separator using the --args-separator option:
salt --args-separator=:: '*' some.fun,test.echo params with , comma :: foo
CLI Completion Shell completion scripts for the Salt CLI are available in the pkg Salt source directory.
Writing Execution Modules Salt execution modules are the functions called by the salt command.
Modules Are Easy to Write! Writing Salt execution modules is straightforward.
A Salt execution module is a Python or Cython module placed in a directory called _modules/ at the root of the Salt fileserver. When using the default fileserver backend (i.e. roots), unless environments are otherwise defined in the file_roots config option, the _modules/ directory would be located in /srv/salt/_modules on most systems.
Modules placed in _modules/ will be synced to the minions when any of the following Salt functions are called:
• state.highstate (or state.apply with no state argument)
• saltutil.sync_modules
Modules placed in _modules/ will be synced to masters when any of the following Salt runners are called:
Note that a module's default name is its filename (i.e. foo.py becomes module foo), but that its name can be overridden by using a __virtual__ function.
If a Salt module has errors and cannot be imported, the Salt minion will continue to load without issue and the module with errors will simply be omitted.
If adding a Cython module the file must be named <modulename>.pyx so that the loader knows that the module needs to be imported as a Cython module. The compilation of the Cython module is automatic and happens when the minion starts, so only the *.pyx file is required.
Zip Archives as Modules Python 2.3 and higher allows developers to directly import zip archives containing Python code. By setting enable_zip_modules to True in the minion config, the Salt loader will be able to import .zip files in this fashion. This allows Salt module developers to package dependencies with their modules for ease of deployment, isolation, etc.
For a user, Zip Archive modules behave just like other modules.  When executing a function from a module provided as the file my_module.zip, a user would call a function within that module as my_module.<function>.
Creating a Zip Archive Module A Zip Archive module is structured similarly to a simple Python package. The .zip file contains a single directory with the same name as the module.  The module code traditionally in <module_name>.py goes in <module_name>/__init__.py.  The dependency packages are subdirectories of <module_name>/.
Here is an example directory structure for the lumberjack module, which has two library dependencies (sleep and work) to be included.
modules $ ls -R lumberjack __init__.py     sleep           work
lumberjack/sleep: __init__.py
lumberjack/work: __init__.py
The contents of lumberjack/__init__.py show how to import and use these included libraries.
# Libraries included in lumberjack.zip from lumberjack import sleep, work
def is_ok(person): """Checks whether a person is really a lumberjack""" return sleep.all_night(person) and work.all_day(person)
Then, create the zip:
modules $ zip -r lumberjack lumberjack adding: lumberjack/ (stored 0%) adding: lumberjack/__init__.py (deflated 39%) adding: lumberjack/sleep/ (stored 0%) adding: lumberjack/sleep/__init__.py (deflated 7%) adding: lumberjack/work/ (stored 0%) adding: lumberjack/work/__init__.py (deflated 7%) modules $ unzip -l lumberjack.zip Archive:  lumberjack.zip Length     Date   Time    Name --------    ----   ----    ---- 0  08-21-15 20:08   lumberjack/ 348  08-21-15 20:08   lumberjack/__init__.py 0  08-21-15 19:53   lumberjack/sleep/ 83  08-21-15 19:53   lumberjack/sleep/__init__.py 0  08-21-15 19:53   lumberjack/work/ 81  08-21-15 19:21   lumberjack/work/__init__.py --------                   ------- 512                   6 files
Once placed in file_roots, Salt users can distribute and use lumberjack.zip like any other module.
$ sudo salt minion1 saltutil.sync_modules minion1: - modules.lumberjack $ sudo salt minion1 lumberjack.is_ok 'Michael Palin' minion1: True
Cross Calling Execution Modules All of the Salt execution modules are available to each other and modules can call functions available in other execution modules.
The variable __salt__ is packed into the modules after they are loaded into the Salt minion.
The __salt__ variable is a Python dictionary containing all of the Salt functions. Dictionary keys are strings representing the names of the modules and the values are the functions themselves.
Salt modules can be cross-called by accessing the value in the __salt__ dict:
def foo(bar): return __salt__["cmd.run"](bar)
This code will call the run function in the cmd module and pass the argument bar to it.
Calling Execution Modules on the Salt Master New in version 2016.11.0.
Execution modules can now also be called via the salt-run command using the salt runner.
Preloaded Execution Module Data When interacting with execution modules often it is nice to be able to read information dynamically about the minion or to load in configuration parameters for a module.
Salt allows for different types of data to be loaded into the modules by the minion.
Grains Data The values detected by the Salt Grains on the minion are available in a Python dictionary named __grains__ and can be accessed from within callable objects in the Python modules.
To see the contents of the grains dictionary for a given system in your deployment run the grains.items() function:
salt 'hostname' grains.items --output=pprint
Any value in a grains dictionary can be accessed as any other Python dictionary. For example, the grain representing the minion ID is stored in the id key and from an execution module, the value would be stored in __grains__['id'].
Module Configuration Since parameters for configuring a module may be desired, Salt allows for configuration information from the  minion configuration file to be passed to execution modules.
Since the minion configuration file is a YAML document, arbitrary configuration data can be passed in the minion config that is read by the modules. It is therefore strongly recommended that the values passed in the configuration file match the module name. A value intended for the test execution module should be named test.<value>.
The test execution module contains usage of the module configuration and the default configuration file for the minion contains the information and format used to pass data to the modules. salt.modules.test, conf/minion.
__init__ Function If you want your module to have different execution modes based on minion configuration, you can use the __init__(opts) function to perform initial module setup. The parameter opts is the complete minion configuration, as also available in the __opts__ dict.
""" Cheese module initialization example """
def __init__(opts): """ Allow foreign imports if configured to do so """ if opts.get("cheese.allow_foreign", False): _enable_foreign_products()
Strings and Unicode An execution  module author should always assume that strings fed to the module have already decoded from strings into Unicode. In Python 2, these will be of type 'Unicode' and in Python 3 they will be of type str. Calling from a state to other Salt sub-systems, should pass Unicode (or bytes if passing binary data). In the rare event that a state needs to write directly to disk, Unicode should be encoded to a string immediately before writing to disk. An author may use __salt_system_encoding__ to learn what the encoding type of the system is.  For example, 'my_string'.encode(__salt_system_encoding__').
Outputter Configuration Since execution module functions can return different data, and the way the data is printed can greatly change the presentation, Salt allows for a specific outputter to be set on a function-by-function basis.
This is done be declaring an __outputter__ dictionary in the global scope of the module.  The __outputter__ dictionary contains a mapping of function names to Salt outputters.
__outputter__ = {"run": "txt"}
This will ensure that the txt outputter is used to display output from the run function.
Virtual Modules Virtual modules let you override the name of a module in order to use the same name to refer to one of several similar modules. The specific module that is loaded for a virtual name is selected based on the current platform or environment.
For example, packages are managed across platforms using the pkg module. pkg is a virtual module name that is an alias for the specific package manager module that is loaded on a specific system (for example, yumpkg on RHEL/CentOS systems , and aptpkg on Ubuntu).
Virtual module names are set using the __virtual__ function and the virtual name.
__virtual__ Function The __virtual__ function returns either a string, True, False, or False with an error string. If a string is returned then the module is loaded using the name of the string as the virtual name. If True is returned the module is loaded using the current module name. If False is returned the module is not loaded. False lets the module perform system checks and prevent loading if dependencies are not met.
Since __virtual__ is called before the module is loaded, __salt__ will be unreliable as not all modules will be available at this point in time. The __pillar__ and __grains__ "dunder" dictionaries are available however.
NOTE: Modules which return a string from __virtual__ that is already used by a module that ships with Salt will _override_ the stock module.
Returning Error Information from __virtual__ Optionally, Salt plugin modules, such as execution, state, returner, beacon, etc. modules may additionally return a string containing the reason that a module could not be loaded.  For example, an execution module called cheese and a corresponding state module also called cheese, both depending on a utility called enzymes should have __virtual__ functions that handle the case when the dependency is unavailable.
""" Cheese execution (or returner/beacon/etc.) module """ try: import enzymes
HAS_ENZYMES = True except ImportError: HAS_ENZYMES = False
def __virtual__(): """ only load cheese if enzymes are available """ if HAS_ENZYMES: return "cheese" else: return ( False, "The cheese execution module cannot be loaded: enzymes unavailable.", )
def slice(): pass
""" Cheese state module. Note that this works in state modules because it is guaranteed that execution modules are loaded first """
def __virtual__(): """ only load cheese if enzymes are available """ # predicate loading of the cheese state on the corresponding execution module if "cheese.slice" in __salt__: return "cheese" else: return False, "The cheese state module cannot be loaded: enzymes unavailable."
Examples The package manager modules are among the best examples of using the __virtual__ function. A table of all the virtual pkg modules can be found here.
Overriding Virtual Module Providers Salt often uses OS grains (os, osrelease, os_family, etc.) to determine which module should be loaded as the virtual module for pkg, service, etc. Sometimes this OS detection is incomplete, with new distros popping up, existing distros changing init systems, etc. The virtual modules likely to be affected by this are in the list below (click each item for more information):
• pkg
• service
• user
• shadow
• group
If Salt is using the wrong module for one of these, first of all, please report it on the issue tracker, so that this issue can be resolved for a future release. To make it easier to troubleshoot, please also provide the grains.items output, taking care to redact any sensitive information.
Then, while waiting for the SaltStack development team to fix the issue, Salt can be made to use the correct module using the providers option in the minion config file:
providers: service: systemd pkg: aptpkg
The above example will force the minion to use the systemd module to provide service management, and the aptpkg module to provide package management.
Logging Restrictions As a rule, logging should not be done anywhere in a Salt module before it is loaded. This rule apples to all code that would run before the __virtual__() function, as well as the code within the __virtual__() function itself.
If logging statements are made before the virtual function determines if the module should be loaded, then those logging statements will be called repeatedly. This clutters up log files unnecessarily.
Exceptions may be considered for logging statements made at the trace level.  However, it is better to provide the necessary information by another means.  One method is to return error information in the __virtual__() function.
__virtualname__ __virtualname__ is a variable that is used by the documentation build system to know the virtual name of a module without calling the __virtual__ function. Modules that return a string from the __virtual__ function must also set the __virtualname__ variable.
To avoid setting the virtual name string twice, you can implement __virtual__ to return the value set for __virtualname__ using a pattern similar to the following:
# Define the module's virtual name __virtualname__ = "pkg"
def __virtual__(): """ Confine this module to Mac OS with Homebrew. """
if salt.utils.path.which("brew") and __grains__["os"] == "MacOS": return __virtualname__ return False
The __virtual__() function can return a True or False boolean, a tuple, or a string. If it returns a True value, this __virtualname__ module-level attribute can be set as seen in the above example. This is the string that the module should be referred to as.
When __virtual__() returns a tuple, the first item should be a boolean and the second should be a string. This is typically done when the module should not load. The first value of the tuple is False and the second is the error message to display for why the module did not load.
def __virtual__(): """ Only load if git exists on the system """ if salt.utils.path.which("git") is None: return (False, "The git execution module cannot be loaded: git unavailable.") else: return True
Documentation Salt execution modules are documented. The sys.doc() function will return the documentation for all available modules:
The sys.doc function simply prints out the docstrings found in the modules; when writing Salt execution modules, please follow the formatting conventions for docstrings as they appear in the other modules.
Adding Documentation to Salt Modules It is strongly suggested that all Salt modules have documentation added.
To add documentation add a Python docstring to the function.
def spam(eggs): """ A function to make some spam with eggs!
salt '*' test.spam eggs """ return eggs
Now when the sys.doc call is executed the docstring will be cleanly returned to the calling terminal.
Documentation added to execution modules in docstrings will automatically be added to the online web-based documentation.
Add Execution Module Metadata When writing a Python docstring for an execution module, add information about the module using the following field lists:
:maintainer:    Thomas Hatch <thatch@saltstack.com, Seth House <shouse@saltstack.com> :maturity:      new :depends:       python-mysqldb :platform:      all
The maintainer field is a comma-delimited list of developers who help maintain this module.
The maturity field indicates the level of quality and testing for this module.  Standard labels will be determined.
The depends field is a comma-delimited list of modules that this module depends on.
The platform field is a comma-delimited list of platforms that this module is known to run on.
Log Output You can call the logger from custom modules to write messages to the minion logs. The following code snippet demonstrates writing log messages:
log.info("Here is Some Information") log.warning("You Should Not Do That") log.error("It Is Busted")
Aliasing Functions Sometimes one wishes to use a function name that would shadow a python built-in.  A common example would be set(). To support this, append an underscore to the function definition, def set_():, and use the __func_alias__ feature to provide an alias to the function.
__func_alias__ is a dictionary where each key is the name of a function in the module, and each value is a string representing the alias for that function.  When calling an aliased function from a different execution module, state module, or from the cli, the alias name should be used.
__func_alias__ = { "set_": "set", "list_": "list", }
Private Functions In Salt, Python callable objects contained within an execution module are made available to the Salt minion for use. The only exception to this rule is a callable object with a name starting with an underscore _.
Objects Loaded Into the Salt Minion def foo(bar): return bar
Objects NOT Loaded into the Salt Minion def _foobar(baz):  # Preceded with an _ return baz
cheese = {}  # Not a callable Python object
Useful Decorators for Modules Depends Decorator When writing execution modules there are many times where some of the module will work on all hosts but some functions have an external dependency, such as a service that needs to be installed or a binary that needs to be present on the system.
Instead of trying to wrap much of the code in large try/except blocks, a decorator can be used.
If the dependencies passed to the decorator don't exist, then the salt minion will remove those functions from the module on that host.
If a fallback_function is defined, it will replace the function instead of removing it
from salt.utils.decorators import depends
try: import dependency_that_sometimes_exists except ImportError as e: log.trace("Failed to import dependency_that_sometimes_exists: {0}".format(e))
@depends("dependency_that_sometimes_exists") def foo(): """ Function with a dependency on the "dependency_that_sometimes_exists" module, if the "dependency_that_sometimes_exists" is missing this function will not exist """ return True
def _fallback(): """ Fallback function for the depends decorator to replace a function with """ return '"dependency_that_sometimes_exists" needs to be installed for this function to exist'
@depends("dependency_that_sometimes_exists", fallback_function=_fallback) def foo(): """ Function with a dependency on the "dependency_that_sometimes_exists" module. If the "dependency_that_sometimes_exists" is missing this function will be replaced with "_fallback" """ return True
In addition to global dependencies the depends decorator also supports raw booleans.
HAS_DEP = False try: import dependency_that_sometimes_exists
HAS_DEP = True except ImportError: pass
@depends(HAS_DEP) def foo(): return True
Executors Executors are used by minion to execute module functions. Executors can be used to modify the functions behavior, do any pre-execution steps or execute in a specific way like sudo executor.
Executors could be passed as a list and they will be used one-by-one in the order. If an executor returns None the next one will be called. If an executor returns non-None the execution sequence is terminated and the returned value is used as a result. It's a way executor could control modules execution working as a filter. Note that executor could actually not execute the function but just do something else and return None like splay executor does. In this case some other executor have to be used as a final executor that will actually execute the function. See examples below.
Executors list could be passed by minion config file in the following way:
module_executors: - splay - direct_call splaytime: 30
The same could be done by command line:
salt -t 40 --module-executors='[splay, direct_call]' --executor-opts='{splaytime: 30}' '*' test.version
And the same command called via netapi will look like this:
curl -sSk https://localhost:8000 \ -H 'Accept: application/x-yaml' \ -H 'X-Auth-Token: 697adbdc8fe971d09ae4c2a3add7248859c87079' \ -H 'Content-type: application/json' \ -d '[{ "client": "local", "tgt": "*", "fun": "test.version", "module_executors": ["splay", "direct_call"], "executor_opts": {"splaytime": 10} }]'
SEE ALSO: The full list of executors
Writing Salt Executors A Salt executor is written in a similar manner to a Salt execution module.  Executor is a python module placed into the executors folder and containing the execute function with the following signature:
def execute(opts, data, func, args, kwargs): ...
Where the args are:
opts:  Dictionary containing the minion configuration options
data:  Dictionary containing the load data including executor_opts passed via cmdline/API.
func, args, kwargs: Execution module function to be executed and its arguments. For instance the simplest direct_call executor just runs it as func(*args, **kwargs).
Returns: None if the execution sequence must be continued with the next executor.  Error string or execution result if the job is done and execution must be stopped.
Specific options could be passed to the executor via minion config or via executor_opts argument. For instance to access splaytime option set by minion config executor should access opts.get('splaytime'). To access the option set by commandline or API data.get('executor_opts', {}).get('splaytime') should be used. So if an option is safe and must be accessible by user executor should check it in both places, but if an option is unsafe it should be read from the only config ignoring the passed request data.
There is also a function named all_missing_func which the name of the func is passed, which can be used to verify if the command should still be run, even if it is not loaded in minion_mods.
Salt contains a robust and flexible configuration management framework, which is built on the remote execution core. This framework executes on the minions, allowing effortless, simultaneous configuration of tens of thousands of hosts, by rendering language specific state files. The following links provide resources to learn more about state and renderers.
States Express the state of a host using small, easy to read, easy to understand configuration files. No programming required.
Full list of states Contains: list of install packages, create users, transfer files, start services, and so on.
Pillar System Contains: description of Salt's Pillar system.
Highstate data structure Contains: a dry vocabulary and technical representation of the configuration format that states represent.
Writing states Contains: a guide on how to write Salt state modules, easily extending Salt to directly manage more software.
NOTE: Salt execution modules are different from state modules and cannot be called as a state in an SLS file. In other words, this will not work:
moe: user.rename: - new_name: larry - onlyif: id moe
You must use the module states to call execution modules directly. Here's an example:
rename_moe: module.run: - name: user.rename - m_name: moe - new_name: larry - onlyif: id moe
Renderers Renderers use state configuration files written in a variety of languages, templating engines, or files. Salt's configuration management system is, under the hood, language agnostic.
Full list of renderers Contains: a list of renderers.  YAML is one choice, but many systems are available, from alternative templating engines to the PyDSL language for rendering sls formulas.
Renderers Contains: more information about renderers. Salt states are only concerned with the ultimate highstate data structure, not how the data structure was created.
State System Reference Salt offers an interface to manage the configuration or "state" of the Salt minions. This interface is a fully capable mechanism used to enforce the state of systems from a central manager.
Mod Aggregate State Runtime Modifications New in version 2014.7.0.
The mod_aggregate system was added in the 2014.7.0 release of Salt and allows for runtime modification of the executing state data. Simply put, it allows for the data used by Salt's state system to be changed on the fly at runtime, kind of like a configuration management JIT compiler or a runtime import system. All in all, it makes Salt much more dynamic.
How it Works The best example is the pkg state. One of the major requests in Salt has long been adding the ability to install all packages defined at the same time. The mod_aggregate system makes this a reality. While executing Salt's state system, when a pkg state is reached the mod_aggregate function in the state module is called. For pkg this function scans all of the other states that are slated to run, and picks up the references to name and pkgs, then adds them to pkgs in the first state. The result is a single call to yum, apt-get, pacman, etc as part of the first package install.
How to Use it NOTE: Since this option changes the basic behavior of the state runtime, after it is enabled states should be executed using test=True to ensure that the desired behavior is preserved.
In config files The first way to enable aggregation is with a configuration option in either the master or minion configuration files. Salt will invoke mod_aggregate the first time it encounters a state module that has aggregate support.
If this option is set in the master config it will apply to all state runs on all minions, if set in the minion config it will only apply to said minion.
Enable for all states:
Enable for only specific state modules:
In states The second way to enable aggregation is with the state-level aggregate keyword. In this configuration, Salt will invoke the mod_aggregate function the first time it encounters this keyword. Any additional occurrences of the keyword will be ignored as the aggregation has already taken place.
The following example will trigger mod_aggregate when the lamp_stack state is processed resulting in a single call to the underlying package manager.
lamp_stack: pkg.installed: - pkgs: - php - mysql-client - aggregate: True
memcached: pkg.installed: - name: memcached
Adding mod_aggregate to a State Module Adding a mod_aggregate routine to an existing state module only requires adding an additional function to the state module called mod_aggregate.
The mod_aggregate function just needs to accept three parameters and return the low data to use. Since mod_aggregate is working on the state runtime level it does need to manipulate low data.
The three parameters are low, chunks, and running. The low option is the low data for the state execution which is about to be called. The chunks is the list of all of the low data dictionaries which are being executed by the runtime and the running dictionary is the return data from all of the state executions which have already be executed.
This example, simplified from the pkg state, shows how to create mod_aggregate functions:
def mod_aggregate(low, chunks, running): """ The mod_aggregate function which looks up all packages in the available low chunks and merges them into a single pkgs ref in the present low data """ pkgs = [] # What functions should we aggregate? agg_enabled = [ "installed", "latest", "removed", "purged", ] # The `low` data is just a dict with the state, function (fun) and # arguments passed in from the sls if low.get("fun") not in agg_enabled: return low # Now look into what other things are set to execute for chunk in chunks: # The state runtime uses "tags" to track completed jobs, it may # look familiar with the _|- tag = __utils__["state.gen_tag"](chunk) if tag in running: # Already ran the pkg state, skip aggregation continue if chunk.get("state") == "pkg": if "__agg__" in chunk: continue # Check for the same function if chunk.get("fun") != low.get("fun"): continue # Pull out the pkg names! if "pkgs" in chunk: pkgs.extend(chunk["pkgs"]) chunk["__agg__"] = True elif "name" in chunk: pkgs.append(chunk["name"]) chunk["__agg__"] = True if pkgs: if "pkgs" in low: low["pkgs"].extend(pkgs) else: low["pkgs"] = pkgs # The low has been modified and needs to be returned to the state # runtime for execution return low
Altering States NOTE: This documentation has been moved here.
File State Backups In 0.10.2 a new feature was added for backing up files that are replaced by the file.managed and file.recurse states. The new feature is called the backup mode. Setting the backup mode is easy, but it can be set in a number of places.
The backup_mode can be set in the minion config file:
Or it can be set for each file:
/etc/ssh/sshd_config: file.managed: - source: salt://ssh/sshd_config - backup: minion
Backed-up Files The files will be saved in the minion cachedir under the directory named file_backup. The files will be in the location relative to where they were under the root filesystem and be appended with a timestamp. This should make them easy to browse.
Interacting with Backups Starting with version 0.17.0, it will be possible to list, restore, and delete previously-created backups.
Listing The backups for a given file can be listed using file.list_backups:
# salt foo.bar.com file.list_backups /tmp/foo.txt foo.bar.com: ---------- 0: ---------- Backup Time: Sat Jul 27 2013 17:48:41.738027 Location: /var/cache/salt/minion/file_backup/tmp/foo.txt_Sat_Jul_27_17:48:41_738027_2013 Size: 13 1: ---------- Backup Time: Sat Jul 27 2013 17:48:28.369804 Location: /var/cache/salt/minion/file_backup/tmp/foo.txt_Sat_Jul_27_17:48:28_369804_2013 Size: 35
Restoring Restoring is easy using file.restore_backup, just pass the path and the numeric id found with file.list_backups:
# salt foo.bar.com file.restore_backup /tmp/foo.txt 1 foo.bar.com: ---------- comment: Successfully restored /var/cache/salt/minion/file_backup/tmp/foo.txt_Sat_Jul_27_17:48:28_369804_2013 to /tmp/foo.txt result: True
The existing file will be backed up, just in case, as can be seen if file.list_backups is run again:
# salt foo.bar.com file.list_backups /tmp/foo.txt foo.bar.com: ---------- 0: ---------- Backup Time: Sat Jul 27 2013 18:00:19.822550 Location: /var/cache/salt/minion/file_backup/tmp/foo.txt_Sat_Jul_27_18:00:19_822550_2013 Size: 53 1: ---------- Backup Time: Sat Jul 27 2013 17:48:41.738027 Location: /var/cache/salt/minion/file_backup/tmp/foo.txt_Sat_Jul_27_17:48:41_738027_2013 Size: 13 2: ---------- Backup Time: Sat Jul 27 2013 17:48:28.369804 Location: /var/cache/salt/minion/file_backup/tmp/foo.txt_Sat_Jul_27_17:48:28_369804_2013 Size: 35
NOTE: Since no state is being run, restoring a file will not trigger any watches for the file. So, if you are restoring a config file for a service, it will likely still be necessary to run a service.restart.
Deleting Deleting backups can be done using file.delete_backup:
# salt foo.bar.com file.delete_backup /tmp/foo.txt 0 foo.bar.com: ---------- comment: Successfully removed /var/cache/salt/minion/file_backup/tmp/foo.txt_Sat_Jul_27_18:00:19_822550_2013 result: True
Understanding State Compiler Ordering NOTE: This tutorial is an intermediate level tutorial. Some basic understanding of the state system and writing Salt Formulas is assumed.
Salt's state system is built to deliver all of the power of configuration management systems without sacrificing simplicity. This tutorial is made to help users understand in detail just how the order is defined for state executions in Salt.
This tutorial is written to represent the behavior of Salt as of version 0.17.0.
Compiler Basics To understand ordering in depth some very basic knowledge about the state compiler is very helpful. No need to worry though, this is very high level!
High Data and Low Data When defining Salt Formulas in YAML the data that is being represented is referred to by the compiler as High Data. When the data is initially loaded into the compiler it is a single large python dictionary, this dictionary can be viewed raw by running:
salt '*' state.show_highstate
This "High Data" structure is then compiled down to "Low Data". The Low Data is what is matched up to create individual executions in Salt's configuration management system. The low data is an ordered list of single state calls to execute. Once the low data is compiled the evaluation order can be seen.
The low data can be viewed by running:
salt '*' state.show_lowstate
NOTE: The state execution module contains MANY functions for evaluating the state system and is well worth a read! These routines can be very useful when debugging states or to help deepen one's understanding of Salt's state system.
As an example, a state written thusly:
apache: pkg.installed: - name: httpd service.running: - name: httpd - watch: - file: apache_conf - pkg: apache
apache_conf: file.managed: - name: /etc/httpd/conf.d/httpd.conf - source: salt://apache/httpd.conf
Will have High Data which looks like this represented in json:
{ "apache": { "pkg": [ { "name": "httpd" }, "installed", { "order": 10000 } ], "service": [ { "name": "httpd" }, { "watch": [ { "file": "apache_conf" }, { "pkg": "apache" } ] }, "running", { "order": 10001 } ], "__sls__": "blah", "__env__": "base" }, "apache_conf": { "file": [ { "name": "/etc/httpd/conf.d/httpd.conf" }, { "source": "salt://apache/httpd.conf" }, "managed", { "order": 10002 } ], "__sls__": "blah", "__env__": "base" } }
The subsequent Low Data will look like this:
[ { "name": "httpd", "state": "pkg", "__id__": "apache", "fun": "installed", "__env__": "base", "__sls__": "blah", "order": 10000 }, { "name": "httpd", "watch": [ { "file": "apache_conf" }, { "pkg": "apache" } ], "state": "service", "__id__": "apache", "fun": "running", "__env__": "base", "__sls__": "blah", "order": 10001 }, { "name": "/etc/httpd/conf.d/httpd.conf", "source": "salt://apache/httpd.conf", "state": "file", "__id__": "apache_conf", "fun": "managed", "__env__": "base", "__sls__": "blah", "order": 10002 } ]
This tutorial discusses the Low Data evaluation and the state runtime.
Ordering Layers Salt defines 2 order interfaces which are evaluated in the state runtime and defines these orders in a number of passes.
Definition Order NOTE: The Definition Order system can be disabled by turning the option state_auto_order to False in the master configuration file.
The top level of ordering is the Definition Order. The Definition Order is the order in which states are defined in salt formulas. This is very straightforward on basic states which do not contain include statements or a top file, as the states are just ordered from the top of the file, but the include system starts to bring in some simple rules for how the Definition Order is defined.
Looking back at the "Low Data" and "High Data" shown above, the order key has been transparently added to the data to enable the Definition Order.
The Include Statement Basically, if there is an include statement in a formula, then the formulas which are included will be run BEFORE the contents of the formula which is including them. Also, the include statement is a list, so they will be loaded in the order in which they are included.
In the following case:
foo.sls
include: - bar - baz
bar.sls
include: - quo
baz.sls
include: - qux
In the above case if state.apply foo were called then the formulas will be loaded in the following order:
1. quo
2. bar
3. qux
4. baz
5. foo
The order Flag The Definition Order happens transparently in the background, but the ordering can be explicitly overridden using the order flag in states:
apache: pkg.installed: - name: httpd - order: 1
This order flag will over ride the definition order, this makes it very simple to create states that are always executed first, last or in specific stages, a great example is defining a number of package repositories that need to be set up before anything else, or final checks that need to be run at the end of a state run by using order: last or order: -1.
When the order flag is explicitly set the Definition Order system will omit setting an order for that state and directly use the order flag defined.
Lexicographical Fall-back Salt states were written to ALWAYS execute in the same order. Before the introduction of Definition Order in version 0.17.0 everything was ordered lexicographically according to the name of the state, then function then id.
This is the way Salt has always ensured that states always run in the same order regardless of where they are deployed, the addition of the Definition Order method mealy makes this finite ordering easier to follow.
The lexicographical ordering is still applied but it only has any effect when two order statements collide. This means that if multiple states are assigned the same order number that they will fall back to lexicographical ordering to ensure that every execution still happens in a finite order.
NOTE: If running with state_auto_order: False the order key is not set automatically, since the Lexicographical order can be derived from other keys.
Requisite Ordering Salt states are fully declarative, in that they are written to declare the state in which a system should be. This means that components can require that other components have been set up successfully. Unlike the other ordering systems, the Requisite system in Salt is evaluated at runtime.
The requisite system is also built to ensure that the ordering of execution never changes, but is always the same for a given set of states. This is accomplished by using a runtime that processes states in a completely predictable order instead of using an event loop based system like other declarative configuration management systems.
Runtime Requisite Evaluation The requisite system is evaluated as the components are found, and the requisites are always evaluated in the same order. This explanation will be followed by an example, as the raw explanation may be a little dizzying at first as it creates a linear dependency evaluation sequence.
The "Low Data" is an ordered list or dictionaries, the state runtime evaluates each dictionary in the order in which they are arranged in the list. When evaluating a single dictionary it is checked for requisites, requisites are evaluated in order, require then watch then prereq.
NOTE: If using requisite in statements like require_in and watch_in these will be compiled down to require and watch statements before runtime evaluation.
Each requisite contains an ordered list of requisites, these requisites are looked up in the list of dictionaries and then executed. Once all requisites have been evaluated and executed then the requiring state can safely be run (or not run if requisites have not been met).
This means that the requisites are always evaluated in the same order, again ensuring one of the core design principals of Salt's State system to ensure that execution is always finite is intact.
Simple Runtime Evaluation Example Given the above "Low Data" the states will be evaluated in the following order:
1. The pkg.installed is executed ensuring that the apache package is installed, it contains no requisites and is therefore the first defined state to execute.
2. The service.running state is evaluated but NOT executed, a watch requisite is found, therefore they are read in order, the runtime first checks for the file, sees that it has not been executed and calls for the file state to be evaluated.
3. The file state is evaluated AND executed, since it, like the pkg state does not contain any requisites.
4. The evaluation of the service state continues, it next checks the pkg requisite and sees that it is met, with all requisites met the service state is now executed.
Best Practice The best practice in Salt is to choose a method and stick with it, official states are written using requisites for all associations since requisites create clean, traceable dependency trails and make for the most portable formulas. To accomplish something similar to how classical imperative systems function all requisites can be omitted and the failhard option then set to True in the master configuration, this will stop all state runs at the first instance of a failure.
In the end, using requisites creates very tight and fine grained states, not using requisites makes full sequence runs and while slightly easier to write, and gives much less control over the executions.
Extending External SLS Data Sometimes a state defined in one SLS file will need to be modified from a separate SLS file. A good example of this is when an argument needs to be overwritten or when a service needs to watch an additional state.
The Extend Declaration The standard way to extend is via the extend declaration. The extend declaration is a top level declaration like include and encapsulates ID declaration data included from other SLS files. A standard extend looks like this:
include: - http - ssh
extend: apache: file: - name: /etc/httpd/conf/httpd.conf - source: salt://http/httpd2.conf ssh-server: service: - watch: - file: /etc/ssh/banner
/etc/ssh/banner: file.managed: - source: salt://ssh/banner
A few critical things happened here, first off the SLS files that are going to be extended are included, then the extend dec is defined. Under the extend dec 2 IDs are extended, the apache ID's file state is overwritten with a new name and source. Then the ssh server is extended to watch the banner file in addition to anything it is already watching.
Extend is a Top Level Declaration This means that extend can only be called once in an sls, if it is used twice then only one of the extend blocks will be read. So this is WRONG:
extend: apache: file: - name: /etc/httpd/conf/httpd.conf - source: salt://http/httpd2.conf # Second extend will overwrite the first!! Only make one extend: ssh-server: service: - watch: - file: /etc/ssh/banner
The Requisite in Statement Since one of the most common things to do when extending another SLS is to add states for a service to watch, or anything for a watcher to watch, the requisite in statement was added to 0.9.8 to make extending the watch and require lists easier. The ssh-server extend statement above could be more cleanly defined like so:
/etc/ssh/banner: file.managed: - source: salt://ssh/banner - watch_in: - service: ssh-server
Rules to Extend By There are a few rules to remember when extending states:
1. Always include the SLS being extended with an include declaration
2. Requisites (watch and require) are appended to, everything else is overwritten
3. extend is a top level declaration, like an ID declaration, cannot be declared twice in a single SLS
4. Many IDs can be extended under the extend declaration
Failhard Global Option Normally, when a state fails Salt continues to execute the remainder of the defined states and will only refuse to execute states that require the failed state.
But the situation may exist, where you would want all state execution to stop if a single state execution fails. The capability to do this is called failing hard.
State Level Failhard A single state can have a failhard set, this means that if this individual state fails that all state execution will immediately stop. This is a great thing to do if there is a state that sets up a critical config file and setting a require for each state that reads the config would be cumbersome.  A good example of this would be setting up a package manager early on:
/etc/yum.repos.d/company.repo: file.managed: - source: salt://company/yumrepo.conf - user: root - group: root - mode: 644 - order: 1 - failhard: True
In this situation, the yum repo is going to be configured before other states, and if it fails to lay down the config file, than no other states will be executed.  It is possible to override a Global Failhard (see below) by explicitly setting it to False in the state.
Global Failhard It may be desired to have failhard be applied to every state that is executed, if this is the case, then failhard can be set in the master configuration file. Setting failhard in the master configuration file will result in failing hard when any minion gathering states from the master have a state fail.
This is NOT the default behavior, normally Salt will only fail states that require a failed state.
Using the global failhard is generally not recommended, since it can result in states not being executed or even checked. It can also be confusing to see states failhard if an admin is not actively aware that the failhard has been set.
To use the global failhard set failhard to True in the master configuration file.
Global State Arguments NOTE: This documentation has been moved here.
Highstate data structure definitions The Salt State Tree A state tree is a collection of SLS files and directories that live under the directory specified in file_roots.
NOTE: Directory names or filenames in the state tree cannot contain a period, with the exception of the period in the .sls file suffix.
Top file The main state file that instructs minions what environment and modules to use during state execution.