neulab/tldr huggingface dataset:
Modular Systems When first working with Salt, it is not always clear where all of the modular components are and what they do. Salt comes loaded with more modular systems than many users are aware of, making Salt very easy to extend in many places.
The most commonly used modular systems are execution modules and states. But the modular systems extend well beyond the more easily exposed components and are often added to Salt to make the complete system more flexible.
Developing New Modules Interactive Debugging Sometimes debugging with print() and extra logs sprinkled everywhere is not the best strategy.
IPython is a helpful debug tool that has an interactive python environment which can be embedded in python programs.
First the system will require IPython to be installed.
# Debian apt-get install ipython
# Arch Linux pacman -Syu ipython2
# RHEL/CentOS (via EPEL) yum install python-ipython
Now, in the troubling python module, add the following line at a location where the debugger should be started:
test = "test123" import IPython
IPython.embed_kernel()
After running a Salt command that hits that line, the following will show up in the log file:
[CRITICAL] To connect another client to this kernel, use: [IPKernelApp] --existing kernel-31271.json
Now on the system that invoked embed_kernel, run the following command from a shell:
# NOTE: use ipython2 instead of ipython for Arch Linux ipython console --existing
This provides a console that has access to all the vars and functions, and even supports tab-completion.
print(test) test123
To exit IPython and continue running Salt, press Ctrl-d to logout.
The Salt Loader Salt's loader system is responsible for reading Special Module Contents and providing the context for the special Dunder Dictionaries. When modules developed for Salt's loader are imported directly, the dunder attributes won't be populated. You can use the Loader Context to work around this.
Loader Context Given the following.
# coolmod.py
def utils_func_getter(name): return __utils__[name]
You would not be able import coolmod and run utils_func_getter because __utils__ would not be defined. You must run coolmod.utils_func_getter in the context of a loader.
import coolmod import salt.loader
opts = {} utils = salt.loader.utils(opts) with salt.loader.context(utils): func = coolmod.utils_func_getter("foo.bar")
Special Module Contents These are things that may be defined by the module to influence various things.
__virtual__ __virtual_aliases__ __virtualname__ __init__ Called before __virtual__()
__proxyenabled__ grains and proxy modules
__proxyenabled__ as a list containing the names of the proxy types that the module supports.
__load__ __func_alias__ __outputter__ Dunder Dictionaries Salt provides several special "dunder" dictionaries as a convenience for Salt development.  These include __opts__, __context__, __salt__, and others. This document will describe each dictionary and detail where they exist and what information and/or functionality they provide.
The following dunder dictionaries are always defined, but may be empty
• __context__
• __pillar__
__opts__ Defined in: All modules
The __opts__ dictionary contains all of the options passed in the configuration file for the master or minion.
NOTE: In many places in salt, instead of pulling raw data from the __opts__ dict, configuration data should be pulled from the salt get functions such as config.get
__salt__["config.get"]("foo:bar")
The get functions also allow for dict traversal via the : delimiter. Consider using get functions whenever using __opts__ or __pillar__ and __grains__ (when using grains for configuration data)
The configuration file data made available in the __opts__ dictionary is the configuration data relative to the running daemon. If the modules are loaded and executed by the master, then the master configuration data is available, if the modules are executed by the minion, then the minion configuration is available. Any additional information passed into the respective configuration files is made available
__salt__ Defined in: Auth, Beacons, Engines, Execution, Executors, Outputters, Pillars, Proxies, Renderers, Returners, Runners, SDB, SSH Wrappers, State, Thorium
__salt__ contains the execution module functions. This allows for all functions to be called as they have been set up by the salt loader.
__salt__["cmd.run"]("fdisk -l") __salt__["network.ip_addrs"]()
NOTE: When used in runners or outputters, __salt__ references other runner/outputter modules, and not execution modules.
__grains__ Filled in for: Execution, Pillar, Renderer, Returner, SSH Wrapper, State.
The __grains__ dictionary contains the grains data generated by the minion that is currently being worked with. In execution modules, state modules and returners this is the grains of the minion running the calls, when generating the external pillar the __grains__ is the grains data from the minion that the pillar is being generated for.
While __grains__ is defined for every module, it's only filled in for some.
__pillar__ Filled in for: Execution, Renderer, Returner, SSH Wrapper, State
The __pillar__ dictionary contains the pillar for the respective minion.
While __pillar__ is defined for every module, it's only filled in for some.
__ext_pillar__ Filled in for: Pillar
The __ext_pillar__ dictionary contains the external pillar modules.
__context__ During a state run the __context__ dictionary persists across all states that are run and then is destroyed when the state ends.
When running an execution module __context__ persists across all module executions until the modules are refreshed; such as when saltutil.sync_all or state.apply are executed.
A great place to see how to use __context__ is in the cp.py module in salt/modules/cp.py. The fileclient authenticates with the master when it is instantiated and then is used to copy files to the minion. Rather than create a new fileclient for each file that is to be copied down, one instance of the fileclient is instantiated in the __context__ dictionary and is reused for each file. Here is an example from salt/modules/cp.py:
if not "cp.fileclient" in __context__: __context__["cp.fileclient"] = salt.fileclient.get_file_client(__opts__)
NOTE: Because __context__ may or may not have been destroyed, always be sure to check for the existence of the key in __context__ and generate the key before using it.
__utils__ Defined in: Cloud, Engine, Execution, File Server, Grain, Pillar, Proxy, Roster, Runner, SDB, State
__proxy__ Defined in: Beacon, Engine, Execution, Executor, Proxy, Renderer, Returner, State, Util
__runner__ Defined in: Engine, Roster, Thorium
NOTE: When used in engines, it should be called __runners__ (plural)
__executors__ Defined in: Executor
__ret__ Defined in: Proxy
__thorium__ Defined in: Thorium
__states__ Defined in: Renderers, State
__serializers__ Defined in: State
__sdb__ Defined in: SDB
Configuration Options A number of configuration options can affect the load process. This is a quick list of them:
• autoload_dynamic_modules (Minion)
• cython_enable (Minion, Master)
• disable_modules (Minion)
• disable_returners (Minion)
• enable_zip_modules (Minion)
• extension_modules (Master)
• extmod_whitelist (Minion, Master)
• extmod_blacklist (Minion, Master)
• whitelist_modules (Minion)
• grains_dirs (Minion)
• module_dirs (Minion, Master)
• outputter_dirs (Minion, Master)
• providers (Minion)
• render_dirs (Minion)
• returner_dirs (Minion)
• runner_dirs (Master)
• states_dirs (Minion)
• utils_dirs (Minion)
Loading Modules Modules come primarily from several sources:
• The Salt package itself
• The Salt File Server
• The extmods directory
• Secondary packages installed
Using one source to override another is not supported.
The Salt Package Salt itself ships with a large number of modules. These are part of the Salt package itself and don't require the user to do anything to use them. (Although a number of them have additional dependencies and/or configuration.)
The Salt File Server The user may add modules by simply placing them in special directories in their fileserver.
The name of the directory inside of the file server is the directory name prepended by underscore, such as:
• _grains
• _modules
• _states
Modules must be synced before they can be used. This can happen a few ways, discussed below.
NOTE: Using saltenvs besides base may not work in all contexts.
Sync Via States The minion configuration contains an option autoload_dynamic_modules which defaults to True. This option makes the state system refresh all dynamic modules when states are run. To disable this behavior set autoload_dynamic_modules to False in the minion config.
When dynamic modules are autoloaded via states, only the modules defined in the same saltenvs as the states currently being run.
Sync Via the saltutil Module The saltutil module has a number of functions that can be used to sync all or specific dynamic modules. The saltutil.sync_* execution functions and runner functions can be used to sync modules to minions and the master, respectively.
The extmods Directory Any files places in the directory set by extension_modules settings (minion, master, default /var/cache/salt/*/extmods) can also be loaded as modules. Note that these directories are also used by the saltutil.sync_* functions (mentioned above) and files may be overwritten.
Secondary Packages Third-party packages may also add modules to Salt if they are installed in the same system and Python environment as the Salt Minion or Master.
This is done via setuptools entry points:
setup( # ... entry_points={"salt.loader": ["module_dirs=spirofs.loader:module"]}, # ... )
Note that these are not synced from the Salt Master to the Minions. They must be installed independently on each Minion.
Module Types The specific names used by each loading method above are as follows. See sections below for a short summary of each of these systems.
[1]  These modules cannot be loaded from the Salt File Server.
NOTE: While it is possible to import modules directly with the import statement, it is strongly recommended that the appropriate dunder dictionary is used to access them instead. This is because a number of factors affect module names, module selection, and module overloading.
Auth The auth module system allows for external authentication routines to be easily added into Salt. The auth function needs to be implemented to satisfy the requirements of an auth module. Use the pam module as an example.
See External Authentication System for more about authentication in Salt.
Beacon • Writing Beacons
Beacons are polled by the Salt event loop to monitor non-salt processes. See Beacons for more information about the beacon system.
Cache The minion cache is used by the master to store various information about minions. See Minion Data Cache for more information.
Cloud Cloud modules are backend implementations used by Salt Cloud.
Engine Engines are open-ended services managed by the Salt daemon (both master and minion). They may interact with event loop, call other modules, or a variety of non-salt tasks. See Salt Engines for complete details.
Execution Execution modules make up the core of the functionality used by Salt to interact with client systems. The execution modules create the core system management library used by all Salt systems, including states, which interact with minion systems.
Execution modules are completely open ended in their execution. They can be used to do anything required on a minion, from installing packages to detecting information about the system. The only restraint in execution modules is that the defined functions always return a JSON serializable object.
Executor Executors control how execution modules get called. The default is to just call them, but this can be customized.
File Server The file server module system is used to create file server backends used by the Salt Master. These modules need to implement the functions used in the fileserver subsystem. Use the gitfs module as an example.
See File Server Backends for more information.
Grains • writing-grains
Grain modules define extra routines to populate grains data. All defined public functions will be executed and MUST return a Python dict object. The dict keys will be added to the grains made available to the minion.
See Grains for more.
Log Handler Log handlers allows the logs from salt (master or minion) to be sent to log aggregation systems.
Matcher Matcher modules are used to define the minion targeting expressions.  For now, it is only possible to override the existing matchers (the required CLI plumbing for custom matchers is not implemented yet).
Metaproxy Metaproxy is an abstraction layer above the existing proxy minion. It enables adding different types of proxy minions that can still load existing proxymodules.
Net API Net API modules are the actual server implementation used by Salt API.
Output The output modules supply the outputter system with routines to display data in the terminal. These modules are very simple and only require the output function to execute. The default system outputter is the nested module.
Pillar External Pillars Salt provides a mechanism for generating pillar data by calling external pillar interfaces. This document will describe an outline of an ext_pillar module.
Location Salt expects to find your ext_pillar module in the same location where it looks for other python modules. If the extension_modules option in your Salt master configuration is set, Salt will look for a pillar directory under there and load all the modules it finds. Otherwise, it will look in your Python site-packages salt/pillar directory.
Configuration The external pillars that are called when a minion refreshes its pillars is controlled by the ext_pillar option in the Salt master configuration. You can pass a single argument, a list of arguments or a dictionary of arguments to your pillar:
ext_pillar: - example_a: some argument - example_b: - argumentA - argumentB - example_c: keyA: valueA keyB: valueB
The Module Imports and Logging Import modules your external pillar module needs. You should first include generic modules that come with stock Python:
And then start logging. This is an idiomatic way of setting up logging in Salt:
Finally, load modules that are specific to what you are doing. You should catch import errors and set a flag that the __virtual__ function can use later.
try: import weird_thing
EXAMPLE_A_LOADED = True except ImportError: EXAMPLE_A_LOADED = False
Options If you define an __opts__ dictionary, it will be merged into the __opts__ dictionary handed to the ext_pillar function later. This is a good place to put default configuration items. The convention is to name things modulename.option.
__opts__ = {"example_a.someconfig": 137}
Initialization If you define an __init__ function, it will be called with the following signature:
def __init__(__opts__): # Do init work here ...
Note: The __init__ function is ran every time a particular minion causes the external pillar to be called, so don't put heavy initialization code here.  The __init__ functionality is a side-effect of the Salt loader, so it may not be as useful in pillars as it is in other Salt items.
__virtual__ If you define a __virtual__ function, you can control whether or not this module is visible. If it returns False then Salt ignores this module. If it returns a string, then that string will be how Salt identifies this external pillar in its ext_pillar configuration. If you're not renaming the module, simply return True in the __virtual__ function, which is the same as if this function did not exist, then, the name Salt's ext_pillar will use to identify this module is its conventional name in Python.
This is useful to write modules that can be installed on all Salt masters, but will only be visible if a particular piece of software your module requires is installed.
# This external pillar will be known as `example_a` def __virtual__(): if EXAMPLE_A_LOADED: return True return False
# This external pillar will be known as `something_else` __virtualname__ = "something_else"
def __virtual__(): if EXAMPLE_A_LOADED: return __virtualname__ return False
ext_pillar This is where the real work of an external pillar is done. If this module is active and has a function called ext_pillar, whenever a minion updates its pillar this function is called.
How it is called depends on how it is configured in the Salt master configuration. The first argument is always the current pillar dictionary, this contains pillar items that have already been added, starting with the data from pillar_roots, and then from any already-ran external pillars.
Using our example above:
ext_pillar(id, pillar, "some argument")  # example_a ext_pillar(id, pillar, "argumentA", "argumentB")  # example_b ext_pillar(id, pillar, keyA="valueA", keyB="valueB")  # example_c
In the example_a case, pillar will contain the items from the pillar_roots, in example_b pillar  will contain that plus the items added by example_a, and in example_c pillar will contain that plus the items added by example_b. In all three cases, id will contain the ID of the minion making the pillar request.
This function should return a dictionary, the contents of which are merged in with all of the other pillars and returned to the minion. Note: this function is called once for each minion that fetches its pillar data.
def ext_pillar(minion_id, pillar, *args, **kwargs):
my_pillar = {"external_pillar": {}}
my_pillar["external_pillar"] = get_external_pillar_dictionary()
return my_pillar
You can call pillar with the dictionary's top name to retrieve its data. From above example, 'external_pillar' is the top dictionary name. Therefore:
salt '*' pillar.get external_pillar
You shouldn't just add items to pillar and return that, since that will cause Salt to merge data that already exists. Rather, just return the items you are adding or changing. You could, however, use pillar in your module to make some decision based on pillar data that already exists.
This function has access to some useful globals:
__opts__ A dictionary of mostly Salt configuration options. If you had an __opts__ dictionary defined in your module, those values will be included.
__salt__ A dictionary of Salt module functions, useful so you don't have to duplicate functions that already exist. E.g.  __salt__['cmd.run']( 'ls -l' ) Note, runs on the master
__grains__ A dictionary of the grains of the minion making this pillar call.
Example configuration As an example, if you wanted to add external pillar via the cmd_json external pillar, add something like this to your master config:
ext_pillar: - cmd_json: 'echo {\"arg\":\"value\"}'
Reminder Just as with traditional pillars, external pillars must be refreshed in order for minions to see any fresh data:
Used to define optional external pillar systems. The pillar generated via the filesystem pillar is passed into external pillars. This is commonly used as a bridge to database data for pillar, but is also the backend to the libvirt state used to generate and sign libvirt certificates on the fly.
Proxy Proxy Minions are a way to manage devices that cannot run a full minion directly.
Renderers Renderers are the system used to render sls files into salt highdata for the state compiler. They can be as simple as the py renderer and as complex as stateconf and pydsl.
Returners Returners are used to send data from minions to external sources, commonly databases. A full returner will implement all routines to be supported as an external job cache. Use the redis returner as an example.
Roster The Roster system is used by Salt SSH to enumerate devices.
Runners Runners are purely master-side execution sequences.
SDB • Writing SDB Modules
SDB is a way to store data that's not associated with a minion. See Storing Data in Other Databases.
Serializer Primarily used with file.serialize.
State State modules are used to define the state interfaces used by Salt States.  These modules are restrictive in that they must follow a number of rules to function properly.
NOTE: State modules define the available routines in sls files. If calling an execution module directly is desired, take a look at the module state.
SPM pkgdb • SPM Development Guide: Package Database
pkgdb modules provides storage backends to the package database.
SPM pkgfiles • SPM Development Guide: Package Database
pkgfiles modules handle the actual installation.
SSH Wrapper SSH Wrapper Salt-SSH Background Salt-SSH works by creating a tar ball of salt, a bunch of python modules, and a generated short minion config. It then copies this onto the destination host over ssh, then uses that host's local python install to run salt-client --local with any requested modules.  It does not automatically copy over states or cache files and since it is uses a local file_client, modules that rely on cp.cache* functionality do not work.
SSH Wrapper modules To support cp modules or other functionality which might not otherwise work in the remote environment, a wrapper module can be created. These modules are run from the salt-master initiating the salt-ssh command and can include logic to support the needed functionality. SSH Wrapper modules are located in /salt/client/ssh/wrapper/ and are named the same as the execution module being extended. Any functions defined inside of the wrapper module are called from the salt-ssh module.function argument command rather than executing on the minion.
State Module example Running salt states on an salt-ssh minion, obviously requires the state files themselves. To support this, a state module wrapper script exists at salt/client/ssh/wrapper/state.py, and includes standard state functions like apply, sls, and highstate. When executing salt-ssh minion state.highstate, these wrapper functions are used and include the logic to walk the low_state output for that minion to determine files used, gather needed files, tar them together, transfer the tar file to the minion over ssh, and run a state on the ssh minion. This state then extracts the tar file, applies the needed states and data, and cleans up the transferred files.
Wrapper Handling From the wrapper script any invocations of __salt__['some.module']() do not run on the master which is running the wrapper, but instead magically are invoked on the minion over ssh.  Should the function being called exist in the wrapper, the wrapper function will be used instead.
One way of supporting this workflow may be to create a wrapper function which performs the needed file copy operations. Now that files are resident on the ssh minion, the next step is to run the original execution module function. But since that function name was already overridden by the wrapper, a function alias can be created in the original execution module, which can then be called from the wrapper.
Example The saltcheck module needs sls and tst files on the minion to function. The invocation of saltcheck.run_state_tests is run from the wrapper module, and is responsible for performing the needed file copy. The saltcheck execution module includes an alias line of run_state_tests_ssh = salt.utils.functools.alias_function(run_state_tests, 'run_state_tests_ssh') which creates an alias of run_state_tests with the name run_state_tests_ssh. At the end of the run_state_tests function in the wrapper module, it then calls __salt__['saltcheck.run_state_tests_ssh'](). Since this function does not exist in the wrapper script, the call is made on the remote minion, which then having the needed files, runs as expected.
Replacement execution modules for Salt SSH.
Thorium Modules for use in the Thorium event reactor.
Tokens Token stores for External Authentication. See the salt.tokens docstring for details.
NOTE: The runner to load tokens modules is saltutil.sync_eauth_tokens.
Tops Tops modules are used to convert external data sources into top file data for the state system.
Util Just utility modules to use with other modules via __utils__ (see Dunder Dictionaries).
Wheel The wheel system is used to manage master side management routines. These routines are primarily intended for the API to enable master configuration.
Salt Extend salt-extend is a templating tool for extending SaltStack. If you're looking to add a module to SaltStack, then the salt-extend utility can guide you through the process.
Command line usage See salt-extend
Choosing a template The following templates are available:
module Creates a new execution module within salt/modules/{{module_name}}.py
module_unit Creates a new execution module unit test suite within tests/unit/modules/test_{{module_name}}.py
state Creates a new state module within salt/states/{{module_name}}.py
state_unit Creates a new state module unit test suite within tests/unit/states/test_{{module_name}}.py
Adding templates 1. Create a directory under <src>/templates
2. Create a file template.yml containing properties for
• description - a description of the template
• questions - a collection of additional questions to ask the user, the name of the item will be used as the key in the context dictionary within the jinja template.
• question - The question to ask the user, as a string
• default - (optional) the default value, can contain Jinja2 template syntax and has access to the default context properties
Example template.yml description: "Execution module" questions: depending_libraries: question: "What libraries does this module depend upon?" virtual_name: question: "What module virtual name to use?" default: "{{module_name}}"
3. Create the files within <src>/templates/<your template> to match the target
NOTE: File names can contain Jinja 2 template syntax, e.g. '{{module_name}}.py}}'
Example file in the template directory print("Hello {{module_name}}") __virtual__ = "{{__virtual_name__}}"
Default context properties The default context provides the following properties
• description - A description of the template
• short_description - A short description of the module as entered by the user
• version - The version name of the next release
• module_name - The module name as entered by the user
• release_date - The current date in the format YYYY-MM-DD
• year - The current year in the format YYYY
As well as any additional properties entered from the questions section of template.yml
API salt.utils.extend module SaltStack Extend A templating tool for extending SaltStack.
Takes a template directory and merges it into a SaltStack source code directory. This tool uses Jinja2 for templating.
This tool is accessed using salt-extend
salt.utils.extend.apply_template(template_dir, output_dir, context) Apply the template from the template directory to the output using the supplied context dict.
• src (str) -- The source path
• dst (str) -- The destination path
• context (dict) -- The dictionary to inject into the Jinja template as context
salt.utils.extend.run(extension=None, name=None, description=None, salt_dir=None, merge=False, temp_dir=None) A template factory for extending the salt ecosystem
• extension (str) -- The extension type, e.g. 'module', 'state', if omitted, user will be prompted
• name (str) -- Python-friendly name for the module, if omitted, user will be prompted
• description (str) -- A description of the extension, if omitted, user will be prompted
• salt_dir (str) -- The targeted Salt source directory
• merge (bool) -- Merge with salt directory, False to keep separate, True to merge trees.
• temp_dir (str) -- The directory for generated code, if omitted, system temp will be used
Salt's Test Suite Salt comes with a powerful integration and unit test suite allowing for the fully automated run of integration and/or unit tests from a single interface. It uses the combination of pytest, nox and Kitchen Salt to run these tests. Nox is used to manage all of the test python dependencies. When you run the test runner with nox, you will be installing the same python dependencies that we use to run our test suite on PRs and branch tests.  Kitchen Salt is used to spin up our virtual machines based off of golden images. These virtual machines use the salt-jenkins sls states to configure any system dependencies.
To learn the basics of how Salt's test suite works, be sure to check out the Salt's Test Suite: An Introduction tutorial.
Nox Nox is used to manage all of the python dependencies used in the test suite and spins up the different nox sessions. You can look at the noxfile.py in the salt repo to view all of the current nox configurations. In that file you will notice various nox sessions. When creating each of these sessions, nox will create a virtualenv with the specified interpreter. Once the virtualenv is created it will also install all of the required python dependencies required for that session and run the tests.
For example if you want to run all of the tests using the zeromq transport on python3 you would need to specify the zeromq transport and python3.
As a contrast, when using the deprecated runtests.py test runner, the command would be:
nox -e 'runtests-zeromq-3(coverage=False)'
To run all the tests but on the tcp transport, you would need to specify the tcp session.
nox -e 'pytest-tcp-3(coverage=False)'
nox -e 'runtests-tcp-3(coverage=False)'
You can view all available sessions by running:
nox --list-sessions
For the most part you will only need nox to run the test suite, as this tool will install the exact same python dependencies we use to run on our test runs.  The exception to this is when a system dependency is required, for example mysql.  These system dependencies are installed with sls states managed in the salt-jenkins repo or you can manually install the dependency yourself.
System Dependencies The system dependencies are installed from the salt-jenkins repo. The golden-images-provision state is what is run to determine what dependencies to install on which platform.  We run this state only when we want to update our current VM images with new dependencies.
Kitchen Salt We also use Kitchen Salt to spin up the VM's used for testing. You can view the kitchen-salt getting started for instructions on how to install and set it up.  Kitchen Salt uses Test Kitchen to spin up the VM or container in the configured provider. Once the VM is spun up, Kitchen Salt can install salt and run a particular set of states. In the case of our branch and PR tests we create "Golden Images" which run the salt-jenkins states and install salt system dependencies beforehand. We only update these "Golden Images" when we need to upgrade or install a system dependency. You can view the kitchen-salt jenkins setup docs for instructions on how to set up Kitchen Salt similar to the jenkins environment we use to run branch and PR tests.
Test Directory Structure Salt's test suite is located in the tests/ directory in the root of Salt's codebase.
With the migration to PyTest, Salt has created a separate directory for tests that are written taking advantage of the full pottential of PyTest. These are located under tests/pytests.
As for the old test suite, it is divided into two main groups:
Within each of these groups, the directory structure roughly mirrors the structure of Salt's own codebase. Notice that there are directories for states, modules, runners, output, and more in each testing group.
The files that are housed in the modules directory of either the unit or the integration testing factions contain respective integration or unit test files for Salt execution modules.
The PyTest only tests under tests/pytests should, more or less, follow the same grouping as the old test suite.
Integration Tests The Integration section of Salt's test suite start up a number of Salt daemons to test functionality in a live environment. These daemons include two Salt Masters, one Syndic, and two Minions. This allows the Syndic interface to be tested and Master/Minion communication to be verified. All of the integration tests are executed as live Salt commands sent through the started daemons.
Integration tests are particularly good at testing modules, states, and shell commands, among other segments of Salt's ecosystem. By utilizing the integration test daemons, integration tests are easy to write. They are also SaltStack's generally preferred method of adding new tests.
The discussion in the Integration vs. Unit section of the testing tutorial is beneficial in learning why you might want to write integration tests vs. unit tests. Both testing arenas add value to Salt's test suite and you should consider adding both types of tests if possible and appropriate when contributing to Salt.
• Integration Test Documentation
Unit Tests Unit tests do not spin up any Salt daemons, but instead find their value in testing singular implementations of individual functions. Instead of testing against specific interactions, unit tests should be used to test a function's logic as well as any return or raises statements.  Unit tests also rely heavily on mocking external resources.
The discussion in the Integration vs. Unit section of the testing tutorial is useful in determining when you should consider writing unit tests instead of, or in addition to, integration tests when contributing to Salt.
• Unit Test Documentation
Running The Tests There is only one requirement to install, to quickly get started running salt's test suite: nox.
Once this requirement is installed, you can use the nox binary to run all of the tests included in Salt's test suite:
For more information about options you can pass the test runner, see the --help option:
nox -e 'pytest-zeromq-3(coverage=False)' -- --help
Running Test Subsections Instead of running the entire test suite all at once, which can take a long time, there are several ways to run only specific groups of tests or individual tests:
• Run unit tests only: nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/, using the deprecated runtests.py the command would be nox -e 'runtests-zeromq-3(coverage=False)' -- --unit-tests
• Run unit and integration tests for states: nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/states/ tests/integration/states/, using the deprecated runtests.py the command would be nox -e 'runtests-zeromq-3(coverage=False)' -- --state-tests
• Run integration tests for an individual module: nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_virt.py, using the deprecated runtests.py the command would be nox -e 'runtests-zeromq-3(coverage=False)' -- -n integration.modules.test_virt
• Run unit tests for an individual module: nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/modules/test_virt.py, using the deprecated runtests.py the command would be nox -e 'runtests-zeromq-3(coverage=False)' -- -n unit.modules.test_virt
• Run an individual test by using the class and test name (this example is for the test_default_kvm_profile test in the tests/integration/module/test_virt.py): nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_virt.py::VirtTest::test_default_kvm_profile, using the deprecated runtests.py the command would be nox -e 'runtests-zeromq-3(coverage=False)' -- -n integration.modules.test_virt.VirtTest.test_default_kvm_profile
For more specific examples of how to run various test subsections or individual tests, please see the pytest documentation on how to run specific tests or the Running Specific Tests section of the Salt's Test Suite: An Introduction tutorial.
Running Unit Tests Without Integration Test Daemons Since the unit tests do not require a master or minion to execute, it is often useful to be able to run unit tests individually, or as a whole group, without having to start up the integration testing daemons. Starting up the master, minion, and syndic daemons takes a lot of time before the tests can even start running and is unnecessary to run unit tests. To run unit tests without invoking the integration test daemons, simply add the unit directory as an argument:
nox -e 'runtests-zeromq-3(coverage=False)' -- --unit-tests
All of the other options to run individual tests, entire classes of tests, or entire test modules still apply.
Running Destructive Integration Tests Salt is used to change the settings and behavior of systems. In order to effectively test Salt's functionality, some integration tests are written to make actual changes to the underlying system. These tests are referred to as "destructive tests". Some examples of destructive tests are changes may be testing the addition of a user or installing packages. By default, destructive tests are disabled and will be skipped.
Generally, destructive tests should clean up after themselves by attempting to restore the system to its original state. For instance, if a new user is created during a test, the user should be deleted after the related test(s) have completed. However, no guarantees are made that test clean-up will complete successfully. Therefore, running destructive tests should be done with caution.
NOTE: Running destructive tests will change the underlying system.  Use caution when running destructive tests.
To run tests marked as destructive, set the --run-destructive flag:
nox -e 'pytest-zeromq-3(coverage=False)' -- --run-destructive
nox -e 'runtests-zeromq-3(coverage=False)' -- --run-destructive
Running Cloud Provider Tests Salt's testing suite also includes integration tests to assess the successful creation and deletion of cloud instances using Salt-Cloud for providers supported by Salt-Cloud.
The cloud provider tests are off by default and run on sample configuration files provided in tests/integration/files/conf/cloud.providers.d/. In order to run the cloud provider tests, valid credentials, which differ per provider, must be supplied. Each credential item that must be supplied is indicated by an empty string value and should be edited by the user before running the tests. For example, DigitalOcean requires a client key and an api key to operate. Therefore, the default cloud provider configuration file for DigitalOcean looks like this:
digitalocean-config: driver: digitalocean client_key: '' api_key: '' location: New York 1
As indicated by the empty string values, the client_key and the api_key must be provided:
digitalocean-config: driver: digitalocean client_key: wFGEwgregeqw3435gDger api_key: GDE43t43REGTrkilg43934t34qT43t4dgegerGEgg location: New York 1
NOTE: When providing credential information in cloud provider configuration files, do not include the single quotes.
Once all of the valid credentials for the cloud provider have been supplied, the cloud provider tests can be run by setting the --cloud-provider-tests flag:
nox -e 'pytest-cloud-3(coverage=False)'
nox -e 'runtests-cloud-3(coverage=False)'
Automated Test Runs SaltStack maintains a Jenkins server to allow for the execution of tests across supported platforms. The tests executed from Salt's Jenkins server create fresh virtual machines for each test run, then execute destructive tests on the new, clean virtual machine.
SaltStack's Jenkins server continuously runs the entire test suite, including destructive tests, on an array of various supported operating systems throughout the day. Each actively supported branch of Salt's repository runs the tests located in the respective branch's code. Each set of branch tests also includes a pylint run. These branch tests help ensure the viability of Salt code at any given point in time as pull requests are merged into branches throughout the day.
In addition to branch tests, SaltStack's Jenkins server also runs tests on pull requests. These pull request tests include a smaller set of virtual machines that run on the branch tests. The pull request tests, like the branch tests, include a pylint test as well.
When a pull request is submitted to Salt's repository on GitHub, the suite of pull request tests are started by Jenkins. These tests are used to gauge the pull request's viability to merge into Salt's codebase. If these initial tests pass, the pull request can then merged into the Salt branch by one of Salt's core developers, pending their discretion. If the initial tests fail, core developers may request changes to the pull request. If the failure is unrelated to the changes in question, core developers may merge the pull request despite the initial failure.
As soon as the pull request is merged, the changes will be added to the next branch test run on Jenkins.
For a full list of currently running test environments, go to https://jenkins.saltproject.io.
Using Salt-Cloud on Jenkins For testing Salt on Jenkins, SaltStack uses Salt-Cloud to spin up virtual machines. The script using Salt-Cloud to accomplish this is open source and can be found here: https://github.com/saltstack/salt/blob/master/tests/jenkins.py
Writing Tests The salt testing infrastructure is divided into two classes of tests, integration tests and unit tests. These terms may be defined differently in other contexts, but for Salt they are defined this way:
• Unit Test: Tests which validate isolated code blocks and do not require external interfaces such as salt-call or any of the salt daemons.
• Integration Test: Tests which validate externally accessible features.
Salt testing uses unittest2 from the python standard library and MagicMock.
• Writing integration tests
• Writing unit tests
Naming Conventions Any function in either integration test files or unit test files that is doing the actual testing, such as functions containing assertions, must start with test_:
def test_user_present(self): ...
When functions in test files are not prepended with test_, the function acts as a normal, helper function and is not run as a test by the test suite.
Submitting New Tests Which branch of the Salt codebase should new tests be written against? The location of where new tests should be submitted depends largely on the reason you're writing the tests.
Tests for New Features If you are adding new functionality to Salt, please write the tests for this new feature in the same pull request as the new feature. New features should always be submitted to the master branch.
If you have already submitted the new feature, but did not write tests in the original pull request that has already been merged, please feel free to submit a new pull request containing tests. If the feature was recently added to Salt's master branch, then the tests should be added there as well. However, if the feature was added to master some time ago and is already present in one or more release branches, please refer to the Tests for Entire Files or Functions section below for more details about where to submit tests for functions or files that do not already have tests.
Tests to Accompany a Bugfix If you are writing tests for code that fixes a bug in Salt, tests will be required before merging the PR. A great option for most bugfixes is to adopt a TDD style approach:
• reproduce the issue
• write a test that exhibits the behavior
• write the bugfix
This helps ensure that known issues are not reintroduced into the codebase.
Tests for Entire Files or Functions Sometimes entire files in Salt are completely untested. If you are writing tests for a file that doesn't have any tests written for it, write your test against the earliest supported release branch that contains the file or function you're testing.
Once your tests are submitted in a pull request and is merged into the branch in question, the tests you wrote will be merged-forward by SaltStack core engineers and the new tests will propagate to the newer release branches. That way the tests you wrote will apply to all current and relevant release branches, and not just the master branch, for example. This methodology will help protect against regressions on older files in Salt's codebase.
There may be times when the tests you write against an older branch fail in the merge-forward process because functionality has changed in newer release branches.  In these cases, a Salt core developer may reach out to you for advice on the tests in question if the path forward is unclear.
NOTE: If tests are written against a file in an older release branch and then merged forward, there may be new functionality in the file that is present in the new release branch that is untested.It would be wise to see if new functionality could use additional testing once the test file has propagated to newer release branches.
Module/Global Level Variables If you need access to module or global level variables, please use a pytest fixture. The use of module and global variables can introduce mutable global objects and increases processing time because all globals are evaluated when collecting tests. If there is a use case where you cannot use a fixture and you are using a type of string, integer, or tuple you can use global/module level variables. Any mutable types such as lists and dictionaries must use pytest fixtures. For an example, if all of your tests need access to a string variable:
FOO = "bar"
def test_foo_bar(): assert FOO == "bar"
def test_foo_not(): assert not FOO == "foo"
We recommend using a pytest fixture:
import pytest
@pytest.fixture() def foo(): return "bar"
def test_foo_bar(foo): assert foo == "bar"
def test_foo_not(foo): assert not foo == "foo"
If you need a class to mock something, it can be defined at the global scope, but it should only be initialized on the fixture:
class Foo: def __init__(self): self.bar = True
@pytest.fixture def foo(): return Foo()
Test Helpers Several Salt-specific helpers are available. A full list is available by inspecting functions exported in tests.support.helpers.
@expensiveTest -- Designates a test which typically requires a relatively costly external resource, like a cloud virtual machine. This decorator is not normally used by developers outside of the Salt core team.
@destructiveTest -- Marks a test as potentially destructive. It will not be run by the test runner unless the -run-destructive test is expressly passed.
@requires_network -- Requires a network connection for the test to operate successfully. If a network connection is not detected, the test will not run.
@requires_salt_modules -- Requires all the modules in a list of modules in order for the test to be executed. Otherwise, the test is skipped.
@requires_system_grains -- Loads and passes the grains on the system as an keyword argument to the test function with the name grains.
@skip_if_binaries_missing(['list', 'of', 'binaries']) -- If called from inside a test, the test will be skipped if the binaries are not all present on the system.
@skip_if_not_root -- If the test is not executed as root, it will be skipped.
@with_system_user -- Creates and optionally destroys a system user within a test case.  See implementation details in tests.support.helpers for details.
@with_system_group -- Creates and optionally destroys a system group within a test case.  See implementation details in tests.support.helpers for details.
@with_system_user_and_group -- Creates and optionally destroys a system user and group within a test case.  See implementation details in tests.support.helpers for details.
Integration Tests The Salt integration tests come with a number of classes and methods which allow for components to be easily tested. These classes are generally inherited from and provide specific methods for hooking into the running integration test environment created by the integration tests.
It is noteworthy that since integration tests validate against a running environment that they are generally the preferred means to write tests.
The integration system is all located under tests/integration in the Salt source tree. Each directory within tests/integration corresponds to a directory in Salt's tree structure. For example, the integration tests for the test.py Salt module that is located in salt/modules should also be named test.py and reside in tests/integration/modules.
Preparing to Write Integration Tests This guide assumes that your Salt development environment is already configured and that you have a basic understanding of contributing to the Salt codebase.  If you're unfamiliar with either of these topics, please refer to the Installing Salt for Development and the Contributing pages, respectively.
This documentation also assumes that you have an understanding of how to run Salt's test suite, including running the test subsections, and running a single integration test file, class, or individual test.
Best Practices Integration tests should be written to the following specifications.
What to Test? Since integration tests are used to validate against a running Salt environment, integration tests should be written with the Salt components, and their various interactions, in mind.
• Isolate testing functionality. Don't rely on the pass or failure of other, separate tests.
• Individual tests should test against a single behavior.