neulab/tldr huggingface dataset:
# There is a shorter form to set the above variables set_variables: module.run: - name: syslog_ng.set_parameters - version: "3.6" - binary_path: "/home/tibi/install/syslog-ng/sbin" - config_file: "/home/tibi/install/syslog-ng/etc/syslog-ng.conf"
# Some global options options.global_options: syslog_ng.config: - config: - time_reap: 30 - mark_freq: 10 - keep_hostname: "yes"
destination.d_log_server: syslog_ng.config: - config: - tcp: - "127.0.0.1" - port: 1234
log.l_log_to_central_server: syslog_ng.config: - config: - source: s_localhost - destination: d_log_server
some_comment: module.run: - name: syslog_ng.write_config - config: | # Multi line # comment
# Another mode to use comments or existing configuration snippets config.other_comment_form: syslog_ng.config: - config: | # Multi line # comment
The syslog_ng.reloaded function can generate syslog-ng configuration from YAML. If the statement (source, destination, parser, etc.) has a name, this function uses the id as the name, otherwise (log statement) its purpose is like a mandatory comment.
After execution this example the syslog_ng state will generate this file:
#Generated by Salt on 2014-08-18 00:11:11 @version: 3.6
options { time_reap( 30 ); mark_freq( 10 ); keep_hostname( yes ); };
source s_localhost { tcp( ip( 127.0.0.1 ), port( 1233 ) ); };
destination d_log_server { tcp( 127.0.0.1, port( 1234 ) ); };
log { source( s_localhost ); destination( d_log_server ); };
# Multi line # comment
Users can include arbitrary texts in the generated configuration with using the config statement (see the example above).
Syslog_ng module functions You can use syslog_ng.set_binary_path to set the directory which contains the syslog-ng and syslog-ng-ctl binaries. If this directory is in your PATH, you don't need to use this function. There is also a syslog_ng.set_config_file function to set the location of the configuration file.
Examples Simple source source s_tail { file( "/var/log/apache/access.log", follow_freq(1), flags(no-parse, validate-utf8) ); };
s_tail: # Salt will call the source function of syslog_ng module syslog_ng.config: - config: source: - file: - file: ''"/var/log/apache/access.log"'' - follow_freq : 1 - flags: - no-parse - validate-utf8
s_tail: syslog_ng.config: - config: source: - file: - ''"/var/log/apache/access.log"'' - follow_freq : 1 - flags: - no-parse - validate-utf8
source.s_tail: syslog_ng.config: - config: - file: - ''"/var/log/apache/access.log"'' - follow_freq : 1 - flags: - no-parse - validate-utf8
Complex source source s_gsoc2014 { tcp( ip("0.0.0.0"), port(1234), flags(no-parse) ); };
s_gsoc2014: syslog_ng.config: - config: source: - tcp: - ip: 0.0.0.0 - port: 1234 - flags: no-parse
Filter filter f_json { match( "@json:" ); };
f_json: syslog_ng.config: - config: filter: - match: - ''"@json:"''
Template template t_demo_filetemplate { template( "$ISODATE $HOST $MSG " ); template_escape( no ); };
t_demo_filetemplate: syslog_ng.config: -config: template: - template: - '"$ISODATE $HOST $MSG\n"' - template_escape: - "no"
Rewrite rewrite r_set_message_to_MESSAGE { set( "${.json.message}", value("$MESSAGE") ); };
r_set_message_to_MESSAGE: syslog_ng.config: - config: rewrite: - set: - '"${.json.message}"' - value : '"$MESSAGE"'
Global options options { time_reap(30); mark_freq(10); keep_hostname(yes); };
global_options: syslog_ng.config: - config: options: - time_reap: 30 - mark_freq: 10 - keep_hostname: "yes"
Log log { source(s_gsoc2014); junction { channel { filter(f_json); parser(p_json); rewrite(r_set_json_tag); rewrite(r_set_message_to_MESSAGE); destination { file( "/tmp/json-input.log", template(t_gsoc2014) ); }; flags(final); }; channel { filter(f_not_json); parser { syslog-parser(
); }; rewrite(r_set_syslog_tag); flags(final); }; }; destination { file( "/tmp/all.log", template(t_gsoc2014) ); }; };
l_gsoc2014: syslog_ng.config: - config: log: - source: s_gsoc2014 - junction: - channel: - filter: f_json - parser: p_json - rewrite: r_set_json_tag - rewrite: r_set_message_to_MESSAGE - destination: - file: - '"/tmp/json-input.log"' - template: t_gsoc2014 - flags: final - channel: - filter: f_not_json - parser: - syslog-parser: [] - rewrite: r_set_syslog_tag - flags: final - destination: - file: - "/tmp/all.log" - template: t_gsoc2014
Salt in 10 Minutes NOTE: Welcome to SaltStack! I am excited that you are interested in Salt and starting down the path to better infrastructure management. I developed (and am continuing to develop) Salt with the goal of making the best software available to manage computers of almost any kind. I hope you enjoy working with Salt and that the software can solve your real world needs!
• Thomas S Hatch
• Salt creator and Chief Developer
• CTO of SaltStack, Inc.
Getting Started What is Salt? Salt is a different approach to infrastructure management, founded on the idea that high-speed communication with large numbers of systems can open up new capabilities. This approach makes Salt a powerful multitasking system that can solve many specific problems in an infrastructure.
The backbone of Salt is the remote execution engine, which creates a high-speed, secure and bi-directional communication net for groups of systems. On top of this communication system, Salt provides an extremely fast, flexible, and easy-to-use configuration management system called Salt States.
Installing Salt SaltStack has been made to be very easy to install and get started. The installation documents contain instructions for all supported platforms.
Starting Salt Salt functions on a master/minion topology. A master server acts as a central control bus for the clients, which are called minions. The minions connect back to the master.
Setting Up the Salt Master Turning on the Salt Master is easy -- just turn it on! The default configuration is suitable for the vast majority of installations. The Salt Master can be controlled by the local Linux/Unix service manager:
On Systemd based platforms (newer Debian, openSUSE, Fedora):
On Upstart based systems (Ubuntu, older Fedora/RHEL):
service salt-master start
On SysV Init systems (Gentoo, older Debian etc.):
/etc/init.d/salt-master start
Alternatively, the Master can be started directly on the command-line:
salt-master -d
The Salt Master can also be started in the foreground in debug mode, thus greatly increasing the command output:
salt-master -l debug
The Salt Master needs to bind to two TCP network ports on the system. These ports are 4505 and 4506. For more in depth information on firewalling these ports, the firewall tutorial is available here.
Finding the Salt Master When a minion starts, by default it searches for a system that resolves to the salt hostname on the network.  If found, the minion initiates the handshake and key authentication process with the Salt master.  This means that the easiest configuration approach is to set internal DNS to resolve the name salt back to the Salt Master IP.
Otherwise, the minion configuration file will need to be edited so that the configuration option master points to the DNS name or the IP of the Salt Master:
NOTE: The default location of the configuration files is /etc/salt. Most platforms adhere to this convention, but platforms such as FreeBSD and Microsoft Windows place this file in different locations.
/etc/salt/minion:
master: saltmaster.example.com
Setting up a Salt Minion NOTE: The Salt Minion can operate with or without a Salt Master. This walk-through assumes that the minion will be connected to the master, for information on how to run a master-less minion please see the master-less quick-start guide:
Masterless Minion Quickstart
Now that the master can be found, start the minion in the same way as the master; with the platform init system or via the command line directly:
As a daemon:
salt-minion -d
In the foreground in debug mode:
When the minion is started, it will generate an id value, unless it has been generated on a previous run and cached (in /etc/salt/minion_id by default). This is the name by which the minion will attempt to authenticate to the master. The following steps are attempted, in order to try to find a value that is not localhost:
1. The Python function socket.getfqdn() is run
2. /etc/hostname is checked (non-Windows only)
3. /etc/hosts (%WINDIR%\system32\drivers\etc\hosts on Windows hosts) is checked for hostnames that map to anything within 127.0.0.0/8.
If none of the above are able to produce an id which is not localhost, then a sorted list of IP addresses on the minion (excluding any within 127.0.0.0/8) is inspected. The first publicly-routable IP address is used, if there is one. Otherwise, the first privately-routable IP address is used.
If all else fails, then localhost is used as a fallback.
NOTE: Overriding the id
The minion id can be manually specified using the id parameter in the minion config file.  If this configuration value is specified, it will override all other sources for the id.
Now that the minion is started, it will generate cryptographic keys and attempt to connect to the master. The next step is to venture back to the master server and accept the new minion's public key.
Using salt-key Salt authenticates minions using public-key encryption and authentication. For a minion to start accepting commands from the master, the minion keys need to be accepted by the master.
The salt-key command is used to manage all of the keys on the master. To list the keys that are on the master:
salt-key -L
The keys that have been rejected, accepted, and pending acceptance are listed.  The easiest way to accept the minion key is to accept all pending keys:
salt-key -A
NOTE: Keys should be verified! Print the master key fingerprint by running salt-key -F master on the Salt master. Copy the master.pub fingerprint from the Local Keys section, and then set this value as the master_finger in the minion configuration file. Restart the Salt minion.
On the master, run salt-key -f minion-id to print the fingerprint of the minion's public key that was received by the master. On the minion, run salt-call key.finger --local to print the fingerprint of the minion key.
On the master:
# salt-key -f foo.domain.com Unaccepted Keys: foo.domain.com:  39:f9:e4:8a:aa:74:8d:52:1a:ec:92:03:82:09:c8:f9
On the minion:
# salt-call key.finger --local local: 39:f9:e4:8a:aa:74:8d:52:1a:ec:92:03:82:09:c8:f9
If they match, approve the key with salt-key -a foo.domain.com.
Sending the First Commands Now that the minion is connected to the master and authenticated, the master can start to command the minion.
Salt commands allow for a vast set of functions to be executed and for specific minions and groups of minions to be targeted for execution.
The salt command is comprised of command options, target specification, the function to execute, and arguments to the function.
A simple command to start with looks like this:
The * is the target, which specifies all minions.
test.version tells the minion to run the test.version function.
In the case of test.version, test refers to a execution module.  version refers to the version function contained in the aforementioned test module.
NOTE: Execution modules are the workhorses of Salt. They do the work on the system to perform various tasks, such as manipulating files and restarting services.
The result of running this command will be the master instructing all of the minions to execute test.version in parallel and return the result. Using test.version is a good way of confirming that a minion is connected, and reaffirm to the user the salt version(s) they have installed on the minions.
NOTE: Each minion registers itself with a unique minion ID. This ID defaults to the minion's hostname, but can be explicitly defined in the minion config as well by using the id parameter.
Of course, there are hundreds of other modules that can be called just as test.version can.  For example, the following would return disk usage on all targeted minions:
salt '*' disk.usage
Getting to Know the Functions Salt comes with a vast library of functions available for execution, and Salt functions are self-documenting. To see what functions are available on the minions execute the sys.doc function:
This will display a very large list of available functions and documentation on them.
NOTE: Module documentation is also available on the web.
These functions cover everything from shelling out to package management to manipulating database servers. They comprise a powerful system management API which is the backbone to Salt configuration management and many other aspects of Salt.
NOTE: Salt comes with many plugin systems. The functions that are available via the salt command are called Execution Modules.
Helpful Functions to Know The cmd module contains functions to shell out on minions, such as cmd.run and cmd.run_all:
salt '*' cmd.run 'ls -l /etc'
The pkg functions automatically map local system package managers to the same salt functions. This means that pkg.install will install packages via yum on Red Hat based systems, apt on Debian systems, etc.:
salt '*' pkg.install vim
NOTE: Some custom Linux spins and derivatives of other distributions are not properly detected by Salt. If the above command returns an error message saying that pkg.install is not available, then you may need to override the pkg provider. This process is explained here.
The network.interfaces function will list all interfaces on a minion, along with their IP addresses, netmasks, MAC addresses, etc:
salt '*' network.interfaces
Changing the Output Format The default output format used for most Salt commands is called the nested outputter, but there are several other outputters that can be used to change the way the output is displayed. For instance, the pprint outputter can be used to display the return data using Python's pprint module:
root@saltmaster:~# salt myminion grains.item pythonpath --out=pprint {'myminion': {'pythonpath': ['/usr/lib64/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages/gst-0.10', '/usr/lib/python2.7/site-packages/gtk-2.0']}}
The full list of Salt outputters, as well as example output, can be found here.
salt-call The examples so far have described running commands from the Master using the salt command, but when troubleshooting it can be more beneficial to login to the minion directly and use salt-call.
Doing so allows you to see the minion log messages specific to the command you are running (which are not part of the return data you see when running the command from the Master using salt), making it unnecessary to tail the minion log. More information on salt-call and how to use it can be found here.
Grains Salt uses a system called Grains to build up static data about minions. This data includes information about the operating system that is running, CPU architecture and much more. The grains system is used throughout Salt to deliver platform data to many components and to users.
Grains can also be statically set, this makes it easy to assign values to minions for grouping and managing.
A common practice is to assign grains to minions to specify what the role or roles a minion might be. These static grains can be set in the minion configuration file or via the grains.setval function.
Targeting Salt allows for minions to be targeted based on a wide range of criteria. The default targeting system uses globular expressions to match minions, hence if there are minions named larry1, larry2, curly1, and curly2, a glob of larry* will match larry1 and larry2, and a glob of *1 will match larry1 and curly1.
Many other targeting systems can be used other than globs, these systems include:
Regular Expressions Target using PCRE-compliant regular expressions
Grains Target based on grains data: Targeting with Grains
Pillar Target based on pillar data: Targeting with Pillar
IP     Target based on IP address/subnet/range
Compound Create logic to target based on multiple targets: Targeting with Compound
Nodegroup Target with nodegroups: Targeting with Nodegroup
The concepts of targets are used on the command line with Salt, but also function in many other areas as well, including the state system and the systems used for ACLs and user permissions.
Passing in Arguments Many of the functions available accept arguments which can be passed in on the command line:
This example passes the argument vim to the pkg.install function. Since many functions can accept more complex input than just a string, the arguments are parsed through YAML, allowing for more complex data to be sent on the command line:
salt '*' test.echo 'foo: bar'
In this case Salt translates the string 'foo: bar' into the dictionary "{'foo': 'bar'}"
NOTE: Any line that contains a newline will not be parsed by YAML.
Salt States Now that the basics are covered the time has come to evaluate States. Salt States, or the State System is the component of Salt made for configuration management.
The state system is already available with a basic Salt setup, no additional configuration is required. States can be set up immediately.
NOTE: Before diving into the state system, a brief overview of how states are constructed will make many of the concepts clearer. Salt states are based on data modeling and build on a low level data structure that is used to execute each state function. Then more logical layers are built on top of each other.
The high layers of the state system which this tutorial will cover consists of everything that needs to be known to use states, the two high layers covered here are the sls layer and the highest layer highstate.
Understanding the layers of data management in the State System will help with understanding states, but they never need to be used. Just as understanding how a compiler functions assists when learning a programming language, understanding what is going on under the hood of a configuration management system will also prove to be a valuable asset.
The First SLS Formula The state system is built on SLS (SaLt State) formulas. These formulas are built out in files on Salt's file server. To make a very basic SLS formula open up a file under /srv/salt named vim.sls. The following state ensures that vim is installed on a system to which that state has been applied.
/srv/salt/vim.sls:
Now install vim on the minions by calling the SLS directly:
salt '*' state.apply vim
This command will invoke the state system and run the vim SLS.
Now, to beef up the vim SLS formula, a vimrc can be added:
/etc/vimrc: file.managed: - source: salt://vimrc - mode: 644 - user: root - group: root
Now the desired vimrc needs to be copied into the Salt file server to /srv/salt/vimrc. In Salt, everything is a file, so no path redirection needs to be accounted for. The vimrc file is placed right next to the vim.sls file.  The same command as above can be executed to all the vim SLS formulas and now include managing the file.
NOTE: Salt does not need to be restarted/reloaded or have the master manipulated in any way when changing SLS formulas. They are instantly available.
Adding Some Depth Obviously maintaining SLS formulas right in a single directory at the root of the file server will not scale out to reasonably sized deployments. This is why more depth is required. Start by making an nginx formula a better way, make an nginx subdirectory and add an init.sls file:
/srv/salt/nginx/init.sls:
nginx: pkg.installed: [] service.running: - require: - pkg: nginx
A few concepts are introduced in this SLS formula.
First is the service statement which ensures that the nginx service is running.
Of course, the nginx service can't be started unless the package is installed -- hence the require statement which sets up a dependency between the two.
The require statement makes sure that the required component is executed before and that it results in success.
NOTE: The require option belongs to a family of options called requisites. Requisites are a powerful component of Salt States, for more information on how requisites work and what is available see: Requisites
Also evaluation ordering is available in Salt as well: Ordering States
This new sls formula has a special name --  init.sls. When an SLS formula is named init.sls it inherits the name of the directory path that contains it.  This formula can be referenced via the following command:
salt '*' state.apply nginx
NOTE: state.apply is just another remote execution function, just like test.version or disk.usage. It simply takes the name of an SLS file as an argument.
Now that subdirectories can be used, the vim.sls formula can be cleaned up.  To make things more flexible, move the vim.sls and vimrc into a new subdirectory called edit and change the vim.sls file to reflect the change:
/etc/vimrc: file.managed: - source: salt://edit/vimrc - mode: 644 - user: root - group: root
Only the source path to the vimrc file has changed. Now the formula is referenced as edit.vim because it resides in the edit subdirectory.  Now the edit subdirectory can contain formulas for emacs, nano, joe or any other editor that may need to be deployed.
Next Reading Two walk-throughs are specifically recommended at this point. First, a deeper run through States, followed by an explanation of Pillar.
1. Starting States
2. Pillar Walkthrough
An understanding of Pillar is extremely helpful in using States.
Getting Deeper Into States Two more in-depth States tutorials exist, which delve much more deeply into States functionality.
1. How Do I Use Salt States?, covers much more to get off the ground with States.
2. The States Tutorial also provides a fantastic introduction.
These tutorials include much more in-depth information including templating SLS formulas etc.
So Much More! This concludes the initial Salt walk-through, but there are many more things still to learn! These documents will cover important core aspects of Salt:
• Pillar
• Job Management
A few more tutorials are also available:
• Remote Execution Tutorial
• Standalone Minion
This still is only scratching the surface, many components such as the reactor and event systems, extending Salt, modular components and more are not covered here. For an overview of all Salt features and documentation, look at the Table of Contents.
Salt's Test Suite: An Introduction NOTE: This tutorial makes a couple of assumptions. The first assumption is that you have a basic knowledge of Salt. To get up to speed, check out the Salt Walkthrough.
The second assumption is that your Salt development environment is already configured and that you have a basic understanding of contributing to the Salt codebase. If you're unfamiliar with either of these topics, please refer to the Installing Salt for Development and the Contributing pages, respectively.
Salt comes with a powerful integration and unit test suite. The test suite allows for the fully automated run of integration and/or unit tests from a single interface.
Salt's test suite is located under the tests directory in the root of Salt's code base and is divided into two main types of tests: unit tests and integration tests. The unit and integration sub-test-suites are located in the tests directory, which is where the majority of Salt's test cases are housed.
Getting Set Up For Tests First of all you will need to ensure you install nox.
pip install nox
Test Directory Structure As noted in the introduction to this tutorial, Salt's test suite is located in the tests directory in the root of Salt's code base. From there, the tests are divided into two groups integration and unit. Within each of these directories, the directory structure roughly mirrors the directory structure of Salt's own codebase.  For example, the files inside tests/integration/modules contains tests for the files located within salt/modules.
NOTE: tests/integration and tests/unit are the only directories discussed in this tutorial. With the exception of the tests/runtests.py file, which is used below in the Running the Test Suite section, the other directories and files located in tests are outside the scope of this tutorial.
Integration vs. Unit Given that Salt's test suite contains two powerful, though very different, testing approaches, when should you write integration tests and when should you write unit tests?
Integration tests use Salt masters, minions, and a syndic to test salt functionality directly and focus on testing the interaction of these components. Salt's integration test runner includes functionality to run Salt execution modules, runners, states, shell commands, salt-ssh commands, salt-api commands, and more. This provides a tremendous ability to use Salt to test itself and makes writing such tests a breeze. Integration tests are the preferred method of testing Salt functionality when possible.
Unit tests do not spin up any Salt daemons, but instead find their value in testing singular implementations of individual functions. Instead of testing against specific interactions, unit tests should be used to test a function's logic. Unit tests should be used to test a function's exit point(s) such as any return or raises statements.
Unit tests are also useful in cases where writing an integration test might not be possible. While the integration test suite is extremely powerful, unfortunately at this time, it does not cover all functional areas of Salt's ecosystem. For example, at the time of this writing, there is not a way to write integration tests for Proxy Minions. Since the test runner will need to be adjusted to account for Proxy Minion processes, unit tests can still provide some testing support in the interim by testing the logic contained inside Proxy Minion functions.
Running the Test Suite Once all of the requirements are installed, the nox command is used to instantiate Salt's test suite:
nox -e 'pytest-zeromq-3(coverage=False)'
The command above, if executed without any options, will run the entire suite of integration and unit tests. Some tests require certain flags to run, such as destructive tests. If these flags are not included, then the test suite will only perform the tests that don't require special attention.
At the end of the test run, you will see a summary output of the tests that passed, failed, or were skipped.
You can pass any pytest options after the nox command like so:
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/modules/test_ps.py
The above command will run the test_ps.py test with the zeromq transport, python3, and pytest. The option after -- can include any pytest options.
Running Integration Tests Salt's set of integration tests use Salt to test itself. The integration portion of the test suite includes some built-in Salt daemons that will spin up in preparation of the test run. This list of Salt daemon processes includes:
• 2 Salt Masters
• 2 Salt Minions
• 1 Salt Syndic
These various daemons are used to execute Salt commands and functionality within the test suite, allowing you to write tests to assert against expected or unexpected behaviors.
A simple example of a test utilizing a typical master/minion execution module command is the test for the test_ping function in the tests/integration/modules/test_test.py file:
def test_ping(self): """ test.ping """ self.assertTrue(self.run_function("test.ping"))
The test above is a very simple example where the test.ping function is executed by Salt's test suite runner and is asserting that the minion returned with a True response.
Test Selection Options If you want to run only a subset of tests, this is easily done with pytest. You only need to point the test runner to the directory. For example if you want to run all integration module tests:
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/
Running Unit Tests If you want to run only the unit tests, you can just pass the unit test directory as an option to the test runner.
The unit tests do not spin up any Salt testing daemons as the integration tests do and execute very quickly compared to the integration tests.
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/
Running Specific Tests There are times when a specific test file, test class, or even a single, individual test need to be executed, such as when writing new tests. In these situations, you should use the pytest syntax to select the specific tests.
For running a single test file, such as the pillar module test file in the integration test directory, you must provide the file path.
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_pillar.py
Some test files contain only one test class while other test files contain multiple test classes. To run a specific test class within the file, append the name of the test class to the end of the file path:
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_pillar.py::PillarModuleTest
To run a single test within a file, append both the name of the test class the individual test belongs to, as well as the name of the test itself:
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_pillar.py::PillarModuleTest::test_data
The following command is an example of how to execute a single test found in the tests/unit/modules/test_cp.py file:
nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/modules/test_cp.py::CpTestCase::test_get_file_not_found
Writing Tests for Salt Once you're comfortable running tests, you can now start writing them! Be sure to review the Integration vs. Unit section of this tutorial to determine what type of test makes the most sense for the code you're testing.
NOTE: There are many decorators, naming conventions, and code specifications required for Salt test files. We will not be covering all of the these specifics in this tutorial. Please refer to the testing documentation links listed below in the Additional Testing Documentation section to learn more about these requirements.
In the following sections, the test examples assume the "new" test is added to a test file that is already present and regularly running in the test suite and is written with the correct requirements.
Writing Integration Tests Since integration tests validate against a running environment, as explained in the Running Integration Tests section of this tutorial, integration tests are very easy to write and are generally the preferred method of writing Salt tests.
The following integration test is an example taken from the test.py file in the tests/integration/modules directory. This test uses the run_function method to test the functionality of a traditional execution module command.
The run_function method uses the integration test daemons to execute a module.function command as you would with Salt. The minion runs the function and returns. The test also uses Python's Assert Functions to test that the minion's return is expected.
Args can be passed in to the run_function method as well:
def test_echo(self): """ test.echo """ self.assertEqual(self.run_function("test.echo", ["text"]), "text")
The next example is taken from the tests/integration/modules/test_aliases.py file and demonstrates how to pass kwargs to the run_function call. Also note that this test uses another salt function to ensure the correct data is present (via the aliases.set_target call) before attempting to assert what the aliases.get_target call should return.
def test_set_target(self): """ aliases.set_target and aliases.get_target """ set_ret = self.run_function("aliases.set_target", alias="fred", target="bob") self.assertTrue(set_ret) tgt_ret = self.run_function("aliases.get_target", alias="fred") self.assertEqual(tgt_ret, "bob")
Using multiple Salt commands in this manner provides two useful benefits. The first is that it provides some additional coverage for the aliases.set_target function.  The second benefit is the call to aliases.get_target is not dependent on the presence of any aliases set outside of this test. Tests should not be dependent on the previous execution, success, or failure of other tests. They should be isolated from other tests as much as possible.
While it might be tempting to build out a test file where tests depend on one another before running, this should be avoided. SaltStack recommends that each test should test a single functionality and not rely on other tests. Therefore, when possible, individual tests should also be broken up into singular pieces. These are not hard-and-fast rules, but serve more as recommendations to keep the test suite simple.  This helps with debugging code and related tests when failures occur and problems are exposed. There may be instances where large tests use many asserts to set up a use case that protects against potential regressions.
NOTE: The examples above all use the run_function option to test execution module functions in a traditional master/minion environment. To see examples of how to test other common Salt components such as runners, salt-api, and more, please refer to the Integration Test Class Examples documentation.
Destructive vs Non-destructive Tests Since Salt is used to change the settings and behavior of systems, often, the best approach to run tests is to make actual changes to an underlying system.  This is where the concept of destructive integration tests comes into play.  Tests can be written to alter the system they are running on. This capability is what fills in the gap needed to properly test aspects of system management like package installation.
To write a destructive test, import and use the destructiveTest decorator for the test method:
import integration from tests.support.helpers import destructiveTest
class PkgTest(integration.ModuleCase): @destructiveTest def test_pkg_install(self): ret = self.run_function("pkg.install", name="finch") self.assertSaltTrueReturn(ret) ret = self.run_function("pkg.purge", name="finch") self.assertSaltTrueReturn(ret)
Writing Unit Tests As explained in the Integration vs. Unit section above, unit tests should be written to test the logic of a function. This includes focusing on testing return and raises statements. Substantial effort should be made to mock external resources that are used in the code being tested.
External resources that should be mocked include, but are not limited to, APIs, function calls, external data either globally available or passed in through function arguments, file data, etc. This practice helps to isolate unit tests to test Salt logic. One handy way to think about writing unit tests is to "block all of the exits". More information about how to properly mock external resources can be found in Salt's Unit Test documentation.
Salt's unit tests utilize Python's mock class as well as MagicMock. The @patch decorator is also heavily used when "blocking all the exits".
A simple example of a unit test currently in use in Salt is the test_get_file_not_found test in the tests/unit/modules/test_cp.py file. This test uses the @patch decorator and MagicMock to mock the return of the call to Salt's cp.hash_file execution module function. This ensures that we're testing the cp.get_file function directly, instead of inadvertently testing the call to cp.hash_file, which is used in cp.get_file.
def test_get_file_not_found(self): """ Test if get_file can't find the file. """ with patch("salt.modules.cp.hash_file", MagicMock(return_value=False)): path = "salt://saltines" dest = "/srv/salt/cheese" ret = "" assert cp.get_file(path, dest) == ret
Note that Salt's cp module is imported at the top of the file, along with all of the other necessary testing imports. The get_file function is then called directed in the testing function, instead of using the run_function method as the integration test examples do above.
The call to cp.get_file returns an empty string when a hash_file isn't found.  Therefore, the example above is a good illustration of a unit test "blocking the exits" via the @patch decorator, as well as testing logic via asserting against the return statement in the if clause. In this example we used the python assert to verify the return from cp.get_file. Pytest allows you to use these asserts when writing your tests and, in fact, plain asserts is the preferred way to assert anything in your tests. As Salt dives deeper into Pytest, the use of unittest.TestClass will be replaced by plain test functions, or test functions grouped in a class, which does not subclass unittest.TestClass, which, of course, doesn't work with unittest assert functions.
There are more examples of writing unit tests of varying complexities available in the following docs:
• Simple Unit Test Example
• Complete Unit Test Example
• Complex Unit Test Example
NOTE: Considerable care should be made to ensure that you're testing something useful in your test functions. It is very easy to fall into a situation where you have mocked so much of the original function that the test results in only asserting against the data you have provided. This results in a poor and fragile unit test.
Add a python module dependency to the test run The test dependencies for python modules are managed under the requirements/static directory. You will need to add your module to the appropriate file under requirements/static.  When pre-commit is run it will create all of the needed requirement files under requirements/static/py3{5,6,7}. Nox will then use these files to install the requirements for the tests.
Add a system dependency to the test run If you need to add a system dependency for the test run, this will need to be added in the salt jenkins repo. This repo uses salt states to install system dependencies.  You need to update the state-tree/golden-images-provision.sls file with your dependency to ensure it is installed. Once your PR is merged the core team will need to promote the new images with your new dependency installed.
Checking for Log Messages To test to see if a given log message has been emitted, the following pattern can be used
# Import logging handler from tests.support.helpers import TstSuiteLoggingHandler
# .. inside test with TstSuiteLoggingHandler() as handler: for message in handler.messages: if message.startswith("ERROR: This is the error message we seek"): break else: raise AssertionError("Did not find error message")
Automated Test Runs SaltStack maintains a Jenkins server which can be viewed at https://jenkins.saltproject.io. The tests executed from this Jenkins server create fresh virtual machines for each test run, then execute the destructive tests on the new, clean virtual machine. This allows for the execution of tests across supported platforms.
Additional Testing Documentation In addition to this tutorial, there are some other helpful resources and documentation that go into more depth on Salt's test runner, writing tests for Salt code, and general Python testing documentation. Please see the follow references for more information:
• Salt's Test Suite Documentation
• Integration Tests
• Unit Tests
• MagicMock
• Python Unittest
• Python's Assert Functions
Troubleshooting The intent of the troubleshooting section is to introduce solutions to a number of common issues encountered by users and the tools that are available to aid in developing States and Salt code.
Troubleshooting the Salt Master If your Salt master is having issues such as minions not returning data, slow execution times, or a variety of other issues, the following links contain details on troubleshooting the most common issues encountered:
Troubleshooting the Salt Master Running in the Foreground A great deal of information is available via the debug logging system, if you are having issues with minions connecting or not starting run the master in the foreground:
# salt-master -l debug
Anyone wanting to run Salt daemons via a process supervisor such as monit, runit, or supervisord, should omit the -d argument to the daemons and run them in the foreground.
What Ports does the Master Need Open? For the master, TCP ports 4505 and 4506 need to be open. If you've put both your Salt master and minion in debug mode and don't see an acknowledgment that your minion has connected, it could very well be a firewall interfering with the connection. See our firewall configuration page for help opening the firewall on various platforms.
If you've opened the correct TCP ports and still aren't seeing connections, check that no additional access control system such as SELinux or AppArmor is blocking Salt.
Too many open files The salt-master needs at least 2 sockets per host that connects to it, one for the Publisher and one for response port. Thus, large installations may, upon scaling up the number of minions accessing a given master, encounter:
12:45:29,289 [salt.master    ][INFO    ] Starting Salt worker process 38 Too many open files sock != -1 (tcp_listener.cpp:335)
The solution to this would be to check the number of files allowed to be opened by the user running salt-master (root by default):
[root@salt-master ~]# ulimit -n 1024
If this value is not equal to at least twice the number of minions, then it will need to be raised. For example, in an environment with 1800 minions, the nofile limit should be set to no less than 3600. This can be done by creating the file /etc/security/limits.d/99-salt.conf, with the following contents:
root        hard    nofile        4096 root        soft    nofile        4096
Replace root with the user under which the master runs, if different.
If your master does not have an /etc/security/limits.d directory, the lines can simply be appended to /etc/security/limits.conf.
As with any change to resource limits, it is best to stay logged into your current shell and open another shell to run ulimit -n again and verify that the changes were applied correctly. Additionally, if your master is running upstart, it may be necessary to specify the nofile limit in /etc/default/salt-master if upstart isn't respecting your resource limits:
limit nofile 4096 4096
NOTE: The above is simply an example of how to set these values, and you may wish to increase them even further if your Salt master is doing more than just running Salt.
Salt Master Stops Responding There are known bugs with ZeroMQ versions less than 2.1.11 which can cause the Salt master to not respond properly. If you're running a ZeroMQ version greater than or equal to 2.1.9, you can work around the bug by setting the sysctls net.core.rmem_max and net.core.wmem_max to 16777216. Next, set the third field in net.ipv4.tcp_rmem and net.ipv4.tcp_wmem to at least 16777216.
You can do it manually with something like:
# echo 16777216 > /proc/sys/net/core/rmem_max # echo 16777216 > /proc/sys/net/core/wmem_max # echo "4096 87380 16777216" > /proc/sys/net/ipv4/tcp_rmem # echo "4096 87380 16777216" > /proc/sys/net/ipv4/tcp_wmem
Or with the following Salt state:
net.core.rmem_max: sysctl: - present - value: 16777216
net.core.wmem_max: sysctl: - present - value: 16777216
net.ipv4.tcp_rmem: sysctl: - present - value: 4096 87380 16777216
net.ipv4.tcp_wmem: sysctl: - present - value: 4096 87380 16777216
Live Python Debug Output If the master seems to be unresponsive, a SIGUSR1 can be passed to the salt-master threads to display what piece of code is executing. This debug information can be invaluable in tracking down bugs.
To pass a SIGUSR1 to the master, first make sure the master is running in the foreground. Stop the service if it is running as a daemon, and start it in the foreground like so:
Then pass the signal to the master when it seems to be unresponsive:
# killall -SIGUSR1 salt-master
When filing an issue or sending questions to the mailing list for a problem with an unresponsive daemon, be sure to include this information if possible.
Live Salt-Master Profiling When faced with performance problems one can turn on master process profiling by sending it SIGUSR2.
# killall -SIGUSR2 salt-master
This will activate yappi profiler inside salt-master code, then after some time one must send SIGUSR2 again to stop profiling and save results to file. If run in foreground salt-master will report filename for the results, which are usually located under /tmp on Unix-based OSes and c:\temp on windows.
Make sure you have yappi installed.
Results can then be analyzed with kcachegrind or similar tool.
On Windows, in the absence of kcachegrind, a simple file-based workflow to create profiling graphs could use gprof2dot, graphviz and this batch file:
:: :: Converts callgrind* profiler output to *.pdf, via *.dot :: @echo off del *.dot.pdf for /r %%f in (callgrind*) do ( echo "%%f" gprof2dot.exe -f callgrind --show-samples  "%%f" -o "%%f.dot" dot.exe "%%f.dot" -Tpdf -O del "%%f.dot" )
Commands Time Out or Do Not Return Output Depending on your OS (this is most common on Ubuntu due to apt-get) you may sometimes encounter times where a state.apply, or other long running commands do not return output.
By default the timeout is set to 5 seconds. The timeout value can easily be increased by modifying the timeout line within your /etc/salt/master configuration file.
Having keys accepted for Salt minions that no longer exist or are not reachable also increases the possibility of timeouts, since the Salt master waits for those systems to return command results.
Passing the -c Option to Salt Returns a Permissions Error Using the -c option with the Salt command modifies the configuration directory. When the configuration file is read it will still base data off of the root_dir setting. This can result in unintended behavior if you are expecting files such as /etc/salt/pki to be pulled from the location specified with -c. Modify the root_dir setting to address this behavior.
Salt Master Doesn't Return Anything While Running jobs When a command being run via Salt takes a very long time to return (package installations, certain scripts, etc.) the master may drop you back to the shell. In most situations the job is still running but Salt has exceeded the set timeout before returning. Querying the job queue will provide the data of the job but is inconvenient. This can be resolved by either manually using the -t option to set a longer timeout when running commands (by default it is 5 seconds) or by modifying the master configuration file: /etc/salt/master and setting the timeout value to change the default timeout for all commands, and then restarting the salt-master service.
If a state.apply run takes too long, you can find a bottleneck by adding the --out=profile option.
Salt Master Auth Flooding In large installations, care must be taken not to overwhealm the master with authentication requests. Several options can be set on the master which mitigate the chances of an authentication flood from causing an interruption in service.
NOTE: recon_default:
The average number of seconds to wait between reconnection attempts.
recon_max: The maximum number of seconds to wait between reconnection attempts.
recon_randomize: A flag to indicate whether the recon_default value should be randomized.
acceptance_wait_time: The number of seconds to wait for a reply to each authentication request.
random_reauth_delay: The range of seconds across which the minions should attempt to randomize authentication attempts.
auth_timeout: The total time to wait for the authentication process to complete, regardless of the number of attempts.
Running states locally To debug the states, you can use call locally.
salt-call -l trace --local state.highstate
The top.sls file is used to map what SLS modules get loaded onto what minions via the state system.
It is located in the file defined in the file_roots variable of the salt master configuration file which is defined by found in CONFIG_DIR/master, normally /etc/salt/master
The default configuration for the file_roots is:
So the top file is defaulted to the location /srv/salt/top.sls
Salt Master Umask The salt master uses a cache to track jobs as they are published and returns come back.  The recommended umask for a salt-master is 022, which is the default for most users on a system. Incorrect umasks can result in permission-denied errors when the master tries to access files in its cache.
Troubleshooting the Salt Minion In the event that your Salt minion is having issues, a variety of solutions and suggestions are available. Please refer to the following links for more information:
Troubleshooting the Salt Minion Running in the Foreground A great deal of information is available via the debug logging system, if you are having issues with minions connecting or not starting run the minion in the foreground:
# salt-minion -l debug