neulab/tldr huggingface dataset:
• Since it occasionally takes some effort to "set up" an individual test, it may be necessary to call several functions within a single test. However, be sure that once the work has been done to set up a test, make sure you are clear about the functionality that is being tested.
Naming Conventions Test names and docstrings should indicate what functionality is being tested.  Test functions are named test_<fcn>_<test-name> where <fcn> is the function being tested and <test-name> describes the behavior being tested.
In order for integration tests to get picked up during a run of the test suite, each individual test must be prepended with the test_ naming syntax, as described above.
If a function does not start with test_, then the function acts as a "normal" function and is not considered a testing function. It will not be included in the test run or testing output.
The setUp and tearDown Functions There are two special functions that can be utilized in the integration side of Salt's test suite: setUp and tearDown. While these functions are not required in all test files, there are many examples in Salt's integration test suite illustrating the broad usefulness of each function.
The setUp function is used to set up any repetitive or useful tasks that the tests in a test class need before running. For example, any of the mac_* integration tests should only run on macOS machines. The setUp function can be used to test for the presence of the Darwin kernel. If the Darwin kernel is not present, then the test should be skipped.
def setUp(self): """ Sets up test requirements """ os_grain = self.run_function("grains.item", ["kernel"]) if os_grain["kernel"] not in "Darwin": self.skipTest("Test not applicable to '{kernel}' kernel".format(**os_grain))
The setUp function can be used for many things. The above code snippet is only one example. Another example might be to ensure that a particular setting is present before running tests that would require the setting.
The tearDown function is used to clean up after any tests. This function is useful for restoring any settings that might have been changed during the test run.
NOTE: The setUp and tearDown functions run before and after each test in the test class that the setUp and tearDown functions are defined.
Be sure to read the Destructive vs Non-Destructive Tests section when using any kind of destructive functions that might alter the system running the test suite in either the setUp or tearDown function definitions.
Testing Order The test functions within a test class do not run in the order they were defined, but instead run in lexicographical order.
Note that if any setUp or tearDown functions are defined in the class, those functions will run before (for setUp) or after (for tearDown) each test case.
Integration Classes The integration classes are located in tests/integration/__init__.py and can be extended therein. There are four classes available to extend:
• ModuleCase
• ShellCase
• SSHCase
• SyndicCase
ModuleCase Used to define executions run via the master to minions and to call single modules and states. The available testing functions are:
run_function Run a single salt function and condition the return down to match the behavior of the raw function call. This will run the command and only return the results from a single minion to verify.
run_state Run the state.single command and return the state return structure.
minion_run Run a single salt function on the 'minion' target and condition the return down to match the behavior of the raw function call.
ShellCase Shell out to the scripts which ship with Salt. The testing functions are:
run_cp Execute salt-cp. Pass in the argument string as it would be passed on the command line.
run_call Execute salt-call, pass in the argument string as it would be passed on the command line.
run_cloud Execute the salt-cloud command. Pass in the argument string as it would be passed on the command line.
run_key Execute the salt-key command. Pass in the argument string as it would be passed on the command line.
run_run Execute the salt-run command. Pass in the argument string as it would be passed on the command line.
run_run_plus Execute the runner function the and return the return data and output in a dict
run_salt Execute the salt command. Pass in the argument string as it would be passed on the command line.
run_script Execute a salt script with the given argument string.
run_ssh Execute the salt-ssh. Pass in the argument string as it would be passed on the command line.
SSHCase Used to execute remote commands via salt-ssh. The available methods are as follows:
run_function Run a single salt function via salt-ssh and condition the return down to match the behavior of the raw function call. This will run the command and only return the results from a single minion to verify.
SyndicCase Used to execute remote commands via a syndic and is only used to verify the capabilities of the Salt Syndic. The available methods are as follows:
Examples The following sections define simple integration tests present in Salt's integration test suite for each type of testing class.
Module Example via ModuleCase Class Import the integration module, this module is already added to the python path by the test execution. Inherit from the integration.ModuleCase class.
Now the workhorse method run_function can be used to test a module:
import os from tests.support.case import ModuleCase
class TestModuleTest(ModuleCase): """ Validate the test module """
The fist example illustrates the testing master issuing a test.ping call to a testing minion. The test asserts that the minion returned with a True value to the master from the test.ping call.
The second example similarly verifies that the minion executed the test.echo command with the text argument. The assertEqual call maintains that the minion ran the function and returned the data as expected to the master.
Shell Example via ShellCase Validating the shell commands can be done via shell tests:
import sys import shutil import tempfile
from tests.support.case import ShellCase
class KeyTest(ShellCase): """ Test salt-key script """
_call_binary_ = "salt-key"
def test_list(self): """ test salt-key -L """ data = self.run_key("-L") expect = [ "Unaccepted Keys:", "Accepted Keys:", "minion", "sub_minion", "Rejected:", "", ] self.assertEqual(data, expect)
This example verifies that the salt-key command executes and returns as expected by making use of the run_key method.
SSH Example via SSHCase Testing salt-ssh functionality can be done using the SSHCase test class:
from tests.support.case import SSHCase
class SSHGrainsTest(SSHCase): """ Test salt-ssh grains functionality Depend on proper environment set by integration.SSHCase class """
def test_grains_id(self): """ Test salt-ssh grains id work for localhost. """ cmd = self.run_function("grains.get", ["id"]) self.assertEqual(cmd, "localhost")
Testing Event System via SaltMinionEventAssertsMixin The fundamentially asynchronous nature of Salt makes testing the event system a challenge.  The SaltMinionEventAssertsMixin provides a facility for testing that events were received on a minion event bus.
import salt.utils.event from tests.support.mixins import SaltEventAssertsMixin
class TestEvent(SaltEventAssertsMixin): """ Example test of firing an event and receiving it """
def test_event(self): e = salt.utils.event.get_event( "minion", sock_dir=self.minion_opts["sock_dir"], opts=self.minion_opts )
e.fire_event({"a": "b"}, "/test_event")
self.assertMinionEventReceived({"a": "b"})
Syndic Example via SyndicCase Testing Salt's Syndic can be done via the SyndicCase test class:
from tests.support.case import SyndicCase
class TestSyndic(SyndicCase): """ Validate the syndic interface by testing the test module """
This example verifies that a test.ping command is issued from the testing master, is passed through to the testing syndic, down to the minion, and back up again by using the run_function located with in the SyndicCase test class.
Integration Test Files Since using Salt largely involves configuring states, editing files, and changing system data, the integration test suite contains a directory named files to aid in testing functions that require files. Various Salt integration tests use these example files to test against instead of altering system files and data.
Each directory within tests/integration/files contain files that accomplish different tasks, based on the needs of the integration tests using those files.  For example, tests/integration/files/ssh is used to bootstrap the test runner for salt-ssh testing, while tests/integration/files/pillar contains files storing data needed to test various pillar functions.
The tests/integration/files directory also includes an integration state tree.  The integration state tree can be found at tests/integration/files/file/base.
The following example demonstrates how integration files can be used with ModuleCase to test states:
# Import python libs from __future__ import absolute_import import os import shutil
# Import Salt Testing libs from tests.support.case import ModuleCase from tests.support.paths import FILES, TMP from tests.support.mixins import SaltReturnAssertsMixin
# Import salt libs import salt.utils.files
HFILE = os.path.join(TMP, "hosts")
class HostTest(ModuleCase, SaltReturnAssertsMixin): """ Validate the host state """
def setUp(self): shutil.copyfile(os.path.join(FILES, "hosts"), HFILE) super(HostTest, self).setUp()
def tearDown(self): if os.path.exists(HFILE): os.remove(HFILE) super(HostTest, self).tearDown()
def test_present(self): """ host.present """ name = "spam.bacon" ip = "10.10.10.10" ret = self.run_state("host.present", name=name, ip=ip) self.assertSaltTrueReturn(ret) with salt.utils.files.fopen(HFILE) as fp_: output = fp_.read() self.assertIn("{0}\t\t{1}".format(ip, name), output)
To access the integration files, a variable named FILES points to the tests/integration/files directory. This is where the referenced host.present sls file resides.
In addition to the static files in the integration state tree, the location TMP can also be used to store temporary files that the test system will clean up when the execution finishes.
Destructive vs Non-Destructive Tests Since Salt is used to change the settings and behavior of systems, one testing approach is to run tests that make actual changes to the underlying system. This is where the concept of destructive integration tests comes into play. Tests can be written to alter the system they are running on. This capability is what fills in the gap needed to properly test aspects of system management like package installation.
Any test that changes the underlying system in any way, such as creating or deleting users, installing packages, or changing permissions should include the @destructive decorator to signal system changes and should be written with care. System changes executed within a destructive test should also be restored once the related tests have completed. For example, if a new user is created to test a module, the same user should be removed after the test is completed to maintain system integrity.
To write a destructive test, import, and use the destructiveTest decorator for the test method:
from tests.support.case import ModuleCase from tests.support.helpers import destructiveTest, skip_if_not_root
class DestructiveExampleModuleTest(ModuleCase): """ Demonstrate a destructive test """
@destructiveTest @skip_if_not_root def test_user_not_present(self): """ This is a DESTRUCTIVE TEST it creates a new user on the minion. And then destroys that user. """ ret = self.run_state("user.present", name="salt_test") self.assertSaltTrueReturn(ret) ret = self.run_state("user.absent", name="salt_test") self.assertSaltTrueReturn(ret)
Cloud Provider Tests Cloud provider integration tests are used to assess Salt-Cloud's ability to create and destroy cloud instances for various supported cloud providers.  Cloud provider tests inherit from the ShellCase Integration Class.
Any new cloud provider test files should be added to the tests/integration/cloud/providers/ directory. Each cloud provider test file also requires a sample cloud profile and cloud provider configuration file in the integration test file directory located at tests/integration/files/conf/cloud.*.d/.
The following is an example of the default profile configuration file for Digital Ocean, located at: tests/integration/files/conf/cloud.profiles.d/digitalocean.conf:
digitalocean-test: provider: digitalocean-config image: Ubuntu 14.04 x64 size: 512MB
Each cloud provider requires different configuration credentials. Therefore, sensitive information such as API keys or passwords should be omitted from the cloud provider configuration file and replaced with an empty string. The necessary credentials can be provided by the user by editing the provider configuration file before running the tests.
The following is an example of the default provider configuration file for Digital Ocean, located at: tests/integration/files/conf/cloud.providers.d/digitalocean.conf:
In addition to providing the necessary cloud profile and provider files in the integration test suite file structure, appropriate checks for if the configuration files exist and contain valid information are also required in the test class's setUp function:
from tests.support.case import ShellCase from tests.support.paths import FILES
class LinodeTest(ShellCase): """ Integration tests for the Linode cloud provider in Salt-Cloud """
def setUp(self): """ Sets up the test requirements """ super(LinodeTest, self).setUp()
# check if appropriate cloud provider and profile files are present profile_str = "linode-config:" provider = "linode" providers = self.run_cloud("--list-providers") if profile_str not in providers: self.skipTest( "Configuration file for {0} was not found. Check {0}.conf files " "in tests/integration/files/conf/cloud.*.d/ to run these tests.".format( provider ) )
# check if apikey and password are present path = os.path.join(FILES, "conf", "cloud.providers.d", provider + ".conf") config = cloud_providers_config(path) api = config["linode-config"]["linode"]["apikey"] password = config["linode-config"]["linode"]["password"] if api == "" or password == "": self.skipTest( "An api key and password must be provided to run these tests. Check " "tests/integration/files/conf/cloud.providers.d/{0}.conf".format( provider ) )
Repeatedly creating and destroying instances on cloud providers can be costly.  Therefore, cloud provider tests are off by default and do not run automatically. To run the cloud provider tests, the --cloud-provider-tests flag must be provided:
./tests/runtests.py --cloud-provider-tests
Since cloud provider tests do not run automatically, all provider tests must be preceded with the @expensiveTest decorator. The expensive test decorator is necessary because it signals to the test suite that the --cloud-provider-tests flag is required to run the cloud provider tests.
To write a cloud provider test, import, and use the expensiveTest decorator for the test function:
from tests.support.helpers import expensiveTest
@expensiveTest def test_instance(self): """ Test creating an instance on Linode """ name = "linode-testing"
# create the instance instance = self.run_cloud("-p linode-test {0}".format(name)) str = "        {0}".format(name)
# check if instance with salt installed returned as expected try: self.assertIn(str, instance) except AssertionError: self.run_cloud("-d {0} --assume-yes".format(name)) raise
# delete the instance delete = self.run_cloud("-d {0} --assume-yes".format(name)) str = "            True" try: self.assertIn(str, delete) except AssertionError: raise
Adding New Directories If the corresponding Salt directory does not exist within tests/integration, the new directory must be created along with the appropriate test file to maintain Salt's testing directory structure.
In order for Salt's test suite to recognize tests within the newly created directory, options to run the new integration tests must be added to tests/runtests.py. Examples of the necessary options that must be added can be found here: https://github.com/saltstack/salt/blob/master/tests/runtests.py. The functions that need to be edited are setup_additional_options, validate_options, and run_integration_tests.
Writing Unit Tests Introduction Like many software projects, Salt has two broad-based testing approaches -- integration testing and unit testing. While integration testing focuses on the interaction between components in a sandboxed environment, unit testing focuses on the singular implementation of individual functions.
Unit tests should be used specifically to test a function's logic. Unit tests rely on mocking external resources.
While unit tests are good for ensuring consistent results, they are most useful when they do not require more than a few mocks. Effort should be made to mock as many external resources as possible. This effort is encouraged, but not required. Sometimes the isolation provided by completely mocking the external dependencies is not worth the effort of mocking those dependencies.
In these cases, requiring an external library to be installed on the system before running the test file is a useful way to strike this balance.  For example, the unit tests for the MySQL execution module require the presence of the MySQL python bindings on the system running the test file before proceeding to run the tests.
Overly detailed mocking can also result in decreased test readability and brittleness as the tests are more likely to fail when the code or its dependencies legitimately change. In these cases, it is better to add dependencies to the test runner dependency state.
Preparing to Write a Unit Test This guide assumes that your Salt development environment is already configured and that you have a basic understanding of contributing to the Salt codebase.  If you're unfamiliar with either of these topics, please refer to the Installing Salt for Development and the Contributing pages, respectively.
This documentation also assumes that you have an understanding of how to run Salt's test suite, including running the unit test subsection, running the unit tests without testing daemons to speed up development wait times, and running a unit test file, class, or individual test.
Best Practices Unit tests should be written to the following specifications.
What to Test? Since unit testing focuses on the singular implementation of individual functions, unit tests should be used specifically to test a function's logic. The following guidelines should be followed when writing unit tests for Salt's test suite:
• Each raise and return statement needs to be independently tested.
• Test functions should contain only one assertion, at most, multiple assertions can be made, but against the same outcome.
• Many Salt execution modules are merely wrappers for distribution-specific functionality. If there isn't any logic present in a simple execution module, consider writing an integration test instead of heavily mocking a call to an external dependency.
Mocking Test Data A reasonable effort needs to be made to mock external resources used in the code being tested, such as APIs, function calls, external data either globally available or passed in through function arguments, file data, etc.
• Test functions should contain only one assertion and all necessary mock code and data for that assertion.
• External resources should be mocked in order to "block all of the exits". If a test function fails because something in an external library wasn't mocked properly (or at all), this test is not addressing all of the "exits" a function may experience. We want the Salt code and logic to be tested, specifically.
• Consider the fragility and longevity of a test. If the test is so tightly coupled to the code being tested, this makes a test unnecessarily fragile.
• Make sure you are not mocking the function to be tested so vigorously that the test return merely tests the mocked output. The test should always be testing a function's logic.
Mocking Loader Modules Salt loader modules use a series of globally available dunder variables, __salt__, __opts__, __pillar__, etc. To facilitate testing these modules pytest-salt-factories has a plugin which will prepare the module globals by patching and mocking the dunders prior to running each test, as long as the test module defines a fixture named configure_loader_modules. Check out the code for the pytest-salt-factories plugin to know how it's internals work.
The reason for the existence of this plugin is because historically one would add these dunder variables directly on the imported module. This, however, introduces unexpected behavior when running the full test suite since those attributes would not be removed once we were done testing the module and would therefore leak to other modules being tested with unpredictable results. This is the kind of work that should be deferred to mock, and that's exactly what this plugin provides.
As an example, if one needs to specify some options which should be available to the module being tested one should do:
import pytest import salt.modules.somemodule as somemodule
@pytest.fixture def configure_loader_modules(): """ This fixture should return a dictionary which is what's going to be used to patch and mock Salt's loader """ return {somemodule: {"__opts__": {"test": True}}}
Consider this more extensive example from tests/pytests/unit/beacons/test_sensehat.py:
import pytest import salt.beacons.sensehat as sensehat from tests.support.mock import MagicMock
@pytest.fixture def configure_loader_modules(): return { sensehat: { "__salt__": { "sensehat.get_humidity": MagicMock(return_value=80), "sensehat.get_temperature": MagicMock(return_value=30), "sensehat.get_pressure": MagicMock(return_value=1500), }, } }
def test_non_list_config(): config = {}
ret = sensehat.validate(config)
assert ret == (False, "Configuration for sensehat beacon must be a list.")
def test_empty_config(): config = [{}]
assert ret == (False, "Configuration for sensehat beacon requires sensors.")
def test_sensehat_humidity_match():
config = [{"sensors": {"humidity": "70%"}}]
ret = sensehat.validate(config) assert ret == (True, "Valid beacon configuration")
ret = sensehat.beacon(config) assert ret == [{"tag": "sensehat/humidity", "humidity": 80}]
def test_sensehat_temperature_match():
config = [{"sensors": {"temperature": 20}}]
ret = sensehat.beacon(config) assert ret == [{"tag": "sensehat/temperature", "temperature": 30}]
def test_sensehat_temperature_match_range():
config = [{"sensors": {"temperature": [20, 29]}}]
def test_sensehat_pressure_match():
config = [{"sensors": {"pressure": "1400"}}]
ret = sensehat.beacon(config) assert ret == [{"tag": "sensehat/pressure", "pressure": 1500}]
def test_sensehat_no_match():
config = [{"sensors": {"pressure": "1600"}}]
ret = sensehat.beacon(config) assert ret == []
What happens in the above example is we mock several calls of the sensehat module to return known expected values to assert against.
Mocking Filehandles NOTE: This documentation applies to the 2018.3 release cycle and newer. The extended functionality for mock_open described below does not exist in the 2017.7 and older release branches.
Opening files in Salt is done using salt.utils.files.fopen(). When testing code that reads from files, the mock_open helper can be used to mock filehandles. Note that is not the same mock_open as unittest.mock.mock_open() from the Python standard library, but rather a separate implementation which has additional functionality.
from tests.support.mock import patch, mock_open
import salt.modules.mymod as mymod
def test_something(): fopen_mock = mock_open(read_data="foo\nbar\nbaz\n") with patch("salt.utils.files.fopen", fopen_mock): result = mymod.myfunc() assert result is True
This will force any filehandle opened to mimic a filehandle which, when read, produces the specified contents.
IMPORTANT: String Types
When configuring your read_data, make sure that you are using bytestrings (e.g. b"foo\nbar\nbaz\n") when the code you are testing is opening a file for binary reading, otherwise the tests will fail.  The mocked filehandles produced by mock_open will raise a TypeError if you attempt to read a bytestring when opening for non-binary reading, and similarly will not let you read a string when opening a file for binary reading. They will also not permit bytestrings to be "written" if the mocked filehandle was opened for non-binary writing, and vice-versa when opened for non-binary writing. These enhancements force test writers to write more accurate tests.
More Complex Scenarios Multiple File Paths What happens when the code being tested reads from more than one file? For those cases, you can pass read_data as a dictionary:
import textwrap
def test_something(): contents = { "/etc/foo.conf": textwrap.dedent( """\ foo bar baz """ ), "/etc/b*.conf": textwrap.dedent( """\ one two three """ ), } fopen_mock = mock_open(read_data=contents) with patch("salt.utils.files.fopen", fopen_mock): result = mymod.myfunc() assert result is True
This would make salt.utils.files.fopen() produce filehandles with different contents depending on which file was being opened by the code being tested.  /etc/foo.conf and any file matching the pattern /etc/b*.conf would work, while opening any other path would result in a FileNotFoundError being raised.
Since file patterns are supported, it is possible to use a pattern of '*' to define a fallback if no other patterns match the filename being opened. The below two mock_open calls would produce identical results:
mock_open(read_data="foo\n") mock_open(read_data={"*": "foo\n"})
NOTE: Take care when specifying the read_data as a dictionary, in cases where the patterns overlap (e.g. when both /etc/b*.conf and /etc/bar.conf are in the read_data). Dictionary iteration order will determine which pattern is attempted first, second, etc., with the exception of * which is used when no other pattern matches. If your test case calls for specifying overlapping patterns, and you are not running Python 3.6 or newer, then an OrderedDict can be used to ensure matching is handled in the desired way:
contents = OrderedDict() contents["/etc/bar.conf"] = "foo\nbar\nbaz\n" contents["/etc/b*.conf"] = IOError(errno.EACCES, "Permission denied") contents["*"] = 'This is a fallback for files not beginning with "/etc/b"\n' fopen_mock = mock_open(read_data=contents)
Raising Exceptions Instead of a string, an exception can also be used as the read_data:
def test_something(): exc = IOError(errno.EACCES, "Permission denied") fopen_mock = mock_open(read_data=exc) with patch("salt.utils.files.fopen", fopen_mock): mymod.myfunc()
The above example would raise the specified exception when any file is opened.  The expectation would be that mymod.myfunc() would gracefully handle the IOError, so a failure to do that would result in it being raised and causing the test to fail.
Multiple File Contents For cases in which a file is being read more than once, and it is necessary to test a function's behavior based on what the file looks like the second (or third, etc.) time it is read, just specify the contents for that file as a list. Each time the file is opened, mock_open will cycle through the list and produce a mocked filehandle with the specified contents. For example:
import errno import textwrap
def test_something(): contents = { "/etc/foo.conf": [ textwrap.dedent( """\ foo bar """ ), textwrap.dedent( """\ foo bar baz """ ), ], "/etc/b*.conf": [ IOError(errno.ENOENT, "No such file or directory"), textwrap.dedent( """\ one two three """ ), ], } fopen_mock = mock_open(read_data=contents) with patch("salt.utils.files.fopen", fopen_mock): result = mymod.myfunc() assert result is True
Using this example, the first time /etc/foo.conf is opened, it will simulate a file with the first string in the list as its contents, while the second time it is opened, the simulated file's contents will be the second string in the list.
If no more items remain in the list, then attempting to open the file will raise a RuntimeError. In the example above, if /etc/foo.conf were to be opened a third time, a RuntimeError would be raised.
Note that exceptions can also be mixed in with strings when using this technique. In the above example, if /etc/bar.conf were to be opened twice, the first time would simulate the file not existing, while the second time would simulate a file with string defined in the second element of the list.
NOTE: Notice that the second path in the contents dictionary above (/etc/b*.conf) contains an asterisk. The items in the list are cycled through for each match of a given pattern (not separately for each individual file path), so this means that only two files matching that pattern could be opened before the next one would raise a RuntimeError.
Accessing the Mocked Filehandles in a Test NOTE: The code for the MockOpen, MockCall, and MockFH classes (referenced below) can be found in tests/support/mock.py. There are extensive unit tests for them located in tests/unit/test_mock.py.
The above examples simply show how to mock salt.utils.files.fopen() to simulate files with the contents you desire, but you can also access the mocked filehandles (and more), and use them to craft assertions in your tests. To do so, just add an as clause to the end of the patch statement:
fopen_mock = mock_open(read_data="foo\nbar\nbaz\n") with patch("salt.utils.files.fopen", fopen_mock) as m_open: # do testing here ... ...
When doing this, m_open will be a MockOpen instance. It will contain several useful attributes:
• read_data - A dictionary containing the read_data passed when mock_open was invoked. In the event that multiple file paths are not used, then this will be a dictionary mapping * to the read_data passed to mock_open.
• call_count - An integer representing how many times salt.utils.files.fopen() was called to open a file.
• calls - A list of MockCall objects. A MockCall object is a simple class which stores the arguments passed to it, making the positional arguments available via its args attribute, and the keyword arguments available via its kwargs attribute.
from tests.support.mock import patch, mock_open, MockCall
def test_something():
with patch("salt.utils.files.fopen", mock_open(read_data=b"foo\n")) as m_open: mymod.myfunc() # Assert that only two opens attempted assert m_open.call_count == 2 # Assert that only /etc/foo.conf was opened assert all(call.args[0] == "/etc/foo.conf" for call in m_open.calls) # Asser that the first open was for binary read, and the # second was for binary write. assert m_open.calls == [ MockCall("/etc/foo.conf", "rb"), MockCall("/etc/foo.conf", "wb"), ]
Note that MockCall is imported from tests.support.mock in the above example. Also, the second assert above is redundant since it is covered in the final assert, but both are included simply as an example.
• filehandles - A dictionary mapping the unique file paths opened, to lists of MockFH objects. Each open creates a unique MockFH object. Each MockFH object itself has a number of useful attributes:
• filename - The path to the file which was opened using salt.utils.files.fopen()
• call - A MockCall object representing the arguments passed to salt.utils.files.fopen(). Note that this MockCall is also available in the parent MockOpen instance's calls list.
• The following methods are mocked using unittest.mock.Mock objects, and Mock's built-in asserts (as well as the call data) can be used as you would with any other Mock object:
• .read()
• .readlines()
• .readline()
• .close()
• .write()
• .writelines()
• .seek()
• The read functions (.read(), .readlines(), .readline()) all work as expected, as does iterating through the file line by line (i.e.  for line in fh:).
• The .tell() method is also implemented in such a way that it updates after each time the mocked filehandle is read, and will report the correct position. The one caveat here is that .seek() doesn't actually work (it's simply mocked), and will not change the position. Additionally, neither .write() or .writelines() will modify the mocked filehandle's contents.
• The attributes .write_calls and .writelines_calls (no parenthesis) are available as shorthands and correspond to lists containing the contents passed for all calls to .write() and .writelines(), respectively.
Examples with patch("salt.utils.files.fopen", mock_open(read_data=contents)) as m_open: # Run the code you are unit testing mymod.myfunc() # Check that only the expected file was opened, and that it was opened # only once. assert m_open.call_count == 1 assert list(m_open.filehandles) == ["/etc/foo.conf"] # "opens" will be a list of all the mocked filehandles opened opens = m_open.filehandles["/etc/foo.conf"] # Check that we wrote the expected lines ("expected" here is assumed to # be a list of strings) assert opens[0].write_calls == expected
with patch("salt.utils.files.fopen", mock_open(read_data=contents)) as m_open: # Run the code you are unit testing mymod.myfunc() # Check that .readlines() was called (remember, it's a Mock) m_open.filehandles["/etc/foo.conf"][0].readlines.assert_called()
with patch("salt.utils.files.fopen", mock_open(read_data=contents)) as m_open: # Run the code you are unit testing mymod.myfunc() # Check that we read the file and also wrote to it m_open.filehandles["/etc/foo.conf"][0].read.assert_called_once() m_open.filehandles["/etc/foo.conf"][1].writelines.assert_called_once()
Naming Conventions Test names and docstrings should indicate what functionality is being tested.  Test functions are named test_<fcn>_<test-name> where <fcn> is the function being tested and <test-name> describes the raise or return being tested.
Unit tests for salt/.../<module>.py are contained in a file called tests/pytests/unit/.../test_<module>.py, e.g. the tests for salt/modules/alternatives.py are in tests/pytests/unit/modules/test_alternatives.py.
In order for unit tests to get picked up during a run of the unit test suite, each unit test file must be prefixed with test_ and each individual test must also be prefixed with the test_ naming syntax, as described above.
If a function does not start with test_, then the function acts as a "normal" function and is not considered a testing function. It will not be included in the test run or testing output. The same principle applies to unit test files that do not have the test_*.py naming syntax. This test file naming convention is how the test runner recognizes that a test file contains tests.
Imports Most commonly, the following imports are necessary to create a unit test:
If you need mock support to your tests, please also import:
from tests.support.mock import MagicMock, patch, call
Evaluating Truth A longer discussion on the types of assertions one can make can be found by reading PyTest's documentation on assertions.
Tests Using Mock Objects In many cases, the purpose of a Salt module is to interact with some external system, whether it be to control a database, manipulate files on a filesystem or something else. In these varied cases, it's necessary to design a unit test which can test the function whilst replacing functions which might actually call out to external systems. One might think of this as "blocking the exits" for code under tests and redirecting the calls to external systems with our own code which produces known results during the duration of the test.
To achieve this behavior, Salt makes heavy use of the MagicMock package.
To understand how one might integrate Mock into writing a unit test for Salt, let's imagine a scenario in which we're testing an execution module that's designed to operate on a database. Furthermore, let's imagine two separate methods, here presented in pseduo-code in an imaginary execution module called 'db.py'.
def create_user(username): qry = "CREATE USER {0}".format(username) execute_query(qry)
def execute_query(qry): # Connect to a database and actually do the query... ...
Here, let's imagine that we want to create a unit test for the create_user function. In doing so, we want to avoid any calls out to an external system and so while we are running our unit tests, we want to replace the actual interaction with a database with a function that can capture the parameters sent to it and return pre-defined values. Therefore, our task is clear -- to write a unit test which tests the functionality of create_user while also replacing 'execute_query' with a mocked function.
To begin, we set up the skeleton of our test much like we did before, but with additional imports for MagicMock:
from salt.modules import db from tests.support.mock import MagicMock, patch, call
def test_create_user(): """ Test creating a user """ # First, we replace 'execute_query' with our own mock function with patch.object(db, "execute_query", MagicMock()) as db_exq:
# Now that the exits are blocked, we can run the function under test. db.create_user("testuser")
# We could now query our mock object to see which calls were made # to it. ## print db_exq.mock_calls
# Construct a call object that simulates the way we expected # execute_query to have been called. expected_call = call("CREATE USER testuser")
# Compare the expected call with the list of actual calls.  The # test will succeed or fail depending on the output of this # assertion. db_exq.assert_has_calls(expected_call)
Modifying __salt__ In Place At times, it becomes necessary to make modifications to a module's view of functions in its own __salt__ dictionary.  Luckily, this process is quite easy.
Below is an example that uses MagicMock's patch functionality to insert a function into __salt__ that's actually a MagicMock instance.
import pytest import salt.modules.my_module as my_module
@pytest.fixture def configure_loader_modules(): """ This fixture should return a dictionary which is what's going to be used to patch and mock Salt's loader """ return {my_module: {}}
def show_patch(self): with patch.dict(my_module.__salt__, {"function.to_replace": MagicMock()}): # From this scope, carry on with testing, with a modified __salt__! ...
A Simple Example Let's assume that we're testing a very basic function in an imaginary Salt execution module. Given a module called fib.py that has a function called calculate(num_of_results), which given a num_of_results, produces a list of sequential Fibonacci numbers of that length.
A unit test to test this function might be commonly placed in a file called tests/pytests/unit/modules/test_fib.py. The convention is to place unit tests for Salt execution modules in test/pytests/unit/modules/ and to name the tests module prefixed with test_*.py.
Tests are grouped around test cases, which are logically grouped sets of tests against a piece of functionality in the tested software. To return to our example, here's how we might write the skeleton for testing fib.py:
import salt.modules.fib as fib
def test_fib(): """ To create a unit test, we should prefix the name with `test_' so that it's recognized by the test runner. """ fib_five = (0, 1, 1, 2, 3) assert fib.calculate(5) == fib_five
At this point, the test can now be run, either individually or as a part of a full run of the test runner. To ease development, a single test can be executed:
nox -e 'pytest-3(coverage=False)' -- -v tests/pytests/unit/modules/test_fib.py
This will report the status of the test: success, failure, or error.  The -v flag increases output verbosity.
To review the results of a particular run, take a note of the log location given in the output for each test run:
...etc... --log-file=artifacts/logs/runtests-20210106103414.685791.log ...etc...
A More Complete Example Consider the following function from salt/modules/linux_sysctl.py.
def get(name): """ Return a single sysctl parameter for this minion
salt '*' sysctl.get net.ipv4.ip_forward """ cmd = "sysctl -n {}".format(name) out = __salt__["cmd.run"](cmd, python_shell=False) return out
This function is very simple, comprising only four source lines of code and having only one return statement, so we know only one test is needed. There are also two inputs to the function, the name function argument and the call to __salt__['cmd.run'](), both of which need to be appropriately mocked.
Mocking a function parameter is straightforward, whereas mocking a function call will require, in this case, the use of MagicMock.  For added isolation, we will also redefine the __salt__ dictionary such that it only contains 'cmd.run'.
import pytest import salt.modules.linux_sysictl as linux_sysctl from tests.support.mock import MagicMock, patch
@pytest.fixture def configure_loader_modules(): return {linux_sysctl: {}}
def test_get(): """ Tests the return of get function """ mock_cmd = MagicMock(return_value=1) with patch.dict(linux_sysctl.__salt__, {"cmd.run": mock_cmd}): assert linux_sysctl.get("net.ipv4.ip_forward") == 1
Since get() has only one raise or return statement and that statement is a success condition, the test function is simply named test_get().  As described, the single function call parameter, name is mocked with net.ipv4.ip_forward and __salt__['cmd.run'] is replaced by a MagicMock function object.  We are only interested in the return value of __salt__['cmd.run'], which MagicMock allows us by specifying via return_value=1.  Finally, the test itself tests for equality between the return value of get() and the expected return of 1.  This assertion is expected to succeed because get() will determine its return value from __salt__['cmd.run'], which we have mocked to return 1.
A Complex Example Now consider the assign() function from the same salt/modules/linux_sysctl.py source file.
def assign(name, value): """ Assign a single sysctl parameter for this minion
salt '*' sysctl.assign net.ipv4.ip_forward 1 """ value = str(value)
tran_tab = name.translate("".maketrans("./", "/."))
sysctl_file = "/proc/sys/{}".format(tran_tab) if not os.path.exists(sysctl_file): raise CommandExecutionError("sysctl {} does not exist".format(name))
ret = {} cmd = 'sysctl -w {}="{}"'.format(name, value) data = __salt__["cmd.run_all"](cmd, python_shell=False) out = data["stdout"] err = data["stderr"]
# Example: #    # sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216" #    net.ipv4.tcp_rmem = 4096 87380 16777216 regex = re.compile(r"^{}\s+=\s+{}$".format(re.escape(name), re.escape(value)))
if not regex.match(out) or "Invalid argument" in str(err): if data["retcode"] != 0 and err: error = err else: error = out raise CommandExecutionError("sysctl -w failed: {}".format(error)) new_name, new_value = out.split(" = ", 1) ret[new_name] = new_value return ret
This function contains two raise statements and one return statement, so we know that we will need (at least) three tests.  It has two function arguments and many references to non-builtin functions.  In the tests below you will see that MagicMock's patch() method may be used as a context manager or as a decorator. When patching the salt dunders however, please use the context manager approach.
There are three test functions, one for each raise and return statement in the source function.  Each function is self-contained and contains all and only the mocks and data needed to test the raise or return statement it is concerned with.
import pytest import salt.modules.linux_sysctl as linux_sysctl from salt.exceptions import CommandExecutionError from tests.support.mock import MagicMock, patch
def test_assign_proc_sys_failed(): """ Tests if /proc/sys/<kernel-subsystem> exists or not """ with patch("os.path.exists", MagicMock(return_value=False)): cmd = { "pid": 1337, "retcode": 0, "stderr": "", "stdout": "net.ipv4.ip_forward = 1", } mock_cmd = MagicMock(return_value=cmd) with patch.dict(linux_sysctl.__salt__, {"cmd.run_all": mock_cmd}): with pytest.raises(CommandExecutionError): linux_sysctl.assign("net.ipv4.ip_forward", 1)
def test_assign_cmd_failed(): """ Tests if the assignment was successful or not """ with patch("os.path.exists", MagicMock(return_value=True)): cmd = { "pid": 1337, "retcode": 0, "stderr": 'sysctl: setting key "net.ipv4.ip_forward": Invalid argument', "stdout": "net.ipv4.ip_forward = backward", } mock_cmd = MagicMock(return_value=cmd) with patch.dict(linux_sysctl.__salt__, {"cmd.run_all": mock_cmd}): with pytest.raises(CommandExecutionError): linux_sysctl.assign("net.ipv4.ip_forward", "backward")
def test_assign_success(): """ Tests the return of successful assign function """ with patch("os.path.exists", MagicMock(return_value=True)): cmd = { "pid": 1337, "retcode": 0, "stderr": "", "stdout": "net.ipv4.ip_forward = 1", } ret = {"net.ipv4.ip_forward": "1"} mock_cmd = MagicMock(return_value=cmd) with patch.dict(linux_sysctl.__salt__, {"cmd.run_all": mock_cmd}): assert linux_sysctl.assign("net.ipv4.ip_forward", 1) == ret
SaltStack Git Policy The SaltStack team follows a git policy to maintain stability and consistency with the repository.
The git policy has been developed to encourage contributions and make contributing to Salt as easy as possible. Code contributors to SaltStack projects DO NOT NEED TO READ THIS DOCUMENT, because all contributions come into SaltStack via a single gateway to make it as easy as possible for contributors to give us code.
The primary rule of git management in SaltStack is to make life easy on contributors and developers to send in code. Simplicity is always a goal!
New Code Entry All new SaltStack code should be submitted against master.
Release Branching SaltStack maintains two types of releases, Feature Releases and Point Releases (also commonly referred to as Bugfix Releases. A feature release is managed by incrementing the first or second release point number, so 2015.5.5 -> 2015.8.0 signifies a feature release and 2015.8.0 -> 2015.8.1 signifies a point release.
Feature Release Branching Each feature release is maintained in a dedicated git branch derived from the last applicable release commit on develop. All file changes relevant to the feature release will be completed in the master branch prior to the creation of the feature release branch. The feature release branch will be named after the relevant numbers to the feature release, which constitute the first two numbers. This means that the release branch for the 2015.8.0 series is named 2015.8.
A feature release branch is created with the following command:
# git checkout -b 2015.8 # From the master branch # git push origin 2015.8
Point Releases As documented in SEP 14, point releases should be rare.
Salt Conventions Writing Salt Documentation Salt's documentation is built using the Sphinx documentation system. It can be built in a large variety of output formats including HTML, PDF, ePub, and manpage.
All the documentation is contained in the main Salt repository. Speaking broadly, most of the narrative documentation is contained within the https://github.com/saltstack/salt/blob/master/doc subdirectory and most of the reference and API documentation is written inline with Salt's Python code and extracted using a Sphinx extension.
Style The Salt project recommends the IEEE style guide as a general reference for writing guidelines. Those guidelines are not strictly enforced but rather serve as an excellent resource for technical writing questions. The NCBI style guide is another very approachable resource.
Point-of-view Use third-person perspective and avoid "I", "we", "you" forms of address. Identify the addressee specifically e.g., "users should", "the compiler does", etc.
Active voice Use active voice and present-tense. Avoid filler words.
Title capitalization Document titles and section titles within a page should follow normal sentence capitalization rules. Words that are capitalized as part of a regular sentence should be capitalized in a title and otherwise left as lowercase. Punctuation can be omitted unless it aids the intent of the title (e.g., exclamation points or question marks).
This is a main heading ======================
This is an exciting sub-heading! --------------------------------
Serial Commas According to Wikipedia: In English punctuation, a serial comma or series comma (also called Oxford comma and Harvard comma) is a comma placed immediately before the coordinating conjunction (usually "and", "or", or "nor") in a series of three or more terms. For example, a list of three countries might be punctuated either as "France, Italy, and Spain" (with the serial comma), or as "France, Italy and Spain" (without the serial comma)."
When writing a list that includes three or more items, the serial comma should always be used.
Documenting modules Documentation for Salt's various module types is inline in the code. During the documentation build process it is extracted and formatted into the final HTML, PDF, etc format.