neulab/tldr huggingface dataset:
Environments The Salt file server defaults to the mandatory base environment. This environment MUST be defined and is used to download files when no environment is specified.
Environments allow for files and sls data to be logically separated, but environments are not isolated from each other. This allows for logical isolation of environments by the engineer using Salt, but also allows for information to be used in multiple environments.
Directory Overlay The environment setting is a list of directories to publish files from. These directories are searched in order to find the specified file and the first file found is returned.
This means that directory data is prioritized based on the order in which they are listed. In the case of this file_roots configuration:
file_roots: base: - /srv/salt/base - /srv/salt/failover
If a file's URI is salt://httpd/httpd.conf, it will first search for the file at /srv/salt/base/httpd/httpd.conf. If the file is found there it will be returned. If the file is not found there, then /srv/salt/failover/httpd/httpd.conf will be used for the source.
This allows for directories to be overlaid and prioritized based on the order they are defined in the configuration.
It is also possible to have file_roots which supports multiple environments:
file_roots: base: - /srv/salt/base dev: - /srv/salt/dev - /srv/salt/base prod: - /srv/salt/prod - /srv/salt/base
This example ensures that each environment will check the associated environment directory for files first. If a file is not found in the appropriate directory, the system will default to using the base directory.
Local File Server New in version 0.9.8.
The file server can be rerouted to run from the minion. This is primarily to enable running Salt states without a Salt master. To use the local file server interface, copy the file server data to the minion and set the file_roots option on the minion to point to the directories copied from the master.  Once the minion file_roots option has been set, change the file_client option to local to make sure that the local file server interface is used.
The cp Module The cp module is the home of minion side file server operations. The cp module is used by the Salt state system, salt-cp, and can be used to distribute files presented by the Salt file server.
Escaping Special Characters The salt:// url format can potentially contain a query string, for example salt://dir/file.txt?saltenv=base. You can prevent the fileclient/fileserver from interpreting ? as the initial token of a query string by referencing the file with salt://| rather than salt://.
/etc/marathon/conf/?checkpoint: file.managed: - source: salt://|hw/config/?checkpoint - makedirs: True
Environments Since the file server is made to work with the Salt state system, it supports environments. The environments are defined in the master config file and when referencing an environment the file specified will be based on the root directory of the environment.
get_file The cp.get_file function can be used on the minion to download a file from the master, the syntax looks like this:
salt '*' cp.get_file salt://vimrc /etc/vimrc
This will instruct all Salt minions to download the vimrc file and copy it to /etc/vimrc
Template rendering can be enabled on both the source and destination file names like so:
salt '*' cp.get_file "salt://{{grains.os}}/vimrc" /etc/vimrc template=jinja
This example would instruct all Salt minions to download the vimrc from a directory with the same name as their OS grain and copy it to /etc/vimrc
For larger files, the cp.get_file module also supports gzip compression. Because gzip is CPU-intensive, this should only be used in scenarios where the compression ratio is very high (e.g. pretty-printed JSON or YAML files).
To use compression, use the gzip named argument. Valid values are integers from 1 to 9, where 1 is the lightest compression and 9 the heaviest. In other words, 1 uses the least CPU on the master (and minion), while 9 uses the most.
salt '*' cp.get_file salt://vimrc /etc/vimrc gzip=5
Finally, note that by default cp.get_file does not create new destination directories if they do not exist.  To change this, use the makedirs argument:
salt '*' cp.get_file salt://vimrc /etc/vim/vimrc makedirs=True
In this example, /etc/vim/ would be created if it didn't already exist.
get_dir The cp.get_dir function can be used on the minion to download an entire directory from the master.  The syntax is very similar to get_file:
salt '*' cp.get_dir salt://etc/apache2 /etc
cp.get_dir supports template rendering and gzip compression arguments just like get_file:
salt '*' cp.get_dir salt://etc/{{pillar.webserver}} /etc gzip=5 template=jinja
File Server Client Instance A client instance is available which allows for modules and applications to be written which make use of the Salt file server.
The file server uses the same authentication and encryption used by the rest of the Salt system for network communication.
fileclient Module The salt/fileclient.py module is used to set up the communication from the minion to the master. When creating a client instance using the fileclient module, the minion configuration needs to be passed in. When using the fileclient module from within a minion module the built in __opts__ data can be passed:
import salt.minion import salt.fileclient
def get_file(path, dest, saltenv="base"): """ Used to get a single file from the Salt master
CLI Example: salt '*' cp.get_file salt://vimrc /etc/vimrc """ # Get the fileclient object client = salt.fileclient.get_file_client(__opts__) # Call get_file return client.get_file(path, dest, False, saltenv)
Creating a fileclient instance outside of a minion module where the __opts__ data is not available, it needs to be generated:
import salt.fileclient import salt.config
def get_file(path, dest, saltenv="base"): """ Used to get a single file from the Salt master """ # Get the configuration data opts = salt.config.minion_config("/etc/salt/minion") # Get the fileclient object client = salt.fileclient.get_file_client(opts) # Call get_file return client.get_file(path, dest, False, saltenv)
Git Fileserver Backend Walkthrough NOTE: This walkthrough assumes basic knowledge of Salt. To get up to speed, check out the Salt Walkthrough.
The gitfs backend allows Salt to serve files from git repositories. It can be enabled by adding git to the fileserver_backend list, and configuring one or more repositories in gitfs_remotes.
Branches and tags become Salt fileserver environments.
NOTE: Branching and tagging can result in a lot of potentially-conflicting top files, for this reason it may be useful to set top_file_merging_strategy to same in the minions' config files if the top files are being managed in a GitFS repo.
Installing Dependencies Both pygit2 and GitPython are supported Python interfaces to git. If compatible versions of both are installed, pygit2 will be preferred. In these cases, GitPython can be forced using the gitfs_provider parameter in the master config file.
NOTE: It is recommended to always run the most recent version of any the below dependencies. Certain features of GitFS may not be available without the most recent version of the chosen library.
pygit2 The minimum supported version of pygit2 is 0.20.3. Availability for this version of pygit2 is still limited, though the SaltStack team is working to get compatible versions available for as many platforms as possible.
For the Fedora/EPEL versions which have a new enough version packaged, the following command would be used to install pygit2:
# yum install python-pygit2
Provided a valid version is packaged for Debian/Ubuntu (which is not currently the case), the package name would be the same, and the following command would be used to install it:
# apt-get install python-pygit2
If pygit2 is not packaged for the platform on which the Master is running, the pygit2 website has installation instructions here. Keep in mind however that following these instructions will install libgit2 and pygit2 without system packages. Additionally, keep in mind that SSH authentication in pygit2 requires libssh2 (not libssh) development libraries to be present before libgit2 is built. On some Debian-based distros pkg-config is also required to link libgit2 with libssh2.
NOTE: If you are receiving the error "Unsupported URL Protocol" in the Salt Master log when making a connection using SSH, review the libssh2 details listed above.
Additionally, version 0.21.0 of pygit2 introduced a dependency on python-cffi, which in turn depends on newer releases of libffi. Upgrading libffi is not advisable as several other applications depend on it, so on older LTS linux releases pygit2 0.20.3 and libgit2 0.20.0 is the recommended combination.
WARNING: pygit2 is actively developed and frequently makes non-backwards-compatible API changes, even in minor releases.  It is not uncommon for pygit2 upgrades to result in errors in Salt. Please take care when upgrading pygit2, and pay close attention to the changelog, keeping an eye out for API changes. Errors can be reported on the SaltStack issue tracker.
RedHat Pygit2 Issues The release of RedHat/CentOS 7.3 upgraded both python-cffi and http-parser, both of which are dependencies for pygit2/libgit2. Both pygit2 and libgit2 packages (which are from the EPEL repository) should be upgraded to the most recent versions, at least to 0.24.2.
The below errors will show up in the master log if an incompatible python-pygit2 package is installed:
2017-02-10 09:07:34,892 [salt.utils.gitfs ][ERROR ][11211] Import pygit2 failed: CompileError: command 'gcc' failed with exit status 1 2017-02-10 09:07:34,907 [salt.utils.gitfs ][ERROR ][11211] gitfs is configured but could not be loaded, are pygit2 and libgit2 installed? 2017-02-10 09:07:34,907 [salt.utils.gitfs ][CRITICAL][11211] No suitable gitfs provider module is installed. 2017-02-10 09:07:34,912 [salt.master ][CRITICAL][11211] Master failed pre flight checks, exiting
The below errors will show up in the master log if an incompatible libgit2 package is installed:
2017-02-15 18:04:45,211 [salt.utils.gitfs ][ERROR   ][6211] Error occurred fetching gitfs remote 'https://foo.com/bar.git': No Content-Type header in response
A restart of the salt-master daemon and gitfs cache directory clean up may be required to allow http(s) repositories to continue to be fetched.
GitPython GitPython 0.3.0 or newer is required to use GitPython for gitfs. For RHEL-based Linux distros, a compatible version is available in EPEL, and can be easily installed on the master using yum:
# yum install GitPython
Ubuntu 14.04 LTS and Debian Wheezy (7.x) also have a compatible version packaged:
# apt-get install python-git
GitPython requires the git CLI utility to work. If installed from a system package, then git should already be installed, but if installed via pip then it may still be necessary to install git separately. For MacOS users, GitPython comes bundled in with the Salt installer, but git must still be installed for it to work properly. Git can be installed in several ways, including by installing XCode.
WARNING: GitPython advises against the use of its library for long-running processes (such as a salt-master or salt-minion). Please see their warning on potential leaks of system resources: https://github.com/gitpython-developers/GitPython#leakage-of-system-resources.
WARNING: Keep in mind that if GitPython has been previously installed on the master using pip (even if it was subsequently uninstalled), then it may still exist in the build cache (typically /tmp/pip-build-root/GitPython) if the cache is not cleared after installation. The package in the build cache will override any requirement specifiers, so if you try upgrading to version 0.3.2.RC1 by running pip install 'GitPython==0.3.2.RC1' then it will ignore this and simply install the version from the cache directory.  Therefore, it may be necessary to delete the GitPython directory from the build cache in order to ensure that the specified version is installed.
WARNING: GitPython 2.0.9 and newer is not compatible with Python 2.6. If installing GitPython using pip on a machine running Python 2.6, make sure that a version earlier than 2.0.9 is installed. This can be done on the CLI by running pip install 'GitPython<2.0.9', or in a pip.installed state using the following SLS:
GitPython: pip.installed: - name: 'GitPython < 2.0.9'
Simple Configuration To use the gitfs backend, only two configuration changes are required on the master:
1. Include gitfs in the fileserver_backend list in the master config file:
fileserver_backend: - gitfs
NOTE: git also works here. Prior to the 2018.3.0 release, only git would work.
2. Specify one or more git://, https://, file://, or ssh:// URLs in gitfs_remotes to configure which repositories to cache and search for requested files:
gitfs_remotes: - https://github.com/saltstack-formulas/salt-formula.git
SSH remotes can also be configured using scp-like syntax:
gitfs_remotes: - git@github.com:user/repo.git - ssh://user@domain.tld/path/to/repo.git
Information on how to authenticate to SSH remotes can be found here.
3. Restart the master to load the new configuration.
NOTE: In a master/minion setup, files from a gitfs remote are cached once by the master, so minions do not need direct access to the git repository.
Multiple Remotes The gitfs_remotes option accepts an ordered list of git remotes to cache and search, in listed order, for requested files.
A simple scenario illustrates this cascading lookup behavior:
If the gitfs_remotes option specifies three remotes:
gitfs_remotes: - git://github.com/example/first.git - https://github.com/example/second.git - file:///root/third
And each repository contains some files:
first.git: top.sls edit/vim.sls edit/vimrc nginx/init.sls
second.git: edit/dev_vimrc haproxy/init.sls
third: haproxy/haproxy.conf edit/dev_vimrc
Salt will attempt to lookup the requested file from each gitfs remote repository in the order in which they are defined in the configuration. The git://github.com/example/first.git remote will be searched first.  If the requested file is found, then it is served and no further searching is executed. For example:
• A request for the file salt://haproxy/init.sls will be served from the https://github.com/example/second.git git repo.
• A request for the file salt://haproxy/haproxy.conf will be served from the file:///root/third repo.
NOTE: This example is purposefully contrived to illustrate the behavior of the gitfs backend. This example should not be read as a recommended way to lay out files and git repos.
The file:// prefix denotes a git repository in a local directory. However, it will still use the given file:// URL as a remote, rather than copying the git repo to the salt cache.  This means that any refs you want accessible must exist as local refs in the specified repo.
WARNING: Salt versions prior to 2014.1.0 are not tolerant of changing the order of remotes or modifying the URI of existing remotes. In those versions, when modifying remotes it is a good idea to remove the gitfs cache directory (/var/cache/salt/master/gitfs) before restarting the salt-master service.
Per-remote Configuration Parameters New in version 2014.7.0.
The following master config parameters are global (that is, they apply to all configured gitfs remotes):
• gitfs_base
• gitfs_root
• gitfs_ssl_verify
• gitfs_mountpoint (new in 2014.7.0)
• gitfs_user (pygit2 only, new in 2014.7.0)
• gitfs_password (pygit2 only, new in 2014.7.0)
• gitfs_insecure_auth (pygit2 only, new in 2014.7.0)
• gitfs_pubkey (pygit2 only, new in 2014.7.0)
• gitfs_privkey (pygit2 only, new in 2014.7.0)
• gitfs_passphrase (pygit2 only, new in 2014.7.0)
• gitfs_refspecs (new in 2017.7.0)
• gitfs_disable_saltenv_mapping (new in 2018.3.0)
• gitfs_ref_types (new in 2018.3.0)
• gitfs_update_interval (new in 2018.3.0)
These parameters can now be overridden on a per-remote basis. This allows for a tremendous amount of customization. Here's some example usage:
gitfs_provider: pygit2 gitfs_base: develop
gitfs_remotes: - https://foo.com/foo.git - https://foo.com/bar.git: - root: salt - mountpoint: salt://bar - base: salt-base - ssl_verify: False - update_interval: 120 - https://foo.com/bar.git: - name: second_bar_repo - root: other/salt - mountpoint: salt://other/bar - base: salt-base - ref_types: - branch - http://foo.com/baz.git: - root: salt/states - user: joe - password: mysupersecretpassword - insecure_auth: True - disable_saltenv_mapping: True - saltenv: - foo: - ref: foo - http://foo.com/quux.git: - all_saltenvs: master
IMPORTANT: There are two important distinctions which should be noted for per-remote configuration:
1. The URL of a remote which has per-remote configuration must be suffixed with a colon.
2. Per-remote configuration parameters are named like the global versions, with the gitfs_ removed from the beginning. The exception being the name, saltenv, and all_saltenvs parameters, which are only available to per-remote configurations.
The all_saltenvs parameter is new in the 2018.3.0 release.
In the example configuration above, the following is true:
1. The first and fourth gitfs remotes will use the develop branch/tag as the base environment, while the second and third will use the salt-base branch/tag as the base environment.
2. The first remote will serve all files in the repository. The second remote will only serve files from the salt directory (and its subdirectories). The third remote will only server files from the other/salt directory (and its subdirectories), while the fourth remote will only serve files from the salt/states directory (and its subdirectories).
3. The third remote will only serve files from branches, and not from tags or SHAs.
4. The fourth remote will only have two saltenvs available: base (pointed at develop), and foo (pointed at foo).
5. The first and fourth remotes will have files located under the root of the Salt fileserver namespace (salt://). The files from the second remote will be located under salt://bar, while the files from the third remote will be located under salt://other/bar.
6. The second and third remotes reference the same repository and unique names need to be declared for duplicate gitfs remotes.
7. The fourth remote overrides the default behavior of not authenticating to insecure (non-HTTPS) remotes.
8. Because all_saltenvs is configured for the fifth remote, files from the branch/tag master will appear in every fileserver environment.
NOTE: The use of http:// (instead of https://) is permitted here only because authentication is not being used. Otherwise, the insecure_auth parameter must be used (as in the fourth remote) to force Salt to authenticate to an http:// remote.
9. The first remote will wait 120 seconds between updates instead of 60.
Per-Saltenv Configuration Parameters New in version 2016.11.0.
For more granular control, Salt allows the following three things to be overridden for individual saltenvs within a given repo:
• The mountpoint
• The root
• The branch/tag to be used for a given saltenv
gitfs_root: salt
gitfs_saltenv: - dev: - mountpoint: salt://gitfs-dev - ref: develop
gitfs_remotes: - https://foo.com/bar.git: - saltenv: - staging: - ref: qa - mountpoint: salt://bar-staging - dev: - ref: development - https://foo.com/baz.git: - saltenv: - staging: - mountpoint: salt://baz-staging
Given the above configuration, the following is true:
1. For all gitfs remotes, files for the dev saltenv will be located under salt://gitfs-dev.
2. For the dev saltenv, files from the first remote will be sourced from the development branch, while files from the second remote will be sourced from the develop branch.
3. For the staging saltenv, files from the first remote will be located under salt://bar-staging, while files from the second remote will be located under salt://baz-staging.
4. For all gitfs remotes, and in all saltenvs, files will be served from the salt directory (and its subdirectories).
Custom Refspecs New in version 2017.7.0.
GitFS will by default fetch remote branches and tags. However, sometimes it can be useful to fetch custom refs (such as those created for GitHub pull requests). To change the refspecs GitFS fetches, use the gitfs_refspecs config option:
In the above example, in addition to fetching remote branches and tags, GitHub's custom refs for pull requests and merged pull requests will also be fetched. These special head refs represent the head of the branch which is requesting to be merged, and the merge refs represent the result of the base branch after the merge.
IMPORTANT: When using custom refspecs, the destination of the fetched refs must be under refs/remotes/origin/, preferably in a subdirectory like in the example above. These custom refspecs will map as environment names using their relative path underneath refs/remotes/origin/. For example, assuming the configuration above, the head branch for pull request 12345 would map to fileserver environment pr/12345 (slash included).
Refspecs can be configured on a per-remote basis. For example, the below configuration would only alter the default refspecs for the second GitFS remote. The first remote would only fetch branches and tags (the default).
gitfs_remotes: - https://domain.tld/foo.git - https://domain.tld/bar.git: - refspecs: - '+refs/heads/*:refs/remotes/origin/*' - '+refs/tags/*:refs/tags/*' - '+refs/pull/*/head:refs/remotes/origin/pr/*' - '+refs/pull/*/merge:refs/remotes/origin/merge/*'
Global Remotes New in version 2018.3.0: for all_saltenvs, 3001 for fallback
The all_saltenvs per-remote configuration parameter overrides the logic Salt uses to map branches/tags to fileserver environments (i.e. saltenvs). This allows a single branch/tag to appear in all GitFS saltenvs.
NOTE: all_saltenvs only works within GitFS. That is, files in a branch configured using all_saltenvs will not show up in a fileserver environment defined via some other fileserver backend (e.g. file_roots).
The fallback global or per-remote configuration can also be used.
This is very useful in particular when working with salt formulas. Prior to the addition of this feature, it was necessary to push a branch/tag to the remote repo for each saltenv in which that formula was to be used. If the formula needed to be updated, this update would need to be reflected in all of the other branches/tags. This is both inconvenient and not scalable.
With all_saltenvs, it is now possible to define your formula once, in a single branch.
gitfs_remotes: - http://foo.com/quux.git: - all_saltenvs: anything
If you want to also test working branches of the formula repository, use fallback:
gitfs_remotes: - http://foo.com/quux.git: - fallback: anything
Update Intervals Prior to the 2018.3.0 release, GitFS would update its fileserver backends as part of a dedicated "maintenance" process, in which various routine maintenance tasks were performed. This tied the update interval to the loop_interval config option, and also forced all fileservers to update at the same interval.
Now it is possible to make GitFS update at its own interval, using gitfs_update_interval:
gitfs_update_interval: 180
gitfs_remotes: - https://foo.com/foo.git - https://foo.com/bar.git: - update_interval: 120
Using the above configuration, the first remote would update every three minutes, while the second remote would update every two minutes.
Configuration Order of Precedence The order of precedence for GitFS configuration is as follows (each level overrides all levels below it):
1. Per-saltenv configuration (defined under a per-remote saltenv param)
gitfs_remotes: - https://foo.com/bar.git: - saltenv: - dev: - mountpoint: salt://bar
2. Global per-saltenv configuration (defined in gitfs_saltenv)
gitfs_saltenv: - dev: - mountpoint: salt://bar
3. Per-remote configuration parameter
gitfs_remotes: - https://foo.com/bar.git: - mountpoint: salt://bar
4. Global configuration parameter
gitfs_mountpoint: salt://bar
NOTE: The one exception to the above is when all_saltenvs is used. This value overrides all logic for mapping branches/tags to fileserver environments. So, even if gitfs_saltenv is used to globally override the mapping for a given saltenv, all_saltenvs would take precedence for any remote which uses it.
It's important to note however that any root and mountpoint values configured in gitfs_saltenv (or per-saltenv configuration) would be unaffected by this.
Serving from a Subdirectory The gitfs_root parameter allows files to be served from a subdirectory within the repository. This allows for only part of a repository to be exposed to the Salt fileserver.
Assume the below layout:
.gitignore README.txt foo/ foo/bar/ foo/bar/one.txt foo/bar/two.txt foo/bar/three.txt foo/baz/ foo/baz/top.sls foo/baz/edit/vim.sls foo/baz/edit/vimrc foo/baz/nginx/init.sls
The below configuration would serve only the files under foo/baz, ignoring the other files in the repository:
gitfs_remotes: - git://mydomain.com/stuff.git
gitfs_root: foo/baz
The root can also be configured on a per-remote basis.
Mountpoints New in version 2014.7.0.
The gitfs_mountpoint parameter will prepend the specified path to the files served from gitfs. This allows an existing repository to be used, rather than needing to reorganize a repository or design it around the layout of the Salt fileserver.
Before the addition of this feature, if a file being served up via gitfs was deeply nested within the root directory (for example, salt://webapps/foo/files/foo.conf, it would be necessary to ensure that the file was properly located in the remote repository, and that all of the parent directories were present (for example, the directories webapps/foo/files/ would need to exist at the root of the repository).
The below example would allow for a file foo.conf at the root of the repository to be served up from the Salt fileserver path salt://webapps/foo/files/foo.conf.
gitfs_remotes: - https://mydomain.com/stuff.git
gitfs_mountpoint: salt://webapps/foo/files
Mountpoints can also be configured on a per-remote basis.
Using gitfs in Masterless Mode Since 2014.7.0, gitfs can be used in masterless mode. To do so, simply add the gitfs configuration parameters (and set fileserver_backend) in the _minion_ config file instead of the master config file.
Using gitfs Alongside Other Backends Sometimes it may make sense to use multiple backends; for instance, if sls files are stored in git but larger files are stored directly on the master.
The cascading lookup logic used for multiple remotes is also used with multiple backends. If the fileserver_backend option contains multiple backends:
Then the roots backend (the default backend of files in /srv/salt) will be searched first for the requested file; then, if it is not found on the master, each configured git remote will be searched.
NOTE: This can be used together with file_roots accepting __env__ as a catch-all environment, since 2018.3.5 and 2019.2.1:
file_roots: base: - /srv/salt __env__: - /srv/salt
Branches, Environments, and Top Files When using the GitFS backend, branches, and tags will be mapped to environments using the branch/tag name as an identifier.
There is one exception to this rule: the master branch is implicitly mapped to the base environment.
So, for a typical base, qa, dev setup, the following branches could be used:
master qa dev
top.sls files from different branches will be merged into one at runtime. Since this can lead to overly complex configurations, the recommended setup is to have a separate repository, containing only the top.sls file with just one single master branch.
To map a branch other than master as the base environment, use the gitfs_base parameter.
gitfs_base: salt-base
The base can also be configured on a per-remote basis.
Environment Whitelist/Blacklist New in version 2014.7.0.
The gitfs_saltenv_whitelist and gitfs_saltenv_blacklist parameters allow for greater control over which branches/tags are exposed as fileserver environments. Exact matches, globs, and regular expressions are supported, and are evaluated in that order.  If using a regular expression, ^ and $ must be omitted, and the expression must match the entire branch/tag.
NOTE: v1.*, in this example, will match as both a glob and a regular expression (though it will have been matched as a glob, since globs are evaluated before regular expressions).
The behavior of the blacklist/whitelist will differ depending on which combination of the two options is used:
• If only gitfs_saltenv_whitelist is used, then only branches/tags which match the whitelist will be available as environments
• If only gitfs_saltenv_blacklist is used, then the branches/tags which match the blacklist will not be available as environments
• If both are used, then the branches/tags which match the whitelist, but do not match the blacklist, will be available as environments.
Authentication pygit2 New in version 2014.7.0.
Both HTTPS and SSH authentication are supported as of version 0.20.3, which is the earliest version of pygit2 supported by Salt for gitfs.
NOTE: The examples below make use of per-remote configuration parameters, a feature new to Salt 2014.7.0. More information on these can be found here.
HTTPS For HTTPS repositories which require authentication, the username and password can be provided like so:
gitfs_remotes: - https://domain.tld/myrepo.git: - user: git - password: mypassword
If the repository is served over HTTP instead of HTTPS, then Salt will by default refuse to authenticate to it. This behavior can be overridden by adding an insecure_auth parameter:
gitfs_remotes: - http://domain.tld/insecure_repo.git: - user: git - password: mypassword - insecure_auth: True
SSH SSH repositories can be configured using the ssh:// protocol designation, or using scp-like syntax. So, the following two configurations are equivalent:
• ssh://git@github.com/user/repo.git
• git@github.com:user/repo.git
Both gitfs_pubkey and gitfs_privkey (or their per-remote counterparts) must be configured in order to authenticate to SSH-based repos. If the private key is protected with a passphrase, it can be configured using gitfs_passphrase (or simply passphrase if being configured per-remote). For example:
gitfs_remotes: - git@github.com:user/repo.git: - pubkey: /root/.ssh/id_rsa.pub - privkey: /root/.ssh/id_rsa - passphrase: myawesomepassphrase
Finally, the SSH host key must be added to the known_hosts file.
NOTE: There is a known issue with public-key SSH authentication to Microsoft Visual Studio (VSTS) with pygit2. This is due to a bug or lack of support for VSTS in older libssh2 releases. Known working releases include libssh2 1.7.0 and later, and known incompatible releases include 1.5.0 and older.  At the time of this writing, 1.6.0 has not been tested.
Since upgrading libssh2 would require rebuilding many other packages (curl, etc.), followed by a rebuild of libgit2 and a reinstall of pygit2, an easier workaround for systems with older libssh2 is to use GitPython with a passphraseless key for authentication.
GitPython HTTPS For HTTPS repositories which require authentication, the username and password can be configured in one of two ways. The first way is to include them in the URL using the format https://<user>:<password>@<url>, like so:
gitfs_remotes: - https://git:mypassword@domain.tld/myrepo.git
The other way would be to configure the authentication in ~/.netrc:
machine domain.tld login git password mypassword
gitfs_remotes: - http://git:mypassword@domain.tld/insecure_repo.git: - insecure_auth: True
SSH Only passphrase-less SSH public key authentication is supported using GitPython. The auth parameters (pubkey, privkey, etc.) shown in the pygit2 authentication examples above do not work with GitPython.
gitfs_remotes: - ssh://git@github.com/example/salt-states.git
Since GitPython wraps the git CLI, the private key must be located in ~/.ssh/id_rsa for the user under which the Master is running, and should have permissions of 0600. Also, in the absence of a user in the repo URL, GitPython will (just as SSH does) attempt to login as the current user (in other words, the user under which the Master is running, usually root).
If a key needs to be used, then ~/.ssh/config can be configured to use the desired key. Information on how to do this can be found by viewing the manpage for ssh_config. Here's an example entry which can be added to the ~/.ssh/config to use an alternate key for gitfs:
Host github.com IdentityFile /root/.ssh/id_rsa_gitfs
The Host parameter should be a hostname (or hostname glob) that matches the domain name of the git repository.
It is also necessary to add the SSH host key to the known_hosts file. The exception to this would be if strict host key checking is disabled, which can be done by adding StrictHostKeyChecking no to the entry in ~/.ssh/config
Host github.com IdentityFile /root/.ssh/id_rsa_gitfs StrictHostKeyChecking no
However, this is generally regarded as insecure, and is not recommended.
Adding the SSH Host Key to the known_hosts File To use SSH authentication, it is necessary to have the remote repository's SSH host key in the ~/.ssh/known_hosts file. If the master is also a minion, this can be done using the ssh.set_known_host function:
# salt mymaster ssh.set_known_host user=root hostname=github.com mymaster: ---------- new: ---------- enc: ssh-rsa fingerprint: 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48 hostname: |1|OiefWWqOD4kwO3BhoIGa0loR5AA=|BIXVtmcTbPER+68HvXmceodDcfI= key: AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== old: None status: updated
If not, then the easiest way to add the key is to su to the user (usually root) under which the salt-master runs and attempt to login to the server via SSH:
$ su - Password: # ssh github.com The authenticity of host 'github.com (192.30.252.128)' can't be established. RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'github.com,192.30.252.128' (RSA) to the list of known hosts. Permission denied (publickey).
It doesn't matter if the login was successful, as answering yes will write the fingerprint to the known_hosts file.
Verifying the Fingerprint To verify that the correct fingerprint was added, it is a good idea to look it up. One way to do this is to use nmap:
$ nmap -p 22 github.com --script ssh-hostkey
Starting Nmap 5.51 ( http://nmap.org ) at 2014-08-18 17:47 CDT Nmap scan report for github.com (192.30.252.129) Host is up (0.17s latency). Not shown: 996 filtered ports PORT     STATE SERVICE 22/tcp   open  ssh | ssh-hostkey: 1024 ad:1c:08:a4:40:e3:6f:9c:f5:66:26:5d:4b:33:5d:8c (DSA) |_2048 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48 (RSA) 80/tcp   open  http 443/tcp  open  https 9418/tcp open  git
Nmap done: 1 IP address (1 host up) scanned in 28.78 seconds
Another way is to check one's own known_hosts file, using this one-liner:
$ ssh-keygen -l -f /dev/stdin <<<`ssh-keyscan github.com 2>/dev/null` | awk '{print $2}' 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48
WARNING: AWS tracks usage of nmap and may flag it as abuse. On AWS hosts, the ssh-keygen method is recommended for host key verification.
NOTE: As of OpenSSH 6.8 the SSH fingerprint is now shown as a base64-encoded SHA256 checksum of the host key. So, instead of the fingerprint looking like 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48, it would look like SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Refreshing gitfs Upon Push By default, Salt updates the remote fileserver backends every 60 seconds. However, if it is desirable to refresh quicker than that, the Reactor System can be used to signal the master to update the fileserver on each push, provided that the git server is also a Salt minion. There are three steps to this process:
1. On the master, create a file /srv/reactor/update_fileserver.sls, with the following contents:
update_fileserver: runner.fileserver.update
2. Add the following reactor configuration to the master config file:
reactor: - 'salt/fileserver/gitfs/update': - /srv/reactor/update_fileserver.sls
3. On the git server, add a post-receive hook
a. If the user executing git push is the same as the minion user, use the following hook:
#!/usr/bin/env sh salt-call event.fire_master update salt/fileserver/gitfs/update
b. To enable other git users to run the hook after a push, use sudo in the hook script:
#!/usr/bin/env sh sudo -u root salt-call event.fire_master update salt/fileserver/gitfs/update
4. If using sudo in the git hook (above), the policy must be changed to permit all users to fire the event.  Add the following policy to the sudoers file on the git server.
Cmnd_Alias SALT_GIT_HOOK = /bin/salt-call event.fire_master update salt/fileserver/gitfs/update Defaults!SALT_GIT_HOOK !requiretty ALL ALL=(root) NOPASSWD: SALT_GIT_HOOK
The update argument right after event.fire_master in this example can really be anything, as it represents the data being passed in the event, and the passed data is ignored by this reactor.
Similarly, the tag name salt/fileserver/gitfs/update can be replaced by anything, so long as the usage is consistent.
The root user name in the hook script and sudo policy should be changed to match the user under which the minion is running.
Using Git as an External Pillar Source The git external pillar (a.k.a. git_pillar) has been rewritten for the 2015.8.0 release. This rewrite brings with it pygit2 support (allowing for access to authenticated repositories), as well as more granular support for per-remote configuration. This configuration schema is detailed here.
Why aren't my custom modules/states/etc. syncing to my Minions? In versions 0.16.3 and older, when using the git fileserver backend, certain versions of GitPython may generate errors when fetching, which Salt fails to catch. While not fatal to the fetch process, these interrupt the fileserver update that takes place before custom types are synced, and thus interrupt the sync itself. Try disabling the git fileserver backend in the master config, restarting the master, and attempting the sync again.
This issue is worked around in Salt 0.16.4 and newer.
MinionFS Backend Walkthrough New in version 2014.1.0.
NOTE: This walkthrough assumes basic knowledge of Salt and cp.push. To get up to speed, check out the Salt Walkthrough.
Sometimes it is desirable to deploy a file located on one minion to one or more other minions. This is supported in Salt, and can be accomplished in two parts:
1. Minion support for pushing files to the master (using cp.push)
2. The minionfs fileserver backend
This walkthrough will show how to use both of these features.
Enabling File Push To set the master to accept files pushed from minions, the file_recv option in the master config file must be set to True (the default is False).
file_recv: True
NOTE: This change requires a restart of the salt-master service.
Pushing Files Once this has been done, files can be pushed to the master using the cp.push function:
salt 'minion-id' cp.push /path/to/the/file
This command will store the file in a subdirectory named minions under the master's cachedir. On most masters, this path will be /var/cache/salt/master/minions. Within this directory will be one directory for each minion which has pushed a file to the master, and underneath that the full path to the file on the minion. So, for example, if a minion with an ID of dev1 pushed a file /var/log/myapp.log to the master, it would be saved to /var/cache/salt/master/minions/dev1/var/log/myapp.log.
Serving Pushed Files Using MinionFS While it is certainly possible to add /var/cache/salt/master/minions to the master's file_roots and serve these files, it may only be desirable to expose files pushed from certain minions. Adding /var/cache/salt/master/minions/<minion-id> for each minion that needs to be exposed can be cumbersome and prone to errors.
Enter minionfs. This fileserver backend will make files pushed using cp.push available to the Salt fileserver, and provides an easy mechanism to restrict which minions' pushed files are made available.
Simple Configuration To use the minionfs backend, add minionfs to the list of backends in the fileserver_backend configuration option on the master:
fileserver_backend: - roots - minionfs
NOTE: minion also works here. Prior to the 2018.3.0 release, only minion would work.
Also, as described earlier, file_recv: True is needed to enable the master to receive files pushed from minions. As always, changes to the master configuration require a restart of the salt-master service.
Files made available via minionfs are by default located at salt://<minion-id>/path/to/file. Think back to the earlier example, in which dev1 pushed a file /var/log/myapp.log to the master. With minionfs enabled, this file would be addressable in Salt at salt://dev1/var/log/myapp.log.
If many minions have pushed to the master, this will result in many directories in the root of the Salt fileserver. For this reason, it is recommended to use the minionfs_mountpoint config option to organize these files underneath a subdirectory:
minionfs_mountpoint: salt://minionfs
Using the above mountpoint, the file in the example would be located at salt://minionfs/dev1/var/log/myapp.log.
Restricting Certain Minions' Files from Being Available Via MinionFS A whitelist and blacklist can be used to restrict the minions whose pushed files are available via minionfs. These lists can be managed using the minionfs_whitelist and minionfs_blacklist config options. Click the links for both of them for a detailed explanation of how to use them.
A more complex configuration example, which uses both a whitelist and blacklist, can be found below:
minionfs_whitelist: - host04 - web* - 'mail\d+\.domain\.tld'
minionfs_blacklist: - web21
Potential Concerns • There is no access control in place to restrict which minions have access to files served up by minionfs. All minions will have access to these files.
• Unless the minionfs_whitelist and/or minionfs_blacklist config options are used, all minions which push files to the master will have their files made available via minionfs.
Salt Package Manager The Salt Package Manager, or SPM, enables Salt formulas to be packaged to simplify distribution to Salt masters. The design of SPM was influenced by other existing packaging systems including RPM, Yum, and Pacman. [image]
NOTE: The previous diagram shows each SPM component as a different system, but this is not required. You can build packages and host the SPM repo on a single Salt master if you'd like.
Packaging System
The packaging system is used to package the state, pillar, file templates, and other files used by your formula into a single file. After a formula package is created, it is copied to the Repository System where it is made available to Salt masters.
See Building SPM Packages
Repo System
The Repo system stores the SPM package and metadata files and makes them available to Salt masters via http(s), ftp, or file URLs. SPM repositories can be hosted on a Salt Master, a Salt Minion, or on another system.
See Distributing SPM Packages
Salt Master
SPM provides Salt master settings that let you configure the URL of one or more SPM repos. You can then quickly install packages that contain entire formulas to your Salt masters using SPM.
See Installing SPM Packages
Building SPM Packages The first step when using Salt Package Manager is to build packages for each of of the formulas that you want to distribute. Packages can be built on any system where you can install Salt.
Package Build Overview To build a package, all state, pillar, jinja, and file templates used by your formula are assembled into a folder on the build system. These files can be cloned from a Git repository, such as those found at the saltstack-formulas organization on GitHub, or copied directly to the folder.
The following diagram demonstrates a typical formula layout on the build system: [image]
In this example, all formula files are placed in a myapp-formula folder. This is the folder that is targeted by the spm build command when this package is built.
Within this folder, pillar data is placed in a pillar.example file at the root, and all state, jinja, and template files are placed within a subfolder that is named after the application being packaged. State files are typically contained within a subfolder, similar to how state files are organized in the state tree. Any non-pillar files in your package that are not contained in a subfolder are placed at the root of the spm state tree.
Additionally, a FORMULA file is created and placed in the root of the folder. This file contains package metadata that is used by SPM.
Package Installation Overview When building packages, it is useful to know where files are installed on the Salt master. During installation, all files except pillar.example and FORMULA are copied directly to the spm state tree on the Salt master (located at \srv\spm\salt).
If a pillar.example file is present in the root, it is renamed to <formula name>.sls.orig and placed in the pillar_path.  [image]
NOTE: Even though the pillar data file is copied to the pillar root, you still need to manually assign this pillar data to systems using the pillar top file. This file can also be duplicated and renamed so the .orig version is left intact in case you need to restore it later.
Building an SPM Formula Package 1. Assemble formula files in a folder on the build system.
2. Create a FORMULA file and place it in the root of the package folder.
3. Run spm build <folder name>. The package is built and placed in the /srv/spm_build folder.
spm build /path/to/salt-packages-source/myapp-formula
4. Copy the .spm file to a folder on the repository system.
Types of Packages SPM supports different types of packages. The function of each package is denoted by its name. For instance, packages which end in -formula are considered to be Salt States (the most common type of formula). Packages which end in -conf contain configuration which is to be placed in the /etc/salt/ directory. Packages which do not contain one of these names are treated as if they have a -formula name.
formula By default, most files from this type of package live in the /srv/spm/salt/ directory. The exception is the pillar.example file, which will be renamed to <package_name>.sls and placed in the pillar directory (/srv/spm/pillar/ by default).
reactor By default, files from this type of package live in the /srv/spm/reactor/ directory.
conf The files in this type of package are configuration files for Salt, which normally live in the /etc/salt/ directory. Configuration files for packages other than Salt can and should be handled with a Salt State (using a formula type of package).
Technical Information Packages are built using BZ2-compressed tarballs. By default, the package database is stored using the sqlite3 driver (see Loader Modules below).
Support for these are built into Python, and so no external dependencies are needed.
All other files belonging to SPM use YAML, for portability and ease of use and maintainability.
SPM-Specific Loader Modules SPM was designed to behave like traditional package managers, which apply files to the filesystem and store package metadata in a local database. However, because modern infrastructures often extend beyond those use cases, certain parts of SPM have been broken out into their own set of modules.
Package Database By default, the package database is stored using the sqlite3 module. This module was chosen because support for SQLite3 is built into Python itself.
Please see the SPM Development Guide for information on creating new modules for package database management.
Package Files By default, package files are installed using the local module. This module applies files to the local filesystem, on the machine that the package is installed on.
Please see the SPM Development Guide for information on creating new modules for package file management.
Distributing SPM Packages SPM packages can be distributed to Salt masters over HTTP(S), FTP, or through the file system. The SPM repo can be hosted on any system where you can install Salt. Salt is installed so you can run the spm create_repo command when you update or add a package to the repo. SPM repos do not require the salt-master, salt-minion, or any other process running on the system.