neulab/tldr huggingface dataset:
Because this is a monitoring state, it will return extra data to code that expects it. This data will always include text and status. Optionally, headers and dict may also be requested by setting the headers and decode arguments to True, respectively.
Using Salt at scale The focus of this tutorial will be building a Salt infrastructure for handling large numbers of minions. This will include tuning, topology, and best practices.
For how to install the Salt Master please go here: Installing saltstack
NOTE: This tutorial is intended for large installations, although these same settings won't hurt, it may not be worth the complexity to smaller installations.
When used with minions, the term 'many' refers to at least a thousand and 'a few' always means 500.
For simplicity reasons, this tutorial will default to the standard ports used by Salt.
The Master The most common problems on the Salt Master are:
1. too many minions authing at once
2. too many minions re-authing at once
3. too many minions re-connecting at once
4. too many minions returning at once
5. too few resources (CPU/HDD)
The first three are all "thundering herd" problems. To mitigate these issues we must configure the minions to back-off appropriately when the Master is under heavy load.
The fourth is caused by masters with little hardware resources in combination with a possible bug in ZeroMQ. At least that's what it looks like till today (Issue 118651, Issue 5948, Mail thread)
To fully understand each problem, it is important to understand, how Salt works.
Very briefly, the Salt Master offers two services to the minions.
• a job publisher on port 4505
• an open port 4506 to receive the minions returns
All minions are always connected to the publisher on port 4505 and only connect to the open return port 4506 if necessary. On an idle Master, there will only be connections on port 4505.
Too many minions authing When the Minion service is first started up, it will connect to its Master's publisher on port 4505. If too many minions are started at once, this can cause a "thundering herd".  This can be avoided by not starting too many minions at once.
The connection itself usually isn't the culprit, the more likely cause of master-side issues is the authentication that the Minion must do with the Master. If the Master is too heavily loaded to handle the auth request it will time it out. The Minion will then wait acceptance_wait_time to retry. If acceptance_wait_time_max is set then the Minion will increase its wait time by the acceptance_wait_time each subsequent retry until reaching acceptance_wait_time_max.
Too many minions re-authing This is most likely to happen in the testing phase of a Salt deployment, when all Minion keys have already been accepted, but the framework is being tested and parameters are frequently changed in the Salt Master's configuration file(s).
The Salt Master generates a new AES key to encrypt its publications at certain events such as a Master restart or the removal of a Minion key. If you are encountering this problem of too many minions re-authing against the Master, you will need to recalibrate your setup to reduce the rate of events like a Master restart or Minion key removal (salt-key -d).
When the Master generates a new AES key, the minions aren't notified of this but will discover it on the next pub job they receive. When the Minion receives such a job it will then re-auth with the Master. Since Salt does minion-side filtering this means that all the minions will re-auth on the next command published on the master-- causing another "thundering herd". This can be avoided by setting the
in the minions configuration file to a higher value and stagger the amount of re-auth attempts. Increasing this value will of course increase the time it takes until all minions are reachable via Salt commands.
Too many minions re-connecting By default the zmq socket will re-connect every 100ms which for some larger installations may be too quick. This will control how quickly the TCP session is re-established, but has no bearing on the auth load.
To tune the minions sockets reconnect attempts, there are a few values in the sample configuration file (default values)
recon_default: 1000 recon_max: 5000 recon_randomize: True
• recon_default: the default value the socket should use, i.e. 1000. This value is in milliseconds. (1000ms = 1 second)
• recon_max: the max value that the socket should use as a delay before trying to reconnect This value is in milliseconds. (5000ms = 5 seconds)
• recon_randomize: enables randomization between recon_default and recon_max
To tune this values to an existing environment, a few decision have to be made.
1. How long can one wait, before the minions should be online and reachable via Salt?
2. How many reconnects can the Master handle without a syn flood?
These questions can not be answered generally. Their answers depend on the hardware and the administrators requirements.
Here is an example scenario with the goal, to have all minions reconnect within a 60 second time-frame on a Salt Master service restart.
recon_default: 1000 recon_max: 59000 recon_randomize: True
Each Minion will have a randomized reconnect value between 'recon_default' and 'recon_default + recon_max', which in this example means between 1000ms and 60000ms (or between 1 and 60 seconds). The generated random-value will be doubled after each attempt to reconnect (ZeroMQ default behavior).
Lets say the generated random value is 11 seconds (or 11000ms).
reconnect 1: wait 11 seconds reconnect 2: wait 22 seconds reconnect 3: wait 33 seconds reconnect 4: wait 44 seconds reconnect 5: wait 55 seconds reconnect 6: wait time is bigger than 60 seconds (recon_default + recon_max) reconnect 7: wait 11 seconds reconnect 8: wait 22 seconds reconnect 9: wait 33 seconds reconnect x: etc.
With a thousand minions this will mean
1000/60 = ~16
round about 16 connection attempts a second. These values should be altered to values that match your environment. Keep in mind though, that it may grow over time and that more minions might raise the problem again.
Too many minions returning at once This can also happen during the testing phase, if all minions are addressed at once with
$ salt * disk.usage
it may cause thousands of minions trying to return their data to the Salt Master open port 4506. Also causing a flood of syn-flood if the Master can't handle that many returns at once.
This can be easily avoided with Salt's batch mode:
$ salt * disk.usage -b 50
This will only address 50 minions at once while looping through all addressed minions.
Too few resources The masters resources always have to match the environment. There is no way to give good advise without knowing the environment the Master is supposed to run in.  But here are some general tuning tips for different situations:
The Master is CPU bound Salt uses RSA-Key-Pairs on the masters and minions end. Both generate 4096 bit key-pairs on first start. While the key-size for the Master is currently not configurable, the minions keysize can be configured with different key-sizes. For example with a 2048 bit key:
With thousands of decryptions, the amount of time that can be saved on the masters end should not be neglected. See here for reference: Pull Request 9235 how much influence the key-size can have.
Downsizing the Salt Master's key is not that important, because the minions do not encrypt as many messages as the Master does.
In installations with large or with complex pillar files, it is possible for the master to exhibit poor performance as a result of having to render many pillar files at once. This exhibit itself in a number of ways, both as high load on the master and on minions which block on waiting for their pillar to be delivered to them.
To reduce pillar rendering times, it is possible to cache pillars on the master. To do this, see the set of master configuration options which are prefixed with pillar_cache.
If many pillars are encrypted using gpg renderer, it is possible to cache GPG data. To do this, see the set of master configuration options which are prefixed with gpg_cache.
NOTE: Caching pillars or GPG data on the master may introduce security considerations. Be certain to read caveats outlined in the master configuration file to understand how pillar caching may affect a master's ability to protect sensitive data!
The Master is disk IO bound By default, the Master saves every Minion's return for every job in its job-cache. The cache can then be used later, to lookup results for previous jobs. The default directory for this is:
cachedir: /var/cache/salt
and then in the /proc directory.
Each job return for every Minion is saved in a single file. Over time this directory can grow quite large, depending on the number of published jobs. The amount of files and directories will scale with the number of jobs published and the retention time defined by
250 jobs/day * 2000 minions returns = 500,000 files a day
Use and External Job Cache An external job cache allows for job storage to be placed on an external system, such as a database.
• ext_job_cache: this will have the minions store their return data directly into a returner (not sent through the Master)
• master_job_cache (New in 2014.7.0): this will make the Master store the job data using a returner (instead of the local job cache on disk).
If a master has many accepted keys, it may take a long time to publish a job because the master must first determine the matching minions and deliver that information back to the waiting client before the job can be published.
To mitigate this, a key cache may be enabled. This will reduce the load on the master to a single file open instead of thousands or tens of thousands.
This cache is updated by the maintanence process, however, which means that minions with keys that are accepted may not be targeted by the master for up to sixty seconds by default.
To enable the master key cache, set key_cache: 'sched' in the master configuration file.
Disable The Job Cache The job cache is a central component of the Salt Master and many aspects of the Salt Master will not function correctly without a running job cache.
Disabling the job cache is STRONGLY DISCOURAGED and should not be done unless the master is being used to execute routines that require no history or reliable feedback!
The job cache can be disabled:
job_cache: False
How to Convert Jinja Logic to an Execution Module NOTE: This tutorial assumes a basic knowledge of Salt states and specifically experience using the maps.jinja idiom.
This tutorial was written by a salt user who was told "if your maps.jinja is too complicated, write an execution module!". If you are experiencing over-complicated jinja, read on.
The Problem: Jinja Gone Wild It is often said in the Salt community that "Jinja is not a Programming Language".  There's an even older saying known as Maslow's hammer.  It goes something like "if all you have is a hammer, everything looks like a nail".  Jinja is a reliable hammer, and so is the maps.jinja idiom. Unfortunately, it can lead to code that looks like the following.
# storage/maps.yaml
{% import_yaml 'storage/defaults.yaml' as default_settings %} {% set storage = default_settings.storage %} {% do storage.update(salt['grains.filter_by']({ 'Debian': { }, 'RedHat': { } }, merge=salt['pillar.get']('storage:lookup'))) %}
{% if 'VirtualBox' == grains.get('virtual', None) or 'oracle' == grains.get('virtual', None) %} {%   do storage.update({'depot_ip': '192.168.33.81', 'server_ip':  '192.168.33.51'}) %} {% else %} {%   set colo = pillar.get('inventory', {}).get('colo', 'Unknown') %} {%   set servers_list = pillar.get('storage_servers', {}).get(colo, [storage.depot_ip, ]) %} {%   if opts.id.startswith('foo') %} {%     set modulus = servers_list | count %} {%     set integer_id = opts.id | replace('foo', '') | int %} {%     set server_index = integer_id % modulus %} {%   else %} {%     set server_index = 0 %} {%   endif %} {%   do storage.update({'server_ip': servers_list[server_index]}) %} {% endif %}
{% for network, _ in salt.pillar.get('inventory:networks', {}) | dictsort %} {%   do storage.ipsets.hash_net.foo_networks.append(network) %} {% endfor %}
This is an example from the author's salt formulae demonstrating misuse of jinja.  Aside from being difficult to read and maintain, accessing the logic it contains from a non-jinja renderer while probably possible is a significant barrier!
Refactor The first step is to reduce the maps.jinja file to something reasonable. This gives us an idea of what the module we are writing needs to do. There is a lot of logic around selecting a storage server ip.  Let's move that to an execution module.
{% if 'VirtualBox' == grains.get('virtual', None) or 'oracle' == grains.get('virtual', None) %} {%   do storage.update({'depot_ip': '192.168.33.81'}) %} {% endif %}
{% do storage.update({'server_ip': salt['storage.ip']()}) %}
{% for network, _ in salt.pillar.get('inventory:networks', {}) | dictsort %} {%   do storage.ipsets.hash_net.af_networks.append(network) %} {% endfor %}
And then, write the module.  Note how the module encapsulates all of the logic around finding the storage server IP.
# _modules/storage.py #!python
""" Functions related to storage servers. """
def ips(): """ Provide a list of all local storage server IPs.
CLI Example::
salt \* storage.ips """
if __grains__.get("virtual", None) in ["VirtualBox", "oracle"]: return [ "192.168.33.51", ]
colo = __pillar__.get("inventory", {}).get("colo", "Unknown") return __pillar__.get("storage_servers", {}).get(colo, ["unknown"])
def ip(): """ Select and return a local storage server IP.
This loadbalances across storage servers by using the modulus of the client's id number.
:maintainer:    Andrew Hammond <ahammond@anchorfree.com> :maturity:      new :depends:       None :platform:      all
salt \* storage.ip
numerical_suffix = re.compile(r"^.*(\d+)$") servers_list = ips()
m = numerical_suffix.match(__grains__["id"]) if m: modulus = len(servers_list) server_number = int(m.group(1)) server_index = server_number % modulus else: server_index = 0
return servers_list[server_index]
Conclusion That was... surprisingly straight-forward.  Now the logic is available in every renderer, instead of just Jinja.  Best of all, it can be maintained in Python, which is a whole lot easier than Jinja.
Using Apache Libcloud for declarative and procedural multi-cloud orchestration New in version 2018.3.0.
NOTE: This walkthrough assumes basic knowledge of Salt and Salt States. To get up to speed, check out the Salt Walkthrough.
Apache Libcloud is a Python library which hides differences between different cloud provider APIs and allows you to manage different cloud resources through a unified and easy to use API. Apache Libcloud supports over 60 cloud platforms, including Amazon, Microsoft Azure, DigitalOcean, Google Cloud Platform and OpenStack.
Execution and state modules are available for Compute, DNS, Storage and Load Balancer drivers from Apache Libcloud in SaltStack.
libcloud_compute - Compute - services such as OpenStack Nova, Amazon EC2, Microsoft Azure VMs
libcloud_dns - DNS as a Service - services such as Amazon Route 53 and Zerigo
libcloud_loadbalancer - Load Balancers as a Service - services such as Amazon Elastic Load Balancer and GoGrid LoadBalancers
libcloud_storage - Cloud Object Storage and CDN - services such as Amazon S3 and Rackspace CloudFiles, OpenStack Swift
These modules are designed as a way of having a multi-cloud deployment and abstracting simple differences between platform to design a high-availability architecture.
The Apache Libcloud functionality is available through both execution modules and Salt states.
Configuring Drivers Drivers can be configured in the Salt Configuration/Minion settings. All libcloud modules expect a list of "profiles" to be configured with authentication details for each driver.
Each driver will have a string identifier, these can be found in the libcloud.<api>.types.Provider class for each API, https://libcloud.readthedocs.io/en/latest/supported_providers.html
Some drivers require additional parameters, which are documented in the Apache Libcloud documentation. For example, GoDaddy DNS expects "shopper_id", which is the customer ID. These additional parameters can be added to the profile settings and will be passed directly to the driver instantiation method.
libcloud_dns: godaddy: driver: godaddy shopper_id: 90425123 key: AFDDJFGIjDFVNSDIFNASMC secret: FG(#f8vdfgjlkm)
libcloud_storage: google: driver: google_storage key: GOOG4ASDIDFNVIdfnIVW secret: R+qYE9hkfdhv89h4invhdfvird4Pq3an8rnK
You can have multiple profiles for a single driver, for example if you wanted 2 DNS profiles for Amazon Route53, naming them "route53_prod" and "route54_test" would help your administrators distinguish their purpose.
libcloud_dns: route53_prod: driver: route53 key: AFDDJFGIjDFVNSDIFNASMC secret: FG(#f8vdfgjlkm) route53_test: driver: route53 key: AFDDJFGIjdfgdfgdf secret: FG(#f8vdfgjlkm)
Using the execution modules Amongst over 60 clouds that Apache Libcloud supports, you can add profiles to your Salt configuration to access and control these clouds. Each of the libcloud execution modules exposes the common API methods for controlling Compute, DNS, Load Balancers and Object Storage.  To see which functions are supported across specific clouds, see the Libcloud supported methods documentation.
The module documentation explains each of the API methods and how to leverage them.
For example, listing buckets in the Google Storage platform:
$ salt-call libcloud_storage.list_containers google
local: |_ ---------- extra: ---------- creation_date: 2017-01-05T05:44:56.324Z name: anthonypjshaw
The Apache Libcloud storage module can be used to synchronize files between multiple storage clouds, such as Google Storage, S3 and OpenStack Swift
salt '*' libcloud_storage.download_object DeploymentTools test.sh /tmp/test.sh google_storage
Using the state modules For each configured profile, the assets available in the API (e.g. storage objects, containers, DNS records and load balancers) can be deployed via Salt's state system.
The state module documentation explains the specific states that each module supports
libcloud_storage - Cloud Object Storage and CDN
• services such as Amazon S3 and Rackspace CloudFiles, OpenStack Swift
libcloud_loadbalancer - Load Balancers as a Service
• services such as Amazon Elastic Load Balancer and GoGrid LoadBalancers
libcloud_dns - DNS as a Service
• services such as Amazon Route 53 and Zerigo
For DNS, the state modules can be used to provide DNS resilience for multiple nameservers, for example:
libcloud_dns: godaddy: driver: godaddy shopper_id: 12345 key: 2orgk34kgk34g secret: fjgoidhjgoim amazon: driver: route53 key: blah secret: blah
And then in a state file:
webserver: libcloud_dns.zone_present: name: mywebsite.com profile: godaddy libcloud_dns.record_present: name: www zone: mywebsite.com type: A data: 12.34.32.3 profile: godaddy libcloud_dns.zone_present: name: mywebsite.com profile: amazon libcloud_dns.record_present: name: www zone: mywebsite.com type: A data: 12.34.32.3 profile: amazon
This could be combined with a multi-cloud load balancer deployment,
webserver: libcloud_dns.zone_present: - name: mywebsite.com - profile: godaddy ... libcloud_loadbalancer.balancer_present: - name: web_main - port: 80 - protocol: http - members: - ip: 1.2.4.5 port: 80 - ip: 2.4.5.6 port: 80 - profile: google_gce libcloud_loadbalancer.balancer_present: - name: web_main - port: 80 - protocol: http - members: - ip: 1.2.4.5 port: 80 - ip: 2.4.5.6 port: 80 - profile: amazon_elb
Extended parameters can be passed to the specific cloud, for example you can specify the region with the Google Cloud API, because create_balancer can accept a ex_region argument. Adding this argument to the state will pass the additional command to the driver.
lb_test: libcloud_loadbalancer.balancer_absent: - name: example - port: 80 - protocol: http - profile: google - ex_region: us-east1
Accessing custom arguments in execution modules Some cloud providers have additional functionality that can be accessed on top of the base API, for example the Google Cloud Engine load balancer service offers the ability to provision load balancers into a specific region.
Looking at the API documentation, we can see that it expects an ex_region in the create_balancer method, so when we execute the salt command, we can add this additional parameter like this:
$ salt myminion libcloud_storage.create_balancer my_balancer 80 http profile1 ex_region=us-east1 $ salt myminion libcloud_storage.list_container_objects my_bucket profile1 ex_prefix=me
Accessing custom methods in Libcloud drivers Some cloud APIs have additional methods that are prefixed with ex_ in Apache Libcloud, these methods are part of the non-standard API but can still be accessed from the Salt modules for libcloud_storage, libcloud_loadbalancer and libcloud_dns.  The extra methods are available via the extra command, which expects the name of the method as the first argument, the profile as the second and then accepts a list of keyword arguments to pass onto the driver method, for example, accessing permissions in Google Storage objects:
$ salt myminion libcloud_storage.extra ex_get_permissions google container_name=my_container object_name=me.jpg --out=yaml
Example profiles Google Cloud Using Service Accounts with GCE, you can provide a path to the JSON file and the project name in the parameters.
google: driver: gce user_id: 234234-compute@developer.gserviceaccount.com key: /path/to/service_account_download.json auth_type: SA project: project-name
LXC Management with Salt NOTE: This walkthrough assumes basic knowledge of Salt. To get up to speed, check out the Salt Walkthrough.
Dependencies Manipulation of LXC containers in Salt requires the minion to have an LXC version of at least 1.0 (an alpha or beta release of LXC 1.0 is acceptable).  The following distributions are known to have new enough versions of LXC packaged:
• RHEL/CentOS 6 and later (via EPEL)
• Fedora (All non-EOL releases)
• Debian 8.0 (Jessie)
• Ubuntu 14.04 LTS and later (LXC templates are packaged separately as lxc-templates, it is recommended to also install this package)
• openSUSE 13.2 and later
Profiles Profiles allow for a sort of shorthand for commonly-used configurations to be defined in the minion config file, grains, pillar, or the master config file. The profile is retrieved by Salt using the config.get function, which looks in those locations, in that order. This allows for profiles to be defined centrally in the master config file, with several options for overriding them (if necessary) on groups of minions or individual minions.
There are two types of profiles:
• One for defining the parameters used in container creation/clone.
• One for defining the container's network interface(s) settings.
Container Profiles LXC container profiles are defined underneath the lxc.container_profile config option:
lxc.container_profile: centos: template: centos backing: lvm vgname: vg1 lvname: lxclv size: 10G centos_big: template: centos backing: lvm vgname: vg1 lvname: lxclv size: 20G
Profiles are retrieved using the config.get function, with the recurse merge strategy. This means that a profile can be defined at a lower level (for example, the master config file) and then parts of it can be overridden at a higher level (for example, in pillar data).  Consider the following container profile data:
In the Master config file:
lxc.container_profile: centos: template: centos backing: lvm vgname: vg1 lvname: lxclv size: 10G
In the Pillar data
lxc.container_profile: centos: size: 20G
Any minion with the above Pillar data would have the size parameter in the centos profile overridden to 20G, while those minions without the above Pillar data would have the 10G size value. This is another way of achieving the same result as the centos_big profile above, without having to define another whole profile that differs in just one value.
NOTE: In the 2014.7.x release cycle and earlier, container profiles are defined under lxc.profile. This parameter will still work in version 2015.5.0, but is deprecated and will be removed in a future release. Please note however that the profile merging feature described above will only work with profiles defined under lxc.container_profile, and only in versions 2015.5.0 and later.
Additionally, in version 2015.5.0 container profiles have been expanded to support passing template-specific CLI options to lxc.create. Below is a table describing the parameters which can be configured in container profiles:
┌──────────┬────────────────────┬──────────────────────┐ │Parameter │ 2015.5.0 and Newer │ 2014.7.x and Earlier │ ├──────────┼────────────────────┼──────────────────────┤ │template1 │ Yes                │ Yes                  │ ├──────────┼────────────────────┼──────────────────────┤ │options1  │ Yes                │ No                   │ ├──────────┼────────────────────┼──────────────────────┤ │image1    │ Yes                │ Yes                  │ ├──────────┼────────────────────┼──────────────────────┤ │backing   │ Yes                │ Yes                  │ ├──────────┼────────────────────┼──────────────────────┤ │snapshot2 │ Yes                │ Yes                  │ ├──────────┼────────────────────┼──────────────────────┤ │lvname1   │ Yes                │ Yes                  │ ├──────────┼────────────────────┼──────────────────────┤ │fstype1   │ Yes                │ Yes                  │ └──────────┴────────────────────┴──────────────────────┘
│size      │ Yes                │ Yes                  │ └──────────┴────────────────────┴──────────────────────┘
1. Parameter is only supported for container creation, and will be ignored if the profile is used when cloning a container.
2. Parameter is only supported for container cloning, and will be ignored if the profile is used when not cloning a container.
Network Profiles LXC network profiles are defined defined underneath the lxc.network_profile config option.  By default, the module uses a DHCP based configuration and try to guess a bridge to get connectivity.
WARNING: on pre 2015.5.2, you need to specify explicitly the network bridge
lxc.network_profile: centos: eth0: link: br0 type: veth flags: up ubuntu: eth0: link: lxcbr0 type: veth flags: up
As with container profiles, network profiles are retrieved using the config.get function, with the recurse merge strategy. Consider the following network profile data:
lxc.network_profile: centos: eth0: link: br0 type: veth flags: up
lxc.network_profile: centos: eth0: link: lxcbr0
Any minion with the above Pillar data would use the lxcbr0 interface as the bridge interface for any container configured using the centos network profile, while those minions without the above Pillar data would use the br0 interface for the same.
NOTE: In the 2014.7.x release cycle and earlier, network profiles are defined under lxc.nic. This parameter will still work in version 2015.5.0, but is deprecated and will be removed in a future release. Please note however that the profile merging feature described above will only work with profiles defined under lxc.network_profile, and only in versions 2015.5.0 and later.
The following are parameters which can be configured in network profiles. These will directly correspond to a parameter in an LXC configuration file (see man 5 lxc.container.conf).
• type - Corresponds to lxc.network.type
• link - Corresponds to lxc.network.link
• flags - Corresponds to lxc.network.flags
Interface-specific options (MAC address, IPv4/IPv6, etc.) must be passed on a container-by-container basis, for instance using the nic_opts argument to lxc.create:
salt myminion lxc.create container1 profile=centos network_profile=centos nic_opts='{eth0: {ipv4: 10.0.0.20/24, gateway: 10.0.0.1}}'
WARNING: The ipv4, ipv6, gateway, and link (bridge) settings in network profiles / nic_opts will only work if the container doesn't redefine the network configuration (for example in /etc/sysconfig/network-scripts/ifcfg-<interface_name> on RHEL/CentOS, or /etc/network/interfaces on Debian/Ubuntu/etc.). Use these with caution. The container images installed using the download template, for instance, typically are configured for eth0 to use DHCP, which will conflict with static IP addresses set at the container level.
NOTE: For LXC < 1.0.7 and DHCP support, set ipv4.gateway: 'auto' is your network profile, ie.:
lxc.network_profile.nic: debian: eth0: link: lxcbr0 ipv4.gateway: 'auto'
Old lxc support (<1.0.7) With saltstack 2015.5.2 and above, normally the setting is autoselected, but before, you'll need to teach your network profile to set lxc.network.ipv4.gateway to auto when using a classic ipv4 configuration.
Thus you'll need
lxc.network_profile.foo: etho: link: lxcbr0 ipv4.gateway: auto
Tricky network setups Examples This example covers how to make a container with both an internal ip and a public routable ip, wired on two veth pairs.
The another interface which receives directly a public routable ip can't be on the first interface that we reserve for private inter LXC networking.
lxc.network_profile.foo: eth0: {gateway: null, bridge: lxcbr0} eth1: # replace that by your main interface 'link': 'br0' 'mac': '00:16:5b:01:24:e1' 'gateway': '2.20.9.14' 'ipv4': '2.20.9.1'
Creating a Container on the CLI From a Template LXC is commonly distributed with several template scripts in /usr/share/lxc/templates. Some distros may package these separately in an lxc-templates package, so make sure to check if this is the case.
There are LXC template scripts for several different operating systems, but some of them are designed to use tools specific to a given distribution. For instance, the ubuntu template uses deb_bootstrap, the centos template uses yum, etc., making these templates impractical when a container from a different OS is desired.
The lxc.create function is used to create containers using a template script. To create a CentOS container named container1 on a CentOS minion named mycentosminion, using the centos LXC template, one can simply run the following command:
salt mycentosminion lxc.create container1 template=centos
For these instances, there is a download template which retrieves minimal container images for several different operating systems. To use this template, it is necessary to provide an options parameter when creating the container, with three values:
1. dist - the Linux distribution (i.e. ubuntu or centos)
2. release - the release name/version (i.e. trusty or 6)
3. arch - CPU architecture (i.e. amd64 or i386)
The lxc.images function (new in version 2015.5.0) can be used to list the available images. Alternatively, the releases can be viewed on http://images.linuxcontainers.org/images/. The images are organized in such a way that the dist, release, and arch can be determined using the following URL format: http://images.linuxcontainers.org/images/dist/release/arch. For example, http://images.linuxcontainers.org/images/centos/6/amd64 would correspond to a dist of centos, a release of 6, and an arch of amd64.
Therefore, to use the download template to create a new 64-bit CentOS 6 container, the following command can be used:
salt myminion lxc.create container1 template=download options='{dist: centos, release: 6, arch: amd64}'
NOTE: These command-line options can be placed into a container profile, like so:
lxc.container_profile.cent6: template: download options: dist: centos release: 6 arch: amd64
The options parameter is not supported in profiles for the 2014.7.x release cycle and earlier, so it would still need to be provided on the command-line.
Cloning an Existing Container To clone a container, use the lxc.clone function:
salt myminion lxc.clone container2 orig=container1
Using a Container Image While cloning is a good way to create new containers from a common base container, the source container that is being cloned needs to already exist on the minion. This makes deploying a common container across minions difficult.  For this reason, Salt's lxc.create is capable of installing a container from a tar archive of another container's rootfs. To create an image of a container named cent6, run the following command as root:
tar czf cent6.tar.gz -C /var/lib/lxc/cent6 rootfs
NOTE: Before doing this, it is recommended that the container is stopped.
The resulting tarball can then be placed alongside the files in the salt fileserver and referenced using a salt:// URL. To create a container using an image, use the image parameter with lxc.create:
salt myminion lxc.create new-cent6 image=salt://path/to/cent6.tar.gz
NOTE: Making images of containers with LVM backing
For containers with LVM backing, the rootfs is not mounted, so it is necessary to mount it first before creating the tar archive. When a container is created using LVM backing, an empty rootfs dir is handily created within /var/lib/lxc/container_name, so this can be used as the mountpoint. The location of the logical volume for the container will be /dev/vgname/lvname, where vgname is the name of the volume group, and lvname is the name of the logical volume. Therefore, assuming a volume group of vg1, a logical volume of lxc-cent6, and a container name of cent6, the following commands can be used to create a tar archive of the rootfs:
mount /dev/vg1/lxc-cent6 /var/lib/lxc/cent6/rootfs tar czf cent6.tar.gz -C /var/lib/lxc/cent6 rootfs umount /var/lib/lxc/cent6/rootfs
WARNING: One caveat of using this method of container creation is that /etc/hosts is left unmodified.  This could cause confusion for some distros if salt-minion is later installed on the container, as the functions that determine the hostname take /etc/hosts into account.
Additionally, when creating an rootfs image, be sure to remove /etc/salt/minion_id and make sure that id is not defined in /etc/salt/minion, as this will cause similar issues.
Initializing a New Container as a Salt Minion The above examples illustrate a few ways to create containers on the CLI, but often it is desirable to also have the new container run as a Minion. To do this, the lxc.init function can be used. This function will do the following:
1. Create a new container
2. Optionally set password and/or DNS
3. Bootstrap the minion (using either salt-bootstrap or a custom command)
By default, the new container will be pointed at the same Salt Master as the host machine on which the container was created. It will then request to authenticate with the Master like any other bootstrapped Minion, at which point it can be accepted.
salt myminion lxc.init test1 profile=centos salt-key -a test1
For even greater convenience, the LXC runner contains a runner function of the same name (lxc.init), which creates a keypair, seeds the new minion with it, and pre-accepts the key, allowing for the new Minion to be created and authorized in a single step:
salt-run lxc.init test1 host=myminion profile=centos
Running Commands Within a Container For containers which are not running their own Minion, commands can be run within the container in a manner similar to using (cmd.run <salt.modules.cmdmod.run). The means of doing this have been changed significantly in version 2015.5.0 (though the deprecated behavior will still be supported for a few releases). Both the old and new usage are documented below.
2015.5.0 and Newer New functions have been added to mimic the behavior of the functions in the cmd module. Below is a table with the cmd functions and their lxc module equivalents:
┌─────────────────────┬────────────────┬────────────────┐ │Description          │ cmd module     │ lxc module     │ ├─────────────────────┼────────────────┼────────────────┤ │Run a command and    │ cmd.run        │ lxc.run        │ │get all output       │                │                │ ├─────────────────────┼────────────────┼────────────────┤ │Run a command and    │ cmd.run_stdout │ lxc.run_stdout │ │get just stdout      │                │                │ ├─────────────────────┼────────────────┼────────────────┤ │Run a command and    │ cmd.run_stderr │ lxc.run_stderr │ │get just stderr      │                │                │ ├─────────────────────┼────────────────┼────────────────┤ │Run a command and    │ cmd.retcode    │ lxc.retcode    │ │get just the retcode │                │                │ ├─────────────────────┼────────────────┼────────────────┤ │Run a command and    │ cmd.run_all    │ lxc.run_all    │ │get all information  │                │                │ └─────────────────────┴────────────────┴────────────────┘
2014.7.x and Earlier Earlier Salt releases use a single function (lxc.run_cmd) to run commands within containers. Whether stdout, stderr, etc. are returned depends on how the function is invoked.
To run a command and return the stdout:
salt myminion lxc.run_cmd web1 'tail /var/log/messages'
To run a command and return the stderr:
salt myminion lxc.run_cmd web1 'tail /var/log/messages' stdout=False stderr=True
To run a command and return the retcode:
salt myminion lxc.run_cmd web1 'tail /var/log/messages' stdout=False stderr=False
To run a command and return all information:
salt myminion lxc.run_cmd web1 'tail /var/log/messages' stdout=True stderr=True
Container Management Using salt-cloud Salt cloud uses under the hood the salt runner and module to manage containers, Please look at this chapter
Container Management Using States Several states are being renamed or otherwise modified in version 2015.5.0. The information in this tutorial refers to the new states. For 2014.7.x and earlier, please refer to the documentation for the LXC states.
Ensuring a Container Is Present To ensure the existence of a named container, use the lxc.present state. Here are some examples:
# Using a template web1: lxc.present: - template: download - options: dist: centos release: 6 arch: amd64
# Cloning web2: lxc.present: - clone_from: web-base
# Using a rootfs image web3: lxc.present: - image: salt://path/to/cent6.tar.gz
# Using profiles web4: lxc.present: - profile: centos_web - network_profile: centos
WARNING: The lxc.present state will not modify an existing container (in other words, it will not re-create the container).  If an lxc.present state is run on an existing container, there will be no change and the state will return a True result.
The lxc.present state also includes an optional running parameter which can be used to ensure that a container is running/stopped. Note that there are standalone lxc.running and lxc.stopped states which can be used for this purpose.
Ensuring a Container Does Not Exist To ensure that a named container is not present, use the lxc.absent state. For example:
web1: lxc.absent
Ensuring a Container is Running/Stopped/Frozen Containers can be in one of three states:
• running - Container is running and active
• frozen - Container is running, but all process are blocked and the container is essentially non-active until the container is "unfrozen"
• stopped - Container is not running
Salt has three states (lxc.running, lxc.frozen, and lxc.stopped) which can be used to ensure a container is in one of these states:
web1: lxc.running
# Restart the container if it was already running web2: lxc.running: - restart: True
web3: lxc.stopped
# Explicitly kill all tasks in container instead of gracefully stopping web4: lxc.stopped: - kill: True
web5: lxc.frozen
# If container is stopped, do not start it (in which case the state will fail) web6: lxc.frozen: - start: False
Remote execution tutorial Before continuing make sure you have a working Salt installation by following the installation and the configuration instructions.
Order your minions around Now that you have a master and at least one minion communicating with each other you can perform commands on the minion via the salt command. Salt calls are comprised of three main components:
salt '<target>' <function> [arguments]
SEE ALSO: salt manpage
target The target component allows you to filter which minions should run the following function. The default filter is a glob on the minion id. For example:
salt '*' test.version salt '*.example.org' test.version
Targets can be based on minion system information using the Grains system:
salt -G 'os:Ubuntu' test.version
SEE ALSO: Grains system
Targets can be filtered by regular expression:
salt -E 'virtmach[0-9]' test.version
Targets can be explicitly specified in a list:
salt -L 'foo,bar,baz,quo' test.version
Or Multiple target types can be combined in one command:
salt -C 'G@os:Ubuntu and webser* or E@database.*' test.version
function A function is some functionality provided by a module. Salt ships with a large collection of available functions. List all available functions on your minions:
salt '*' sys.doc
Show all currently available minions:
Run an arbitrary shell command:
salt '*' cmd.run 'uname -a'
SEE ALSO: the full list of modules
arguments Space-delimited arguments to the function:
salt '*' cmd.exec_code python 'import sys; print sys.version'
Optional, keyword arguments are also supported:
salt '*' pip.install salt timeout=5 upgrade=True
They are always in the form of kwarg=argument.
Multi Master Tutorial As of Salt 0.16.0, the ability to connect minions to multiple masters has been made available. The multi-master system allows for redundancy of Salt masters and facilitates multiple points of communication out to minions. When using a multi-master setup, all masters are running hot, and any active master can be used to send commands out to the minions.
NOTE: If you need failover capabilities with multiple masters, there is also a MultiMaster-PKI setup available, that uses a different topology MultiMaster-PKI with Failover Tutorial
In 0.16.0, the masters do not share any information, keys need to be accepted on both masters, and shared files need to be shared manually or use tools like the git fileserver backend to ensure that the file_roots are kept consistent.
Beginning with Salt 2016.11.0, the Pluggable Minion Data Cache was introduced. The minion data cache contains the Salt Mine data, minion grains, and minion pillar information cached on the Salt Master. By default, Salt uses the localfs cache module, but other external data stores can be used instead.
Using a pluggable minion cache modules allows for the data stored on a Salt Master about Salt Minions to be replicated on other Salt Masters the Minion is connected to. Please see the Minion Data Cache documentation for more information and configuration examples.
Summary of Steps 1. Create a redundant master server
2. Copy primary master key to redundant master
3. Start redundant master
4. Configure minions to connect to redundant master
5. Restart minions
6. Accept keys on redundant master
Prepping a Redundant Master The first task is to prepare the redundant master. If the redundant master is already running, stop it. There is only one requirement when preparing a redundant master, which is that masters share the same private key. When the first master was created, the master's identifying key pair was generated and placed in the master's pki_dir. The default location of the master's key pair is /etc/salt/pki/master/. Take the private key, master.pem, and copy it to the same location on the redundant master. Do the same for the master's public key, master.pub. Assuming that no minions have yet been connected to the new redundant master, it is safe to delete any existing key in this location and replace it.
NOTE: There is no logical limit to the number of redundant masters that can be used.
Once the new key is in place, the redundant master can be safely started.
Configure Minions Since minions need to be master-aware, the new master needs to be added to the minion configurations. Simply update the minion configurations to list all connected masters:
master: - saltmaster1.example.com - saltmaster2.example.com
Now the minion can be safely restarted.
NOTE: If the ipc_mode for the minion is set to TCP (default in Windows), then each minion in the multi-minion setup (one per master) needs its own tcp_pub_port and tcp_pull_port.
If these settings are left as the default 4510/4511, each minion object will receive a port 2 higher than the previous. Thus the first minion will get 4510/4511, the second will get 4512/4513, and so on. If these port decisions are unacceptable, you must configure tcp_pub_port and tcp_pull_port with lists of ports for each master. The length of these lists should match the number of masters, and there should not be overlap in the lists.
Now the minions will check into the original master and also check into the new redundant master. Both masters are first-class and have rights to the minions.
NOTE: Minions can automatically detect failed masters and attempt to reconnect to them quickly. To enable this functionality, set master_alive_interval in the minion config and specify a number of seconds to poll the masters for connection status.
If this option is not set, minions will still reconnect to failed masters but the first command sent after a master comes back up may be lost while the minion authenticates.
Sharing Files Between Masters Salt does not automatically share files between multiple masters. A number of files should be shared or sharing of these files should be strongly considered.
Minion Keys Minion keys can be accepted the normal way using salt-key on both masters.  Keys accepted, deleted, or rejected on one master will NOT be automatically managed on redundant masters; this needs to be taken care of by running salt-key on both masters or sharing the /etc/salt/pki/master/{minions,minions_pre,minions_rejected} directories between masters.
NOTE: While sharing the /etc/salt/pki/master directory will work, it is strongly discouraged, since allowing access to the master.pem key outside of Salt creates a SERIOUS security risk.
File_Roots The file_roots contents should be kept consistent between masters. Otherwise state runs will not always be consistent on minions since instructions managed by one master will not agree with other masters.
The recommended way to sync these is to use a fileserver backend like gitfs or to keep these files on shared storage.
IMPORTANT: If using gitfs/git_pillar with the cachedir shared between masters using GlusterFS, nfs, or another network filesystem, and the masters are running Salt 2015.5.9 or later, it is strongly recommended not to turn off gitfs_global_lock/git_pillar_global_lock as doing so will cause lock files to be removed if they were created by a different master.
Pillar_Roots Pillar roots should be given the same considerations as file_roots.
Master Configurations While reasons may exist to maintain separate master configurations, it is wise to remember that each master maintains independent control over minions.  Therefore, access controls should be in sync between masters unless a valid reason otherwise exists to keep them inconsistent.
These access control options include but are not limited to:
• external_auth
• publisher_acl
• peer
• peer_run
Multi-Master-PKI Tutorial With Failover This tutorial will explain, how to run a salt-environment where a single minion can have multiple masters and fail-over between them if its current master fails.
The individual steps are
• setup the master(s) to sign its auth-replies
• setup minion(s) to verify master-public-keys
• enable multiple masters on minion(s)
• enable master-check on  minion(s) Please note, that it is advised to have good knowledge of the salt- authentication and communication-process to understand this tutorial.  All of the settings described here, go on top of the default authentication/communication process.
Motivation The default behaviour of a salt-minion is to connect to a master and accept the masters public key. With each publication, the master sends his public-key for the minion to check and if this public-key ever changes, the minion complains and exits. Practically this means, that there can only be a single master at any given time.
Would it not be much nicer, if the minion could have any number of masters (1:n) and jump to the next master if its current master died because of a network or hardware failure?
NOTE: There is also a MultiMaster-Tutorial with a different approach and topology than this one, that might also suite your needs or might even be better suited Multi-Master Tutorial
It is also desirable, to add some sort of authenticity-check to the very first public key a minion receives from a master. Currently a minions takes the first masters public key for granted.
The Goal Setup the master to sign the public key it sends to the minions and enable the minions to verify this signature for authenticity.