saltstack/salt — pull request #10273 by kenatbasis (2014-02-07 20:42:53+00:00)
Title: Adds master module for populating pillars via external module

### Problem: No good way to distribute static cluster information to a set of nodes at deploy time, ie ip addresses for iptables.

#### Current solution 1: Salt Mine
This doesn't scale; you have thousands of nodes all pinging the salt master at a set interval. Also, for static information (like ip addresses or hostname), updating the information doesn't matter. You can't access the mine inside a pillar, so the minions must call back to the master to get the mined data. You can't store arbitrary values, and the key is a function name, instead of a function name with args (I'd like to be able to look up mine data for network.ipaddrs eth0 only, instead of parsing all of network.interfaces minion side every time).

#### Current solution 2: Peer publish
First, peer publish is slow. 2 nodes asking for each other's ip takes on average 3 seconds. It is also inefficient and doesn't scale: a node must respond once for every other node in the cluster. Putting the publish in the pillars still doesn't scale, and putting the publish in the state files means extra calls for every appearance in a state file (no caching).

#### Current solution 3: Cmd.run + Jinja magic
This involves embedding a saltutil.cmd to grains in a top level pillar file, then avoiding calling it again by including other pillar files and passing them via defaults. For example:
```
{% set all_grains = salt['saltutil.cmd']('cluster:' ~ grains['cluster'], 'grains.items', expr_form='grain').values()|map(attribute='ret')|list %}
{% set nodes_by_role = all_grains | groupby('role') %}
{% set app_node_ips = nodes_by_role[0][1:][0]|map(attribute='ip_interfaces')|map(attribute='eth1')|map('first')|list|sort %}
include:
- app:
defaults:
app_node_ips: {{ app_node_ips }}
```
This is quite hacky. This is what we have been using, and it scales better than the others but is still quite inefficient. Also, this requires many worker processes to avoid timeouts on larger clusters, which requires a lot of memory (we run with 128 workers, which is around 8GB). This could probably be done better with cmd.run + salt-run cache.grains, but dropping into a subshell seems wrong.

#### Proposed solution: Master module
The biggest problem is that the salt variable in jinja doesn't include _modules, so any module you want to include in jinja has to be in the base salt code. This module will correctly allow you to call _modules from jinja, allowing for flexible development of modules specifically for populating pillars.

For example, this is how to distribute ip addresses to a cluster at deploy time:

file_roots/_modules/cache.py:
```
'''
Module version of salt-run cache

Meant to be called by master.mmodule

Most of these functions will output strings instead of python lists/dicts
because salt's jinja config autoescapes and adds all sorts of \
'''

# Import Salt libs
import salt
import salt.utils.master

def grain(tgt, key, expr_form='glob', default=''):
'''
Attempt to retrieve the named value from cached grains, if the named value is not
available return the passed default. The default return is an empty string.

The value can also represent a value in a nested dict using a ":" delimiter
for the dict. This means that if a dict in grains looks like this::

{'pkg': {'apache': 'httpd'}}

To retrieve the value associated with the apache key in the pkg dict this
key can be passed::

pkg:apache
'''
pillar_util = salt.utils.master.MasterPillarUtil(tgt, expr_form,
use_cached_grains=True,
grains_fallback=False,
opts=__opts__)
cached_grains = pillar_util.get_minion_grains()
ret = {}
for node in cached_grains:
ret[node] = salt.utils.traverse_dict(cached_grains[node], key, default)

return str(ret)
```

pillar_roots/included_pillar.sls:
```
cluster_ips: {{ salt['master.mmodule'](grains['cluster'], 'cache.grain', 'G@cluster:mycluster', 'ip_interfaces:eth1', expr_form='compound' ) }}
```

This code needs better error reporting/handling, but since jinja mangles it all anyways...