Portal:Cloud VPS/Admin/notes/Puppet Possibilities

Talking Puppet, Hiera, Openstack and process
= Meeting Agenda =
== Old universe ==
* Nova in many places for (nearly) all things
* intermingled repo management
* Current install specific groupings (nova-api and nova-network, etc)
* Per host values for labtest in hiera and regex madness
* hiera lookups buried in: roles, modules, templates
* Overlapping requirements for components handled via require_package() a lot
* Passing dicts of huge numbers of unrelated params to get to a few
* Duplicate params at various levels of the hiera tree
* Convoluted component interdependence
* Associated configuration lumped into large blocks in roles or other
:* ferm rules that make it difficult to reason on cleanup as it's unclear why it existed
* Inconsistent strategy on hostname => ip esp for ferm
* data structured mingled in with code (labsdb, etc)
* The same value would be named multiple things throughout a config tree
== General ==
* Three failover scenarios
:* labcontrol (host and cold) via hiera
:* controlled via hiera params setting a boolean in a profile and passed to modules 
class openstack2::keystone::service(
 $active,
:* labservices (partial host and cold)
:* labnet (host and cold) role swapping <== looking to deprecate
* Deployment separations (deployments to come)
:* deployments may end up being regions? 
:* deployments are made up of components
:* components can be upstream openstack specific or not
* Why not use inheritance:
:* https://puppet.com/docs/puppet/4.10/lang_classes.html#inheritance
:* so much implicit reasoning already
* We are working on a holistics openstack / VPS product profile model
* consistent naming across components glance/service.pp, keystone/service.pp
:* If it is parameterized per deployment is worth doing base and deployment profiles potentially under profiles/manifests/openstack
* Understanding our Hiera use cases:
:* https://phabricator.wikimedia.org/T171494#3479169
:* Hiera: per site, and our special base instead of defunct common. Hiera consistency must be our bread and butter.
:* hiera_lookup
* Labs hiera tree is (also) a nightmare.
:* expand_paths vs not
:* labs/private is an overloaded dual purpose shim
:* back end split
:* Labs hiera is a nightmare
:* Per deployment labs/private or equiv may be needed
* Puppet norms for new style are not always set up for sucess
:* base_firewall, standard, remaining implicit override, hiera adding functionality, etc
== Upstream to us Style ==
* https://wikitech.wikimedia.org/wiki/Puppet_coding#Coding_Style
* https://wikitech.wikimedia.org/wiki/Puppet_coding/Style_guide
* https://puppet.com/docs/puppet/5.3/style_guide.html
* I want to get more consistent with our own style
== To Be Done ==
* Issue of cross site deployment variables (db1047)
* Issue of special failover with labnet
* firewall rules buried in site.pp
* tools stub zone on labservices persistence
* Issue of cleanup for hiera
* openstack2 => openstack
* test failover for pairs
== Moving forward ==
* resolve() within ferm
:* per ferm update resolution instead of per puppet run
:* human readable in diffs
* Our own sub-style consistency that is an extension but not an override
:* File and resource consistence etc
:** owner,group,mode,source,requires
:* manifest flow: include, package, file(s), service
* This results in per deployment per component profiles and per deployment roles
:* Much easier to test in labtest without effecting main, and we SHOULD/need to/must
:* staging models mean more of an opportunity for productive meaningful refactor
:* testplan designations
* "foo: bar for x". "testplan: baz?"
* modules/roles/manifests/wmcs
:* potentially: modules/roles/manifests/wmcs/tenants
* Anything
* Functional approach to hiera and parameters. Ironic? :)
* In the mid-term we must consolidate on WMCS managed hiera strategy
* A large and intimidating Toolforge cleanup must happen
* We will have to solve the puppet multi-tenancy problem in the next 2 years or die trying