Spicerack/Cookbooks

== Cookbook Operations ==
To list or run a Cookbook you must be logged into one of the cumin hosts ().
=== List available Cookbooks ===
To show a tree of the available cookbooks run:
 $ cookbook -l
To include a brief one-line description of each cookbook in the tree use:
 $ cookbook -lv
=== Run a single Cookbook ===
The generic way of running a cookbook is:
 $ sudo cookbook [-d] [-v] COOKBOOK [COOKBOOK_ARGS]
The full dotted name as shown in the tree of the available cookbooks (e.g. sre.hosts.downtime) must be used.
To see the global options of the cookbook executable:
 $ cookbook -h
To see the specific options of a given cookbook:
 $ sudo cookbook sre.hosts.downtime -h
To run the cookbook in dry-run mode (it will not !log and will not execute any RW operation in the infrastructure):
 $ sudo cookbook -d sre.hosts.downtime [COOKBOOK_ARGS]
And finally, to actually run the cookbook:
 $ sudo cookbook sre.hosts.downtime [COOKBOOK_ARGS]
=== Logs ===
How the cookbook store their logs is described in the Spicerack documentation.
== Development ==
=== Documentation ===
See the Cookbooks API section of the Spicerack documentation.
Make sure to set the ownership field of the cookbook if not set at the parent package level. The accepted values are the SRE teams from Puppet's Wmflib::Team define. They are currently listed also in the cookbooks repository tests so that CI can ensure only valid values are used.
=== Test before merging ===
Gerrit patches can be tested on the cumin hosts () before actually merging them using the test-cookbook binary. It allows to checkout the patch from Gerrit and run spicerack using that checkout of the cookbooks. It manages automatically the checkout and allows to clean it up too. Here's the current (June 2023) help message for the script:
$ test-cookbook -h
usage: test-cookbook [-c CHANGE] [--ps PS] [--delete] [--no-sal-logging] [-h]
Cookbook testing script
It allows testing a Gerrit patch on the cookbook repository checking out the change locally
on the cumin host where the command is run and running the cookbook with the change code instead
of the officially deployed one. It also allows to specify a given patch set (PS) to use.
The cookbook binary will be invoked with the -c/--config argument already set to point to the
custom configuration pointing to the testing checkout.
After a first checkout it is possible to modify the files in place and test it again. The script
will automatically detect that there are local changes and ask the operator what to do, as long
as the local modifications are on the same branch (hence the same PS) requested.
It should be used with caution but at least provides a standard way to test cookbook changes.
All the extra parameters are passed to the cookbook binary.
Example usage:
 # Use the latest PS of Gerrit change 12345 to make a DRY-RUN of the downtime cookbook
 test-cookbook -c 12345 --dry-run sre.hosts.downtime -h
 # Use a specific PS of Gerrit change 12345 to make a REAL run of the downtime cookbook
 test-cookbook -c 12345 --ps 3 sre.hosts.downtime -h
 # Cleanup a previously tested change
 test-cookbook --delete -c 12345
The generated files structure is as follows:
 ~/cookbooks_testing/ # Parent directory of all modified files
 ~/cookbooks_testing/config.yaml # The configuration file that is passed to the cookbook binary
 ~/cookbooks_testing/cookbooks # Symlink that points to the currently tested change
 ~/cookbooks_testing/cookbooks-$CHANGE_ID # The git checkout of a given the change ID
 ~/cookbook_testing/logs # The log directory where all cookbooks will log into
optional arguments:
 -c CHANGE, --change CHANGE
 The Gerrit change ID to fetch.
 --ps PS The optional patch set to fetch. If not specified the last one will be fetched.
 --delete Delete any existing environment for the given change.
 --no-sal-logging Do not log to SAL. Use with CAUTION!
 -h, --help show this help message and exit
=== Deployment ===
Once a patch is +2 on Gerrit, CI will take care of the actual merge and at the next Puppet run the cookbook will be updated automatically on the cluster management hosts (). Forcing a Puppet run after the merge will update the cookbooks immediately.
==== Renaming/Deleting a cookbook ====
Due to the current way of deploying cookbooks, if a cookbook gets renamed or deleted, its compiled files (.pyc), if present, are not automatically deleted and need a manual deletion.
Find the existing files with this command, assuming the cookbook being renamed/deleted was called sre.somedir.example-action, and then proceed to delete them, for example by appending -delete to the find command.
sudo cumin 'A:cumin' 'find /srv/deployment/spicerack/cookbooks/sre/somedir/ -name "example-action*.pyc"'
===Creating your local environment ===
==== Pre-requisites ====
* tox
==== Installation ====
* Clone the required repositories from Gerrit under ~/wmf/gerrit for simplicity
 $ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/operations/software/spicerack
 $ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/operations/cookbooks
* Change to the spicerack directory and run tox. It will create the proper virtualenv for us with all the dependencies required:
 $ tox -e py3N-unit # where N is picked from any of environment listed with `tox -av | grep unit` for which the related Python version is available on the system
* Note: We could just run 'tox' at this point, it will be just a bit slower because it will run all the tests with all the compatible python versions that are installed in the system and configured in the tox.ini file to run the tests
* There should be a .tox/py3N-tests/ directory with the virtualenv created, we can source the activate script
 $ source .tox/py3N-tests/bin/activate
* Create a configuration file for spicerack called config.yaml:
 cookbooks_base_dirs:
 - ~/wmf/gerrit/cookbooks
 logs_base_dir: ~/wmf/gerrit/spicerack/logs
* The cookbook entry point script should be available, and you should be able to see the list of available cookbooks
 $ cookbook -c config.yaml -l
===== Additional steps for WMCS cookbooks =====
* Clone the additional repository for WMCS-specific cookbooks:
$ git clone ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/cloud/wmcs-cookbooks
* Add the new checked out repository to the list of available cookbooks in the config.yaml file:
 cookbooks_base_dirs:
 - ~/wmf/gerrit/cookbooks
 - ~/wmf/gerrit/wmcs-cookbooks
 logs_base_dir: ~/wmf/gerrit/spicerack/logs
== Also see ==
* Runbook
Category:Deployment
Category:SRE Infrastructure Foundations