Package: cfgtool
A configuration management tool that provides for self-documenting hosts
and coherently versioned configurations.What doescfgtooldo?cfgtoolallows applications to separate their configuration files
from the host specific values that go into those files. These values,
calledbeliefs, can be deployed either by hand or by common
infrastructure orchistration toolsSaltStackorAnsible.DisclaimerWhilecfgtoolshouldn’t have any issue running on most Linux
distributions, it has only been tested on Debian based Linux systems.How do I get it?cfgtoolcan be installed viapip. It should be included as a
dependency for each application you install that is expected to have its
configuration managed by it.pip install cfgtoolHow does it work?Upon installation (assuming with root permissions),cfgtoolcreates
the following configuration file and directories./etc/cfgtool/cfgtool.conf
/etc/cfgtool/module.d/
/etc/cfgtool/belief.d/module.dcontains a file for each installed application usingcfgtool. This file contains a list of configuration files we expectcfgtoolto generate for that application. Each of these
configuration files is expected to have a corresponding.templfile
at the same directory as it is going to be installed.Example:/etc/my_application/application.conf
/etc/my_application/application.conf.templThe.templfile is the template configuration file it will seed with
beliefs to generate the actual configuration file.belief.dcontains JSON files, which hold ourbeliefsused in
generating application configurations.We can change the locations of ourmodule.dandbelief.ddirectory in ourcfgtool.conffile if we wish.belief_dir = "/my/new/belief/dir/..."
module_dir = "/my/new/module/dir/..."An exampleLet’s say we have a Python application calledreporter.reporteris a program that generates a report every hour about the statistics for
the machine it runs on and sends them to a reporting server. Reporter
has two configuration files:/etc/reporter/reporter.confdefines what we want the report of our
machine to include.temperature=<boolean>
system_load=<boolean>
disk_space=<boolean>
.../etc/reporter/report_send_init.shdefines where we want the report
to go.exportREPORTER_USER=<string>exportREPORTER_PASS=<string>exportFTP_SERVER=<string>exportREPORT_PREFIX=<string>This is used in a/etc/cron.hourly/reportercron script that runs
every hour to generate a report and upload it to an FTP server.#! /bin/bash./etc/reporter/report_send_init.shfilename="${REPORT_PREFIX}_$(date+%Y-%m-%d-%H-%M)"reporter-c/etc/reporter/reporter.conf>/tmp/${filename}.txtftp-n<<EOF
open ${FTP_SERVER}
user ${REPORTER_USER} ${REPORTER_PASS}
put /tmp/${filename}.txt
EOFrm-f/tmp/${filename}.txtNow let’s say we are given the following scenario; we have to deploy ourreportertool over two server clusters at a company creatively namedcompany.company_overviewAfter careful analysis,companyhas determined that the temperature of
each machine in theirletterscluster needs to be carefully monitored.
They have another cluster callednumbersthat has a state of the art
cooling system with its own reporting, but has frequent disk space and
system load issues that need to be monitored.The machines of each cluster periodically send their reports to their
master.How are we going to configure all of these machines?Fortunately for us,companywas diligent enough to implement some
orchistration software last month to deploy config files on each of
their machines. Their head of operations would like to have as few
configuration files as possible deployed across all installed
applications. Let’s try and minimize the number of managed
configurations usingcfgtool!Modifying the configurations ofreporterSeeing asreporteris an in-house tool, we can modify it to usecfgtoolas a dependency.The head of operations has agreed to start usingcfgtoolto simplify
configuration across all servers. A unique ‘environment’ beliefs file
will be created and deployed for each of the machines in each cluster
under/etc/cfgtool/belief.d/env.json. This will contain beliefs that
may be useful to many applications usingcfgtool.Here is what that file would look like for server A in our diagram:{"master":{"domain":"letters.company.com","username":"machine_a","password":"very_secret_password"},"host":{"name":"letters-a","address":"lettersa.company.com"}}Our head of operations has also deployedreporterspecific
configurations to each cluster onto each machine under the file name/etc/reporter/belief.d/reporter.json. Here is what that would look
like for machines in thenumbers.company.comcluster.{"reporter":{"temperature":false,"system_load":true,"disk":true}}Let’s createtemplfiles for each of our configuration files. In
each of ourtemplfiles, we can reference these beliefs underbelief.d/using${...}syntax.cfgtoolmerges all of our
belief files together (traversing them alphabetically) into one big
dictionary.Warning:If a belief is specified twice, the later one (the one
in a file whose name is alphabetically greater) will be whatcfgtooluses.Let’s change our configuration files intotemplfiles./etc/reporter/reporter.conf.templtemperature=${reporter.temperature}system_load=${reporter.system_load}disk_space=${reporter.disk}.../etc/reporter/report_send_init.sh.templexportREPORTER_USER=${master.username}exportREPORTER_PASS=${master.password}exportFTP_SERVER=${master.domain}exportREPORT_PREFIX=${host.name}Now let’s assume our directory structure forreporterlooks like
this:├── LICENSE
├── README.md
├── setup.py
├── requirements.txt
├── reporter
│   ├── __init__.py
│   ├── reporter.py
│   ├── ...
├── config
│   ├── reporter
│   ├── reporter.conf.templ
│   ├── report_send_init.sh.templ
│   ├── reporter.sh
├── install.shOur newinstall.shscript will look like this:#! /bin/shinstall-D-groot-oroot-m0644-pconfig/reporter/etc/cfgtool/module.d/reporterinstall-D-groot-oroot-m0644-pconfig/reporter.conf.templ/etc/reporter/reporter.conf.templinstall-D-groot-oroot-m0644-pconfig/report_send_init.sh.templ/etc/reporter/report_send_init.sh.templinstall-D-groot-oroot-m0644-pconfig/reporter.sh/etc/cron.hourly/reporterThereporterfile contains the names of the configuration files thatreportershould generate:/etc/reporter/reporter.conf/etc/reporter/report_send_init.shCreating therealconfig filesAt this point we have fully integratedcfgtoolintoreporter,
deployed our beliefs and have our software installed, butnotyet
usable because our config files do not exist yet. We still have to tellcfgtoolto generate our configurations.To generate the configurations, we run the following in the terminal:$cfgtoolwrite<module>[--force]We replace<module>with the application we would like to create
configuration files for.--forcemeans we wouldactuallylike to
write (to keep it differentiated from non-destructive commands).Let’s create the configuration files forreporter. This command
needs to be run on each machinereporteris installed on and should
be part of your deployment process.$cfgtoolwritereporter--forceModule:reporterGenerate...File:/etc/reporter/reporter.confFile:/reporter/report_send_init.shLet’s look at what was produced on machine A:/etc/reporter/reporter.conftemperature=truesystem_load=falsedisk_space=false.../etc/reporter/report_send_init.shexportREPORTER_USER="machine_a"exportREPORTER_PASS="very_secret_password"exportFTP_SERVER="letters.company.com"exportREPORT_PREFIX="letters-a"Wow, each machine is configured and ready to report just like that!What if our configuration changes?Now it is a safe assumption that our setup will not necessarily stay the
same forever. Let’s say two years after starting to usecfgtool, a
new cooling system is introduced into our letters cluster and more
machines are added. We no longer have temperature issues we need to
monitor, but system load is now something we need to watch for some
reason.A newreporterbeliefs file has been deployed onto this cluster./etc/reporter/belief.d/reporter.json{"reporter":{"temperature":false,"system_load":true,"disk":false}}To update our configurations, we simply run ourcfgtool writecommand on the cluster, as we did with the initial installation.$cfgtoolwritereporter--forceAnd once again, our configurations are up to date!Wait, what if I made a mistake and want my old configuration back?cfgtoolis careful in that it always leaves a time stamped copy of
whatever it overwrites behind. Here’s what our directory on machine A
looks like after runningcfgtool writeagain.├── etc
│   ├── reporter
│   │   ├── reporter.conf.templ
│   │   ├── report_send_init.sh.templ
│   │   ├── reporter.conf
│   │   ├── reporter.conf-backup.2016-01-20_0019.23
│   │   ├── report_send_init.sh
│   │   ├── report_send_init.sh-backup.2016-01-20_0019.23
│   │   ├── ...Eventually this may really start to pile up after many consecutive
redeployments:reporter.conf
reporter.conf-backup.2016-01-10_0019.23
reporter.conf-backup.2016-01-11_0112.01
reporter.conf-backup.2016-01-12_1202.26
reporter.conf-backup.2016-01-13_0311.04
reporter.conf-backup.2016-01-14_1049.45
reporter.conf-backup.2016-01-15_0059.15
reporter.conf-backup.2016-01-16_5001.02
reporter.conf-backup.2016-01-17_0019.21
...We can havecfgtooltoss all of our backups with thecleancommand.$ cfgtool clean reporterAnd now all the backups are gone.reporter.conf
...Hmm, I can’t recall if I (re)generated my configurations alreadyWe can check if our existing configurations files match our beliefs (or
even exist) by running thecheckcommand.$cfgtoolcheckreporterModule:reporterGenerate...File:/etc/reporter/reporter.conf-checkFile:/reporter/report_send_init.sh-checkCheck...File:/etc/reporter/reporter.conf-checkFile:/reporter/report_send_init.sh-checkIf anything is inconsistent, the checks will not pass. Leave out the
module name to do a check forallinstalled applications.What if I want to generate configurations to see what they look like butnotimmediately use them?Runcfgtoolwith thesamplecommand to generate config files
with a.sampleextension.$cfgtoolsamplereporterModule:reporterGenerate...File:/etc/reporter/reporter.conf.sampleFile:/reporter/report_send_init.sh.sampleIf everything looks good, just runcfgtoolwithwrite.Wait a minute, going back to this thing about all the beliefs being combined bycfgtool, doesn’t that expose secrets?Now say for example we install another application calleduploaderwhich takes files that our vast client base uploads and puts them in
Amazon S3. Among its beliefs is an AWS key:/etc/cfgtool/belief.d/uploader.json{"uploader":{"aws_secret_key":"..."}}We don’t want tools likereportergetting access to that information
by simply putting${uploader.aws_secret_key}somewhere in their
configurationtemplfiles. What do we do?Fortunately,cfgtoolis smart enough to realize who should know
what. As long as there is a top level belief calleduploader,cfgtoolwill realize it should only be seen by theuploaderapplication and hide it.We can confirm this by checking the beliefs that are exposed toreporterafter installinguploaderwith thebeliefcfgtoolcommand in machine A.$cfgtoolbeliefdb_reportsModule:db_reports{"master":{"domain":"letters.company.com","username":"machine_a","password":"very_secret_password"},"host":{"name":"letters-a","address":"lettersa.company.com"},"reporter":{"temperature":false,"system_load":true,"disk":true}}The beliefs ofuploaderhave not been exposed. You can be assured
yourcfgtoolutilizing apps only know what they are supposed to
know!What’s next?This page summarizes the major functionality ofcfgtool. To learn
more about other features ofcfgtool, check out the help section via
your terminal.$cfgtool-help