catkin/xylem — pull request #17 by NikolausDemmel (2014-07-29 20:37:50+00:00)
Title: Config utilities for describing config files and CLI arguments

- new module `config_utils` for a general purpose tool to describe config files, parse
yaml config files and expose specific arguments to the command line; includes some
unit tests for the config item types

- new module `config` describes xylem's global configuration; includes the logic to
compute sources and cache folders given command line arguments and config files

- new module `arguments` for global command line arguments (moved from `utils`);
interactis with config to include and process global config arguments

- new module `yaml_utils` contains customization to loading and dumping yaml (moved
(from `utils`)

- `--help` argument now uses pydoc.pager to display the help in a `less`-like scrollable
manner on pty's

- help output includes new section `config arguments`, describing the arguments
that correspond to a config file entry

- commands have a new hook to add to the config description; some logic common
to all commands was moved to `commands.main`

- context objects like `InstallerContext` and `SourcesContext` now take a config
dictionary as argument, which defaults to the global config sigleton if not
passed explicitly

- high-level API used by the commands (`update`, `resolve`, `lookup`) is also
parameterized by an optional config dict argument, which defaults to the
global config

- Introduce XylemError as common Exception base class. Having a common
base class allows users to catch all xylem related exceptions in one handler.

Output of `xylem resolve --help`:

```
usage: xylem resolve [--show-trumped] [--all] [--show-priority]
[--show-default-installer] [--os name:version]
[--os-features "feature1,feature2,..."]
[--core-installers "inst1,inst2,..."]
[--use-additional-installers yes|no]
[--install-from "inst1:[key1,key2,...] ..."]
[--installer-options "inst1:{opt1:val1,...}, ..."]
[--user-sources yes|no] [--cache-dir PATH]
[--sources-dir PATH] [-h] [--version] [-v] [-d]
[--print-config] [--no-user-config]
[--prefix XYLEM_PREFIX] [--xylem-dir XYLEM_DIR]
[xylem_key [xylem_key ...]]

Lookup a xylem key and resolve to unique, parsed rule based on priorities.

positional arguments:
xylem_key

optional arguments:
--show-trumped Show all possible resolutions for key, also for
trumped installers.
--all Resolve all keys with resolution for this OS.
--show-priority Show priority of installer.
--show-default-installer
Show installer even if it is the default installer.

config arguments:
The following typed arguments correspond to entries in the config file.
Command line argument values are interpreted as YAML and override the
corresponding entries in user/system config files.

There are some special cases and short hand notation for parsing config
arguments:
* List: the outer '[]' can be omitted; multiple occurrences of the same
command line argument are concatenated
* MergingDict: the outer '{}' can be omitted; multiple occurrences of the
same command line argument are merged; value is merged with the config
file entries
* Path: string parsed as-is; path is expanded (e.g. `~`)
* String: string parsed as-is

--os name:version override os detection; if no ':' is present, the
entire string is interpreted as the os name and the
version is detected
(config: `os_override`, type: String)
--os-features "feature1,feature2,..."
OS features to be used. The list of possible values
and the default choice is defined by the os plugin for
the selected OS.
(config: `os_options/features`, type: List)
--core-installers "inst1,inst2,..."
core installers to be used. The default list is
defined by the OS plugin for the selected OS.
Additional installers as defined by installer plugins
may be used on top of the core installers unless the
`use_additional_installers` option is set to False.
(config: `os_options/installers`, type: List)
--use-additional-installers yes|no
if `True`, use additional installers as defined by
installer plugins on top of the core installers
(type: Boolean, default: True)
--install-from "inst1:[key1,key2,...] ..."
mapping installer names to list of xylem keys;
overwrites the installer priority such that the given
keys are only ever installed with the specified
installer
(type: MergingDict)
--installer-options "inst1:{opt1:val1,...}, ..."
options passed to installer plugins; valid options are
specific to each installer plugin
(type: MergingDict)
--user-sources yes|no
if `True`, look for sources and cache in user
directory instead of system-wide locations
(type: Boolean, default: False)
--cache-dir PATH override the cache location
(type: Path)
--sources-dir PATH override the sources directory
(type: Path)

global arguments:
By default xylem operates with system-wide configuration of sources and
cache. The path under which those are found can be set with the
XYLEM_PREFIX environment variable (or `--prefix` argument). To instead use
sources and directories in the user's home directory, make use of the
`--user- sources` argument. An alternative mode of configuration is by
setting the XYLEM_DIR environment variable (or `--xylem-dir` argument).
With that, only config files in that directory are used and sources/cache
folder is local to the XYLEM_DIR. This can be used by third party tools to
utilize xylem with config/sources/cache in a temporary directory.

-h, --help show this help message and exit
--version print xylem version and exit
-v, --verbose verbose console output
-d, --debug enable debug messages (overwrites the XYLEM_DEBUG
environment variable)
--print-config print the effective configuration (from command line
and config files) and exit
--no-user-config disable user config (system-wide config only)
--prefix XYLEM_PREFIX
Set the FHS prefix for finding system-wide
configurations and caches. System-wide configuration
files will be looked for in 'XYLEM_PREFIX/etc/xylem'
and cache files will be placed in
'XYLEM_PREFIX/var/cache/xylem'. The default prefix is
'/'. The prefix can also be set by the XYLEM_PREFIX
environment variable, but the command line option
takes precedence.
--xylem-dir XYLEM_DIR
Specify the path of directory to use for configuration
and caches. Configuration files will be looked for in
'XYLEM_DIR' and caches will be placed in
'XYLEM_DIR/cache'. This is intended to be used by
tools that want to create a local configuration and
cache independent from the system-wide setup. If set,
only the configuration files in XYLEM_DIR are used
instead of system and user configuration files, i.e.
XYLEM_PREFIX will be ignored. This can also be set by
the XYLEM_DIR environment variable, but the command
line option takes precedence.
```

Given the following conifg file in `~/.config/xylem/config.yaml`,

```
install_from:
homebrew: [foo, bar, baz]
installer_options:
apt:
repositories: [{sourceline: "foo bar"}]
os_options:
features: []
os_override: osx
user_sources: yes
```

the output of two example commands to illustrate the config functionality follows:

```
$ xylem resolve boost --os "ubuntu:trusty&python3" --install-from "apt: [foo, bar]" --install-from "pip: [quux, quax]" --print-config
cache_dir: /Users/demmeln/.config/xylem/cache
install_from:
apt: [foo, bar]
homebrew: [foo, bar, baz]
pip: [quux, quax]
installer_options:
apt:
repositories: [{sourceline: foo bar}]
os_options:
features: [python3]
installers:
os_override: [ubuntu, trusty]
sources_dir: /Users/demmeln/.config/xylem/sources.d
use_additional_installers: true
user_sources: true
```

```
$ xylem resolve boost --os "ubuntu:trusty&python3" --install-from "apt: [foo, bar]" --install-from "pip: [quux, quax]" --debug --verbose
overriding OS to [ubuntu:trusty]
initializing database with sources dir `/Users/demmeln/.config/xylem/sources.d` and cache dir `/Users/demmeln/.config/xylem/cache/sources`
No configs found in configured source dir '/Users/demmeln/.config/xylem/sources.d', using default sources.
Loading default source urls
boost --> libboost-all-dev
```