Metadata Files and Associated Verbs

### Overview


See documentation for the changes pending here: http://jbohren-ct.readthedocs.org/en/metadata-file/index.html

This PR (wip) covers a multitude of usability improvements to catkin_tools. This is in part motivated by features which used to be supported by `rosbuild` such as context-aware building and partial cleaning of the workspace.

The key architectural feature of this patch is the creation of a metadata directory called `.catkin_tools` in the root of a catkin workspace. This directory is not only used by the verbs to determine the location of the root of the workspace, but also it is used to store persistent information between builds. The API for interacting with this information is presented below.

The information stored in the `.catkin_tools` directory is contained in independent "configuration profiles" between which a user can switch without having to create multiple workspaces or do his or her own bookkeeping of command-line arguments.

Another focus of this patch is the organization and clarification of the command-line interface to catkin_tools verbs. This involves the addition of argument groups and the re-organization of arguments between verbs. For example, several of the "workspace configuration" arguments like the specification of source, build, and devel spaces have been moved from `catkin build` to `catkin config`. This simplifies the parameters to `catkin build` and makes it easier for newer users to figure out the important command-line options.

This patch also adds several new verbs which take advantage of this metadata utility (more details for each verb in the following sections):

* **init** Create / reset a workspace (via creating / removing metadata directory)
* **config** Set or view configuration (mostly build stuff)
* **clean** Clean various parts of the catkin workspace
* **profile** Manage various metadata profiles

### Example Workflow

This PR currently enables an initialization and build workflow like so (try it yourself):

```sh
$ mkdir -p ~/my_ws/src
$ cd ~/my_ws
$ catkin config # declare this a catkin workspace with a default configuration
$ ls -a
.  ..  .catkin_tools src
$ cd src
$ git clone https://github.com/ros/catkin.git
$ git clone https://github.com/ros/rospack.git
$ git clone https://github.com/ros/cmake_modules
$ git clone https://github.com/ros/ros.git
$ pwd
~/my_ws/src
$ catkin build # call `build` from anywhere in the workspace 
<building everything...>
$ catkin clean --build --devel # clean products without dangerous `rm -rf`
$ cd rospack
$ catkin build --this # context-aware building based on working directory
<building rospack and its dependencies...>
```

Some cool context-aware aliases that this PR curently enables:

```yaml
these: build --this              # Build this package and its dependencies
this: build --this --no-deps     # Build only this package
those: build --start-with --this # Build this package and all those which depend on it
```

### Related Issues

Once complete, this PR should address the needs of the following issues:
-  (#27) *build: Add marker file to workspace root*
  - Adds the marker file.
-  (#67) *new verb proposal: catkin info*
  - Creates the verb which dumps the marker file info along with other information.
-  (#10) *build: allow execution from anywhere under the root of the workspace*
  - Uses the marker file to find the root of the workspace, and then build from there.
- (#68) *build: Add "--clear-cmake-cache" argument`
  - Adds a `clean` verb.
- (#11) *build: prevent mixing catkin_make and catkin build on same build folder`
  - Reports an error if someone tries calling `catkin build` with a buildspace created by `catkin_make`
- (#2) *move common functionality out of catkin_build and into a neutral module*
  - Moves most of the common functionality into the `catkin_tools` module

### IRC Discussions

- [#ros 2014-07-21](https://botbot.me/freenode/ros/2014-07-21/?msg=18362909&page=1)
- [#ros 2014-07-29](https://botbot.me/freenode/ros/2014-07-29/?msg=18818922&page=1)

### TODO(NE)

- [x] PEP8 silliness (use https://pypi.python.org/pypi/autopep8/)
- [ ] document all new features in RST documentation
- [x] enable building from anywhere in the source tree
- [x] add `init` verb to mark a directory as a catkin_tools workspace
- [x] add `clean` verb which deletes `buid/`, `devel/` and re-initializes `.catkin_tools.yml`
- [x] add `--cmake-cache-only` argument to `catkin clean`
- [x] add `info` verb to dump workspace information
- [x] make workspace arguments in new verbs non-positional
- [x] place metadata in individual profile directories, with a `config.yml` file under `catkin_tools` which lists the default profile
- [x] create `profile` or `configure` verb or something similar for managing profiles
- [x] ~~separate "configuration-like" arguments out of `build` verb~~
- [x] add `configure` verb to set build configuration without actually building
- [x] enable context-aware building of a package (only that package, starting with that package, or that package and deps)
- [x] add `--this` argument to `build` to build the package defined by the currently-enclosing directory
- [x] clean up refactor of `Context` class (includes separating out verb-independent functionality of `common.py`, `color.py` and `run_unix.py`)
- [x] ~~require `config`/`init` call to use `catkin build` with stored configuration~~
- [x] automatically generate `.catkin_build` marker if `build` `config` or `init` are called
- [x] ~~don't store cmake/make args (for now)~~
- [x] store cmake/make args via `catkin config`
- [x] unify context configuration
- [x] make build (config) options only apply tmporarily
- [x] replace all user-facing description of "metadata" as "profiles"
- [x] sort profiles alphabetically in `catkin -h`
- [x] remove `catkin info` verb (duplicate behavior with `catkin config`) this will be resurrected later in time
- [x] ~~remove `catkin init` verb (duplicate behavior with `catkin config`) -- replace with alias to `config --init`~~ leave in `catkin init`
- [x] ~~have `catkin config --init` create directories like source space~~ don't add directory-creating functionalities, maybe at some point later add a`catkin mkspaces` for making the source space dir
- [x] add checks to detect incompatible build configs like merged vs unmerged spaces, or install vs noinstall
- [x] add checks to detect if the user has used catkin_make or catkin_make_isolated previously in a workspace
- [x] verify that `--workspace` and  `--profile` arguments work with all verbs
- [x] make `catkin clean` display help if run with no arguments
- [x] fix `--start-with --this` so that you can run `catkin build --this --start-with foo_pkg`
- [x] document python API functions & classes
- [ ] create a consistent appearance (colors/formatting/spacing) for all verbs
  - [x] build appearance
  - [x] config appearane
  - [ ] clean appearance
  - [ ] profile appearance
  - [x] init appearance
- [x] add helpful output to `catkin init` (like `initialized empty catkin workspace in...`) and then `To build your workspace blah blah...`) 
- [x] simplify `CMAKE_PREFIX_PATH` persistence in context

### Other Ideas

- add a `catkin build --from-paths DIR` argument for building all packages in a directory
- store a list of the last set of packages which were built. if they've changed or their paths have changed since the last run, prompt the user to re-source the setup file in the develspace (or just always put this line at the end of build)

### Configuration Profile (Metadata) Details

- Metadata is stored in a `.catkin_tools` directory 
- This directory contains the following:
 - A simple instructive README file explaining what the directory is
 - A `profiles.yml` file describing the active profile (if different from `default`)
 - One or more directories containing configuration values, where each directory corresponds to a "configuration profile"
- Each configuration profile directory contains individual yaml files for verb-sepcific metadata. I.e. `catkin build` creates/updates the file `.catkin_tools/<PROFILE-NAME>/build.yml`
- When initializing the metadata directory (either automatically when calling a verb which writes to it, or explicitly using `catkin init`), the metadata module checks to see if that directory is contained within another catkin_tools-based workspace.

See https://github.com/jbohren-forks/catkin_tools/blob/metadata-file/catkin_tools/metadata.py for metadata API documentation.

### New Command Line Interface Features

```
usage: catkin [-h] [-a] [--test-colors]
              [build | clean | config | init | list | profile] ...

catkin command

optional arguments:
  -h, --help            show this help message and exit
  -a, --list-aliases    lists the current verb aliases and then quits, all
                        other arguments are ignored
  --test-colors         prints a color test pattern to the screen and then
                        quits, all other arguments are ignored

catkin command:
  Call `catkin VERB -h` for help on each verb listed below:
  
    build	Builds a catkin workspace
    clean	Deletes build, devel, and/or workspace configuration
    config	Configures a catkin workspace's context.
    init	Initializes a given folder as a catkin workspace
    list	Lists catkin packages in a given folder
    profile	Manage metadata profiles for a catkin workspace

  [build | clean | config | init | list | profile]
```

### Modified Verbs

#### catkin build

```
usage: catkin build [-h] [--workspace WORKSPACE] [--profile PROFILE]
                    [--list-only] [--this] [--no-deps]
                    [--start-with [PKGNAME]] [--force-cmake]
                    [--no-install-lock] [--save-config]
                    [--parallel-jobs PARALLEL_JOBS]
                    [--cmake-args ARG [ARG ...] | --no-cmake-args]
                    [--make-args ARG [ARG ...] | --no-make-args]
                    [--catkin-make-args ARG [ARG ...] | --no-catkin-make-args]
                    [--force-color] [--verbose] [--interleave-output]
                    [--no-status]
                    [PKGNAME [PKGNAME ...]]

Build one or more packages in a catkin workspace. This invokes `CMake`,
`make`, and optionally `make install` for either all or the specified packages
in a catkin workspace. Arguments passed to this verb can temporarily override
persistent options stored in the catkin profile config. If you want to save
these options, use the --save-config argument. To see the current config, use
the `catkin config` command.

optional arguments:
  -h, --help            show this help message and exit
  --workspace WORKSPACE, -w WORKSPACE
                        The path to the catkin_tools workspace or a directory
                        contained within it (default: ".")
  --profile PROFILE     The name of a config profile to use (default: active
                        profile)
  --list-only, --list   List packages in topological order, then exit.

Packages:
  Control which packages get built.

  PKGNAME               Workspace packages to build, package dependencies are
                        built as well unless --no-deps is used. If no packages
                        are given, then all the packages are built.
  --this                Build the package containing the current working
                        directory.
  --no-deps             Only build specified packages, not their dependencies.
  --start-with [PKGNAME]
                        Build a given package and those which depend on it,
                        skipping any before it.

Build:
  Control the build behaiovr.

  --force-cmake         Runs cmake explicitly for each catkin package.
  --no-install-lock     Prevents serialization of the install steps, which is
                        on by default to prevent file install collisions

Config:
  Parameters for the underlying buildsystem.

  --save-config         Save any configuration options in this section for the
                        next build invocation.
  --parallel-jobs PARALLEL_JOBS, --parallel PARALLEL_JOBS, -p PARALLEL_JOBS
                        Maximum number of packages which could be built in
                        parallel (default is cpu count)
  --cmake-args ARG [ARG ...]
                        Arbitrary arguments which are passes to CMake. It must
                        be passed after other arguments since it collects all
                        following options.
  --no-cmake-args       Pass no additional arguments to CMake.
  --make-args ARG [ARG ...]
                        Arbitrary arguments which are passes to make.It must
                        be passed after other arguments since it collects all
                        following options.
  --no-make-args        Pass no additional arguments to make (does not affect
                        --catkin-make-args).
  --catkin-make-args ARG [ARG ...]
                        Arbitrary arguments which are passes to make but only
                        for catkin packages.It must be passed after other
                        arguments since it collects all following options.
  --no-catkin-make-args
                        Pass no additional arguments to make for catkin
                        packages (does not affect --make-args).

Interface:
  The behavior of the command-line interface.

  --force-color         Forces catkin build to ouput in color, even when the
                        terminal does not appear to support it.
  --verbose, -v         Print output from commands in ordered blocks once the
                        command finishes.
  --interleave-output, -i
                        Prevents ordering of command output when multiple
                        commands are running at the same time.
  --no-status           Suppresses status line, useful in situations where
                        carriage return is not properly supported.
```

### New Verbs

#### catkin init

```
usage: catkin init [-h] [--workspace WORKSPACE] [--profile PROFILE] [--reset]

Initializes a given folder as a catkin workspace

optional arguments:
  -h, --help            show this help message and exit
  --workspace WORKSPACE, -w WORKSPACE
                        The path to the catkin_tools workspace or a directory
                        contained within it (default: ".")
  --profile PROFILE     The name of a config profile to use (default: active
                        profile)
  --reset               Reset (delete) all of the metadata for the given
                        workspace.
```

#### catkin config

```
usage: catkin config [-h] [--workspace WORKSPACE] [--profile PROFILE]
                     [--extend EXTEND_PATH] [--init] [--mkdirs]
                     [-s SOURCE_SPACE | --default-source-space]
                     [-b BUILD_SPACE | --default-build-space]
                     [-d DEVEL_SPACE | --default-devel-space]
                     [-i INSTALL_SPACE | --default-install-space]
                     [-x SPACE_SUFFIX] [--isolate-devel | --merge-devel]
                     [--install | --no-install]
                     [--isolate-install | --merge-install]
                     [--parallel-jobs PARALLEL_JOBS]
                     [--cmake-args ARG [ARG ...] | --no-cmake-args]
                     [--make-args ARG [ARG ...] | --no-make-args]
                     [--catkin-make-args ARG [ARG ...] |
                     --no-catkin-make-args]

This verb is used to configure a catkin workspace's configuration and layout.
Calling `catkin config` with no arguments will display the current config and
affect no changes if a config already exists for the current workspace and
profile.

optional arguments:
  -h, --help            show this help message and exit
  --workspace WORKSPACE, -w WORKSPACE
                        The path to the catkin_tools workspace or a directory
                        contained within it (default: ".")
  --profile PROFILE     The name of a config profile to use (default: active
                        profile)

Workspace Context:
  Options affecting the context of the workspace.

  --extend EXTEND_PATH, -e EXTEND_PATH
                        Explicitly extend the result-space of another catkin
                        workspace, overriding the value of $CMAKE_PREFIX_PATH.
  --init                Initialize a workspace if it does not yet exist.
  --mkdirs              Create directories required by the configuration (e.g.
                        source space) if they do not already exist.

Spaces:
  Location of parts of the catkin workspace.

  -s SOURCE_SPACE, --source-space SOURCE_SPACE
                        The path to the source space.
  --default-source-space
                        Use the default path to the source space ("src")
  -b BUILD_SPACE, --build-space BUILD_SPACE
                        The path to the build space.
  --default-build-space
                        Use the default path to the build space ("build")
  -d DEVEL_SPACE, --devel-space DEVEL_SPACE
                        Sets the target devel space
  --default-devel-space
                        Sets the default target devel space ("devel")
  -i INSTALL_SPACE, --install-space INSTALL_SPACE
                        Sets the target install space
  --default-install-space
                        Sets the default target install space ("install")
  -x SPACE_SUFFIX, --space-suffix SPACE_SUFFIX
                        Suffix for build, devel, and install space if they are
                        not otherwise explicitly set.

Devel Space:
  Options for configuring the structure of the devel space.

  --isolate-devel       Build products from each catkin package into isolated
                        devel spaces.
  --merge-devel         Build products from each catkin package into a single
                        merged devel spaces.

Install Space:
  Options for configuring the structure of the install space.

  --install             Causes each package to be installed to the install
                        space.
  --no-install          Disables installing each package into the install
                        space.
  --isolate-install     Install each catkin package into a separate install
                        space.
  --merge-install       Install each catkin package into a single merged
                        install space.

Build Options:
  Options for configuring the way packages are built.

  --parallel-jobs PARALLEL_JOBS, --parallel PARALLEL_JOBS, -p PARALLEL_JOBS
                        Maximum number of packages which could be built in
                        parallel (default is cpu count)
  --cmake-args ARG [ARG ...]
                        Arbitrary arguments which are passes to CMake. It must
                        be passed after other arguments since it collects all
                        following options.
  --no-cmake-args       Pass no additional arguments to CMake.
  --make-args ARG [ARG ...]
                        Arbitrary arguments which are passes to make.It must
                        be passed after other arguments since it collects all
                        following options.
  --no-make-args        Pass no additional arguments to make (does not affect
                        --catkin-make-args).
  --catkin-make-args ARG [ARG ...]
                        Arbitrary arguments which are passes to make but only
                        for catkin packages.It must be passed after other
                        arguments since it collects all following options.
  --no-catkin-make-args
                        Pass no additional arguments to make for catkin
                        packages (does not affect --make-args).
```

#### catkin clean

```
usage: catkin clean [-h] [--workspace WORKSPACE] [--profile PROFILE] [-a] [-b]
                    [-d] [-i] [-c] [-o]

Deletes build, devel, and/or workspace configuration

optional arguments:
  -h, --help            show this help message and exit
  --workspace WORKSPACE, -w WORKSPACE
                        The path to the catkin_tools workspace or a directory
                        contained within it (default: ".")
  --profile PROFILE     The name of a config profile to use (default: active
                        profile)

Basic:
  Clean workspace subdirectories.

  -a, --all             Remove all of the *spaces associated with the given or
                        active profile. This will remove everything but the
                        source space and the hidden .catkin_tools directory.
  -b, --build           Remove the buildspace.
  -d, --devel           Remove the develspace.
  -i, --install         Remove the installspace.

Advanced:
  Clean only specific parts of the workspace.

  -c, --cmake-cache     Clear the CMakeCache for each package, but leave build
                        and devel spaces.
  -o, --orphans         Remove only build directories whose source packages
                        are no longer enabled or in the source space. This
                        might require --force-cmake on the next build.
```

#### catkin profile {info, set, remove}

```
usage: catkin profile [-h] [--workspace WORKSPACE]
                      {info,set,add,remove,list} ...

Manage metadata profiles for a catkin workspace

positional arguments:
  {info,set,add,remove,list}
                        sub-command help
    info                Get information about the active and available
                        profiles.
    set                 Set the active profile by name.
    add                 Add a new profile by name.
    remove              Remove a profile by name.
    list                List the available profiles.

optional arguments:
  -h, --help            show this help message and exit
  --workspace WORKSPACE, -w WORKSPACE
                        The path to the catkin workspace. Default: current
                        working directory
```

Example `catkin profile info` output:

```
[moldy-crow:~/ws/hydro/rrm]$ catkin profile info     
# Information for profile: `debug`
#  Profile is ACTIVE
#  Profile has not been initialized, it contains no information.
#
# Available profiles:
#  `default`
```
Needs a space after the `,`.