deploy: Add config module

This module reads the configuration and includes
several methods for extracting info. It gets
initialized with the command line arguments.

It reads the DEFAULT section of synnefo, vcluster, and deploy conf
files and evaluates their options in a sense that once it gets
imported, one has easy access to generic configuration variables,
e.g. config.synnefo_user.

On the other hand the setups, nodes, ganeti and packages conf files
are proper ConfigParser files with sections and default values.

nodes: All info needed for each node.
ganeti: All info needed for each cluster.
setups: All role-nodes mappings for each setup and each cluster.
packages: All special package versions for each OS.

The get_info() helper method gets a node or cluster identifier
(e.g. node2, ganeti1) and returns a FQDN object with all related
options as attributes.

The get_package() helper method is used to get the special version
(if any) of a package.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Add status module

This is used to keep track of the deployment status. The status file
is kept under the 'state' dir. The check() method is used to find
conflicting components or the ones that have already been installed.
The update() method adds an entry in the node's section. If in dry-run
this file is not written back to permanent storage (kept in memory).

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Add new base module

This defines the base classes 'FabricRunner', 'CompnentRunner' and
'Component'. These classes will be used by the classes defined in
the components module.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Add context module

This module will be used as a common execution context
just like fabric's env. A context is basically the node, role,
cluster, and setup that a components gets installed. The new
Context class is a configuration snapshot for the given context.
It exports helper methods to get the node's and cluster's info, etc.
This class is used to initialize the target component class.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Add roles module

This is used to get the target component for each node.
It has a role-component mapping and should be used like:

NS = roles.get(constants.NS, ctx)

where the ctx is a Context() with proper node, role, cluster, setup.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Add vcluster module

..with the relevant helper methods for creating a local
virtual cluster with KVM. snf-deploy base script will import
this module and use it if vcluster command is requested.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Cleanup 'lib' module

Remove unused Env, Status, Conf classes. Host and Alias classes have
been substituted by FQDN. The latter will be used for Node and
Cluster info.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Make snf-deploy script use new modules

Use config, context, status and vcluster modules.

The main method of snf-deploy script will initialize them with
the args passed. Additionally make some minor changes related
to the command line:

 - snf-deploy all -> snf-deploy synnefo.
 - add --setup option
 - add --cluster option
 - use --<name>-dir for template, state, package dirs

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Refactor components module

Until now components were simple objects decoupled from fabric
execution. Their methods were just returning a list of bash commands
which utils module was executing in the correct context.

This patch changes this rational. Specifically:

A Component() gets initialized with an execution context that is a
configuration snapshot of the target setup, cluster and node. A
component implements the following helper methods: check, install,
prepare, configure, restart, initialize, test. All those methods
will be executed on target node with this order during setup.
Additionally it implements admin_pre, and admin_post methods which
invoke actions on different components on the same execution context
before and after installation. For example before a backend gets
installed its FQDN must resolve so we have to run some actions on
nameserver and after installation we must add it to cyclades
(snf-manage backend-add on the cyclades node).

Component() inherits ComponentRunner() which practically exports the
setup() method. This will first check if the required components are
installed, will install them if not, update the status of target
node.

ComponentRunner() inherits FabricRunner() which practically wraps
basic fabric commands (put, get, run) with the correct execution
environment.

The fabfile practically gets the target nodes for each role from the
configuration and uses the roles module that knows the
role-component mapping to get the target component with the proper
execution context. Then it just invokes the setup() method.

Each component gets initialized with an execution context.
For global wide options uses config module to get the required
info. The context provides node, cluster, setup related info.

We introduce some helper decorators that wrap methods of Component
class and update its execution context (self, self.ctx):

- parse_* will execute the wrapped method and parse the output
  The desired vars (user_id, backend_id, user_uuid, etc.) will
  be stored in the execution context and made available to other
  methods

- update_admin will initialize the admin roles (NS, Astakos,
  Cyclades, etc.) of the current context and make the available
  under self.NS, self.ASTAKOS, etc. These have the same execution
  context of the current components besides the target node
  which gets derived from the corresponding config.

- update_cluster_admin will initialize the cluster's admin role
  and make it available under self.MASTER.

- run_cmds will run the list of returned commands in the target node

The update_ns() method of NS component will get the
self.ctx.admin_fqdn and invoke nsupdate accordingly.

The fabfile now is responsible to find the target nodes for each
execution context. It practically gets the corresponding component
for each role and invokes fabric's execute() with hosts the target
nodes. Some things can be done concurrently (setup of each VMC
node). We use the fabric's @parallel decorator.

Refactor packages configuration

Change wheezy.conf to packages.conf

Each section denotes the node's os.

If a package is not found in packages.conf the default
version of the node's distribution is used.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Use archipelago related components

Make Cyclades and Pithos require PithosBackend and Archip.
Make VMC require also ExtStorage and ArchipGaneti.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Make ganeti1 a synnefo backend

This is done by making True the synnefo option in the ganeti1
section. This way the VMC component will require some other
components (Image, GanetiCollectd, ArchipGaneti, etc.) that are
needless in a regular Ganeti cluster.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy/ci: Update conf files for ci schema

..and invoke `snf-deploy synnefo` instead of `snf-deploy all`.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
docs: Update snf-deploy docs

..to reflect the latest changes.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Use cname instead of fqdn

..for astakos, cyclades, pithos, cms, db, mq, nfs.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Add simplejson as requirement

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Install blktap in Achip component

Currently latest version of archipelago uses blktap instead of
xseg. Install official blktap-dkms and our custom blktap-utils
for this purpose.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>