deploy: Add lock for status file

This is needed because some components are being installed
in parallel.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Use passwords found in conf files

The following passwords and secrets are needed for each deployment:

db, rapi, mq, vnc, cyclades, oa2, webproject, stats, and collectd.

Provide all of them in synnefo.conf file. Components can get them
via the config module e.g., config.synnefo_db_passwd,
config.cyclades_secret, etc.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Small changes in status module

Make check() and update() methods take a component as an argument
plus let the module decide whether to write the file to
permanent storage or not (it checks config.dry_run).

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Fix status in case of parallel context

Fabric's @parallel creates separate processes in order to run the
requested tasks in parallel. Thus the CofigParser object held in memory
(status.cfg) is not the same across different processes.

To ensure validity, any read/update/write
operation must be done by using the status file in permanent storage
and not the status.cfg object.

For atomicity this must be done after acquiring the corresponding
lock. Introduce @_lock_read_write for this purpose.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Create random passwords if --pass-gen

In case --pass-gen option is used, snf-deploy instead of using
passwords found in conf files, it will create random passwords
and save them in status file under the section related to context's
setup (e.g. synnefo).

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Modify conf files to include a qa setup

One could have three nodes with IPs 10.1.2.2-4, and run

snf-deploy ganeti-qa --setup qa

This will create a ganeti cluster (master.qa.synnefo.live) on
second and third node while the first one will be the development
node as well as the NFS and NS server.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Fix context issues in fabfile

Ensure that the context that each component is setup with, is
the correct one and does not include stale entries.

Use fabric's execute() method with hosts argument the proper
target nodes derived from current context. Introduce three
new decorators for this purpose:

 - with_ctx gets the current context
 - with_node updates the context's node with env.host
 - with_cluster updates the context's cluster with env.host

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Extract ArchipSynnefo from Archip

Extract configuration settings related to gunicorn from the Archip
Component and put them in the new ArchipSynnefo Component that
is required only by Pithos and Cyclades.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Restart nfs instead of exportfs -a

..otherwise on multinode setups the Mount component hungs
during the 'mount' command.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Update NS for NFS component

..so that Mount components can refer to it with its CNAME (e.g.
nfs.synnefo.live)

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: image_dir -> vcluster_dir in deploy.conf

..so that it does not get overridden by image_dir in synnefo.conf.

This dir is used to keep locally the disks of the virtual cluster VMs.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Use one shared dir instead of three

Until now we had pithos_dir, archip_dir and image_dir explicitly
exported by the NFS component. This is practically useless.

Let NFS server export only shared_dir (e.g. /srv) so that everything
under it is shared across NFS clients.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
ci: Update schema configuration files

..to comply with the latest deploy changes.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy/ci: Let extra_disk be /dev/vdb by default

In case of --autoconf the deployment is done on node1. This
node should not have an extra disk (currently ci cannot create
instances with 2 disks).

This way the only difference between snf-deploy and snf-ci conf
files is the testing_vm option.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Fixes related to the GanetiDev component

Install the deb and cabal packages required for the latest ganeti
version (master branch of official repo).

Clone both official ganeti and snf-ganeti under config.shared_dir
(i.e. /srv/src). In ganeti-qa dev node is the same with nfs
so all qa nodes will see the same src and will be able to install
it from source (make install).

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Exit if required components fail to setup

Do not try: except: during installation of the required components
otherwise deploy could silently fail and continue execution.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Introduce helper context methods

..for getting all nodes, all ips and cluster nodes of each context.

Use find_all_nodes() config method that recursively parses
corresponding section options of the requested setup.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Fix qa-sample.json template

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
deploy: Add ganeti-instance-debootstrap template

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
docs: Update snf-deploy docs

Update sections related to configuration files, and especially
the setups.conf and synnefo.conf.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>