Multiple Changes:

1) For YCSB (multiple backends) and Hadoop Virtual Application types, the default behavior
is now to generate a whole new data set for every new load id (i.e., the load generating
process always gets executed against a fresh, newly-generated data set).

2) Improvements on VM ssh keypair manipulation. If a user does not set a default SSH key on
his/her cloud configuration file, then CB will create an ssh keypair on the cloud using
the "cbtool_rsa" pair as source. This should make CB deployments easier/faster.
IMPORTANT: The keypair cbtool_rsa is created by calling the "install" utility.

3) For OpenStack clouds, the parameter OSK_CREDENTIALS can now receive a 4th element,
indicating the path CA bundle file to use in verifying a TLS (https) server certificate.
Therefore, the two possible values for the OSK_CREDENTIALS are:

<user>-<password>-<tenant> (for connections through HTTP)

<user>-<password>-<tenant>-<cacert> (for connections through HTTPS)

4) Improvements and fixes for the API. Now the API calls get_latest_system_data,
get_latest_management_data and get_latest_app_data can be used independently from the
"refresh_vms" API call.
Added automatic keypair creation functionality for the Amazon EC2 cloud adapter.
Merge branch 'master' of https://github.com/bowers/cbtool into bowers-master

Conflicts:
	configs/templates/PUBLIC_dependencies.txt
Merge branch 'bowers-master'
At long last, a functional implementation of the SoftLayer (CCI) Cloud Adapter
for CBTool. At the moment, only vmattach and vmdetach operations are
implemented. The code for vmcapture and vmrunstate operations are
implemented, but not tested.
Re-factored the code that propagated VApp-wide attributes down the VMs.
An example usage would be, in OpenStack, the deployment of new VApps
using tenants/projects different from the one used during the initial cloud
attachment.
Several bugfixes and improvements

1) The checking for passwordless ssh access to the JUMP_HOST (when configured) happens only
after the connection to the cloud (and creation of the appropriate SSH key pair there).
This manner a user can right after the cloud attachment failure, just create a new VM to be
the JUMP_HOST, passing the appropriate public ssh key (cbtool_rsa) by default.

2) Now, by default, logs go to /var/log/cloudbench. Each one of the multiple log files are
prefixed by the USERNAME that launched cloudbench (e.g., cbuser_operations.log), a measure
to allow multiple user to run CB on the same host simultaneously.

3) When an non-existing cloud model is indicated during a cldattach (e.g., "cldattach xyz
TESTCLOUD"), provide an appropriate error message.

4) The default cloud configuration file (i.e., USERNAME_cloud_definitions.txt) can now
reside within a subdirectory inside the configs directory.

5) During the post attachment operations on the VM, a new separate step, before the copy
of the code is rsynced and the post-boot script is run remotely, is executed. In this step
CB just checks to make sure that the ssh is really inject (or already inside) the VM by
remotely running "/bin/true"

6) Fixes an error during the CB startup where the private rsyslog configuration file was
deleted, but the rsyslogd process is still running.

7) The configuration parser now supports nested INCLUDE statements
Multiple changes:

1) Attributes in configuration files now support multi-line values by using the "+=" sign.
For instance an excerpt like the below is now perfectly legal:

...
MY_MULTILINE_ATTRIBUTE = first line
MY_MULTILINE_ATTRIBUTE += second line
MY_MULTILINE_ATTRIBUTE += one more

2) In an effort to make CB more "self-documenting", and the workloads and cloud adpaters
more self-contained, their specific configuration files are being moved. For instance,
gone are the *_application_instances.txt files in configs/templates/ dir. Now, each
Virtual Application type in scripts/<vapp type>/ has a virtual_application.txt file that
contains all the configuration for that particular VApp. The configurations now include a
brief text describing the Virtual Application, shown through the "typeshow <vapptype>"
command.

3) Record the time zone/time difference between VMs and the CB Orchestrator Node (ON).
This information is highly relevant for at least two situations: a) CB ON and VMs just
happen to be actually located in different geographical regions or b) the timezone on the
VMs is not properly set. In CB, the python call (time.time()) is used to create timestamps.
This call is supposedly *defined* to be 1970, 1, 1 in UTC, but in at least one scenario
it could be experimentally verified to be using the localtime. In order to provide the
experimenter with the correct information to allow him to fix any potential errors (due to
timezone shifting) during the production of plots, now the "utc_offset_delta" is provided.
It is provided both as an attribute on the Object Store and as a column in the .csv files
extracted from the metric store.

4) In cb_common.sh, Ganglia (gmond) start/stop operations are split in their own separate
functions, making it easier to restart gmond during the Virtual Application's lifetime.
ProvisionVMs.py Used to Dynamically add workloads
Based on established SLAs
Small bugfix for Redis automated install on PUBLIC_dependencies.txt
WIP - Provision VMs for automated workload development
Remove the table drop... Caused issues
Merge pull request #49 from jtaleric/master

WIP
Important (and big) commit

1) While checking for VM boot completion by running a passwordless ssh command going through
the JUMP_HOST, make sure that the ssh command includes the -o ConnectTimeout. Without this
option, the first time the ssh command is attempted through the JUMP_HOST will cause it to
block until the connection times out (since the VM might not have been even created yet),
which is normally set to 120s. Now the ConnectTimeout will be set to half of the time
specified in the VM's attribute "update_frequency".

2) In case any of the methods used to check if a VM is ready/started (to boot) or to check
if the VM completed its boot takes longer than the VM's attribute "update_frequency" to
return from the polling attempt, log a warning message informing the user of this fact.
This was prompted by the scenario described on item 1) (i.e., an ssh timeout that is much,
much longer the "update_frequency").

3) CB now has rudimentary support for SLA tracking. Great care was taken to make sure that
this new capability is non-intrusive to users that prefer to have their SLA tracking done
from the outside. Two kinds of SLA tracking are possible, provisioning and runtime.
If a VM is deployed with the attribute "sla_provisioning_target" set, then CB will check,
at the end of the provisioning, if the total time taken to provision is smaller than the
specified value. It then adds a new attribute - "sla_provisioning" - with values "ok" or "violated"
to the VM. In addition to it, CB will also add the VM to a "view" (i.e., a list of objects
that share a common attribute) that can be queried at any time.
Example for SLA provisioning
The command "aiattach nullworkload tinyvm_sla_provisioning_target=5" (the API equivalent of
this command is "api.appattach("TESTSIMCLOUD", "nullworkload", temp_attr_list="tinyvm_sla_provisioning_target=1")
") will cause CB to automatically check if the total provisioning time is smaller than 5,
and then place the VM on a list of VMs with violated or obeyed SLA provisioning. These
lists (also known as "views" internally by CB) can be accessed by the command viewshow.
A list of VMs who violated the provisioning SLA can be obtained with "viewshow vm sla_provisioning violated",
while a list of VMs who fulfilled the SLA can be with "viewshow vm sla_provisioning ok".
The API equivalent of these commands is "api.viewshow("TESTSIMCLOUD", "VM", "SLA_PROVISIONING", "OK")".
Example for SLA runtime
The command "aiattach nullworkload sla_runtime_target_bandwidth=100-gt" will cause CB to
automatically check if the reported application bandwidth is smaller than 100 (that is
controlled by "gt" or "greater than" and "lt"  or "less than"), and then place the VM on a
list of VMs with violated or obeyed SLA runtime. These lists (also known as "views"
internally by CB) can be accessed by the command viewshow. A list of VMs who violated the
SLA can be obtained with "viewshow vm sla_runtime violated", while its a list of VMs who
fulfilled the SLA can be with "viewshow vm sla_runtime ok". The API equivalent of these
commands is "api.viewshow("TESTSIMCLOUD", "VM", "SLA_RUNTIME", "OK")".
Small bugfix for configure/install (redis-server version not being correctly detected)
Changes:

1) (For OpenStack clouds) During initial cloud attachment, if the value of the VM_DEFAULTS
attribute JUMP_HOST is "$True", CB will create a VM named "cb_jumphost", attach
a floating IP to it, and then automatically use it as the jumphost. If, instead of
"$True", the value of the JUMP_HOST attribute is an already existing IP address, then
the creation of the "jumphost" VM will be bypassed.

2) Added proper description (plus additional code cleanup) for Virtual Workloads
cassandra_ycsb, coremark, ddgen, filebench, hadoop and nullworkload.

3) Introduced a new regression test targetting "real" Virtual Application instances
on a "real" (OpenStack) cloud. Up until this point, all regression tests only covered
a simulated cloud, and thus no actual testing of the Virtual Application load generating
code was performed. With this commit, all Virtual Application instances (starting
by the ones described in #2) will be tested on a real OpenStack cloud before commit.
1) Added proper description (plus additional code cleanup) for Virtual Workloads
redis-ycsb and hpcc

2) The functions service_restart_enable, and services_stop_disable were re-written to support
services in Ubuntu and RHEL with multiple different mechanisms (e.g., systemctl, service/rc.d,
service/chkconfig)

3) Improvements on "stats" API call. First of all, the data returned by the API is a JSON
object, instead of a list of lists (much easier to parse and use). Second, the API call
now supports two options: "object_type" (default is "all") and "output" (default is "print").
When calling stats from the API, the user can specify output to be "json" and then he will
just receive back a JSON object. The default value "print" is used by the CLI, where
printing a properly formatted response is important. The second improvement is the capability
to, if allowed by the cloud, to include the number of VMs reported directly by it on the
list of counters for the "VM" object.

4) Implemented a primitive "backoff" algorithm for cases where the instance boot completion
is checked through ssh over a jumphost. After one third of attempts is made, the default
timeout for the ssh (which is half of the value of the "update_frequency" for each VM)
will doubled, being doubled again after two thirds of the attempts are made.

5) Updated automated installer. Now the installer requires the specification of a "role"
("-r", "--role") to the node where it is run (possible values are "orchestrator" and "workload").
The automated installer now takes care of deactivating firewalld, iptables and ipfw on
workload VMs.

6) Refactoring on the core "operations" code. a) Now, a particular instantiation
of a cloud operations class, for a given user and experiment identifier is stored on
shared dictionary. This will ensure that this instantiation can be re-utilized across
multiple operations, reducing the need for repeated dynamic module imports. b) A new
shared variable, "expid", caches the result of query to the "time" global object on the
Object Store. This means a reduced number of queries across multiple operations.

7) Support for PubSub through both CLI and API, including setting of timeouts for
subscribers. The command "msgpub VM mychannel mymessage" (the API equivalent is
api.msgpub("TESTSIMCLOUD", "VM", "mychannel", "mymessage")) will publish the message "mymessage"
on the channel "VM:mychannel". The command "waiton VM  mychannel mymessage 120" (the API
equivalent is api.waiton("TESTSIMCLOUD", "VM", "mychannel", "mymessage", 120)) will cause
the CLI/API client to block waiting for the message "mymessage" to be publish on the channel
"VM:mychannel" with a timeout of 120 seconds.

8) A new VM attribute, "comments", is now automatically stored on the Metric Store. This
is very handy if, during an experiment, an user wishes to "annotate" certain VMs for later
analysis.

9) In OpenStack clouds, extract the total time to complete the provisioning request directly
from nova API, instead of relying on the value determined by CB (e.g., through polling).
Since the total time reported by CB and by nova API differ, the actual time spent by CB
waiting for the provisioning request to complete is added to VM's "comments" attribute.
Bugfix for the CB API call "stats" when used against an OpenStack cloud.
Update ProvisionVM work
Merge pull request #50 from jtaleric/master

Update ProvisionVM work