Added several checks on deleted cluster to prevent error logs

Change-Id: I8f867d7cc89abbdf407a170344be062f00ed50a2
Closes-Bug: #1274739
Closes-Bug: #1286391
Use in-memory sqlite DB for unit tests

Currently unit tests use temporary file for DB. This causes
several problems:
1. temp files need to be cleared even after crash
2. temp folder need to be properly configures in OS
3. temp folder could be modified in code (e.g.
   oslotest.base.BaseTestCase)

Using in-memory DB fixes all these problems.

Partly implements: blueprint oslotest

Change-Id: I8be76efdb2fa56f5e8865dcafbc3c240a6be0f64
Minor EDP refactoring

* Made run_job method more structored.
* Got rid of workflow creator object (we don't use factory externaly)
* Created unit tests for extracted methods

Change-Id: I4c45cd0d583110c8a5d610f5a69d9d9f9a178f8d
Added secondary name node heap size param to vanilla plugin

* added check to integration tests for vanilla 1.2.1
* there is no params testing in IT for vanilla 2+

Change-Id: I75e5df9c4a7a6e644ef85d6142112c3a378e0b3b
Closes-Bug: #1217245
Added jobhistory address config to vanilla 2

Change-Id: Ia04c2e4901cb03bac3487960092fd603c4c32e12
Closes-Bug: #1328675
Remove usage of remote from HDP Instance constructor

When Sahara is run in distributed mode, the Instance object is used in
sahara-api context. You can see the full stack trace in the bug description.
Here it is in the simplified form:
  * User requests HDP cluster scaling
  * Scaling validation (in HDP plugin)
  * HDP Instance objects are constructed
  * Sahara's instance.remote() is called inside the constructor

The remote is unavalable in sahara-api context and that causes the issue.
Moving invokation of instance.remote() out of constructor fixes it.

Change-Id: I0d09c7a19e4df7fce948994fb21028fe147d2782
Closes-bug: #1325989
Fixed remote call in external HDFS configuration method

Before the change code was failing with exception. Probably
parentheses were not added during remote engine refactoring for
heat engine preparation.

Blueprint to create integration test: it-external-hdfs

Change-Id: If9aef3be0cc88133d6e20c9e3ffb1ef75e16443c
Closes-Bug: #1329079
Merge "Use in-memory sqlite DB for unit tests"
Merge "Added jobhistory address config to vanilla 2"
Merge "Added secondary name node heap size param to vanilla plugin"
Update job execution status on cluster deletion

On cluster creation Sahara now does:
1. remove cluster_id (as before)
2. don't change status and end_time for finished jobs
3. change status to 'FAILED' and update end_time for not finished jobs

Change-Id: Ia26cb20565d50af71032d0e739c272b3ade0e8ee
Closes-Bug: #1329109
Fixed status update for job execution

Fixed wrong status update for job execution introduced at
https://review.openstack.org/#/c/97637/

Change-Id: I4067667c19c8fd2e3eea72b2c1d64bba6016c19f
Closes-Bug: #1306720
Closes-Bug: #1317205
Fixed /etc/hosts update for external hdfs

Before the change /etc/hosts were growing infinitely

Change-Id: I303c013fc0fa85f1826aa60106846a3b7b214be3
Closes-Bug: #1326114
Sync up oslo log module

Main changes:
* removed annoying warning about RFCSysLogHandler.super
* improved password screening
* default log level for oslo.messaging

Closes-Bug: #1306559
Change-Id: I65ebf85e648cd5a1388ea77f6036d1f79f5b335b
Migrated unit tests to testtools

Main changes:
* testtools checks that base setUp is called
* assertRaises doesn't return context
* testtools inherits from unittest (not unittest2) and doesn't have
  specific compare functions in py2.6

Notes:
* Integration tests will be tracked separately (bp: it-testtools)
* Migration to oslotest will be tracked separately (bp: oslotest)
* bug #1325108 is out of scope of this CR

Implements blueprint: ut-testtools
Closes-Bug: #1179009

Change-Id: I85f0ea1aa972ff68ddd1817ec936a8c26312f041
Hided not found logger messages in unit tests

Hided messages "No handlers could be found for logger ..." from
unit tests output.

Change-Id: I2f997a589407c45a60848297513da95f847bd66c
Closes-Bug: #1264005
Updated from global requirements

I6ec51fa400a037cad06e4714dbe59a8627048635 is incorrect due to the
hacking version update that needs new checks manual disable.

Change-Id: Ica0c8d6972c4128f92505b9988a523cf244ee4a6
Merge "Fixed remote call in external HDFS configuration method"
Merge "Fixed /etc/hosts update for external hdfs"
Merge "Updated from global requirements"