Fix up installation guide

 + added a new page dedicated to Sahara configuration

Partial-Implements: blueprint update-docs-icehouse
Change-Id: If9a65ab165e1ecf52f0e71d7f1dbedeea1661470
(cherry picked from commit db8eb67d5e6539c00791da0f83fb442f0b462e15)
Add examples of upstream files that we should not change

Partial-Bug: #1277584
Partial-Implements: blueprint update-docs-icehouse
Change-Id: I37b25b4cec49dce9f3237953fe84f953c63eb849
(cherry picked from commit 083003cf6a30ba3a73b30131ce8f3ebd1c09ae3d)
Updating the setup development environment docs for icehouse

Changes
* rewording title paragraph to be more descriptive of the document
* adding a link to devstack.org in the "... inside DevStack" section
* adding a sentence about developing on OSX to the "... inside DevStack"
section
* changing the toctree link to the OSX VM install doc to be inline
* minor grammar cleanup on section titles
* adding a header paragraph to the ui dev environ page
* rewording the "... as part of DevStack" section to be similar to dev
environ doc
* minor cleanups to ui dev environ page
* correcting sahara-dashboard location in /opt/stack

Partial-Implements: blueprint update-docs-icehouse

Change-Id: Ideed96512ee7f28280577f2520bacc620952e305
(cherry picked from commit a693dc0584851685885bdb572c1d0eb0a847a8e3)
Added rackawareness to Hadoop 2 in vanilla plugin

* Now Hadoop 2.3.0 in vanilla is able to push topology data for
  datanodes and nodemanagers
* Only 3-level topology (rackawareness) pushes to nodes. Why 4-level is
  not? Because this is not implemented in YARN yet (see YARN-18 and YARN-19):
  https://issues.apache.org/jira/browse/HADOOP-8468
* Even if admin configures 4-level topology configs Sahara will force
  down to 3-level.

Co-Authored-By: Sergey Reshetnyak <sreshetniak@mirantis.com>

Partially implements: blueprint vanilla-plugin-hadoop-2

Change-Id: If5b93dde1f6aaaf619def60634da8c8819a3c75a
(cherry picked from commit 066cc7b8b43075907cd293e304aa6a0e16b3a11c)
Removed possibility to run job w/o Oozie

* Added a validation function which checks if cluster contains oozie
  service in its node groups
* Added unit-test covers both valid and invalid cases when cluster
  contains and doesn't oozie service in attempt of job execution
* Fixed small issue in unit tests in validation utils where pathced
  api.get_cluster function had never been unpatched.

Change-Id: I5c5de6cfb1812ff6795797df63b8ec0292395fd1
Closes-Bug: #1291649
(cherry picked from commit cd583756a0d7befccb71f82059d4b7b8e1718d12)
Update EDP requirements for hadoop v2

Change-Id: Ib59a6fb200e0e47a3070c2509ae7e84c8ef1fd15
Partial-Implements: blueprint update-docs-icehouse
(cherry picked from commit fce2bb6d5ab97f9426e5321921f62438b27daa89)
Fix big job binary objects in mysql

Fixes bug: #1302579

Change-Id: I3acc13864c41127c341875e350b4611ff7035e29
(cherry picked from commit 3e317ed73d9e2188ad2693157a75afcfcacbb6e0)
Add a custom filter method to scan wrapped dict results

Basic filtering is available via Resource._filter_fields and
Resource._children but partial filtering of fields is not supported.

This change allows a Resource to define custom sanitization methods
for individual fields. The specified method is applied after the
field has been generated.

For nested structures, particularly when elements are not Resource
types, it is easier to scan the field after generation and remove
specific elements than it is to modify the recursive descent already
implemented in to_wrapped_dict().

JobExecutions are filtered with this mechanism to set swift credential
values to empty strings and to remove the 'conf' section reported
for running Oozie actions.

Closes-Bug: #1300291
Change-Id: Ifb0b99a3e13d40306139b0d0021aead69e870205
(cherry picked from commit 3367d9c3632629b900b618c80261d8b6de8086ee)
Update links for vanilla images in doc

* Update links for vanilla 1 plugin images
* Add links to Hadoop 2 images

Change-Id: I71cf69d424c7b53082d0980479967a8f4128a167
(cherry picked from commit 4ff36b320eb2cdf7146197913f41efbc917c5e1b)
Minor fixes to Sahara UI Installation Guide

 * removed item #7 - it is not needed and is misleading
 * rephrased clarification for AUTO_ASSIGNMENT_ENABLED
 * renamed 'Nova Network' to 'Nova-Network' - seems like the later is
   the only name which is used

Partial-Implements: blueprint update-docs-icehouse
Change-Id: Ibb177820da8d44e1f70a8fd7273dadd57e81ad1e
(cherry picked from commit 253ab59a3bfb0066b46935edc18cb252f962877b)
Doc's update for integration tests

* Description of tests was changed and complemented;
* List of tests to each plugin and its version was added.

Change-Id: I8de087d745cb83dfc6da67d1bc5036cacd1b6b80
(cherry picked from commit 3a00dd05e3a1df5ecfc3e0423c439db4d4153eda)
Cleanup of docs for integration tests

Some parts of text from https://github.com/openstack/sahara/blob/master/sahara/tests/integration/README.rst
are displayed not correctly. This patch corrects that.

Change-Id: Iec7b6b51530d96739f640a3fdfe1e894a67ed4a9
(cherry picked from commit 1fa25cd90a4997cc01356dd51b3d2d9bd31e1fbd)
Fix up Sahara UI installation guide

Partial-Implements: blueprint update-docs-icehouse
Change-Id: I1840f87f931812ecf3327661f58b56e6d5920cb6
(cherry picked from commit 5b3f47ee1f159792a7e5a6553a862bf323a8a875)
REST API 1.1 corresponds to Icehouse as well

Previously it mentioned that it corresponds to 0.3 only

Change-Id: I8c870f27e81422377a0c1b070ae4ecc3d8b0a776
(cherry picked from commit f2232588e211cac58ea5386505588f5929cee46a)
Updated validation section for Vanilla Plugin

Change-Id: I20232d5a8fec8e485ccb81640150c39b3ce8da79
Partial-Implements: blueprint update-docs-icehouse
(cherry picked from commit 06531df98e399a9bf9f39e78ba9434a292e4bc65)
Change links to images in Quick Start guide

 * the old links point to Savanna 0.3 images
 * + fixed a small typo

Change-Id: Icce1358fcbaad5b73e82f78d9c86d5285afba13b
(cherry picked from commit 2cccc06b7506d9289bf9164171ba5bf68addee14)
Updated hdp_plugin features to align with current capabilties

* Minor re-write of Swift integration feature
* Extracted plugin capabilities into an easy to update/view table

Co-Authored-By: Alexander Ignatov <aignatov@mirantis.com>

Partially Implements: blueprint  update-docs-icehouse

Change-Id: I4014e9549467b4ad3d3fe888aad98e1079796013
(cherry picked from commit 158e897e1f185d9b17aec31fcfeff5b7c7d190cd)
Saharaclient must be installed for UI to work in dev environment

Since in dev install guide we create a symlink to saharadashboard instead
of installing it, its dependencies are not installed automatically. So we
need to manually install saharaclient. Thanks to Yaroslav for finding
that issue.

Change-Id: I319ff3aaf6df86e473d4b0206f9769938074c6f8
(cherry picked from commit 01cc56d25d04a2f5a7b0c1ff32cf11b16c06edd0)
Fix up DevStack guide

For some reason DevStack instructions were on page
"Setup VM for DevStack on OSX" which is rather weird. I've shuffled the
instructions and section names to make them more consistent.

Change-Id: Ib48808b202c9a640f1e673611bd9f521b03368c7
(cherry picked from commit f39726cc4ddff12faf0db87bd96d6cb22220c8c4)
Fix storing binaries in Swift

 * Swift does not work with Keystone API v3, so we need to always provide
   auth_url for API v2 to Swift.
 * We incorrectly parsed swift URL, which resulted in Sahara trying
   to get binaries from containers with '.sahara' suffix, while the
   suffix should be stripped.

Closes-Bug: #1305210
Change-Id: I76dee756165e089bc408e584f3aa476c45ed6067
(cherry picked from commit 9ad556f5984510b843b2b52c127af2fa804d98cf)