Support for Neutron Traffic Steering service plugin

This commit adds suport for the Traffic Steering Abstraction blueprint.

Implements: blueprint traffic-steering-abstraction
Change-Id: Ib7ba2b7ece897b27569c40918c82c5c5ac02e7ce
add is_keystone_enabled

This is missing, and makes logic much harder for the grenade use
case.

Change-Id: Ib6670096d190e69273256b87e6a67e2b46945a0d
(cherry picked from commit 4f3b66f538937cf0c7f9fdc97e5472f4c77bbe69)
debian: do not enable grizzly repository from gplhost

There is not reason to enable this additional repository anymore.
All the important packages are either in wheezy or wheezy-backports.

Change-Id: I47af16ba6b507fc929b0d2948c757f4060ba93a5
(cherry picked from commit 5420992a2cad4c0a23e436f5f416011f46c97ecd)
Fix Swift config for Ceilometer usage

Ceilometer filter was previously removed from the Swift
configuration to avoid useless Ceilometer logs in the Swift
ones. This was fixed by setting only warning log level for
this Ceilometer part of the pipeline to keep only important
ones.

that is needed in stable/icehouse to allow swift+ceilometer
in devstack and devstack gate (otherwise grenade job, that is
using stable/icehouse devstack, won't pass the gate)

Change-Id: I8c41355bb98dbf3bb59ec792221b05ea936086b7
Partial-Bug: #1294789
(cherry picked from commit eedfdee060b17d38b14628b7e706967cb15a35af)
Adds support for Hyper-V image formats

Adds support for VHD and VHDX image formats, including gz compression.

Note: variables naming case changed to match the target branch

(cherry picked from ca8239449fbdbc3a73f691683be2ec9d4a941ff6)

Change-Id: I9b143c9883b947397b435a671f13703f78019d23
Closes-bug: #1353726
Don't set tunnel_types on VLAN network type

The current code sets tunnel types for the neutron agent
even when the network type is 'vlan'. This results in the
agent not starting because 'vlan' is not a valid tunnel type.
This patch changes it so tunnel_types is not populated when
vlans are used.

This bug just now surfaced because of commit
8feaf6c9516094df58df84479d73779e87a79264 requiring Q_ML2_TENANT_NETWORK_TYPE
to be set to avoid being set to the 'local' type.

Conflicts:
	lib/neutron_plugins/ml2

Change-Id: If2bc57b36dad2bfb34df573581acce176604812e
(cherry picked from commit e3cfbf1580d4e0b77191a2f4bcb61bde9f8d9815)
Fix tools/xen/functions for bashate errors

Change I205d675ff231c0403dd135eacc8d57b45317859f modified the matching
for bashate and it has started running on this file, whereas before it
wasn't.  This fixes the errors it is now picking up.

(cherry picked from commit 62cae13d7e7e13102a02aa385ae63b8e2afb7c83)

(We need to squash these 2 patches together to pass tests)

Use tox for bashate testing

Create a tox.ini file to install bashate into a virtualenv for running
tests. It can be run with a command such as "tox -v -ebashate". The find
command being used is equivalent to the default list of files discovered
by bashate plus the exclusion of the .tox/ directory. Also add the .tox/
directory to the .gitignore file.

Conflicts:
	.gitignore

Change-Id: I7876b5c94d90f15e3b0228999c0750bfbebcaa1c
(cherry picked from commit 0c256abd766265b8fdb11d354be7115c05bcaeb5)
Add docs environment to tox.ini

As per the governance resolution https://review.openstack.org/119875,
projects should have a docs environment in their tox.ini testing
interface. Add one.

Change-Id: Ib3548bc23713e120089ae29b5c4e851de48cf8d3
(cherry picked from commit 81fa6c7b4bd5a1d57cc095395b9d2ba3eeb6c20f)
Run processes without screen

This introduces new run_process() and screen_service() functions and sets the
groundwork to change how DevStack starts services.  screen_service() is simply a
direct call to the screen portion of the old screen_it() function and is intended
to run commands that only need to run under screen, such as log file watchers.

run_process() is a replacement for screen_it() (which remains until all of the
services are updated).  The usage is similar but requires updates to every current
screen_it() call to remove everything that requires the command to be interpreted
by a shell.

The old run_process() and _run_process() functions are still present as
old_run_process() and _old_run_process() to support the deprecated screen_it()
function.  These will all go away in the future once all services have been
confirmed to have been changed over.

There is a similar new set of stop process functions stop_process() and
screen_stop_service().  The old screen_stop() will also remain for the deprecation
period.

As an initial test/demostration this review also includes the changes for
lib/cinder to demonstrate what is required for every service.

I included the scripts I used to test this; tests/fake-service.sh and
tests/run-process.sh are quite rough around the edges and may bite.  They should
mature into productive members of the testing ecosystem someday.

Conflicts:
	functions-common
	lib/cinder

Change-Id: I03322bf0208353ebd267811735c66f13a516637b
(cherry picked from commit 3159a821c2e0662278746c5311e187bcfd4d8b75)
Replace screen_it() with run_process() throughout

run_process will use screen if USE_SCREEN=True (the default),
otherwise it will simply start the requested service. Therefore
wherever screen_it used, run_process can be instead.

Where stop_screen was found it has been replaced with stop_process.

A tail_log function has been added which will tail a logfile in a
screen if USE_SCREEN is True.

lib/template has been updated to reflect the use of the new
functions.

When using sg the quoting in run_process gets very complicated.
To get around this run_process and the functions it calls accepts
an optional third argument. If set it is a group to be used with sg.

Conflicts:
	functions-common
	lib/ceilometer
	lib/keystone
	lib/neutron
	lib/sahara
	lib/zaqar
	stack.sh

Change-Id: Ia3843818014f7c6c7526ef3aa9676bbddb8a85ca
(cherry picked from commit 2f27a0ed3c609bfcd6344a55c121e56d5569afc9)
ceilometer compute agent not started correctly

command isn't properly parsed after switch to run_process. this
patch corrects the regression

Change-Id: Ib28d4bbf6f150854fc5aa04e3fa4eed92005ce19
Closes-Bug: #1370673
(cherry picked from commit bdeea69d726eca7a899b1d2225d213724b31df22)
Add identity_feature_enabled.xml_api config

This change allows us to continue to test identity XML support for
stable icehouse. XML support in Keystone has been deprecated and staged
for removal in Kilo.

Change-Id: Ibb402640c5b744f1ea38ce548994079996c72508
Reserve Keystone ports from the ephemeral range

Reserve Keystone ports from the ephemeral range as early as reasonably
possible in the fixup_stuff.sh process to reduce the likelihood that the
port will be in use. This does not completely resolve the issue
where Keystone's IANA assigned port falls into Linux's ephemeral
range, but this should reduce the occurrences. The default ports
are 35357 and 35358.

Change-Id: I8cfb53d8f90c1ff1fb1083c59fefabca3d14323b
Partial-Bug: #1253482
(cherry picked from commit 6cae83efd72133adae891da0665f51f526705592)
set xml_api flag for Network tests

XML api support is being removed in Kilo, so we need to enable here to
ensure that the API is still tested in this stable release.

Change-Id: Iaf0cb93fda607c62911c57132eced14a33f94f97
Move to download.fedoraproject.org

download.fedoraproject.org is the mirror system, while
dl.fedoraproject.org is just one system. We have seen performance issues
with dl.fedoraproject.org and using download.fedoraproject.org was a
recommended as a solution by the folks at #fedora-admin.
Local testing on a rackspace server with dl.fedoraproject.org got about
70K/s while download.fedoraproject.org got over 1M/s

Note this is not a cherry-pick of I318d8844cfcfbc42c1e552e443c6b793d20f321f, because that wouldn't work

Change-Id: Ic68efcef0ed62f7867114d6992aee1610d6e8eb9
Related-Bug: #1383928
Enable Swift's newer feature of container-sync

Newer version of container-sync feature is introduced in Swift ver. 1.12.0.

The spec:
http://docs.openstack.org/developer/swift/overview_container_sync.html

Before this commit, Devstack does not configure any realm used in
container-sync, therefore this feature does not work.
To test this feature in CI system, moreover to show the sample
configuration of realms, Devstack now edits realms configuration file.

Change-Id: I9f1e3224403e08e725a989162729470357fe90b0
Closes-Bug: 1378646
(cherry picked from commit d03915f9c08ff91c5d39f8cadb064718079bdb4b)
Set xml_api_v2 for Compute tests

Nova XML V2  api support is being removed in Kilo, so we need to
enable here to ensure that the API is still tested in this stable release.

Change-Id: I85ca8f5a7140a2637e64e980015ee30b2a23a46d
make run_tests.sh run devstack unit tests

run_tests.sh was created as an entry point for bashate. We stopped
using it now that we know how to run bashate directly in the gate.
We lost running an unrelated bashate test when that happened.

Devstack does have unit tests. We don't run them. We should.
Especially when working on things like the LIBS_FROM_GIT which is
sufficiently tricky that we really need to add unit tests to ensure
that we don't break it.

Conflicts:
	run_tests.sh

Change-Id: Ife067855569e2eae4c085471d326e8086de37332
(cherry picked from commit 9f20ea13826234ac2c508ea1d50a0822b68d42dc)
use released library versions by default

This patch provides a new path for installing libraries in devstack so
that it's possible to either test with upstream released libraries, or
with git versions of individual libraries.

Libraries are added by name to 3 associative arrays GITREPO,
GITBRANCH, GITDIR. When we get to the library install phase we inspect
LIBS_FROM_GIT and look for libraries by name (i.e. "oslo.config") and
if they exist we'll clone and install those libraries from
git. Otherwise we won't, and just let pip pull them as dependencies
when it needs them.

This patch provides the conversion of the oslo libraries, including
pbr.

Devstack-gate jobs for these libraries will need to change to support
actually forward testing their content.

restructure stackrc into groupings

in order to support installing from stable libraries we first need to
actually sort out all the categories our giant list of git repos fit
into. This will make it much easier to not lose one in the process.

support installing clients and supporting libraries at released versions

expand the devstack support for libraries from released versions to
support python-* clients and supporting libraries. This is a collapse
of the master patch series that took a while to get correct.

(This is a collapsed patchset for backport to make backport fixups easier)

add unit tests for GIT* definitions

This adds unit tests for all the GIT* definitions, ensuring that for
libraries we think should be defined, they are. It exposed a bug in
glance_store definitions in the process.

The GITDIR definition for python-openstackclient is moved to stackrc
for testability.

Conflicts:
	lib/glance

Change-Id: I26fac0ccf8fd4818e24618d56bf04b32306f88f6
Source functions from stackrc

stackrc now requires GITREPO, GITBRANCH, GITDIR and has been
dependent on functions for a while (is_package_installed).  Ensure
we source the required functions file when stackrc is loaded.  Avoids
unexpected issues in grenade where they may or may not have been loaded
depending on the configuration.

Closes-bug: #1377274

Change-Id: I5027cfad07af0de7ff39f424601d6f7ec5dcadae
(cherry picked from commit 0e4cd038287bcf36ff31c4e7b22266051198b44c)