Fix arguments for method expected_success

In expected_success method we expect to get status code of
API response, but we use incorrect argument 'resp' instead
of 'resp.status' in several files.

Change-Id: I28f9e4862ddd42322757f5b206ddfb6ddabcfe0f
Closes-Bug: #1354389
Do not run fwaas API tests in smoke

Now that we are running full tempest against neutron, and because fwaas
is not part of core functionality (and not fully baked). Remove these
test from smoke as part of the effort to only test advanced services
against neutron and not against all projects.

Related patch: Ib315d30ff3c4463203f7ebe00e159a9680cf1616

Change-Id: I2e0ce9641becabbeed5f6ad82ad478d84a37f36f
Fwaas API Test Enhancement

Add Test to insert and remove firewall rule to firewall policy

-Create a firewall rule
-Create a firewall policy
-Insert a firewall rule to policy
-Validate the insertion of rule to policy
-Remove the firewall rule from policy
-Validate the removal of firewall rule from policy

Change-Id: I3c9711294e1e6323bdad4ff783de1ed19e133c1b
Add test list_servers_filtered_by_name_regex

The nova servers API supports full regex filtering on names. Add a test
for basic regex support.

Change-Id: Iddd5908d7daa5d5b20cb29dd0f403fcdd3ecefcf
Add subnet tests for extra attributes

* Tests added for ipv4/6
test_create_delete_subnet_with_allocation_pools
test_create_delete_subnet_with_gw_and_allocation_pools
test_create_delete_subnet_with_host_routes_and_dns_nameservers
test_create_delete_subnet_with_dhcp_enabled
test_update_subnet_gw_dns_host_routes_dhcp
test_create_delete_subnet_all_attributes

* Adds helper to create/verify/delete subnet
* Adds helper to delete network

Implements: bp subnet-extra-attributes

Change-Id: I7c1b00f4fad797eb5dd904f1123b5bcd5e5af649
Add tempest post-run cleanup script

Implemented all features defined in blueprint.

Change-Id: If3c9b82f85095eeb22bbf32841be534024213567
Implements: blueprint post-run-cleanup
Easier to read console output

The console output was hard to read because it was displayed on one
line as a string as part of a tuple. Split them up so it's readable.

Change-Id: Ieb59909802a587027c4691924e70923c61476591
Fix use of code name in services decorator

This commit fixes an incorrect usage of ceilometer in the services
decorator, telemetry is the correct service tag name. This also adds
a unit test to catch the incorrect usage of a service tag.

Change-Id: I5c231ec3ba064013f8d5fc2f49dbdf28e5c637c1
Fix resource cleanup in server rescue tests

If server rescue tests are skipped then the resource cleanup method
terminates with AttributeError: type object has no attribute 'volume'.
The fix checks whether volume exist before operating with it.

Change-Id: I6ae7f7bb2ba73313db676bafd597c0934704959a
Migrate orchestration API tests to resource_* fixtures

Partially-implements bp resource-cleanup

Change-Id: I58b5f9f17d166c762a1091f4ecb2c560bff8e977
Migrate queuing API tests to resource_* fixtures

Partially-implements bp resource-cleanup

Change-Id: I5c2773e1e0d475dcbd77007121a33ae06a436c22
Migrate telemetry API tests to resource_* fixtures

Partially-implements bp resource-cleanup

Change-Id: I47e007ef70b79044d78c08b07b222183e1025f70
Migrate scenario utils to tempest client

All scenario tests have been migrated to tempest client.
Migrate scenario utils as well, so that we may drop
the OfficialClientManager completely.

Change-Id: I3ec80de961ba289734f9bc811e4789e7d1b5903c
Partially-implements: bp/tempest-client-scenarios
Clean up tox.ini and make tempest and unit test distinct

This commit creates a separate base env for tempest tests and
tempest's unit tests. However, to keep the same interface the tempest
env is setup by overloading the testenv using substitution from a
common base section.

As part of this internal reorganization in the tox.ini several other
things are cleaned up or removed. Usless or duplicate job definitions
are removed. The unit tests are switched to pretty_tox. Previously
they were run without any output which made it difficult to figure out
exactly what was running. Additionally, this removes the unnecessary
regex filter on the unit test job. The unit tests use a different
discovery path so there is no need to also filter it on the same path
this will allow the use of more specific regexes with tox to filter
tests.

Change-Id: I996d81dec2739bbe907f5fe87a335994c01697b3
Fix MismatchError for LB scenario test

It has been observed that at a given iteration only one of the two HTTP
services responds to requests made to the VIP. This is somewhat rare and
it may be caused by the fact that the number of requests being sent is
not high enough for the ROUNDROBIN algorithm to equally distribute requests
across all the members in the pool.

Increasing the number of retries makes the test runtime only slightly
longer, but it should give us more confidence that, if we keep observing
the same type of failure, this can be ruled out.

At the same time, this patch also relaxes the check so that it just verifies
that member pools are load balanced at least once. Verifying that the number
of requests are spread evenly across is always going to be prone to transient
condition and it is beyond the scope of a scenario test that validates whether
Neutron LBaaS is wiring things correctly.

Closes-bug: #1358857

Change-Id: Ibc0aeb092a4643eb151cef8d7d942d9e3d1ed71c
javelin: Detach the volume before destruction

In Icehouse, a volume needs to be detached before it can be deleted.

Closes-Bug:  #1375476

Change-Id: Ic1c8e79fc1efa46ac9a70e288166cd5e0d823cfa
Add a test for deleting a suspended server

A server should be able to be deleted at any time.
This patch adds a test for deleting a suspended server.

Change-Id: I21ab4eb204f1d85ef53d239afc92fd6a3359a97e
Fixes image create utf-8 multibyte test

Fixes test_create_image_specify_multibyte_character_image_name
so that it actually sends a UTF-8 multibyte character rather
than unicode. Also changes it to a 3 byte multibyte character
rather than a 4 byte multibyte character because glance currently
doesn't handle 4 byte utf-8 and will 500.

Change-Id: I064b563024a090899a4adcbbd462a18c5450830c
Related-Bug: 1006725
Add an internal test method for rebooting a server

test_reboot_server_hard and test_reboot_server_soft contain almost
the same code and they are redundant.
This patch adds an internal test method and makes them use the method
for cleanup.

Change-Id: I98fc519b5bb2de8cc47007ba9138247f4c0f3369
Reduce complexity during import phase

In the current state of the negative testing framework the import phase
consists in many complex operation during import phase. This fix reduces
the complexity during that phase to improve the performance,
debugability and robustness.

In this fix we only determine the test scenarios in the import phase. The
payloads are created while test execution. This also improves the
flexibility since the test executor has full access to the test variables.

Change-Id: I8deed33d37c5bf07c26118d6d66ada105d259c89
Partially-implements: bp autogen-negative-tests