Deduplicate status call checks

This commit updates the jsonschema validation method validate_response()
to use the rest client method expected_success() to check that the
correct status code response is received. Previously validate response
was doing the same check despite there already being a method available
for doing the check. As a part of using expected_success() to validate
the status codes using the jsonschema it was necessary to expand
expected_successs() to handle both single expected codes as well as a
list of codes. This is because it's perfectly valid for the jsonschema
to specify a list of expected responses.

Change-Id: I400bb9aff89c18f3964a1d8ec45bfb8dfd75301d
Enhanced Trove (database) flavor API tests

 - Added test to make sure trove flavors matched OS (nova) flavors
 - Added response status checks to existing tests

Partially implements: blueprint trove-tempest

Change-Id: I59cc551ef5ec58fe43b794ae8dbd6a89c40c774b
Reworked scenario tests to use their own non-default security groups

Moved _create_security_group_nova to base OfficialClientTest class, so
that now it can be used by all scenario tests. All scenario tests now
create security group with "loginable" rules explicitly and attach it to
servers instead of using "default" security group.

Change-Id: If82c681b4a7f62808f46b03c57d113f8f5c97f6f
Closes-Bug: #1247060
Add branchless tempest section to README.rst

This commit adds a section to the tempest README outlining some things
that have to be watched when pushing changes to tempest because of
branchless tempest

Change-Id: I9dbf042cef6af51c92d48d42ba04c1582690bda7
Add README section about API stability

This commit adds a section to the README outlining that tempest tests
should only be added for stable APIs. If the API is tested the
expectation is that it complies with the API stability guidelines.

Change-Id: I02a93f66ff45e4d687b7395fb24ee5bda6e52fb0
make logformat more similar to oslo

make the logformat actually match what we're used to reading in
the other logs, especially including the levels.

Because the logging fixture doesn't include the oslo logging
context processor, we are limited in what we can do here. A good
follow on would be to make a fixture which does understand oslo
logging processing so we could get request_id in the logs.

Change-Id: Ibb95aebd2df714fae331df13ad7789d6341af265
Add unittests for remote client

Remote client is used in servers tests to provide some common operations into a
remote server. It's backended by the ssh class, which already has tests, so all
of its calls are mocked and due to that there are some 'useless' tests that are
pretty much checking response's format and checking if the ssh call didn't
change to avoid inconsistencies in the future.

Partially implements bp unit-tests

Change-Id: I5dabc170a557c81bb9b0605bf101719585d7c53b
Adds test for Floating Ips bulk Nova V2 API

This patch adds the test for following Nova V2 APIs-
- Create Floating Ips bulk
- Delete Floating Ips bulk
- List Floating Ips bulk

This patch also adds the config option for unallocated
Floating IP Range

Change-Id: I6af52bb2ce2caadf8491740097df71d041643e78
Add total runtime to summary output

This commit adds the total runtime in secs to the summary view from
subunit-trace.py

Change-Id: If56f2a048b3d1ab3d51425d5b3560e1f365a6c3a
Sync versionutils from oslo-incubator

This is needed for comparing client versions from the command line to
required versions for CLI tests.

Tempest already has the immediate oslo dependencies of gettextutils and
log and those are up-to-date enough for what's needed in versionutils,
so this is a --nodeps sync.

Part of blueprint minversion-check-for-cli-tests

Change-Id: I19784499a9bb1135a9c8dc8b26fc56c53489ed22
Use known flavor ID as marker

The flavors marker test races with the flavors admin test because we
can list the flavors, pick the first flavor ID as a marker, then the
flavors admin test deletes that flavor and finally our attempt to list
flavors with that marker fails.

Avoid this by using a known flavor ID as our marker.

Change-Id: I56d043b58a403fc28a13b6f45540831ae8e0331d
Closes-Bug: #1330830
Verify list_instance_action attributes of Nova API

This patch adds the JSON schema for Nova V2 & V3 list_instance_actions
API response and validate the response with added JSON schema
to block the backward incompatibility change in the future.

The response body of list_instance_actions V2 API is below:

{
    "instanceActions": [
        {
            "action": "reboot",
            "instance_uuid": "b48316c5-71e8-45e4-9884-6c78055b9b13",
            "request_id": "req-3293a3f1-b44c-4609-b8d2-d81b105636b8",
            "user_id": "789",
            "project_id": "147",
            "start_time": "2012-12-05 00:00:00.000000",
            "message": ""
        }
    ]
}

The response body of list_instance_actions V3 API is below:

{
    "server_actions": [
        {
            "action": "reboot",
            "server_uuid": "b48316c5-71e8-45e4-9884-6c78055b9b13",
            "request_id": "req-3293a3f1-b44c-4609-b8d2-d81b105636b8",
            "user_id": "789",
            "project_id": "147",
            "start_time": "2012-12-05T00:00:00.000000",
            "message": ""
        }
    ]
}

Partially implements blueprint nova-api-attribute-test

Change-Id: I81a21a4ec0522318fd20c3db3e534fdcea7b8715
Set default to false for nova v3_api config option

This commit switches the default value for the nova v3 api config
option from true to false. This changes the default behavior to not
run the Nova v3 API tests. This is because the v3 is still marked as
experimental and is able to change in a non-backwards compatible way
which will cause issues when running with branchless tempest.
Additionally, the nova team decided not to do a standalone v3 api so
at some point this will disappear and instead become v2.1 of the api
with microversions. When you couple this with the fact that tempest's
gate budget is basically exhausted (tests shouldn't take over an hour
to run in the gate) this should be defaulted off. For more discussion
see:

http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html

Change-Id: I13dbc4b433a0f6defbeebce958b2e1cbcf64ba7a
Unbreak baremetal API tests that were not updated

A recent merge (88233) changed the baremetal test base to return (resp, body)
tuples from its create_* methods instead of the created objects.  This updates
the test cases that were missed in this patch and unbreaks all Ironic CI jobs
that were overlooked in that review.

Change-Id: I691e36d0678a15737614b7867a4c6778e714f299
Closes-bug: #1330834
Merge "Enhanced Trove (database) flavor API tests"
Merge "Add unittests for remote client"
Merge "Set default to false for nova v3_api config option"
Merge "Unbreak baremetal API tests that were not updated"
Merge "Add total runtime to summary output"
Merge "Reworked scenario tests to use their own non-default security groups"