Fwaas API Test Enhancement

Add Test to insert and remove firewall rule to firewall policy

-Create a firewall rule
-Create a firewall policy
-Insert a firewall rule to policy
-Validate the insertion of rule to policy
-Remove the firewall rule from policy
-Validate the removal of firewall rule from policy

Change-Id: I3c9711294e1e6323bdad4ff783de1ed19e133c1b
Easier to read console output

The console output was hard to read because it was displayed on one
line as a string as part of a tuple. Split them up so it's readable.

Change-Id: Ieb59909802a587027c4691924e70923c61476591
Clean up tox.ini and make tempest and unit test distinct

This commit creates a separate base env for tempest tests and
tempest's unit tests. However, to keep the same interface the tempest
env is setup by overloading the testenv using substitution from a
common base section.

As part of this internal reorganization in the tox.ini several other
things are cleaned up or removed. Usless or duplicate job definitions
are removed. The unit tests are switched to pretty_tox. Previously
they were run without any output which made it difficult to figure out
exactly what was running. Additionally, this removes the unnecessary
regex filter on the unit test job. The unit tests use a different
discovery path so there is no need to also filter it on the same path
this will allow the use of more specific regexes with tox to filter
tests.

Change-Id: I996d81dec2739bbe907f5fe87a335994c01697b3
Fixes image create utf-8 multibyte test

Fixes test_create_image_specify_multibyte_character_image_name
so that it actually sends a UTF-8 multibyte character rather
than unicode. Also changes it to a 3 byte multibyte character
rather than a 4 byte multibyte character because glance currently
doesn't handle 4 byte utf-8 and will 500.

Change-Id: I064b563024a090899a4adcbbd462a18c5450830c
Related-Bug: 1006725
Reduce complexity during import phase

In the current state of the negative testing framework the import phase
consists in many complex operation during import phase. This fix reduces
the complexity during that phase to improve the performance,
debugability and robustness.

In this fix we only determine the test scenarios in the import phase. The
payloads are created while test execution. This also improves the
flexibility since the test executor has full access to the test variables.

Change-Id: I8deed33d37c5bf07c26118d6d66ada105d259c89
Partially-implements: bp autogen-negative-tests
Fix schema definition admin_flavor_create

The api of flavor_create has a missing property level. Property "id"
is noneable and need to be excluded from negative testing.

Change-Id: I411eb8c29d0c00d6d64358ede26992ee2fe380c9
Closes-bug: #1375142
Fix finding router for scenario tests

Finding router return tuple resp,body and body only contains router details.
Closes-Bug: 1375859
Change-Id: Ic405d20991561e8773153d829750826a82ed751c
Add the start message for debugging net info

When a network scenario test fails due to networking timeout, we can
get both server console log and network information for debugging.
Current network information doesn't contain the start message and
it seems a little confusion where starts network info.
This patch adds the message.

Change-Id: I8490c08b7073302d299f7b63dd9d6380d8a8d27e
Related-Bug: #1323658
Fix create_user parameters for v3

When testing against the v3 identity API, the create_user function
definition doesn't sync up to how it's being called. The result is
errors about mismatches in the number of arguments.

This patch updates the function to sync up with how it's being
called, which is also how the definition is written for the v2 API.

Change-Id: I7fd68e774e54933e49b98b9690d803475fcb5d62
Closes-Bug: #1376830
Remove Resp status code checks in stress tests

As response code checks is done at client side,
tests case do not need to check the same again.

This patch removes status code checks in stress tests.
All removed status code checks are present on client side.

Partially Implements blueprint: client-checks-success

Change-Id: I1f09b25d6d6d1c966a0635509d670c55d9415468
Separate security group rule test for each args

If passing "group_id" argument to "add security group rule" API, the
API ignores "cidr" argument.
In test_security_group_rules_create_with_optional_arguments, both
arguments are specified and that is meaningless.
This patch separates it for each purpose.

This patch removes success code checks from each test because the code
is already checked in the client method create_security_group_rule().

Change-Id: I1af1f37eb2d4b4f182d7ab69bcb0e9c37a356e35
Related-Bug: #1373832
Revert "Skip telemetry_notification_api test due to bug 1336755"

As https://review.openstack.org/#/c/114514/ has been merged, we can
try to check if changing of Ceilometer backend to the MongoDB makes the
situation better with running Ceilometer notification tests. Locally the
timeout issues were not reproduced both on MySQL and MongoDB, so this
change aims to check if the issue was solved by trying experimental runs
of MongoDB devstack Ceilometer testing.

This reverts commit 88004daa7b0fd746a60dd3b17d68ed4427b40e58.

Closes-Bug: #1276157

Change-Id: I41fce7e56a0e9fb111a95a266def786cb0f90c5e
Remove setUpClass added after cleanup

After the cleanup of setUpClass a new one managed to sneak
in. Removing it now, and enforcing the hacking rule in next
patchset.

Change-Id: If53b3a7d299f01f7c45739f5b2f023fbdb73aa45
Adds status check for FloatingIP in network scenarios

Verifies FloatingIP changes status correctly on association and dis-association

Scenarios:
test_network_basic_op
test_network_advanced_server_ops

Status is verified after connectivity check and without a waiter in order to
not mask unknown races in Neutron

Changes show() to refresh() to keep consistency

Partially Implements: blueprint fip-op-status (this is a Neutron bp:
https://blueprints.launchpad.net/neutron/+spec/fip-op-status)
Partially Implements: blueprint neutron-advanced-scenarios

Change-Id: Ibd45f3092cebc8f9fd0d16a3a543c2ef72156409
Remove OfficialClient dependency from HACKING.rst

This commit migrates OfficialClientTest to ScenarioTest about
TestVolumeBootPattern in HACKING.rst. We should not have such an example
in the HACKING.rst because we don't use OfficialClientTest class in the
scenario tests anymore.

Partially-implements: bp:tempest-client-scenarios

Change-Id: Id95d011f3d080effc7bfee6232578451f7ba72d9
Log AttributeError as 'info' instead of 'exception'

When failure happens in setUpClass, tearDownclass is being called
from base class irrespective of safe_setup decorator used or not in
specific test class. Which results to lot of instance of AttributeError
raised from tearDownclass. For example when tests case is skipped.

This patch provides a workaround by logging AttributeError as 'info'
instead of 'exception'.

Permanent fix for cleaning up those in more appropriate way will
be by split-up resource_setup & resource_cleanup in more structural way
as part of second phase of resource cleanup BP.

Change-Id: Ia49c9e171338e22f8d605fc53d251ef3d3471bb1
Hacking rule to forbid resource unsafe fixtures

Extend the existing T105, which was missing checks for
tearDownClass anyways. Forbid overriding setUpClass
and tearDownClass except for tempest/test.py where the
test base class is defined.

To be able to enforce the rule, fixing setUpClass
which was added with new tests before this patch could
merge.

Partially-implements bp resource-cleanup

Change-Id: Ib4d98c2ff8776ea1379a044b5a30fb02e351ce75
Drop unused safe_setup decorator

The safe_setup decorator was introduced so that tests could
enforce tearDownClass being called in case of failures during
the setUpClass.

With the implementation of bp resource-cleanup, the test base
class automatically provides this facility for all tests,
so the safe_setup decorator is not needed anymore.

Partially-implements bp resource-cleanup

Change-Id: I3a4f34324420ee3a3ecc9a5e4e33d6ae27f64299
Fix use of kwargs instead of **kwargs

This but did not show well because it is used inside a handler that
ignores and logs the exception. But the domain update call fails.

Closes-Bug: #1375415

Change-Id: I16f01b6c2b6cc71ad60f42f4c875cbb85d2e016b
Fix verify_tempest_config with disabled services

The script silently fails when verifying API versions for services
that are not deployed or configured in the catalog.  This fixes it
by only verifying versions for services that actually exist in
keystone.

Change-Id: Iee3724f8bbbfe767e3c6e0b9ee0f9f818e87f68b