fwts_test now fails based on user specified threshold
accidentally removed the enhanced failure output, so tweaked that and added it back in
suspend/suspend_advanced and suspend/suspend_advanced_auto now only fail on CRITICAL events, while all other tests using fwts_test will still fail on HIGH or CRITICAL events.
removed extra print() statements that were not necessary
"[r=zkrynicki][bug=1099554][author=bladernr] automatic merge by tarmac"
Remove unused variables

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Fix indent on one line

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Fix mk-interesting-graphs after adding special command

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
"[r=sylvain-pineau][bug=][author=zkrynicki] automatic merge by tarmac"
Add a simple testing script

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add TestCaseWithParameters to testing_utils

This new subclass of TestCase allows construction of test cases
that multiply themselves to get better coverage for various values
of specified variables.

Implementation is a cross between the well-known TestCaseWithScenarios
and my own TestCaseWithInvariants. It allows the developer to
name variable parameters to a test case and provide all the distinct
values they (as a collection) should take. Using this as the base class
it's easy to construct the more limited TestCaseWithScenarios (I have that
as a separate patch) as well as the special-cased invariants.

The code is heavily documented and seems to work well in causual testing.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add TestCwd to plainbox.testing_utils.cwd

This small class simply creates a context manager for working in
a different directory, temporarily

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add initial integration tests

This warrants some explanations. Integration tests won't be running
as a part of normal unit testing, hence the test_*.py patter is not
used. The code is correct but explicitly focused on one arbitrary
checkbox job (cpu/scaling_test). Proper implementation will need
test multiplication with something similar to testscenarios from
python 2.x world. Work on that is ongoing.

The actual test just compares the outcome of the job with reference
known-good expected values. Those are stored in the source tree and
are packaged along with everything else. To make discovery painless
and irrelevant of installation path I'm using pkg_resource which matches
python3 packaging evolution and the documented best strategy for accessing
package data.

There are a number of dummy packages (plainbox.data.*) that may not be
needed after all, I'll investigate this and edit this commit.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add plainbox self-test command.

This command discovers, loads and executes all integration tests

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Use TestCaseWithParameters for IntegrationTests

This patch allows integration tests to discover all the checkbox
jobs that should be started by inspecting package data (looking
for reference results).

This works very well but after implementing it and giving it some thought
I cannot help to wonder if we need more than one "set" of reference
answers. (in practice the data directory would get a sub-directory
per each data-set). This could be used to test responses from resource
scripts (so reference data would contain actual resources) and other
generators that don't produce one valid data-set but instead slightly
different data-sets depending on the hardware (or software and hardware)
they run on.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

Conflicts:
	plainbox/plainbox/impl/integration_tests.py
Drop dummy __init__.py files from the data directory

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Run integration tests when testing for coverage

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>