Add simple support for plainbox subcommands

The code around plainbox job evaluation has gotten seriously unwieldy
and in consequence hard to understand, let alone test.

This large patch splits the monolithic code into PlainBox and a number
of PlainBoxCommand subclasses. The old plainbox command is pretty much in
the new 'plainbox run' subcommand. Everything else is now in the new
'plainbox special'.

Each command has basically two methods it must implement:

The "entry" method is invoked(self, ns) where the command needs to
do it's job, having been passed the namespace object produced by argparse.

The "support" method is register_parser(self, subparsers) where the
command must add itself to a parent parser's subparsers instance.

Unlike other such systems __init__() is free-form and commands are actual
instances, not just classes. This allows the system to be somewhat easier
to test and configure as each command can be "fed" on a per-use-case basis

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add tests for plainbox special

Not everything is tested but this gives a pretty good smoke test
and high executed code coverage.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add return values to --output-*=? and plainbox run

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add a simple testing script

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Move TestIO to testing_utils module

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Fix indent on one line

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add TestDirectory to testing_utils

This small class simply creates a context manager for working with
a temporary directory

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Remove unused variables

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add TestCwd context manager

This class is another small utility for testing. It can reliably
change the current working directory for the duration of the context
manager scope.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add initial integration tests

This warrants some explanations. Integration tests won't be running
as a part of normal unit testing, hence the test_*.py patter is not
used. The code is correct but explicitly focused on one arbitrary
checkbox job (cpu/scaling_test). Proper implementation will need
test multiplication with something similar to testscenarios from
python 2.x world. Work on that is ongoing.

The actual test just compares the outcome of the job with reference
known-good expected values. Those are stored in the source tree and
are packaged along with everything else. To make discovery painless
and irrelevant of installation path I'm using pkg_resource which matches
python3 packaging evolution and the documented best strategy for accessing
package data.

There are a number of dummy packages (plainbox.data.*) that may not be
needed after all, I'll investigate this and edit this commit.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Fix unit tests after adding self-test

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add plainbox self-test command.

This command discovers, loads and executes all integration tests

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Split unit tests for testing_utils into dedicated module

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add TestCaseWithParameters to testing_utils

This new subclass of TestCase allows construction of test cases
that multiply themselves to get better coverage for various values
of specified variables.

Implementation is a cross between the well-known TestCaseWithScenarios
and my own TestCaseWithInvariants. It allows the developer to
name variable parameters to a test case and provide all the distinct
values they (as a collection) should take. Using this as the base class
it's easy to construct the more limited TestCaseWithScenarios (I have that
as a separate patch) as well as the special-cased invariants.

The implementation does not clone/deepcopy each test case instance, this
may be a design bug that I'll correct once I do some testing. Each time
run() is called the special attribute _parameters is assigned with
a new value of the TestCaseParameters class. All of the special methods
that were defined in TestCase are adapted to understand this new argument
but use cases such as storing a TestCase in a collection and comparing
it to another TestCase probably won't work as intended as the special
parameter (which affects all of the special methods, including __hash__)
is only assigned, temporairly, in the run() method loop.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Add untested implementation of TestCaseWithScenarios

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Drop dummy __init__.py files from the data directory

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Use TestCaseWithParameters for IntegrationTests

This patch allows integration tests to discover all the checkbox
jobs that should be started by inspecting package data (looking
for reference results).

This works very well but after implementing it and giving it some thought
I cannot help to wonder if we need more than one "set" of reference
answers. (in practice the data directory would get a sub-directory
per each data-set). This could be used to test responses from resource
scripts (so reference data would contain actual resources) and other
generators that don't produce one valid data-set but instead slightly
different data-sets depending on the hardware (or software and hardware)
they run on.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Fix mk-interesting-graphs after adding special command

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Run integration tests when testing for coverage

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>