results.py: Add abstract backend class

This patch adds an abc for Backends, redefining the API that backends
(currently only JSONWriter) produce. In subsequent patches JSONWriter
will be adapted into a Backend derived class and one additional Backend
will be added, for writing JUnit xml.

This is a fairly substantial change, and deserves some explanation. At
Intel we are deploying Jenkins servers to assist our development team,
and piglit's native JSON format isn't supported by Jenkins. VMware
already uses JUnit to interface with Jenkins, but does so by converting
the native piglit JSON into JUnit, there are a couple of problems with
this solution. The biggest problem is that it requires extra time to
convert from one format to another, the second problem is that the
script used to do the conversion is not Python3 ready, or even close,
which is step forward that Intel would like to make. By converting
piglit to have an abstracted backend layer we remove the need to convert
piglit's JSON into another format, and we remove another roadblock
toward python3.

The goal of the Backend class it to present a fairly high level API that
is "just enough" API for piglit's use, not a feature complete general
purpose API as JSONWriter provides. This will make writing backends
simpler.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
python: Don't call methods that are not equivalent to Backend API externally

This patch removes external calls to JSONWriter methods that are not
part of the Backend API (or closely equivalent), since those methods
will be changed from public to protected.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Change JSONWriter methods to be more like Backend

This changes initialize_json and close_json to be more like the
constructor and finalize methods of Backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Make fsync a keyword argument

Since fsync is not relevant to all backends, changing it to file_fsync
makes it fit better into the Backend API

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Change JSONWriter to take a string rather than file object

Since not every backend wants a file type not passing one is much nicer
than passing a file. In addition, it removes code duplication between
run and resume.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Merge initialize_json into constructor

This moves the Initial metadata write into the constructor, like Backend
expects.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Add write_test method

This is the test writing method provided by Backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Mark numerous methods as protected

This marks open_dict, close_dict, write_dict_key, and write_dict_item as
protected (with a single leading underscore). Since these methods have
no external users and are not provided by the Backend API.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Rename JSONWriter.close_json to JSONWriter.finalize

This makes the API consistant with Backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: change JSONWriter locking.

Use an RLock directly rather than synchronized_self.

This is simple because only the three public methods need to lock,
rather than half a dozen methods.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Make JSONWriter a Backend derived class

This makes JSONWriter actually inherit from Backend, and everything
works with no warnings or errors.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
framework: Don't pass json_writer to Test.execute

This is a really sill pass, since all of the information that we want
to write is actually available to profile.run(), so not passing it
reduces some code complexity.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
python: Rename json_writer instances to backend

Since we are adding additional backends it makes sense to give the
backend instances a more generic name.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results_tests.py: Add utils.nose_generator decorator to a test

test_generate_initialize is a generator that was not decorated, meaning
that if any test other than the last one failed they would have the
wrong name in the description.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
python: Allow additional backends

This adds the boilerplate code required to make multiple backends
actually work, a backend option and a helper to facilitate getting the
requested backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Rename JSONWriter to JSONBackend

It's a mostly private class, so renaming it has very little public
effect, and it makes it clearer what the class is.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
framework: Add basic support for a JUnit Backend

This adds a very basic JUnit backend. It currently passes all of its
tests, but we still need to know that jenkins will consume said xml.

This needs to have some refactoring done in profile to make it work
correctly, (although I would be comfortable for now pushing with the
known broken behavior if Jenkins will accept it), because JUnit expects
to be given the number of tests for the initial metadata block, but we
have no way to calculate that number until after the run has started.
This is because of the flattening pass in profile, so removing that pass
or pushing it back to the profile modules is necissary.

This also adds the jenkins xsd, Jenkins is a GPL or LGPL project, so
that needs to be looked into.
results.py: Refactor fysnc support for code sharing

Having the option to fsync the JUnitBackend is obviously useful, and
code duplication is bad, so being able to share code between JSONBackend
and JUnitBackend is what we want.

I see two possible ways to handle this, with a decorator or with a mixin
class. The decorator is nice in that it doesn't create multiple
inheritance and all of the potential pitfalls that come from that,
although using a mixin neutralizes many of those problems. The mixin is
nice because it would allow sharing the init between the two, and it
would be able to dictate some of the structure of the class.

I first implemented the decorator, but felt that the mixin was cleaner
and easier to understand.