tests/all.py: Fix name of arb_tessellation_shader test

'/' is not allowed in the name of tests because it is the delimiter for
group names. This test includes '/' which causes the HTML summary to
render in odd ways.

v2: - use nested group structure rather than adding a test directly to
      the test list (Ilia)

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
tests/all.py: Add assertions for bad characters in test names

'/' is not allowed in test names since it is used as the separator for
groups.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
results.py: Add abstract backend class

This patch adds an abc for Backends, redefining the API that backends
(currently only JSONWriter) produce. In subsequent patches JSONWriter
will be adapted into a Backend derived class and one additional Backend
will be added, for writing JUnit xml.

This is a fairly substantial change, and deserves some explanation. At
Intel we are deploying Jenkins servers to assist our development team,
and piglit's native JSON format isn't supported by Jenkins. VMware
already uses JUnit to interface with Jenkins, but does so by converting
the native piglit JSON into JUnit, there are a couple of problems with
this solution. The biggest problem is that it requires extra time to
convert from one format to another, the second problem is that the
script used to do the conversion is not Python3 ready, or even close,
which is step forward that Intel would like to make. By converting
piglit to have an abstracted backend layer we remove the need to convert
piglit's JSON into another format, and we remove another roadblock
toward python3.

The goal of the Backend class it to present a fairly high level API that
is "just enough" API for piglit's use, not a feature complete general
purpose API as JSONWriter provides. This will make writing backends
simpler.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
python: Don't call methods that are not equivalent to Backend API externally

This patch removes external calls to JSONWriter methods that are not
part of the Backend API (or closely equivalent), since those methods
will be changed from public to protected.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Change JSONWriter methods to be more like Backend

This changes initialize_json and close_json to be more like the
constructor and finalize methods of Backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Make fsync a keyword argument

Since fsync is not relevant to all backends, changing it to file_fsync
makes it fit better into the Backend API

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Change JSONWriter to take a string rather than file object

Since not every backend wants a file type not passing one is much nicer
than passing a file. In addition, it removes code duplication between
run and resume.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Merge initialize_json into constructor

This moves the Initial metadata write into the constructor, like Backend
expects.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Add write_test method

This is the test writing method provided by Backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Mark numerous methods as protected

This marks open_dict, close_dict, write_dict_key, and write_dict_item as
protected (with a single leading underscore). Since these methods have
no external users and are not provided by the Backend API.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Rename JSONWriter.close_json to JSONWriter.finalize

This makes the API consistant with Backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: change JSONWriter locking.

Use an RLock directly rather than synchronized_self.

This is simple because only the three public methods need to lock,
rather than half a dozen methods.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Make JSONWriter a Backend derived class

This makes JSONWriter actually inherit from Backend, and everything
works with no warnings or errors.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
framework: Don't pass json_writer to Test.execute

This is a really sill pass, since all of the information that we want
to write is actually available to profile.run(), so not passing it
reduces some code complexity.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
python: Rename json_writer instances to backend

Since we are adding additional backends it makes sense to give the
backend instances a more generic name.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results_tests.py: Add utils.nose_generator decorator to a test

test_generate_initialize is a generator that was not decorated, meaning
that if any test other than the last one failed they would have the
wrong name in the description.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
python: Allow additional backends

This adds the boilerplate code required to make multiple backends
actually work, a backend option and a helper to facilitate getting the
requested backend.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Rename JSONWriter to JSONBackend

It's a mostly private class, so renaming it has very little public
effect, and it makes it clearer what the class is.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
framework: Add a JUnit backend

This adds a mostly compliant JUnit backend. It currently passes all of
its tests and jenkins happily consumes the xml we produce.

This needs to have some refactoring done in profile to make it 100%
compliant with the junit-7.xsd from upstream jenkins, (although I would
be comfortable for now pushing with the known non-optimal behavior if
Jenkins will accept it), because JUnit expects to be given the number of
tests for the initial metadata block, but we have no way to calculate
that number until after the run has started.  This is because of the
flattening pass in profile that flattens the nested directory structure
into a flat dictionary.

There are two options to solve this problem:
1) Flatten all.py and other modules. This is a lot of work and I have
   many work-in-progress branches to do just hat
2) Push the pass out to a public method and call it ahead of time. This
   seems really hacky to me, and I'd rather not do something that ugly.

Currently this patch just passes 0 for the test count unconditionally,
jenkins does not seem to have a problem with this.

This includes JUnit.xsd from the jenkins svn repository for piglit
framework unit testing. This is only used in the piglit python framework
unit tests.

v2: - Fix some spelling errors
    - Remove asserts that were used for development
    - Replace any '\\' in test names for the JUnit backend as well as
      '/'. This covers that case that Windows os.path.join() is used on
      test names; this uses an re.sub() instead of str.replace()

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
results.py: Refactor fysnc support for code sharing

Having the option to fsync the JUnitBackend is obviously useful, and
code duplication is bad, so being able to share code between JSONBackend
and JUnitBackend is what we want.

I see two possible ways to handle this, with a decorator or with a mixin
class. The decorator is nice in that it doesn't create multiple
inheritance and all of the potential pitfalls that come from that,
although using a mixin neutralizes some of those problems. The mixin is
nice because it would allow sharing the init between the two, and it
would be able to dictate some of the structure of the class.

I first implemented the decorator, but felt that the mixin was cleaner
and easier to understand.

Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>