summary.py: Don't use os.environ to set the template directory

This makes summary more robust, since it doesn't rely on
PIGLIT_SOURCE_DIR being set correctly in the environment.

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
framework: Don't rely on os.environ so much

There are some problems with using os.environ, all of which are caused
shell child-variable relationships. Most of these problems are easy to
solve, since we don't actually need to have environment variables set
except during test execution, and that can be passed to Popen (or it's
helper wrappers). This gives us much better assurance that things are
happening in the way we expect.

v7: - use itertools.chain instead of nested iterations
    - fix piglit.conf default case
    - reorder environment overrides so that variables passed in by the
      user (or exported) override the builtin values instead of
      vice-versa

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
core: replace Environment.collectData()

Environment.collectData() is strange and awful for a host of reasons.
1) Its implementation was such that it should have been a static method,
   but it wasn't
2) It wasn't at all related to Environment
3) It used a public helper which it was the sole consumer of.

Replacing it with a top level function makes a lot more sense for all of
these reason. This function has another major advantage, it assumes
nothing. In the former implementation it was assumed that specific
operating systems had specific tools available, and that those tools
were only available on that platform. This implementation doesn't assume
that, instead it makes a blind leap that the tools is available, and
then is caught if its wrong.

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
core.py: Simplify Environment constructor

This replaces assignment followed by for loops with list comprehensions.
This is less code, simpler, and faster.

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
core.py: rename Environment to Options

Environment doesn't really describe what the class is (although,
honestly it's a pretty bad class design), but Options comes much closer
to what it is.

v7: - update piglit-print-commands.py
    - replace instance name 'env' with 'opts'
    - rename Test.ENV to Test.OPTS

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
framework: Split code out of framework.core

This splits results code out of core into its own module. This makes the
code cleaner and easier to understand, and is groundwork for coming
patches.

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Acked-by: Ilia Mirkin <imirkin@alum.mit.edu>
results.py: Replace JSONEncoder subclass with default method

Both methods are equally viable solution when using the builtin json
module, but using defaults is superior when using simplejson (a C based
implementation of the json module providing an identical interface but
vastly better performance) which can be used with piglit if it's
installed.

v7: - fix one framework test

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
results.py: Modernize some things

Use super() instead of an explicit call to the parent class in one case,
and make all classes descend from object (make them new-style classes)

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Acked-by: Ilia Mirkin <imirkin@alum.mit.edu>
results.py: Cleanup pylint warnings

These are mostly from one of two sources:
1. overriding builtin names
2. pep8 compliance issues

A couple of the other errors:
- an unused keyword argument
- extra grouping parens
- long lines

v7: - rename some variables differently

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
results.py: Simplify TestrunResult.write()

This simplifies TestrunResult.write() to use a generator expression.
This method isn't currently used, but it will be useful in a patch later
in this series.

This also adds a testcase for this function to ensure that what it
writes is equivalent to what it reads.

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
framework: consolidate boilerplate in JSNOWriter class

This moves all of the code for writing the name, the options, etc out of
run and resume and into a single method of JSONWriter. This should
reduce errors, code duplication, and help abstract a lot of problems
with future changes to the json away.

v7: - add close_json() to JSONWriter
v9: - simplify close_json()

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
framework: Update results to use version numbers

This patch updates our json to version 1. Changes from version 0 to
version 1 are as follows:

- renamed 'main' to 'results.json'
- dmesg must be a string (It was stored a list in some version 0
  results)
- subtests are never stored as duplicate entries, a single instance of
  the test is recorded, (In version 0 both are possible)
- there is no info entry in version 1, err, out, and returncode are
  always split into separate entries

This patch adds support to the results module for handling updates to
the results in a sane way. It does this by adding a result_version
attribute to the TestrunResult (which is stored as json), and
implementing the ability to incrementally update results between
versions.

It does this automatically on load, non-destructively, moving the old
results to results.json.old, but does write the updated results to disk,
making the cost of this update a one time cost.

v2: - Handle non-writable directories and files, this also fixes using
      file-descriptors as inputs
v4: - Fix bug that caused update_results to always return early
    - Fix bug that caused tests with one subtest to have their name
      mangled
    - Set TestrunResult.results_version by default, set it to None
      during load if the results file doesn't have a results_vesrion
      value. Otherwise results could end up with no version when they
      should have one
    - Fix a bug that raises an uncaught exception when a test's info has
      more elements than the expected 3 (mostly glean)
    - split the version 0 to 1 tests out of the results_tests module.
      There are enough tests and enough data specific to these tests it
      makes more sense to make a separate test module
v5: - Fix handling of subtest with / in their name
v6: - Properly handle results with proper one-entry-per-test
v7: - set TestrunResult to 0 instead of None when loading results that
      don't have a version
    - Use a function to loop over the update functions instead of
      writing each one out in a long list
    - Add two more framework tests that test rests.update_results()
    - Remove some unnecessarily careful variable checking, we have
      assurances earlier in the code path that the values we expect are
      there
    - update comments in _update_results() to include expected input and
      output
    - don't compare dicts with .iterkeys()
    - rework info splitting to be more robust

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
summary.py: Remove work-arounds for version 0 result bugs

These work-arounds are no longer necessary since they are handled by the
result format updater on load.

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
arb_arrays_of_arrays: fix required extensions

Signed-off-by: Timothy Arceri <t_arceri@yahoo.com.au>
Reviewed-by: Matt Turner <mattst88@gmail.com>
arb_texture_query_lod: allow some tolerance for comparisons

This is required for radeonsi to pass.

Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
core.py: Add additional catch to collect_system_info()

Catch subprocess.CalledProcessError, which is an exception that
subprocess.check_call() raises if the binary returns a non 0 status.
The usual culprit for this is glxinfo on systems not running X

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Tested-by: Tom Stellard <thomas.stellard@amd.com>
arb_texture_buffer_object-get: use piglit_get_gl_enum_name() in error msg

Reviewed-by: Charmaine Lee <charmainel@vmware.com>
shader_runner: Add support for layered framebuffer

This allows attaching an entire 2darray texture as a layered
framebuffer.

Signed-off-by: Chris Forbes <chrisf@ijw.co.nz>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
shader_runner: Support creating uninitialized 2d array textures

Signed-off-by: Chris Forbes <chrisf@ijw.co.nz>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
arb_fragment_layer_viewport: Add tests for not-written cases

Signed-off-by: Chris Forbes <chrisf@ijw.co.nz>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>