framework/tests/summary.py: Tests Summary.py's assignment of statues

This code tests the way Summary.py assigns statuses to tests.
Specifically it tests regressions, fixes, changes, and problems,
assuring that only the correct permutations of these tests are added to
the list.

It uses python's builtin unittest framework, and a magical decorator to
limit code duplication.

This already reveals ~20 problems in the current implementation

v2: - Move framework_tests to framework/tests
    - Spelling corrections
v3: - More spelling corrections
    - Address a number of concerns from Ken Graunke
    - Make exceptions more robust by using the ``except <error> as''
      syntax
    - Refactor framework tests to reduce code duplication
    - Use python's tempfile module to get the system dependent temporary
      file location instead of hard-coding /tmp
    - Add test for the skip list

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
piglit-framework-tests.py: Adds a utility for running piglit unit tests

This program gives the option to run the tests. These tests are meant to
test internal features of the piglit framework. Currently there are only
tests for the summary module's handling of regressions, fixes, changes,
and problems.

Acked-by: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
framework/summary.py: Fix problems with status handling

Fixes status handling (fixes, regressions, changes) to be compliant with
the status handling in framework_tests/summary.py. Mainly this addresses
notrun -> * and * -> notrun, as well as a few corner cases.

v2: - Spelling corrections
v3: - More spelling corrections
    - fix skipped page generation

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
status.py: Adds a new status class and subclasses

These status classes are intended to replace the string statuses
currently used by piglit summary.

They give a couple of very nice advantages, first, they can be directly
compared to each other to determine whether a status is worse than
another status. These statuses can also be compared to any object
implementing the __int__ magic method.

This will allow us to remove a number of special functions for sorting
representations of statuses.

v2: - Use larger numbers for the statuses, this should make adding
      additional statuses easier, since there is room between the
      statuses.
    - Add a class for the "Not Run" status
    - Adds classes for dmesg-warn and dmesg-fail
    - Change statuses to match the standard set in the framework test
    - Spelling corrections
v3: - add a docstring showing status order
    - correct a couple of comments
    - Adds a function for converting a string into a status

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
status: Make use of the status objects

This adds code in framework/summary.py and framework/core.py to make use
of the status classes in status.py. This makes comparisons between
statuses much simpler and cleaner

v2: - Instead of importing all of the classes into the local namespace
      use the ``import ... as'' syntax. This means that if additional
      statuses are added they can easily be used without updating
      imports
    - Correct the sorting for fixes and regressions
    - Fix bug in TestResult that caused Fail, Warn, and Crash to be
      assigned the wrong status Objects
    - Adds support for dmesg-warn and dmesg-fail
v3: - Use the status.status_lookup() function
    - Replace use of map with list comprehension

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
summary: Remove optimization with no impact

Generating these lists is rather trivial, so just generate them all the
time.

v2: - Fix html page generation
v3: - Stop framework summary tests from trying to generate these lists

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
summary: Fold __generate_lists into __init__

Since there is nothing special about __generate_lists() anymore (it
always generates the same set of lists when called), and we need to call
it in every instance just fold it back into the constructor, so that it
no longer need to be explicitly called.

v2: - Address some spelling problems in the commit message
v3: - More spelling isues

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
piglit-run.py: Fix missing import

A late revision of commit 06b1a851874a75440458037543077002c31a510a
accidentally included the hunk:

-from threadpool import ThreadPool

which broke piglit-run.py completely.  It looks like a rebase mistake.
sso: Verify that rendezvous by location also works

Separate shader objects can either connect outputs to inputs by matching
names (rendezvous by name) or by assigning explicit locations via the
layout(location=...) (rendezvous by location).  This is a simple test
that a vertex shader with a different name than the fragment shader
input can by matched by the location.

v2: Fix some dumb bugs in the test.  Use multiple values to ensure we're
not matching using rendezvous-by-dumb-luck.

v3:
* Fix a bug in the ordering of cross product parameters.  Noticed by
  Paul.
* Add another test case where the fragment shader variables are listed
  in a different order.  Add some explanitory text with rationalization
  for the tests chosen.
* Dynamically set the #version line depending on driver support.  This
  works around some bugs in the NVIDIA driver noticed by Paul.

Signed-off-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Paul Berry <stereotype441@gmail.com>
sso: Combine 20 vertex shaders and 20 fragment shaders in various ways.

Verify that the right shaders are used in the right combinations several
ways.

* The vertex shader has information baked-in that determines the X position
  of the block on the screen.

* The fragment shader has information baked-in that determines how the
  block is colored.  This is combined with data passed from the vertex
  shader.

Since data is passed from the vertex shader to the fragment shader, the
test can use either rendezvous-by-name (default) or rendezvous-by-location
(with --by-location command line parameter).

This test passes in both modes on NVIDIA (304.64 on GTX 260).

v2: Ensure that the window is sized large enough for the tests.  Use a
much simpler shuffling algorithm.  Both suggested by Paul.

Signed-off-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Paul Berry <stereotype441@gmail.com>
sso: Add compile tests for the location layout qualifier

NVIDIA (304.64 on GTX 260) fails all of the 1.10, 1.20, and 1.30 tests
because they fail to recognize the layout keyword at all.  Paul has
suggested that adding

    #extension GL_ARB_fragment_coord_conventions: require

may work around this issue in some cases.

Signed-off-by: Ian Romanick <ian.d.romanick@intel.com>
Reviewed-by: Paul Berry <stereotype441@gmail.com>
Acked-by: Wen Su <WSu@nvidia.com>
GL 1.50: Test that UNIFORM_BLOCK_REFERENCED_BY_GEOMETRY_SHADER is accepted by GetActiveUniformBlockiv()

v2: Incorporate this test into referenced-by-shader.c

Reviewed-by: Eric Anholt <eric@anholt.net>
GLSL 1.50: Test that version 140 and version 150 shaders can be linked together

Reviewed-by: Paul Berry <stereotype441@gmail.com>
GLSL 1.50: Test that multiple shaders of version 140 and 150 can be linked together

Reviewed-by: Paul Berry <stereotype441@gmail.com>
GL 3.2: Test that GetTexLevelParameterfv() generates an error when passed TEXTURE_BORDER

v2: Multiple fixes and added to all.tests

Reviewed-by: Eric Anholt <eric@anholt.net>
junit: Update script for status class hierarchy.

Also, remove all the summary stuff, which we never used.
built-in-constants: don't try to create GS with version < 150.

Previously, built-in-constants looked at required_glsl_version to
determine what to include in the "#version" directive, but it looked
at glsl_version (the maximum version supported by the implementation)
to decide whether to test geometry shaders.  As a result, it when
testing a built-in constant whose required_glsl_version was less than
150, it would try to create a geometry shader using a "#version"
directive less than 150, leading to GL errors.

This patch prevents the problem by only testing geometry shaders when
required_glsl_version >= 150.

Reviewed-by: Matt Turner <mattst88@gmail.com>

v2: Don't try to create GS on GLES either.

Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
gs: Check GL errors in primitive-id-restart test.

Previously, if drawing failed due to a GL error, no primitives would
be generated, so the test would have nothing to check, and it would
erroneously pass.  Avoid that by checking for GL errors at the end of
the test.

Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
glut framework: skip if implementation doesn't support required GL version.

In effect, we are already doing this when Waffle is in use, since
Waffle uses a context creation mechanism that ensures that the context
won't be created if the implementation doesn't support the GL version
needed by the test.  But for GLUT we need to do the check manually,
since GLUT always just creates the highest GL version context it can.

Tested-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Chad Versace <chad.versace@linux.intel.com>
piglit-summary-html.py: Fix the -e/--exclude-details option

This was broken by a previous commit 06b185187, this patch corrects
that oversight allowing exclude to continue working.

Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>