TechEmpower/FrameworkBenchmarks — pull request #1129 by hamiltont (2014-10-24 11:11:06+00:00)
Title: Objectify FrameworkTest Types

This is a big code cleanup PR that breaks out test types (json, update, plaintext, etc) into objects and refactors most of the verification code to be much easier to read and debug. None of the existing behavior has been changed, just the code is cleaner.

Makes each test type a subclass of `FrameworkTestType`, which cleans up a lot of duplication in `FrameworkTest`. As tests `db`, `query`, and `update` are all variants of the same theme, I was able to reuse a lot of the testing code once I had split it out, so we now have much more thorough verification checks on some items. For example, we now check 4 different URLs for `update` instead of the previous 1. Also, every single JSON object returned by `query` is checked for correctness using the object description outlined by the `database` test. I was also able to find and add a few minor new verification edge cases.

The only user-facing change introduced is that verification results are now printed for each URL tested while the verification is running, which looks like this if things are good:

```
INFO:root:Verifying framework URLs
PASS for http://127.0.0.1:8080/db
PASS for http://127.0.0.1:8080/fortune
PASS for http://127.0.0.1:8080/queries?queries=2
PASS for http://127.0.0.1:8080/queries?queries=0
PASS for http://127.0.0.1:8080/queries?queries=foo
PASS for http://127.0.0.1:8080/queries?queries=501
```

like this if things are bad:

```
INFO:root:Verifying framework URLs
PASS for http://127.0.0.1:8080/mysql-orm
PASS for http://127.0.0.1:8080/fortune
WARN for http://127.0.0.1:8080/mysql-orm-update?queries=0
JSON array length of 0 != expected length of 1
WARN for http://127.0.0.1:8080/mysql-orm-update?queries=foo
JSON array length of 0 != expected length of 1
PASS for http://127.0.0.1:8080/mysql-orm?queries=2
PASS for http://127.0.0.1:8080/mysql-orm?queries=0
PASS for http://127.0.0.1:8080/mysql-orm?queries=foo
PASS for http://127.0.0.1:8080/mysql-orm?queries=501
```

and this if things are really bad:

```
INFO:root:Verifying framework URLs
FAIL for http://127.0.0.1:16969
Caused Exception in TFB
This almost certainly means your return value is incorrect,
but also that you have found a bug. Please submit an issue
including this message: global name 'htmlString' is not defined
Traceback (most recent call last):
File "/FrameworkBenchmarks/toolset/benchmark/framework_test.py", line 297, in verify_type
results = test.verify(base_url)
File "/FrameworkBenchmarks/toolset/benchmark/test_types/fortune_type.py", line 30, in verify
parser.feed(htmlString)
NameError: global name 'htmlString' is not defined
```

[Travis Run](https://travis-ci.org/hamiltont/FrameworkBenchmarks/builds/38913522)