basho/riak_test — pull request #118 by joedevivo (2012-12-06 19:14:19+00:00)
Title: a la carte test reporting

ok people, this is what you've been waiting for! The ability to report a customized subset of the giddyup defined suite to giddyup without running every test.

I know @cmeiklejohn will be excited about it. Maybe he'll even stop suggesting that Bishop should go.

Here's how it works:

* If you're not using `-r` then nothing changes. These aren't the droids you're looking for, move along.
* If you're using `-r` and you were adventurous enough to try doing something crazy like using `-b eleveldb` or `-t client_verify_ruby` you may have noticed that riak_test didn't care, it ignored those flags and just used the suite from giddyup. Well, now it cares.

New use of `-r /config`:

if you use `-r` without `-t` or `-b`, it pulls the suite from giddyup like the old days.
If you add `-t` and/or `-b` it will still pull from giddyup, but then filter out tests you didn't specify.

It also does some smart things for you. Like if you don't use `-b` to specify a backend, it will use the backend(s) that giddyup already specifies for the test(s) you chose with `-t`. e.g. `-t client_ruby_verify` only works with `-b memory`, and this will be smart enough to use the memory backend for this test. If you try and force the issue and run with `-b bitcask` it will ignore the request. What you do in the privacy of your own terminal is fine, but don't you go polluting my giddyup database with test failures that can't ever pass.

If a test in the giddyup suite has two possible backends, like secondary_index_tests, then this feature will run both backends if none is specified. If one is specified and it's bad it won't run any. If a backend that is supported by that test is specified, it skips the others.

```
➜ riak_test git:(jd-a-la-carte-test-reporting) ✗ ./riak_test -r osx-64 -t secondary_index_tests
...
Tests to run: [
{secondary_index_tests,
[{id,210},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{secondary_index_tests,
[{id,209},
{backend,eleveldb},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]}]
➜ riak_test git:(jd-a-la-carte-test-reporting) ✗ ./riak_test -r osx-64 -t secondary_index_tests -b bitcask
...
Tests to run: []

➜ riak_test git:(jd-a-la-carte-test-reporting) ✗ ./riak_test -r osx-64 -t secondary_index_tests -b memory
...
Tests to run: [
{secondary_index_tests,
[{id,210},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]}]
```

Here are some more examples! The output is a little verbose here, and I apologize for that. However, I wanted to be explicit as possible about how this works.

```
➜ riak_test git:(jd-a-la-carte-test-reporting) ✗ ./riak_test -r config
...
Tests to run: [{gh_riak_core_154,
[{id,192},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{gh_riak_core_155,
[{id,193},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{gh_riak_core_176,
[{id,194},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{mapred_verify_rt,
[{id,195},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{riaknostic_rt,
[{id,196},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{rolling_capabilities,
[{id,197},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{rt_basic_test,
[{id,198},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_build_cluster,
[{id,201},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_capabilities,
[{id,202},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_claimant,
[{id,203},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_down,
[{id,204},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_leave,
[{id,205},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_riak_lager,
[{id,206},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_riak_stats,
[{id,207},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_staged_clustering,
[{id,208},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{secondary_index_tests,
[{id,209},
{backend,eleveldb},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{secondary_index_tests,
[{id,210},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{basic_command_line,
[{id,241},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{client_java_verify,
[{id,253},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{partition_repair,
[{id,265},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_listkeys,
[{id,277},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_basic_upgrade,
[{id,200},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{client_ruby_verify,
[{id,289},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{loaded_upgrade,
[{id,312},
{backend,bitcask},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{loaded_upgrade,
[{id,346},
{backend,bitcask},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{loaded_upgrade,
[{id,313},
{backend,eleveldb},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{loaded_upgrade,
[{id,347},
{backend,eleveldb},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_basic_upgrade,
[{id,359},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_busy_dist_port,
[{id,373},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{verify_commit_hooks,
[{id,385},
{backend,undefined},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{client_python_verify,
[{id,397},
{backend,eleveldb},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]}]

➜ riak_test git:(jd-a-la-carte-test-reporting) ✗ ./riak_test -t client_ruby_verify -r config -b memory -t secondary_index_tests -t basic_command_line -b eleveldb
...
Tests to run: [{client_ruby_verify,
[{id,289},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{basic_command_line,
[{backend,eleveldb},
{id,241},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{basic_command_line,
[{backend,memory},
{id,241},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{secondary_index_tests,
[{id,210},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{secondary_index_tests,
[{id,209},
{backend,eleveldb},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]}]
...
➜ riak_test git:(jd-a-la-carte-test-reporting) ✗ ./riak_test -t client_ruby_verify -r config -b memory -t secondary_index_tests -t basic_command_line
...
Tests to run: [{client_ruby_verify,
[{id,289},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{basic_command_line,
[{backend,memory},
{id,241},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{secondary_index_tests,
[{id,210},
{backend,memory},
{platform,<<"osx-64">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]}]

➜ riak_test git:(jd-a-la-carte-test-reporting) ✗ ./riak_test -t client_ruby_verify -b memory -t secondary_index_tests -t basic_command_line
...
Tests to run: [{secondary_index_tests,
[{id,-1},
{backend,memory},
{platform,<<"local">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{client_ruby_verify,
[{id,-1},
{backend,memory},
{platform,<<"local">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]},
{basic_command_line,
[{id,-1},
{backend,memory},
{platform,<<"local">>},
{version,<<"riak-1.2.1-85-g69b5e9e-gh251-riak-stop-bsd">>},
{project,<<"riak">>}]}]
```