basho/riak_test — pull request #267 by joedevivo (2013-04-25 16:09:57+00:00)
Title: A (not so) clear separation of church and state

I think Thomas Jefferson (of $2 bill 'Jeffy' infamy) said it best

```
I contemplate with sovereign reverence that act of the
whole [Engineering Team] which declared that their
[validation process] should "make no law respecting
an establishment of [a test], or prohibiting the
free exercise thereof," thus building a wall of separation
between [actual tests] and [the test framework].
```

The separation of tests from the test framework comes down to a very simple premise. There are two types of code in this project: code that is tied directly to a version of Riak and code that is not. The problem then arises: how do you make sure you're running the correct version of a test against the same version of Riak. A great example of that is PR #262.

PR #262 contains a fix"" for 'basic_command_line' that basically says "remember that test for `riak attach`? well, now that's `riak attach-direct`". Ok so once that PR is merged into riak_test, great, we can validate 1.4. But what happens if we need to validate a 1.3.X? well, this test will fail (unless by pure chance, that feature is backported to 1.3.X). That's not 1.4's fault, and it's not 1.3.X's fault. It's certainly not MY fault! But here it is on my plate to solve and I'm going to solve it by moving the tests (and test infrastructure that relies on riak working a certain way) into the riak repository's ./riak_test directory. riak_test can now be run from that directory, and you can check out a tag/branch of riak that corresponds to the version you're testing! Hooray!

Now that you've digested that, and thinking "Oh man, I'd sure like to take Joe out for a taste since he solved all of my problems and maybe I've heard of him." Let me drop the databass on you.

Right now there is a slight issue of the rt module being in bolth 'riak/riak_test' and 'riak_test'. The one in 'riak/riak_test' is the one those tests will use, but the genericization of the rt module is incomplete; however, it is complete enough to start versioning tests for 1.4 and that's pretty gosh darn important.

This PR is mostly deletes, with some partial refactoring of things out of `rt` the module. This PR should not be merged without also merging similar PRs against the 'riak' repo, and the 'buildbot-tester' repo. I'll update this PR after those are generated.