riptano/cassandra-dtest — pull request #93 by knifewine (2014-09-23 18:16:44+00:00)
Title: Paging tests and util

These changes introduce the paging tests formerly running with the cassandra java driver on jython. This also includes some helpers which I'd like some feedback on before merging (see below).

I removed 3 tests which were covering changing page size during a query because this is not exposed in the python driver api, and I'm not certain they were very useful. We should reintroduce these tests later if needed. I can see changing page size being useful for user-driven paging where a frontend app user can change their page size preference, but without driver support there's not much we can do at this point.

I changed one test from sending a zero page size to not sending a page size at all (java made me send an int of course), to check that the driver defaults to 5k/page. Sending zero may be a useful test but I was getting non-deterministic test results so I'm not sure if this is a python driver problem, misuse of the API, or maybe a C* issue. I'll follow up there.

These tests include a paging harness altered from the jython version to work with the python async paging api. I tried to keep it simple and not do too much magic. The basic goal with this is to allow the tests to work/assert in terms of discrete pages instead of trying to deal with automatic results paging (which is great, but makes testing a bit more difficult).

Finally, these changes introduce another module from the jython tests, datahelp.py -- here's where I need the most feedback. The goal of this tool is to allow a declarative style for creating test data, which you can see used in the paging tests themselves where there are markdown-style tables. The tool is used both for creating data and for generating data expectations as well, so it can be used in making test assertions. The dilemma here is whether this tool deserves its own module (so it can be used by other test modules, which I think would be cool), or if it should be moved/kept only as part of the paging tests. I'd like to see it available for general use, but I probably think it's cooler than it actually is, so I need some level headed feedback here.

My ultimate goal for this data tool is to allow faster data creation and the ability to create an indefinite number of rows in a memory efficient way (the version included here is likely to OOM python with a sufficiently large dataset). I've got some experimental changes for a memory efficient version of datahelp, which includes a prototype for logging test expectations to disk but I'll push them on a separate branch for now (I think memory use is still escalating despite my efforts, but in a test I was able to create 20 million rows in about 40 minutes).