variety/variety — pull request #41 by todvora (2014-09-16 18:53:22+00:00)
Title: tests refactoring (rewritten to java), tested all given examples and more

Hi all,
Recently I decided to extend test cases for variety and discovered, that previous version of tests is very hard to extend and manage. The shell runner was not robust enough and init/cleanup scripts too far from tests itself.

I rewrote it to language and framework I know much better - Java / JUnit. After that, writing new tests become very easy. I added lots of new tests. For every example given in readme, for data types recognition, standard output of script, parsing input params and more. Tests are running without problems on Travis.

There are several bugs / ambiguities I found:

### Unnamed objects not analysed (UnnamedObjectsAnalysisTest.java)
Due to issue https://github.com/variety/variety/issues/29 I assume, that all unnamed objects should be analysed, key prefixed with .XX. But latest version of variety ignores unnamed objects.

### MaxDepth (MaxDepthAnalysisTest.java)
According to readme, when object has this structure: ```{name:"Walter", someNestedObject:{a:{b:{c:{d:{e:1}}}}}}``` and we set maxDepth with value 3, innermost processed key should be ```someNestedObject.a.b```. But current variety processes also ```someNestedObject.a.b.c```. In readme is also recorded ```"percentContaining" : 16.66666666666666```. That should be probably 100, since there is only one object at all.

### Limit - strange percents and total counts (LimitResultsAnalysisTest.java)
When limiting variety to certain number of results, total counts and percents are incorrect. For first example from readme and limit set to 1, variety says that there are ```someBinData``` with count 5 and 100 percents.

### Query (QueryLimitedAnalysisTest.java)
Unlike limit, query seems to filter both analysed and processed documents. When analysing first example from readme with query ```{someBinData:{$exists: true}}```, variety returns this for ```id``` key: ```{ "_id" : { "key" : "_id" }, "value" : { "types" : [ "ObjectId" ] }, "totalOccurrences" : 1, "percentContaining" : 20 }```. I would expect 5 occurrences / 100%.

### Null data type (DatatypeRecognitionTest.java)
This is just a minor note, unlike all other data types, null type first letter is written lowercase. It seems little bit inconsistent.

Please look at this problems and tell me, what should correct and expected results look like. After that, I will update tests (and variety.js, if my javascript skills will be good enough). Now all tests are passing, problematic asserts are commented out.

Thank you,
Tomas