housejester/d8a-conjure — pull request #5 by dhruvp (2013-07-31 22:22:30+00:00)
Title: Cardinality Nodes

We needed to create a way to generate data where we could customize the following: 1. Cardinality
2. Type (String, int, long, etc. )
3. Number of columns
This fork creates the capability to do so. In addition, it allows you to print each data event to an external object queue as well.
The code was all formatted to follow a uniform formatting spec. To ignore the whitespace changes in the diff, just add ? w=1 at the end of the pull request url

By cardinality I mean the number of different values a given column can take. Druid's performance can vary based on the cardinality of a column. So, to make sure a column can have a certain cardinality (say 100), I just have a simple counter that ensures that the given column outputs 100 different values in 100 events.

If I'm correct, weighted nodes are used to give a node a probability of occurring. For our use case, we weren't really concerned with probability. The reason we didn't follow how weighted nodes were added was because we wanted users to be able to provide a json spec. Here's an example of our spec file:

{
"specs":
[
{
"cardinality":10,
"type":"int",
"count":5
},
{
"cardinality":100,
"type":"string",
"count":3,
"name":"HundredCardinalityColumn"
}
]
}
Specs is a list of different specifications. Each specification specifies a cardinality, a type, and a count (the number of columns of that specification you want). You can also optionally specify a name.

Finally, the general overview is as follows

1. Specifications get passed in via a json file and is deserealized using Jackson.
2. ACardinalityNode is created for each column and stored in CardinalityNodeList.
3. On every print call, a Map is created by CardinalityNodeList. The key is the column name, the object is the value of
the column. This will be used as an event in Druid.