Shopify/verdict — pull request #4 by wvanbergen (2014-01-21 14:56:53+00:00)
Title: Segmenter refactor

A bunch of work to support different segmenting/assignment models in Verdict.

Segmenters were always pluggable in the library, but it shipped with only the fixed percentage segmenter. I added two basic other segmenters, and made the segmentation model a bit more flexible so we can implement for advanced concepts like multi-armed bandit.

## New segmenter possibilities

- A `Segmenter#assign` should normally return a group. Now, you can also return `nil`, which means "temporarily disqualified". This means that the disqualification will not be stored, and if the subject returns, we will try to segment it to a group again. This allows us to implement rollout scenarios with increasing percentages. (Compare this to the `qualify` block, which will cause somebody to be disqualified permanently.)
- If you call `Experiment#convert(subject, goal)`, this is forwarded to segmenter. The segmenter can use this information to adjust itself based on results. This feedback loop allows us to implement multi-armed bandit scenarios.

## Segmenter implementations

The `Verdict::Segmenter` base class now is documented with comments, to make it easier to implement new segmenters. I also implemented two new segmenters, `Verdict::StaticSegmenter` and `Verdict::RolloutSegmenter`

#### Verdict::StaticSegmenter

This allows you to specify a non-random list of subjects per group.

``` ruby
Verdict::Experiment.define('my_experiment') do
groups(Verdict::StaticSegmenter) do
group :a, ['1234', '2345', '3456']
group :b, ['0987', '9876', '8765']
end
end
```
This will assign subjects to groups if their identifier is mentioned in the list. This is of course not a randomized experiment, but it is useful to beta-test experiments on a couple of subjects. If a subject identifier is not included in any group, it will be temporarily disqualified. This means that you can always qualify a subject later, e.g. by changing the StaticSegmenter by a normal percentage segmenter with the same group names. In that case, the assignments of the originally hardcoded subject identifiers will remain in their original group.

In this case the list is hardcoded, but potentially it could be loaded from a database. (Note: identifiers are always strings)

#### Verdict::RolloutSegmenter

The rollout segmenter is exactly the same as percentage segmenter, except it allows you to assign less than 100%. Every subject that falls outside the assigned range is temporarily disqualified. This allows you to increase the percentages later to slowly rollout a new feature.

I also added a shortcut:

``` ruby
Verdict::Experiment.define('test') do
rollout_percentage 10
end
```

This will create a single group of 10%, called "enabled". The other 90% will be temporarily disqualified.

Review @peterjm @CamDavidsonPilon @BenDoyle