airbnb/trebuchet — pull request #7 by nelgau (2013-07-22 22:55:38+00:00)
Title: An internal stubbing interface for Trebuchet

I'm not here to convince you of the value of stubbing Trebuchet. That practice is essential and well-established. Instead, I want to talk about the advantages of dropping our ad-hoc ```rspec-mocks``` stubs by moving the stubbing abstraction into Trebuchet itself.

Why? The current situation is gnarly and the variety is manifold, but I have an ulterior motive as well.

I'd like to make it easy to support and maintain hacks like this — http://www.github.com/airbnb/trebuchet-roulette.

### A short survey of the gnarly

We have at least four distinct patterns for stubbing Trebuchet in Monorail:

* ```stub_all_trebuchet```
* ```stub_trebuchet( ... )```
* ```Trebuchet.any_instance.stub(:launch?) { ... }```
* ```Trebuchet.stub_chain(:current, :launch?).and_return( ... )```

And the always popular not-a-stub-stub:

* ```Trebuchet.aim( ... , :users, [2, 3])```

Let's put aside the fact that ```stub_all_trebuchet``` is a demon spawn, at least for the moment.

While each of these methods has the desired effect, it makes it very difficult for other parts of our testing infrastructure to determine when a feature is stubbed. In some cases—I'm looking at you, ```stub_all_trebuchet```—the method is so blunt as to defeat the purpose of testing controlled examples.

### A blessed stubbing interface

My ideal Trebuchet stubbing interface would:

* Be singular.
* Permit introspecting which features are currently stubbed.
* Actively interfere with and/or resist any stubbing by other means.

I want to propose a new way to stub Trebuchet. The syntax is simple:

Trebuchet.feature('my_darling_feature').stub(:launched)

To reset stubbing, you call:

Trebuchet.dismantle_stubs

And that's it.

I made several attempts to introduce the final behavior I listed above, but ```rspec-mocks``` is a wily beast. Disallowing stubbing on any particular instance is trivial and in fact, you can look back at these commits for an example. However, defeating the ```#any_instance``` stubber is more involved. I'm willing to admit that this kind of tomfoolery doesn't belong in Trebuchet.

Though, if there's interest, I can continue to investigate what it would take to do it in Monorail...

### But wait, there's more

Once we start to use this interface (even in just a few places), many interesting possibilities open before us. At any time, we can inquire as to which features (and which states) are under *deliberate* consideration by any particular test example:

Trebuchet::Feature.stubbed_features
=> { 'my_darling_feature' => :launched }

... and which are not! This, of course, allows ```trebuchet-roulette``` to work its magic.

### Spin the wheel

The rationale for ```trebuchet-roulette``` is written up in its repository, but in the event that you're terminally lazy or just late for a wedding, I've reproduced one of the examples here. :)

Consider the following test:

it "should succeed due to an assumption" do
some_condition = trebuchet.launch?('false_by_default')
some_condition.should be_false
end

This test passes because all Trebuchet experiments are unlaunched by default, but the very same expectation will fail when that particular feature is launched.

Let's see what happens when we enable ```trebuchet-roulette```:

describe 'the panoply of all possibility' do
with_trebuchet_roulette

it "should succeed due to an assumption" do
some_condition = trebuchet.launch?('false_by_default')
some_condition.should be_false
end
end

The output now reveals the truth for what it is:

Failures:

1) the panoply of all possibility should succeed due to an assumption (false_by_default=true)
Failure/Error: some_condition.should be_false
expected: false value
got: true

Let's put an end to surprise assaults from Trebuchet!

*"Green Builds should be safe to deploy."* —Anonymous