Project: Puppet (Provisioning / Automation & Configuration)
File: rspec_tutorial.md

Puppet relies heavily on automated testing to ensure that Puppet behaves as expected and that new features don't interfere with existing behavior. There are three primary sets of tests that Puppet uses: unit tests, integration tests, and acceptance tests. - - Unit tests are used to test the individual components of Puppet to ensure that they function as expected in isolation. Unit tests are designed to hide the actual system implementations and provide canned information so that only the intended behavior is tested, rather than the targeted code and everything else connected to it. Unit tests should never affect the state of the system that's running the test. - - Integration tests serve to test different units of code together to ensure that they interact correctly. While individual methods might perform correctly, when used with the rest of the system they might fail, so integration tests are a higher level version of unit tests that serve to check the behavior of individual subsystems. All of the unit and integration tests for Puppet are kept in the spec/ directory. - - Acceptance tests are used to test high level behaviors of Puppet that deal with a number of concerns and aren't easily tested with normal unit tests. Acceptance tests function by changing system state and checking the system after the fact to make sure that the intended behavior occurred. Because of this acceptance tests can be destructive, so the systems being tested should be throwaway systems. All of the acceptance tests for Puppet are kept in the acceptance/tests/ directory. Running the acceptance tests is much more involved than running the spec tests. Information about how to run them can be found in the [acceptance testing documentation](https://github.com/puppetlabs/puppet/blob/master/acceptance/README.md). Puppet is only compatible with a specific version of RSpec. If you are not using Bundler to install the required test libraries you must ensure that you are using the right library version. Using an unsupported version of RSpec will probably display many spurious failures. The supported version of RSpec can be found in the project Gemfile. GitHub Actions (spec tests only): https://github.com/puppetlabs/puppet/actions Jenkins (spec and acceptance tests): https://jenkins.puppetlabs.com/view/Puppet%20FOSS/ Puppet uses RSpec to perform unit and integration tests. RSpec handles a number of concerns to make testing easier: Executing examples and ensuring the actual behavior matches the expected behavior (examples) Grouping tests (describe and contexts) Setting up test environments and cleaning up afterwards (before and after blocks) Isolating tests (mocks and stubs) At the most basic level, RSpec provides a framework for executing tests (which are called examples) and ensuring that the actual behavior matches the expected behavior (which are done with expectations) ```ruby specify "one equals one" do expect(1).to eq(1) end it "one doesn't equal two" do expect(1).to_not eq(2) end ``` Good examples generally do as little setup as possible and only test one or two things; it makes tests easier to understand and easier to

More complete documentation on expectations is available at https://www.relishapp.com/rspec/rspec-expectations/docs Note Puppet supports the API, so please do not use RSpec 2 "should" syntax like `1.should == 1`. Example groups are fairly self explanatory; they group similar examples into a set. ```ruby describe "the number one" do it "is larger than zero" do expect(1).to be > 0 end it "is an odd number" do expect(1).to be_odd # calls 1.odd? end it "is not nil" do expect(1).to be end end ``` Example groups have a number of uses that we'll get into later, but one of the simplest demonstrations of what they do is how they help to format documentation: ``` rspec ex.rb --format documentation the number one is larger than zero is an odd number is not nil Finished in 0.00516 seconds 3 examples, 0 failures ``` Examples may require some setup before they can run, and might need to clean up afterwards. `before` and `after` blocks can be used before this, and can be used inside of example groups to limit how many examples they affect. ```ruby describe "something that could warn" do before :each do $VERBOSE = nil end after :each do $VERBOSE = true end it "doesn't generate a warning" do MY_CONSTANT = 1 MY_CONSTANT = 2 end end ``` Some examples may require setting up data before hand and making it available to tests. RSpec provides helper methods with the `let` method call that can be used inside of tests. ```ruby describe "a helper object" do let(:my_helper) do ['foo', 'bar', 'baz'] end it "is an array" do expect(myhelper).to beakindof Array end it "has three elements" do expect(my_helper.size).to eq(3) end end ``` Like `before` blocks, helper objects like this are used to avoid doing a lot of setup in individual examples and share setup between similar tests. RSpec allows you to provide fake data during testing to make sure that individual tests are only running the code being tested. You can stub out entire objects, or just stub out individual methods on an object. When a method is stubbed the method itself will never be called. The RSpec Mocks documentation can be found at https://relishapp.com/rspec/rspec-mocks/v/3-8/docs/ ```ruby describe "stubbing a method on an object" do let(:my_helper) do ['foo', 'bar', 'baz'] end it 'has three items before being stubbed' do expect(my_helper.size).to eq(3) end describe 'when stubbing the size' do before :each do allow(myhelper).to receive(:size).andreturn(10) end it 'has the stubbed value for size' do expect(my_helper.size).to eq(10) end end end ``` Entire objects can be stubbed as well. ```ruby describe "stubbing an object" do let(:my_helper) do double(:notanarray, :size => 10) end it 'has the stubbed size' expect(my_helper.size).to eq(10) end end ``` It's possible to combine the concepts of stubbing and expectations so that a method has to be called for the test to pass (like an expectation), and can return a fixed value (like a

```ruby describe "mocking a method on an object" do let(:my_helper) do ['foo', 'bar', 'baz'] end describe "when mocking the size" do before :each do expect(myhelper).to receive(:size).andreturn(10) end it "adds an expectation that a method was called" do my_helper.size end end end ``` Like stubs, entire objects can be mocked. ```ruby describe "mocking an object" do let(:my_helper) do double(:notanarray) end before :each do expect(notanarray).to receive(:size).and_return(10) end it "adds an expectation that the method was called" do notanarray.size end end ``` When properly written each test should be able to run in isolation, and tests should be able to be run in any order. This makes tests more reliable and allows a single test to be run if only that test is failing, instead of running all 17000+ tests each time something is changed. However, there are a number of ways that can make tests fail when run in isolation or out of order. If you do have a test that passes in isolation but fails when run as part of a full spec run, you can often narrow down the culprit by a two-step process. First, run: ``` bundle exec rake spec ``` which should generate a spec_order.txt file. Second, run: ``` util/binarysearchspecs.rb <full path to failing spec> ``` And it will (usually) tell you the test that makes the failing spec fail. The 'usually' caveat is because there can be spec failures that require specific ordering between > 2 spec files, and this tool only handles the case for 2 spec files. The > 2 case is rare and if you suspect you're in that boat, there isn't an established best practice, other than to ask for help on IRC or the mailing list. Puppet has a number of older tests that use `before` blocks and instance variables to set up fixture data, instead of `let` blocks. These can retain state between tests, which can lead to test failures when tests are run out of order. ```ruby RSpec.configure do |c| c.mock_with :rspec end describe "fixture data" do describe "using instance variables" do before :all do @fixture = double('test data') end it "can be stubbed" do allow(@fixture).to receive(:foo).and_return(:bar) expect(@fixture.foo).to eq(:bar) end it "does not keep state between tests" do expect { @fixture.foo }.to raise_error end end describe "using `let` blocks" do let(:fixture) { double('test data') } it "can be stubbed" do allow(fixture).to receive(:foo).and_return(:bar) expect(fixture.foo).to eq(:bar) end it "does not keep state between tests" do expect { fixture.foo }.to raise_error end end end ``` ``` bundle exec rspec test.rb -fd fixture data using instance variables can be stubbed should not keep state between tests (FAILED - 1) using `let` blocks can be stubbed should not keep state between tests Failures: 1) fixture data using instance variables should not keep state between tests Failure/Error: expect { @fixture.foo }.to raise_error expected Exception but nothing was raised Finished in 0.00248 seconds 4 examples, 1 failure Failed examples: rspec ./test.rb:16 # fixture data using instance variables should not keep state between tests ``` RSpec core docs: https://www.relishapp.com/rspec/rspec-core/docs RSpec guidelines with Ruby: http://betterspecs.org/