Reset the configuration after tests

Fixes the breakage at http://jenkins.inscitivops.com/job/api-ruby/199