Fix rails new tests

This fixes some of the tests failing here: https://build.go.cd/go/pipelines/build-new-rails/138/run-rspec/1