Pr 1248 merge prep

This PR is meant to take #1248 across the finish line and get it merged.

If you want to see what my additional changes are on top of @jalkoby's, here's a compare URL:

https://github.com/rspec/rspec-core/compare/f52e7f90275a0a17fb30415a809de9107df30b0e...pr-1248-merge-prep

/cc @JonRowe