Handle exception message with invalid UTF-8; For rbx

See https://travis-ci.org/mikel/mail/jobs/33707769#L129

```plain
An exception occurred running /home/travis/build/mikel/mail/gemfiles/vendor/bundle/rbx/2.1/gems/rspec-core-3.0.4/exe/rspec:
    invalid byte sequence in UTF-8 (ArgumentError)
Backtrace:
    Rubinius::Splitter.valid_encoding? at kernel/common/splitter.rb:17
    Rubinius::Splitter.split at kernel/common/splitter.rb:22
    String#split at kernel/common/string.rb:515
    RSpec::Core::Notifications::FailedExampleNotification#failure_lines at \
          gemfiles/vendor/bundle/rbx/2.1/gems/rspec-core-3.0.4/lib/rspec/core/notifications.rb:220
```

To reproduce with just the one test

```sh
 rspec spec/rspec/core/formatters/base_text_formatter_spec.rb:108
```
1. good point re: memoizing booleans and `||=`. that was kind of dumb
2. re: `defined?(Rubinius)` seemed the more common check to me.  I think `defined?(RUBY_ENGINE) && RUBY_ENGINE == 'rbx'` would be my next choice.