seattlerb/minitest — pull request #468 by jasonrclark (2014-10-20 15:34:14+00:00)
Title: Handle encoding failures on writing test results

Ran into a failure where mismatched string encodings caused Minitest to not write out test results. While the actual test was more complicated, the root cause looked like this:

```
require 'minitest/autorun'

# Encoding was set when running in guard
$stdout.set_encoding(::Encoding::UTF_8) if defined?(::Encoding)

class Boo < MiniTest::Test
def test_poor_encoding_choices
# Actual error came from more complex JSON parsing on a bad string, but this works.
raise [106, 199, 95].pack("c*")
end
end
```

Running this, Minitest gave no indication which tests had failed:

```
Run options: --seed 37739

# Running:

E

Finished in 0.001153s, 1734.6054 runs/s, 0.0000 assertions/s.
/Users/jclark/.rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/minitest-5.4.2/lib/minitest.rb:553:in `write': "\xC7" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
from /Users/jclark/.rbenv/.../minitest-5.4.2/lib/minitest.rb:553:in `puts'
from /Users/jclark/.rbenv/.../minitest-5.4.2/lib/minitest.rb:553:in `report'
from /Users/jclark/.rbenv/.../minitest-5.4.2/lib/minitest.rb:620:in `each'
from /Users/jclark/.rbenv/.../minitest-5.4.2/lib/minitest.rb:620:in `report'
from /Users/jclark/.rbenv/.../minitest-5.4.2/lib/minitest.rb:128:in `run'
from /Users/jclark/.rbenv/.../minitest-5.4.2/lib/minitest.rb:55:in `block in autorun'
```

While you can run with `--verbose` to figure out which tests failed, this seemed less than ideal to me.

Given that, I've written up some changes to `SummaryReporter` to be more resilient to failures during writing test results. With these changes, you'd get this output instead:

```
Run options: --seed 26986

# Running:

E

Finished in 0.001009s, 1982.1606 runs/s, 0.0000 assertions/s.

1) Oops:
Minitest failed writing test results with error:
Encoding::UndefinedConversionError: "\xC7" from ASCII-8BIT to UTF-8

Error:
Boo#test_poor_encoding_choices:
RuntimeError:
puts_carefully.rb:11:in `test_poor_encoding_choices'

1 runs, 0 assertions, 0 failures, 1 errors, 0 skips
```

Building the fallback message is the ugliest bit here. Most of that complexity came from trying to build a complete message minus the exception text. Maybe that's overboard? Things would certainly be much simpler if we just wrote the test name and the error Minitest rescued instead.

I also wasn't super happy adding a conditionally-run test, but with the encoding changes between 1.8 and 1.9, these issues just don't show up on older Rubies. Along similar lines, didn't love the need for a different `IO` type than the other tests, hence the comment.

Appreciate any feedback, and I'm happy to revise. I also understand if this is too niche and/or ugly to accept--it was totally worth my time trying out a real change in Minitest in any case. Super fun codebase to work with. :heart: