What I mean is 
```
    for name, metrics in THRESHOLDED_METRICS_ITEMS:
          assert_raises(ValueError, metrics, y1_str, y2_str)
```

should be  (or at least the following lines of code added)

```
    for name, metrics in THRESHOLDED_METRICS_ITEMS:
          assert_raises(ValueError, metrics, y1_str, y2)
```
Because y2_str can't be a valid input for thresholded metrics. The goal of the `test_invariance_string_vs_numbers_labels`in my opinion is to check that metrics can handle string labels correctly. 

If log_loss supports string input for labels, it should also be tested with y1_str.astype('O'). Either the invariance test could be enhanced, either one more assertion could be added in `test_log_loss`.

Is it clearer?

> I think the test is correct at master, and hinge_loss should also raise ValueError

Sorry, I wasn't clear. I wasn't talking about the exception to return.