Add failing test with bad message

We incorrectly state that some errors were "Caused by redirect" when they aren't, for example if you retry and get a 500 a few times in a row. 

This adds a failing test to demonstrate this. Not sure how you'd like to handle it, I don't have great suggestions at the moment but I'll keep looking.
Should use self.assertNotIn for consistency.