Don't exit in other places than CLI

Raise errors instead and let CLI handle it.
Remove usage of the have_error_message matcher

It's a bit over-the-top to have "ERROR" displayed, and it's not used
enough to warrant its existence.