Stop returning ASCII error messages to binary clients

In a couple out-of-memory conditions, out_string() was being called
regardless of the client's protocol.

It's never appropriate to return malformed garbage (which is what an
ASCII response is to a binary client). In one case we can try to return
an error using the binary protocol, and in the other it's preferable to
just fail silently than to try to return an ASCII response, especially
since the attempt would probably also fail with an out-of-memory error.