Stop returning ASCII error messages to binary clients

In a couple out-of-memory conditions, out_string() was being called
regardless of the client's protocol.  Wrap all the out-of-memory
out_string() calls with a helper function that returns a binary
response if appropriate. (In most cases, this is in ASCII protocol
handling code and thus is a no-op change, but using the wrapper in some
places and not others is an invitation to future bugs of this type.)