Check that malloc or asprint was successful

Fixes bug RUBY-760

I wanted to pick out a minor bug, and opted for 760 https://jira.mongodb.org/browse/RUBY-760?jql=project%20%3D%20RUBY%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20%22Minor%20-%20P4%22%20ORDER%20BY%20key%20DESC. I know the contribution should only happen against master, but maybe this is of use. If not, that is okay :)