fix failure of malloc to set errno on heap (brk) exhaustion

I wrongly assumed the brk syscall would set errno, but on failure it
returns the old value of the brk rather than an error code.