Return ENOMEM from nn_allocmsg (fixes #175)

Signed-off-by: Martin Sustrik <sustrik@250bpm.com>