vslkv: return sane error value on ioctl mem error

Instead of returning a pointer, let user-space know it was a memory
error.