Return ENOMEM when failing to allocate memory

When vr_btable_alloc fails, it should get reported back up the call
stack as ENOMEM, not EINVAL. This is also what happens in vr_flow.

Change-Id: I250b272c4274904cadb2c1e9a6dc92fc6bc4c7b5