MB-8828: Check memory condition before returning enomem for withmeta ops

If we run into a memory condition when we are trying to put something
into the hashtable then we return enomem even though we might actually
be able to free memory. This change makes sure we check the memory
condition before actually returning an error code.

Change-Id: I3d10641764653c971aaeb9e8089fb21fe891a699
Reviewed-on: http://review.couchbase.org/28269
Reviewed-by: David Liao <david.liao@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>