Support case where off-heap mem buffer can not be allocated