Make sure that an EmptyByteBuf will not get forwarded on remove/replace