rgw: add data_extra pool to bucket

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: zone placement info includes extra data pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: multipart meta object uses extra data pool

Fixes: #7676
We need the extra data pool since EC backends cannot handle the omap
operations that are needed for the multipart meta object.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: don't modify rgw_bucket for data extra pool

This cleans up the interface, as we don't have this ugly implicit
structure modification when we deal with object that resides on the data
extra pool. Replcae lot's of calls to get_obj_bucket_and_oid_key() to a
cleaner get_obj_ref() that uses a struct to hold the updated fields, and
update the relevant callers.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Merge branch 'wip-7676-firefly' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
add write trace point for performance analysis