osd/ReplicatedPG: allow whiteout deletion with IGNORE_CACHE flag

If the client specifies IGNORE_CACHE, allow a regular DELETE to zap a
whiteout.  Expand test case to verify this works.

Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_rados_api_tier: fix cleanup of whiteouts

Doing a normal DELETE on a whiteout returns ENOENT.  Use the IGNORE_CACHE
flag to make sure it removes the whiteout, too.

Fixes: #10052
Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2926 from ceph/wip-10052

ceph_test_rados_api_tier: fix whiteout cleanup (10052)

Reviewed-by: Samuel Just <sjust@redhat.com>