object-storage: rework dir obj creation from leaf

Rework the directory object creation from the leaf up, only setting
metadata on the directory objects if explicitly requested.

This refactoring also isolates the directory object creation so that
it can be individually serialized in the future.

Change-Id: I564e50679067507ba3f3549a303dfa4cd3266c89
BUG: 906966 (https://bugzilla.redhat.com/show_bug.cgi?id=906966)
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: work around for mkdir() race condition

This is a work around for a race condition in certain Gluster versions
where two or more threads issuing an mkdir() system call for the exact
same directory can end up in D-state, symptomatic of the created
directory having two different gfids.

The work around uses a file lock applied to the parent directory to
ensure that only one thread can issue an mkdir for a directory in the
parent.

BUG: 906966 (https://bugzilla.redhat.com/show_bug.cgi?id=906966)
Change-Id: Ifb9b210821828e2f28863f6756efbff9e6885b3b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: remove work-around for mkdir race

We no longer need this work-around for mkdir race as a fix has been
developed for the GlusterFS side.

BUG: 907072 (https://bugzilla.redhat.com/show_bug.cgi?id=907072)
Change-Id: Ib0b0aa467f41d2cce449f1d68ff0e9c24ff538e5
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: add config option mount per server

Add a configuration option to allow each object server to use a unique
mount point for its operations on the file system. This gives us a bit
of a boost for both bandwidth and IOps.

We create a lock file for each unique instance of an object-server
using a lock file, and when a server acquires that lock file, it can
use or create the corresponding numbered mount point.

Change-Id: I538241711d0f542971e467b663799332a5b8b09f
Signed-off-by: Peter Portante <peter.portante@redhat.com>