object-storage: do not list directories before delete

Do not list the entire directory contents of the directory containing
the file before deleting it. We have already verified the files
existence constructing the Gluster_DiskFile object.

This fixes a customer issue. See BZ 885281
(https://bugzilla.redhat.com/show_bug.cgi?id=885281).

Backport of commit 622ea301205063307034ee306d8b0565316edbde.

Change-Id: Iade38479d7667fbf7f948fa055378fdd72c2221d
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: backport starvation fix

Backport: "Fix starvation in object server with fast clients"

See https://github.com/openstack/swift/commit/783f16035a8e251d2138eb5bbaa459e9e4486d90

See also BZ 903396,
(https://bugzilla.redhat.com/show_bug.cgi?id=903396)

Change-Id: Ide99a201f44aa18cb1aac6f97b87c758a3eb1675
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: yield during directory tree walks

Have the co-routine (greenlet) yield during long running directory
tree walks to avoid starving out other greenlets that might be
available to run.

A directory walk involves statting every file in the tree to get it
size. For large numbers of files, this can get expensive. So we yield
after every stat call, and after processing every directory.

This is a backport of commit 488d6ac4e992a86a314efb76230ec490c756e073.

See BZ 894674,
(https://bugzilla.redhat.com/show_bug.cgi?id=894674)

Change-Id: I2118ce21b1f24cbdc614e85e92b3827068258e4f
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: turn off stat() for container list

Turn of stat() system calls used to fetch the file size during a
container listing operation since these system calls can swamp Gluster
and the result is most often not used.

When a GET or HEAD request is made on a container, stat() system calls
are made during the Python standard library method, os.walk, to
determine if a given directory entry is another directory to recurse
into, and then utils._update_list() will stat() each file to get it
size, and finally utils.get_container_details_from_fs() will stat()
each directory encountered.

For most installations we have seen so far, we don't need the
container listing to accurately return the size of all the objects in
the container, so we can reduce the number of stat() system calls by
not fetching the size of the object.

For now, turn it off by default, and provide an /etc/swift/fs.conf
configuration parameter to turn it back on:

  accurate_size_in_listing = yes

The default for the above is "no".

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I8617e5781da9ccfbe8695a7ee45af701ad560182
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: cleanup err handling in Glusterfs

Clean up the error handling in this module. This is a back-port of
work done as part of upstream refactoring work to remove the
dependency on the Swift differences we carry, see commit
6194691528d487417288e4dd0fa377fa06e85a04.

This reduces a number of errors seen at customer sites that just added
noise to the log files since the errors are recoverable situations.

See BZ 904370 (https://bugzilla.redhat.com/show_bug.cgi?id=904370).

Change-Id: Ic759fcab47c24e890a8655f0cdd79ffe14064330
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: turn off act and cont updates

For gluster, we don't need to update an account or a container when a PUT
operation occurs for a container or object, respectively.

Since the account is just the gluster volume (one-to-one mapping of account to
gluster volume mount point), the PUT of a container just creates a directory
in the root of the volume, so it has already "put" it into the account.

Since the object is just a file or directory placed into a container which, in
turn, is just a directory, the act of creating it has already "put" it into
the container.

Removing these two operations saves a round trip on PUT operations for
containers and objects, which is a big savings for objects.

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: Ic9b37defef3611d9ef342a17cbfe8040df9cca4e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: add worker max_clients parameter

Adding the "max_clients" parameter for a worker allows us to handle
one client at a time for a worker. This is very useful for debugging
behaviors.

But it is also useful for working around a starvation issue when the
file system underneath becomes very slow. By setting max_clients to 1,
and bumping up the number of workers significantly, we can attempt to
mitigate that starvation effect.

See BZ 844805 (https://bugzilla.redhat.com/show_bug.cgi?id=844805).

Change-Id: I4d7db1e0c0eb05da73f19d5abd5f32e4cec11475
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: provide more common cfg defaults

Provide more common configuration defaults, as seen and recommended across
many large customer installations, as well as provide some guidance on how to
set the parameters.

We also provide the new max_clients parameter for the object-server
configuration to help situations where long running file system operations end
up starving other requests in-flight in the same worker in other co-routines.

See BZ 904629 (https://bugzilla.redhat.com/show_bug.cgi?id=904629).

Change-Id: If9b38859a46fe23fc607ed6c318894484dd2b3be
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: update spec files to match RHS 2.0

Update the spec files to match what is shipped with RHS 2.0 and add a
trailer to the revision to mark these as special builds "pdq".

This was performed to be sure test RPMs handed to customers don't
conflict with the original product builds. See the following BZ
covering most of those customer issues: BZ 903396
(https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I408945ad69398cb7ed3d8511e0d64b63d1a5d51e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: remove stat of directories

It was painfully slow to gather a list of all the files in a directory tree
when there were as many directories as files, since we also stat'd all of the
directories as well. We only did that to cache using memcache the result, to
know when it changed. However, there was no way for a memcache object to be
passed down to this code. So we have removed the memcache support entirely,
and removed the stat of the directories as a result.

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I1bda65102fb15b34cbc52c9b2580d17917a9a1dd
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: don't perform listing for HEAD req

For both the account and container HEAD listing APIs don't walk the
entire tree to update the return values, as they don't need to be
accurate (for now). By removing the tree walk and accompanying flood
of stat() system calls, the system does not become stressed as a
result.

This was a change the original Gluster/Swift integration created, this
is not part of the behavior of Swift (since each PUT/DELETE operation
on a container/object update a single row in either the account or
container databases).

Note this change does not use the accurate_size_in_listing behavior
parameter since that is intended for GET requests on
containers/accounts.

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I702f58c2c0058cfa74101cf38ba4d24c5aaae3f7
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: avoid fetching metadata for lists

Only fetch the metadata for objects in a container, or containers in an
account, when the REST API asks for them (using an XML or JSON content-type).

We save many thousands of stat() and xattr system calls as a result.

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I053e5421f36beb5b947ca5add4e9a0c727dc2488
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: update the version to cover latest changes

This marks the commit covering the work for *.pdq.1.* test RPMs.

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I5fe8c1bf863d518220e239b44db4d42d62a91791
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: Refactor to use pyxattr's get/set/remove methods

These methods are not deprecated, the old ones with the xattr suffix are
considered deprecated starting with 0.4 of pyxattr.

Backport of upstream commit: 1105b880d06128e1428578dcf73c6680a98b35b8
See BZ 865858 (https://bugzilla.redhat.com/show_bug.cgi?id=865858).

Change-Id: Ie8fd9a98e7e117f555076dadfa5e587f75280d3c
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: remove unnecessary system calls around xattr ops

The following is a simple refactoring of the read and write metadata methods
to just make calls to get/set/remove xattr operations instead of trying to
stat/open/close the file. For a single GET operation, this brings down the
number of system calls made from 13 to 3 for a file object with no
significantly sized metadata (meaning, only 1 xattr key/value pair is needed
to store the metadata).

Backport of upstream commit: 51eace9fe9420ee3626f2622c282d00d23489d1b
See BZ 865858 (https://bugzilla.redhat.com/show_bug.cgi?id=865858).

Change-Id: Id509acc8e6e75cfd36cea890316cadaa69d7077a
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: reduce the number of getxattr system calls by one

Initial step towards addressing BZ 865619
(https://bugzilla.redhat.com/show_bug.cgi?id=865619).

Prior to the patch, reading metadata for a given object or container
involves at minimum three getxattr system calls for objects that use
only one xattr key/value pair:

    1. (via pyxattr)  getxattr() to see if key exists and get its value
       length (so it can allocate memory for second call below)
    2. (from pyxattr) getxattr() to get actual value data
    3. (via pyxattr)  getxattr() to see if following key exists

For objects and containers that only have to use one xattr key/value
pair, this patch reduces the number system calls by one. This can be
significant given that almost every Swift API operation requires reads
of the object or containers metadata.

This patch is mostly a change to plugins.utils.read_metadata() to try
to unpickle the accumulated metadata as each key/value pair is read,
rather than trying to accumulate all the key/value pairs and unpickle
at the end. Once we get enough data to form the pickle, we no longer
keep trying to get more keys, even if those keys exist.

The extra keys can exist when the size of the stored metadata shrinks
below a key threshold.

See https://bugzilla.redhat.com/show_bug.cgi?id=865619 for more
details. See also https://bugzilla.redhat.com/show_bug.cgi?id=865858.

This patch also adds a unit test infrastructure, and uses it to test
with full coverage, the read_metadata, write_metadata and
clean_metadata functions. As a result, we found that an infinite loop
would occur in clean_metadata() when an unexpected IOError would occur
trying to remove a key (this patch contains a fix).

Backport of upstream commit: 1f3ee888df53d0b22b4d732850f206e20af619a4

Change-Id: Iddd6931eaf0be67d5cfa562ae818244161e4ab87
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: further reduce extd attr rds/wrs

More refactoring towards reducing the number of extended attribute
reads/writes to the file system.

Basically the redundant routines restore_object, restore_account and
restore_container have been collapsed to one routine,
restore_metadata, which will only write out metadata if the new
metadata is different from what was originally read.

Backport of commit: 8ede7774470259e7849b11c1d3fa2d13daa06068
See BZ 868120 (http://bugzilla.redhat.com/show_bug.cgi?id=868120).

Change-Id: I6d037e90432f1c7be1b8dfaa55ad88257bf8a1a3
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: reduce stat calls gen'ing metadata

Reduce the number of stat/fstat system calls made while generating
metadata.

This refactoring consolidates the code a bit, and uses os.stat calls
to get all the information in one call when gathering the metadata for
an object. The five stat system calls (invoked via all the os.path.*()
calls) have been reduced to one.

Backport of commit: a79e6f63dc3f4dc10d1c2993231659a0387a9a33
See BZ 868086 (https://bugzilla.redhat.com/show_bug.cgi?id=868086).

Change-Id: I0fd16b75845f561cc7d9e12496a94b0a2ca1d43f
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: fix ismount; don't read conf on req

Write our own ismount() method which only calls lstat() 2 times
instead of 3.

Read the swift.conf and fs.conf files once per module instead of for
every request.

This is a partial backport of upstream commit:
    fe083592e91f2505ddb536a9df29d6dc84c4f2b3

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I51bef6139a545c7c98ee6cdadbb4bd86ffc1fee8
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: don't call fallocate() for Gluster FS

Swift always calls fallocate(), even though it always returns
ENOTSUPP. When using Gluster, don't make that call.

See BZ 903396 (https://bugzilla.redhat.com/show_bug.cgi?id=903396).

Change-Id: I1284fc0131dcac39635b7a64f52bca3190da8b5d
Signed-off-by: Peter Portante <peter.portante@redhat.com>