object-storeage: refactor to use one memcache key

Address BZ 868087: https://bugzilla.redhat.com/show_bug.cgi?id=868087

Store all of the data needed to generate the correct set of container and
account details in one object, respectively, rather than using three seperate
memcache keys.

Change-Id: I46bf60c405b37cdb22727965bfd67bc5c410e77c
BUG: 868087
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4139
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
object-storage: use constants for directory names

Change-Id: I6109ee55061eae53c4c6762ca31fb5c4b13f0071
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4140
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
api/examples: updated examples to use new calls

Change-Id: I865d0c52de268b27bbdb8786d3f4af5ef3ce3d9a
BUG: 839950
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4023
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
POSIX sh uses = and not ==

BUG: 815227
Change-Id: I5a498f1b917fb658914133ee071783e7b8e0d025
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Reviewed-on: http://review.gluster.org/4151
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
object-storage: remove glusterfs filter requirement

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=870589

Remove the Glusterfs object, transforming it into a module providing module
data fields (like swift.common.constraints) and module methods for
mounting/unmounting and access the gluster volume information. As a result, we
can then remove the glusterfs filter from the pipeline since we no longer need
to provide the Glusterfs object through all the plugin code paths.

This is one more step closer to removing our dependency on modifying the Swift
code directly with these changes. See It is also the first step to acknowledging
that we are not a plugin, but a layering on top of Swift.

The major piece of work here is based on a recognition that the
plugins/Glusterfs.py module provided a Glusterfs class that instantiated
instances of an object that always contained the same data from the
configuration file. The fields of such an object were not being changed and
were treated as read-only in all cases. Since the object's data was the same
for all instantiations there was no need to pass the data from the glusterfs
filter all the way down into the bowels of the Gluster_DiskFile and DiskDir
objects.

Taking advantage of the nature of that data, we now just have those fields
read into module variables, and change the Glusterfs object methods into
module level functions. Much of the changes result from the consequence of
making that switch from object to module.

Here are a few other changes made along the way:

  * Bump the release numbers in the spec files in recognition of these changes

  * Create the plugins/fs_utils.py module so that the methods in the
    plugins/Glusterfs.py module don't have to include plugins/utils.py, which
    would create a circular dependency

    * Note that this dependency comes from methods in plugins/utils.py
      depending on the module level constructs in plugins/Glusterfs.py so that
      we only store those values in one place

  * Changed plugins/DiskDir.py:DiskDir class to not check for, and/or
    optionally create, the /etc/swift/db_file.db at run time, just create it a
    module init time

  * Removed the duplicate strip_obj_storage_path() from plugins/DiskDir.py and
    utils.py and move it to the Glusterfs module

  * Used os.path.join in plugins/DiskDir.py where possible

  * Renamed the .conf files to .conf-gluster so that we don't clobber existing
    config files

    * This is not a complete change, as the spec file also needs to be
      modified to avoid the clobbering

    * See also https://bugzilla.redhat.com/show_bug.cgi?id=865867

  * Removed the redundant DIR_TYPE definition in plugins/utils.py

  * Removed MOUNT_PATH from plugins/utils.py replacing references with that from
    Glusterfs

    * This actually fixes a bug if a user every used a different mount path
      from the default in fs.conf

  * Added ASYNCDIR definition to plugins/utils.py until such time as another
    refactoring can rely on the one from swift.obj.server

  * Renamed plugins/utils.py's plugin_enabled() function to Gluster_enabled()

  * The diffs we carry for Swift are now a bit smaller in that we no longer
    have to add the plugin() method, we don't have to keep a fs_object field
    in these objects, and we can reference the Glusterfs module directly

  * Unit tests were modified appropriately, but now need to be run in the
    context of a Swift tree; this is unfortunate, but further refactoring will
    address this

Change-Id: Id5d2510d56364761c03b3979bc71187dbe2f82fe
BUG: 870589
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4141
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
object-storage: minor code cleanups

Apply a few minor code cleanups to help reduce future refactoring diffs.

  * Remove extra lines between methods so that all methods are consistently
    spaced by one line

  * Remove unused field meta_file

  * Add missing documentation for uid and gid parameters

Change-Id: I0a38eb06680eb0709edfaa1828d09657823f778e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: add missing docs for params

Add missing documentation for constructor parameters.

Change-Id: I3a656cd22f6a2a61e0b4d563c75c2f7fe68d3c54
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: unused parameter cleanup & handl'n

Unsed parameter cleanup and handling changes. The device and partition
parameters are not used. The account parameter is used in place of the device
parameter. Rather than just making the device parameter reference the value of
the account parameter, we just don't use the device parameter at all and make
references to account. We also remove the device_path field from the class
since it is unused.

Additionally, we assert that we have a logger since the rest of the relies on
that fact.

Change-Id: Iea556319744ab4f729b1ce23d987650a164749db
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: rename self.name to self.container

Minor refactoring ahead of further diffs to add clarity to the code.

Rename the DiskDir field from "name" to "container", which is what it is.

Along the way, add a note regarding the one-to-one mapping of account name to
gluster volume name, and fix the get_info() container method documentation to
list all the dictionary keys returned.

Change-Id: I659e1a9a070c16671287f17a170f9d77198534e7
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: fix bad refs to REMOTE_CLUSTER

Change-Id: I07cd997ef9a98645a87792632f9dd612928678b7
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: fix bad refs to check_acct_exists

Fix bad references to check_account_exists: this is a module local method, not
provided by Glusterfs. Not sure how we missed this. More unit tests would
help.

Also added the ability to run the Swift tree unit tests with gluster
artificially enabled to test as many code paths as possible ahead of full
functional tests.

Change-Id: Iaf696d34db8f841c887f03569e65321fca8894d1
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: DiskAccount subclassing of DiskDir

Changes to make DiskAccount subclass DiskDir complete. This change helps
reduce the diffs ahead of further refactoring. Note that DiskDir initializes
all the same fields as DiskAccount, plus a few more, and ends up reading the
metadata properly, while only adding a couple of fields not used for account
purposes.

This change also passes the account controller's logger for use by the
DiskAccount class.

Along the way we:

    * Remove the redundant update_account() method in the DiskCommon base
      class, since the update_container() method in DiskDir performs the exact
      same functionality

    * Remove the initialization of variables ahead of assignments in two
      cases, and also move the initialization of the list variables closer to
      where they are used

    * Collapse the imports from swift.plugins.utils into one statement

    * Use a temporary directory for the db_file.db when we find ourselves in a
      unit test environment

Change-Id: I4dc59b416a66393adfe0ed32c7a62c79731d70c0
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: only update metadata that changes

See BZ 868120 (https://bugzilla.redhat.com/show_bug.cgi?id=868120)

Change-Id: Ia7f5562cff32eee30c5ca18652d5d9a5a2b18bf4
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: remove the device and part params

Change-Id: I60b4b1b976ad8359ba072d5d0ed6a8dd2087a738
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: fix trailing whitespace

Change-Id: Ib3c671e693c2c332af98a593ca14c42c36f5ca76
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: swift-diff removal: hide obj only

Towards swift-diff removal: hide object_only behavior inside DiskDir class
objects. See BZ 870589 (https://bugzilla.redhat.com/show_bug.cgi?id=870589)

One of the last few reasons for having the swift diff is the need to update
the account and container information from the file system when the
configuration is not for object-only use (meaning, other updates to the file
system can come from clients other than UFO, which means UFO currently has to
generate the necessary metadata for each request instead of caching it).

This change moves that logic into the DiskDir class methods so that the
container and account server code does not have to track that.

Change-Id: If0970c77c72245facd09c2035083e681d6d80303
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: reduce diffs, bury metadata update

Further reduce the diffs we have to carry by moving the Gluster specific
metadata stored when we create an object to the DiskFile class's put() method.

Change-Id: I8cef5412627d3ddfdadbf78ef947e067db5bf620
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: make proper refs to module vars

Not sure how this worked before this, but mount_ip and mount_path do not
exist, their uppercase versions do.

Change-Id: I53e9b550991d17b5541ed4ada5f1bc9b3cb1b8b1
Signed-off-by: Peter Portante <peter.portante@redhat.com>
object-storage: refactor to use swift devices

Refactor code to use the devices configuration file setting for account,
container, and object servers, dropping mount_path from the fs.conf file, and
constructing new account, container and object server rings.

This removes the next to last set of diffs in the swift diff file. See BZ
870589 (https://bugzilla.redhat.com/show_bug.cgi?id=870589).

The key to the change is the dropping of the pre-built account, container and
object rings, instead providing a script that will generate them for the user
given the gluster volume name in use. In addition, we override the Swift
check_mount() method and replace it with Gluster's which attempts to
"auto-mount" if it is not already mounted.

The following is an enumeration of the changes contained in this refactoring:

    * The refactoring to override the Swift check_mount() involved condensing
      a lot of code, removing redundancies and simplifying methods across a
      number of modules

    * Drop checking the mount point in the low level DiskDir, DiskAccount and
      DiskFile objects now that Swift's normal mount checking is used, and
      enable it by default in the template configuration file

    * Add missing get_container_timestamp() method for DiskAccount objects

    * Fix the plugin RPM spec file to provide the new ring builders, and while
      we were at it, finally fix the over-writing of the configuration files
      on install

    * Bug fixes

        * The monkey patched version of check_object_creation was not working
          correctly due to a missing import of HTTPBadRequest and a bad
          reference to validate_obj_name_component()

        * Only have the utils module import the plugins constraints module for
          the side effect of monkey patching if gluster is enabled

        * Removed the db_file.db file in the tree since it is created on the
          fly

        * Updated 2011 copyright notices to 2012

Change-Id: I8f4454576b1423021c9bbf3c36176f8db51e62c0
Signed-off-by: Peter Portante <peter.portante@redhat.com>