cluster/dht: Add "afr.readdir-failover=off" option the rebalance process

By failing over readdir (default behaviour), rebalance could get duplicate
files, as readdir would re-read from offset 0. Rebalance should not attempt
to migrate these files again.

Additionally, we need to handle these cases as failure in rebalance crawl.

No test case provided, as we cannot determine the read child in afr.

Change-Id: If07508b4f92dacc17e0f695b48a866c7c66004be
BUG: 859387
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/4300
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
object-storage: Initial unittest of DiskFile class

If we had this ahead of time, we could have avoided the errors that
were encountered leading to the fix-account-mapping fix (see
http://review.gluster.org/4222).

This represents 100% coverage of the DiskFile module, but the coverage
report says otherwise, unfortunately. That is because the put() method
invokes eventlets during the test run, and coverage is not able to
accurately track the coverage lines properly. If one comments out the
"tpool.execute()" line in DiskFile.put() the coverage report then
reports 100% for the DiskFile module.

Additionally, we changed DiskFile.put() in four ways that should not
change its behavior:

  1. Comments were added to explain various code paths and mark
     potential issues / fixes

  2. It no longer returns a boolean value, matching the behavior of
     swift.obj.server.DiskFile.put()

  3. It no longer logs a message when it detects a directory that
     already exists, instead is raises an exception

     We believe this is okay because we cannot find a code path that
     would lead to his condition. As a result, it makes it easier to
     test all the code paths in that routine.

  4. It no longer logs a message when create_dir_object() fails, since
     create_dir_object() raises an exception on failure only

     We also modified create_dir_object() to not return a boolean as a
     result of the above behavior.

Note that by implementing these tests up to this point we found three
code paths that would have failed if encountered due to missing
imports. We also made changes to the DiskFile module to make it a bit
easier to test, also eliminating an extra stat system call when
deleting directory objects.

Change-Id: I3286de83c1ec7c5e8d6cab9354e1c4397cee7497
BUG: 870589
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4284
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
object-storage: a set of cleanups to DiskFile

Here are a set of cleanups for things noticed while working on the
unit test coverage:

  * Remove unused constants, adding a constant for default disk chunk
    size

  * Document missing logger parameter from constructor

  * Add iter_hook paramater to constructor in anticipation of 1.7.4
    support

  * Add back meta_file field even though it is not used in our version
    of the constructor for paranoid compatibility

  * Remove is_valid field as it is not referenced anywhere, and it is
    not a field of the super class

  * Rename fields only used internally by DiskFile with leading
    underscores

  * Convert to using os.path.join() instead of hard coded '/'
    references

  * Use data_file field where possible

  * Assert that put_metadata() will only work when the file exists

  * Remove update_object() method since it is identical to
    put_metadata() and a direct write will suffice

  * Create the directory object only if it does not exist when a
    marker directory is requested

Change-Id: If207fed4c0f423e6bd3232e4507f87de877057c4
BUG: 876660
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4287
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
object-storage: apply upstream DiskFile refactor

Apply the upstream DiskFile refactoring in ahead of its use to easiliy apply
the temp file optimization.

Change-Id: I2708733eed3d87759c70eb3d9e6cd74ef91d0c7b
BUG: 876660
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4288
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
object-storage: do not list directory for delete

Do not list the entire directory contents of the directory containing the file
before deleting it. We have already verified the files existence constructing
the Gluster_DiskFile object.

This fixes a customer issue. See 885281
(https://bugzilla.redhat.com/show_bug.cgi?id=885281).

Change-Id: I4425c42af3e03624370c779d74b7f073e6eef024
BUG: 885281
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4302
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
object-storage: don't sort object list, not req'd

The REST API for listing objects in a container does not require that the list
of objects be sorted (the API for listing containers in an account does
require it). Since we can have thousands and thousands of objects in a
container, don't sort them when it is not required.

Change-Id: I6939ef3fec3ea3814a49e3a3046273304889831c
BUG: 887301
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4312
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
object-storage: only open file when requested

Only open a file when the caller of the constructor indicates that it
expects to have a file pointer to access in the "fp" field.

Found by Junaid during review http://review.gluster.org/4284, Patch
set 6.

We also bump the version number to mark a line in the set of changes
we want to perform a mini release with to double check for performance
regressions.

Change-Id: I13cf336bb519088890192ee111f4ece85a5982c4
BUG: 886730
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4303
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>