Faster swift-dispersion-populate

- Makes swift-dispersion-populate a bit faster when using a larger
  dispersion_coverage with a larger part_power.
- Adds option to only run population for container OR objects
- Adds option to let you resume population at given point (useful if you
  need to resume population after a previous run error'd out or the
  like) by specifying which suffix to start at.

The original populate just randomly used uuid4().hex as a suffix on the
container/object names until all the partition's required where covered.
This isn't a big deal if you're only doing 1% coverage on a ring with a
small part power but takes ages if you're doing 100% on a larger ring.

Change-Id: I52f890a774412c1d6179f12db9081aedc58b6bc2
Handle X-Copy-From header in container_quota middleware

Content length of the copied object is checked before
allowing the copy request according to the container quota.

Closes-Bug: #1201875
Change-Id: If44b916791e94ac6c66eee04a5727186ce0e56ae
Make test_translations test our tree

In order to run the correct classes, Python test framework adjusts
sys.path. However, these changes are not propagated to subprocesses.
Therefore, the test actually tries to test installed Swift, not
the one in which it is running.

The usual suggestion is to run "python setup.py develop" before
testing, but it's annoying and error-prone. If you forget it,
you may test the code in /usr very easily, and never know.

Let's just pass the correct path to subprocess. Much safer.

Change-Id: Ic71314e8462cf6e0579d704ffe9fbbfac7e6ba24
Fix 500 from account-quota middleware

If a user had set X-Account-Meta-Quota-Bytes to something non-integer
prior to the installation of the account-quota middleware, then the
quota check would choke on it. Now a non-integer value is treated as
"no quota".

Change-Id: I5c38911be1f66fa293aea9c78590d4ce7d184113
Small efficiency improvement in account quotas

If you were performing an object PUT with X-Copy-From and you had
account_quotas in your pipeline but your account did not have a quota
set, then we will now avoid an unnecessary call to get_object_info().

Change-Id: I8592d4ad65e45ecf9e0557bc10ce2c2db19732e8
Merge "Handle X-Copy-From header in container_quota middleware"
Small optimization to container_quotas

When handling an object PUT with X-Copy-From, we only need to fetch
the source object's size if the container has a byte quota, but we
were always fetching it. Now we only fetch the size when we need it.

Change-Id: I0c2c6820cdf0022ef57df5fe7dcb2dd466665a4f
return value on memcache client decr call

Closes-Bug: 1238204
Change-Id: I7d2ece118159637e5c5bf4bf5dd4ce4d2f6e4523
Removes vim comment

Change-Id: Ibc930638ee2d839e8499a13018c205b615f8b854
Partial-Bug: 1229324
Merge "Removes vim comment"
Merge "Fix 500 from account-quota middleware"
Handle COPY verb in account quota middleware

Since COPY verb allow to copy an existing object
we must check the size of the source object before
allowing the copy.

Fixes bug: #1201884

Change-Id: Ia37bc716be0c3e5a3174dc4370bb5084f81073ad
Proxy: use just one greenthread per client

Instead of using two greenthreads and an eventlet.queue.Queue of size
1 to stream response bodies from backends to clients, just do it with
a single greenthread that reads and writes in a loop. This should
lower the amount of CPU used by the proxy in its response streaming.

Client fairness used to be provided implicitly; since the queue only
held 1 item, the read-from-backend greenthread would block after each
chunk until the write-to-client greenthread got a chance to run. Now
fairness is provided by an explicit eventlet.sleep() call, just as it
is in the object server.

Change-Id: Iae27109f5a3d109ad21ec9a972e39f22150f6dbb
DiskFile API, with reference implementation

Refactor on-disk knowledge out of the object server by pushing the
async update pickle creation to the new DiskFileManager class (name is
not the best, so suggestions welcome), along with the REPLICATOR
method logic. We also move the mount checking and thread pool storage
to the new ondisk.Devices object, which then also becomes the new home
of the audit_location_generator method.

For the object server, a new setup() method is now called at the end
of the controller's construction, and the _diskfile() method has been
renamed to get_diskfile(), to allow implementation specific behavior.

We then hide the need for the REST API layer to know how and where
quarantining needs to be performed. There are now two places it is
checked internally, on open() where we verify the content-length,
name, and x-timestamp metadata, and in the reader on close where the
etag metadata is checked if the entire file was read.

We add a reader class to allow implementations to isolate the WSGI
handling code for that specific environment (it is used no-where else
in the REST APIs). This simplifies the caller's code to just use a
"with" statement once open to avoid multiple points where close needs
to be called.

For a full historical comparison, including the usage patterns see:
https://gist.github.com/portante/5488238

(as of master, 2b639f5, Merge
 "Fix 500 from account-quota     This Commit
 middleware")
--------------------------------+------------------------------------
                                 DiskFileManager(conf)

                                   Methods:
                                     .pickle_async_update()
                                     .get_diskfile()
                                     .get_hashes()

                                   Attributes:
                                     .devices
                                     .logger
                                     .disk_chunk_size
                                     .keep_cache_size
                                     .bytes_per_sync

DiskFile(a,c,o,keep_data_fp=)    DiskFile(a,c,o)

  Methods:                         Methods:
   *.__iter__()
    .close(verify_file=)
    .is_deleted()
    .is_expired()
    .quarantine()
    .get_data_file_size()
                                     .open()
                                     .read_metadata()
    .create()                        .create()
                                     .write_metadata()
    .delete()                        .delete()

  Attributes:                      Attributes:
    .quarantined_dir
    .keep_cache
    .metadata
                                *DiskFileReader()

                                   Methods:
                                     .__iter__()
                                     .close()

                                   Attributes:
                                    +.was_quarantined

DiskWriter()                     DiskFileWriter()

  Methods:                         Methods:
    .write()                         .write()
    .put()                           .put()

* Note that the DiskFile class   * Note that the DiskReader() object
  implements all the methods       returned by the
  necessary for a WSGI app         DiskFileOpened.reader() method
  iterator                         implements all the methods
                                   necessary for a WSGI app iterator

                                 + Note that if the auditor is
                                   refactored to not use the DiskFile
                                   class, see
                                   https://review.openstack.org/44787
                                   then we don't need the
                                   was_quarantined attribute

A reference "in-memory" object server implementation of a backend
DiskFile class in swift/obj/mem_server.py and
swift/obj/mem_diskfile.py.

One can also reference
https://github.com/portante/gluster-swift/commits/diskfile for the
proposed integration with the gluster-swift code based on these
changes.

Change-Id: I44e153fdb405a5743e9c05349008f94136764916
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Merge "Small optimization to container_quotas"
Merge "DiskFile API, with reference implementation"
Adding more unit tests for audit_location_generator

Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
Merge "Proxy: use just one greenthread per client"
Merge "Small efficiency improvement in account quotas"
Merge "Make test_translations test our tree"