Handle X-Copy-From header in account_quota mw

Content length of the copied object
is checked before allowing the copy
request according to the account
quota set by Reseller.

Fixes: bug #1200271
Change-Id: Ie4700f23466dd149ea5a497e6c72438cf52940fd
Optimizations to Ring.get_more_nodes()

When we're looking for handoffs, we first try to span all the regions,
then zones, then devices. However, the search loops would always go
over a big subset of the ring, regardless of whether or not there was
anything left to find.

In particular, this would result in long runtimes for the first call
to get_more_nodes() if you didn't have more regions than replicas, as
the region-search loop would chew through a ton of devices and not
find any (since there weren't any). Now we do a little arithmetic,
think about the pigeonhole principle a little bit, and bail out when
there's nothing left to find, rather than waiting until there's no
space left to search.

Similar changes were made for the different-zone and different-device
search loops.

On a 4800-device test ring (4 regions, 5 zones each, 20 node each, 12
drives each), the time to get all handoffs dropped to about 5% of its
previous value (a 95% improvement).

More usefully, on a 1200-device test ring (same as above but only 1
region), the time to get just the first 6 handoffs dropped to about
0.01% of its original runtime, or a 10,000x speedup.

Fixes bug 1225018.

Change-Id: I4c77094186f0032a3e19a099a1a0e71b2ba06719
Add a user variable to templates

This is a common configuration option and allows devstack (and others)
to configure a mod_wsgi user to run the daemon process.

Change-Id: Idf134b3bc6b08e3c3a80dde8830d5a4f3da5a06c
Fixes: bug 1226346
Add HTTPSeeOther(303) and HTTPTemporaryRedirect(307) to swob

RFC 1945 and RFC 2068 specify that the client is not allowed to change the
method on the redirected request.  However, most existing user agent
implementations treat 302 as if it were a 303 response, performing a GET on
the Location field-value regardless of the original request method. The status
codes 303 and 307 have been added for servers that wish to make unambiguously
clear which kind of reaction is expected of the client.

HTTP/1.1 RFC for 302: http://tools.ietf.org/html/rfc2616#section-10.3.4
HTTP/1.1 RFC for 307: http://tools.ietf.org/html/rfc2616#section-10.3.8

Change-Id: I354e2f4f3e3eb6a1553b3d9c60b613d8f0c37531
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Merge "Optimizations to Ring.get_more_nodes()"
Merge "Handle X-Copy-From header in account_quota mw"
Merge "Refactor common/utils methods to common/ondisk"
Merge "Add HTTPSeeOther(303) and HTTPTemporaryRedirect(307) to swob"
Fix probe tests

Change-Id: I03573bf24baf031b1874c3ff2e2d89d34473c266
add seek() to CompressingFileReader

Change-Id: I33d9119684497b53101db3bc380953e86bdf25f0
Merge "add seek() to CompressingFileReader"
Merge "Add a user variable to templates"
SLOs broken for range requests

Change-Id: I21175a4be0cda9a8a98c425bff11c80895cd6d3e
Fix unsuitable assertTrue

assertTrue accepts a parameter msg which will be printed when
assertion fails, usually msg is a str. This patch fixes unsuitable
usage of assertTrue which set msg to bool type True.

Change-Id: I731f8ea553c935eba0e112ffded16f41a5ea86c0
Fixes-Bug: #1226374
Merge "SLOs broken for range requests"
Refactor on-disk knowledge out of object server

Refactor on-disk knowledge out of the object server by pushing the
async update pickle creation to the new DiskFileManager class (name is
not the best, so suggestions welcome), along with the REPLICATOR
method logic. We also move the mount checking and thread pool storage
to the new ondisk.Devices object, which then also becomes the new home
of the audit_location_generator method.

For the object server, a new setup() method is now called at the end
of the controller's construction, and the _diskfile() method has been
renamed to get_diskfile(), to allow implementation specific behavior.

Change-Id: Id945276037dbbeb8a778753e47a7316cd408eac7
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Use new ondisk.Devices where possible

We also remove the unused mount_check attribute from the recon
middleware.

Change-Id: Icc593fd548a6e2d9bc5c8178f78424b1d6473f05
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Refactor DiskFile hiding quarantine adding reader

This set of refactorings is proposed as the last step before
officiallly declaring the DiskFile class, and its helpers, to be the
backend API servicing the Object Server REST API implementation code.
The goal is to ensure that the REST API layer in the object server
does not have any access to internal knowledge of how the DiskFile
class is implemented.

We hide the need for the REST API layer to know how and where
quarantining needs to be performed. There are now two places it is
checked internally, on open() where we verify the content-length,
name, and x-timestamp metadata, and in the reader on close where the
etag metadata is checked if the entire file was read.

We add a reader class to allow implementations to isolate the WSGI
handling code for that specific environment (it is used no-where else
in the REST APIs). This simplifies the caller's code to just use a
"with" statement once open to avoid multiple points where close needs
to be called.

For a full historical comparison, including the usage patterns see:
https://gist.github.com/portante/5488238

(as of master 8a255a3 Merge
 "Refactor finalize_put as an    This Commit
 object method")
--------------------------------+------------------------------------
                                 DiskFileManager(conf)

                                   Methods:
                                     .pickle_async_update()
                                     .get_diskfile()
                                     .get_hashes()

                                   Attributes:
                                     .devices
                                     .logger
                                     .disk_chunk_size
                                     .keep_cache_size
                                     .bytes_per_sync

DiskFile(a,c,o,keep_data_fp=)    DiskFile(a,c,o)

  Methods:                         Methods:
   *.__iter__()
    .close(verify_file=)
    .is_deleted()
    .is_expired()
    .quarantine()
    .get_data_file_size()
                                     .open()
                                     .read_metadata()
    .create()                        .create()
                                     .write_metadata()
    .delete()                        .delete()

  Attributes:                      Attributes:
    .quarantined_dir
    .keep_cache
    .metadata
                                *DiskFileReader()

                                   Methods:
                                     .__iter__()
                                     .close()

                                   Attributes:
                                    +.was_quarantined

DiskWriter()                     DiskFileWriter()

  Methods:                         Methods:
    .write()                         .write()
    .put()                           .put()

* Note that the DiskFile class   * Note that the DiskReader() object
  implements all the methods       returned by the
  necessary for a WSGI app         DiskFileOpened.reader() method
  iterator                         implements all the methods
                                   necessary for a WSGI app iterator

                                 + Note that if the auditor is
                                   refactored to not use the DiskFile
                                   class, see
                                   https://review.openstack.org/44787
                                   then we don't need the
                                   was_quarantined attribute

Change-Id: Iefbbe7aada972b1ff2fc9fd5492646ab01fa192c
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Document DiskFile API, provide reference impl

Add sphinx documentation for the DiskFile pluggable backend API. We
also updated references to the other account and container backend
docs.

** We have removed the "upload_size" attribute of the DiskFileWriter,
   changing it to be a return value of the "write()" method instead
   (one step simpler API).

As part of documenting the interface, all of the attributes have had
underscores ("_") appended to define them as internal to the reference
implementation.

A reference "in-memory" object server implementation of a backend
DiskFile class in swift/obj/mem_server.py and
swift/obj/mem_diskfile.py.

One can also reference
https://github.com/portante/gluster-swift/commits/diskfile for the
proposed integration with the gluster-swift code based on these
changes.

Change-Id: I44e153fdb405a5743e9c05349008f94136764916
Signed-off-by: Peter Portante <peter.portante@redhat.com>