fog/fog — pull request #1800 by burns (2013-05-05 17:04:48+00:00)
Title: [rackspace|storage] add large object container and prefix options

When storing an OpenStack Large Object, the object is split into segments which share the same
prefix and are suffixed such that they will sort in the order they should be concatenated back
together. `#put_object` is used to upload each segment, then `#put_object_manifest` is used to store
a manifest object with a `X-Object-Manifest` header that contains the prefix used for the segments.

`#put_object_manifest(container, object)` stores the manifest as `container/object` and uses this
same name for `X-Object-Manifest`. There is no way to specify a separate segment prefix.

Being able to use a name for the manifest object that's different from the prefix used by the
segments would make the manifest file easier to identify within the container. Also, while all
segments must be in the same container, the manifest file may be in a different container.

Being the same as the segments' prefix also creates a problem when you want to use the `Etag`
returned by a `#head_object` for the manifest object. This `Etag` is a MD5 hexdigest of the
concatenated `Etag`s for all the segments for the manifest.

```rb
etags = []
prefix = File.basename(large_file)
File.open(large_file, 'r') do |file|
while data = file.read(chunk_size)
etags << Digest::MD5.hexdigest(data)
suffix = etags.size.to_s.rjust(5, '0')
connection.put_object(container, "#{ prefix }-#{ suffix }", data, { 'Etag' => etags.last })
end
connection.put_object_manifest(container, prefix)
end

# etags => ["82d82ca1dbf1deb04f433fc213a57638", "0b7af2259525d8101f313fa6244ef97b"]
expected = Digest::MD5.hexdigest(etags.join) # => "7447e423f30e20a83ccb0882b90e1405"

# I don't know why this is returned with quotes, but it is.
actual = connection.head_object(container, prefix).headers['Etag'].gsub('"','')
```

You would expect `actual` to equal `expected`.
However, `actual` here is `f74ae5c6ab8115a9ed993c963dc67531`.

Since the manifest object's path is the same as the prefix for the segments, OpenStack sees the
manifest object as the first segment. So to compute the correct `expected` value, you must prepend
the manifest object's `Etag`:

```rb
etags.unshift "d41d8cd98f00b204e9800998ecf8427e" # => Digest::MD5.hexdigest('')
expected = Digest::MD5.hexdigest(etags.join) # => "f74ae5c6ab8115a9ed993c963dc67531"
```

This pull request gives you the option to create a manifest object with a name that will not match
the segments' prefix and avoid this work-around.

```rb
connection.put_object_manifest(container, 'large_object', 'segments_prefix' => 'large_object-')
```

You may also create the manifest file in a separate container, in which case it doesn't matter if
the manifest name and the segments' prefix is the same or not.

```rb
connection.put_object_manifest('container_a', 'large_object', 'segments_container' => 'container_b')
```