richleland/django-cumulus — pull request #137 by bennylope (2014-01-24 20:16:39+00:00)
Title: Try requesting object, handle missing object by exception

**Summary**: The `_get_object` method tests whether an object should be returned by first requesting the entire list of object names from the container. It does this for every call to `_get_object`. If the object is present a subsequent call is made for the object, else a `False` value is returned. I think this can be simplified and performance improved by trying to retrieve and return the object and returning a falsifiable value if the service cannot find the object, rather than requesting a list of all object names from the container.

------

**In this change**: if an object is present this means one request is made to get the object, vs one request to get a list of object names and then another to get the object. The number of requests required for a missing object is still one.

The time required for these requests is the real difference. I've noticed in a development site that pages with CloudFile sourced images take much longer to load after updating to the latest version of Django Cumulus using Pyrax (from a custom fork with cached containers, but that's a different issue), and suspect that it's excessive calls to the CloudFiles API. If that's the case, then anything we can be done to minimize those calls would be great.

I tested the `get_object_names` and `get_object` methods to see what kind of time they're taking. While these are methods on the Pyrax container, they're used by Cumulus.

Here's what I started with from the Django shell of my local project with configuration already set:

```python
from cumulus.storages import SwiftclientStorage
client = SwiftclientStorage()

def pyrax_time(kallable, iters, sets, *args, **kwargs):
"""
Reports cumulative time for `iters` number of
repetitions over `sets` number of sets
"""
for x in range(sets):
start_time = time.time()
for y in range(iters):
try:
kallable(*args, **kwargs)
except:
pass
print time.time() - start_time, "seconds"
```

For each test I ran 5 sets.

First I tested against the list of objects from this container (782 objects, total):

```python
>>> pyrax_time(client.container.get_object_names, 5, 5)
7.47269201279 seconds
7.66448092461 seconds
4.50465297699 seconds
3.44787693024 seconds
3.24015784264 seconds
```
I originally started at 100 repetitions for 5 sets but had to interrupt while waiting for the first set to complete.

Then I tested against an object I knew existed:

```python
>>> pyrax_time(client.container.get_object, 5, 5, 'photos/who-framed-roger-rabbit.jpg')
3.60012054443e-05 seconds
0.000520944595337 seconds
0.000123023986816 seconds
1.50203704834e-05 seconds
1.50203704834e-05 seconds
```

I ran it again at 100 repetitions per set:

```python
>>> pyrax_time(client.container.get_object, 100, 5, 'photos/who-framed-roger-rabbit.jpg')
0.000938892364502 seconds
0.000381946563721 seconds
0.000221967697144 seconds
0.000216007232666 seconds
0.000180006027222 seconds
```

And for a missing object?

```python
>>> pyrax_time(client.container.get_object, 5, 5, 'photos/who-framed-roger-rabbit.png')
7.78252291679 seconds
7.19900798798 seconds
7.43221020699 seconds
7.945759058 seconds
6.46154594421 seconds
```

This is closer to the time for getting the list of object names, at least within the same order of magnitude.

(I extracted those times from a lot of error output in the console, that's not what the exact output looked like).

Now if a requested object is present the response time is miniscule. If it's missing the response time is roughly about the response time (here at least) for listing all object names in the previous code. So overall the time required should be much less. On an individual level these numbers are pretty tiny, but cumulatively I think they're noticeable.

**So this reduces the number of requests, and should reduce the overall request time to the CloudFiles API.**

Almost lastly: it's not clear to me that a `swiftclient.exceptions.ClientException` would be raised since the swiftclient's `get_object` method doesn't take the same arguments, but for 'completeness' I've covered that here.

Lastly: I changed the return value of `_get_object` to `None` as this seems more consistent with a method that returns an object. It still evaluates to `False` in the `bool` test against this return value. However this change isn't necessary to the thrust of the patch.