kylemacfarlane/django-cb-storage-s3 — pull request #12 by whitews (2013-09-07 21:24:12+00:00)
Title: Incomplete downloads due to InvalidRange response from AWS

Hi Kyle,

I started getting incomplete downloads using cuddlybuddly. I'm pretty sure this wasn't happening before, maybe AWS changed something? Anyway, the files were missing bytes at the end of the file, around a dozen to a couple hundred bytes. I checked the server logs, and noticed S3Error messages reporting InvalidRange:

```
File "/.../cuddlybuddly/storage/s3/storage.py", line 326, in read,
data, etags, content_range = self._storage._read(self.name, *args),
File "/.../cuddlybuddly/storage/s3/storage.py", line 196, in _read,
raise S3Error(response.message),
S3Error:
,

InvalidRange
The requested range is not satisfiable
12884984
......
bytes=12884984-12950519
,
```

Not sure why I'm seeing this and no one else has reported it, perhaps my setup is different somehow, but I don't think I'm doing anything too wacky. I'm using Django 1.5.2 and the latest cuddlybuddly 2.18. I wrote a script to reproduce the error locally so I could debug easier. Here's that code:

```
from django.conf import settings

settings.configure(AWS_HEADERS=[])

import cuddlybuddly.storage.s3.storage as cb

AWS_ACCESS_KEY_ID = '...'
AWS_SECRET_ACCESS_KEY = '...'
AWS_STORAGE_BUCKET_NAME = 'some-bucket'
S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME

DEFAULT_CHUNK_SIZE = 2**16 * 4

storage = cb.S3Storage(
bucket=AWS_STORAGE_BUCKET_NAME,
access_key=AWS_ACCESS_KEY_ID,
secret_key=AWS_SECRET_ACCESS_KEY,
calling_format=2,
cache=None,
base_url=S3_URL
)

s3_file = cb.S3StorageFile(
name='path/to/file/some_file.bin',
storage=storage,
mode='r',
)

# imitate Django's file read behaviour
# see File.chunks() in base.py in django.core.files.base
def get_data(s3_file):
data = ''
while True:
new_chunk = s3_file.read(DEFAULT_CHUNK_SIZE)
if not new_chunk:
break
data = data + new_chunk

return data

print "Getting file without chunking..."
s3_file.seek(0)
file_string_1 = s3_file.read()
file_1 = open('file_1', 'wb')
file_1.write(file_string_1)
file_1.close()

print "Getting file with chunking..."
s3_file.seek(0)
file_string_2 = get_data(s3_file) # <-- Kablamo!
file_2 = open('file_2', 'wb')
file_2.write(file_string_2)
file_2.close()
```

After looking at File.chunks() in base.py in django.core.files.base, I put together this pull request which should be compatible with what Django expects, since it checks for None when reading a new chunk. I also updated the version to 2.19.

I'm curious as to your thoughts, please let me know if there's more info I can provide.

Thanks,
Scott