WindowsAzure/azure-sdk-for-python — pull request #92 by duggan (2013-06-19 17:16:54+00:00)
Title: Request chunking for blobs (a suggestion with POC)

This PR is more to start a conversation than anything else, but working code is nice :)

Blobs can be big. Really big. Currently, the SDK expects to be able to buffer blobs in memory before making them available, which is not really feasible with (for example) OS disk images.

A typical error might be:

```python
File "/usr/local/lib/python2.7/dist-packages/azure/storage/blobservice.py", line 380, in get_blob
response = self._perform_request(request)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 126, in _perform_request
resp = self._filter(request)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 117, in _perform_request_worker
return self._httpclient.perform_request(request)
File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 112, in perform_request
respbody = resp.read(resp.length)
File "/usr/lib/python2.7/httplib.py", line 567, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
OverflowError: signed integer is greater than maximum
```

The attached code adds some simple request chunking that doesn't interfere with existing functionality, though I think doing so is a little untidy. I'd prefer that chunking be integrated into the primary flow (`get_blob`), but don't see an obvious way to do so without breaking backwards compatibility.

In theory, a temp file could be used to do this a bit more seamlessly, but if `/tmp/` is on a small partition then we're back to square one.

I haven't tried uploading large blobs (yet), though I imagine there's a similar issue.

An example of this (POC) patch being used:

```python
from azure.storage import BlobService

blob_service = BlobService('',
'')

blob = blob_service.get_blob_chunked('', '', '')
```

PS: this PR contains some commits of whitespace fixes, something I have automated in vim - I'd recommend a project-wide cleanup, but add `?w=1` to the end of the URL for this PR to see the diff without whitespace.