rackspace/pyrax — pull request #449 by aztlan2k (2014-08-19 12:45:45+00:00)
Title: fix object mgr _fetch_chunker() to use chunk_size appropriately

for your consideration...

- the get request was using the `chunk_size` incorrectly and therefore
causing **data corruption** on fetched objects.
- request was ending up retrieving `chunk_size+1` bytes (`0 -> chunk_size`; inclusive)
- the next request would start at `chunk_size`, thus overlapping
the previous chunk by 1 byte. (`0-4096`, `4096-8192`, etc...)
[last byte (4096) of previous read was re-read as the first byte
on next fetch]

Here's what my trace of a `fetch_object(container, obj, chunk_size=4096)` call (_within `_fetch_chunker()`_):

```
pyrax: chunk_size = 4096
pyrax: obj_size = 102400
pyrax: headers = {'Range': 'bytes=0-4096'}
CF: len(chunk) = 4097
CF: chunk[0] = 05 ; chunk[-1] = 68
pyrax: headers = {'Range': 'bytes=4096-8192'}
CF: len(chunk) = 4097
CF: chunk[0] = 68 ; chunk[-1] = 76
pyrax: headers = {'Range': 'bytes=8192-12288'}
...
```

notice that even though we asked for chunks of 4096 we're actually retrieving 4097 bytes. And, each subsequent read starts at the last byte of the previous read. This causes data corruption with bytes being duplicated. Below is a hexdump of the two files i used to compare. On the left is the original. On the right is the copy that's be uploaded and then downloaded once again using `fetch_object()`:

```
0000fe0 1d 8d 59 6a 3e 39 b0 7f d0 c9 b2 72 3c 23 51 70 |0000fe0 1d 8d 59 6a 3e 39 b0 7f d0 c9 b2 72 3c 23 51 70
0000ff0 25 08 fc c9 33 4f 64 fb 0b cb 2b 00 ec 52 58 fa |0000ff0 25 08 fc c9 33 4f 64 fb 0b cb 2b 00 ec 52 58 fa
> 0001000 68 a5 7c 45 a7 a9 a2 ed 94 b5 25 78 da f8 f4 e2 |0001000 68 68 a5 7c 45 a7 a9 a2 ed 94 b5 25 78 da f8 f4
0001010 85 fd d8 b1 40 f4 34 04 8d 23 f6 fa 57 8e 7b cb |0001010 e2 85 fd d8 b1 40 f4 34 04 8d 23 f6 fa 57 8e 7b
0001020 ed 2b db ac 25 4d 54 53 0b 1d 6f 6b 24 ee 1d cf |0001020 cb ed 2b db ac 25 4d 54 53 0b 1d 6f 6b 24 ee 1d
```

Note that on byte 0x1000 (byte 4096) we see the byte value 68 repeated and then everything shifts over by a single byte.