incomplete file upload

On Windows 7 64bit, uploading a file using requests via python-onedrive does not complete, to be more precise, only the first few bytes/kilobytes are uploaded.

Modified models.py for debugging as follows:

```
 def prepare(self, method=None, url=None, headers=None, files=None,
        data=None, params=None, auth=None, cookies=None, hooks=None):
    """Prepares the entire request with the given parameters."""

    ### ADDED FOR DEBUGGING
    with open(r'C:\Python\requests_debug.log', 'ab') as log:
      for k, file_tuple in files.viewitems():
        log.write('BEFORE: Key: {}, File tuple: {}, position: {}\n'.format(k, file_tuple, file_tuple[1].tell()))
    ### END

    self.prepare_method(method)
    self.prepare_url(url, params)
    self.prepare_headers(headers)
    self.prepare_cookies(cookies)
    self.prepare_body(data, files)
    self.prepare_auth(auth, url)
    # Note that prepare_auth must be last to enable authentication schemes
    # such as OAuth to work on a fully prepared request.

    # This MUST go after prepare_auth. Authenticators could add a hook
    self.prepare_hooks(hooks)

    ### ADDED FOR DEBUGGING
    with open(r'C:\Python\requests_debug.log', 'ab') as log:
      for k, file_tuple in files.viewitems():
        pos = file_tuple[1].tell()
        file_tuple[1].seek(0, os.SEEK_END)
        log.write('AFTER: Key: {}, position: {}, size: {}\n'.format(k, pos, file_tuple[1].tell()))
      log.write('Body size: {}\n'.format(len(self.body)))
    ### END
```

Here's an example log:

BEFORE: Key: file, File tuple: (u'nv2-pc.zip', <open file u'nv2-pc.zip', mode 'r' at 0x028B4EE8>, u'application/octet-stream'), position: 0
AFTER: Key: file, position: 4784128, size: 4787658
Body size: 522

As you can see, the body size is much smaller than the size indicated in AFTER.


Out of interest, once you've sent the request what's the value of `len(r.request.body)`?


I should note, it was python-onedrive's dev who determined that the cause of the issue was likely in requests and provided the debugging code. So, how would I go about getting that value?


It can be printed out [here](https://github.com/mk-fg/python-onedrive/blob/master/onedrive/api_v5.py#L161) (as `len(res.request.body)`). 


How do I access that value? Is it recorded in a file?

On Mon, May 12, 2014 at 12:56 PM, Cory Benfield notifications@github.comwrote:

> It can be printed out herehttps://github.com/mk-fg/python-onedrive/blob/master/onedrive/api_v5.py#L161(as
> len(res.request.body)).
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/2039#issuecomment-42879900
> .


Ah, sorry, yeah, the code will be on your machine. You can locate it by running `import onedrive; print onedrive.__file__`.


Confused. I uncommented that log.debug code, added your edit, then uploaded a file with onedrive-cli. Where's the log?


Heh, sorry, you didn't need to uncomment the log code. Just add my new line.


Uploaded the same file. I got 522.


Now that is interesting. Replace it with `print res.request.body`?


Getting this:
    print res.request.body
            ^
SyntaxError: invalid syntax


I apologise, you're clearly using Python 3: try `print(res.request.body)`.


I'm using Python 2.7.6. The new code works; here's what I got:

--b4bc63616bcd472992f40eada5ad72de
Content-Disposition: form-data; name="file"; filename="nv2-pc.zip"
Content-Type: application/octet-stream

PK♥♦¶  ┬c█BIò-åI 2!Ñ
↓╪âÄ8Ω¿úÄmtb←MN←£h‼╚L♠H┬Σ6      Æ¢▬cL◄ó¥A╘\áô╪l▬â╢Ñ¡÷σT{Äº╟₧cï╢∟èòΩLé╣ BBD2É@@╘↔
!\
        ♦µ{₧╡÷₧L☻j{╬{╛∩ √■ⁿÿ∞╜εk=k¡τ╢₧⌡∞▄ç^Σ╘∟╟iα↨♫s▄VÄ²╦α╛■▼»Γ╕Ö╖■e&╖y┌╖mUσ|p[Q
═π?4╒=╡Γ▒º▲~┬⌠²çù/_ß6=≥¿Θ)╧r╙π╦MYï
º╛rÆ>∩|σ₧╫⌂≤J≈╪?╜≥◄-≈↔Z╧í╛d╛√╩>xΩ_←º╧æ'♫╝rÄª╜·╩δ┐├≡«W♫B°P▀φ»╝·▼¼L↨
--b4bc63616bcd472992f40eada5ad72de--


This is very weird. The file appears to be very, very short from requests' perspective.

Can you try opening the file in Python in the normal way and see how long it appears to be?


How would I do that?


``` python
f = open(path_to_file, 'rb')
data = f.read()
f.close()

print len(data)
```


I got 4787658, which is the correct size of the file.


Well, that's utterly perplexing.

Let's try something new. What happens when you do this:

``` python
import requests

r = requests.post('http://httpbin.org/post', files={'file': (filename, open(path_to_file, 'rb'), 'application/octet-stream')})
print r.status_code
print r.json()['headers']
print len(r.request.body)
```


200
{u'X-Request-Id': u'ccda5e91-e5e5-49f2-b6c6-4616c2a3c5a9', u'Accept-Encoding': u
'gzip, deflate, compress', u'Content-Length': u'4787843', u'Host': u'httpbin.org
', u'Accept': u'_/_', u'User-Agent': u'python-requests/2.2.1 CPython/2.7.6 Windo
ws/7', u'Connection': u'close', u'Content-Type': u'multipart/form-data; boundary
=7af78a8f9a7e4bfba3562dae9882f3b1'}
4787843


Right, that's what I expected. When you use requests itself, it correctly uploads the entire file. That puts the responsibility for this behaviour _either_ on python-onedrive's code (which messes about with quite a few things) or on whatever is happening on the network connection to Microsoft.

I think you'll have to go back to python-onedrive's maintainer and ask them to dig further into what's going on there, because requests itself is fine.


Thanks very much for your time and efforts. So I guess this issue can be closed.


Depends. If you want to leave it open until your particular problem is resolved in case you need to contact us, I'm happy to do that. Alternatively, you can always email me if you need help: my email is on my GitHub profile page.


Feel free to CC me as well. :)


I'd like to continue discussion here, if you don't mind.

@cmasc, can you replace the `onedrive/api_v5.py` file (that you've edited earlier) with the one from here: https://gist.githubusercontent.com/mk-fg/c3de257baa034181b6dc/raw/b1b891f1946ecc2c0a6ee921fcc5764a29a1d44b/gistfile1.txt

Diff for these changes there is:

``` diff
diff --git a/onedrive/api_v5.py b/onedrive/api_v5.py
index 9bf8e73..b1b891f 100644
--- a/onedrive/api_v5.py
+++ b/onedrive/api_v5.py
@@ -120,11 +120,11 @@ class OneDriveHTTPClient(object):

         import requests  # import here to avoid dependency on the module

-        if not getattr(requests, '_onedrive_tls_fixed', False):
-            # temp fix for https://github.com/mk-fg/python-onedrive/issues/1
-            patched_session = self._requests_tls_workarounds(requests)
-            if patched_session is not None:
-                self._requests_session = patched_session
+        # if not getattr(requests, '_onedrive_tls_fixed', False):
+        #     # temp fix for https://github.com/mk-fg/python-onedrive/issues/1
+        #     patched_session = self._requests_tls_workarounds(requests)
+        #     if patched_session is not None:
+        #         self._requests_session = patched_session

         if session is None:
             try:
@@ -151,13 +151,17 @@ class OneDriveHTTPClient(object):
                 if len(file_tuple) == 2:
                     files[k] = tuple(file_tuple) + ('application/octet-stream',)
                 # rewind is necessary because request can be repeated due to auth failure
+                file_tuple[1].seek(0, os.SEEK_END)
+                print('file', file_tuple, file_tuple[1].tell())
                 file_tuple[1].seek(0)
             kwz['files'] = files
         if headers is not None:
             kwz['headers'] = headers
         code = res = None
         try:
+            print('request', url, kwz)
             res = func(url, **kwz)
+            print(len(res.body))
             # log.debug('Response headers: {}'.format(res.headers))
             code = res.status_code
             if code == requests.codes.no_content:
```

It should print exact arguments passed to requests, file size before it gets passed and same body len afterwards, plus disable TLSv1Adapter (which I don't see how might affect things much, but to avoid mess with default requests behavior altogether).

If this updated version will hang on upload (OneDrive servers you get redirected to still hang on TLS 1.2) - please try this api_v5.py version: https://gist.githubusercontent.com/mk-fg/9e20e898ea029d77bb3c/raw/e9288643371d1a66b9550e2f566ef3881b2a395b/gistfile1.txt (it leaves TLSv1Adapter in)

Also, can you run the following python script:

```
import requests
print 'requests', requests.__version__
```

It should print requests module version, I wonder if it's pre-1.0.0, where more serious monkey-patching was involved to avoid TLS 1.2.


I wonder if OneDrive cares that you're sending a zip file as `application/octet-stream` instead of `application/zip`.


file (u'nv2-pc.zip', <open file u'nv2-pc.zip', mode 'r' at 0x027D4EE8>) 4787658
request https://apis.live.net/v5.0/me/skydrive/files?access_token=EwBoAq1DBAAUGC
CXc8wU%2FzFu9QnLdZXy%2BYnElFkAAfHGYPxg6L%2BT1FaY%2B%2B3MV497j0sK3FIYEUNTgoDXVee6
HLinq%2Fsf38dZOTlde%2F4zec41yJ3WWwu78NUvQJEEB33yF4iDcYGxbIrjzLmmCGUQFWgBS4%2B6JT
3eGALkWHFt9Z43PA%2FVDBBpZLImPvQo0OVHB0aBmg3uelQjDsj1C0puhxoRc8zmHhhhq%2FgPgtPB7J
Sj%2BKiqxK2bAml62nJfmIb75eCtE5GmjaUdIEWU%2F2rax9kZcPlwCthzxVKt4vgB0TIovV2EHXg1Ti
J4pFRwKdWlRJNdY0HcdQYvii6hWsrfK64ddk4EEqD3hiDL%2F1BUIU1AXrEZle41NvzSLpAlOYADZgAA
CDrzqMfE%2BtPlOAHfRxpkEOZtksCKtZbeNzYRrIRCT4qKROd1DvFs3p8UcZ0Awe97Mir%2BLB9IvUN8
Ncqfz1cZ002My6PAjyrNEu3EqoblF26Z8DD4FkD9FZf2CUxiGTv05xSn3ntxjMMuZpv9NQxgWpThDcD7
MmXLRsk5G%2FMy%2F45kAWNx8em6NgLlpFat%2B3YcAGstRa1Z7WTJ2YpNf%2FscVVtiKUgtv%2FPH0%
2F5maIHxg%2BnMNbO7y6%2BOUOcFpYfD263T2uR%2BDrqBCq%2FAZs3rorQbuwMRTxMKElyCBbxTW24W
JwmnTb0759U8tpL4MmGDXHOWAXHbdQ%2FZKVu3a3YhlkUdpjejUw5rKkDjQRJBWdKjuZ1cf2vccRAeHg
LH3hA%2F%2FIRBqHfNv%2Fnq2NaxnzGFc3Bt8j0MhSdzw%2Fym5gzsCJWhR6Y1TXJGAQ%3D%3D {u'fi
les': {'file': (u'nv2-pc.zip', <open file u'nv2-pc.zip', mode 'r' at 0x027D4EE8>
, u'application/octet-stream')}, u'headers': {}}
Traceback (most recent call last):
  File "C:\Python\Scripts\onedrive-cli-script.py", line 9, in <module>
    load_entry_point('python-onedrive==14.05.0', 'console_scripts', 'onedrive-cl
i')()
  File "C:\Python\lib\site-packages\onedrive\cli_tool.py", line 307, in main
    resolve_path(optz.folder), overwrite=not optz.no_overwrite)
  File "C:\Python\lib\site-packages\onedrive\api_v5.py", line 394, in put
    method='post', files=dict(file=(name, src)))
  File "C:\Python\lib\site-packages\onedrive\api_v5.py", line 332, in **call**
    return self.request(api_url(), **kwz)
  File "C:\Python\lib\site-packages\onedrive\api_v5.py", line 164, in request
    print(len(res.body))
AttributeError: 'Response' object has no attribute 'body'

requests 2.2.1


@cmasc 
Looks like github is snipping out somewhat important `<open file ...>` bits from the output you're pasting, would be more useful if you'd wrap these things in ``` (three backticks, put before and after text that should be preserved as-is) next time.

And my bad with that `res.body` line - should rather be `print(len(res.request.body))`, same as suggested by @Lukasa above. But that line was tested above already, no big deal here.

I went back to output you've (@cmasc) sent me in the mails earlier, and it looks like snipped bits look like `<open file u'nv2-pc.zip', mode 'r' at 0x027E1EE8>`, which is notably differs from open/read/len code tried above in mode='r' instead of 'rb' (which is consistently suggested above).

Afaik it doesn't matter on unixes, but might mean something on windows (and I've never used python there myself), so can you please try running the following snippet (with `path_to_file` replaced by path to that same file):

``` python
import requests
r = requests.post('http://httpbin.org/post', files={'file': (filename, open(path_to_file, 'r'), 'application/octet-stream')})
print r.status_code
print r.json()['headers']
print len(r.request.body)
```


200
{u'Content-Length': u'521', u'Accept-Encoding': u'gzip, deflate, compress', u'X-
Request-Id': u'6edd4720-5ecb-4a57-9df9-f0644e6c6144', u'Connection': u'close', u
'Accept': u'_/_', u'User-Agent': u'python-requests/2.2.1 CPython/2.7.6 Windows/7
', u'Host': u'httpbin.org', u'Content-Type': u'multipart/form-data; boundary=15f
a5b09837443d6b9eef589850685d3'}
521


Problem found. =)


This isn't a requests bug though. All we can do is take the file object we've been given and read from it. I think python-onedrive needs to take the lead here and ensure it uploads binary files correctly. =)


Yeah, definitely wouldn't want requests to mess with passed objects like that ;)

Might there be a valid use-case for passing default file objects with 'r' (on windows or otherwise), btw?
If not, maybe would do users a world of good to check and warn about such behavior?

And indeed, bug is in the cli tool that does the `open()`.