Broken pipe Exception

I post a large file to server, and it raise `ConnectionError` exception.

```
Traceback (most recent call last):
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/worker.py", line 479, in perform_job
    rv = job.perform()
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/job.py", line 466, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "./jobs/office.py", line 97, in notice_file
    open('test.pdf', 'rb')))
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/enterprise/client/api/media.py", line 15, in upload
    'media': media_file
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/api/base.py", line 20, in _post
    return self._client._post(url, **kwargs)
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 74, in _post
    **kwargs
  File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 39, in _request
    **kwargs
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
```

The `test.pdf` is 4.2M, when I post it as a file, I get ConnectionError Exception.
But I use curl to post it to server, it can be uploaded successful.


Can you reproduce it every time? What curl command are you using? Does the
server have any useful log messages?

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Fri, Jan 23, 2015 at 6:03 AM, Cloverstd notifications@github.com wrote:

> I post a large file to server, and it raise ConnectionError exception.
> 
> Traceback (most recent call last):
>   File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/worker.py", line 479, in perform_job
>     rv = job.perform()
>   File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/rq/job.py", line 466, in perform
>     self._result = self.func(_self.args, *_self.kwargs)
>   File "./jobs/office.py", line 97, in notice_file
>     open('test.pdf', 'rb')))
>   File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/enterprise/client/api/media.py", line 15, in upload
>     'media': media_file
>   File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/api/base.py", line 20, in _post
>     return self._client._post(url, *_kwargs)
>   File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 74, in _post
>     *_kwargs
>   File "/Users/cloverstd/Dropbox/WorkSpace/gench/enterprise/wechatpy/client/base.py", line 39, in _request
>     *_kwargs
>   File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/api.py", line 49, in request
>     response = session.request(method=method, url=url, *_kwargs)
>   File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
>     resp = self.send(prep, *_send_kwargs)
>   File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
>     r = adapter.send(request, *_kwargs)
>   File "/Users/cloverstd/.virtualenvs/enterprise/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
>     raise ConnectionError(err, request=request)
> ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
> 
> The test.pdf is 4.2M, when I post it as a file, I get ConnectionError
> Exception.
> But I use curl to post it to server, it can be uploaded successful.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2422.


Have you tried without using `rq`? Are you using curl within `rq`?


Same here. I'm using requests to post large files (about 8 Mb in total) to a Django endpoint. Getting same error. It does work fine with smaller files.


@barseghyanartur perhaps you can give us more details than @cloverstd did, for example, what code will reproduce this?

I have a suspicion as to the cause but I haven't been able to reproduce this yet.


@sigmavirus24 

I have gone a bit further in research and this is what I have found.

Basically, it's this:

```
requests.post(some_url, data=data_dict, files=files_dict)
```

Environment details:
- Python==2.7.3
- requests==2.5.1

I managed to reproduce the problem exactly. In my case, the endpoint is a multi-lingual Django site. Requests to URL without the language prefix (like "http://example.com/foo/endpoint/") do get redirected to the same URL with the language prefix (like "http://example.com/en/foo/endpoint/").

If `some_url` is "http://example.com/foo/endpoint/" I get an error:

('Connection aborted.', error(32, 'Broken pipe'))

If `some_url` is "http://example.com/en/foo/endpoint/" it all goes fine. Likely, `requests` doesn't handle large post requests to endpoints which do redirect them to some other endpoints.

I have just tried it with requests==2.6.0 and I do get the same problem.


@barseghyanartur can you give us the status code from the redirect?


I wonder if the remote end isn't expecting a body but we're sending it anyway, so we get our connection torn down.


@Lukasa or if we change the method (i.e., in a case like 7231 Section 6.6.4) without stripping the body/content-length. (or strip the body and not the content-length).


Really any information @barseghyanartur can give us about the redirect response and the subsequent request will be helpful


@sigmavirus24 

Hey, thanks for getting back on this. I'll post it here quite soon after I get home. :) That's gonna be in about 5 hours.


@barseghyanartur no rush. I'd like to fix this once and for all. That can be today or next week. As long as it's fixed, I'll be happy.


@sigmavirus24 

I don't get any response back, since it fails hard.

``` py
Traceback (most recent call last):
  File "requests_test.py", line 29, in <module>
    response = requests.post(url, data=data, files=files)
  File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/api.py", line 108, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/home/me/.virtualenvs/env/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
```

And this is the code:

``` py
import os

import requests

url = 'http://localhost:8001/foo/endpoint/'

data = dict(['input_{0}'.format(index), 'some value {0}'.format(index)] for index in range(10))

FILES = (
    '/home/me/bbrepos/requests_test/media/1.gif',
    '/home/me/bbrepos/requests_test/media/2.gif',
)

files = dict(['file_{0}'.format(index), (os.path.basename(file_path), open(file_path, 'rb'))] for index, file_path in enumerate(FILES))

response = requests.post(url, data=data, files=files)
```


@barseghyanartur you can determine what the redirect codes, etc. are with `requests.post(url, data=data, files=files, allow_redirects=False)`


@sigmavirus24 

Ah, sorry, I didn't realise you were asking for that. It was 301, as far as I remember.


Given that this is all plaintext, you should be able to get a packet capture of the whole transaction. Running `tcpdump -nnvvXSs 1514 port 8001 and host localhost` should do the trick.


I'm having the same problem:  small (1MB) data works, but large (50MB) data breaks ('Connection aborted.', error(32, 'Broken pipe')).   

The complications in my case are that (1) I don't control the Web service -- it's the property of another company; (2) the protocol is HTTPS, but perhaps the packet capture would still be informative, even if the payload data is encrypted from tcpdump's point of view.

I will get a packet capture of a broken large transaction and reply back.


I created the tcpdump file https://gist.github.com/JazzFan/fc55e28c525518ada71f with this command:
    `sudo tcpdump -nnvvXSs 1514 port 443 and host 54.200.58.94`

My program ran under Python 2.7.8, with `requests` 2.5.3 (but I feel pretty confident that I can get it to break under other versions of Python or `requests`, if you would prefer others).

The relevant lines of the Python program:

```
    import requests
    ...
    headers = {'Content-Type':  'application/json'}
    response = requests.post(URL, data=wrapped_chunk, headers=headers)
```

The idea of the program is to split a large file into 50MB chunks, wrap each chunk with a bit of JSON, then POST it to the Web service API that a vendor company provides to my company.

The stack trace (home dir redacted with `~`)

```
Traceback (most recent call last):
  File "~/bin/progname.py", line 399, in <module>
    main()
  File "~/bin/single_process.py", line 33, in _
    return f(*a, **kw)
  File "~/bin/progname.py", line 392, in main
    max_return_code = send_file_chunks(fn, api_key, args.parallelism)
  File "~/bin/progname.py", line 347, in send_file_chunks
    chunk_filenames))
  File "~/anaconda/envs/py278/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "~/anaconda/envs/py278/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
```

I realize the stack trace has everything but `requests` in it; let me know if that's a problem and I'll write a simpler program in order to get a clean stack trace.


@jazzfan thanks for doing that. I'll look tonight and I suspect that @Lukasa will look when he wakes up


@sigmavirus24 You forget that I am in NYC at the minute, and so am (relatively) awake!

As I've said elsewhere, EPIPE almost always comes when we attempt to send on a connection that has been remote-closed.

I think it would be most interesting if you could run a patch in requests' vendored copy of urllib3 that would throw the socket object up with the `ConnectionError`, and then read from it. The exact patch to apply here is a bit unclear and overlaps with some other stuff we've been thinking about when it comes to EPIPE.

Alternatively, a non-SSL repro would work!


@Lukasa, does this mean that the SSL tcpdump file is essentially useless?  Also, I'm afraid I'm too much of a newbie to be able to figure this patch you're describing, especially if it's unclear to you.  

I guess I'll rig up some kind of little test HTTP server on my localhost and see if I can reproduce the error using that.


@JazzFan Sadly, it's not very easy to tell from it whether or not a response was sent. That said, we can definitely see that the remote end tore the connection down at 15:17:24.141410 (sending a TCP FIN). Shortly thereafter we emitted another packet, which caused the remote end to send a barrage of TCP RSTs (though most of these appear to be retransmits, and in practice appear to be in response to the other TCP packets that were in flight when the FIN was emitted (as shown by the sequence/acknowledgement numbers).

What we really want to understand is what triggered the connection teardown.


@Lukasa 

Well, one thing is clear - it happens only when large amounts of data (files) and transferred.


@barseghyanartur Yup, but that doesn't necessarily mean much. You can only get an EPIPE if the remote end forcibly closes the connection, which is only likely to happen for large data files.


@Lukasa 

Are you interested to know what happens if 20 Mb plain text (textarea) is posted?


I'm really interested in seeing something happening here in plaintext. =D


Well, for whatever reason I just can't get tcpdump to actually capture the packets.  I get output like this:

```
0 packets captured
303 packets received by filter
0 packets dropped by kernel
```

But this example breaks for me, again with Python 2.7.8 and requests 2.5.3:

```
#!/usr/bin/env python

import requests

line = '.' * 79 + '\n'
data = line * 10000000
response = requests.post('http://127.0.0.1:8000', data=data)
print(response)
```

Here is the stack trace (again with home dir redacted as `~`):

```
$ ./requests_test.py 
Traceback (most recent call last):
  File "./requests_test.py", line 7, in <module>
    response = requests.post('http://127.0.0.1:8000', data=data)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/api.py", line 99, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "~/anaconda/envs/py278/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
```

I can capture the packets with Wireshark.  I know that you're looking for a particular kind of plaintext output from tcpdump, but would you happen to know how to get the equivalent plaintext output from Wireshark?  
On the off chance that it would work for you, I've created a gist, wireshark.txt, https://gist.github.com/JazzFan/8517983ff5eed2d95b10.


@JazzFan In addition to what you sent me I need the actual packet data. There's a 40-byte TCP payload coming from your server in frame 60 that Wireshark is claiming is HTTP data, but it hasn't broken out the data itself so I can't see it.

What tcpdump command are you running?


I'm running

```
sudo tcpdump -nnvvXSs 1514 port 8000 and host 127.0.0.1 > tcpdump.txt
```

and variations on that (e.g., omitting the `and host 127.0.0.1`).  After triggering the bug in another window and hitting `Ctrl-C` on the `tcpdump` command, I get something like

```
0 packets captured
492 packets received by filter
0 packets dropped by kernel
```

and an empty tcpdump.txt file.  I interpret the output to mean that the filter saw the packets but didn't capture them, but I don't know why it didn't capture them.


Also, FWIW, the 40 bytes of TCP data payload on packet 60 are

```
<html><body><h1>POST!</h1></body></html>
```

This is just the standard response that my dummy HTTP server makes to any post request.   FWIW, here is my HTTP server:

``` python
#!/usr/bin/env python
"""

Very simple HTTP server in python.

Usage::
    ./dummy-web-server.py [<port>]

Send a GET request::
    curl http://localhost

Send a HEAD request::
    curl -I http://localhost

Send a POST request::
    curl -d "foo=bar&bin=baz" http://localhost

"""
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import SocketServer


class S(BaseHTTPRequestHandler):
    def _set_headers(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_GET(self):
        self._set_headers()
        self.wfile.write("<html><body><h1>hi!</h1></body></html>")

    def do_HEAD(self):
        self._set_headers()

    def do_POST(self):
        # Doesn't do anything with posted data
        self._set_headers()
        self.wfile.write("<html><body><h1>POST!</h1></body></html>")


def run(server_class=HTTPServer, handler_class=S, port=8000):
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    print 'Starting httpd...'
    httpd.serve_forever()


if __name__ == "__main__":
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()
```


Oops, I forgot to include @Lukasa in my previous comments.