OpenSSL.SSL.SysCallError: (104, 'Connection reset by peer')

I'm using the python package `requests` to send requests to https://mobile.twitter.com/username/following.

At first, I encounter the exception : requests.exceptions.SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol. To solve that, I follow this solution to write a SSLAdapter to specify PROTOCOL_TLSv1.

After that, I encounter another exception : requests.exceptions.SSLError: [Errno bad handshake] (-1, 'Unexpected EOF’). And, I found this, but i send request and receive data in the same process.

And then, I use `requests` to send requests to https://api.twitter.com/1.1/friends/ids.json. The second exception is gone(still didn't understand why). But I encounter third exception : OpenSSL.SSL.SysCallError: (104, 'Connection reset by peer'). I found [this](http://stackoverflow.com/questions/383738/104-connection-reset-by-peer-socket-error-or-when-does-closing-a-socket-resu) in SO. And I add `time.sleep(10)` before send requests. But the third exception still happen.

So the second and the third exception still happen. Maybe the response content is too big too read? Or it's the problem of twitter server(some SO [solutions](http://stackoverflow.com/questions/23397460/error-handling-boto-error-104-connection-reset-by-peer) said about it ).


All of those exceptions indicate that Twitter is closing the connection on you. You should check whether your data is valid.


the data means the request header?


Could be the request headers, could be the request body. 


the requests send to twitter need to be authorized. Maybe the bearer token is invalid. Thank for your reply.


the second exception is the problem of mine. because the program run out of the memory so the program wait for read data from the connection, but there is no space to store, so it seems dead lock. and the server wait till the time out, and the exception throws. but i still can not figure out why leads to the third exception.


I just had exactly the same error - but not with Twitter. In my case it's communication with the web interface of a local modem, which I interrogate for statistics. The request is exactly the same each time, and the program has been running for over 24 hours before the problem, so I suspect it is not an authorization problem. This is the complete backtrace:

```
Traceback (most recent call last):
 File "/usr/local/bin/mon_quality.py", line 255, in <module> main()
 File "/usr/local/bin/mon_quality.py", line 251, in main
    m.status_log(logfname, 15)
 File "/usr/local/bin/mon_quality.py", line 123, in status_log
    response = self.status_csv()
 File "/usr/local/bin/mon_quality.py", line 137, in status_csv
     stat = self.status()
 File "/usr/local/bin/mon_quality.py", line 190, in status
    data = payload, headers = self.hdrs)
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 494, in post
     return self.request('POST', url, data=data, **kwargs)
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 452, in request
    resp = self.send(prep, **send_kwargs)
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 581, in send
      history = [resp for resp in gen] if allow_redirects else []
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 175, in resolve_redirects
      allow_redirects=False,
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 555, in send
      r = adapter.send(request, **kwargs)
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/adapters.py", line 391, in send
      r.content
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/models.py", line 690, in content
      self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/models.py", line 628, in generate
      for chunk in self.raw.stream(chunk_size, decode_content=True):
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/packages/urllib3/response.py", line 240, in stream
      data = self.read(amt=amt, decode_content=decode_content)
 File "/usr/lib64/python2.7/site-packages/requests-2.2.1-py2.7.egg/requests/packages/urllib3/response.py", line 187, in read
      data = self._fp.read(amt)
 File "/usr/lib64/python2.7/httplib.py", line 543, in read
      return self._read_chunked(amt)
 File "/usr/lib64/python2.7/httplib.py", line 612, in _read_chunked
      value.append(self._safe_read(chunk_left))
 File "/usr/lib64/python2.7/httplib.py", line 658, in _safe_read
      chunk = self.fp.read(min(amt, MAXAMOUNT))
 File "/usr/lib64/python2.7/socket.py", line 380, in read
      data = self._sock.recv(left)
 socket.error: [Errno 104] Connection reset by peer
```


Again, to be clear, `Connection reset by peer` means that the remote end closed the connection while we were expecting to read data from it. In your case @JohnCC330, we're trying to read a chunked body and the chunk is not complete.

You should be trying to investigate why the modem closed the connection in a non-graceful manner.


> You should be trying to investigate why the modem closed the connection in a non-graceful manner.

These things can happen. Power can fall out, connections get broken. My question: is there a way to recuperate from such a situation in a graceful way? I have the impression that this is not solveable with a try...except - am I correct here? 


It depends, because what 'recuperate' means depends on your use-case.

If this is a streaming API, then you should just reopen the connection and keep going. Try except is fine here.

If it's a whole request, you can retry the request, but don't retry forever because it may be that the request is malformed.

I'd need to know your specific use case before I can be more helpful.


I am interrogating the modem stats each 15 seconds. Each time I log in, and download just one page. So, the requests are totally independent. And they are also identical, so there is no chance that they are malformed (except for hardware problems). The problem I'm investigating is that the modem switches speed because of poor line quality, so the connection may not be up all the time. 

The problem has presented itself - until  now - always when I send the login information with a POST.

I've added a try...except around the POST and the following GET, hoping to catch the problem, but I have the impression that this error is not generating exceptions (from the error messages).


This error should absolutely be generating exceptions. What makes you think it isn't?


I was looking for an error message which said "unhandled exception"... which didn't appear, but I'm not quite sure where I got that idea... A try... except does seem to capture the problem, but I can't seem to re-enable the communication without actually exiting and re-entering the program. I do restart a new session, and try a new connection, but the program hangs. 

I've been quite occupied with more pressing issues, and the problem doesn't appear frequently (only after 3-5 hours of working normally). I tried cause the error by unplugging the phone line, but there the software recovers as expected.


If the software is hanging, it's possible we're re-using a broken socket. However, it's hard to know when you're using a version of requests that's so many versions behind the current release. Is it possible for you to test with a more recent version?


Wow... Installed requests.master from Github, and now the POST doesn't return any content. Only change was updating requests. I'll investigate the changelog.


With the new (2.6.1) version of requests, the communication ends with the POST (each time). The modem replies to the POST with a 303, which causes a ConnectionError exception, even though I have allow_redirects = True in the post request.

```
            r = s.post("http://10.0.0.2/Forms/login_security_1",    # s is the session
                        data = payload, headers = self.hdrs,
                        allow_redirects = True)
```

In wireshark:

```
HTTP/1.1 303 See Other
Location: http://10.0.0.2/rpSys.html
Content-Length: 0
Server: RomPager/4.07 UPnP/1.0
Set-Cookie: C0=21232f297a57a5a743894a0e4a801fc3; path=/
Set-Cookie: C1=21232f297a57a5a743894a0e4a801fc3; path=/
EXT:
```

and I get the ConnectionError at this point. Strangely, the status_code of the POST is 200.


Do you see a TCP FIN in the wireshark output following the response?


Oh hang on, that appears to be an ill-formed header block. You can't have a name with no value.


Yep.... I noticed that, but had to leave for work. Back now, so I checked older captures I saved on disk, and the EXT name is always empty. Requests 2.2.1 seemed to be Ok with that though. I've checked through several specs and cannot find a requirement for this field to have anything. 

RFC 2616 http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 seems to specify:

```
   message-header = field-name ":" [ field-value ]
   field-name     = token
   field-value    = *( field-content | LWS )
   field-content  = <the OCTETs making up the field-value
                    and consisting of either *TEXT or combinations
                    of token, separators, and quoted-string>
```

So, it seems that the LWS (linear white space) and CR/LF after the EXT: should be sufficient. Could it be that requests is generating the exception on that field? I suspect that is not the problem - the error reported is really ConnectionError.

BTW, no FIN. I send the POST, get an ACK and the 303 in return. In case of requests 2.2.1, the communication redirected to the Location from the 303. In case of 2.6.1 the communication stopped. I'll test this tomorrow. It's 02:00 now...


Sorry, 7230 disagrees:

```
     header-field   = field-name ":" OWS field-value OWS

     field-name     = token
     field-value    = *( field-content / obs-fold )
     field-content  = field-vchar [ 1*( SP / HTAB ) field-vchar ]
     field-vchar    = VCHAR / obs-text

     obs-fold       = CRLF 1*( SP / HTAB )
                    ; obsolete line folding
                    ; see Section 3.2.4
```

Can I see the message in the ConnectionError?


Note that the '*' in

```
field-value    = *( field-content / obs-fold )
```

indicates, by default, 0 to infinite times. So empty is allowed (RFC5234 in http://tools.ietf.org/html/rfc5234, section 3.4)

There seems to be quite a bit of discussion around this... But the tendency is to allow empty values.
In IBM's http://www-01.ibm.com/support/docview.wss?uid=swg1PK96911 is a patch to allow empty values:

> An empty Supported header is present in the message. This is a
> valid message, specifying the client supports no
> protocol extensions.

Also, Apache has this patch (https://issues.apache.org/jira/browse/TS-3495):

> If I understand correctly, empty header value is allowed except pseudo-headers.

The message of the exception:

> Error:  `<class 'requests.exceptions.ConnectionError'>`
> Message:  ('Connection aborted.', ResponseNotReady())
> Status: 200


Apologies, you are quite right.

Are you sure the ConnectionError comes out of the original request? I suspect it comes out of the 303, and it would be caused by httplib not thinking the response was finished.


As I said earlier, the problem _is_ the 303:

> I send the POST, get an ACK and the 303 in return. In case of requests 2.2.1, the communication redirected to the Location from the 303. In case of 2.6.1 the communication stopped. I'll test this tomorrow. It's 02:00 now...

but the httplib is the same in both cases (requests 2.2.1 and 2.6.1) I did not change the python version. So why do I have problems with requests 2.6.1? 

"ResponseNotReady()" comes from httplib though. Is there a difference in the calling of of httplib by r 2.2.1 and 2.6.1?


This is the validation of the header line in rfc822.py:

```
def isheader(self, line):
    i = line.find(':')
    if i > 0:
        return line[:i].lower()
    return None
```

It doesn't seem to check for empty values. 

Could this be an unchecked special situation, where an empty header is the _last_ header in the header section?


Let me be clear.

I'm not pointing the finger of blame at httplib. My theory is that this is another manifestation of #2568. What's confusing me is why that would happen if chunked transfer encoding is not specified.


I have to correct something I said earlier: There _is_ a FIN just after the POST. When I said there wasn't, that was because I was relying (erronously) on a saved wireshark capture. I was re-testing all this, and I now see that a FIN is sent 1ms after the POST goes out. This is a similar situation as #2568. Mikkelsen does two GETs, I do a POST immediately followed by a GET. I suspect he might have a FIN in between the GETs too.

$64,000 question: why the FIN?

> My theory is that this is another manifestation of #2568

It would a hell of a coincidence if it wasn't related... Same exception... Same version...


@JohnCC330 How urgent is a fix for this? Are you ok to wait until we have a proposed fix for #2568?


Well, I switched back to 2.2.1 for now. So, I'm back to the original problem I reported, which was _not_ covered by that issue. There's no life-or-death situation here...


I am noticing OpenSSL.SSL.SysCallError errors when Twitter API server has closed the connection. I was surprised to get it and was expecting a requests.exceptions.ConnectionError instead. It would be nice if requests could hide some of these details. But my understanding of how all this is working is admittedly limited.


@edsu what version of Python is installed and what version of requests is in use?


Sorry, I'm actually the proxy for someone else (@remagio) who reported the exception getting thrown over at https://github.com/edsu/twarc/issues/72 From the stack trace it looks like he is using python 2.7, but it wasn't clear what the version of requests was at play. I've asked him to check. Perhaps it's an older version that is behaving this way?