python requests runs infinitely if the called script executes more than 2 minutes

I am setting up a vps. Everything has worked well with python requests module until I had to run a script on the server that exceeded two minutes. As soon as any script executes through python requests on the remote server for more than 2 minutes, the request goes on infinitely. It never returns or terminates. If I call the script from any browser, it executes normally and returns the appropriate response IRRESPECTIVE OF THE DURATION. The only means of terminating any scripting lasting more than two minutes and executed through python requests is by including the timeout parameter in the requests call, which normally gives the following traceback:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 376, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 378, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Python34\lib\http\client.py", line 1148, in getresponse
    response.begin()
  File "C:\Python34\lib\http\client.py", line 352, in begin
    version, status, reason = self._read_status()
  File "C:\Python34\lib\http\client.py", line 314, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Python34\lib\socket.py", line 371, in readinto
    return self._sock.recv_into(b)
  File "C:\Python34\lib\ssl.py", line 708, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Python34\lib\ssl.py", line 580, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\adapters.py", line 376, in send
    timeout=timeout
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\util\retry.py", line 247, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\packages\six.py", line 310, in reraise
    raise value
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 380, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='domain1.com', port=443): Read timed out. (read timeout=130)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "all-test.py", line 32, in <module>
    cool = requests.post("https://domain1.com/test3.php", stream=True, verify=False, timeout=130)
  File "C:\Python34\lib\site-packages\requests\api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "C:\Python34\lib\site-packages\requests\api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python34\lib\site-packages\requests\sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python34\lib\site-packages\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python34\lib\site-packages\requests\adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='domain1.com', port=443): Read timed out. (read timeout=130)

Note, everything works perfectly with no issue if called from browser.
Also, checking the access.log file on the remove server shows that the script completed at the right time with no issue and responded with 200 status code even when called by python requests that ran infinitely.

To eliminate some probable causes, i have set up and run similar script on my local wamp, calling the same script via python's requests with no issue.

Please, I will highly appreciate any help anybody can give me to resolve the above issue. It has taken me 4 days with no success. 


This sounds very much like it relates to the web server you're running. Can you try with a different web server?

I think you might be right because I have a shared server somewhere where I normally upload files lasting more than 15 minutes using python requests. It has always worked well.
On the other hand, I want to believe it is not web server, but python requests, because the code completes well irrespective of duration when called from a browser. Moreover, checking the access.log file on the web server shows that the script completed at the right time with no issue and responded with 200 status code even when called by python requests that ran infinitely.
I would easily have tried this on another server but like I said earlier, it's my first vps experience and have not got another somewhere
Please, Lukasa, can u think of any setting on the web server (apache2, php5.6 and Ubuntu 14.04) that might cause this?

I think I should add that the script in question is designed to echo output only at the end. So, could it be that if requests has to wait for more than two minutes without a response, something goes wrong?

Is your script sending its response using HTTP or HTTPS? If HTTP, can you use wireshark or tcpdump to intercept the request/response data?

it sends through HTTPS

Hrm. Can you set `stream=True` and print `response.headers` immediately after, before doing anything with the body?

Ok, I ill try that and get back soon. But note, it does not return except with timeout.

@jj-one My suspicion is that this is being caused by a parsing error on the response such that we are waiting for connection close. `stream=True` will cause it to return once the header block is complete, which should probably work.

Unfortunately, it crashed out because the supposed response object is NoneType. Remember, it was stopped by the timeout parameter and trying to response.headers in the except requests.exceptions.Timeout block cause another exception "AttributeError: 'NoneType' object has no attribuete 'headers'.

The most confusing issue here is the fact that the requests never returns by itself once two minutes is exceeded except by timeout parameter which make the response object to still remain as NoneType. If it could stop by itself, we could investigate the response object.

Does it not return by itself even if you set `stream=True`, without a timeout?

No. It only stop when the timeout value is reached.

Ok, that suggests something really bad is going on, because it means the data we're receiving must be *extremely* ill-formed: it must really not be HTTP at all.

Ok, testing with stream=True, without a timeout.

Just terminated it through Windows Task Manager after 5 minutes (i.e while running with stream=True, without timeout).

Ok, so this strongly suggests that the data coming from the server is pretty much not valid HTTP. 

I strongly suspect that requests abandons response after waiting for two minutes and goes into whatever mode which never ends. 

The server access.log still shows that all the call calls so far completed successfully at the right time.

Remember that if I reduce the script's iterations such that it ends in less than two minutes, it completes and requests receives appropriate response.

This is really driving me crazy. PLEASE, PLEASE, PLEASE, SOMEBODY HELP!!!

Requests does not abandon anything, it's simply not how the codebase works. Please try to remain calm: we are busy people and are all only volunteers.  

At this point I think we need to see the plaintext of the server response 

I really appreciate your effort both in dev and volunteering to help users troubleshoot Requests. Don't misunderstand my cry for help. This has stalled my project for four days and I don't know how soon I can be able to resolve it. Requests is awesome. I have used it in several other project: sending data to  and receiving data from remote servers, tt hasn't given me any issue before. The current project that has developed the above issue is almost at completion level. All other scripts in the same project have run and still run successfully and return the expected results (Array of five) because each one of them complete within two minutes. It is only this one that has taken up to two minutes to execute and has caused me this much delay.

So it would be helpful to see if you can try to remove code to get a minimal reproducer. Try pulling chunks of code out one at a time and see if the problem keeps happening. If we can get a small code example that'll help us narrow the problem down, and if at any point you remove some code and the problem goes away you have a bit of an idea of where the problem might be. 

Here is the response header when successful:
{'Content-Length': '52', 'Date': 'Sun, 21 May 2017 20:27:28 GMT', 'Server': 'Apache/2.4.7 (Ubuntu)', 'Keep-Alive': 'timeout=5', 'Connection': 'Keep-Alive', 'Content-Type': 'text/html; charset=UTF-8'}

And the following the response.content when successful:
b'["no", "dummy39", "dummy40", "dummy41", "BEST CASE"]'



Also, here is the content of test3.php, which is a file I created only to test the above issue:
`<?php
$start = $end = time();
while(($end - $start) < 50){
	$nt = "";
	$end = time();
}
echo '["no", "dummy39", "dummy40", "dummy41", "BEST CASE"]';
exit;`

Of course, the above code is successful, but if I increase 50 to anything 120 or more, Requests never returns unless by timeout parameter or I forcefully end task via Windows Task Manager.

Can you thing of any setting on Apache2, PHP5.6 or Ubuntu 14.04 that will interfere with Requests  but not browser's execution time?

No, but can you show the raw data the browser saw?

Sorry, it was late night at my place. Meanwhile, the following code would go infinitely in Requests:

`<?php $start = $end = time(); while(($end - $start) < 140){ $nt = ""; $end = time(); } echo '["no", "dummy39", "dummy40", "dummy41", "BEST CASE"]'; exit;`

But with browser, here is the output:

["no", "dummy39", "dummy40", "dummy41", "BEST CASE"]

I just attempted this locally with ubuntu 14.04, apache2 and PHP5. Using your exact script, and Requests 2.14.2, I didn't reproduce the problem: the request returns fine.

Can you provide us with more details about what your Python environment looks like? What Python version, what version of Requests, etc?

Python 3.4, Requests 2.9.1, Window8 (64Bit) and calling the request script via command prompt.
On the remote server, Ubuntu 14.04, PHP5.6, Apache2, HTTPS requests and response.

Also, I tried this locally on wamp server. It failed first. But when I increased Timeout in php.ini in Apache directory of wamp (that of php directory has always been at 600), it worked perfectly; both via Requests and browser. 

The same thing happened with the browser. The first time above two minutes, the browser was timed out, but setting timeout in /etc/apache2/apache2.conf to 600 solved the problem.

If it will be of use, I could upload the php.ini and apache2.conf on the remote server.