Running a single connection with pool_block == True, hangs the application

I have a problem where I'm trying to create a python script to test a single-threaded web server on an embedded device.  I was having performance problems until I realized that I could only support one connection per session.  When I made my own HTTP adapter to only use one connection with pool_block equal to True, one request got sent, then the  script hung trying to send the next request. 

If I comment out line 533 (the "if release_conn:") in request/package/urllib3/connectionpool.py, my problems go away. The script performances super duper fast as expected compared against a cURL bash script.

However, I'm not sure this is the proper fix for the problem, since I'm a special case.  I need a way to configure this to run this single connection from the top-level api without getting blocked.

BTW, if I turn blocking off (pool_block == False), I get the same horrible performance.  Basically, it has to do retries and slows down, because the web server can only handle  one connection at a time.


Are you reading the full response from the request?


As far as I know, I thought I was.  I just get or post, and parse the
response in a separate variable.  Is there something else that needs to be
done?

On Wed, Mar 19, 2014 at 1:52 PM, Cory Benfield notifications@github.comwrote:

> Are you reading the full response from the request?
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1967#issuecomment-38091631
> .


That's ok, I just wanted to confirm that you were.

As a quick fix, calling `Response.close()` should fix this problem. We need to work out why the connection isn't being released back to the pool.


I'll give that a try.

On Wed, Mar 19, 2014 at 2:53 PM, Cory Benfield notifications@github.comwrote:

> That's ok, I just wanted to confirm that you were.
> 
> As a quick fix, calling Response.close() should fix this problem. We need
> to work out why the connection isn't being released back to the pool.
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1967#issuecomment-38098836
> .


I put the "close()" call in the methods where I get a response from the requests calls.

Still hangs.


Hmm. Can I see some sample code?


```
    def post_entry(self, url, ylist, entry):                                                                                                                                 
         """                                                                                                                                                                  
         Create an instance of a yang list                                                                                                                                    

         :param: url - the base rest url                                                                                                                                      
         :param: ylist - the yang list this is operating on                                                                                                                   
         :param: entry - the yang instance you're creating(dictionary)                                                                                                        

         :return: True for success, false for failure                                                                                                                         
         """                                                                                                                                                                  
         if not url.endswith("/"):                                                                                                                                            
             url += "/"                                                                                                                                                       
         url_string = url + ylist                                                                                                                                             
         headers = {"Content-type":"application/json"}                                                                                                                        
         try:                                                                                                                                                                 
             r = call_requests(self.session.post, url=url_string, data=json.dumps(entry), timeout=5, headers=headers)                                                         
         except Exception as ex:                                                                                                                                              
             logger.error("Failure Post Exception: {0}".format(str(ex)))                                                                                                      
             return False                                                                                                                                                     

         r.close()                                                                                                                                                            
         if r.status_code != requests.codes.created:                                                                                                                          
             logger.error("Failure - Status: {0}".format(r.status_code))                                                                                                      
             return False                                                                                                                                                     

         return True   


 @RetryIt                                                                                                                                                                     
 def call_requests(func, **kwargs):                                                                                                                                           
     """                                                                                                                                                                      
     Intermediate call to the requests methods with retry decorator                                                                                                           
     """                                                                                                                                                                      
     return func(**kwargs)                                                                                                                                                         
```


ugh, don't know why this posted twice. I blame the laptop:-)


Hmm. I wonder if the retry decorator is doing something weird here.


it only checks for the "retries" in the exception and sends again for maximum amount.  passes everything back to the caller.


Right, but 'sends again' is the bit that I find interesting. Can you remove it?


Here's the meat of the decorator

```
     def __call__(self, *args, **kwargs):                                                                                                                                     
          """                                                                                                                                                                  
         Handle the function call and retry it                                                                                                                                
         """                                                                                                                                                                  
         try:                                                                                                                                                                 
             results = self.f(*args, **kwargs)                                                                                                                                
         except Exception as ex:                                                                                                                                              
             if "retries" in str(ex):                                                                                                                                         
                 self.retry_cnt -= 1                                                                                                                                          
                  if self.retry_cnt >= 0:                                                                                                                                      
                      logger.info("Retrying {0} on {1} # {2}".format(self.f.__name__,                                                                                          
                                                                    args[0].__name__,                                                                                          
                                                                    str(self.max_retry - self.retry_cnt)))                                                                     
                      time.sleep(0.001)                                                                                                                                        
                      results = self.__call__(*args, **kwargs)                                                                                                                 
                  else:                                                                                                                                                        
                      logger.error("Out of Retries for  {0}".format(self.f.__name__))                                                                                          
                      #reset                                                                                                                                                   

                     self.retry_cnt = self.max_retry
                    raise                                                                                                                                              
             else:                                                                                                                                                            
                  #reset                                                                                                                                                       
                 self.retry_cnt = self.max_retry                                                                                                                              
                 raise                                                                                                                                                        

         #reset                                                                                                                                                               
         self.retry_cnt = self.max_retry                                                                                                                                      
         return results                                                                                                                                                       

```


Do I need a close inside there? I wasn't sure the Response existed in the exception.


just saw your question.   let me try


too the hard to remove, rest of code expects a response object returned


Let me remove the entry retry decorator.  I probably don't need it if I configure the adaptor correctly


Okay, I removed the decorator entirely.  Still hangs even with "close()" being called.


Right. So we need to work out why the connection isn't being returned to the pool.


I walked through the debugger on the "close()" method.  It's hits line 118 of /package/urllib3/response.py and exit's out on the if statement.

```
def release_conn(self):
    if not self._pool or not self._connection:
        return
```

The self._connection variable appears to be None.


So in that situation we should have dropped the connection, which is what I'd expect if we'd exhausted it on the read (which we do).

@shazow, want to run your eye over this quickly?


So, I changed the if statement to just check if the pool is None. When it goes to put the connection back in the pool, it gets a "Full" exception.  The comment on that exception says that it should never happen if self.block == True (which it is in this case).


Hmm I'm not sure how to reproduce this with plain urllib3. Could be something with the Requests wrapping?

``` python
In [8]: import urllib3

In [9]: urllib3.add_stderr_logger()
2014-03-20 11:30:34,125 DEBUG Added an stderr logging handler to logger: urllib3
Out[9]: <logging.StreamHandler at 0x106673550>

In [10]: pool = urllib3.HTTPConnectionPool('httpbin.org', block=True, maxsize=1)

In [11]: r = pool.request('GET', '/')
2014-03-20 11:30:44,571 INFO Starting new HTTP connection (1): httpbin.org
2014-03-20 11:30:45,345 DEBUG "GET / HTTP/1.1" 200 7700

In [12]: r = pool.request('GET', '/')
2014-03-20 11:30:46,106 DEBUG "GET / HTTP/1.1" 200 7700

In [13]: r = pool.request('GET', '/')
2014-03-20 11:30:47,640 DEBUG "GET / HTTP/1.1" 200 7700
```


It's worth noting that we don't use `pool.request()`, we use `urlopen` and grab the connection by hand ourselves.


Perhaps we should be using `request` on an `HTTPConnectionPool` instance then?


There's shouldn't be much difference, all it does is handle determining the different request types (urlencoded or multipart etc) and basic headers: https://github.com/shazow/urllib3/blob/master/urllib3/request.py#L58

It's a convenience function over urlopen. I think Requests has something similar already (though feel free to use urllib3's if you'd rather not maintain your logic for it).


Another strange behavior problem, if I set "release_conn=True" in the call to url_open in the "send" method of requests/adapters.py.  I no longer hang on the second requests call.  However, it's still way too slow.  At the same time, if I go and comment out the "if release_conn:" in the finally statement of urlopen, it still works but is slow.  If I remove the "release_conn=True" in the url_open function call leaving the "if release_conn:" commented out like my previous example, it's back to behaving the way I expect ... super fast. I'm not sure why the release_conn=True would behave differently.


Ok, I think what we need to do is work out what exactly is happening. @jgatkinsn, are you familiar with Wireshark/tcpdump? I'd like to see some packet captures of each of the different permutations so I can get an idea of how the connections are proceeding.


I'm somewhat familiar(not an expert).  I was already using tshark to capture the packets between my script and a simple bash script that calls cURL to do the same thing.  It was leading me to believe the problem was somewhere in the python implemenation versus the super fast cURL bash script.  I'll try and grab my old captures and post them here.


I can't figure out how to attach the capture files to the Git site.  Do  you want me to email them directly to you?


Feel free. =)