Socket leakage - CLOSE_WAIT - multithreading

Any thoughts please as I ran out of ideas what could be wrong. The problem is very possibly in the way I use requests and unfortunately I can't provide detailed configuration or the code itself but here we go.

Design: 

Note: All system parts are currently running on the same server so the requests.sessions are to localhost.

The application has multiple virtual encryption classes (VCS), each can use one of upstream encryption services (ES). One of ESes has a RESTful API -> the virtual encryption class uses requests.get() for queries.

When the application starts, it creates an instance of VCS and this in turn creates its instance of a  configuration class for an ES.

there is a main static method, which listens on downstream connections. On each such connection, it will create a WorkingThread which is given the VCS instance as a parameter. The WorkingThread:
1. asks ES to create a requests.session (static method)
2. runs a bunch of get() requests
3. closes the requests.session

The downstream client serializes connections so at the moment there's only one WorkingThread running at any one time. All upstream connections are to the same URL, what changes are HTTP GET parameters.

What happens is that when there are, e.g., 10 connections created back-to-back, the application leaves 9 sockets in the CLOSE_WAIT state. If there is just 1 downstream connection, the socket is correctly closed.

## Expected Result

Correctly closed sockets.

## Actual Result

When a new requests.Session() is created immediately after the previous has been closed, it leaks a socket (indefinite CLOSE_WAIT).

## Reproduction Steps

Not available at the moment - I tried run get() calls within "with", with "Connection: close" header, explicitly closing responses as well as the session, disabling selinux to no avail.

The dataflow summary: TCP server accepts a connection, creates a workingthread, which is given an global persistent object instance with an upstream config. the workingThread opens a requests.session, runs several get() requests, closes the session and sends a TCP response back.

## System Information

I can't access the server at the moment, but as of 20th Feb, 2019

Centos7.6,  Python 2.7.5 requests and other packages are up to date, installed with pip


## System Information

{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": ""
  }, 
  "idna": {
    "version": "2.6"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.5"
  }, 
  "platform": {
    "release": "3.10.0-693.17.1.el7.x86_64", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "", 
    "version": null
  }, 
  "requests": {
    "version": "2.21.0"
  }, 
  "system_ssl": {
    "version": "1000105f"
  }, 
  "urllib3": {
    "version": "1.24.1"
  }, 
  "using_pyopenssl": false
}



I'm still trying to replicate with a small python script. So far, I have managed to do that once (several consecutive runs but I lost it after changing the code slightly). It seemed like the thread doing get() requests was terminated while the session was still in the ESTABLISHED state. It would then go directly into infinite CLOSE_WAIT.

As I mentioned, if there are 10 tasks (each = 1 thread and multiple get() requests), only the last one leaves its socket in CLOSE_WAIT state. These threads have to be running spawned immediately after each other (first responds downstream -> a new thread is spawn, so 2 threads can overlap). If there is a gap between following threads, all sockets close correctly.

If I don't close the session, I get CLOSE_WAIT for all tasks I execute.

The most recent version uses simply requests.get(....) calls -> session should be torn down immediately after the response is received (stream=False ... i.e., default value).



BTW, the server to which the Python script connects to is a Java Spring app.
all requests are to localhost



ok, so the problem remains even when I use urllib/urllib2.

 If anyone has any thoughts, I'd really appreciate to hear any ideas.

... I should have mentioned that I'm opening a few hundred connections within a couple of seconds.

Looks like I ran into the similar (or same problem). Many threads, one session..

have you tried using the Connection:close header....HTTP 1.1 by default uses Connection:keep-alive, so if a  socket ends its job, and the server did nor response back with a connection:close or when chucked responses you the client dont  read the last chunked bite...the tcp socket will still open until the handler closes it...you can simple read the data using the response.content property, you can copy the response object or is you are using session,  try a request ina context.