[Tracking] slow leak in latest requests with https connections

This seems unrelated to https://github.com/kennethreitz/requests/issues/1685 as this is specific to https connections and I'm specifically closing the session and response connections.

I've spent a few days investigating a memory leak we've seen with the requests module and narrowed it down to three issues:

1) when using pyOpenSSL with requests 2.13.0 in python 3.5.2 there's one leak:
    1) with pyOpenSSL with a local server there's a leak, removing pyOpenSSL resolves leak
2) when using pyOpenSSL with requests 2.13.0 in python 3.5.3 there's two leaks:
    1) same as 1.1
    2) when connecting to servers on different hosts there's a leak
3) in both 3.5.2 and 3.5.3 with external hosts the script is periodically unable to contact the host due to "network is unreachable" errors, presumably because some resource has been temporarily exhausted...eventually recovering and repeating the error.

I've investigated this some and it doesn't seem to be a leak python objects (see tracemalloc util in script), or a leak sockets/file descriptors, so I believe it's a native leak in either python or openssl given the behavior changed between 3.5.2. and 3.5.3.  In our 3.5.2 tests we were using libssl1.0.1 and 3.5.3 libssl1.1.0 and upgrading libssl did not fix the issue.

I've created a test file here: https://gist.github.com/thehesiod/5dbd7f2bffbe0b850980e865f5649338

and a docker cloud image here: `amohr/testing:stretch_request_leak` based on:
```
FROM debian:stretch

COPY request_https_leak.py /tmp/request_https_leak.py

RUN apt-get update && \
    apt-get install -y python3.5 python3-pip git

RUN python3 -m pip install requests git+git://github.com/thehesiod/pyca.git@fix-py3#egg=calib setproctitle requests psutil
```

it constantly updates three files in the directory it's run from:
- mprofile_*.dat: memory usage by number of requests
- nprofile_*.dat: number of net connections by number of requests
- fprofile_*.dat: number of file descriptors by number of requests

you can plot the memory usage with the python module: memory_profiler with `mprof plot [FILENAME]`

Here's an example run from the image above running for 15,000 requests....while it seems to stabilize I've noticed that running it for a long time it continues with a fairly constant long term line...I think the python malloc just starts requesting larger chunks over time.

![stretch-google](https://cloud.githubusercontent.com/assets/4001789/24168210/ba4da3a4-0e36-11e7-8518-586c9cb0e595.jpg)

btw x-axis in the plots generated is request num, not time.

Also in the script is the ability to switch to a local server (which is one of the issues), when it generates the temp certs, hit enter for everything except common-name, and just put the name it tells you to put there.

Next steps I think are attempting to try a native python ssl socket to see if the problem reproduces there to see if this is a native ssl issue (would be good to test aiohttp as well to see if it doesn't reproduce if they have a work-around, and if it does they should know), and then a full valgrind investigation to see what's leaking the memory.

Thanks for this report! It's very detailed, which is a good start. I'd like to confirm some things:

1. You used 3.5.2 and 3.5.3, with libssl 1.0.1 and 1.1.0 respectively. You said, however, that "upgrading libssl didn't fix the problem". Did that mean just moving to the latest libssl in those releases, or moving 1.0.1 to 1.1.0 on 3.5.2.
2. Does bug 2 go away when you remove pyopenssl?
3. Does this reproduce with a simpler scenario? That is, if you change the response to, for example, be a simple 200 OK? What about if you change the SSL method to SSLv23_METHOD?
4. Does this reproduce on different distributions? Your docker image is Debian stretch: if you use, say, Fedora Rawhide, does that problem go away?

I'm not super well at the moment so I probably won't be looking at this in the next day or two, so if anyone else wants to dive in then feel free. Also, I'm going to CC some folks associated with PyOpenSSL: @hynek @alex @reaperhulk.

1. Since 3.5.2 doesn't exhibit problem 2.2 I didn't try upgrading libssl there (from base of 1.0.1t), for 3.5.3 I tried with both 1.0.1t and 1.1.0e and both leaked for issue 2.2
2. The only way to get rid of bug 2.2 is using 3.5.2 w/o pyopenssl
3. I was able to reproduce this with our internal server that simply does a 301 w/o body.  Also note that I'm doing `head` in the test, and not `get`.  I haven't tried with different SSL methods.  The core is simply: 
```python
def do_one_req():
    global _req_num

    try:
        response = requests.head(URL, verify=VERIFY,  allow_redirects=False, timeout=1)
        for r in response.history:
            r.close()
        response.close()
        return response
    except:
        traceback.print_exc()
    finally:
        _req_num.value += 1
```
4. that is a good question, I haven't tried different distributions yet.

I hope you feel better soon btw!  These last few months have been pretty terrible for colds/flus :(

also btw, perhaps in general requests shouldn't be trying to pull in pyOpenSSL for 3+ since from what I understand it's not needed in those releases anymore?

So we pull in PyOpenSSL generally because it often has access to newer copies of OpenSSL, especially if installed via pip. 

ah, following the trail I see how it can dynamically load the openssl library, but couldn't find the code where it picks the newest version if multiple versions are installed: https://github.com/pyca/cryptography/blob/master/src/_cffi_src/build_openssl.py.

I believe I've verified this is a bug in SSLSocket.  I've updated the gist to support raw sockets and aiohttp and I see the leak there as well.

I've validated that calling `close()` vs `shutdown()` doesn't seem to make a difference, nor does using a global `ssl.create_default_context()`  (as this doesn't have a `close()` method I was curious).  I just opened: http://bugs.python.org/issue29870.  Hopefully someone looks into it.

I've validated that this happens in Fedora:Rawhide w/ python 3.6.1rc1 as well, and 3.5.2's behavior is  night and day.  There were a few interesting SSL changes in 3.5.3 (https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-3) so probably caused by one of them.  Take a look at this, Deb 3.5.2 on bottom vs fedora 3.6 on top (similar to all 3.5.3+ versions).

![fedora_36_vs_deb_352](https://cloud.githubusercontent.com/assets/4001789/24178327/af8855de-0e65-11e7-8cd7-7c7bfa8876fd.jpg)


Interesting. Let's leave this issue open while upstream takes a look at theirs and see what happens. 

leak found!  they're working on the correct fix now

btw, think it's worth adding a python warning when we know the definitive list of versions that leak?

It's a leak in the code that handles CRL distribution points extension, https://github.com/python/cpython/pull/1085

I think a warning might be a bit aggressive: the leak is a bug that should get fixed, so users should deal with that in the standard manner they deal with all bugs in dependencies. If we take it upon ourselves to warn for all bugs in dependencies we'll find that takes basically all our maintenance time. 😉

Also, well done to all involved to track down and diagnose this leak! ❤️

ok closing issue then