Possible Memory Leak 

I'm crawling a lot of different URLs using the requests library and I encountered that the process takes more and more RAM over time. 
Basically all I do is calling this iteratively from multiple threads:
```{python}
r  = requests.get(url=url, timeout=timeout)
content = r.text
```
when I comment out the second line this issue does not occur ... am I using the library fundamentally wrong or could this actually be a bug? 

What are you doing with the content? What's the rate of memory growth? Can you try following [this blog post](http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-python-memory-leaks/) to gather data about *what* objects are being retained, ideally after your memory has grown very substantially (e.g. to more than 10 times the initial process memory size).

Also, I have a hunch, so humor my following question: Does the server provide a charset/character encoding? If not, does the leak persist if you do `content = r.content`?

@Lukasa I wonder if there's a leak in chardet.

Good hunch. Checking if the problem appears with a change of `text` to `content` would also work to isolate that.

@Lukasa 
> What are you doing with the content?

I store  _content_ in a queue and have a worker thread that dumps the data in a csv file. 
However I isolated the download workers and skipped the part of storing it in the queue. 
So basically all I do now is this (using multiple threads):
```
while True:
    r  = requests.get(url=url, timeout=timeout)
    content = r.text
```
If I delete the assignment to _content_ the memory does not grow anymore.
```
while True:
    r  = requests.get(url=url, timeout=timeout)
```
> What's the rate of memory growth?

The initial RAM usage is at about 100MB and grows to 2GB within ~24 hours.

@sigmavirus24 
> Does the server provide a charset/character encoding?

Good question. Since my URL list contains more than 100 servers I would have to check for each. 
Besides what exactly do you mean by "charset/character encoding". Are you referring to the HTTP response header's field _Content-Encoding_? 
 
> If not, does the leak happen if you do content = r.content?

I will give that a try. 

No, @sigmavirus24 is referring to having a `charset` parameter in the `content-type` header field.

Ah okay. I wrote a little script that prints the content-type field of all responses. 
Content types are:
```
Content-Type: text/xml
Content-Type: text/xml;charset=UTF-8
Content-Type: application/rss+xml;charset=UTF-8
```
Additionally, some of those servers don't even provide a Content-Type field.


Ok, so the ones with no `content-type` and the ones with no `charset` are the ones we're interested in. Try changing `text` to `content` to see if it reproduces.

Alright, I compared both versions now:

| time in h   | content        | text |
| ------------- |-------------| -----|
| 00:00       | 0.9%| 1.0% |
| 00:30       | 1.3%|1.7% |
| 01:00       | 1.4% |2.1% |
| 01:30       | 1.4% | |
| 02:00       | 1.4% | 2.3%|
| 02:30       | 1.4% |2.3% |
| 03:00       | 1.4% |2.4% |
| 03:30       | 1.5% |2.4% |
| 04:00       | 1.5% | 2.4%|
|4:30| | 2.5%|
|...|||
|~6:00h|1.6%| |

In case you're interested in absolute values: my notebook has 10GB of RAM. 
It seems like RAM usage growing something like a root function. 
This behavior is equivalent to what I have encountered when using my crawler although there it's taking GBs of RAM. 

This what my requests test script looks like:
https://gist.github.com/le1nux/c94b3707f076edbd982299906737b1ca 

Ok, so the leaking does appear to occur more *slowly* when using `content`, but it is vaguely present. In both of these cases, though, this may not be a leak: given that the growth is asymptotic it may be a property of the Python allocator, which does not always return object memory to the OS. In this case the process with `content` is asymptotically growing to about 160MB: large, but not completely absurd, especially for large response bodies.

I think we need to see `objgraph` results from a long run to see how much data is actually floating around.

>  In this case the process with content is asymptotically growing to about 160MB: large, but not completely absurd, especially for large response bodies.

Well, I only requested XML data which is not really big. 

Character length:

|   max   |   min   |   avg   |   median   |
|-----------|---------|----------|----------------|
| 610869 | 876 | 51329 | 21592|

I will replace my URL list by URLs that direct to bigger websites and see if I can reproduce the huge RAM usage of my crawler. 




> it may be a property of the Python allocator, which does not always return object memory to the OS.

So actually, the last time I spent a few weeks trying to track down a memory leak in requests what I learned is that the Resident Size measure (RSS) is actually not the amount of memory *actively* used by your program but the maximum amount of memory it has allocated thus far. And if your OS doesn't need to reclaim any of that, that value can stay the same and generally does not shrink.

Not sure how relevant it is, but there was an interesting thread a month ago on Python-Dev about CPython's memory allocation approach, and some of its consequences. The thread starts here:
https://mail.python.org/pipermail/python-dev/2017-June/148105.html



@sigmavirus24 
Good point. However I'm still asking myself why my crawler (i.e. the requests library probably) allocates that amount of RAM in the first place.  

@le1nux btw, can you please fill out the template you had to delete to report this issue in the first place. We *did* have a leak a couple years back with the functional API you're using and without knowing what version you're on, I don't have a good way of telling you if that's the issue.

Hey @le1nux I'm going to close this since the investigation has stalled and we don't really have any concrete things to chase down. We'll be happy to reopen it if you can bring us the details we need to continue the investigation. Cheers!

Hey @sigmavirus24 I'm sorry for not having provided any further information, yet. At the moment I'm busy working on something else but I will get back to it in approx. two weeks. I will investigate that issue further and if it persists I will come back to you guys. 

I have a similar issue where additional objects are created everytime r=requests.get() is called and r.content is retrieved in multithreaded program
                       <class 'str |           4 |    275     B
  <class 'urllib.parse.SplitResult |           1 |     88     B


Using python 2.7.15 and requests 2.21.0
And I encountering the exact same memory leak.

Since this issue seems to persist for others, I set up my old crawler again and updated to the latest requests library version (2.21.0). I ran it for a couple of hours and the memory keeps still increasing. In my opinion my light-wight rss-feed-crawler might be an "easy" starting point to track it down? 

https://github.com/le1nux/crawly

In any case, I will be happy to assist tracking down this possible memory leak. 

Python 3.7.2 and requests 2.21.0
And I also have the exact same memory leak.

09/12/2019
Windows-7 python38 requests 2.22.0 -- the same leakage found while on threadpool execution

27 Dec 2020
Think I've found the same: Python 3.7, requests 2.22.0. 

I'm also seeing this exact same memory leak - Python 3.6.9, requests 2.23.0. Has anyone found any work arounds?