Memory Leak

### I just started applications

![start](https://cloud.githubusercontent.com/assets/6700330/7748290/be4461ce-fff5-11e4-90d4-f44b29312325.JPG)

---
### After 24h

![after24h](https://cloud.githubusercontent.com/assets/6700330/7748331/29d13b9c-fff6-11e4-8bc5-823bd8272a3b.JPG)


At the beginning, the memory usage is 32MB.
After 24h, the memory usage is 1.2G.


Unfortunately, this 'leak' is somewhat unclear. Can you reduce this to a single bit of reproducible code for me to see?


Sorry.
My code is very simple.

Like:

```
class GetHtml():
    ...
    def get(self, headers)
        self.response = requests.get('http://'+self.target_host+self.uri, headers=headers)         
        self.body, self.headers = self.response.content, self.response.headers
    ...
```


Running at:
-   CentOS7.1
-   Python3.4.3

Other module:
-   chardet
-   flask


Do you have any active `GetHtml` classes lying around, or any references to `self.response` still active?


Absolutely no.


```
    def get(self, **headers):
        try:
            headers['Referer'] = 'http://' + self.target_host + '/'
            self.response = requests.get('http://'+self.target_host+self.uri, headers=headers)
            self.body, self.headers = self.response.content, self.response.headers
            if self.headers.get('Content-Type', '').startswith('text') and self.headers.get('Content-Type', '') != 'text/css':
                if self.target_host in self._encodings and self._encodings[self.target_host] in self.body:
                    encoding = self._encodings[self.target_host].decode('utf-8')
                else:
                    encoding = chardet.detect(self.body[100:1000])['encoding']
                    self._encodings[self.target_host] = encoding.encode('utf-8')
                self.body = self.body.decode(encoding, 'ignore')
                self.replace()
                self.body = self.body.encode(encoding, 'ignore')
            return self.response.status_code, self.body, self.headers
        except Exception as e:
            return 500, str(e), {'Content-Type':'text/html'}
```


## in flask

```
code, body, headers = rp.get(
    **{
        'User-Agent' : request.headers.get(
            'User-Agent',
            '''Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET4.0E; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)'''
            ),
    }
)
resp = make_response(body, code)
#for name, value in headers.items():
    #self.set_header(name, value)
if 'Content-Type' in headers:
    resp.headers['Content-Type'] = headers['Content-Type']

return resp
```


With a very basic code example, I cannot reproduce this leak. objgraph shows no extra allocations.

Can you try using objgraph to find out what is keeping those objects alive?

FYI, the code I used is below:

```
import requests
import signal
import threading


def handler(*args):
    import pdb
    pdb.set_trace()


def req_thread():
    while True:
        r = requests.get('http://www.http2bin.org/get')
        r.content


signal.signal(signal.SIGBREAK, handler)

ts = [threading.Thread(target=req_thread) for _ in range(0, 8)]
for t in ts:
    t.start()

for t in ts:
    while True:
        try:
            success = t.join(5)
        except IOError:
            continue
        if success:
            break
```


How about 599 error?  
There are so many 599 error.  
And I control source address and port by hook http.client.HTTPConnection.**init** method.  

```
def HTTPConnection_init_hook(HC):
    def wraps(*args, **kwargs):
        global IPS #It is list of my server ip.
        if IPS:
            kwargs["source_address"] = (random.choice(IPS), random.randint(10000,30000))
        return HC(*args, **kwargs) 
    return wraps


http.client.HTTPConnection.__init__ = HTTPConnection_init_hook(http.client.HTTPConnection.__init__)
```

**The port is possible be used, it will raise a Error.**


No more code,other code is not important.
I'm sorry, I can not do more.


Once again, it's going to be easiest if we can track down what is keeping these objects alive. If it's inside requests we can deal with it, and if it's inside your code I can provide guidance on how to improve things. Are you familiar with objgraph?


# rproxy.py

```
import re
import random
import http.client

import chardet
import requests

from . import host
from . import config


IPS_PATH = 'ips'
IPS = []

with open(IPS_PATH, 'rt') as f:
    for line in f.read().replace(' ', '').split('\n'):
        if line:
            IPS.append(line)


class ReverseProxy():
    _encodings = {}
    _titles = {}
    def __init__(self, local_host, target_host, uri):
        self.local_host = local_host
        self.target_host = target_host
        self.uri = uri
        self.is_index = uri == '/'

    def get(self, **headers):
        try:
            headers['Referer'] = 'http://' + self.target_host + '/'
            self.response = requests.get('http://'+self.target_host+self.uri, headers=headers)
            self.body, self.headers = self.response.content, self.response.headers
            if self.headers.get('Content-Type', '').startswith('text') and self.headers.get('Content-Type', '') != 'text/css':
                if self.target_host in self._encodings and self._encodings[self.target_host] in self.body:
                    encoding = self._encodings[self.target_host].decode('utf-8')
                else:
                    encoding = chardet.detect(self.body[100:1000])['encoding']
                    self._encodings[self.target_host] = encoding.encode('utf-8')
                self.body = self.body.decode(encoding, 'ignore')
                self.replace()
                self.body = self.body.encode(encoding, 'ignore')
            return self.response.status_code, self.body, self.headers
        except Exception as e:
            return 500, str(e), {'Content-Type':'text/html'}

    def replace(self):
        self.replace_title()
        self.replace_keywords()
        self.replace_description()
        self.replace_meta()
        self.replace_host()
        self.replace_js()
        self.replace_outlink()

    def replace_host(self):
        body = self.body
        body = re.sub('http://'+self.target_host + '/', '/', body, 0, re.IGNORECASE)
        body = re.sub(self.target_host, self.local_host, body, 0, re.IGNORECASE)
        self.body = body

    def replace_js(self):
        body = self.body
        body = re.sub(r'<script[\s\S]+?/script.*?>', '', body, 0, re.IGNORECASE)
        body = re.sub(r'<noscript[\s\S]+?/noscript.*?>', '', body, 0, re.IGNORECASE)
        if self.is_index:
            if re.search('</head', body, re.IGNORECASE):
                body = re.sub(r'</head', '<script type="text/javascript" src="/local.js"></script>\n</head', body, 1, re.IGNORECASE)
            elif re.search('<body', body, re.IGNORECASE):
                body = re.sub(r'<body', '<script type="text/javascript" src="/local.js"></script>\n<body', body, 1, re.IGNORECASE)
            else:
                raise Exception('no </head> and <body>')
        self.body = body

    def replace_outlink(self):
        self.body = re.sub(r'''<a[^<>]*?href=['"]?http://[^<]*?</a>''', '', self.body, 0, re.IGNORECASE)
        #add outlink
        self.body += ''.join([
            '<div id="footer"><b style="color:#03F; font-size:14px;">友情链接：</b>',
            '<ul>',
            ''.join([
                '<li><a href="http://'+h+'">'+config[h].get('host', 'title').split(',', 1)[0]+'</a></li>' for h in host.get_outlink_host(self.local_host)
            ]),
            '</ul>',
            '</div>',
        ])

    def replace_title(self):
        body = self.body

        title = config[self.local_host].get('host', 'title')
        main_key = title.split(',', 1)[0]

        if self._titles.get(self.target_host, None):
            replaced_title = self._titles.get(self.target_host)
        elif self.is_index:
            replaced_title = re.search('<title.*?>(.*?)</title', body, re.IGNORECASE).groups()[0]
            replaced_title = re.sub(r'[a-zA-Z0-9\s\-\.\*\\\/\,]', '', replaced_title, 0)
            replaced_title = ''.join(set(replaced_title))
            self._titles[self.target_host] = replaced_title
        else:
            replaced_title = '囃'

        if self.is_index:
            body = re.sub('<title.+?</title', '<title> '+ title +' </title', body, 0, re.IGNORECASE)
            body = re.sub('<h1.+?</h1', '<h1> '+ main_key +' </h1', body, 0, re.IGNORECASE) # replace h1
        body = re.sub('[' + replaced_title + ']{3,}', main_key, body)   #替换页面中的标题相关关键词

        self.body = body


    def replace_keywords(self):
        if self.is_index:
            self.body = re.sub(r'''<meta[^\n<>]+?name\s*=\s*(['"]?)keywords?\1[^\n]*?>''', '<meta name="keywords" content="'+ config[self.local_host].get('host', 'keywords') +'" />', self.body, 1, re.IGNORECASE)

    def replace_description(self):
        if self.is_index:
            self.body = re.sub(r'''<meta[^\n<>]+?name\s*=\s*(['"]?)descriptions?\1[^\n]*?>''', '<meta name="description" content="'+ config[self.local_host].get('host', 'description') +'" />', self.body, 1, re.IGNORECASE)

    def replace_meta(self):
        if self.is_index:
            body = self.body
            body = re.sub(r'''<meta[^\n<>]*?refresh[^<>]*?/>''', '', body, 0, re.IGNORECASE)
            if self.is_index:
                if re.search('</head', body, re.IGNORECASE):
                    body = re.sub(r'</head', '<meta http-equiv="Cache-Control" content="no-transform" />\n<meta http-equiv="Cache-Control" content="no-siteapp" />\n</head', body, 1, re.IGNORECASE)
                elif re.search('head>', body, re.IGNORECASE):
                    body = re.sub(r'head>', 'head>\n<meta http-equiv="Cache-Control" content="no-transform" />\n<meta http-equiv="Cache-Control" content="no-siteapp" />', body, 1, re.IGNORECASE)
            self.body = body


def block_ext(url):
    if '.' in url and url.split('?', 1)[0].rsplit('.', 1)[1] in \
       config.get('other', 'block_ext').replace(' ', '').split(','):
        return True
    else:
        return False

def HTTPConnection_init_hook(HC):
    def wraps(*args, **kwargs):
        global IPS
        if IPS:
            kwargs["source_address"] = (random.choice(IPS), random.randint(10000,30000))
        return HC(*args, **kwargs)
    return wraps


http.client.HTTPConnection.__init__ = HTTPConnection_init_hook(http.client.HTTPConnection.__init__)
```

# view.py

```
from flask import redirect, request, abort, make_response

from . import app
from . import host
from . import rproxy
from . import config


@app.before_request
def all():
    host_ = '.'.join(request.host.split('.')[-2:])
    if host_ != request.host:
        return redirect('http://'+host_)

    if not host.is_allow(request.host):
        return abort(400)

    uri = request.url[len(request.url_root)-1:]
    if rproxy.block_ext(uri):
        return abort(403)

    # local.js
    if uri == '/local.js':
        with open('local.js', 'rt') as f:
            return f.read()

    # reverse_proxy
    rp = rproxy.ReverseProxy(request.host, config[request.host].get('host', 'target'), uri)
    code, body, headers = rp.get(
        **{
            'User-Agent' : request.headers.get(
                'User-Agent',
                '''Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET4.0E; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)'''
                ),
        }
    )
    resp = make_response(body, code)
    #for name, value in headers.items():
        #self.set_header(name, value)
    if 'Content-Type' in headers:
        resp.headers['Content-Type'] = headers['Content-Type']

    return resp
```

# That is my code.


I use 'dozer' to trace memory leak.  
It gave me the above two pictures.  


The memory leak is slow.


So I need to say again that I agree that you have a memory leak, but without using `objgraph` we're going to find it very difficult to track it down. That's something you can do just as easily as I can. =)


For the requests you're making is there a content encoding? If not, this may be coming from chardet.


Thank you.
I remove the chardet module, and reinstall CentOS6.x.
Let me see tomorrow.


@Lukasa  It is very difficult to using `objgraph` in my code.
Because the memory leak is slow and `objgraph` can't track every `requests` instance,
 it will create a lot of pictures, Which picture Useful?

Thank You for your help.
If I find the code of memory leaks, I will contact you again.


# It's still here.

![unnamed qq screenshot20150524143820](https://cloud.githubusercontent.com/assets/6700330/7786962/b979daaa-0222-11e5-9851-b3c3eb066fab.png)


@030io can you please tell us what version of requests you're using? As you're using it, I see absolutely no way that this is coming from us. Since I don't know how your app is used, I can't tell you for sure, but you do realize that those class attributes will continue to accumulate information. Specifically `ReverseProxy._encodings` and `ReverseProxy._titles`. Those are class level attributes and will continue to grow over time even though you access them as `self._encodings`.


@sigmavirus24 
I use `http.client` instead of `requests`. And the memory leak is gone.


I'll ask again, can you please tell us what version of requests you're using?


sorry.
the newest version of requests in pypi.
running at: python3.4.3