Issue 1:
While processing a large set of requests, apparent_encoding() works well for quite a while but eventually hangs. All I have at this point is a traceback:

File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/models.py", line 637, in apparent_encoding
return chardet.detect(self.content)['encoding']
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/__init__.py", line 30, in detect
u.feed(aBuf)
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/universaldetector.py", line 128, in feed
if prober.feed(aBuf) == constants.eFoundIt:
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/charsetgroupprober.py", line 64, in feed
st = prober.feed(aBuf)
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/sjisprober.py", line 74, in feed
self._mContextAnalyzer.feed(aBuf[i + 1 - charLen:i + 3

I appreciate anything you can do to help.

Issue 2:
I am using chardet as part of a web crawler written in python3. I noticed that over time (many hours), the program consumes all memory. I narrowed down the problem to a single call of chardet.detect() method for certain web pages.

After some testing, it seems that chardet has problem with some special input and I managed to get a sample of such an input. It consumes on my machine about 220 MB of memory (however, the input is 2.5 MB) and takes about 1:22 minutes to process (in contrast to 43 ms when the file is truncated to about 2 MB). It seems not to be limited to python3, in python2 the memory consumption is even worse (312 MB).

Versions:

Fedora release 20 (Heisenbug) x86_64

chardet-2.2.1 (via pip)

python3-3.3.2-11.fc20.x86_64

python-2.7.5-11.fc20.x86_64

How to reproduce:

I cannot attach any files to this issue so I uploaded them to my dropbox account: https://www.dropbox.com/sh/26dry8zj18cv0m1/sKgP_E44qx/chardet_test.zip Please let me know of a better place where to put it if necessary. Here is an overview of the content and the results:

setup='import chardet; html = open("mem_leak_html.txt", "rb").read()'
python3 -m timeit -s "$setup" 'chardet.detect(html[:2543482])'
# produces: 10 loops, best of 3: 43 ms per loop
python3 -m timeit -s "$setup" 'chardet.detect(html[:2543483])'
# produces: 1 loops, best of 3: 1min 22s per loop
python3 mem_leak_test.py
# produces:
# Good input left 2.65 MB of unfreed memory.
# Bad input left 220.16 MB of unfreed memory.

python -m timeit -s "$setup" 'chardet.detect(html[:2543482])'
# produces: 10 loops, best of 3: 41.7 ms per loop
python -m timeit -s "$setup" 'chardet.detect(html[:2543483])'
# produces: 10 loops, best of 3: 111 sec per loop
python mem_leak_test.py
# produces:
# Good input left 3.00 MB of unfreed memory.
# Bad input left 312.00 MB of unfreed memory.

mem_leak_test.py:

import resource
import chardet
import gc

mem_use = lambda: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
html = open("mem_leak_html.txt", "rb").read()

def test(desc, instr):
gc.collect()
mem_start = mem_use()
chardet.detect(instr)
gc.collect()
mem_used = mem_use() - mem_start
print('%s left %.2f MB of unfreed memory.' % (desc, mem_used))

test('Good input', html[:2543482])
test('Bad input', html[:2543483])