netty/netty — Issue #163: ZlibDecoder and HttpContentCompressor consumes too much memory
Author: trustin
Created: 2012-01-27 08:20:24+00:00
State: open
Labels: improvement

Got the following message:

Hi Trustin - need your help to add a change to HttpContentCompressor. I need to set bits and memLevel along with compression level. Currently, compression level is the only value that can be set. Here is what the issue is -
In our cluster we have 10-20K long lived user connections. Once I enabled compression on then I noticed that more than 70% requests were using compression consuming about 5G additional memory. A heap dump showed that there are 4 x 64K buffers per each connection held in Deflate.java. More specifically the members "window", "pending_buf", "prev" and "head" are each growing to 64K and holding on to the memory. What I have done is taken that current classes and modified so that I can set "bits" to 12 from 15 and "memLevel" to 5 form 8. This bring down memory consumption to 8K per buffer. I am about to check how much compression I lose due to this but seems like it's not that bad. On other clusters we have fewer connections so the defaults should be fine. If these values are adjustable we can tune it based on our cluster requirement.

I need to see if both bits and memLevel needs to be set by user or bits can be calculated from memLevel (or the reverse).

Even better, if would be great if we can release those large buffers when unused.