bumptech/palm — pull request #30 by dowski (2013-07-09 15:44:25+00:00)
Title: Less memory

## Memory Efficiency Changes

A couple tweaks to help PALM use less memory.

### Repeated fields grow more slowly now

Previously they grew in N * 64 chunks, starting at 64. This meant that if you pushed over 4096 items in a repeated field, the next step was to allocate room for 262144 items. That required 25 MB of RAM (the `pbf_mark` structure it was making space for is 96 bytes on a 64 bit machine). Now they grow in N * 2 chunks.

### Fields aren't over allocated now

Previously when writing a field, we allocated 100 fields beyond the field that was being written. That means if you were writing sub messages with just a few fields, each one was reserving enough space for 100+. The over allocation is the reason that PALM was using hundreds of MB of RAM to serialize in memory `ProtoBase` instances with many repeated sub-messages. I tracked this down using Valgrind's [massif](http://valgrind.org/docs/manual/ms-manual.html) tool.

### tl;dr

The practical upshot of these changes is the `normal-append` memory check uses 25% of the memory that it previously did. The `fast-append` check also uses less memory due to the more gradual repeated growth.

```
master $ python check_mem.py
baseline 32828 6892
normal-append 633684 607304
fast-append 59040 33356

less-memory $ python check_mem.py
baseline 32828 6892
normal-append 155868 129476
fast-append 40812 15104
```