basho/riak_cs — pull request #552 by kellymclaughlin (2013-05-03 22:20:47+00:00)
Title: Memory not released from access archiver process

This [issue](http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-April/011805.html) was reported on the mailing list and I was able to reproduce using the provided python code.

My testing shows that the problem is that when the access archiver process receives ownership of an ets table, it retains the memory used by that table even after the table is deleted.

Forcing gc on the process immediately free the memory, but this does not happen automatically for some reason.

The short-term fix is make the process go into hibernation when it goes to the idle state, which is where is spends the majority of its time.

### Testing

I used the following python script provided by Stefan Silasi on the mailing list to provoke the condition:

```
#!/usr/bin/env python

import boto
import boto.s3
import boto.s3.connection

conn = boto.connect_s3(
aws_access_key_id = 'KEYID',
aws_secret_access_key = 'KEYSECRET',
host = "s3.amazonaws.com",
proxy = "127.0.0.1",
proxy_port = 8080,
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
debug = 2,
is_secure = False,
)

bucket = conn.create_bucket('bucket1')

for row in range(1000000):
key=bucket.new_key('testkey')
if key.exists():
key.open_read()
```

It is not important that `testkey` actually exist. It is simply the quantity of requests that matters. As it runs you should see memory usage increasing. Some of this is due to access information being stored in an ets table. If the script completes it should trigger an automatic flushing of the access information; otherwise, it can be forced by running `riak-cs-access flush`.

Prior to this change, after the archiving is completed the memory usage should remain elevated (indefinitely from what I saw).

Closer inspection should show that prior to the archiving step, there is an ets table consuming a large chunk of memory and the access archiver process is using a normal amount (`process_info(whereis(riak_cs_access_archiver), memory)`). To see this the python script needs to be killed prior to completion so the automatic archiving does not happen. I used to following to spit out all the ets table names with owners and memory usage:

```
[io:format("~p ~p ~p~n", [T, ets:info(T, owner), ets:info(T, memory)]) || T <- ets:all()].
```

After flushing the access data with `riak-cs-access flush`, the large ets table should be deleted. without this change the `riak_cs_access_archiver` process should now have a unexpectedly large heap size, but using this change the archiver process should have gone back to hibernating and in doing so freed up most of its heap.