rakshasa/rtorrent — Issue #7: High memory usage leads to fork errors
Author: ntg
Created: 2011-11-18 05:01:46+00:00
State: open

Sorry about the format of this bug report, it's kind of vague because I don't have much detail to give. This seems to be an isolated occurrence, but it seems serious enough to report in case others are having similar problems.

Quick baseline:
rtorrent/libtor svn1272, Ubuntu 11.04, 16GB of RAM

Description:
On my machine with 16GB of RAM, rtorrent is the only thing running and it somehow got itself into a spot where it was using 50.1% (8.0001GB) of resident memory (as shown by top). At that point it started spitting out fork errors (ERR_NOMEM) every time it tried to spawn a scheduled process (which happens a few times a minute). I didn't notice this until 6 hours later, at which point I had to kill and restart rtorrent to free the RAM. In that entire time, rtorrent continued with its data transfers at ~2MB/s up&down, but all other scheduled activity stopped due to fork errors. Obviously the forks were failing because each child requires exactly as much memory as the parent, but...

Questions:
1) why would rtorrent require 8GB in the first place? I could understand if there were high speeds and thousands of buffers involved, but transfer speeds were slow--at only 2MB/s up and down--with 17 active torrents. If it makes a difference, the total size of the active torrents was about 500GB, the size of ALL seeding torrents was over a terabyte, and the active torrents included 4 slow downloads totalling 100GB, and downloading at a total of 2MB/s. rtorrent reported memory usage of ~150MB, so there must have been almost 8GB of memmapped files. I'm afraid that running memtest slipped my mind until after I killed/restarted rtorrent, but 8GB seems extremely high no matter what it was trying to do. I lowered the bufsize to 256k, maxOpenFiles to 4, and stopped all torrents except the downloads, but there was no change in memory usage so I just shut down rtorrent and restarted.

2) Has rtorrent always forked the scheduled processes this way? Because if so, then rtorrent could never have used more than half the available memory, which seems to be a huge waste (and it also should have caused problems for me and many other people). And, it also means that there could only be one forked process running at a time, or the memory problem goes up exponentially. Either way, would it be possible to fix this by running just a small controller as the main program, which would delegate all the memmaps and scheduled processes into separately-forked child processes? This way the main program and all of its forks would be tiny, although I assume it could take a fair amount of rewriting to structure it this way (I haven't looked at the code yet).

3) Although I didn't get to run memstat, is there anything that seems likely to be causing this? Could the memmapped regions be leaking memory instead of getting freed, or any other likely explanation? I've watched the resident memory, and in normal use the RAM usage occasionally jumps up to ~2GB when I'm downloading above 50MB/s (400Mbps)...but within seconds it flushes the data and the RAM usage returns to just a couple hundred MB. This time it seems to have kept rising until it hit 8GB, at which point it couldn't rise any further or apparently even drop, and it just got stuck until I shut down rtorrent.

4) ...Or is it maybe something that's just configured wrong with my setup, and not rtorrent at all?

Again, sorry for the report with lots of talk and few facts, but I'm hoping this description is enough to give an idea of what happened.