gmcquillan/firetower — Issue #30: Even Idle Firetower Server Spends All Its Time Archiving
Author: gmcquillan
Created: 2012-01-25 20:47:52+00:00
State: open
Labels: bug

I noticed something strange on my larger installation of Firetower (which has about 3380 categories) -- the server is consistently taking up about 16% of a CPU, even when no new events are coming in to be categorized.

This is the strace output for that process:
```recvfrom(3, "*0\r\n", 8192, 0, NULL, NULL) = 4
write(4, "[2012-01-25 20:42] DEBUG: Fireto"..., 108) = 108
sendto(3, "*2\r\n$7\r\nHGETALL\r\n$48\r\ncounter_9d"..., 72, 0, NULL, 0) = 72
recvfrom(3, "*0\r\n", 8192, 0, NULL, NULL) = 4
write(4, "[2012-01-25 20:42] DEBUG: Fireto"..., 108) = 108
sendto(3, "*2\r\n$7\r\nHGETALL\r\n$48\r\ncounter_bb"..., 72, 0, NULL, 0) = 72
recvfrom(3, "*0\r\n", 8192, 0, NULL, NULL) = 4
select(0, NULL, NULL, NULL, {1, 0}^C
```

Clearly, it's spending all its time doing HGETALL calls for the various categories to check and see if they have any data to archive. This is hardcoded to happen every 2 seconds. The problem now is that this process takes more than 2 seconds to complete, so it's pretty much constant.

Two pronged solution:
- make the default a little longer (say 60 seconds), and configurable.
- This many categories isn't ideal. Need to address the category dispersion.