BrightcoveOS/Diamond — pull request #372 by lambdafu (2013-05-13 23:03:10+00:00)
Title: Do not use abs mode for threaded or forked interval tasks.

Threading and interval tasks do not mix with abs mode, as the next
job is scheduled immediately after spawning the thread for the current
job. Not only does it not make sense (a job running significant
time is scheduled too often), it is actually harmful (a long running
job will cause the calculated delay to be zero, which creates a tight
loop that spawns threads until the system runs out of process slots,
essentially making diamond a temporary fork bomb).

This may do something positive for Issues #203 and #206 (hey, maybe even fix them!).

I actually saw this happening when running diamond with a 10 seconds default interval logging through circus, which only reads once from the logging pipe every second (of course that's a separate bug). The amount of logging from the disk space collector overwhelms circus (ha!) and makes it run longer than 10 seconds. Hilarity ensues as diamond spawns 1000 disk collector threads which cause other parts of the system to randomly fail until they all squeezed their output through the circus logging channel, and the whole process starts again.

Here is an example program that forces the problem. It limits itself to 10 threads, to protect you from total desaster.

```#!/usr/bin/python
# coding=utf-8
################################################################################

import unittest
import configobj

from diamond import scheduler
import time
from itertools import count

class BaseCollectorTest(unittest.TestCase):

def test_ThreadedIntervalTask(self):
counter = count(start=1)

def ping(sched, msg):
cnt = counter.next()
print cnt, time.asctime(time.localtime()), msg
if cnt == 10:
sched.stop()
time.sleep(3)
print cnt, "done"

sched = scheduler.Scheduler()
sched.add_interval_task(ping, "ping", 0, 2, scheduler.method.threaded,
[sched, 'hello'], None, abs=True)
sched.start()

self.assertEquals('test', 'test')
print "HALLO"

if __name__ == '__main__':
unittest.main()
```