binarydud/pyres — Issue #91: forking question
Author: yashh
Created: 2011-11-28 04:13:45+00:00
State: open

Hey I been using pyres for quite some time but only as a secondary queuing infrastructure. I have a couple of questions I want to raise since I am giving serious thoughts on using pyres as our primary queue.

- Forking, I see that every worker forks himself and the forked worker performs the task. But in our situation we have a lot of
cache connections, sql connections. We pool all these connections to reuse the open connection, since we are facing high
traffic. I want to know whats the advantage of forking ? Can we just make workers BLPOP on redis and pick up task from redis
process and perform BLPOP ?

- Restarting workers. Is there a graceful way of restarting workers? Currently we are using supervisors to manage workers. We
are killing all workers and then restarting the supervisor. Can we make sure `kill pid` will not abruptly kill the task but kill after
completing one?

Pretty much forking has been slow in my opinion. I see that tasks which are blazing fast take about a second on pyres since it has to open a bunch of connections etc. Wht do u think?