manager: remove timer fd explicitly from epoll

Forked processes can keep the old fd alive triggering epoll over and
over again else.

https://bugs.freedesktop.org/show_bug.cgi?id=61697