Fix timers properly when reloading configuration