Disable TimeoutManager on workers

TimeoutManager is only enabled if is a distributor or not a worker