Make sure to remove the timeout task when shutting down the session manager