Option to not start runner after setup