Adding config option to enable/disable rerun