Handle reload timeouts correctly

Should not load plugins again if we timed out reloading web configs.