Only skip the main config

Otherwise read only additional configs might not be processed