Regenerate configs before upstreaming