Add existing certificate to CA store on upgrade