Apple Pay Certificate Rotation
By: Daniel Book
Tags: Black Friday, Ecommerce, Engineering, Certification, Apple Pay

Introduction

Black Friday Scare

In late October 2020, Bolt discovered that our Apple Pay certificates were set to expire in a month — just days before the busiest shopping day of the year: Black Friday.

Notification of an expiring certificate in our sandbox environment

If Bolt unsuccessfully replaced these certificates, all Bolt merchants would experience a complete Apple Pay outage.

With Apple Pay accounting for millions of dollars of transaction volume in the Bolt Network, a complete Apple Pay outage in the days leading up to Black Friday would be unacceptable. We assumed that Apple Pay would temporarily allow two active certificates to facilitate a standard rotation procedure; why would replacing the certificates be anything besides straightforward?

Overview of Apple Pay at Bolt

Bolt and Apple Pay checkout buttons

Before diving deeper into this problem, let’s take a step back to understand how Bolt uses Apple Pay. There are three Apple Pay certificates that Bolt manages in order to process an Apple Pay transaction:

Domain Verification Certificate — to render the Apple Pay button on a merchant’s website, the merchant’s domain must be registered with an Apple Merchant ID and verified by Apple. Verifying the domain with Apple entails uploading a merchant-domain-specific file provided by Apple to an Apple-specified location on the merchant’s website. Domain verification is used to prove ownership of a domain to Apple. Merchant Identity Certificate — when a customer clicks the Apple Pay button, Bolt sends a start session request to the Apple Pay servers with an Apple Pay Merchant Identity certificate. This request must return successfully in order for the Apple Pay checkout to proceed. The start session request is used to prove that a valid merchant domain is associated with the entity trying to process a transaction. There can be multiple active Merchant Identity certificates associated with a single Merchant ID at any given time. Payment Processing Certificate — once a customer decides to pay, Apple Pay encrypts the Payment Data using the public key associated with the certificate received from the start_session response and sends the encrypted data to Bolt’s servers. Bolt uses the Payment Processing certificate to decrypt and extract the necessary information to perform an authorization request. Encrypting/decrypting the Payment Data with the Payment Processing certificate protects data integrity in transit and ensures only valid entities can process transactions with the data. There can only be one active Apple Pay Payment Processing certificate associated with an Apple Merchant ID at any given time.

Apple Pay certificate flow

Bolt uses a few different environments for Apple Pay: staging (for internal testing), sandbox (for merchant testing), and production . Each environment has its own Apple Merchant ID with an associated Merchant Identity certificate and Payment Processing certificate. To onboard a merchant into our Apple Pay flow, Bolt creates and manages the merchant-domain-specific files under the Merchant ID for the environment. As a result, Bolt can use one Payment Processing certificate to decrypt the payment data for all merchants in a single environment.

Bolt originally built an Apple Pay integration in 2018, and, in the two years leading up to October 2020, we had received notifications from Apple Developer Relations whenever the Domain Verification for one of our merchants using Apple Pay was set to expire. The only step required to re-verify a domain is uploading a new certificate; there is no risk of downtime unless the new certificate is uploaded after the old certificate expires.

When Bolt was notified of the impending expiration of both our Merchant Identity certificates and Payment Processing certificates, we incorrectly assumed that updating these certificates would be similar to reverifying a domain — limited in both scope and impact. However, after investigating the usages of these certificates in our core Apple Pay code (which hadn’t been touched in almost 2 years), we realized that any mistakes in updating either of the certificates would result in a full Apple Pay outage for all Bolt merchants.

Rotation

For both the Merchant Identity certificate and Payment Processing certificate, we planned to test and formalize our rotation process on staging before rotating the certificates on sandbox on production. However, as mentioned above, per environment there can be multiple active Merchant Identity certificates at any given time, but there can only be one active Payment Processing certificate. This difference significantly impacted our approach to rotating the certificates.

Merchant Identity Certificate

This was the easier of the two certificates to rotate. Because multiple certificates could be active simultaneously, we were able to generate a new certificate for the environment and test that it was working with one-off create session requests before performing the rotation. The rotation occurred in a 4 step process:

Generate a new Merchant Identity certificate Securely generate the key pair and store the private key in AWS KMS or Hashicorp’s Vault under a new name (i.e. merchant_identity_cert_<date> ) Modify the code to fetch and use the new certificate Revoke the old certificate in Apple’s developer portal

Payment Processing Certificate

Because only one certificate could be active at a time, we needed to figure out how to instantly switch from the old certificate to the new certificate as soon as the new certificate was functional. To do this, we built a function that accepts primary and secondary certificates. If the primary certificate fails, we fall back to the secondary certificate.

Using decryptDataWithFallback, we executed the rotation in a 5 step process:

Generate new Payment Processing certificate Securely generate the key pair and store the private key in AWS KMS or Hashicorp’s Vault under a new name (i.e. payment_processing_cert_<date> ) Modify the code to use the new certificate as the primary certificate and the old certificate as secondary. Apple did not provide a time estimate for the delay between activating the new certificate and being able to process transactions with the new certificate. During the rotations, we found the delay to be around 60 minutes; the old certificate needs to be accessible for the duration of the transition period. Once we see that our logic is correctly falling back to the secondary certificate, activate the new certificate from Apple’s developer portal (this automatically revokes the old certificate). Once we see that the logic is successfully using the primary certificate and never falling back to the secondary, remove the old certificate as secondary.

Outcome

In the month leading up to Black Friday, we were able to rotate the Merchant Identity Certificate and Payment Processing Certificate across all of our environments without any downtime. Furthermore, we built a reusable framework that will facilitate a much quicker rotation when these certificates expire again in two years.

Key Takeaways

As mentioned above, we found that the amount of time it takes for a new certificate to start working is around an hour. Because the switch isn’t immediate or exact, having a key rotation strategy proved to be even more important. Without our rotation framework, we would have been looking at around an hour of downtime across all merchants using Apple Pay.

Payment Processing certificate activation notification