RCA for about.gitlab.com 503s for certificate expired

**Please note:** if the incident relates to sensitive data, or is security related consider
labeling this issue with ~security and mark it confidential.
***

## Summary

For approximately 20 minutes on 2018-11-16 unauthenticated requests to https://gitlab.com and all requests to https://about.gitlab.com were returning 503 errors because of an expired certificate. 


## Impact & Metrics

The following services were impacted:

* https://about.gitlab.com
* unauthenticated requests to https://gitlab.com (because of the redirect to about.gitlab.com)


The following services remained unaffected during the outage:

* authenticated logins to https://gitlab.com
* pages, registry, api, all gitlab.com services


## Detection & Response

Start with the following:

- How was the incident detected?
- Did alarming work as expected?
- How long did it take from the start of the incident to its detection?
- How long did it take from detection to remediation?
- Were there any issues with the response to the incident? (i.e. bastion host used to access the service was not available, relevant team memeber wasn't page-able, ...)

## Timeline

2018-11-16

- 00:03 UTC - we started seeing expired certificate errors for about.gitlab.com
- 00:06 UTC - curl for `about.gitlab.com` cert reveals that it's good
- 00:08 UTC - curl for `about-src.gitlab.com` cert reveals that it's expired at 00:00 2018-11-16
- 00:10 UTC - started purchase process for cert renewal of `about-src.gitlab.com`
- 00:15 UTC - update chef vault for `about-gitlab-com _default` with new certs
- 00:18 UTC - manually run chef-client on `about-src.gitlab.com` to roll in new certificate
- 00:21 UTC - `about-src.gitlab.com` serving new cert now
- 00:25 UTC - modify Fastly configuration for Origin validation to accept `about-src.gitlab.com` as TLS name
- 00:27 UTC - Roll out Fastly configuration for Origin SSL TLS name change
- 00:28 UTC - about.gitlab.com is operating normally.


## Root Cause Analysis

The certificate on the origin server for `about.gitlab.com` had expired causing the CDN provider to throw errors. We didn't realize the cert was expiring or about to expire because when we moved the production site `about.gitlab.com` to be fully CDN resolved we never re-branded the origin server, in stead leaving the old certificate on the server while the CDN provider generated a new one. This then was further compounded by the fact that our certificate monitoring in prometheus for advanced alerting was never reconfigured either, so it was still looking at `about.gitlab.com` rather than the origin server of `about-src.gitlab.com` for cert expiration alerting.


## What went well

- Alerting triggered immediately in both Slack ad PagerDuty that there was an issue

## What can be improved

- We need a more automated means to iterate over the domains that we have certs on and monitor when they're going to expire.


## Corrective actions


## Guidelines

* [Blameless Postmortems Guideline](https://about.gitlab.com/handbook/infrastructure/#postmortems)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 2.0