Scheduled maintenance for CustomersDot on 2022-11-13 03:00 UTC

Zuora Billing will undergo planned maintenance on 2022-11-13 from 03:00 UTC to 04:00 UTC.  This means CustomersDot (CDot) should be in maintenance mode during this window to avoid any errors for customers.  CDot can automatically put itself into maintenance mode based on health checks (which include Zuora) but we should still have someone on standby to ensure that CDot goes into and out of maintenance mode as expected.

## Maintenance window

Sunday 2022-11-13 03:00 - 04:00 UTC

- https://gitlab.com/gitlab-com/business-technology/enterprise-apps/financeops/finance-systems/-/issues/952+
- See [official release](https://community.zuora.com/communities/community-home/digestviewer/viewthread?GroupId=475&MessageKey=ec7a72cc-934e-45ae-8987-aa0961991be0&CommunityKey=c91d3142-7e65-4715-9e11-7eb5c768c213&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3dc91d3142-7e65-4715-9e11-7eb5c768c213) from Zuora regarding this episode of planned maintenance**

## Process

Toggle `maintenance_mode` in https://gitlab.com/gitlab-org/customers-gitlab-com/-/feature_flags/111/edit

[ASSIGNED-USERNAME] - On Stand-by to monitor maintenance mode activities and jump in to assist as needed.

1. [x] Created scheduled maintenance issue in https://gitlab.com/gitlab-com/gl-infra/production for the CMOC to update the GitLab status page
    - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7929
1. [ ] Set maintenance mode in CustomersDot at 2022-11-13 03:00 UTC
    - This step could be done automatically by CDot.  An engineer could just check that maintenance mode was enabled after Zuora health checks started failing.
1. [ ] Test the maintenance page is working at 2022-11-13 03:00 UTC
1. [ ] Quiet Sidekiq process from [Sidekiq Admin UI](https://customers.gitlab.com/sidekiq/busy) at 7PM PT
    -  This will prevent Sidekiq from processing new queued jobs but jobs will continue to enqueue.
1. [ ] [Ensure the CMOC](https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/#how-to-engage-the-cmoc) has updated status.gitlab.com
1. [ ] Remove maintenance mode in CustomersDot at 2022-11-13 04:00 UTC
    -  This step could be done automatically by CDot.  An engineer could just check that maintenance mode was disabled after Zuora health checks began passing again.
1. [ ] Test the maintenance page is gone at 2022-11-13 04:00 UTC
1. [ ] Manually restart Sidekiq at 2022-11-13 04:00 UTC
    - Connect to CDot production server and run `sudo supervisorctl restart customers-sidekiq` as [documented here](https://gitlab.com/gitlab-org/customers-gitlab-com#deployments)
1. [ ] [Ensure the CMOC](https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/#how-to-engage-the-cmoc) has updated status.gitlab.com
1. [ ] Manually enqueue scheduled jobs from [the Recurring Jobs tab](https://customers.gitlab.com/sidekiq/recurring-jobs) missed during maintenance window
    - These were likely the jobs missed based on the planned outage window: 
      - `ExpireOrderCronJob`
      - `BillingRateUpdateCronJob`
      - `ReconciliationCronJob`
      - `UpcomingReconciliationNotificationCronJob`
      - `AutoRenewCronJob`

Story points: 0.0