Remove CI infrastructure from DigitalOcean

DO has served us well for first years of having autoscaled Shared Runners on GitLab.com. But with our transition to GCP, connected with transition of CI infrastructure also there, it became a backup environment.

After migrating the CI infrastructure to GCP we've added several updates to increase the efficiency of that environment and to make the configuration simpler. So at this moment configuration of the CI infrastructure in DO is not the same as in GCP.

During last year since the transition, DO environment was used twice - both times during some problems with GCP environment. And both times unsuccessful. The bigger load that we've got since migrating to GCP and the different configuration of the infrastructure made DO environment in current shape to be no more usable for GitLab.com CI infrastructure. At this moment we get a lot of alerting noise (e.g. related to cache server) from an infrastructure part that is totally not used.

We still should have some backup strategy. But we should rather think about how we can quickly (re)create the CI infrastructure in another region in GCP (e.g. with the idea like #4813).

With all of this said, I think it's time to send our CI infrastructure in DO for a well-deserved retirement.

---

What should be done for DigitalOcean CI infrastructure termination:

- [x] Cleanup Prometheus configuration:
    - [x] `prometheus-server` role, `node` job, `public_hosts` list:
        - [x] remove `runners-cache-5.gitlab.com` 
    - [x] `prometheus-server` role, `prometheus` job, `public_hosts` list:
        - [x] remove `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`

    - [x] `gprd-infra-prometheus-server.json` role, `ci-node` job, `role_name` list:
        - [x] remove `gitlab-runners-consul`
        - [x] remove `runners-cache-server`
    - [x] `gprd-infra-prometheus-server.json` role, `ci-node` job, `public_hosts` list:
        - [x] remove `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
    - [x] `gprd-infra-prometheus-server.json` role, `blackbox` job, `static_configs->targets` list:
        - [x] remove `http://runners-cache-5.gitlab.com/minio/login`
        - [x] remove `http://runners-cache-5.gitlab.com:1443/v2`
        - [x] remove `http://runners-cache-5.gitlab.com:5000/v2`
        - [x] remove `http://runners-cache-5.gitlab.com:9000/minio/login`
        - [x] remote `https://prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
    - [x] `gprd-infra-prometheus-server.json` role, `shared-runners` job, `static_configs->targets` list:
        - [x] remove `shared-runners-manager-1.gitlab.com:9402`
        - [x] remove `shared-runners-manager-2.gitlab.com:9402`
    - [x] `gprd-infra-prometheus-server.json` role, `shared-runners-gitlab-org` job, `static_configs->targets` list:
        - [x] remove `gitlab-shared-runners-manager-1.gitlab.com:9402`
        - [x] remove `gitlab-shared-runners-manager-2.gitlab.com:9402`
    - [x] `gprd-infra-prometheus-server.json` role, `private-runners` job, `static_configs->targets` list:
        - [x] remove `private-runners-manager-1.gitlab.com:9402`
        - [x] remove `private-runners-manager-2.gitlab.com:9402`
    - [x] `gprd-infra-prometheus-server.json` role, `prometheus` job, `public_hosts` list:
        - [x] remove `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
    - [x] `gprd-infra-prometheus-server.json` role, `ci-prometheus-fleet` job, `static_configs->targets` list:
        - [x] remove `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
    - [x] `gprd-infra-prometheus-server.json` role
        - [x] remove `digitalocean_gitlab_bv` job
        - [x] remove `digitalocean_gitlab_ci` job
        - [x] remove `digitalocean_hanging_droplets_cleaner` job
        - [x] remove `digitalocean_droplet_zero_machines_cleaner` job
        - [x] remove `runners-cache-registry` job
        - [x] remove `runners-cache-server` job
        - [x] remove `runners-cache-minio` job

    - [x] `ops-infra-prometheus-server` role, `node` job, `public_hosts` list:
        - [x] remove `shared-runners-manager-1.gitlab.com`
        - [x] remove `shared-runners-manager-2.gitlab.com`
        - [x] remove `gitlab-shared-runners-manager-1.gitlab.com`
        - [x] remove `gitlab-shared-runners-manager-2.gitlab.com`
        - [x] remove `private-runners-manager-1.gitlab.com`
        - [x] remove `private-runners-manager-2.gitlab.com`
        - [x] remove `runners-cache-5.gitlab.com`
        - [x] remove `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
        - [x] remove `consul-01.nyc1.do.gitlab-runners.gitlab.net`
        - [x] remove `consul-02.nyc1.do.gitlab-runners.gitlab.net`
        - [x] remove `consul-03.nyc1.do.gitlab-runners.gitlab.net`
    - [x] `ops-infra-prometheus-server` role, `prometheus` job, `public_hosts` list:
        - [x] remove `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
    - [x] `ops-infra-prometheus-server` role, `thanos` job, `public_hosts` list:
        - [x] remove `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
- [x] Chef cleanup:
    - [x] remove nodes:
        - [x] `shared-runners-manager-1.gitlab.com`
        - [x] `shared-runners-manager-2.gitlab.com`
        - [x] `gitlab-shared-runners-manager-1.gitlab.com`
        - [x] `gitlab-shared-runners-manager-2.gitlab.com`
        - [x] `private-runners-manager-1.gitlab.com`
        - [x] `private-runners-manager-2.gitlab.com`
        - [x] `runners-cache-5.gitlab.com`
        - [x] `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
        - [x] `consul-01.nyc1.do.gitlab-runners.gitlab.net`
        - [x] `consul-02.nyc1.do.gitlab-runners.gitlab.net`
        - [x] `consul-03.nyc1.do.gitlab-runners.gitlab.net`
    - [x] remove roles:
        - [x] `gitlab-runner-srm-do`
        - [x] `gitlab-runner-gsrm-do`
        - [x] `gitlab-runner-prm-do`
        - [x] `runners-cache-5-gitlab-com`
        - [x] `runners-cache-server`
        - [x] `prometheus-blackbox-runners-cache`
        - [x] `gitlab-runners-prometheus-do-nyc1`
        - [x] `gitlab-runners-consul`
        - [x] `gitlab-runners-consul-firewall`
        - [x] `gitlab-runners-consul-do-nyc1`
    - [x] remove chef vaults:
        - [x] `gitlab-runner-srm-do ci-prd`
        - [x] `gitlab-runner-gsrm-do ci-prd`
        - [x] `gitlab-runner-prm-do ci-prd`
        - [x] `runners-cache-5-gitlab-com ci-prd`
        - [x] `gitlab-runners-prometheus-do-nyc1 ci-prd`
        - [x] `gitlab-runners-consul client`
        - [x] `gitlab-runners-consul cluster`
- [x] unregister Runners:
    - [x] dev.gitlab.org:
        - [x] Shared Runners:
            - [x] `private-runners-manager-1.gitlab.com` (ID: 75)
            - [x] `private-runners-manager-2.gitlab.com` (ID: 96)
            - [x] `gitlab-shared-runners-manager-1.gitlab.com` (ID: 110)
            - [x] `gitlab-shared-runners-manager-2.gitlab.com` (ID: 111)
    - [x] GitLab.com:
        - [x] Shared Runners:
            - [x] `shared-runners-manager-1.gitlab.com` (ID: 40786)
            - [x] `shared-runners-manager-2.gitlab.com` (ID: 40788)
            - [x] `gitlab-shared-runners-manager-1.gitlab.com` (ID: 37398)
            - [x] `gitlab-shared-runners-manager-2.gitlab.com` (ID: 37397)
        - [x] `gitlab-org` Group Runners:
            - [x] `private-runners-manager-1.gitlab.com` (ID: 395501)
            - [x] `private-runners-manager-2.gitlab.com` (ID: 395502)
        - [x] `gitlab-com` Group Runners:
            - [x] `private-runners-manager-1.gitlab.com` (ID: 395497)
            - [x] `private-runners-manager-2.gitlab.com` (ID: 395498)
        - [x] `charts` Group Runners:
            - [x] `private-runners-manager-1.gitlab.com` (ID: 396095)
            - [x] `private-runners-manager-2.gitlab.com` (ID: 396096)
- [x] DigitalOcean resources cleanup:
    - [x] remove nodes:
        - [x] from `GitLab Prod` team:
            - [x] `shared-runners-manager-1.gitlab.com`
            - [x] `shared-runners-manager-2.gitlab.com`
            - [x] `gitlab-shared-runners-manager-1.gitlab.com`
            - [x] `gitlab-shared-runners-manager-2.gitlab.com`
            - [x] `private-runners-manager-1.gitlab.com`
            - [x] `private-runners-manager-2.gitlab.com`
            - [x] `runners-cache-5.gitlab.net` + volumes
            - [x] `prometheus-01.nyc1.do.gitlab-runners.gitlab.net` + volumes
            - [x] `consul-01.nyc1.do.gitlab-runners.gitlab.net`
            - [x] `consul-02.nyc1.do.gitlab-runners.gitlab.net`
            - [x] `consul-03.nyc1.do.gitlab-runners.gitlab.net`
    - [x] cleanup runner base images:
        - [x] in `GitLab B.V.` team
        - [x] in `GitLab CI` team
- [x] update our [base image building configuration](https://dev.gitlab.org/cookbooks/packer-runner-machines/) to stop using DO
- [x] other:
    - [x] Grafana cleanup:
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m&fullscreen&panelId=62 - remove panel
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m&fullscreen&panelId=65 - remove panel
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m&fullscreen&panelId=68 - remove panel
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m&fullscreen&panelId=61 - remove panel
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m - remove whole `Cache servers` row
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m - remove whole `Hanging droplets cleaner` row
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m - remove whole `Droplet zero machines cleaner` row
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m - remove `Cache Server` variable
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m - remove `Hanging doplets cleaner` variable
        - [x] https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m - remove `Droplet zero machines cleaner` variable
        - [x] https://dashboards.gitlab.net/d/sXVh89Imk/ci-autoscaling-providers?orgId=1&refresh=5m - remove whole `DigitalOcean` row

Story points: 5.0