giseldo/neodataset huggingface dataset:
Automate postgres checkup report

Let's automate running https://gitlab.com/postgres-ai-team/postgres-checkup and produce a new checkup report every week (?). The result can be pasted into an issue in the infrastructure queue and reviewed from there.

[An example report](https://gitlab.com/gitlab-com/gl-infra/production/snippets/1845548#postgres-checkup_K003) led to lots of great insights, e.g. in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6590 et al.

Story points: 2.0
Update some credentials for https://customers.stg.gitlab.com

We need to update the [following fields](https://gitlab.com/gitlab-cookbooks/cookbook-customers-gitlab-com/blob/master/attributes/default.rb#L18-19) on the Chef vault:

`zuora_api_user` and `zuora_api_password`

The values are stored on the `Subscription Portal` vault that has been shared with the Infrastructure team.

Story points: 1.0
Switch to hourly rotation for some Elastic indices

Some of our elastic indices are very big - up to a 1TB in size.

This may be causing some of the problems we're seeing with our Elastic cluster.

In order to fix the cluster, @ahmadsherif @jarv @mwasilewski\-gitlab and I discussed solutions, and switching over to hourly logging seems to be fairly straight-forward approach which may help.

We aim to do this for the following logs:

* `rails`
* `gitaly`
* `workhorse`

The `nginx` logs are also quite large, but we intend to stop sending these to ElasticSearch: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6542 (and only storing them in GCS)


**Note** changes may need to be made to the retention tools in https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/esc-tools/

Story points: 3.0
Rebuild invalid index: projects_mirror_user_id_idx

Index `projects_mirror_user_id_idx` on the `projects` table is invalid:
```
    "projects_mirror_user_id_idx" btree (mirror_user_id) INVALID
```
as detected by recent checkup: 

It was not so a week ago (the report made 6 days ago [shows only one invalid index](https://gitlab.com/gitlab-com/gl-infra/production/snippets/1845548#postgres-checkup_H001), another one, being expected since at that time pg_repack was operating -- it uses names `index_**` when rebuilding indexes).

An open question is what caused it.

Action: rebuild it. Perhaps pg_repack is a good option for this.

/cc @abrandl

Story points: 1.0
open port 9091 for ops prometheus server

I was trying to setup pushgateway on chef.gitlab.com, license.gitlab.com, version.gitlab.com, and customers.gitlab.com  
Ops prometheus server (prometheus-01-inf-ops.c.gitlab-ops.internal) will be scraping them. Currently, ops prometheus server is not able to connect to the 9091 port of those servers. 

Anyone can help open up the port for ops Prometheus server?

Story points: 2.0
Apply GCP sizing recommendations/downsize nodes in DR

In looking at GCP sizing recommendations in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6239, it appears that prod may be difficult, but we do have a similar potential for savings in DR and in the case of the file-nn-stor nodes, we have already moved the persistent disks to standard so the cpus should be able to scale down.

Recommendation is to scale down file-nn nodes to n1-standard-16 from 32 and see how things perform as we turn on Geo.  We may be able to downsize further after Geo has been running and we see more recommendations.  This will initially save us 32 * 32 = 1024 vcpu per month.

cc @glopezfernandez @andrewn @devin.  Assigning devin given that he has been working DR and Geo items.

Story points: 2.0
Turn of SSL compression on PostgreSQL ZFS replication slot

The #database channel is getting alerted to a replication slot that is currently using SSL compression:

```sql
gitlabhq_production=# select * from pg_stat_ssl where compression = 't';
-[ RECORD 1 ]----------------------------
pid         | 4391
ssl         | t
version     | TLSv1.2
cipher      | ECDHE-RSA-AES256-GCM-SHA384
bits        | 256
compression | t
clientdn    |

gitlabhq_production=# select * from pg_replication_slots;
-[ RECORD 1 ]-------+----------------
slot_name           | postgres_zfs_01
plugin              |
slot_type           | physical
datoid              |
database            |
active              | t
active_pid          | 4391
xmin                | 3997015837
catalog_xmin        |
restart_lsn         | D869/D000000
confirmed_flush_lsn |
```

Story points: 1.0
Daily runner usage

Extracted from : https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6360#note_157441581

if we could get another report into CI, which gives us daily runner usage (aggregated on total CI runner minutes per day) by the following categories?

1. `gitlab`: GitLab and GitLab forks (this could be as simple as checking for "gitlab-ce"/"gitlab-ee" in the project path)
1. `free-public`: shared runners for non-paid public projects
1. `free-nonpublic`: shared runners for non-paid non-public projects
1. `paid`: shared runners for non-free projects
1. `private-runners`: non-shared runner projects

Story points: 2.0
Databases reviews

* [x] @abrandl https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/866
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10676
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26490
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10756

Story points: 1.0
Consider increasing Runner timeouts for gitlab-shared-runners-manager-X.gitlab.com runners to 90 minutes

Currently it seems that the runners from `gitlab-shared-runners-manager-X.gitlab.com` are using a 60 minutes timeout which can lead to rspec jobs to timeout even if all the tests actually passed (e.g. https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/193827374, https://gitlab.com/gitlab-org/gitlab-ee/-/jobs/193827346). We should consider increasing this timeout to something like 90 minutes.

@tmaczukin What do you think?

/cc @meks @godfat

Story points: 1.0
setup an Elastic Cloud cluster for indexing in staging.gitlab.com

as per: https://gitlab.com/groups/gitlab-org/-/epics/853 we need to set up an ELK cluster to enable Elasticsearch integration on `staging.gitlab.com` for `gitlab-org` group

~~* [ ] extend our subscription in Elastic Cloud~~ turns out the subscription limit is per deployment so I could create the cluster
* [x] create a cluster for staging
* [x] connect it to `staging.gitlab.com` (limit to `gitlab-org`, do not enable searching for now, let indexing run first)
  * [x] check indexing is done, after ~12h:
```bash
$ sudo gitlab-rake gitlab:elastic:index_repositories_status
Indexing is 26.30% complete (811186/3084890 projects)
```
* [x] indexing is done
* [x] if needed, manually trigger indexing: https://docs.gitlab.com/ee/integration/elasticsearch.html#indexing-large-instances
* [x] enable searching with ELK on `staging.gitlab.com`
* [x] test resizing of the cluster

Story points: 16.0
Investigate 10TB Directory on share-01

In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6467 we began to investigate why the `share-01` server filled up so much and so quickly.

It was determined that the directory `/var/opt/gitlab/gitlab-rails/uploads/-/system/tmp` has 10.1TB of data in it and we need to figure out how it got there and decide if it can be deleted. Given that it is `tmp`, I would imagine we can just delete it, but we need to verify its purpose and why it got to this point before doing so.

Story points: 3.0
Blueprint for MachineResourceManager

We want to be able to upgrade OS and/or Kernel on our compute machines whenever we want and downgrade if we find issues. The traditional way of doing `apt-get` directly on our hosts is succeeded with a process like `unattended-upgrades`. However, we are taking it to the next level and going with base image creation process. The idea is that we will have a fully automated process that can build a new image with newer OS/kernel and can test it to make sure basic functionalities work without any issues. There is already work being done for this. 

What is needed then? We still need to come up with a process that will take the base image, do all the orchestration work of configuring it the way we want for a given service and replace an existing node with the new one and it should be able to do it for all of our hosts. 

This issue is to keep track of the blueprint for this initiative. It is called: "MachineResourceManager". In the work field, "Human Resource" is a function that deals with hiring, firing, providing career growth/development, promotion...etc for employees. When demand increases, HR hires more people temporarily. HR lets go off employees and onboards new employees. These are very similar to what we are trying to achieve with the work - the only difference is that it is for machines. We want to be able to replace machines, add some and remove some when needed.

Story points: 8.0
LoggingVisibilityDiminished Alerts

```
Firing 2 - LoggingVisibilityDiminished
PubSub messages are queuing up.  Unacked messages older than 60 seconds exist in the queue
for the last 5 minutes. This will lead to a loss of log data.


:desktop_computer: *Services:*

 **
 PubSub queuing high

 PubSub messages are queuing up.  Unacked messages older than 60 seconds exist in the queue
for the last 5 minutes. This will lead to a loss of log data.



 **
 PubSub queuing high

 PubSub messages are queuing up.  Unacked messages older than 60 seconds exist in the queue
for the last 5 minutes. This will lead to a loss of log data.


:label: *Labels*:

 *Alertname*: LoggingVisibilityDiminished
 *Channel*: production
 *Env*: gprd
 *Monitor*: default
 *Provider*: gcp
 *Region*: us-east
 *Severity*: warn
 *Subscription_id*: pubsub-workhorse-inf-gprd-sub
```

https://prometheus.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=sum%20by(subscription_id)%20(stackdriver_pubsub_subscription_pubsub_googleapis_com_subscription_oldest_unacked_message_age)%20%3E%2060&g0.tab=0

![Screen_Shot_2019-04-02_at_1.15.52_PM](/uploads/002e9829d311cb471a3fcda6fd4a2e72/Screen_Shot_2019-04-02_at_1.15.52_PM.png)

Story points: 1.0
Cleanup or fix obsolete alerts

The expressions backing the following alerts doesn't return any data, so either we need to fix them or remove them:

* [x] HighGitCatFileCount
* [x] BlackBoxGitPullHttps
* [x] BlackBoxGitPullSsh
* [x] BlackBoxGitPushHttps
* [x] BlackBoxGitPushSsh
* [ ] CICDTooManyRunningJobsPerNamespaceOnSharedRunners
* [ ] CICDRunnerMachineCreationRateHigh
* [x] CICDRunnersCacheDown
* [x] GitLabComLatencyWeb
* [x] GitLabComLatencyWebCritical
* [x] GitLabComLatencyAPI
* [x] GitLabComLatencyAPICritical
* [x] GitLabComLatencyGit
* [x] GitLabComLatencyGitCritical
* [x] GitlabComDown
* [x] WWWGitlabComDown
* [x] MonitorGitlabNetPrometheusDown
* [x] MonitorGitlabNetNotAccessible
* [x] OtherPrometheusDown
* [x] PostgreSQL_ReplicationStopped
* [x] PostgreSQL_ReplicaStaleXmin
* [x] PostgreSQL_DiskUtilizationMaxedOut
* [x] PostgreSQL_PGBouncer_maxclient_conn
* [x] SnitchHeartBeat
* [x] StagingGitlabComDown
* [x] StagingPostgresIO_High
* [x] FrontEndWorkerDown

Story points: 3.0
investigate the difference between redirect behaviour in nginx and Fastly

redirects were broken in three cases, they were fixed in the following MRs: [self-managed](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20865), [categories](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20867), [codefresh](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20868)

investigate why they behaved differently from nginx redirects

Story points: 2.0
Move Zoom sync script out of gitlab-server

Currently the zoom sync script is housed in the [gitlab-server cookbook](https://gitlab.com/gitlab-cookbooks/gitlab-server/blob/master/files/default/zoom_sync.rb). We need to create a repo for it and move it out of the cookbook and make it public. Ultimately we want to run it via CI/CD or similar and decommission the current `cron` server that runs it.

Story points: 1.0
Upload Nessus package to aptly.gitlab.com

Per: https://gitlab.com/gitlab-com/runbooks/blob/master/howto/aptly.md

Requesting to have the Nessus package added to our aptly repo.  

File is available here: https://drive.google.com/file/d/1gRJ3MbYIrHbxemvqKlnlU44eCzGPt1Kk/view?usp=sharing

Story points: 1.0
Upgrade Packagecloud to 2.0.6

packagecloud 2.0.6 has been released which should resolve an issue we are having with backups uploading to S3 by implementing multi-part upload. 

Time to upgrade again!

This should be basically no big deal, certainly in comparison to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6317 :smile:

Story points: 1.0
Analyze checkpoint frequency

~~We currently peak at 15 checkpoints per minute (per [Grafana](https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1&fullscreen&panelId=194&from=now-7d&to=now)).~~

~~The peak up to 45 opm was exceptional:~~

This was based on a wrong reading of the graph. The *primary* actually has low checkpoint frequency (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6477#note_159060272). 

![Screenshot_from_2019-03-28_17-54-07](/uploads/ea74601587d7e55114551a503f73a121/Screenshot_from_2019-03-28_17-54-07.png)

This strikes as too frequent checkpoints. We should increase `max_wal_size` in order to reduce the frequency.

Concerns to watch out for:
* Decreasing checkpoint frequency means an increase in recovery time. What do we deem tolerable here? We need to measure this too.
* Would be interesting to measure the IO impact a decrease in frequency has.

cc @NikolayS @cshobe @yguo

Story points: 1.0
Lower the repository size limit on GitLab.com to 10GB

As the final step on https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5160, let's:

* [x] Mail affected users of repositories above 10GB: https://gitlab.com/gitlab-com/marketing/digital-marketing-programs/issues/440
* [x] Adjust the limit to 10GB

Story points: 2.0
Many staging alerts still paging production

Staging alerts are paging production, even though staging is silenced.  This is happening because many alerts are tagged incorrectly.

This MR fixed things for one alert: https://gitlab.com/gitlab-com/runbooks/merge_requests/1010

The same thing needs to be done for all others.

We may need to come up with a better way to make sure that all alerts are tagged by default, now that they all get sent to the same alert manager.

Story points: 2.0
Monitor osqueryd

Depending on the osquery profile and the amount of server activity, osqueryd can consume a significant amount of system resources. When caching too many events locally, it even breaks the local db which leads to filling up the disk because old data files will not be cleaned up anymore (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6434).

We need to monitor and alert on data dir size and CPU usage.

Story points: 5.0
Database Reviews

* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26490
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26057#note_149890203
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9861
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10411 with https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26496
* [x] @yguo https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10331#note_152692635
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9815#note_152275676
* [ ] @fjsanpedro https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9634#note_154356948
* [x] @yguo https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26582#note_154264004
* [x] @fjsanpedro -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26540
* [x] @yguo, @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10057#note_154636936 along with https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25975#note_154636914
* [x] @fjsanpredro -> @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10446#note_154605666
* [ ] @fjsanpedro -> @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9634#note_154536921
* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25973#note_153842418 ~"Community Contribution" 
* [x] @NikolayS -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25432#note_153437855
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26146#note_153144338
* [x] @yguo -> @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10353#note_154967144
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26675
* [x] @abrandl https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/866#note_158632
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25973
* [x] @fjsanpedro -> @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10437#note_15637594510437
* [x] @abrandl https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2997#note_158700
* [x] @fjsanpedro -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26554
* [x] @NikolayS -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25299#note_155593259
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10552#2cda0c6171fa7d04989507a1dd112e34c40df46d
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26823#3c92371994fe8eb4b866e721926eab2c2be2f44d
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10446#note_157598053
* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10609
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26212
* [ ] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24512

Story points: 5.0
ChatOps access request for @jedwardsjones on ops.gitlab.net

## What

I'd like to be given chatops access so I can enable a feature flag on staging. I think this entails being added to a group on ops.gitlab.net

## Why

In the manage team we are using per-group feature flags for the first time, and I'd like to test both the features themselves and the process of enabling feature flags. In particular we've added the ability for chatops to enable features per group instead of per project, but I've been unable to verify that this actually works because I don't have access.

## From slack

https://gitlab.slack.com/archives/C101F3796/p1552617972681000
>>>
How do I get access to chatops? When I try `/chatops run feature list --match=saml --staging` I just get the error "Whoops! This action is not allowed. This incident will be reported."

A slack search (https://gitlab.slack.com/archives/C3JJET4Q6/p1547028248032500) suggests I should ask here and include my ops.gitlab.net username, `@jedwardsjones`.
>>>

Story points: 1.0
setup staging environment for about.gitlab.com

this came up as part of discussion in: https://gitlab.com/gitlab-com/www-gitlab-com/issues/3952

* [x] manually create a VM in Azure
* [x] manually create a DNS entry for the VM
* [x] register the VM with Chef
* [x] create a separate role called `about-staging-gitlab-com.json`, similar to `about-gitlab-com.json` and apply it
  * [ ] edit the `cookbook-about-gitlab-com` cookbook so that nginx config is generated from a template and the domain can be specified in the role, created a separate issue for this: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6503
  * [x] configure a CI job to deploy site from master
* [x] register the runner
* [x] create a service in Fastly
* [x] configure TLS certs, enforce TLS
* [x] create DNS entries (IPv4, IPv6) for Fastly about.staging.gitlab.com
* [x] configure `staging.gitlab.com` to use `about.staging.gitlab.com` for its redirects

Story points: 15.0
PullMirrorsOverdueQueueTooLarge in Staging

We are getting these errors going to PagerDuty.  They seem to be correct alerts this time, and they are going to the right place. 

It is not going to production this time like it was in: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5172

And it doesn't seem to be in error as it was in this: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5401

The threshold is 5K overdue updates, and the graph looks like this:

![Screen_Shot_2019-03-20_at_8.34.37_AM](/uploads/fbaf2649780c677b8f06a1724cd839cc/Screen_Shot_2019-03-20_at_8.34.37_AM.png)

[Dashboard link](https://dashboards.gitlab.net/d/_MKRXrSmk/pull-mirrors?refresh=30s&orgId=1&var-environment=gstg&var-prometheus=prometheus-01-inf-gstg&var-prometheus_app=prometheus-app-01-inf-gstg&from=1552847993728&to=1553107193728)

Story points: 1.0
restore project generated too many one time ssh keys

the pipeline failed with this error `ERROR: (gcloud.compute.scp) INVALID_ARGUMENT: Login profile size exceeds 32 KiB. Delete profile values to make additional space.` https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/-/jobs/373652

seems like there are lots of ssh key that were used once in the pipeline still exists. we can either clean up all the obsoleted sshkeys with 
```for i in $(gcloud compute os-login ssh-keys list | grep -v FINGERPRINT); do echo $i; gcloud compute os-login ssh-keys remove --key $i; done```

also, I think we should consider publish the scripts to a GCS bucket, and let the instance download the script from GCS bucket and run as startup script instead of `scp` and `ssh` into the instance to execute the script.

Story points: 2.0
Add Outreach to the CNAME for a branded URL

To help with deliverability of emails sent by Outreach, creating a branded URL has been suggested by our CSM 

Instructions to set up are here: [Outreach Support Article](https://support.outreach.io/hc/en-us/articles/115001092234-Setting-up-Branded-URLs)     

And **Step 1** is below to point the CNAME at:  
![Screen_Shot_2019-03-18_at_3.05.30_PM](/uploads/ef28569a3aed9b9b84a15a9936f2de1c/Screen_Shot_2019-03-18_at_3.05.30_PM.png)  


Branded URL is going to be `enable.gitlab.com`  


Please let me know if there are questions.

Story points: 1.0
Improve PostgreSQL configuration files

Presently the Chef recipes result in two files being placed in /var/opt/gitlab/postgresql:
1.  postgresql.base.conf - this appears to be more or less a default configuration file, from the version 9.2 (or earlier) distribution, as it contains commented parameters that were removed in later releases.  It also lacks many of the newer settings, and is full of comment noise and is frankly a mess that's difficult to read.
2.  postgresql.conf - this is Patroni's overrides to the base configuration, for variables that Patroni needs to be able to control.  It includes the previous file and then overrides/adds to that configuration.

There are also some effective settings that don't match either of these files or the default settings, which is confusing.

I would like the configuration files to be clear, correct for the current PostgreSQL version, have all settings explicitly defined, and be written to match the order in the documentation.  Chef templating should be able to handle this fairly easily.  Ideally our Chef recipes would not be hardcoded to use a specific version either, but instead set a default major version that can be overridden.

Story points: 3.0