giseldo/neodataset huggingface dataset:
Inventory Catalogue - Load Balancer (FE)

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Load Balancer (FE)

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - InfluxDB

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
InfluxDB

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Create new gitaly storage shard node to replace `nfs-file46`

Gitaly storage shard `nfs-file46` (`file-46-stor-gprd.c.gitlab-production.internal`) is currently at `62.27%` usage.

A new gitaly node should be created, and added to the list of shards configured to be included in consideration for storing new project repositories.

That way, `nfs-file46` can be removed from rotation without any concern that the node's removal from the configuration will put any additional burden on the remaining nodes.

It is important to avoid acceleration of usage growth on the remaining nodes accepting new repositories.

Story points: 3.0
Create new gitaly storage shard node to replace `nfs-file43`

Gitaly storage shard `nfs-file43` (`file-43-stor-gprd.c.gitlab-production.internal`) is currently at `60.38%` usage.

A new gitaly node should be created, and added to the list of shards configured to be included in consideration for storing new project repositories.

That way, `nfs-file43` can be removed from rotation without any concern that the node's removal from the configuration will put any additional burden on the remaining nodes.

It is important to avoid acceleration of usage growth on the remaining nodes accepting new repositories.

Story points: 2.0
Resolve redis config discrepancy for redis-cache between gstg and gprd

Apart from credentials, the only discrepancy between the production and staging config for redis-cache nodes is the disk persistence options.  The redis-cache nodes in the staging environment enabled periodically writing an RDB dump, whereas production does not.  Since this dataset is meant to be ephemeral and gets persisted to disk anyway on clean shutdown, this may be a reasonable setting.  We can re-enable it later if we decide it's too expensive to repopulate this cache from empty, but anecdotally several people have mentioned that they think it's tolerable.  For now, let's just get the staging config to match production.

For reference, here's the discrepancy we're going to resolve:

```shell
msmiley@saoirse:~$ ssh redis-cache-01-db-gprd.c.gitlab-production.internal -- sudo cat /var/opt/gitlab/redis/redis.conf | egrep -v '^ *($|#)' | perl -pe 's/(masterauth|requirepass) .*/$1 **REDACTED**/' > /tmp/redis.conf.gprd

msmiley@saoirse:~$ ssh redis-cache-01-db-gstg.c.gitlab-staging-1.internal -- sudo cat /var/opt/gitlab/redis/redis.conf | egrep -v '^ *($|#)' | perl -pe 's/(masterauth|requirepass) .*/$1 **REDACTED**/' > /tmp/redis.conf.gstg

msmiley@saoirse:~$ diff -U0 /tmp/redis.conf.{gstg,gprd}
--- /tmp/redis.conf.gstg	2020-03-27 10:53:49.830460831 -0700
+++ /tmp/redis.conf.gprd	2020-03-27 10:53:26.821766688 -0700
@@ -12,3 +11,0 @@
-save 900 1
-save 300 10
-save 60 10000
```

Story points: 1.0
Configure WAL-G's "wal-push" and "backup-push" to test Postgres backups creation with WAL-G on production

In https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8903, we started to test WAL-G for both the creation and use of backups on **staging**.

After 3 weeks of successful testing (see `Daily staging restore using WAL-G, from WAL-G staging archive` in https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/pipeline_schedules), it is time to start testing on a production replica.

1. [x] Decide how replica will be chosen and how it will be configured using Chef (note that patroni-01 is the master in gprd right now). Such replica is to be marked as unavailable for failover.
1. [x] In addition to WAL-E, install WAL-G 0.2.14 to all Postgres instances (how it can be done: https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/blob/master/bootstrap.sh#L84). It is okay to have both WAL-E and WAL-G installed.
1. [x] Configure WAL-G. Use some new GCS bucket, different from the existing ones.
1. [ ] on the chosen replica in `gprd`, put WAL-G's `wal-push` to `archive_command`, and set `archive_mode` to `'always'`.
1. [ ] Ensure that wal-push log does not have errors, but shows successful WAL-pushing and the new GCS bucket/folder is being filled with WALs.
1. [ ] Configure daily `backup-push` on the same replica (a cronjob similar to WAL-E's `backup-push` cronjob on the master), to have full backups daily.

Additional TODOs:

- [ ] https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitlab-wale-backups.yml#L24
- [ ] adjust runbooks

Story points: 8.0
Incorrect runbook link for alert `service_apdex_slo_out_of_bounds_lower_15m`

Link for `runbook` ("`troubleshooting/service-ci-runners.md`") on `service_apdex_slo_out_of_bounds_lower_15m` alert is incorrect.

Correct link is: https://ops.gitlab.net/gitlab-com/runbooks/-/blob/master/docs/ci-runners/service-ci-runners.md

Story points: 1.0
Terraform artifacts too large in CI

CI jobs for terraform are failing with `413 Request Entity Too Large` errors.

https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/jobs/1029505

```
 environments/gstg/.terraform: found 3365 matching files 
 ERROR: Uploading artifacts to coordinator... too large archive  id=1029505 responseStatus=413 Request Entity Too Large status=413 Request Entity Too Large token=bjnHYt3n
 FATAL: too large                                   
 ERROR: Job failed: exit code 1
```

Story points: 3.0
Execute PostgreSQL upgrade on staging

Using the template: https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/issues/10

We would like to execute the first test migration for the PostgreSQL upgrade. The date would be at 31 of March of 2020 12:30 UTC

The maintenance time for staging should be from 30 to 40 minutes.

Please upgrade 3 nodes, we will keep 3 secondaries hosts for the rollback.

Story points: 2.0
Redis PCAP files in /tmp

@gitlab-com/gl-infra 

There are many pcap files that are quite large on `redis-cache-01-db-gprd.c.gitlab-production.internal` in the `/tmp` directory. This has put the filespace free down to 6% or so on the root filesystem.

If you need any of these files for posterity, please move them and delete them. And if you made some and don't need them, consider deleting them ASAP.

Thank you!

Story points: 1.0
create the rollback script for the postgresql upgrade

We need to script the rollback of the Postgresql upgrade if we need to use it.

We would consider using GCP snapshots for it.

Also, we would keep 2 database nodes that are read-only that we will not execute the upgrade. With those nodes, we could restore the environment quickly.

Please consider that we will be in a scenario after the upgrade script described in the file: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9277

I would like share with you the steps for the rollback script:

At the begining of the rollback we will have:

* A snapshot consistent from the database before the upgrade.
* Excluded instances group - 4 nodes not upgraded. If we are successful with the upgrade, we would not upgrade them, reducing the cluster number of nodes.

At this point we are ready to run the rollback upgrade.

- Run Chef `merge_and_apply` to rollback the `patroni.yml` file.
- Stop services on failed instances like `chef-client`, `pgbouncer`, `patroni`, and `postgresql` (check manually).
- On failed instances change config to false values and avoid start service accidentally.
- Stop `pgbouncer` on "excluded instances" to avoid connections.
- Start `chef-client` services on "excluded instances".
- Resume patroni service --wait
- Check if patroni elect a new leader.
- Start `pgbouncer` 
- [continue...]

Story points: 4.0
New storage setting for Terraform state

In https://gitlab.com/groups/gitlab-org/-/epics/2673, the Configure:System group is creating a new feature to store Terraform state in GitLab. This involves adding a new storage setting, since Terraform state doesn't correlate to any of the other settings.

https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26619 introduces an uploader that uses this new storage setting. It also adds default values for this setting.

https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3983 adds the storage setting to Omnibus.

https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/1208 adds the storage setting to GitLab Helm Chart.

- [x] Add buckets for the feature in our module https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/storage-buckets/-/merge_requests/23
- [x] Setup buckets on gstg https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1608
- [x] Enable feature on gstg https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3163
- [x] Setup buckets on gprd https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1610
- [x] Enable feature on gprd https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3164

/cc @nicholasklick @mattkasa

Story points: 2.0
DNS record for `*.asaba-sandbox.gitlap.com`

Please add the following A record to DNS:

```
    "*.asaba-sandbox.gitlap.com.": {
      "records": [
         "35.208.230.243"
       ],
      "ttl": 300
    }
```

Related to gitlab-com/gl-security/engineering#884.

@dawsmith @AnthonySandoval for assignment.

Story points: 1.0
Execute database benchmarking on Postgresql 11

The goal is to check our load on PostgreSQL 11 and verify that the performance is similar or better. We could find some queries that the performance could change with the new planner, so we would like to reduce these cases.

We would execute the benchmark with the same concept that we have planned to test our primary database.

The main steps will be:

- List the queries from pg_stat_statements

- Execute a JMeter from a different host to reproduce the queries against the database, to generate similar traffic as production.

Story points: 4.0
Update customers.gitlab.com cheat sheet to prevent outages

Originating incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1763
Incident review: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9488

Update the page in the dotcom documentation to reflect running the console as the `gitlab-customers` user and not as root.

The location that needs updated: https://gitlab.com/gitlab-com/support/dotcom/dotcom-internal/-/wikis/Customers-Console-Rails-Cheat-Sheet
A source showing the proper commands: https://gitlab.com/gitlab-org/customers-gitlab-com/#accessing-production-as-an-admin-and-logs-and-console

Story points: 1.0
change requests to improve checkpoint setup

As mentioned in the issues : 
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5189

and 

https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2848

We would like to reconsider checkpoint-related PostgreSQL settings:

- `max_wal_size`: `5GB` → `64GB` for less frequent "forced" checkpoints;
- `checkpoint_timeout`: `5min` → `10min` for less frequent "planned" checkpoints;
- `checkpoint_completion_target`: `0.7` → `0.9` for smoother checkpointer
behavior.

And we would need to create the following changes, with 1-2 days of observations between them, on each step performing the change only on staging, and then on production:

1. change `checkpoint_timeout` from `5min` to `10min` and change `checkpoint_completion_target` from `0.7` to `0.9`.
   1. apply to gstg
   1. apply to gprd
1. change `max_wal_size` from `5GB` to `8GB`
   1. apply to gstg
   1. apply to gprd
1. change `max_wal_size` from `8GB` to `16GB`
   1. apply to gstg
   1. apply to gprd
1. change `max_wal_size` from `16GB` to `32GB`
   1. apply to gstg
   1. apply to gprd
1. change `max_wal_size` from `32GB` to `64GB`
   1. apply to gstg
   1. apply to gprd

For the 1st step and staging @NikolayS created: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2871

Story points: 2.0
Implement elasticsearch operational logging for utility scripts like `storage_rebalance.rb` and `storage_revert.rb`

Implement elasticsearch operational logging for utility scripts like `storage_rebalance.rb` and `storage_revert.rb`.

Got the idea for this one from @cmiskell.

> Just wondering if you're aware of SemanticLogger (https://github.com/rocketjob/semantic_logger)?  Delivery use this by preference, and I did also in https://gitlab.com/gitlab-com/gl-infra/infra-vault/-/tree/master/backup_job when prompted.   It does a lot of handy things, including logging to ElasticSearch if desired.  Not critical, but might be nice.

https://gitlab.com/gitlab-com/runbooks/-/merge_requests/1935#note_294926778

Story points: 3.0
Re-factor the information runbook utility commands into a bash function, and support an `--include-wiki` cli argument flag

Re-factor the information commands into a bash function, and support an `--include-wiki` cli argument flag.

This will mean that the `print_info` function will be invoked once for the main `.git` repo directory, and then again for the `.wiki.git` directory when the `--include-wiki` flag is given.

This is dependent on implementation of proper options parsing, which is tracked by this issue: https://gitlab.com/gitlab-com/runbooks/-/issues/35

Story points: 1.0
Update Disaster Recovery page in Handbook to reflect current state

While [answering some customer questions](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9402), we noticed at least one piece of information is out of data on the [Handbook page describing our DR posture](https://about.gitlab.com/handbook/engineering/infrastructure/library/disaster-recovery/#repository-data).

This task is to review that document and update it where needed with regard to current state of recoverability.

Rendered: https://about.gitlab.com/handbook/engineering/infrastructure/library/disaster-recovery/

Source: https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/source/handbook/engineering/infrastructure/library/disaster-recovery/index.html.md

Story points: 2.0
Update/create runbooks for increase/reduce disks for the patroni cluster

After increasing the disks from the patroni cluster we observed a restart from postgresql on the node Patroni-07.

To increase the security from the platform, we would like to update/create our runbooks to stop the traffic on the node. 

I would suggest adding the tag on the node to drain the traffic:

```
noloadbalance: true
```

Story points: 2.0
Regular and predictable database latency spikes are leading to web latency slowdowns (or "the {00,04,08,12,16,20}:05 spike")

Spun out of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9248#note_294148937

Within ~1 minute of the following times, on a daily basis, we experience a slowdown on our web (and possibly other services)

* `00:05`
* `04:05`
* `08:05`
* `12:05`
* `16:05`
* `20:05`

# p99.99 `db` durations over 24 hours

![image](/uploads/0b763fd835f60e788df09e3d3d5cc422/image.png)

https://log.gprd.gitlab.net/goto/6f240764af89c33cbaf08ff91d92fe53

# p99.9 web duration over 3 minutes during a spike

![image](/uploads/10bf79d5ea3db922bcf780c320e305fa/image.png)

https://log.gprd.gitlab.net/goto/c414837b2116e33e0809ec871bafa946

cc @Finotto

Story points: 4.0
Postgresql Migration test on staging

In this issue, I would like to plan and list the steps that we will execute to generate and restore the staging environment.

Nowadays we have a cluster with Postgresql 9.6

We will need to execute the following steps : 
- Install PostgreSQL 11. ( chef )
- Setup the new instance ( Chef )
- Stop the traffic on PG 9.6
- Execute the migration ( the migration plan that will include the scripted steps on ansible ) - take a backup/dump from the database cluster 9.6 *BEFORE THE UPGRADE*
- Extra tests with QA
- Stop the traffic on PG 11
- After finish, the migration tests, remove PG11 and restore the database backup to the instance PG 9.6.
- restore the traffic to PG 9.6

Story points: 4.0
Create process or feature to stop or control the impact of the background migrations on the database

We have faced some background migrations that generated load on the database cluster. We would like to investigate the possibilities to manage better this situation in the future.

The idea of this issue is to find out or improve/generate documentation about the background migrations and how to manage them.

Would be a goal to have the following features : 

- Be possible to stop the migration.
- Setup the intervals when the migrations are executed.
- Change the chunks of data that will be affected. 
- Pause the migration.

Story points: 8.0
Fix terraform warnings regarding deprecations

We've been updating our target terraform versions in several of our environments at https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/ for different reasons, and this has left behind a cumulus of warnings in the output of our terraform `plan`s. Some of them are quite simple to address, like removing quotes around data types in variable deffinitions (e.g. `"string"` => `string`), but others might require refactoring.

- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/generic-stor/-/merge_requests/20
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/generic-stor-redis/-/merge_requests/17
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/generic-stor-with-group/-/merge_requests/16
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/generic-sv-sidekiq/-/merge_requests/24
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/generic-sv-with-group/-/merge_requests/14
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/gke/-/merge_requests/26
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/monitoring-with-count/-/merge_requests/20
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/project/-/merge_requests/28
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/tcp-lb/-/merge_requests/8
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/pubsubbeat/-/merge_requests/20
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/cloudflare_workers/-/merge_requests/3
- [x] https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/static-objects-cache/-/merge_requests/21
- [x] https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1510

Story points: 3.0
New dedicated `gitlab-qa-mirror-runner` runner manager

As part of https://gitlab.com/gitlab-org/gitlab-qa/issues/261, we'd like to setup a dedicated `gitlab-qa-mirror-runner` runner manager for the https://gitlab.com/gitlab-org/gitlab-qa-mirror project.

The set up / specifications would be similar to the `omnibus-gitlab-mirror-runner` runner manager.

I originally opened a MR in the chef-repo, but then was pointed to Infrastructure as this would involve more steps than I anticipated: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2490#note_43048

The runner manager should be under the `GitLab QA Projects` GCP folder and probably in a new dedicated `quality-runners` project?

Any help is appreciated, thanks! :pray_tone2: 

/cc @gl-quality/eng-prod

Story points: 1.0
502 Bad Gateway error while testing purchase in staging customers portal

Amanda Rueda encounters a `502 Bad Gateway` while trying to complete a simple purchase test in the staging customers portal.

Occurs in all browsers.  Clearing cache has no effect.

Steps to reproduce:

Staging gitlab page: https://customers.stg.gitlab.com/plans

Then select the Bronze plan to purchase.

Then when the nginx returns, the url is: https://customers.stg.gitlab.com/subscriptions/new?plan_id=2c92c0f95a24621b015a259a50307881&transaction=create_subscription (edited) 

Customer ID for testing in the customers portal is `4013`.

Story points: 2.0
Migrate Gitlab's Production Project to `ops.gitlab.net`

To further efforts to dogfood incident management product features–and to remove our dependency on the installation (GitLab.com) that we're supporting–we will export the `production` project from gitlab.com and import it onto ops.gitlab.net.

A prerequisite for moving ops.gitlab.net (ops) will be to assess the production readiness of the installation. While we have no intent of publishing public issues or pages from ops.gitlab.net, the intent is for the ops installation to take over responsibility for more of our operational workflows (e.g. status page).

Additionally, adjustments will require substantial changes to the handbook instructing the Engineer On-Call to open production incidents on ops. We'll begin a campaign to inform the company of the switch, encouraging engineers and business stakeholders to verify or request access to ops. This will be necessary to view and contribute to ongoing incidents.

Lastly, we'll redirect Incident Management Automation workflows (&100) and all other automation to use ops as the API endpoint for interaction with incident issues.

There will be a number of situations where we lose the convenience of cross referencing issues to other gitlab.com projects. We manage to handle this well enough with projects that primarily require participation from the infrastructure department alone. It could become more problematic when we need to increase visibility on issues that impact other business units.

Cc @marin @dawsmith @brentnewton @glopezfernandez

Story points: 5.0
Simplify and standardize path-based haproxy blocking

Using file-based block lists simplifies the most common forms of ACL additions, making urgent changes safer and quicker to apply than writing new ACLs from scratch.

Useful lists we could standardize include:
* regexp match on request path
* regexp match on beginning of request path (implemented today by "blacklist-uris.lst")
* substring match on request path (computationally cheaper but less flexible than the equivalent regexp matcher)
* CIDR match on client IP (after Cloudflare compatibility transformations)
* regexp match on request path rate-limited to configurable X requests per second per client IP (after Cloudflare compatibility transformations)

Story points: 2.0
Verify SRE access to Azure

The following SREs need to verify their access to Azure portal:

- [x] aamarsanaa@gitlab.com @aamarsanaa 
- [x] ahmad@gitlab.com @ahmadsherif 
- [x] cbarrett@gitlab.com @craig 
- [x] cfurman@gitlab.com @craigf 
- [x] cmiskell@gitlab.com @cmiskell 
- [x] dsylva@gitlab.com @devin 
- [x] hmeyer@gitlab.com @T4cC0re 
- [x] hphilipps@gitlab.com @hphilipps 
- [x] msmiley@gitlab.com @msmiley 
- [x] nnelson@gitlab.com @nnelson 

Please reachout to me if you do not have access to the portal.

Cc @dawsmith

Story points: 1.0