giseldo/neodataset huggingface dataset:
Verifying gitlab domain for One trust

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details** 
 - Point of contact for this request: @Karuna16 
 - If a call is needed, what is the proposed date and time of the call:  [+ Date and Time +]
 - Additional call details (format, type of call): [+ additional details +]

**SRE Support Needed**
[+ Support Request Details +]

New txt record on `gitlab.com`
Verifying domain name for gitlab.com for One Trust SSO setup. Here's the [link](https://my.onetrust.com/s/article/UUID-3ec2fcd6-37f9-a0e3-20a1-acb6c64f99c3#UUID-3ec2fcd6-37f9-a0e3-20a1-acb6c64f99c3_bridgehead-idm458761989491843162723680452) on how to verify domain. Attaching the screenshot in case link doesn't work.![Screenshot_2021-03-01_at_9.04.10_PM](/uploads/3b7cf7fa46101c34e13c48ac61cdbf43/Screenshot_2021-03-01_at_9.04.10_PM.png)

The TXT token from One trust: onetrust-domain-verification=af5b5fda116e45a9b4c4abcd9e571923
Alias can @ or blank.

<!--
please do not edit the below
-->![Screenshot_2021-03-01_at_9.38.48_PM](/uploads/18a6e3bdc131df47aa2c7c386675dad7/Screenshot_2021-03-01_at_9.38.48_PM.png)

Story points: 1.0
Transfer meltano.{com,io,net,org} domains to Meltano's AWS account

As a separate business unit in the company, the Meltano team mostly manages its own assets (for example its website, accounts for Google Workspace, Slack, Zendesk, and SendGrid, and domain names meltanodata.com and singerhub.io), that are billed to a Meltano-specific corporate credit card and budgeted to the "R&D - Meltano" department.

The one exception are the `meltano.{com,io,net,org}` domains that are registered under GitLab's AWS account.
To give the Meltano team full control over the assets it's responsible for and make sure expenses are taken out of the correct budgets,
I think it would be appropriate for these domains to be [transferred](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-transfer-from-route-53.html) to Meltano's AWS account with ID 292927715491.

Since this was requested and rejected two years ago in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/6316, the Meltano team has become more independent and now manages more of its own infra assets, with the ownership of these domains a remnant of its early days as "just" a team in GitLab.

Story points: 2.0
Add deployment version info to K8s dropdown menu for each service in Grafana

Use case: We want to get information about the deployed version of a service in Grafana.

Right now there is a `pod-info` dashboard for each service where you can see the deployment versions e.g. [this](https://dashboards.gitlab.net/d/sidekiq-pod/sidekiq-pod-info?orgId=1) for sidekiq. But it would be easier to find, if we

1. would add this to the "Kubernetes Detail" dropdown menu for each service overview dashboard or
2. add the deployment version information directly to the "Kube Deployment Detail" dashboard, which already is linked in the "Kubernetes Detail" dropdown menu

I would prefer option 2.

Story points: 3.0
Review and update the Services-Base README

Per this incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3398

The [README.md](https://ops.gitlab.net/gitlab-com/services-base/-/blob/master/README.md) for the Services-Base project should be reviewed and updated to better reflect the intended workflow of making a change, promoting the change through the Staging and Production environments, and cleaning up any remaining environments.

Story points: 3.0
TLS renewal or certificate for airflow.gitlabdata.com

This was originally set up with https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9205 and was implemented as described in [our handbook](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/#tls)

Looks like we either need access to Route 53 to do this or the renewed certificate. 

The data team work is tracked in https://gitlab.com/gitlab-data/analytics/-/issues/7994

Story points: 2.0
Grow production API fleet by 6 nodes (or more)

[![Screen_Shot_2021-02-12_at_2.52.48_PM](/uploads/97561837174e5c2969e4a676824ce8ec/Screen_Shot_2021-02-12_at_2.52.48_PM.png)](https://dashboards.gitlab.net/d/api-main/api-overview?viewPanel=66&orgId=1&from=1612986771345&to=1613159571346&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-sigma=2)

[![Screen_Shot_2021-02-12_at_2.56.10_PM](/uploads/62cd2c700ef8577f275fd664a6b67421/Screen_Shot_2021-02-12_at_2.56.10_PM.png)](https://dashboards.gitlab.net/d/api-main/api-overview?viewPanel=1217942947&orgId=1&from=1612986960558&to=1613159760558&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-sigma=2)

The API VM fleet is starting to show loads that place the VMs load/core value over 1 during peak use times. This coupled with a deploy can lead to poor apdex. Until API is moved into Kubernetes, we should increase the fleet size to minimize customer facing performance issues.

The outcome of this work should be that deploys during peak times should not spike single node puma worker component to 100%.

Story points: 3.0
Update packagecloud to latest version

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details** 
 - Point of contact for this request: @twk3 @joshlambert @mendeni 
 - If a call is needed, what is the proposed date and time of the call: (likely not needed)
 - Additional call details (format, type of call):

**SRE Support Needed**

We would like to have packages.gitlab.com upgraded to the latest version of packagecloud 3.0.3 as it brings support for additional package versions the Distribution team would like to provide to customers.

We would like for the upgrade to be available for us to use with the GitLab 13.10 release.

Omnibus issue: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5979

cc\ @ahanselka @dawsmith 




<!--
please do not edit the below
-->

Story points: 5.0
Provision read only console servers

Write Terraform to provision read only console servers including Teleport and ephemeral sessions from PoC

Once this is complete we can:

- Deploy Teleport on Read Only console servers: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12565
- Deploy Ephemeral Sessions on Read Only console servers: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12566

Story points: 8.0
Unable to expand Cloud NAT IP routing any further

In recent incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3448

We needed to add additional IP addresses to our Cloud NAT device.  Luckily we were able to easily add two ip addresses that were next in line with our IP reservations.  Should we need to expand any further, our terraform module may not support this.

Utilize this issue to accomplish the following:

1. [x] Learn how IP allocation works with respect to Cloud NAT devices - during the incident, we appeared to use IP addresses that are not technically allowed to be used in the CIDR range
1. [ ] Determine how we can expand the set of IP's to the Cloud NAT device - currently we apply a CIDR range and a count of IP addresses - the current allocated CIDR has no free IP addresses
1. [ ] Documentation updated to reflect how these are configured and how to expand in the future
1. [ ] Consider expanding our current Cloud NAT IP Allocation

Marking this as high priority because the next time we run out of ports, we currently do not have an identified mitigation strategy.

Story points: 13.0
Fluentd pod restarts should not cause many old, already indexed log lines to be sent to elasticsearch

<!--
I see you're filing an issue for the observability SRE team. Please write a clear description of the problem in the issue description, and acceptance criteria. Give the issue a terse but descriptive title.

Don't forget to:

- Put the issue in an epic, if one fits
- assign a weight based on perceived complexity and how much of the problem space is unknown: 1, 2 or 4

See https://about.gitlab.com/handbook/engineering/infrastructure/team/reliability/observability/ for a description of our workflow.

Remember to keep your epics in order of priority / necessity, to indicate to the team lead and engineering manager that you would like it to be inserted into the backlog.
-->

## Problem

It's been observed (e.g. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3435#note_498652734) that restarting fluentd (by reprovisioning the daemonset pod) causes log files to be read in full from the beginning, entering many old and duplicated lines into elasticsearch. This skews logs-based aggregations, and puts strain on the elasticsearch nodes.

## Desired outcome

Fluentd makes a best-effort attempt to pick up where it left off. The pos_file feature of the tail input plugin (https://docs.fluentd.org/input/tail#read_from_head), which we do configure, is meant to allow fluentd to resume from its previous position. The pos files are kept on `/var/log`, which in the daemonset pod is the same mountpoint is in the host's mount namespace and so files written there persist across pod restarts. 

## Acceptance criteria

- [ ] As few as possible duplicate lines are entered into elastic in most fluentd pod restart scenarios

Story points: 2.0
ILB for thanos-query-frontend, to bypass IAP for grafana

<!--
I see you're filing an issue for the observability SRE team. Please write a clear description of the problem in the issue description, and acceptance criteria. Give the issue a terse but descriptive title.

Don't forget to:

- Put the issue in an epic, if one fits
- assign a weight based on perceived complexity and how much of the problem space is unknown: 1, 2 or 4

See https://about.gitlab.com/handbook/engineering/infrastructure/team/reliability/observability/ for a description of our workflow.

Remember to keep your epics in order of priority / necessity, to indicate to the team lead and engineering manager that you would like it to be inserted into the backlog.
-->

## Problem

We need to access thanos-query-frontend in kubernetes internally, inside and outside GKE (because we have not migrated grafana to k8s yet).

## Desired outcome

A DNS entry that resolves to a private IP, that allows unauthenticated access to thanos QFE from inside the VPC.

## Acceptance criteria

- [x] A DNS entry that resolves to a private IP, that allows unauthenticated access to thanos QFE from inside the VPC

---

FYI @bjk-gitlab

Story points: 1.0
CI Cleaner script seems to have stopped working

For some reason, the CI cleaner script has stopped working but doesn't error out. I'm not quite sure why as there is no error. Nonetheless, this should be fixed.

My initial suspicion is that something changed about our naming scheme or the GCP API, but I have done minimal investigation thus far.

[link to pipelines](https://ops.gitlab.net/gitlab-com/gl-infra/ci-project-cleaner/-/pipelines)

Story points: 8.0
Delete extra server entry from Okta ASA

During the course of fixing https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12441, an additional server entry for Okta ASA was created. This is unnecessary and should be deleted, but I don't have access to. The only person I know of that has access to delete is @cmiskell, but there may be others.

[this is the server that needs deleting](https://app.scaleft.com/t/gitlab-poc/project/windows_ci/servers/28409349-186f-4079-a036-9faf2da9a92a)

Story points: 1.0
Support pg_ctl_timeout attribute in the patroni service configuration

Support `pg_ctl_timeout` attribute in the patroni service configuration.

## Why

The present situation in the production patroni hosts is as follows:

The command `sudo systemctl stop patroni` when invoked in production does not wait long enough for the postgresql PostgreSQL database server to terminate, and sends it a `sigkill` signal, which forces the process to halt in an untidy way.

This would also happen if a patroni host was instructed to shutdown or reboot -- the `systemd` service manager would attempt to stop all its unit daemons, including the `patroni` service.

## Expectations

This is an experiment with no bad consequences.  The experiment is to put in place a configuration attribute which allows us to adjust the amount of time which patroni waits for the PostgreSQL database server to cleanly shutdown before patroni gives up and sends postgresql a `sigkill`.

It is NOT KNOWN how long the PostgreSQL database server is required to shut down cleanly in production.  We have not yet conducted any tests to determine what this interval actually is.

## Possible outcomes

The worst possible outcome is that patroni takes longer than the default timeout of `30` seconds for the PostgreSQL database server to shut down cleanly.  If the PostgreSQL database server still has not shut down by the end of the new timeout of `120` seconds, then we are no worse off than we were before.

But now we will have in place a mechanism to control this timeout interval configuration and adjust it upwards as we encounter new situational information around host system and patroni service shutdown commands, whereas presently we have no method by which to control this mechanism using our configuration management tooling.

## Material references

Discussions around discovery of problem which this issue intends to iteratively solve.

- https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12066#note_464281824
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12066#note_463307997
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12066#note_463768785

## Alternative proposals

This is not strictly an alternative option, as it can be combined with the current approach to reduce anxiety around system and service stop instructions.  It is possible to also disable the patroni systemd unit configuration to use the `sigkill` method at all.

- https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12066#note_465064619

Story points: 1.0
replace `module.gcp-tcp-lb-spectrum` with `module.gcp-tcp-lb` in gprd

As part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12313 we cleaned up `gprd` terraform to have a clean resource for the main spectrum app. To reach full alignment with `gstg`, we need to create a new `module.gcp-tcp-lb` with identical configuration and migrate the spectrum app over to that.

Story points: 13.0
Woodhouse is @-ing unrelated people in incident issues

Example: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3381#note_490373672

It appears as thought woodhouse is sometimes @-ing the incorrect GitLab handle for incident openers in the issue description, leading to this situation in which @brentnewton opened an issue which tagged `@bnewton`. We can see this was the case, from this subsequent description edit:

![Screenshot_2021-01-21_at_07.43.03](/uploads/d633389241bfe4a3d9aa8e85ba19f32e/Screenshot_2021-01-21_at_07.43.03.png)

This may be due to incorrectly assuming some relationship between the user's Slack handle and their GitLab handle.

Story points: 1.0
Woodhouse mentions the wrong GitLab user

I noticed after I created production#3338, that the initial description populated by Woodhouse at-mentioned `cbarrett`, but my gitlab.com username is @craig. I'm not sure where this identity is managed, since my slack handle is also @craig, so it's not pulling from there. My email is either cbarrett or craig at gitlab.com, depending on how/where you look.

Story points: 3.0
Update our WAL-G info in our Runbooks

This Runbook information is outdated and we should update it as a result of our successful implementation of WAL-G in production (retiring WAL-E)

https://ops.gitlab.net/gitlab-com/runbooks/-/blob/master/docs/patroni/postgresql-backups-wale-walg.md

We may want to iterate on this in 2-3 different phases, to get results asap (clean the wrong info first), given the size and the amount of info of this Runbook.

Acceptance Criteria:
- [ ] All the mentions to WAL-E are removed, or state clearly the ones that are not valid anymore.
- [ ] Update the mentions to WAL-G in the runbook, so they represent what we have in production at the moment.
- [ ] Clean the sections that are not needed, and the pointers to external links.

Story points: 5.0
Bump omnibus to Redis 6.0.10

We'll also want to make sure to allow the new threaded-io settings to be configured:

From [annotated `redis.conf`](https://raw.githubusercontent.com/redis/redis/6.0/redis.conf):

```
################################ THREADED I/O #################################

# Redis is mostly single threaded, however there are certain threaded
# operations such as UNLINK, slow I/O accesses and other things that are
# performed on side threads.
#
# Now it is also possible to handle Redis clients socket reads and writes
# in different I/O threads. Since especially writing is so slow, normally
# Redis users use pipelining in order to speed up the Redis performances per
# core, and spawn multiple instances in order to scale more. Using I/O
# threads it is possible to easily speedup two times Redis without resorting
# to pipelining nor sharding of the instance.
#
# By default threading is disabled, we suggest enabling it only in machines
# that have at least 4 or more cores, leaving at least one spare core.
# Using more than 8 threads is unlikely to help much. We also recommend using
# threaded I/O only if you actually have performance problems, with Redis
# instances being able to use a quite big percentage of CPU time, otherwise
# there is no point in using this feature.
#
# So for instance if you have a four cores boxes, try to use 2 or 3 I/O
# threads, if you have a 8 cores, try to use 6 threads. In order to
# enable I/O threads use the following configuration directive:
#
# io-threads 4
#
# Setting io-threads to 1 will just use the main thread as usual.
# When I/O threads are enabled, we only use threads for writes, that is
# to thread the write(2) syscall and transfer the client buffers to the
# socket. However it is also possible to enable threading of reads and
# protocol parsing using the following configuration directive, by setting
# it to yes:
#
# io-threads-do-reads no
#
# Usually threading reads doesn't help much.
#
# NOTE 1: This configuration directive cannot be changed at runtime via
# CONFIG SET. Aso this feature currently does not work when SSL is
# enabled.
#
# NOTE 2: If you want to test the Redis speedup using redis-benchmark, make
# sure you also run the benchmark itself in threaded mode, using the
# --threads option to match the number of Redis threads, otherwise you'll not
# be able to notice the improvements.
```

We'll need to review the [CHANGELOG](https://raw.githubusercontent.com/antirez/redis/6.0/00-RELEASENOTES) for potential BC breaks.

Story points: 2.0
Create lifecycle for docker images and containers on runner-01-inf-ops

The `runner-01-inf-ops` runner has a lot of disk space that is consumed by docker images/containers/volumes/etc.

We should add some cron jobs to perform routine clean-ups on these resources to keep the node operating with plenty of disk space for new images, etc.

Minimum DOD:
* A weekly cron job to clean up dangling images `docker image prune`

Nice to have DOD:
* A more elaborate cron job that also cleans up images not attached to containers, old containers, old volumes, and old networks.
* Consider moving `/var/lib/docker` to it's own filesystem to prevent this disk usage from endangering the node's root filesystem.

Story points: 2.0
Align altssh on gstg with gprd & cleanup terraform residue

**Current Situation**
<!--Describe the current situation which you are intending to iterate on with the work described in this issue.-->
Currently on gprd we redirect the altssh traffic to our regular SSH LBs in Cloudflare Spectrum. We don't do this for gstg.

**Desired Outcome**
<!--Describe the intended outcome of the work represented in this issue. How does it improve upon the current situation above?-->
gprd and gstg both re-map `altssh.gitlab.com:443` to `$gitlab_com_origin_ip:22` in Cloudflare spectrum.

**Acceptance Criteria**
These are intertwined, thus not separate issues!
<!--List out work items necessary to arrive at the desired outcome above, as atomically as possible.-->
- [x] Copy the terraform definition for altssh from gprd to gstg
- [x] Apply the changes
- [x] Clean up the old terraform code

Story points: 13.0
Wrap up GSM epic and milestone

The [project to launch Google Secrets Manager](&343) and initial set of related processes is effectively done, and needs to be wrapped up. The [epic](&343) and [milestone](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/113) both need to be updated and closed, with references to leftover follow-up tasks to be addressed individually or as part of other, subsequent efforts (e.g. fixing instance-level service accounts during &231)

Story points: 1.0
Create new gitaly storage shard nodes to replace `nfs-file52` and `nfs-file53`

Gitaly storage shard `nfs-file52` (`file-52-stor-gprd.c.gitlab-production.internal`) is at `75.46%` usage as of `2021-01-07`.  Gitaly storage shard `nfs-file53` (`file-53-stor-gprd.c.gitlab-production.internal`) is at `72.68%` usage as of `2021-01-07`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on both `nfs-file52` and `nfs-file53`.  The automatic weights assignment tool will soon eliminate `nfs-file52` and also `nfs-file53` as a candidate shard receiver for new project residencies.

Note: This is partially a corrective action for incident [`2020-12-21: Gitaly nodes being removed from weight pool too soon`](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3225): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3225

It is no longer the case that nodes are being removed too soon, and so the initial five additional nodes were reduced to only two, instead.

Story points: 1.0
Evaluate infracost for inclusion in CI for terraform repositories

The [infracost](https://www.infracost.io) utility seems like a very useful addition to our standard terraform workflows, and [this template](https://gitlab.com/infracost/infracost-gitlab-ci) may provide a useful starting place for incorporating the addition into our terraform project workflows (gitlab-com-infrastructure, gitlab-services, environments)

Story points: 3.0
Improve wal-g backup job alerting

The wal-g backup job metrics caused bogus alerts, because the backup is running from a random node each time, which made the metrics look like there was no backup for several days on a single node, while we are only interested on one successful backup job per env each day.

This was tracked within this incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3263

The fixes:

* https://gitlab.com/gitlab-com/runbooks/-/merge_requests/3088
* https://ops.gitlab.net/gitlab-cookbooks/gitlab-walg/-/merge_requests/39
* https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/4813
* https://gitlab.com/gitlab-com/runbooks/-/merge_requests/3093

Adding this issue for tracking in our milestone.

## Acceptance criteria

* [x] Do not cause "wal-g backup delayed" false alerts for single patroni nodes
* [x] runbook doc for how to use central pushgateways for job metrics with environment focus

Story points: 8.0