giseldo/neodataset huggingface dataset:
Issue for investigation of gitlab_com_db_sync errors for data team

This issue will be for investigation of data sync issues from our replica to snowflake for the data team for gitlab.com syncs.

It is related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7141 which covers the syncs for license, version, and customers.  

Facts:
- The data extracts run in Airflow from the gitlab-analysis project in GCP in a scheduled set of K8s jobs.
- https://gitlab.com/gitlab-data has the projects, docker files and config
- Connections are to: `postgresql://analytics:***@10.217.7.101:5432/gitlabhq_production` to connect to the analytics replica - which is GCP production instance postgres-dr-archive-01-db-gprd
- logs of errors are in the slack channel #analytics-pipelines
- We'll add example errors and date's times in comments.


cc @kathleentam @tlapiana @jjstark from the data team. I'm trying to separate threads of investigation.  This issue will be for gitlab.com sync and we'll keep investigating the sync for customers, license, and version on #7141 .  The backends / gcp projects are different and we can better manange priority/ solutions if we split investigation.

Story points: 1.0
The chef secrets bucket is created in 2 places

it looks like the secrets bucket is created in 2 places:

1. https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/project/blob/master/main.tf#L28, which is called from env-projects (in which there is an entry for CI)
1. https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/storage-buckets/blob/master/main.tf#L59

Since env-projects and each gitlab env (e.g. gprd) are different TF envs, with separate tfstate, they can both track this bucket. @craig, when you created env-projects, did you have to import this bucket? If so we should decide whether or not this bucket belongs to the meta-env (env-projects) or a gitlab-google-project-tf-env (e.g. gprd, gstg).

Since the secrets bucket must be pre-populated for any chef'ed VMs provisioned with terraform to work, I vote env-projects. @craig feel free to reassign this, but I just wanted to make sure we are on the same page before I make any change.

cc @hphilipps (who I was chatting about this with)

Story points: 1.0
Automate and document unsealing process for Vault

Once we have a production Vault cluster we will need to provide documentation and automation around the unsealing process

1. Document the basic process to seal/unseal the cluster
1. Add keys for new engineers
1. Remove keys after engineers change roles / leave the team
1. Rotate keys for existing engineers
1. Monitor/report on key expiration, last rotation, etc.

Story points: 3.0
Fix thanos object store errors across envs

ThanosCompactBucketOperationsFailed is firing regularly for thanos-store-0{1,2}-inf-gprd.

Graphs show:
1. High consistent failure rates in testbed + pre environments (not alerting due to their environment)
1. Scattering of failures in ops + gstg + dr
1. Regular failures of the 'get' operation in gprd, leading to these alerts.

This needs cleaning up for testbed + pre (to eliminate noise), and either fixing the errors, or increasing the threshold if they are normal and recoverable.

Story points: 1.0
SLO for background processing jobs with a target to reach an upper-bound of 1 min

Currently, for Sidekiq we have an SLA status. You can see a 30 day running average of the status available at https://dashboards.gitlab.net/d/general-slas/general-slas. 

![Screen_Shot_2019-09-24_at_12.08.10](/uploads/573f434d38a8d01e28788d4e067709f3/Screen_Shot_2019-09-24_at_12.08.10.png)

@andrewn has done a considerable amount of work setting up alerts based on SLO violation. If we don't have them already setup for background processing jobs, we should. A considerable amount of work has been done to separate and split jobs, priorities, and queues appropriately, which hopefully allows us to place monitoring granularly for each specific category of job.

https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1 shows the metrics in detail. It's been raised that certain–maybe even all?–background processing jobs should adhere to a 1 min latency average. Though, at first glance, this may not be possible for all jobs–project exports take substantially longer than others.

Story points: 2.0
first practice incident - rough agenda

Basic summary

This is meant to be a simple problem to solve and a table top scenario.
First, we are testing incident response and basic group of host interactions.

Open questions:
1.  Could/Should we use staging to do this?
1.  Should we make a project/repo (like readiness) to make md files for practice scenarios?

Scenario:
service stop haproxy on all LB - current status no haproxy running (front door is closed)

Incident Start:
EOC - execute command(s) to stop LB in gstg

Start of incident handling
1. EOC use `/incident declare` in `#incident management`

Validate:
1. [x] EOC page
2. [x] IMOC page
3. [x] CMOC page
4. [x] Creation of incident gdoc
5. [x] Creation of incident issue?
6. [x] EOC, IMOC, CMOC join incident zoom

Once manager and cmoc join
1. IMOC/CMOC - talk through any comments to understand the issue
1. cmoc log into status.io and talk through what they would do to create incident [Link to tests status page](https://app.status.io/statuspage/5bedc0c2a394fc04c9ccc974)
1. cmoc - talk through your update to status.io

Resolution actions
1. EOC - talk through actions you would do to get load balancers restarted
1. EOC/ Manager - talk about how to escalate to engineer on call
1. Verify incident is resolved
1. cmoc - confirm resolution and talk through status.io update
1. Follow up with action items
1. Create Incident Review issue
1. How to escalate action items to infradev. 

cc @marin and @AnthonySandoval for feedback

Story points: 5.0
Run rake task to update usage data entries on version.gitlab.com

We need to execute the following rake task on the production instance of `version.gitlab.com`:
`rake usage_data:fix_usage_data_stats`

The goal of this rake task is to change any `usage_data` record that has non integers in the `stats` `json` column to be compliant. i.e. having only integers for values in the `stats` column.

This rake task can be ran 2 ways:
1.  Against all `usage_data` records that have a non empty `stats` column at once (in batches of 1000):
*  `rake usage_data:fix_usage_data_stats`
2.  Against predefined `id` ranges (to lessen the individual execution times per batch):
*  `rake usage_data:fix_usage_data_stats[1,1000]` - where `1` is the starting `id` in `usage_data` table, and `1000` is the ending `id` in the `usage_data` table.  In this scenario, this would need ran in chunks up until the last `id` in the `usage_data` table, which is printed out in each run if arguments are passed as above.

Decision on which one to run is up to the infrastructure team to decide.

Story points: 1.0
Investigate making haproxy metrics more robust

https://dashboards.gitlab.net/d/AkOdlrSmk/imported-haproxy-stats uses haproxy_backend_current_sessions and haproxy_frontend_current_sessions which are point-in-time sampled values.  This leads to some anomalous and misleading graphs when the sampling misses active sessions, e.g. we've seen graphs drop to 0 for 429_rate_limit, when there are provably requests being rate-limited.  It's probably worse for rate-limited sessions that are dropped immediately, compared to any even slightly-longer-running normal HTTP requests which are going to result in a more consistent sampling.

The anomalous graphs have caused us to go down a rabbit hole of investigations at least once, and we need to do better.

Options include other haproxy exporter metrics, existing mtail metrics, or at worst, enhancing mtail to capture more data.

Story points: 2.0
Stabilize Cloud NAT in CI

See https://gitlab.com/gitlab-org/gitlab/issues/32433#note_220111066 for original context.

Cloud NAT was rolled out to private runner machines in CI on 2019-09-20. There were then several job failures attributable to NAT failures: connection failures coinciding with error bursts at the NAT.

Since then several increases in NAT port:VM ratio have been rolled out in order to mitigate these failures (while ensuring there are still enough IPs to support the private runner fleet). There have been no reports of job failures attributable to NAT since that ratio was set to 256 (it had originally been 64, the Cloud NAT default).

There have still been occasional period of elevated errors but it isn't clear whether or not these result in higher-level failures / slowdowns or are at an acceptable level to be fixed by higher-layer retries (TCP, or application layer).

Story points: 4.0
Tuning Puma settings on ops.gitlab.net

There is a small background rate of Puma worker terminations with the default memory limit of 650MiB in combination with 4 worker threads.

![puma-memory-max-2019-09-20](/uploads/2eed4c9667664bd5c5d14e54a126f534/puma-memory-max-2019-09-20.png)

![puma-worker-terminations-2019-09-20](/uploads/0fcc69b87dd166cb09b566f5f3fe4c40/puma-worker-terminations-2019-09-20.png)

Story points: 2.0
View stackdriver metrics from the CI project

In order to implement https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7728, we need to be able to alert on stackdriver metrics in the CI project. This will involve:

- [x] rolling out a sd-exporter there
- [x] Ensuring that exporter is scraped by prometheus
- [x] Being able to view those metrics in thanos
- [x] Being able to send alerts based on these metrics to Alertmanager.

cc @hphilipps

Story points: 3.0
Revert any manual work created from db user creation gstg-deploy to bypass statement_timeout

Yesterday the staging deploy was failing due to a statement timeout.  See the MR here: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1849

This is being reverted.  See the MR for additional context.

Any work that was done to create this dedicated user now needs to be removed as well.  This issue can be closed when that user is removed from the system.

Story points: 1.0
Track the cost of enabling serial port logging into Stackdriver

Enabled in https://gitlab.com/gitlab-com/gl-infra/production/issues/1156, this will cause additional (duplicated) logging into stackdriver, and we would like to understand the cost implications of this.  We can estimate it (and have, with an upper limit of around ~3GB/day in prod), but it will likely be easier to simply measure it after the fact to be really sure.

Story points: 1.0
Move Docker Build stage from GitLab Services to ci-images repository

There is a docker file which is built at the beginning of each pipeline in the GitLab Services project.  This image rarely changes, and should be moved to our [CI Images Repository](https://gitlab.com/gitlab-com/gl-infra/ci-images/). It is currently slowing down work by adding a lot of time to every pipeline run (even no-op), for no good reason.

The docker file is in `docker files/environments_client/Dockerfile` and is built in the [build_client_image](https://ops.gitlab.net/gitlab-com/services-base/blob/master/.gitlab-ci.yml#L19) stage.

This image is based on `google/cloud-sdk:alpine` and includes:

- `git`
- `openssh`
- `curl`
- `jq`
- `kubectl`
- `Helm`

The version of the Docker image should reflect the version of `Helm` that is installed.

Once the image is built correctly, the `.gitlab-ci.yml` file needs to be updated with the correct [DOCKER_IMAGE_TAG](https://ops.gitlab.net/gitlab-com/services-base/blob/master/.gitlab-ci.yml#L4)

Story points: 2.0
Gather input from Engineers regarding the content of a Grafana 101 training video

Objective: To identify content Engineers would like included in a Grafana 101 Training Video

Goals: GitLab Engineers need a comprehensive tutorial video to empower them to use Grafana to ensure they have the background to use this robust analytical and visualization tool.  The goal for this issue to identify the content that will be included in this video.

Story points: 1.0
Terraform automation blog post

It feels like we've hit a milestone in our Terraform configs, so it seems like a good time to do a write-up.

Initial ideas/brainstorming

- Using GitLab CI to run terraform
  - Leverage GitLab Environments feature
  - Clean up drift, upgrade to 0.12
- Greenfield (ephemeral-environments, group-projects, services) vs legacy (gitlab-com-infrastructure)
  - Are we using AutoDevOps for any of these?
- Mono-repo to versioned modules, tagging pipeline
- Cleanup/link to [design doc](https://about.gitlab.com/handbook/engineering/infrastructure/library/terraform-automation/)
- Next steps / future plans
  - [GitLab Flow](https://about.gitlab.com/handbook/engineering/infrastructure/library/git-workflow/)
  - [Incorporate Vault](https://about.gitlab.com/handbook/engineering/infrastructure/library/vault/)

/cc @ansdval for prioritization / scheduling
/cc @devin @craigf @hphilipps @ggillies FYI

Story points: 3.0
Remove Geo config in staging

The staging environment is set up as a Geo primary.  This is probably left over from previous work.  It is causing unnecessary jobs, and likely also a lot of unnecessary database records.

I'm going to go ahead and remove the primary config. Since there is no secondary, this shouldn't hurt anything.  If someone needs it, they can set it up again.

I'm creating this issue so that there is a record of what was done.

![Screen_Shot_2019-09-16_at_1.33.59_PM](/uploads/6cf442f3dd08b1e292b1ce482ab5ed37/Screen_Shot_2019-09-16_at_1.33.59_PM.png)

Story points: 1.0
Group Project Request: gitlab-qa-50k

# Group Project Request

- Project / Group Name (<17 characters): `gitlab-qa-50k`
- Project Administrator (email): gyoung@gitlab.com

### Provide a brief overview of the reason for this project and why it is needed and for how long it will be used.

This project will contain the environment used to test the 50k reference architecture which will be built and maintained by the Quality Department. 

This project will be used indefinitely - as long as the 50k reference environment stays relevant and supported. 

Quality's issue to create the 50k environment: https://gitlab.com/gitlab-org/quality/performance/issues/66

## Security

### Provide a list of data and the corresponding classification that will be used in this project and how it will be accessed.

The data used is a sterile copy of GitLab CE. There is no sensitive data. The data is used during the load tests to simulate an appropriately complex project. @grantyoung can you verify that what I've described here is valid and correct?

## Group Project Access Checklist

Make sure the following criteria is met and understood by the project administrator. 

- [-] If the gitlab.com database is copied, that data has been processed by the [pseudonymization script]( https://gitlab.com/gitlab-com/runbooks/blob/master/howto/pseudonymization-gitlab-db.md).
- [x] Regular security updates are applied to all nodes in the project.
- [x] Unused instances will be removed in a timely manner
- [x] The Project Administrator is responsible for any users or additional administrators that they add to the project
- [x] The Project Administrator is responsible for justifying any cloud spend within the project.
- [x] Group Projects are intended for development, test, or demo work. Everything in these projects is considered temporary.

## Infrastructure Tasks

- [x] Create file in https://ops.gitlab.net/gitlab-com/group-projects named `environments/(group name from above).tf` by copying an existing file and changing the Administrator and Group Name variables
- [x] Merge the change to master
- [x] Create a branch from master named `(group name from above)` and push
- [x] Verify that the pipeline completed successfully at https://ops.gitlab.net/gitlab-com/group-projects/pipelines

Story points: 5.0
Group Project Request: gitlab-qa-25k

# Group Project Request

- Project / Group Name (<17 characters): `gitlab-qa-25k`
- Project Administrator (email): gyoung@gitlab.com

### Provide a brief overview of the reason for this project and why it is needed and for how long it will be used.

This project will contain the environment used to test the 25k reference architecture which will be built and maintained by the Quality Department. 

This project will be used indefinitely - as long as the 25k reference environment stays relevant and supported. 

Quality's issue to create the 25k environment: https://gitlab.com/gitlab-org/quality/performance/issues/57

## Security

### Provide a list of data and the corresponding classification that will be used in this project and how it will be accessed.

The data used is a sterile copy of GitLab CE. There is no sensitive data. The data is used during the load tests to simulate an appropriately complex project. @grantyoung can you verify that what I've described here is valid and correct?

## Group Project Access Checklist

Make sure the following criteria is met and understood by the project administrator. 

- [-] If the gitlab.com database is copied, that data has been processed by the [pseudonymization script]( https://gitlab.com/gitlab-com/runbooks/blob/master/howto/pseudonymization-gitlab-db.md).
- [x] Regular security updates are applied to all nodes in the project.
- [x] Unused instances will be removed in a timely manner
- [x] The Project Administrator is responsible for any users or additional administrators that they add to the project
- [x] The Project Administrator is responsible for justifying any cloud spend within the project.
- [x] Group Projects are intended for development, test, or demo work. Everything in these projects is considered temporary.

## Infrastructure Tasks

- [x] Create file in https://ops.gitlab.net/gitlab-com/group-projects named `environments/(group name from above).tf` by copying an existing file and changing the Administrator and Group Name variables
- [x] Merge the change to master
- [x] Create a branch from master named `(group name from above)` and push
- [x] Verify that the pipeline completed successfully at https://ops.gitlab.net/gitlab-com/group-projects/pipelines

Story points: 5.0
Take a screenshot about the group hook under gitlab-org

Following up from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7855#note_217393324

@cmiskell I just tried to manually trigger it, but I don't think it's working. Could you please take a screenshot on the hook editing page so I can see if it's properly configured?

Test was done at: https://gitlab.com/gitlab-org/quality/triage-serverless/issues/11#note_217393880

Marking as confidential because it contains tokens.

Story points: 1.0
file-15-stor-gprd rebooted

file-15-stor-gprd rebooted at 9/14 09:37 UTC.

Opened this issue to track repos with zeroed files. Currently searching through all repos:

```
cd /var/opt/gitlab/git-data/repositories/@hashed
ionice -n 5 find . -regextype sed -regex ".*/objects/.*" -size 0 > /var/tmp/zerofiles.txt
```

Story points: 3.0
Investigate creating a kubectl wrapper script with a production warning

Decide if it makes sense to create a kubectl wrapper script to include a check of the current context and the command line.

- If the current context is production **AND**
- If the current command is not **get**
- Then confirm whether to continue modifying production (y/N)

We will have to discuss how to make sure all users are using the wrapper script and not calling `kubectl` directly

Story points: 3.0
Set Retention period for cloudtrail bucket

We enabled AWS CloudTrail organization wide in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7565. This means that CloudTrail logs go to an S3 bucket in our primary org account, and that bucket will never clean itself up.

How long do we want to keep CloudTrail logs for AWS? I was thinking that 1 year should be sufficient. Alternatively, we can keep them longer and just use a transition to cold storage for them.

Story points: 1.0
file-33-stor-gprd rebooted

file-33-stor-gprd rebooted at 9/11 15:19 and 11/11 15:42 UTC.

Opened this issue to track repos with zeroed files. Searching through all repos:

```
cd /var/opt/gitlab/git-data/repositories/@hashed
ionice -n 5 find . -regextype sed -regex ".*/objects/.*" -size 0 > /var/tmp/zerofiles.txt
```

Results below.

Story points: 3.0
Help delete a group webhook under gitlab-org

The context is at https://gitlab.com/gitlab-org/quality/triage-serverless/issues/11#note_215742570

We want to delete the webhook under https://gitlab.com/groups/gitlab-org/-/hooks

![image](https://gitlab.com/gitlab-org/quality/triage-serverless/uploads/d0b62d599053d89a74f33cec57fb096b/image.png)

The one for http://triage-customers.triage-serverless-12690061.triage.serverless.grzegorz.co

However we cannot do so via the web UI because of https://gitlab.com/gitlab-org/gitlab-ee/issues/29215

Could someone help us delete it? Then @rspeicher can help us create another one for https://triage-serverless.gitlab.com/

I am not sure why we cannot edit it, but it seems we can only delete and create a new one.

Story points: 1.0
Potential bug with our GKE module related to node pool instance counts

Two recent events brought up an issue with our GKE module: https://gitlab.com/gitlab-com/gl-infra/terraform-modules/google/gke

During testing, an engineer was unable to get a node to be created with nodes participating in the cluster.  And another time when an Engineer was adding an additional node pool, both times the Engineers ran into the same issue.

The most probable cause may be the removal of the `node_count` option: https://gitlab.com/gitlab-com/gl-infra/terraform-modules/google/gke/commit/8e49364128f22b9b4ef4de8b029e4562de62c154

Utilize this issue to investigate/test/fix whatever may be wrong.

Story points: 3.0
Git Data Loss on Gitaly server restart

When a user is pushing a git repository and the Gitaly server which is hosting that repo shuts down for any reason, that push fails leaving 0 byte files on the disk. When the server comes back up, any subsequent pushes to those repositories fail until someone manually intervenes.

The recent Infrastructure issu is: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7837

It's also being discussed in the EE tracker: https://gitlab.com/gitlab-org/gitlab-ee/issues/24267

Some suggestions are:

- `core.fsyncObjectFiles=true` Git Option
- Set the `data=journal` mount option to prevent zeroing files on crashes. Now we have `data=ordered`.

The ultimate solution will have to be Gitaly HA, but we will need to mitigate this problem before that is ready.  Use this issue to discuss other ideas.

/cc @gitlab-com/gl-infra

Story points: 3.0