giseldo/neodataset huggingface dataset:
Add GCS storage to Thanos service

In order to reduce / offload local storage requirements for Prometheus, Thanos can upload TSDB data to GCS (Also S3, or similar).

TODO:

* [x] Add a GCS bucket to the "ops" project for Thanos to read/write.
* [x] Configure the sidecars to upload data.
* [x] Add a [Store Gateway](https://github.com/improbable-eng/thanos/blob/master/docs/getting_started.md#store-gateway) to provide read access.
* [x] Reduce Prometheus local retention to 7 days.
* [x] Add a [Compactor](https://github.com/improbable-eng/thanos/blob/master/docs/getting_started.md#compactor) to add downsample data.

Story points: 3.0
Setup no-data alerts

We need to come up with a strategy for creating no-data alerts, this was recently seen in https://gitlab.com/gitlab-com/production/issues/459 where an alert was defined but due to a name refactoring the alert was no longer valid.

This is problematic in the current configuration because we have the same alerting rules for all environments, should we? this will cause problems when we hardcode environment specific labels.

Story points: 3.0
BigQuery sink for `logName="projects/gitlab-production/logs/haproxy"

`haproxy` logs are no longer being forwarded to ElasticCloud due to size, but we continue to run into cases where basic queries would be helpful to investigating traffic.  I'd to explore using a log sink directly to BigQuery to support these queries.

In particular, bulk queries for patterns of paths and user-agents have come up a few times in the last few months.  While we now have user agent in more logs in ELK, we're still not sure of the storage and stability impact.

My primary concern looking at the `haproxy` tagged logs in stackdriver is that there are appear to be 3 different formats for `jsonPayload`, so I think just trying it and seeing results in BigQuery is the fastest way to iterate and determine feasability.

 * [x] Infra team: Set up a sink in staging for `haproxy` logs in staging and collect 2 days worth of logs
   * [x] Security: examine result, determine if changes need to be made to filter or request other changes
   * [x] Iterate
 * [x] Infra: create filter and sink in production and collect for a few hours
 * [x] Both teams: Evaluate efficacy and size


/cc @jurbanc @jarv @andrewn

Story points: 5.0
Update Pagerduty api key to use v2 for cog

Got an email from PagerDuty that the v1 api is being deprecated:

In Pagerduty, there are 2 keys still using v1:

|Key Name|Created|Last Used|
|--------|-------|---------|
|ZenDesk Integration|Jan 25, 2016 at 7:04 PM  by Patricio Cano|Sep 5, 2016|
|GitLab Cog|Sep 13, 2016 at 7:59 AM by GitLab Admin|Sep 7, 2018 at 3:05 PM|

We can probably check with support and remove the zendesk, but likely need to update the GitLab Cog key whereever it is used

----------- email: ----------

PagerDuty’s v1 REST API will be decommissioned and no longer operational starting on October 19, 2018, at 22:00 PDT (UTC+7).

Your PagerDuty account has had at least one v1 REST API key in use over the last 90 days. This message has only been sent to the administrators of your PagerDuty account. Please forward this message to your team and ask them to migrate to our v2 REST API.

Click here to learn more about our v1 REST API decommissioning on this FAQ page. Please note that the PagerDuty v1 Events API is not being deprecated. Read the FAQ for steps to identify your active v2 REST API keys.

Feel free to contact support@pagerduty.com if you have unanswered questions or need assistance migrating.

Story points: 1.0
Email user list for ending GitLab.com Early Adopter program

We're ending the Early Adopter program for GitLab.com and are discussing in https://gitlab.com/gitlab-com/marketing/general/issues/3050.

### Problem to solve

We'd like to email all users/groups who will be impacted before making the switchover on October 1st, 2018.

### What's needed

CSV of all GitLab.com users and groups on the Early Adopter plan. Columns should be `name`, `email`, `state`, `username`, `created_at`.

Story points: 1.0
Rename postgres-01 to postgres-archive-replica-01 (Was: postgresql-01 production replica differs from other replicas)

The replication lag postgres-01 (replica) started to grow significantly today: https://prometheus.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=(pg_replication_lag%20%3E%2043200)%20and%20on(instance)%20(pg_replication_is_replica%7Bfqdn%3D%22postgres-01-db-gprd.c.gitlab-production.internal%22%7D%20%3D%3D%201)&g0.tab=0

@skarbek also pointed that repmgr isn't running on it.

`recovery.conf` differs from the other replicas, streaming replication is not enabled, it uses WAL shipping from S3 (perhaps it was left as is since the times of the Azure->GCP migration, when postgres-01 node intentionally didn't use SR): 

```
$ sudo cat /var/opt/gitlab/postgresql/data/recovery.conf
# recovery file for creating the standby server
# uses both restore_command to fetch wal chunks
# and pimary_conninfo to transition to secondary
# when possible

# Specifies whether to start the PostgreSQL server as a standby.
# If this parameter is on, the server will not stop recovery when the end of archived WAL is reached,
# but will keep trying to continue recovery by fetching new WAL segments using restore_command and/or
# by connecting to the primary server as specified by the primary_conninfo setting.
standby_mode = 'on'

# By default, recovery will recover to the end of the WAL log.
# So we don't need any recovery_* options

# If any option is unspecified in this string, then the corresponding environment variable (see Section 32.14) is checked.
# https://www.postgresql.org/docs/9.6/static/libpq-envars.html
# TL;DR: export PGPASSWORD=XXX
#primary_conninfo = 'user=gitlab_repmgr host=''postgres-01.db.prd.gitlab.com'' password=XXX port=5432 fallback_application_name=repmgr sslmode=prefer sslcompression=1 application_name=''postgres-01.db.gprd.gitlab.com'''
#primary_slot_name = secondary_gprd

# lastly, the restore command that will be run until we can switch
restore_command = '/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-fetch -p 32 "%f" "%p"'
recovery_target_timeline='latest'
```

Questions:

1) why it is still using WAL shipping instead of streaming replication, why do we have not symmetric setup?
2) why is it lagging (higher lags are possible because WAL shipping from AWS is way longer and less reliable way than SR, but this time the lag is too high)
3) do we really need 5 replicas? if yes, what are reasons for that?

/cc @abrandl @Finotto

Story points: 5.0
Create IO dashboard for Postgres disks

We'd like to gain more insight into IO behavior for postgres disks.

Let's add a dashboard to monitor this in one place:

* IO wait
* IOPS
* throughput read
* throughput write

As @skarbek pointed out, those metrics are in `node_disk_*`.

Some of these exist already, see https://dashboards.gitlab.net/d/bd2Kl9Imk/host-stats?refresh=1m&orgId=1&var-node=postgres-01-db-gprd.c.gitlab-production.internal and https://gitlab.slack.com/archives/C101F3796/p1536167065000100.


Story points: 1.0
Assets in canary don't work any more

When using [canary](https://about.gitlab.com/handbook/engineering/#canary-testing), no assets work any more.

This appears to be because we're using the production CDN, but if canary has assets that production doesn't.

From staging, I guessed that the CDN host was supposed to be gl-canary.global.ssl.fastly.net. However, https://gl-canary.global.ssl.fastly.net/assets/application-367ee28873f9e3f90b56617182f9eded9ad2dbd834f7279d5771980e1858b411.css doesn't work either.

@andrewn checked, and this is using canary.gitlab.com as a backend, which no longer exists. Should we bring it back and fix the canary config? cc @toon as you wanted this back for other reasons :slight\_smile: 

(Posted without using canary.)

Story points: 1.0
Automation for infrastructure deployments

In a bid to increase reliability, consistency, transparency, and asynchronous workflows, we should begin automating our current infrastructure deployment processes. We will start with simple scripted automation in our CI pipeline to start, and consider whether other changes and/or more advanced tools are necessary based on the results of that work. Since the [Atlantis project](https://www.runatlantis.io/) does not have sufficient security controls to be used on a public repository, we are unable to use it at this time. Terraform Enterprise also provides similar advanced features, though we will need to evaluate the cost after we have a better understanding of where the pain points are using GitLab CI, first.

## Objectives

- Master branch + current state are always consistent with live environment
    - `terraform plan` on master should show zero changes, except while `terraform apply` is running during merge
- All changes to environment are made via code and automation

- All changes in `gprd` have been applied/validated in `gstg`

_**We will obviously need to define how we will monitor and report on these_

## Next steps

1. Update CI pipeline to run `terraform plan` for each environment - #5113 

1. Add automatic apply stage for `gstg` - #5114

1. Add optional manual apply stages for `ops` and `gprd` - #5114

1. Setup required/blocking manual apply stage for `ops` and `gprd` - #5115

Story points: 3.0
New users, new projects, and MAU for GitLab.com for August 2018

We track users and projects for GitLab.com for internal use and for reports to other interested parties. I'd like to obtain counts for August.

For GitLab.com, may I have the output for:

```
user_count = User.where(created_at: Date.new(2018,8,1).beginning_of_day..Date.new(2018,9,1).beginning_of_day).count
project_count = Project.where(created_at: Date.new(2018,8,1).beginning_of_day..Date.new(2018,9,1).beginning_of_day).count
```
and

```
SELECT COUNT(DISTINCT "audit_events".author_id)
FROM "audit_events"
WHERE ("audit_events"."created_at" BETWEEN '2018-08-01 00:00:00.000000' AND '2018-08-31')
```

Thanks a lot. :smile:

Story points: 1.0
Week 4 hashed storage - begin migration of gitlab-com & gitlab-org projects on gitlab.com

Related: https://gitlab.com/gitlab-com/infrastructure/issues/4772
Closes: https://gitlab.com/gitlab-com/infrastructure/issues/3955

Begin the migration process for `gitlab-com` and `gitlab-org` projects on gitlab.com

This is going to be better off scripted
* [x] Gather list of project ID's to be migrated
* [x] Perform the migration
* [x] Validate all projects have completed their migration

Story points: 5.0
Week 3 - enable hashed storage permanently for new projects on gitlab.com

Related: https://gitlab.com/gitlab-com/infrastructure/issues/4772

In Week 2, we temporarily enabled hashed storage for new projects. No issues were found. In this issue we will be re-enabling hashed storage for new projects and leaving the feature as enabled. We will continue to monitor for any problems. 

* [x] Enable hashed storage for new projects (with automatic migration when renaming/moving)
  - this is one of our easier stories to be completed :joy:
* [x] Monitor for any issues with the renaming/moving
* [x] If anything pops up with the renaming/moving migration, Flip the feature toggle back, but keep the "Hashed Storage for new projects enabled"

Story points: 1.0
Week 2 - temporarily enable hashed storage for new projects on GitLab.com

Related: https://gitlab.com/gitlab-com/infrastructure/issues/4772

* [x] Flip the feature toggle to disable the migration when renaming/moving a project (this step listed is on the parent ticket)
* [x] Enable the Hashed Storage option for 2 hours and monitor.
* [x] After 2 hours disable the Hashed Storage option for new projects.
* [x] Flip the feature toggle back to the initial state (this step listed is on the parent ticket)

Story points: 1.0
Sometimes GCS returns 5XX (Server error) status code

GitLab-rails have the object storage integration feature. It's used by various kind of areas, such as LFS, Avatar, Job artifacts, etc.

It seems sometimes GCS returns 5XX status code (Server error) when GitLab-rails tried to access to it. You can see a bunch of errors happen on our servers.

- Production: https://sentry.gitlap.com/gitlab/gitlabcom/?query=Google%3A%3AApis%3A%3AServerError
- Staging: https://sentry.gitlap.com/gitlab/staginggitlabcom/?query=Google%3A%3AApis%3A%3AServerError
- dev.gitab.org: https://sentry.gitlap.com/gitlab/devgitlaborg/?query=Google%3A%3AApis%3A%3AServerError

From what I saw, this happens transiently. At the moment, we don't have any clues of the cause.

We don't know which this issue falls into Infrastructure or GitLab-CE. For now, we create this on infrastrcture project.

Story points: 1.0
Setup a Prometheus Pushgateway

Having a Prometheus Pushgateway would help us solve many observability issues:

1. **Allow `ops.gitlab.net` CI jobs to post information related to CI runs**
   - As an example: if we setup a CI job which routinely performs a restore of our postgres backup, at the end of the job we can post the time it takes to the push gatewat
   - We can then decide (or better yet, let exec/business decide) on an acceptable restore time for our Postgres backup
   - This time limit can then be codified as a alert in AlertManager. If the CI job takes longer than X hours, we raise an alert
   - Additionally, we have trend data to predict and plan ahead
   - (cc @abrandl as this scenario came from discussions between us)

1. **Configure takeoff to send metrics related to**:
   1. Deployment time (per host) as a latency histogram
   1. Deployment in progress (as a gauge): `gitlab_takeoff_deployment_in_progress{fqdn="...", gitlab-version="..."} --> 1` 
   1. Given these metrics, we would be able to annotate our grafana dashboards with details of deploys. 
   1. We would also be able to compare error rates during and after deployments with much greater ease than we can at present.

1. **Allow for easier integration with non-prometheus components**:
   1. For example, Redis Sentinel allows CLI commands to run when a failover occurs. We could use this to increment a counter on each failover via the push gateway and then setup more robust alerting than we have at present.


Before starting to build - estimate out how much effort is it to do this with Chef vs K8s and update the team with the estimate.


cc @bjk\-gitlab @jarv

Story points: 3.0
Disable Transparent Huge Pages (THP) on Redis machines

I don't think we've had to do this before, and I don't see warnings in the Redis logs but Redis strongly recommends disabling THP to avoid latency:

* https://redis.io/topics/latency 
* https://github.com/antirez/redis/issues/3176 
* http://antirez.com/news/84

It can be disabled simply by:

```sh
echo never > /sys/kernel/mm/transparent_hugepage/enabled
```

Right now it's set as the following:

```sh
# cat /sys/kernel/mm/transparent_hugepage/enabled 
always madvise [never]
```

/cc: @jarv, @andrewn

Story points: 1.0
Unmount NFS for Gitaly

The tripswitch for for path access from Rails/Sidekiq has been enabled for a while now[1], so the next steps could be taken to remove NFS from our architecture.

Now, dev.gitlab.org doesn't use NFS to the best of my knowledge, so we can do testing there, but I think we should move forward by unmounting the NFS drive for staging and do manual testing there. And afterwards move on to production.

/cc @jramsay @jacobvosmaer\-gitlab 

[1]: https://gitlab.com/gitlab-org/gitaly/issues/1261

Story points: 2.0
Configure Snowplow integration on production for GitLab.com

We added our Snowplow configuration to staging in https://gitlab.com/gitlab-com/infrastructure/issues/4731, and nothing exploded. We also see events successfully populating in our Snowflake dw, so we should configure and turn on the integration in production as well.

In admin settings, we need to enable Snowplow and set the following values:
* Collector URI: `snowplow.trx.gitlab.net`
* Site ID: `gitlab`
* Cookie domain: `.gitlab.com` 

![snowplow](https://gitlab.com/gitlab-org/gitlab-ee/uploads/2b0bd6866097a38f7f985500578c00fe/snowplow.png)

Story points: 1.0
Environment running EE master refreshed nightly

### Problem
- We currently have `dev.gitab.org` running CE master which is refreshed nightly. In addition to serving as a production environment for GitLab security-related merge requests, it's a great place to test features that are already in master in a live environment with real data.
- `staging.gitlab.com` has early RCs of EE with GitLab.com production data, but it is never running master. So if you wanted to test out features already on master, you cannot do so if those features are not yet in an RC, which oftentimes, means several weeks later.
- Testing out features already on master is super helpful for pretty much everyone at GitLab. For PMs, we can review features, help out on QA, and do FA. This is especially helpful to see a new feature integrated with the latest and greatest code, and not on a branch that might not have the newest functionality. 
- Currently, the alternative is setting up GDK locally, which is not super easy to do for many GitLabbers.

### Proposal
- Have an environment running EE master.
- Having it refreshed nightly would be great, or at least weekly.
- Having some type of real data would be great, and having that data refreshed would be awesome as well. But that is less crucial, and should be a second iteration. Just having the environment with master running means users can create their own test data manually.

Story points: 1.0
Some users are having trouble performing a deploy

```
✖ Stopping chef-client on nfs, sidekiq, mailroom, web, api, git, registry, deploy-node (26s)
ssh_exchange_identification: Connection closed by remote host
ssh_exchange_identification: Connection closed by remote host
ssh_exchange_identification: Connection closed by remote host
ssh_exchange_identification: Connection closed by remote host
ssh_exchange_identification: Connection closed by remote host
ERROR: Errno::EPIPE: Broken pipe
/Users/jivanvl/gitlab/takeoff/lib/steps/base.rb:69:in `abort': Failed to execute command: bundle exec knife ssh -e -a fqdn 'roles:gprd-base-stor-nfs OR roles:gprd-base-be-sidekiq OR roles:gprd-base-be-mailroom OR roles:gprd-base-fe-web OR roles:gprd-base-fe-api OR roles:gprd-base-fe-git OR roles:gprd-base-fe-registry OR roles:gprd-base-deploy-node' 'sudo service chef-client stop' (ScriptError)
    from /Users/jivanvl/gitlab/takeoff/lib/steps/base.rb:60:in `post_checks'
    from /Users/jivanvl/gitlab/takeoff/lib/steps/base.rb:34:in `run!'
    from /Users/jivanvl/gitlab/takeoff/lib/step_runner.rb:44:in `block in run'
    from /Users/jivanvl/gitlab/takeoff/lib/step_runner.rb:42:in `each'
    from /Users/jivanvl/gitlab/takeoff/lib/step_runner.rb:42:in `run'
    from bin/takeoff-deploy:130:in `block in <main>'
    from /Users/jivanvl/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/semantic_logger-4.3.0/lib/semantic_logger/base.rb:355:in `measure_internal'
    from /Users/jivanvl/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/semantic_logger-4.3.0/lib/semantic_logger/base.rb:97:in `measure_info'
    from bin/takeoff-deploy:128:in `<main>'
```

At least two members have reported nearly the same error message.

Story points: 2.0
pgbouncer authentication is incorrectly configured in vault

pgbouncer has the incorrect credentials stored in our vault.  The next time a `gitlab-ctl reconfigure` is run on the db servers, is the next time we'll cause an outage as pgbouncer will not be able to properly authenticate with the master postgres node.

In order to close this issue:
1. Correct the password in our vault
1. Ensure chef has placed the proper credentials on it's next run
1. Ensure a `gitlab-ctl reconfigure` has completed on all database/pgbouncer servers
1. Validate pgbouncer is still successful at connecting to the master node
1. The above is done in both staging and production

Story points: 5.0
Update the infrastructure section of the handbook - first iteration

https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/ needs a refresh.
@andrewn has done a bit of work in https://gitlab.com/gitlab-com/infrastructure/issues/4675 which would serve as a great starting point to update the diagrams.  I think we should probably keep the scaffolding and just do a big update to make it relevant for GCP.

cc @gl\-infra @andrewn 


* `## On this page`
* `## Other Related Pages`
* `## Current Architecture` - needs an update
* `## Proposed Cloud Native Architecture` - needs an update
* `### GitLab.com Production Environment` - consolidate with current architecture
* `### High Level Components View` - Borrow diagrams from the readiness review?
* `### Pods Definition` - This can probably stay as-is but maybe could use an update?
* `### Database Architecture` - Needs an update
* `### Monitoring Architecture` - Needs an update
* `### Logging Architecture` - Either reference or put link to the logging page
* `## Infrastructure "Services" and Their SLx's` - I think we should remove this until we have a solid plan, cc @andrew
* `## Host Naming Standards` - Update for gcp
* `### Hostnames` - Update for gcp
* `### Service Tiers` - This is pretty much the same
* `### Environments` - Needs update
* `### Locations` - update for GCP
* `### TLD Zones` needs update for internal and .gitlab.net addresses
* `### Examples`- needs update
* `## Internal Networking Scheme` - replace with pointer to terraform
* `### Production` - remove
* `### Canary` - remove
* `### Staging` - remove
* `### GitLabGeoPrd` - remove
* `### GitLabGeoStg` - remove
* `### GitLabOps` - remove
* `### Remote Access` - Replace with pointer to howtos for bastion access
* `## Azure` - remove
* `### Load Balancers` - Remove, covered in high level diagrams
* `### Service Nodes` - remove
* `## Digital Ocean` - needs update
* `## AWS` - needs update
* `## Google Cloud` - remove as we are on gcp now
* `## Monitoring` - keep, reference to monitoring section
* `## Technology at GitLab` - more or less the same.

Story points: 1.0
Week 1 - test limited repos on GitLab.com with hashed storage

Week 1 issue for #4772 - 

Following: https://gitlab.com/gitlab-com/infrastructure/issues/4174 and as a step to https://gitlab.com/groups/gitlab-org/-/epics/75 I discussed some additional steps with @stanhu to have an initial rollout in gitlab dot com:

- First week: Let's enable it for new projects for a short period of time (~2 hours) and monitor

Coordinate with @stanhu and geo team for which repos and time window

Story points: 1.0
Change GITLAB_CDN_HOST to the IPv6-enabled endpoint on staging

**Reason:**  
[IPv6 support on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/645) is being prioritized. Static resources ought to also be reachable over IPv6.

**Problem:**  
Staging uses `gl-staging.global.ssl.fastly.net`/`gl-staging-canary.global.ssl.fastly.net` as the CDN endpoint, but that doesn't support IPv6.

**Fix:**  
Change the [`GITLAB_CDN_HOST`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/administration/environment_variables.md#supported-environment-variables) environment variable to `gl-staging.freetls.fastly.net`/`gl-staging-canary.freetls.fastly.net`. As an added benefit this will enable HTTP/2. See [Fastly's Free TLS IPv6 documentation](https://docs.fastly.com/guides/securing-communications/setting-up-free-tls#support-for-http2-ipv6-and-tls-12).

Story points: 1.0