giseldo/neodataset huggingface dataset:
PostgresSQL_XIDConsumptionTooLow in staging

Alert in staging
```
*postgres-06-db-gstg.c.gitlab-staging-1.internal*
 Postgres seems to be consuming transaction IDs very slowly

 TXID/s is 0.2 on postgres-06-db-gstg.c.gitlab-staging-1.internal:9187 which is unusually low. Perhaps the application is unable to connect
```

Possibly related to: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5402

Story points: 1.0
301 Redirect: handbook/product/pricing/ to /handbook/ceo/pricing

We've moved the page previously hosted at /handbook/product/pricing/ to /handbook/ceo/pricing/, and I'd appreciate some help setting up a redirect to catch any traffic pointed at the old URL. See https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/15799/diffs for reference.

This may be a redundant request as I thought I'd opened this issue already, but I didn't see anything when I went back and searched through open and closed issues. Apologies in advance if that's the case.

Thanks!

Story points: 1.0
Get Ephemeral Environments ready for Postgres work

The Environments project is nearly ready for use in deploying arbitrary temporary environments.  This task is to make the minimal changes necessary to make this project usable for deploying arbitrary environments.

The intention is to use these temporary workspaces to do work on Chef recipes for provisioning Postgres with patroni clustering. In the near term it will also be used for migrating terraform code to versioned modules.

This implements a first iteration of design document https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5285

Story points: 8.0
301 redirect for /2016/11/30/setting-up-gitlab-ci-for-android-projects/

Last week we published an updated and expanded version of an old blog post and need to redirect the link from the old post to the new post.

| 404 link | redirect URI |
| ------ | ------ |
| https://about.gitlab.com/2016/11/30/setting-up-gitlab-ci-for-android-projects/ | https://about.gitlab.com/2018/10/24/setting-up-gitlab-ci-for-android-projects/ |

Thanks!

Story points: 1.0
Import big projects for customers

## TL;DR  

Customers and potential ones may want to import a large project into GitLab.com. This isssue is to provide a workaround in a timely manner for them.

## Background

In the past, for big imports such as the K8S import, we had a dedicated instance that helped us achieve this.

Large imports, will either timeout (we have a timeout of a few hours) or will get killed by the Sidekiq Memory Killer. This is becoming a bit more frequent as the app grows larger, and the Sidekiq process eats more memory.

We improved the Import mechanism so it doesn't use as much memory at the cost of getting a bit slower (executing less transactions in a single commit, keeping less objects in memory). This can be tweaked, but either way, we'll either hit a memory issue or a timeout problem. The next big step would be to separate this into different independent workers in order to save memory, but it's not a small refactor.

## Why?

This will help Support/Sales

## Current workarounds

For a self-hosted instance, this is easy to workaround by either increasing the Sidekiq RSS memory allowed and/or disabling the worker that kills the imports when a certain timeout is reached.

For GitLab.com, we can't tweak that. So we normally get rid of `pipelines` or heavy objects in the exported project in order to free a bit more memory. But we could still encounter the problem.

## Example

I managed to import a ~900MB export in my DO instance, averaging an RSS of 850MB, peak at 950MB (note that practically half of this is just loading the Rails app). The default max RSS at GitLab.com is ~~1GB~~ (2GB now). But this wouldn't work at GitLab.com - What's the difference? The thread where the import ran in my DO instance wasn't shared by any other thread, as that was the only thing the process was doing (the other threads were free).

## Proposal

For prospects or customers, use the deploy node to run a script that does this for us, without the memory limits we would normally hit.

We'll need to:

1. (James) Provide a script that calls the I/E logic provided an export archive and a target (easy).
1. (James) Document somewhere how this works. We may need to use tmux/screen.
1. (Infra) Confirm this is OK. I would suggest every time we have to do this we ping the oncall.
1. (Support) Confirm the identity of the customer to verify access to the target namespace.
1. (Support) Ping the oncall when this is required from a customer.

## Alternative

Maybe we can automate this hooking it into ChatOps and use a runner with enough memory :thinking: But haven't put much thought about it.

## Related: 

https://gitlab.com/gitlab-com/support/dotcom/dotcom-escalations/issues/2 

https://gitlab.com/gitlab-org/gitlab-ce/issues/52956

cc @cynthia @dawsmith @lmcandrew

Story points: 1.0
Ephemeral Environments Demo Video

Rather than trying to schedule a bunch of zoom calls to show how CI/CD works for [Ephemeral Environments](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5285), I've created this video so that everyone can watch it when it's convenient.  I'm using this issue as a place to gather feedback. Please comment here, rather than youtube.

[![Environments-Video](/uploads/e1e29bf2015ac24b268fb2f233bb7393/Environments-Video.png)](https://www.youtube.com/watch?v=XCIz75ffhlY&t=6s)

A few things to note:

- This initial iteration is in the context of building a [POC environment for patroni](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5342), but it can be useful as an easy way to experiment with [Kubernetes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5158) and other things. 

- The Worflow in this demo doesn't yet take into account the [Desired git workflow](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5276) of using branches to create test environments.  Issue #5276 and it's merge requests are the appropriate place for that conversation.

- The code that the video skims over is visible at: https://ops.gitlab.net/gitlab-com/environments and if you are curious what is actually happening in the pipelines you can see the output at https://ops.gitlab.net/gitlab-com/environments/pipelines

- We could use a lot more discussion on how this should all work. The design doc issue is #5285 and it's merge requests are still pending comments.

- This project probably needs to move from my workspace to the ops instance

/cc @gitlab\-com/gl\-infra - please click a reaction so I can keep track of who has watched this.

Story points: 3.0
AWS admin access to staging for new SREs

Per comments in #5322, some new SREs do not have access to the `staging` AWS account, with account ID `4745-2183-0347`.

Please create IAM user accounts and assign the necessary role(s)/group(s) so that @ctbarrett, @skarbek, @dsylva, and @aamarsanaa are able to manage this account

/cc @gitlab\-com/gl\-infra

Story points: 1.0
Remove workaround to not send params to elasticsearch for the api_json.log

we currently strip the `params` field because of indexing issues on elasticsearch.
This issue is to remove that workaround when the fix makes it into a release, once https://gitlab.com/gitlab-org/gitlab-ce/issues/53155 .

Putting this in the next milestone but we may need to push it out further depending on how long it takes for the fix to land.

Story points: 1.0
rebalance repositories on storage shards

Currently we have a few storage shards that are much more full than others. I think we should consider rebalancing repos across shards to be more evenly spread out. Ideally everything will be less than 70%.

https://dashboards.gitlab.com/d/W_Pbu9Smk/storage-stats

cc/ @gitlab\-com/gl\-infra for your consideration

Story points: 5.0
create alert for failing consul service check

We should always have one postgresql service check passing in consul for our postgres HA topology

https://dashboards.gitlab.net/d/a988f2tmz/consul?orgId=1&from=now-30m&to=now&var-environment=gprd&var-prometheus=prometheus-01-inf-gprd&var-pg_service_nodes=All&var-pg_node_status=All&var-consul_server=consul-01-inf-gprd.c.gitlab-production.internal

If we don't, this should definitely page the oncall as it means we will be unable to failover properly after a failure.

Story points: 2.0
postgresql service check is failing on production

It looks like this has been failing for some time:

https://dashboards.gitlab.net/d/a988f2tmz/consul?orgId=1&from=1535089028546&to=1540313847444&var-environment=gprd&var-prometheus=prometheus-01-inf-gprd&var-pg_service_nodes=All&var-pg_node_status=All&var-consul_server=consul-01-inf-gprd.c.gitlab-production.internal

What this means is that we are probably never going to succeed with a failover which puts our database HA topology at risk.

The consul definition for the postgresql service:

```
"service":{"name":"postgresql","address":"","port":5432,"checks":[{"script":"/opt/gitlab/bin/gitlab-ctl repmgr-check-master","interval":"10s"}]},"watches":[{"type":"keyprefix","prefix":"gitlab/ha/postgresql/failed_masters/","handler":"/opt/gitlab/bin/gitlab-ctl consul watchers handle-failed-master"}]}
```

on postgres-01 in production:

```
gitlab-+ 41999  2353  0 14:43 ?        00:00:00 /bin/sh -c /opt/gitlab/bin/gitlab-ctl repmgr-check-master
gitlab-+ 42000 41999  0 14:43 ?        00:00:00 /bin/bash /opt/gitlab/bin/gitlab-ctl repmgr-check-master
gitlab-+ 42001 42000  0 14:43 ?        00:00:00 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/omnibus-ctl gitlab /opt/gitlab/embedded/service/omnibus-ctl* repmgr-check-master
gitlab-+ 42083  2353  0 14:44 ?        00:00:00 /bin/sh -c /opt/gitlab/bin/gitlab-ctl repmgr-check-master
gitlab-+ 42084 42083  0 14:44 ?        00:00:00 /bin/bash /opt/gitlab/bin/gitlab-ctl repmgr-check-master
gitlab-+ 42085 42084  1 14:44 ?        00:00:00 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/omnibus-ctl gitlab /opt/gitlab/embedded/service/omnibus-ctl* repmgr-check-master
gitlab-+ 42251  2353  0 14:45 ?        00:00:00 /bin/sh -c /opt/gitlab/bin/gitlab-ctl repmgr-check-master
gitlab-+ 42252 42251  0 14:45 ?        00:00:00 /bin/bash /opt/gitlab/bin/gitlab-ctl repmgr-check-master
gitlab-+ 42253 42252  2 14:45 ?        00:00:00 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/omnibus-ctl gitlab /opt/gitlab/embedded/service/omnibus-ctl* repmgr-check-master
root     42357 42192  0 14:45 pts/2    00:00:00 grep repmgr-check-master
```

It appears this command is just hanging on production.

Credentials are correct so the problem is with the command:

```
jarv@postgres-01-db-gprd.c.gitlab-production.internal:~$ sudo -u gitlab-consul /opt/gitlab/embedded/bin/psql -d gitlab_repmgr -h /var/opt/gitlab/postgresql  -p 5432 -U gitlab-consul
could not change directory to "/home/jarv": Permission denied
psql (9.6.8)
Type "help" for help.

gitlab_repmgr=>
```


Story points: 2.0
Refactor PagerDuty Cog Functions into GitLab ChatOps

There's little sense in putting effort into rewriting the Cog PagerDuty function that we all use as that's what it would require given the changes in gems, API, and the abandonment of Cog.  Rather than refactor the function into something we need to dispose of, it needs to be written in format commutable with our CI driven ChatOps.

Current Requirements for matched functionality are:
- [x] It should accept a bare request for 'on call' and return who is on call for all services.
- [x] It should accept a service name or partial name match and return who is on call for the service.

Story points: 4.0
Investigation for new haproxy draining deployment

The new haproxy draining logic was used for the first time and it appears to be successful as we did not see a normal spike in errors during deployment.

There are a few other anomolies that we should investigate to follow-up.  These traffic changes may be related to healthchecks which due account for a large amount of workhorse and backend traffic. The spikes in api traffic are still a bit of a mystery.



* Traffic spikes on the api at 07:47:45
![Screen_Shot_2018-10-22_at_10.54.25_AM](/uploads/0a7c91af005d81b9271c085c22be1305/Screen_Shot_2018-10-22_at_10.54.25_AM.png)


* Dip in workhorse web traffic, web 2xx and web 3xx at 07:40:25 and ~07:53 and ~07:55
![Screen_Shot_2018-10-22_at_10.54.02_AM](/uploads/b88f0005340f1ee1418f3eab337428ce/Screen_Shot_2018-10-22_at_10.54.02_AM.png)
![Screen_Shot_2018-10-22_at_10.53.39_AM](/uploads/d4046e00b2d44b6e3a53cb9868accacc/Screen_Shot_2018-10-22_at_10.53.39_AM.png)
![Screen_Shot_2018-10-22_at_10.53.11_AM](/uploads/01147c37dc4ea0a7fb823a46b99c3026/Screen_Shot_2018-10-22_at_10.53.11_AM.png)

Story points: 1.0
301 Redirect for partners

Previously, we redirected `/partners/` to `/applications/` but `/partners/` should be the path. `/integrations/` should `/partners/integrate/` https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/15698 fixes the paths. We need some 301s put in place and the deployment coordinated. 

## Create chef MR
- [x] remove 301 for `/partners` to `/applications`
- [x] create 301 for `/applications` to `/partners`
- [x] create 301 for `/integrations` to `/partners/integrate`

## Deploy list
- [x] Stop chef client
- [x] Merge chef MR
- [x] Merge website MR https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/15698
- [x] When website MR is close to deployed, start chef client to deploy the 301s

Story points: 2.0
Create chef cookbook for Patroni Install and setup

* Install the package 
* Setup each node
  * Add all the actual parameters from production cluster
  * Setup all the monitoring metrics for the new postgresql instances ( move / duplicate the actual ones )
  * Define rules for autofailover ( patroni.yml)
  * Create pgbouncer users
* Create /check mechanism to restart in case of the service dies ( runit / initd/ systemd)
* Create monitoring for the service, each node and health status.
* Create process to setup the first node getting data from Prod ( cookbook ) 
  * Could be using pg_basebackup 
  * Could be using e-wally
* Create process to create patroni nodes ( cookbook )  
Pgbouncer : 
* Add all the actual parameters from production cluster
* Create /check mechanism to restart in case of the service dies ( runit / initd)
* Setup all the monitoring metrics for the new pgbouncer instances ( move / duplicate the actual ones )
* Create/check cronjob to updage the access of new users to connect by pgbouncer to avoid direct connections or authquery.
* Update Watcher script to consul to update the pgbouncer entry points for pgobuncers to master database only.

The ongres document how to install is here : 
https://docs.google.com/document/d/1NTqnRzT-elPEmPyq_sNC80sULFsCUXX7F10EVOFyZK0/edit?usp=sharing



Story points: 4.0
Create chef cookbook for consul server Install

* Install standard version of Consul server. 
* Create /check mechanism to restart in case of the service dies ( runit / initd)
* Initially setup the cluster without autospawn of instances ( create 5 servers considering 3 for consensus for consul).
* Add monitoring metrics of each consul server and his health.
* Usage of vips meaning Ip’s reserved. In gcp we should use Static Internal Ip’s.
  * Seems we need to setup this in the startup of the server. 
  * The first node of consul - the bootstrap - is a good practice to have a fixed IP.
* Explore if Instance group makes is vital for the first iteration.


- Can we use be terraform consul module?
https://registry.terraform.io/browse?provider=google  
https://registry.terraform.io/modules/hashicorp/consul/google/0.3.0

Story points: 1.0
Database Reviews

* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22430#note_109954169
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7806#note_109966838
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8000#note_109977095
* [x] https://gitlab.com/gitlab-org/gitlab-ce/issues/45522#note_110188925

* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22380#note_109456537
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22404
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8012
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7990
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22482
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22563
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22433#note_110337738
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22307
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22143
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22307
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22292#note_111772401
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22292#note_111766801
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22143
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22713
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22606 and https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8105
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8016
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22482#note_113802530
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22013#note_110236879
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6947#note_111236178
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22694
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22433/diffs#note_113259877
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22522 / https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8144/diffs#note_113443391
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6878#note_113587368 
* [x] Monday/Kamil https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22380
* [x] Monday/Kamil https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22433
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22795

Story points: 8.0
301 GitHub URL

https://about.gitlab.com/comparison/github-com-vs-gitlab.html redirects to 404. 

A mistake I made on [line 161](https://docs.google.com/spreadsheets/d/17cU2VUlIIaw9LEU1VnPuNrA2uBos3UAD8CItZP9ESzo/edit#gid=0)

/comparison/github-com-vs-gitlab.html	/devops-tools/github-com-vs-gitlab.html

Should be 

/comparison/github-com-vs-gitlab.html	/devops-tools/github-vs-gitlab.html

@ahanselka can you fix?

Story points: 2.0
Get packaged Omnibus downloads counts for September '18

Please obtain the packaged Omnibus downloads count for the month of September, 2018.

- Instructions are in [usage statistics sheet](https://docs.google.com/spreadsheets/d/1ujEmxvQQXjqFHHylZUwJw8RdvLdGBnKFzZLwtAB7dYo/edit#gid=463675429) shows (among others). Goal is cell AU3.

![image](/uploads/b73b0fe569450ac1182d6f2ffa42b94c/image.png)

Thanks a lot!

Story points: 1.0
Add alerting for repositories mirrored on github

Assigning to @nolith since he solved https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/3703. @erushton, please reassign in case there is someone more 
appropriate.

CC @marin to investigate if this is related to a recent release

CC @sytses @markpundsack @eliran.mesika @bjung @dzaporozhets @axil @dhavens @pritianka 

Story points: 4.0
Set up a redirect for /handbook/people-operations/group-conversations

We've renamed Functional Group Updates (FGUs) to Group Conversations.

In [this merge request](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/15506) you'll see that I've updated the URL for the People Operations page on Group Conversations and attempted to fix any links broken as a result, but I think we should set up a redirect just in case other links point to the page that I can't see.

The page /handbook/people-operations/functional-group-updates/ should redirect **to** /handbook/people-operations/group-conversations/

Story points: 1.0
Delivery: MTTP Metric

Define, develop and track `Mean-Time-to-Production` (MTTP) metric as a KPI for both Infrastructure as a department and Scalability as a team. Iterate as necessary.

Update URL, Health, Maturity and Next Steps on the Handbook's [Infrastructure Performance Indicators](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#mean-time-to-production- mttp) page as necessary.

Story points: 2.0
Update to chef 14.x

* Update chef-client to v14.x on the `pre` environment: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1558
* Identify and fix all issues on `pre`: No issue identified
* Deploy all fixes to `pre`: No fixes required
* Verify `pre` works: :white_check_mark: 
* Deploy across all our environments all non-breaking fixes: No fixes required
* Update chef-client to v14.x and breaking fixes across all our environments: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1559

First step of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7406

Story points: 3.0
rsyslog not creating system log files at bootstrap

Noticed while building sidekiq nodes for https://gitlab.com/gitlab-com/gl-infra/production/issues/963: after initial bootstrap, /var/log/syslog, /var/log/auth.log and so on did not exist, even though rsyslog was running.  It required a further restart of rsyslog (or a reboot) for them to be created and logged to.  This means system logs are neither logged locally, nor shipped to ElasticSearch, until manual activity occurs.

I have done no further investigation yet, no idea how simple or complex this will be to fix.

Story points: 3.0
Review PgBouncer Pool configuration and architecture

Related issue [1014](https://gitlab.com/gitlab-com/gl-infra/production/issues/1014), from July 31st.

It is being observed that the current Database Pool configuration may be exhausting current Datanode CPU and memory capacity. That is, the settings of the number of connections allowed from each node and pool can be potentially harmful to the leader's performance.

Right now, `max_connections` in the cluster is set at 300, a conservative number yet very above what is the best theoretical connection/throughput relation peak. The way that we calculate the _theoretical connection capacity_ is: `(cores / % effective usage) * scale_factor`. Effective usage is the client busy percentage, which can be calculated around 95% as this is a queue processing; scale_factor is a coefficient between 2 and 4. Within this, the best suitable amount of maximum _active_ connections would be around 130 and 200.

But, in the PgBouncer side, we currently have two nodes, each with 3 pools (2 intensive: production and sidekiq). Production pool is 50 and sidekiq is 75 on each node, meaning that there are potentially 250 active connections. 

From OnGres we need to:

- Define the pool size on sidekiq and production for reducing the performance degradation occurrences that happened during the last days. 
- Splitting pools can offer better resilience when one of the pools is generating waits that affect other queues in the node. 
- Revisit other PgBouncer configuration. For instance: 
  - Current: `min_pool_size = 0`, recommended `min_pool_size = 20`. This opens a minimum amount of persistent connections, decreasing any possible startup time when issuing new connections.
- If the split is necessary, establish the necessary amount of nodes per each pool and its configuration.


Related graphs:

https://prometheus.gprd.gitlab.net/graph?g0.range_input=2w&g0.expr=sum(sidekiq_queue_size)%20by%20(fqdn)&g0.tab=0
https://dashboards.gitlab.com/d/9GOIu9Siz/sidekiq-stats?orgId=1&fullscreen&panelId=71&from=now-10d&to=now


IOWait on Pgbouncer https://dashboards.gitlab.com/d/PwlB97Jmk/pgbouncer-overview?from=1564563814425&to=1564571034282&fullscreen&panelId=6

Story points: 0.0