giseldo/neodataset huggingface dataset:
2PC via pre-receive hook

As a result of the experiments in #2466 and #2529, we have concluded that the most promising way to implement strong consistency for reference updates is by going via Git hooks: given a reference update, a hook will execute on each Gitaly node that reports back to Praefect. Praefect will collect these reports from all Gitaly nodes that take part in the current update and, if all nodes post the same update, send them a message to go ahead.

While the mid-term goal is to hook directly into the reference transaction handling code in order to handle all kinds of reference updates and not only those invoked via git-receive-pack(1), this requires a new set of hooks on Git's side. We thus decided to implement a first POC of this mechanism by using the Git pre-receive hook, which executes after all reference updates have been announced by the Git client. This should give us a better picture of how the mechanism will work in the end.

The following diagram depicts the 3PC via a pre-receive hook:

```mermaid
sequenceDiagram
  Praefect->>+Gitaly: ReceivePack
  Gitaly->>+Git: git receive-pack
  Git->>+Hook: update HEAD master
  Hook->>+Gitaly: TX: update HEAD master
  Gitaly->>+Praefect: TX: update HEAD master
  Praefect->>+Praefect: TX: collect votes
  Praefect->>+Gitaly: TX: commit
  Gitaly->>+Hook: TX: commit
  Hook->>+Git: exit 0
  Git->>+Gitaly: exit 0
  Gitaly->>+Praefect: success
```

The 2PC voting protocol will start as soon as a first "TX" message is received on the Praefect node. Each of the pre-receive hooks will block until it receives a message from Praefect telling to to either go on with the update or to abort. In case the vote was successful, the hook will exit with `0` to indicate success, otherwise it will return an error code and thus abort the reference update.

The main goal of this issue is to establish a communication channel between hook and Praefect to allow for transaction handling. The communication channel should be implemented transparent to Gitaly nodes as much as possible so that Gitaly does not need to know if and how many transactions a given Git transaction is going to start. This ensures we can swap out the hooks in the future and start multiple transactions for a single Git command.

Story points: 7.0
Remove primary in praefect config once SQL election is merged and the default

SQL leader election has now been merged https://gitlab.com/gitlab-org/gitaly/-/issues/2547, meaning we no longer need the idea of a "default" primary in the config.

We should remove this in:

- [x] Praefect example config toml in gitaly
- [x] GDK
- [ ] Omnibus
- [x] Gitaly Cluster admin docs

Story points: 2.0
Option for failover to put out-of-date repositories in read-only mode

## Problem to solve

Praefect has a failover mechanism, where if it detects that a Gitaly node is unresponsive, it will promote a secondary storage server to primary. If this secondary server could be missing data still from the old primary. This in itself would allow Gitaly to serve stale data. However, after serveral mutations on the repository, it could end up in a state where data-loss occurs. As such, the current implementation is great for demo's, or instances where velocity is more important than data integrity. The current approach will not work for gitlab.com

## Solution

In <issue link> we're working on reporting of potential data loss, this issue proposes the next iteration of it; putting each of those repositories in a read only state, until manual intervention from an admin to mark it operational again. Allowing all administrators to use the current failover mechanism, and resolve data integrety issues manually.  


### Additional benefits

When moving a repository on a shard, it's put in read only mode for a brief period of time. This is a loosely defined contract right now. Having a praefect managed read only mode for repositories put stronger data integrity.

This approach could remove the need for https://gitlab.com/gitlab-org/gitaly/-/issues/2601

Story points: 5.0
Stale packed-refs.new files prevents branches from being deleted

We saw in https://gitlab.com/gitlab-org/gitlab/-/issues/213620#note_318864514, that somehow the `gitlab-org/gitlab` repository had a stale/aborted `packed-refs.new` file that prevented branches from being deleted inside the `MergeRequestWorker`, causing that worker to retry 4 times unnecessarily:

```
root@file-cny-01-stor-gprd.c.gitlab-production.internal:/var/log/gitlab/gitlab-rails# sudo ls -al /var/opt/gitlab/git-data/repositories/@hashed/a6/80/a68072e80f075e89bc74a300101a9e71e8363bdb542182580162553462480a52.git
total 34168
drwxr-s---   7 git root     4096 Apr  7 02:50 .
drwxr-s---   5 git root     4096 Dec 12 09:21 ..
-rw-r--r--   1 git root      409 Apr  7 00:05 config
-rw-r--r--   1 git root       73 Dec 12 09:17 description
-rw-r--r--   1 git root   156636 Apr  7 02:54 FETCH_HEAD
drwxr-sr-x   2 git root     4096 Apr  7 00:05 gitlab-worktree
-rw-r--r--   1 git root       23 Feb  5 07:35 HEAD
drwxr-sr-x   2 git root     4096 Dec 12 09:17 hooks
drwxr-sr-x   2 git root     4096 Apr  7 02:54 info
-rw-r--r--   1 git root   301751 Apr  7 02:13 language-stats.cache
drwxr-sr-x 260 git root     4096 Apr  7 02:56 objects
-rw-r--r--   1 git root 30427806 Apr  6 11:07 packed-refs
-rw-r--r--   1 git root  4055040 Apr  6 11:08 packed-refs.new
drwxr-sr-x  11 git root     4096 Dec 12 09:21 refs
root@file-cny-01-stor-gprd.c.gitlab-production.internal:/var/log/gitlab/gitlab-rails# date
Tue Apr  7 02:56:39 UTC 2020
```

Attached are the files:

[packed-refs.tar.gz](/uploads/d12f6d32ef38a805943bff296b25a722/packed-refs.tar.gz)

It almost looks like the writes to `packed-refs.new` got aborted at 11:08:

![image](/uploads/4e1d6697a1c911a5e0963dab6ed7a7b4/image.png)

Few questions/issues:

1. What left this `packed-refs.new` around? I presume it was a `git pack-refs`. I see a `PackRefs` RPC that completed successfully at 11:05: https://log.gprd.gitlab.net/goto/b30757c2d0feb12c7ea3dd7590d93760 
1. Should we run `Cleanup()` on stale `packed-refs.new` files?
1. https://gitlab.com/gitlab-org/gitlab/-/issues/213620: `MergeRequestWorker` should be idempotent here: why did the deletion of the branch cause duplicate merge commits? 
1. These error messages only went to `githost.log`: https://gitlab.com/gitlab-org/gitaly/blob/981ff509c05475c65480f11c92101f0a4d5f5158/ruby/lib/gitlab/git/operation_service.rb#L209. Can we just port this RPC to Go and use structured logging here?

Story points: 2.0
Documentation: two sources of truth for GL_REPOSITORY

When receiving a git push, Gitaly will make HTTP callbacks to gitlab-rails. This is done via the Git `pre-receive` and `post-receive` hooks.

In these callbacks, Gitaly must tell gitlab-rails what repo the hooks are running in. We use an opaque token called GL_REPOSITORY for this. This token comes in as an RPC request field, and gets handed down into the Git hook via an environment variable on the `git receive-pack` process.

There are three groups of RPC's where we need GL_REPOSITORY to make callback requests:

- SSHService.SSHReceivePack (git push via SSH)
- SmartHTTPService.PostReceivePack (git push via HTTP)
- OperationsService.UserCommitfiles, OperationsService.UserMergeBranch, etc. (RPC's that modify a repo on while impersonating a user)

In the case of OperationsService, we always pass GL_REPOSITORY [via the Repository protobuf message](https://gitlab.com/gitlab-org/gitaly/-/blob/dca26f3a6d09e3cea5d05373bd2c35ddc03c05c8/proto/shared.proto#L42).

In the case of SSHReceivePack and PostReceivePack, however, we have a confusing situation. These RPC's were implemented before we had a GL_REPOSITORY field on the Repository message. What we do there instead is that we explicitly pass in the value via the request message ([example 1](https://gitlab.com/gitlab-org/gitaly/-/blob/master/proto/smarthttp.proto#L82)) and [example 2](https://gitlab.com/gitlab-org/gitaly/-/blob/master/proto/ssh.proto#L66)).

The purpose of this issue is to discuss if this is a problem or not, and if so, what we want to do about it.

Story points: 2.0
Automatically clean up after moving a project to a different shard

## Problem to solve

Moving a project from one shard to another leaves behind the repo on the original shard, and it isn't cleaned up automatically. This means manual work is required to reclaim space

## Proposal

If a project is moved to a new shard successfully, the space should be reclaimed on the old shard by deleting the repo

## Links / references

Story points: 2.0
Praefect: Enable database replication queue by default

As of now by default Praefect uses in-memory implementation of the replication queue.  
This can be switched to Postgres implementation by providing configuration parameter [`postgres_queue_enabled`](https://gitlab.com/gitlab-org/gitaly/-/blob/cc314adf47d3eed96d824f35ae601e891e2e70e2/internal/praefect/config/config.go#L33) with value `true`.
Once Postgres implementation tested enough this parameter must be removed and Postgres implementation must become default.

/cc @zj-gitlab @jramsay

Story points: 2.0
Praefect: same storage name can't be used for different virtual storages

If you have setup with multiple virtual storages and each storage has node with name `node-1` only one node will be used for processing.  
Another node will be treated as duplicate and skipped, [here is the code](https://gitlab.com/gitlab-org/gitaly/-/blob/19fd0676954fd87cee2f0108efd3caaf20c5c266/internal/praefect/datastore/datastore.go#L168-170).

Expected: all nodes must be bound to proper virtual storage so all operations can be routed properly.

/cc @zj-gitlab @jramsay

Story points: 3.0
WalkRepos should handle a directory being deleted during file walking

[`service/internalgitaly.WalkRepos`](https://gitlab.com/gitlab-org/gitaly/-/blob/f3f4ae8bc2945b8363835e6c6862a7044311aa9e/internal/service/internalgitaly/walkrepos.go#L36) should handle a possible `not exists` errors. This can happen if a repository is deleted by another RPC during the file walk.

Story points: 1.0
Detect node failure by read/write failures

### Problem to solve

Health checks can often succeed even if there is a problem with the server, Gitaly or Git. For example, the Gitaly process may be running just fine and respond with a healthy signal even though the persistent disk is not available which would prevent any Git operation from succeeding.

### Further details

Health checks are relatively slow to detect failures (seconds), and are performed less frequently to the total number of Git operations. For example, a busy Gitaly node may be handle hundreds of requests per second (get commit, get file, list something etc). If multiple health checks must fail before failover, this means thousands of operations have failed before the system notices a failure.

### Proposal

Praefect should monitor Gitaly RPC success/failures and mark a node as bad if there multiple failues in a row. For example, if 5 or 10 RPCs fail in a row, this is a strong signal that the node isn't working.

Worst case scenario we fail over to an up to date replica because strong consistency means we always have an up to date replica https://gitlab.com/groups/gitlab-org/-/epics/1189.

### Testing

Functional end-to-end coverage will be provided by a new [failover test](https://gitlab.com/gitlab-org/quality/testcases/-/issues/836). And broad coverage by [running the entire end-to-end test suite against environments using Praefect for storage](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationpraefect-ceeefull-image-address) (including Staging).

Performance end-to-end testing will be implemented as part of https://gitlab.com/gitlab-org/quality/performance/-/issues/231 (see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/451 for a more detailed plan of performance tests under failure conditions).

### Links / references

Story points: 3.0
Expose node level metrics even if automatic failover is disabled

Currently, node metrics such as `gitaly_praefect_primaries` and `gitaly_praefect_node_latency` get populated when `failover_enabled = true`. We should populate these metrics regardless.

This involves refactoring the node manager so that it does healthchecks on the nodes even if `failover_enabled = false`.

Here is where we start the health checks: https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/praefect/nodes/manager.go#L164

The easiest way is to probably prevent node promotion with `failoverEnabled` in https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/praefect/nodes/manager.go#L164

Story points: 2.0
On startup, Praefect can't talk to gitaly nodes before it picks up replication jobs

During our demo on 3/20/20, we observed that when praefect is brought up when the sql replication queue has jobs in `ready`, praefect picks up the jobs immediately and tries to replicate, but failed because it couldn't talk to gitaly.

To reproduce:

1. bring praefect down
1. create replication jobs on the sql database
1. bring praefect up

in the logs, praefect will complain that it cannot talk to gitaly, and the jobs will fail.

Story points: 2.0
Drop Golang 1.12 support

With the release of Go 1.14, the support stops for Go 1.13 as per the Golang policy to support the last two releases. This means that Gitaly now supports being compiled by an unsupported Golang version.

Action items:
- [x] Remove Go 1.12 from our Pipeline
- [ ] Remove Go 1.12 from the docs at this project, and GitLab

Story points: 1.0
Enable partial clone with blob size filter by default

## Problem to solve

Once it is possible to enable blob and sparsity filters independently, we should enable blob filters by default since they are low risk and make it easy for customers to try out partial clone.

## Proposal

Currently we have one feature flag `gitaly_upload_pack_filter` to control both blob and sparsity filters.

- reuse the existing feature flag `gitaly_upload_pack_filter` for blob filters, and default to enabled
- create a new feature flag `gitaly_upload_pack_filter_sparsity` for sparsity filters, and default to false

## Links/references

Story points: 4.0
SQL for failover and leader election

## Problem to solve

It is important that multiple Praefect proxies/routers can be run at the same time so that there is no single point of failure. These need to be consistent in choosing the same primary node else different Praefects will route requests inconsistently causing data loss.

## Further details

There have been investigations in Consul https://gitlab.com/gitlab-org/gitaly/issues/2037 https://gitlab.com/gitlab-org/gitaly/issues/2458 for GitLab Omnibus, since this is a good application of Consul. However, Consul isn't likely to be ideal in a Kubernetes environment.

Since we already have a PostgreSQL database as part of the current architecture, and is seems like viable first iteration (spike https://gitlab.com/gitlab-org/gitaly/-/merge_requests/1883) we will use this for providing failover and leader election. We plan to keep open the possibility for supporting Consul and K8s features for these problems.

## Proposal

1. We do some lightweight refactoring to prepare the Praefect code to support both SQL and Consul
1. We aim to incorporate the SQL PoC approach first and roll out two Praefect nodes on GitLab.com.
1. Focus on getting failover metrics etc. so we have more data on how this performs

## Links / references

Story points: 6.0
Document how to keep Praefect's SQL schema up to date

Praefect has its own SQL database. Every time you deploy a new version of Praefect you need to make sure you run any pending SQL migrations.

The status quo in multi-node GitLab installations is that the local administrators must have their own process for running database migrations. In the gitlab-rails case this means running `rake db:migrate`. https://docs.gitlab.com/omnibus/update/#zero-downtime-updates

I think we should extend this documentation with information on how to run Praefect migrations.

Story points: 2.0
Add praefect replication delay metric

Per demo on 3/10, a useful metric would be the total time from when a replication job is created to when it is completed. This would give us total "replication delay".

Additionally, this metric should be graphed in Grafana: https://dashboards.gitlab.net/d/8EAXC-AWz/praefect?orgId=1&from=now-24h&to=now


Story points: 2.0
Add GetFullestShard, GetBusiestShard, GetFastestGrowingProject, GetBusiestProject grpc methods

As a gitlab admin, I would like to balance git repository storage between shards based on project activity and growth.

Gitaly now provides information about current disk usage and availability per shard.

The following GRPC methods could be very useful in selecting specific projects for relocation.

- `GetFullestShard`
- `GetBusiestShard`
- `GetFastestGrowingProject`
- `GetBusiestProject`

While this information *could* be examined using monitoring, it would be useful to integrate such informatics into the `gitlab-rails` app, in order to ultimately expose this information through the admin api, supporting admin tooling.

Story points: 8.0
Add praefect storage to the performance bar

## Problem to solve

When using Gitaly HA it is hard to reconcile what you see in the browser, with what is happening on the server. Which node is primary? If I stop a node, which node is promoted to primary. This would be easy to know if there was information in the performance bar about this.

## Proposal

Include information in the performance bar about which Gitaly node in the cluster responded to the request.

## Links / references

Story points: 2.0
Gitaly should perform better Rails-side validation

For example, this API query alone causes dozens of 500 errors per minute: https://log.gitlab.net/goto/bb1a0ad5013edecd5f0e31daa4c12df0

The Gitaly ruby client should do more basic client side validation, to ensure refs are valid (don't contain whitespace) etc. 

This will prevent a remote GRPC call, git spawn etc, only to error with a basic validation error. 

More examples of this: https://log.gitlab.net/goto/a5a4ce0c321fb5c3703286ced55d1df3

Story points: 2.0
Remove old praefect config virtual_storage_name and nodes

Once https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/3754 and gdk is updated with the change to allow multiple praefect virtual storages, we can remove the backwards compatible support in praefect.

also we need to remove the text fixture in `internal/praefect/config/testdata/single-virtual-storage.config.toml`

Story points: 2.0
Observe which projects suffered dataloss from failover

## Problem to solve

As an administrator, when a failover occurs, I want to know which repositories have experienced dataloss. This provides me an indication of which developers have been effected and where to target efforts for recovering missing data where possible. For example, letting a team know that they might need to re-push their feature branches.

## Further details

At a minimum, we should be able to which specific projects suffered dataloss, caused by replication operations that cannot be completed due to incomplete replication queue.

A possible future improvement could also be to summarize the number of replication jobs lost for each repo, and perhaps even count per ref, per repository.

## Proposal

Store failed replication jobs for some period of time in the Praefect database, and provide a method for reporting on this.

Perhaps the MVC, would be command that can be run on any Praefect server to:

- dump all the failed replication jobs (this could be useful for analysis)
- summarize failed replication jobs


Story points: 4.0
Use `--end-of-options` in the GitDSL

Requires the minimum version of Git 2.24: https://gitlab.com/gitlab-org/gitaly/issues/2170

From the GH release blog: https://github.blog/2019-11-03-highlights-from-git-2-24/#tidbits

>>>
Git 2.24 has a new way to prevent this sort of option injection attack using --end-of-options. When Git sees this as an argument to any command, it knows to treat the remaining arguments as such, and won’t interpret them as more options.

So, instead of the string previously mentioned, you could write the following to get the history of your (admittedly, pretty oddly named) branch:

$ git log --end-of-options --super-dangerous-option

Not using the standard -- was an intentional choice here, since this is already a widely-used mechanism in Git to separate reference names from files. In this example, you could have also written git log --end-of-options --super-dangerous-option ^master -- path/to/file to get only the history over that range which modified that specific file.
>>>

Example of the Git behaviour:

```
$ git diff --end-of-options --name-only HEAD~ HEAD
fatal: option '--name-only' must come before non-option arguments
```

Story points: 2.0
Add GetStorageDiskStatistics grpc

It may also be useful to implement a method for obtaining a collection of disk space statistics.  In fact, this could reduce overall load on the server, by avoiding the need to invoke multiple remote procedure calls to obtain separate details.

So this issue is to suggest the feature addition of a `grpc` resembling `GetStorageDiskStatistics` which would return a collection, ideally a dictionary, containing at least the disk `avail` and disk `used` portions of the output of a `df /var/opt/gitlab` invocation on the given target storage node system.

Story points: 3.0
Add GetStorageDiskUsed grpc

Currently, `gitlab-rails` acquires disk usage information for individual project repositories from `gitaly` which ultimately appears to use `du` (https://gitlab.com/gitlab-org/gitaly/blob/master/internal/service/repository/size.go#L36).

As an SRE, I am responsible for managing storage node configuration through the gitlab rails web application admin section, and also adding new nodes as needed.  Automating these procedures would be very helpful for our team.  To accomplish this, some data is required.

I would like to have reliable access to the disk `used` (amount of disk in current use) statistic.

Only as an example, not an implementation recommendation: `ssh file-34-stor-gprd.c.gitlab-production.internal 'df /var/opt/gitlab --output=used | tail -n1' #=> 13710976516`

After having spoken with @pokstad1 and @jacobvosmaer-gitlab , I think the best suggestion is to implement a `grpc` resembling `GetStorageDiskUsed` which will return the parsed results of such an `df` operation with the relevant detail or details.

Story points: 3.0
Add GetStorageDiskAvail grpc

Currently, `gitlab-rails` acquires disk usage information for individual project repositories from `gitaly` which ultimately appears to use `du` (https://gitlab.com/gitlab-org/gitaly/blob/master/internal/service/repository/size.go#L36).

As an SRE, I am responsible for managing storage node configuration through the gitlab rails web application admin section, and also adding new nodes as needed.  Automating these procedures would be very helpful for our team.  To accomplish this, some data is required.

I would like to have reliable access to the disk `avail` (amount of remaining -- unused -- disk space available for use) statistic.

Only as an example, not an implementation recommendation: `ssh file-34-stor-gprd.c.gitlab-production.internal 'df /var/opt/gitlab --output=avail | tail -n1' #=> 2933905600`

After having spoken with @pokstad1 and @jacobvosmaer-gitlab , I think the best suggestion is to implement a `grpc` resembling `GetStorageDiskAvail` which will return the parsed results of such an `df` operation with the relevant detail or details.

Story points: 3.0
Praefect as pass through proxy by default in the GDK

Currently one could run Praefect in the GDK, after running `rake praefect:enable`. Given we're confident that praefect works, especially if there's only one Gitaly storage node, the daemon should be enabled by default.

The thing to consider is that users don't like to reseed their GDK, as it takes long. So the changes should have good forward compatibility.

Story points: 2.0