giseldo/neodataset huggingface dataset:
Fixup reference-transaction hook name based on arguments

While the reference-transaction hook was released with Git v2.28.0, it's got a bug which will cause Git to accidentally execute the wrong hook if there's been interleaving calls to both the reference-transaction and another hook. This bug currently prohibits us to use the hook for reference transaction voting until a new Git version is released with the upstreamed fix. Given that this could be as late as release v2.29.0, it's potentially still quite a while until we can use the hook.

As a temporary workaround, we may manually fixup hook names in our `gitaly-hooks` binary. GitLab systems only have at most four hooks installed: `reference-transaction`, `pre-receive`, `update` and `post-receive`. Because of how these hooks get called, we should be able to tell the case when Git meant to execute the `reference-transaction` hook instead of any of the other three by inspecting the hook's arguments:

- `reference-transaction` will receive as first arguments one of `prepared`, `committed` or `aborted`
- `pre-receive` never gets any arguments
- `update` will get as first argument the reference that is to be updated in its fully-qualified form (e.g. `refs/heads/master`) and thus should never clash with any of the three above verbs
- `post-receive` never gets any arguments

As a consequence, none of these hooks except for the `reference-transaction` hook will ever get executed with one of `prepared`, `committed` or `aborted`. If they were executed with such an argument, we can be sure that Git wanted to execute the `reference-transaction` hook instead.

By making use of this info, we should be able to fix up the hook name without any casualties and unblock ourselves until Git v2.29.0 is released with a fix.

Story points: 3.0
Ignore common name from x509 certificates

A temporary mitigation of a breaking change was introduced in https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2472.

This should be removed in the next major version of GitLab. Additionally, all users of the Gitaly Go-client should be upgraded.

Relates to https://gitlab.com/gitlab-org/gitaly/-/issues/3036

Story points: 2.0
Re-include reference-transaction hook

The reference-transaction hook has been removed from Gitaly due to a regression in Git v2.28.0, causing the wrong hooks to be invoked with inputs from the reference-transaction hook if there's interleaving hook calls. We need to re-include the reference-transaction hook as soon as Git v2.28.1 has been released with a fix to this issue.

Depends on https://gitlab.com/gitlab-org/gitlab-git/-/issues/1144

Story points: 2.0
Call reference-transaction hook from Ruby sidecar

While we already call the pre-receive, update and post-receive hooks manually in the Ruby sidecar, we don't do so for the reference-transaction hook. This is a necessity when we want to drop the pre-receive hook in favor of the reference-transacton hook, though.

This depends on the re-inclusion of Gitaly's reference-transaction hook and thus on the release of Git v2.28.1.

Story points: 2.0
Rename white/blacklist references

```
❯ git grep whitelist
internal/praefect/replicator.go:308:    // whitelist contains the project names of the repos we wish to replicate
internal/praefect/replicator.go:309:    whitelist map[string]struct{}
internal/praefect/replicator.go:342:            whitelist:       map[string]struct{}{},
internal/praefect/replicator.go:372:// WithWhitelist will configure a whitelist for repos to allow replication
internal/praefect/replicator.go:373:func WithWhitelist(whitelistedRepos []string) ReplMgrOpt {
internal/praefect/replicator.go:375:            for _, repo := range whitelistedRepos {
internal/praefect/replicator.go:376:                    r.whitelist[repo] = struct{}{}
internal/rubyserver/proxy.go:15:// Headers prefixed with this string get whitelisted automatically
internal/rubyserver/proxy.go:86:                // Automatically whitelist any Ruby-specific feature flag
internal/rubyserver/proxy_test.go:26:   require.False(t, ok, "outgoing MD should not contain non-whitelisted key")
internal/rubyserver/proxy_test.go:43:   require.Equal(t, []string{value}, outMd[key], "outgoing MD should contain whitelisted key")
internal/rubyserver/proxy_test.go:60:   require.Equal(t, []string{value}, outMd[key], "outgoing MD should contain whitelisted feature key")
```

Gitaly should not use black vs white, but instead use block and allow lists.

Story points: 1.0
Praefect server metadata not resolved before passing to Ruby sidecar

When receiving a call with Praefect server metadata, we need resolve contained connection information first based on the current peer's connection information. This will fill in the remote's IP address, which is something that may change depending on how Praefect has connected to us.

While we already to this for "normal" calls via `PraefectFromContext()`, we don't for proxied calls intended for our Ruby sidecar. As a result, the sidecar may receive invalid connection information which it cannot use. To fix this, we need to resolve address information on Gitaly's side and inject resolved metadata into the Ruby sidecar's metadata.

References https://gitlab.com/gitlab-org/gitlab/-/issues/233838

Story points: 3.0
Praefect subcommands miss user-visible documentation

When executing `praefect -help`, we list the switches and subcommands available to Praefect. While switches have some short documentation available, none of the subcommands actually explain what they're doing and why an administrator would want to use them. As many of them will probably be of use in stressful scenarios where there's an incident happening, having some documentation handy for the admin as part of `praefect -help` would definitely be very helpful.

Story points: 2.0
Praefect distributed reads counter has mislabeled storage

During the 2020-07-07 demo, we observed strange behavior from the distributed reads counter (see area in red rectangle):

![Screen_Shot_2020-07-07_at_9.40.20_PM](/uploads/a3292e5a89dcd5d1d5b5f896736641e7/Screen_Shot_2020-07-07_at_9.40.20_PM.png)

Note how all three counters are labeled with the same storage `gitaly-1`.

The counters originate from three different Praefects. The objective of read distribution is to distribute reads across multiple storages of the same virtual storage. We expect to see each Praefect distribute reads to different storages. Given enough time, we expect to see counters labeled with all storages (in this case there were three: gitaly-1, gitaly-2, gitaly-3).

Story points: 2.0
Implement support for subtransactions

With the upcoming reference-transaction hook implemented in gitlab-org/gitlab-git!2, we need to extend transactions to support subtransactions. Right now, we know that each transaction will be invoked once only from the pre-receive hook. But with the reftx-hook, each Git operation may call out to a given transaction arbitrarily many times. As we cannot know how often that's going to happen, the transaction needs to cope and allow for multiple votes.

This is where subtransactions will come in. Each vote of a given node is going to create a new subtransaction iff all previous subtransactions for that node have succeeded. If the current transaction is still ongoing or has failed, it is not allowed to take part in new subtransactions. The outcome for each given node is then either `success` if all subtransactions have reached quorum on the same thing as the node voted for, `failure` if any subtransaction arrived at a different thing or `not started` if none of the nodes has created a subtransaction.

Story points: 4.0
Improve the performance of CountCommits by using commit-graph

(Placeholder issue, Gitaly team know details more than I do)

In https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/470, the main performance issue with the Gitaly cny node appears to be the CountCommits GRPC method.

@chriscool pointed out that this could be sped up using `core.commitGraph`

```
$ git config core.commitGraph true
$ time git rev-list --count --all
67482
real    0m0,100s
user    0m0,088s
sys     0m0,012s
$ git config core.commitGraph false
$ time git rev-list --count --all
67482
real    0m0,587s
user    0m0,567s
sys     0m0,020s
```

If possible, we should consider enabling this. 

cc @zj-gitlab @pokstad1 @chriscool

Story points: 3.0
Refactoring of the listening address configuration

The following discussion from !2276 should be addressed:

- [ ] @8bitlife started a [discussion](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2276#note_362382231): 

    > I think we should refactor it and allow to set a list of addresses that needs to be served with mandatory schema each `listen = ['tcp://0.0.0.0:2609', 'tls://0.0.0.0:2609', 'unix:///opt/var/praefect.socket']`.   
    > It will simplify out code and we won't need to support both: 'tcp://0.0.0.0:2609' and '0.0.0.0:2609' for same `listen_addr` and in a couple of other places: `PraefectServer` type

/cc @zj-gitlab

Story points: 2.0
Filter environment variables when executing custom hooks

Our hooks use a set of environment variables to pass along information across processes that's required to be operational. While we cannot help using these variables, we should make sure to filter out any envvars when executing custom hooks such that these custom hooks won't accidentally start depending on internal variables that weren't meant to be public in the first place.

Thus, executing custom hooks with the equivalent of `execve(3P)` would avoid us breaking customer's scripts if they unknowingly started using internal variables, allowing us to keep iterating here without fear of breakage.

I'm labelling this as a bug as exposing internal variables to custom hooks is unintentional at best.

Story points: 3.0
Push to gitlab.com fails with zeroPaddedFilemode: contains zero-padded file modes

Customer has started to mirror some of their repos from their local gitlab (self-managed) to the new remote one (gitlab.com), but running into an issue with one of their repositories, Jquery, which returns this error: 

```
13:close stream to gitaly-ruby: rpc error: code = Unknown desc = Gitlab::Git::CommandError: remote: error: object a98f1b7a267865bb78ef2a54ed54b67fbad6c839: zeroPaddedFilemode: contains zero-padded file modes       
remote: fatal: fsck error in packed object       
error: remote unpack failed: index-pack abnormal exit
To
 ! [remote rejected]   master -&gt; master (unpacker error)
! [remote rejected]   3.x-stable -&gt; 3.x-stable (unpacker error)
! [remote rejected]   secure-buildFragment -&gt; secure-buildFragment (unpacker error)
! [remote rejected]   1.12-stable -&gt; 1.12-stable (unpacker error)
! [remote rejected]   2.2-stable -&gt; 2.2-stable (unpacker error)
! [remote rejected]   feature/Patch.2.2 -&gt; feature/Patch.2.2 (unpacker error)
! [remote rejected]   mgol/test-flakiness -&gt; mgol/test-flakiness (unpacker error)
error: failed to push some refs to &#39;[FILTERED]@gitlab.com/<customername>/3rdpartytools/jquery.git&#39;
```
They also tried mirroring this manually as follows: 

```
C:\projects\jquery [master ≡]> git remote rename origin old-origin
C:\projects\jquery [master ≡]> git remote add origin https://gitlab.com/<customername>/3rdpartytools/jquery.git
C:\projects\jquery [master ≡]> git push -u origin --all
Enumerating objects: 37001, done.
Counting objects: 100% (37001/37001), done.
Delta compression using up to 4 threads
Compressing objects: 100% (9856/9856), done.
Writing objects: 100% (37001/37001), 21.80 MiB | 1.80 MiB/s, done.
Total 37001 (delta 26510), reused 36878 (delta 26419)
remote: error: object a98f1b7a267865bb78ef2a54ed54b67fbad6c839: zeroPaddedFilemode: contains zero-padded file modes
remote: fatal: fsck error in packed object
error: remote unpack failed: index-pack abnormal exit
To https://gitlab.com/<customername>/3rdpartytools/jquery.git
! [remote rejected]   master -> master (unpacker error)
error: failed to push some refs to 'https://gitlab.com/<customername>/3rdpartytools/jquery.git'
C:\projects\jquery [master ≡]>
```
The following are two issues where it looks like the error was reported before, but no clear indication as to the resolution without having to necessarily repair the source repo.

- [Pushing fails if local repo has problems with zero-padded file modes - 22095](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22095)
- [Pushing fails if local repo has problems with zero-padded file modes - 1602](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/1602)

Story points: 5.0
Implement voting strategies for the transaction manager

The transaction manager currently supports a single voting strategy, only: all registered nodes need to cast the same vote, otherwise the transaction will be aborted. In order to support more scenarios and become more resilient to failures, we want to implement additional strategies:

- Implement a mechanism to reach quorum such that a subset of nodes may fail.
- Implement weighted voting such that it's possible to say e.g. that a primary node may never fail, but a subset of secondaries may.

Story points: 4.0
On hosts without the 'en_US.UTF-8' locale, output from popen_with_timeout will be US-ASCII encoded rather than UTF-8

Gitaly sets [environment variable `LANG=en_US.UTF-8`](https://gitlab.com/gitlab-org/gitaly/-/blob/v13.0.5/internal/command/command.go#L27-30) for all child git processes, as well as Gitaly-Ruby. On hosts that do not have the `en_US.UTF-8` locale installed this causes Ruby to fall back to `US-ASCII` (7-bit) encoding by default, even if they are using a different UTF-8 locale.

As a result, UTF-8 encoded output from [`popen_with_timeout`](https://gitlab.com/gitlab-org/gitaly/-/blob/v13.0.5/ruby/lib/gitlab/git/popen.rb#L50) will trigger `ArgumentError: invalid byte sequence in US-ASCII` errors when string parsing methods are executed on it. A customer experienced this with [`sanitize_url`](https://gitlab.com/gitlab-org/gitaly/-/blob/v13.0.5/ruby/lib/gitaly_server/utils.rb#L74-76) when `UpdateRemoteMirror` returned an error, but there are probably other places this could occur as well.

On a host using `de_DE.UTF-8` as its locale:

```sh
$ LANG=de_DE.UTF-8 /opt/gitlab/embedded/ruby -e 'p Encoding.default_external'
#<Encoding:UTF-8>

$ LANG=en_US.UTF-8 /opt/gitlab/embedded/ruby -e 'p Encoding.default_external'
#<Encoding:US-ASCII>

$ locale -a
C
C.UTF-8
de_DE.utf8
POSIX

$ locale -v
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
```

On a host using `en_US.UTF-8` and `de_DE.UTF-8` installed both work:

```sh
$ LANG=en_US.UTF-8 /opt/gitlab/embedded/ruby -e 'p Encoding.default_external'
#<Encoding:UTF-8>

LANG=de_DE.UTF-8 ruby -e 'p Encoding.default_external'
#<Encoding:UTF-8>

LANG=ar_EG.UTF-8 ruby -e 'p Encoding.default_external'  # Using another UTF-8 encoding as an example
#<Encoding:US-ASCII>

$ locale -a
C
C.UTF-8
POSIX
de_DE.utf8
en_US.utf8

$ locale -v
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES=en_US.UTF-8
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
```

Story points: 3.0
Remove multi-byte  support in the `lines` package

The lines package takes some input, and splits it on a delimiter, and sends the output as defined by a `Sender()`. There's currently support for multi byte delimiters, which is both unused, and broken.

Broken because for example using twice the same byte as delimiter will not send any results. There's currently no call site using this feature, nor test coverage for it. Therefor we should remove it.

Story points: 2.0
Lock worktrees while they are still in use

The following discussion from !2202 should be addressed:

- [ ] @pks-t started a [discussion](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2202#note_353463326):  (+5 comments)

> I also wonder whether it would make sense to start using worktree locks. Currently, the logic will always remove worktrees older than a given threshold. If the threshold is small enough, I could see us removing worktrees that are still in use currently. Iff worktrees are strictly tied to a given operation (are they?), then we could tell the operation to first lock the worktree via git worktree lock <MSG> and consequentially unlock the worktree after the operation has finished. That'd guarantee that we do not delete any worktrees that are still in use. The message could also be chosen in such a way that we know what operation had it locked and when it was locked to give additional context. Naturally, it could happen that a worktree still doesn't get removed cause the operation failed mid-way through and didn't unlock, but we could have another, higher, threshold for that.

- @johncai notes:

> We could easily add something in the `def with_worktree` function that adds a worktree lock. If this change makes sense on its own, maybe we can create a follow up issue?

Story points: 3.0
Omnibus Gitaly dashboard should support multiple nodes

During the 2020-05-27 demo, it was observed that the Omnibus Gitaly dashboard does not show more than one Gitaly at a time. Some of the panels, such as the current version, don't even support more than one Gitaly node (should instead be a table). The dashboard only allows viewing Gitalies with job set to "gitaly", while Praefect Gitaly nodes have a job of "praefect-gitaly". This should also be an option when selecting Gitalies.

Story points: 1.0
Praefect: Coordinator.StreamDirector doesn't handle STORAGE scope correctly

Current implementation of [`StreamDirector`](https://gitlab.com/gitlab-org/gitaly/-/blob/d08b7024a0a2882dc55d8a480d891fc0ded5bb9b/internal/praefect/coordinator.go#L217) doesn't properly handles requests with `scope_level: STORAGE` option, such as [`RemoveNamespace`](https://gitlab.com/gitlab-org/gitaly/-/blob/d08b7024a0a2882dc55d8a480d891fc0ded5bb9b/proto/namespace.proto#L16) in case virtual storage name and storage name for gitaly differs, like: 
```
[[virtual_storage]]
name = 'praefect'
[[virtual_storage.node]]
  storage = "gitaly-1"
```
Coordinator doesn't re-write storage for such RPCs, so request will contain virtual storage name instead of mapped gitaly storage name.

There is an open https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2234 that can be used to verify this bug.  
You can check it with `gdk` and `TestPraefectHandlesStorageScopeRequests` test locally.

/cc @zj-gitlab @jramsay

Story points: 3.0
Configure Gitaly to schedule nightly repo optimization

We have an RPC `OptimizeRepository` that is designed to be called to improve performance of repos. Currently, it is not being utilized by Rails. It may be very complex to figure out the best way to invoke it from Rails. A more incremental approach might be invoking it internally from Gitaly on a routine schedule (e.g. nights, weekends) similar to a cron job.

Story points: 2.0
Connection caching in ReplicateRepository breaks due to v2 tokens

In https://gitlab.com/gitlab-org/gitaly/-/commit/54465275b27178af0268d1281a7f933585f29b0f we dropped support for v1 tokens. V2 tokens have a timestamp constraint. 

It seems that this has broken replication in production. It's unclear if this is due to clock skew, but we are getting some `timestamp too old` errors

Story points: 3.0
Praefect migration to drop gitaly_ tables

We have old migrations in some of our environments with the outdated `gitaly_` tables. We should write a migration that removes these. eg: here is the production environment:

```
praefect_production=> \d+
                                     List of relations
 Schema |               Name                |   Type   |  Owner  |    Size    | Description
--------+-----------------------------------+----------+---------+------------+-------------
 public | gitaly_replication_queue          | table    | default | 8192 bytes |
 public | gitaly_replication_queue_id_seq   | sequence | default | 8192 bytes |
 public | gitaly_replication_queue_job_lock | table    | default | 8192 bytes |
 public | gitaly_replication_queue_lock     | table    | default | 8192 bytes |
 public | hello_world                       | table    | default | 8192 bytes |
 public | node_status                       | table    | default | 8192 bytes |
 public | node_status_id_seq                | sequence | default | 8192 bytes |
 public | replication_queue                 | table    | default | 8192 bytes |
 public | replication_queue_id_seq          | sequence | default | 8192 bytes |
 public | replication_queue_job_lock        | table    | default | 8192 bytes |
 public | replication_queue_lock            | table    | default | 8192 bytes |
 public | schema_migrations                 | table    | default | 16 kB      |
 public | shard_primaries                   | table    | default | 8192 bytes |
 public | shard_primaries_id_seq            | sequence | default | 8192 bytes |
(14 rows)
```

Story points: 1.0
Improve Gitaly connection error messages

A customer using a stand-alone Gitaly server recently ran into an issue where Gitaly-Ruby processes could not communicate with the main Gitaly process when executing `UserCommitFiles`.  Specifically, DNS was not responding when querying for the Gitaly hostname, but the error `rpc error: code = Unavailable desc = failed to connect to all addresses` gave no indication where the problem was occurring.

Including the target that could not be connected to would make problems like this significantly easier to debug.

/cc @zj-gitlab

Story points: 2.0
Enable read-only mode only when there is data loss after a failover

## Problem to solve

Currently read-only mode is enabled after any failover event. Read-only Git access is a reduction in availability and should be avoided where possible.

## Proposal

We should only enable read-only mode if the virtual storage has data loss, that is, any repositories for which the last replication job is not in `completed` state.

cc @zj-gitlab

Story points: 5.0
Replication queue size should have virtual storage label

The Prometheus metric `gitaly_praefect_replication_jobs` should be broken down by virtual storages. This will allow us to compare replication queue pressure between virtual storages on the same Praefect or to aggregate virtual storage queue pressure between Praefects. This requires adding a label for the virtual storage to the metric.

Relates to https://gitlab.com/gitlab-org/gitaly/-/issues/2659

Story points: 2.0
Explore use of Git's trace2 mechanism

When spawning Git executables, we currently cannot evaluate what Git actually does except for observing its runtime and usual stdout/stderr. In order to allow better traceing of performance and events in Git, the trace2 mechanism was introduced in v2.22. We should evaluate whether hooking into this mechanism makes sense to get more insight into Git, improve visibility and ultimately to allow for better debugging of pathological cases.

Story points: 3.0
Documentation: How to remove the filterspec for a local repository

During the Partial clone demo of 2020-05-12, the question was asked how to remove the filterspec from a repository and `git pull` to ensure everything is present. Currently the documentation do not provide this.

### Proposal

Add a section to the documentation: https://docs.gitlab.com/ee/topics/git/partial_clone.html on how to unfilter a repository.

/cc @chriscool @jramsay

Story points: 1.0