giseldo/neodataset huggingface dataset:
Praefect: Use millisecond precision for JSON logs time field

### Problem to solve

A customer [reported](https://gitlab.com/gitlab-com/account-management/emea/manomano/-/issues/53) that times in Praefect logs are not standardized. Some items have millisecond precision (thanks to https://gitlab.com/gitlab-org/gitaly/-/merge_requests/1887), but others don't.

From the customer:
```
We are trying to ingest logs on Datalog and are facing dates on logs that aren't standardized.  Example on the same file (/var/log/gitlab/praefect/current):

{"gitaly":15927,"level":"warning","msg":"forwarding signal","signal":23,"time":"2020-12-09T15:24:49Z","wrapper":15937}
{"correlation_id":"M2qIAurFTg7","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.method":"FindCommit","grpc.request.deadline":"2020-12-09T14:39:10Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommit","grpc.service":"gitaly.CommitService","grpc.start_time":"2020-12-09T14:38:41Z","grpc.time_ms":4.497,"level":"info","msg":"finished streaming call with code OK","peer.address":"REDACTED:11670","pid":15927,"relative_path":"@hashed/e7/f6/REDACTED.git","span.kind":"server","system":"grpc","time":"2020-12-09T14:38:41.010Z","virtual_storage":"distributed"}
```

Notice that the first `time` does not have millisecond precision, but the last `time` does.

### Proposal

Praefect log entries should have times with millisecond precision (ISO 8601.3).

Story points: 1.0
Enforce default limit for recursive calls to GetTreeEntries

## Background

In some repositories with very large trees (https://gitlab.com/gitlab-org/gitlab/-/issues/232072), recursive calls to GetTreeEntries can consume too much system resources to complete.

## Proposal

Enforce a default limit for the number of tree entries that can be recursively fetched by GetTreeEntries. Require an explicit override for requests that want more than the default limit.

Story points: 3.0
[Feature flag] Enable Go implementation of FetchSourceBranch

## What

Enable the `gitaly_go_fetch_source_branch` feature flag, which switches over from the Ruby implementation of FetchSourceBranch to the Go implementation. FetchSourceBranch is used 

- when creating merge requests across forks, FetchSourceBranch is used to get changes from the source repo into the target repo.
- when doing cross-repository diffs.

The feature flag has been rolled out in the past but caused severe performance regressions https://gitlab.com/gitlab-org/gitaly/-/issues/2741#note_374138071. There's been some changes to mitigate those regressions and avoid needless duplication of work when objects are already contained in the target repository https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2980.

## Owners

- Team: Gitaly
- Most appropriate slack channel to reach out to: `#g_create_gitaly`
- Best individual to reach out to: @pks-t

## Expectations

### What release does this feature occur in first?

%13.2 via f0a5bdd7f04fb2819b62ed7afd466daabf672d15, improved performance in %13.8 via f382aa9588e4c818a4a70eefe7b3a8ebf790f91c.

### What are we expecting to happen?

No change in behaviour is expected. Ideally, performance metrics for `FetchSourceBranch` shouldn't change.

### What might happen if this goes wrong?

- Errors when creating merge requests from forks.
- Errors when diffing across repositories.
- Latency regresses.

### What can we monitor to detect problems with this?

- https://dashboards.gitlab.net/d/000000199/gitaly-feature-status?var-environment=gprd&var-method=FetchSourceBranch

## Roll Out Steps

- [x] [Read the documentation of feature flags](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-flags)
- [x] Add ~"featureflag::staging" to this issue
- [x] Is the required code deployed? ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#is-the-required-code-deployed))
- [x] Enable on staging ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-on-staging))
- [x] Test on staging ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#test-on-staging))
- [ ] Ensure that documentation has been updated
- [x] Announce on this issue an estimated time this will be enabled on GitLab.com
- [x] Add ~"featureflag::production" to this issue
- [x] Enable on GitLab.com by running chatops command in `#production` ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-in-production))
- [ ] Cross post chatops slack command to `#support_gitlab-com` and in your team channel
- [x] Announce on the issue that the flag has been enabled
- [x] Remove feature flag and add changelog entry
- [ ] Close this issue

Story points: 1.0
Fix  resolving Praefect address in mixed-TLS deployments

In deployements where Gitaly and Praefect have different TLS configurations, we may fail to correctly resolve injected Praefect addresses. The root cause is that in `resolvePraefectAddress`, we decide how to resolve the peer address by inspecting the incoming TCP connection: if it is unencrypted, we dial back with an unencrypted connection. If it is encrypted, we dial back with an encrypted connection. This doesn't make a lot of sense though, as we're deciding how to dial back depending on how we're reachable ourselves as a Gitaly node. We really shouldn't care for that and instead only go by the injected information we got.

Story points: 3.0
Praefect should only dial to Gitaly for transactions

For transactions, we require that Gitaly dials back to Praefect in order to vote on transactions. This introduces a few problems:

- Gitaly is not configured with Praefect's security token. Praefect must send call-back information, including security token, to Gitaly for all transaction related calls.
- It is not clear to customers how this dial back requirement affects supported network topologies for GitLab deployments. Sometimes, Praefect may be behind a load balancer or NAT. Being able to dial back to Praefect is not always desirable.

However, Praefect is already configured with the Gitaly node security tokens and it is clear that Praefect must be able to reach individual Gitalies on the network. Therefore, it would be more robust to have Praefect initiate the gRPC call to Gitaly for all transaction votes. This would free Gitaly from needing to know network and security details for Praefect since Praefect is already a trusted client of Gitaly.

Story points: 3.0
Improve input validation of hook inputs

The ported UserMergeBranch RPC caused pipelines to not get triggered in production anymore. The root cause was that the PostReceive hook was executed with empty input, where it should've contained a list of all updated references. As a result, the hook pretended no changes existed and thus no pipelines had to be triggered.

This error was hard to diagnose via logs or metrics as this is essentially about the absence of something. If we had stricter input verification in our hooks (e.g. expecting at least one change to exist), then the issue might've been a lot more visible and thus caught earlier.

Story points: 2.0
Add FetchSHA abstraction to LocalRepository

Originally reported in https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2693#note_435231130

Fetching a single SHA is done in multiple places across Gitaly and is very verbose and error prone. Having a definitive and easy to use function will improve our code quality. Adding a new method `FetchSHA` to `localrepo` could enable this.

Story points: 2.0
Inject single client connection pool from main to all Gitaly services

Currently, each service instantiates their own client pool for use in dialing across Gitalies. To improve performance by reducing connection dialing, the same pool should be used everywhere. This ideally should be dependency injected from the main module.

Originally reported in https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2693#note_435231124

Story points: 2.0
Enable verbose logging of git2go subcommands

Currently, git2go subcommands contain very complex logic for various tasks (e.g. squashing commits/resolving conflicts/commiting multiple changes). The subcommands lack observability since STDERR is used to return error messages. An alternative file descriptor should be used to allow git2go commands to relay back fine grain logs. When a git2go command exits with non-zero, or the logging level is set to a certain level, these fine grain logs should be emitted.

Story points: 2.0
Refactor NewFeatureSets to remove unused error return value

The test helper function `NewFeatureSets` is not capable of returning a non-nil error. The helper should be fixed to never return an error and all affected code locations should be updated to accept a single return value. Any error assertions for that function call should be removed.

The function is defined here: https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/testhelper/testhelper.go#L959-980

Story points: 1.0
GIT_DIR missing in hook environment

### Summary

As reported in https://gitlab.com/gitlab-org/gitlab/-/issues/258828, the `$GIT_DIR` is not present anymore in the environment for custom server hooks.

I was able to reproduce by following the instructions at https://docs.gitlab.com/ee/administration/server_hooks.html#create-a-global-server-hook-for-all-repositories. The `$GIT_DIR` variable is not set at all for any of the `pre-receive`, `post-receive`, and `update` hooks (this was on a normal project code repository, I assume the same applies for other repository types).

This seems related to recent work to port the hooks from Ruby to Go. I can see the variable was set in the Ruby code at https://gitlab.com/gitlab-org/gitaly/-/blob/dfd9c3e1a79f6a222b742b1978e45785b06a9440/ruby/lib/gitlab/git/hook.rb#L133.

### Proposal

`$GIT_DIR` is not explicitly documented at https://docs.gitlab.com/ee/administration/server_hooks.html#environment-variables, but for compatibility reasons we probably want to keep supporting it, and should also document this.

We might also want to double-check that we're not missing any of the other variables that were set in the previous Ruby code: https://gitlab.com/gitlab-org/gitaly/-/blob/dfd9c3e1a79f6a222b742b1978e45785b06a9440/ruby/lib/gitlab/git/hook.rb#L118-138

Story points: 1.0
Follow-up from "Pass correlation_id over to gitaly-ssh"

The following discussion from !2530 should be addressed:

- [ ] @ash2k started a [discussion](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2530#note_428727639):  (+3 comments)

    > @8bitlife 
    > 
    > Just discovered that these interceptors have been added. I understand the motivation but now I cannot customize neither of those interceptors:
    > 
    > - For the tracing one I'd like to inject a custom `Tracer` by using `grpc_opentracing.UnaryClientInterceptor(grpc_opentracing.WithTracer())` directly, not via LabKit (see the discussion here https://gitlab.com/gitlab-org/labkit/-/merge_requests/66)
    > - For the correlation one I'd like to set the client name like that `grpccorrelation.StreamClientCorrelationInterceptor(grpccorrelation.WithClientName("kas"))` as suggested here https://gitlab.com/gitlab-com/gl-infra/readiness/-/merge_requests/46#note_428087602 by @jacobvosmaer-gitlab.
    > 
    > I have looked at the code and both Workhorse are Shell add these interceptors to the client:
    > 
    > - Workhorse: https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/master/internal/gitaly/gitaly.go#L158-173 As you can see, it sets client name there.
    > - Shell: https://github.com/gitlabhq/gitlab-shell/blob/master/internal/handler/exec.go#L112-L120 Also sets the name but is missing the tracing interceptor (maybe intentionally).
    > 
    > So the current behavior is not ideal - two traces/spans from the same place (right?) and redundant context creation because the correlating interceptor is executed twice (so it adds the correlation ID twice).
    > 
    > p.s. another bug that I've noticed is that `connOpts` is mutated. It's not safe to append to a slice that has been passed as the caller likely assumes that the method is not mutating it. The slice might have larger capacity and so this method will mutate the original backing array. The code should copy it first.
    > 
    > p.p.s. It might be nicer to change the signature to varargs to match `grpc.Dial()` and all the other places where we take functional options. This will also address the bug ^.

@8bitlife @zj-gitlab

Story points: 1.0
Intercept CreateRepository in Praefect to set up repository database state

Currently we do not have a record of every repository a virtual storage is supposed to contain. This will change with #3033. We should intercept `CreateRepository` calls and set up the expected database records a repository should have. Related to #3133, we should select a primary for the repository when it is created. Related to #2971, we should select the nodes that will host the repository when it is created. Additionally, we can make generation related queries stricter once we can expect every repository to be present in the database and not allow for example incrementing generation of a repository that does not exist. Stricter queries prevent certain class of errors, such as https://gitlab.com/gitlab-org/gitaly/-/issues/3183, and make the logic simpler as we don't have to handle cases where the records are missing.

Story points: 2.0
UserMerge RPC should error with FailedPrecondition on merge conflicts

### Problem

When a user merges a branch while it can't be merged, the Gitaly service will detect this, and return an error. This error is an Unknown error currently, which is surprising to operators of Gitaly. Additionally, the error is wrong as the why of the error is known.

Last reason it should be another error, is that Unknown influences SL{A,O}s. Though it shouldn't.

### Solution

Gitaly should return a FailedPrecondition when it can't be merged due to conflicts.

Story points: 1.0
CreateRepositoryFromSnapshot will fail with error 'Promoting temporary directory failed' if parent directories of target path do not already exist

If `CreateRepositoryFromSnapshot` is called when the parent directory of the target path does not exist, the request will fail with error:

```
rpc error: code = Internal desc = Promoting temporary directory failed: rename /var/opt/gitlab/git-data/repositories/+gitaly/tmp/repo446863582 /var/opt/gitlab/git-data/repositories/@geo-temporary/@hashed/
19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git: no such file or directory
```

Stracing this, we find that the `renameat` call fails with ENOENT:

```
18192 22:01:24.772985 renameat(AT_FDCWD, "/var/opt/gitlab/git-data/repositories/+gitaly/tmp/repo446863582", AT_FDCWD, "/var/opt/gitlab/git-data/repositories/@ge
o-temporary/@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git") = -1 ENOENT (No such file or directory) <0.000013>
```

Generally this RPC is only called on a second attempt to create a repository, so the directory has already been created, but due to https://gitlab.com/gitlab-org/gitlab/-/issues/9803#note_422414267 these calls were the first attempt, exposing this issue.

`ReplicateRepository` will [ensure the directories are present](https://gitlab.com/gitlab-org/gitaly/-/blob/d76d10b540d474d950103aa3303a89a9aad9dd88/internal/gitaly/service/repository/replicate.go#L142-144) before renaming the tmp directory, we should do the same in [this RPC](https://gitlab.com/gitlab-org/gitaly/-/blob/d76d10b540d474d950103aa3303a89a9aad9dd88/internal/gitaly/service/repository/create_from_snapshot.go#L106-112).

Story points: 1.0
Gitaly module version cannot be updated in gitlab-workhorse and gitlab-shell

On a clean Go v1.15.2 installation, I get this while trying to update the Gitaly module in gitlab-workhorse and gitlab-shell:

```
$  go get -u gitlab.com/gitlab-org/gitaly@v13.5.0-rc1
go: downloading gitlab.com/gitlab-org/gitaly v0.0.0-20201001041716-3f5e218def93
go: gitlab.com/gitlab-org/gitaly v13.5.0-rc1 => v0.0.0-20201001041716-3f5e218def93
cat go.mod
go get: inconsistent versions:
	gitlab.com/gitlab-org/gitaly@v0.0.0-20201001041716-3f5e218def93 from gitlab.com/gitlab-org/gitaly@v13.5.0-rc1 requires gitlab.com/gitlab-org/gitaly@v1.68.0 (not gitlab.com/gitlab-org/gitaly@v0.0.0-20201001041716-3f5e218def93 from gitlab.com/gitlab-org/gitaly@v13.5.0-rc1)
```

According to https://github.com/golang/go/issues/35732 and https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning, this is because Go is trying to enforce semantic compatibility:

>>>
In semantic versioning, changing the major version number indicates a lack of backwards compatibility with earlier versions. To preserve import compatibility, the go command requires that modules with major version v2 or later use a module path with that major version as the final element. For example, version v2.0.0 of example.com/m must instead use module path example.com/m/v2, and packages in that module would use that path as their import path prefix, as in example.com/m/v2/sub/pkg. Including the major version number in the module path and import paths in this way is called "semantic import versioning". Pseudo-versions for modules with major version v2 and later begin with that major version instead of v0, as in v2.0.0-20180326061214-4fc5987536ef. 
>>>

I think this is saying that Gitaly should be using `m/v13` in the module path.

A workaround may be to tag another v1 tag with the latest master:

>>>
Code written before the semantic import versioning convention was introduced may use major versions v2 and later to describe the same set of unversioned import paths as used in v0 and v1. To accommodate such code, if a source code repository has a v2.0.0 or later tag for a file tree with no go.mod, the version is considered to be part of the v1 module's available versions and is given an +incompatible suffix when converted to a module version, as in v2.0.0+incompatible. The +incompatible tag is also applied to pseudo-versions derived from such versions, as in v2.0.1-0.yyyymmddhhmmss-abcdefabcdef+incompatible. 
>>>

v1.87.0 was tagged in February 2020, and then we switched versioning schemes.

@pokstad1 @samihiltunen What do you think we should do here?

Story points: 2.0
Gitaly HTTP client should specify TLSClientConfig to support mutual TLS auth

Gitaly currently doesn't specify a TLSClientConfig for the HTTP client used to make requests to the GitLab internal API. This prevents interoperation with services that require mutual TLS auth (e.g. nginx). The non-gRPC clients in Gitaly should be updated to use the configured Gitaly certificates for client connections.

Story points: 2.0
Leverage core.bigFilesTreshold for better memory utilization

From `man git-config`:

```
       core.bigFileThreshold
           Files larger than this size are stored deflated, without attempting delta compression. Storing large files without delta compression avoids excessive memory usage, at the slight expense of
           increased disk usage. Additionally files larger than this size are always treated as binary.

           Default is 512 MiB on all platforms. This should be reasonable for most projects as source code and other text files can still be delta compressed, but larger binary media files won’t be.

           Common unit suffixes of k, m, or g are supported.
```

Gitaly doesn't set the `core.bigFileThreshold`, so it defaults to 512MiB. I think `50m` would be a better threshold and creates a better trade off for us. As I think the number of source files that are over 50MB is limited, so until that point each blob will still be compressed, and they get optimized packs for it. 

This config should be dynamically inject on the relevant Git commands/RPCs to improve `*UploadPack` performance, as well as `Repack*` performance.

* [Tradeoffs from a user perspective](https://kscherer.github.io/linux%20git/2014/09/26/git-server-option-bigfilethreshold)

Story points: 2.0
Reference transaction hook votes twice per reference update

The reference-transaction hook will be called for every transition in Git's reference transaction mechanism. Currently, this will include exactly two transitions: either from `START -> prepared -> committed` or `START -> prepared -> aborted`. As our own hook will ignore which state we're transitioning to, we'll thus call out to the hook service twice and not pass along which state we've transitioned to. As a result, the hook service will vote twice per each reference update.

One could argue that this is a feature and not a bug in order to assert that we've actually performed both transitions as expected. But in any case, voting in the second transition from `prepared -> {committed,aborted}` is quite useless to us as Git will ignore the Git hook's return value in these transitions. After all, the transaction has already been wrapped up, so there's no way it can be changed after the fact.

As a result, we should stop voting in the second transition and only vote when moving to prepared state.

Story points: 2.0
No use of checksum comparison after replication is done

Once the replication is done (`update` of the repository at the gitaly storage) praefect runs [`CalculateChecksum`](https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/praefect/replicator.go#L263) for source and target gitaly nodes.  
The result of the execution compared and if there is no match the error message would be written into the log (example):
```json
{
      "msg": "checksums do not match",
      "hostname": "praefect-01-stor-gprd",
      "shard": "default",
      "level": "error",
      "primary": {
        "storage_name": "file-praefect-02",
        "relative_path": "@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git"
      },
      "replica": {
        "relative_path": "@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git",
        "storage_name": "file-praefect-01"
      },
      "environment": "gprd",
      "pid": 32412,
      "time": "2020-09-03T20:58:57.840Z",
      "tier": "sv",
      "type": "praefect",
      "fqdn": "praefect-01-stor-gprd.c.gitlab-production.internal",
      "tag": "praefect",
      "stage": "main"
}
```

This happens pretty often:
Start of the replication: ![image](/uploads/45c46f2939516b0ce4d5913a810ee493/image.png)  
Errors about not matching checksums: ![image](/uploads/86763ea999d149bc58e23af9599f12d7/image.png)

As a first step we could properly propagate `correlation_id` to all logging entries to get more info about what is going on as it is not used in all places where we log something.  

It is not clear if we should take any actions in regard of different checksums for the repository.

/cc @zj-gitlab @derekferguson

Story points: 1.0