giseldo/neodataset huggingface dataset:
Remove get-commit-signatures feature flag

`GetCommitSignatures` has been rewritten using Go in https://gitlab.com/gitlab-org/gitaly/issues/1604. The new RPC is under a feature flag and already [enabled in gitlab.com](https://gitlab.com/gitlab-org/gitaly/issues/1907). The feature flag can now be removed from both gitaly and gitlab-ce.  

#### TODO 

- [ ] Remove the feature flag from go first: https://gitlab.com/gitlab-org/gitaly/merge_requests/1484
- [ ] Remove feature flag from gitlab-ce: https://gitlab.com/gitlab-org/gitlab/merge_requests/16550

Story points: 1.0
Remove ruby implementation for GetCommitSignatures RPC

GetCommitSignatures RPC has been rewritten using Go in https://gitlab.com/gitlab-org/gitaly/issues/1604. [The old RPC that uses ruby can be removed](https://gitlab.com/gitlab-org/gitaly/blob/master/internal/service/commit/commit_signatures.go#L37).  

Method signature:  

``` go
func rubyGetCommitSignatures(s *server, request *gitalypb.GetCommitSignaturesRequest, stream gitalypb.CommitService_GetCommitSignaturesServer) error {
```  

## TODO

- [ ] Remove `rubyGetCommitSignatures` go function
- [ ] Fix broken specs
- [ ] if possible remove all ruby code related to`GitalyServer::CommitService#get_commit_signatures`

Story points: 1.0
Roll out get_commit_signatures feature flag

<!-- Title suggestion: [Feature flag] Enable description of feature -->

## What

Remove the `get_commit_signatures` feature flag

## Owners

- Team: Gitaly
- Most appropriate slack channel to reach out to: `#g_gitaly`
- Best individual to reach out to: @felipe_artur 

## Expectations

Commit signatures request should use use go version of the RPC  
which is expected to be more performant.

### What release does this feature occur in first?

12.3

### What are we expecting to happen?

Better performance of the RPC

### What might happen if this goes wrong?

Worst case scenario: Stop showing if commit signatures are verified  
because of some exception.  

We may not see commits page broken because that RPC is made async.

### What can we monitor to detect problems with this?

* https://dashboards.gitlab.net/d/000000199/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=GetCommitSignatures&var-job=gitaly&refresh=5m  

* https://sentry.gitlab.net/gitlab/gitaly-production/

* https://sentry.gitlab.net/gitlab/gitaly-production/?query=GetCommitSignatures

## Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

- `gitlab-org/gitlab-ce`/`gitlab-org/gitaly` projects
- `gitlab-org`/`gitlab-com` groups
- ...

## Roll Out Steps

- [x] [Read the documentation of feature flags](https://docs.gitlab.com/ee/development/rolling_out_changes_using_feature_flags.html)
- [x] Enable on staging
- [x] Test on staging
- [x] Ensure that documentation has been updated
- [x] Enable on GitLab.com for individual groups/projects listed above and verify behaviour
- [x] Announce on the issue an estimated time this will be enabled on GitLab.com
- [x] Enable on GitLab.com by running chatops command in `#production`
- [ ] Cross post chatops slack command to `#support_gitlab-com` and in your team channel
- [x] Announce on the issue that the flag has been enabled
- [x] Remove feature flag and add changelog entry

Story points: 1.0
Remove `ensure_storage_path_exists` from gitlab-ce

as discussed in https://gitlab.com/gitlab-org/gitaly/issues/1807#note_208197834, the first step to deprecating the Namespace service is to get rid of `ensure_storage_path_exists` calls

There are two parts to this issue:

- Modify RenameNamespace to create the parent directory of the destination dir
- Remove calls to `ensure_storage_path_exists` calls in gitlab-ce

Story points: 1.0
Remove deprecated message handling code

The following discussion from !1410 should be addressed:

- [ ] @zj-gitlab started a [discussion](https://gitlab.com/gitlab-org/gitaly/merge_requests/1410#note_205645542): 

    > Post merge I'll create an issue to remove this code in %"12.4"

Remove in %"12.4", because gitlab-ce/gitlab-ee must be using the new `:messages` param (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31640 is currently targeting %"12.3").

Story points: 1.0
Deprecate ListDirectories RPC

This RPC is only used in cleanup rake tasks, as far as I can tell. E.g. https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/cleanup.rake#L7

This is problematic for Gitaly HA because we there will be directories on multiple different Gitaly servers and we don't want to directly expose those Gitaly servers.

We should take a look at the things that use ListDirectories, and reimplement whatever it is they do inside Gitaly. In some cases it might be automated cleanup; in other cases we might need a new RPC. In the latter case that RPC should express what we want to happen, and leave the actual work to Gitaly.

Story points: 1.0
remote: warning: ignoring extra bitmap file when pulling from a deduplicated repo

Seen today on the gitlab-org/gitlab-pages repository:

```
$ git fetch upstream master
remote: warning: ignoring extra bitmap file: /var/opt/gitlab/git-data/repositories/@pools/81/17/811786ad1ae74adfdd20dd0372abaaebc6246e343aebd01da0bfc4c02bf0106c.git/objects/pack/pack-1d59d5e04b0538cfc7fb23125f81c317aafd0b10.pack
```

I think we try to avoid displaying server-side paths to the client in other cases. Other than that, it didn't seem to affect anything - the fetch succeeded and had all the objects I expected.

Story points: 4.0
Remove Rugged patches from gitlab

We have been forced to walk back part of the Gitaly migration project because it turned out that in certain scenarios, the old gitlab-ce code that used Rugged is still significantly more performant than the new Gitaly code. Specifically, this is the case for some GitLab installations that host their Git repository data on slow NFS servers.

The code in question can be found here: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/git/rugged_impl

This is blocking us from completing the Gitaly migration project. These rugged patches should be removed again. This issue is meant to track their removal.

## Current Plan of Record for 14.3 and beyond

_Summary of the discussion in the below thread_

We don't believe that the Rugged patches are necessary for our customers utilizing NFS as we believe that [performance issues necessitating these patches are mitigated](https://gitlab.com/gitlab-org/gitaly/-/issues/1690#note_584475955).

As the Rugged patches create an additional maintenance concern for the Gitaly team, as well as not supporting the direction of moving customers requiring HA towards Gitaly Cluster. We therefore feel that they should be removed.

Starting in %"14.3", we will default to disabling the Rugged patches. As stated above, we don't believe that this will cause any performance issues for our users.

### Mitigations pending customer issues

- The Rugged patches are currently behind a feature flag. Starting in %"14.3" we will be defaulting this feature flag to disable Rugged, but we will not be removing the feature flag. If customers notice performance degradation, they can re-enable the Rugged patches via the feature flag. 

- Pending significant customer performance issues, the current plan is to remove the Rugged patches a couple of releases past %"14.3"  and the associated feature flag.

Story points: 1.0
GetCommitSignatures' messages can be too large

https://sentry.gitlab.net/gitlab/gitlabcom/issues/736414/

```
GRPC::ResourceExhausted: 8:Received message larger than max (6302368 vs. 4194304)
  grpc/generic/active_call.rb:31:in `check_status'
    fail GRPC::BadStatus.new_status_exception(
  grpc/generic/active_call.rb:181:in `attach_status_results_and_complete_call'
    recv_status_batch_result.check_status
  grpc/generic/active_call.rb:170:in `receive_and_check_status'
    attach_status_results_and_complete_call(batch_result)
  grpc/generic/active_call.rb:338:in `each_remote_read_then_finish'
    receive_and_check_status
  gitlab/gitaly_client/commit_service.rb:440:in `each'
    response.flat_map do |message|
...
(158 additional frame(s) were not displayed)

Gitlab::Git::CommandError: 8:Received message larger than max (6302368 vs. 4194304)
  gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
    raise Gitlab::Git::CommandError.new(e)
  gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
    def wrapped_gitaly_errors(&block)
  gitlab/metrics/instrumentation.rb:161:in `block in wrapped_gitaly_errors'
    .measure { super }
  gitlab/metrics/method_call.rb:36:in `measure'
    retval = yield
  gitlab/metrics/instrumentation.rb:161:in `wrapped_gitaly_errors'
    .measure { super }
...
(150 additional frame(s) were not displayed)

Gitlab::Git::CommandError: 8:Received message larger than max (6302368 vs. 4194304)
```

Story points: 2.0
CalculateChecksum: invalid memory address or nil pointer

Seen 553 instances of this in 4 hours of customer's Gitaly logs (ZenDesk: https://gitlab.zendesk.com/agent/tickets/111902):

```
2019-02-04_22:21:04.33575 time="2019-02-04T22:21:04Z" level=error msg="grpc panic" error="runtime error: invalid memory address or nil pointer dereference" method=/gitaly.RepositoryService/CalculateChecksum
```

They are running GitLab 11.5.4.

Unfortunately I don't see a backtrace, but I think we need to fix this ASAP.

/cc: @dbalexandre

Story points: 1.0
Stop symlinking the hooks directory

Part of: https://gitlab.com/gitlab-org/gitaly/issues/1226

The hooks are now executed by setting them explicitly each git shellout, instead of executing the `path/to/repo.git/hooks` which is symlinked to the `gitlab-shell/hooks` dir.

The symlinking still happens, but this could stop. Downside of stopping doing so, is that a client could not downgrade the GitLab version with a minor release as the projects that got created in the mean time wouldn't have the symlink.

The cost to keep symlinking is low, but I'd recommend removing the symlinking around %"12.0" to avoid confusion and have one way of doing so.

Added this to that milestone, feel free to remove @jramsay :)

Story points: 1.0
Importing Project on same instance from URL fails

### Summary

Import a repository from the same GitLab instance during project creation is failing with the error message below displayed on the empty repository's import screen.

```
Every import attempt has failed: Error importing repository https://ckgltest.s.co.zw/root/greenie.git into root/greenie-import - 13:CreateRepositoryFromURL: clone cmd wait: exit status 128. Please try again.
```

### Steps to reproduce

* Create a new project and select import by Git URL.
* Enter the URL of another project within our GitLab instance and run the import.

### Example Project

Failed to reproduce on GitLab.com but reproduced successfully on 11.0.3-ee

### What is the expected *correct* behavior?

Project should be imported without errors. This was working for the [customer](https://gitlab.zendesk.com/agent/tickets/100000)  before upgrading to 11.0

### Relevant logs and/or screenshots

![Screen_Shot_2018-07-10_at_09.56.45](/uploads/f979d9588b839bce7509f23948d0c01e/Screen_Shot_2018-07-10_at_09.56.45.png)

Story points: 3.0
Go 1.8

This is a placeholder issue to discuss upgrading Go to 1.8, now that https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2279 is merged. 

- Update CI to stop testing on Go 1.5 / 1.6 and 1.7. 
- Update documentation for developer requirements
- Discuss any potential code changes that may follow on from this change in future (for example, related to context)

Story points: 2.0
Test how GitLab behaves when Gitaly is Down

@pcarranza's idea:
> test how does GitLab behaves when a Gitaly server is down - we need better resiliency when an NFS server goes down, since it will happen again and we will be running Gitaly on those hosts, we need to be sure that the application will survive when they go dark. For this I propose that we actually test how gRPC behaves and we code for resiliency from now and not as an afterthought.

Story points: 2.0
Define the GitLab Shell <-> Gitaly CLI Stub interface

Blocking: #124 

Related: https://gitlab.com/gitlab-org/gitlab-shell/issues/80

----------------------------------

As part of https://gitlab.com/gitlab-org/gitaly/issues/124 we need to have a way of communicating between GitLab-Shell and a GitLab CLI interface.

Do we use environment variables? Configuration file? Arguments?

https://gitlab.com/gitlab-org/gitaly/issues/124 is blocked on this but work will be able to continue once it's done.

Story points: 1.0
Gitaly-on-NFS staging test plan

How will we test this on staging with 9.1. RCx.

- [x] make sure Gitaly over TCP works in omnibus
- [x] test with localhost
- [x] test with remote host and different repository root
- [ ] instructions to prepare staging
- [x] wait for package with all required capabilities to be deployed on staging
- [ ] schedule time with someone with production access to run experiments on staging

Parallel production issue: https://gitlab.com/gitlab-com/infrastructure/issues/1777

Merge Request with (some of) the required staging configuration changes https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/637

Story points: 3.0
Operation `CommitDiff ` on service `Diff` doesn't implement the full functionality expected by Gitlab CE

~Conversation: gitlab-org/gitaly#222

The following flags are not supported:

- [ ] deltas_only
- [ ] ignore_whitespace_change
- [ ] paths

Because of this reason, we can't fully migrate this operation to Gitaly. We should implement support for these flags. This will require changes in gitaly-proto.

/cc @jacobvosmaer-gitlab @andrewn @ahmadsherif 

Story points: 4.0
Forward Git SSH sessions from gitlab-shell to Gitaly Server Implementation

~Conversation: gitlab-org/gitaly#218

Depends on ~"Migration Analysis" #91

Depends on ~"RPC Design" https://gitlab.com/gitlab-org/gitaly-proto/merge_requests/5

-------------------

### ~"Server Implementation" Forward Git SSH sessions from gitlab-shell to Gitaly

Goal: allow Gitaly to monitor and control the git upload-pack and git receive-pack server-side processes spawned by the Git SSH transport.


We already have a protocol for this. https://gitlab.com/gitlab-org/gitaly-proto/blob/v0.5.0/ssh.proto 

We want to be able to run two commands on behalf of SSH clients: `git upload-pack` and `git receive-pack`. Each has its own RPC `SSHUploadPack` and `SSHReceivePack`.

The idea behind the current proto is:

- read first request message to get Repository and some metadata (GL_ID variable) in case of receive-pack
- start the git process requested for the correct git repository path
- in a goroutine: copy the rest of the request stream to stdin
- in two goroutines: copy the stdout and stderr streams to the response
- wait for the git command, out and err streams to finish
- send final response message containing the git exit code

Story points: 4.0
`make test` requires all files to be checked in

Currently, the test suite is checking that all files are checked in and failing if there are any local changes:
 
```
notice-up-to-date:	notice
	git ls-files --error-unmatch NOTICE # NOTICE is a tracked file
	git diff --exit-code # there are no changed files
```

This is happening here: `git diff --exit-code`

I would question why we have this at all:
1. In a local development environment, we would expect there to be uncommitted changes.
2. In a CI environment, this would not serve a purpose either

We should be able to run `make test` without requiring all changes to be committed.

Story points: 1.0
FindChangedPaths returns paths that are not changed from merge commit

<!--
If possible, please select a relevant template from the "Choose a template" dropdown above.
-->

<!-- template sourced from https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab/issue_templates/Default.md -->

Related Gitlab Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/23625

When a user pushes a list of changes to gitlab we perform diff checks on the changes. If those changes include a merge commit the returned changed paths may also include changes that are already in history.

We can solve the gitlab issue linked above by using [-c](https://git-scm.com/docs/git-diff-tree#Documentation/git-diff-tree.txt--c) instead of [-m](https://git-scm.com/docs/git-diff-tree#Documentation/git-diff-tree.txt--m) in the [FindChangedPaths git-diff-tree cmd](https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/gitaly/service/diff/find_changed_paths.go#L61). `-c` combines the diffs and only returns changes that are different in all parents e.g. conflict resolutions (where different from all parents) and other manual changes introduced during the merge.

I discussed this in slack with @jcaigitlab and @pks-gitlab. 

@pks-gitlab pointed out that this would be a breaking change so rather than changing the flag we should introduce an enum to allow the client to specify how the output should be formatted. 

Note: the [output format for a merge commit](https://git-scm.com/docs/git-diff-tree#_diff_format_for_merges) when using `-c` is different to `-m` (each parent has a mode and sha and a status) so the `nextPath` function will need to return multiple `gitalypb.ChangedPaths` for each merge commit.

Story points: 3.0
Add support for generating patch-ids

Users who interact with MR pages are often distanced from the actual mechanics of git that underlie the diffs, which is a strength of our product. However, by gently obfuscating the concept of commits and branches, we find users are often confused about the implications of those concepts. One area this arises in particular is approval rules behavior around rebases. Often users want approval rules to be reset when new code enters the diff; this is a fairly basic security requirement, as it helps ensure that unreviewed/unapproved code doesn't enter an MR.

This sensible behavior is broken in a situation where a rebase occurs, as git generates new commit SHAs, which look like "new code" to our app. While we can work around this with rebase commit SHAs and disabling approval rule resets for UI-generated rebases, if someone rebases via the command line, we have no way of detecting that -- we only see new commits coming in via the push. This would lead to a disjointed UX and rebase changes behavior depending on which execution method is chosen.

+ https://gitlab.com/gitlab-org/gitlab/-/issues/216144+
+ https://gitlab.com/gitlab-org/gitlab/-/issues/337888+

The real question we're attempting to answer is "has a meaningful, programmatic change been made in the diff" which is only detectable by comparing `merge_request_diff` data. Rather than attempt to compare 2 diffs themselves (potentially quite large text blobs) we would instead like the ability to generate a representation of the raw diff, allowing us to compare for _actual_ changes in the code. This would allow us to detect not only for the case of "this is just a rebase, no need to reset approvals" but potentially to trigger (or not..) additional management behavior in the review cycle (such as notifying previous reviewers that a change has been made..)

One approach towards accomplishing this would be to add support for [`git patch-id`](https://git-scm.com/docs/git-patch-id) as a new RPC, allowing us (Rails) to request as needed, storing the result in the DB. This would allow us to quickly and efficiently compare the gestalt of a series of commits, rather than giant diffs, or even commits themselves (which potentially can be added, removed, squashed, etc during rebasing without impacting the resulting diff..)

(This was previously opened in https://gitlab.com/gitlab-org/gitlab/-/issues/378954 and includes some initial discussions and feedback from Rails-side stakeholders here https://gitlab.com/gitlab-org/gitlab/-/issues/378954#note_1149000248)

Story points: 3.0
Introduce new WhitespaceChanges enum and deprecate ignore_whitespace_change

Following from this comment: https://gitlab.com/groups/gitlab-org/-/epics/9028#note_1138747850 from @pks-t 

ultimately this is a backwards-incompatible change in Gitaly that might also impact other users of it. How about we amend the `CommitDiff` RPC to expose a new enum `WhitespaceChanges` or similar that deprecates the `ignore_whitespace_change` field? It could either be `Unspecified` (we look at `ignore_whitespace_change`), `DontIgnore`, `Ignore` or `IgnoreAll` so that you can then control yourself how exactly to render the diff?


### Availability & Testing
As the change in this issue is adding a new field, and potentially marking another as deprecated there shouldn't be any major issue here. Any change in behaviour in rails shouldn't happen until https://gitlab.com/gitlab-org/gitlab/-/issues/378213.  
- We should ensure that `omnibus-gitlab-mirror` trigged by the `trigger-qa` job passes to confirm no unexpected downstream impact

Story points: 2.0
Cleanup fallback logic for gitaly-git2go binary path resolution

https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3587 is merged and deployed on the gitlab.com.  
It is safe to remove [fallback logic](https://gitlab.com/gitlab-org/gitaly/-/blob/8a6d0e26de9d584941267d2b68c94b37bc30e092/internal/git2go/command.go#L31-39) to define the path to the `gitaly-git2go` binary. 

/cc @zj-gitlab

Story points: 1.0
Geo: Help Praefect recover after it does an internal failover

From https://gitlab.com/gitlab-org/gitaly/-/issues/3493#note_536442781:

> Praefect responds with code `FailedPrecondition` and message `repository is in read-only mode`: https://gitlab.com/gitlab-org/gitaly/-/blob/f1b141704f05163c5cf95185151570b844cc8b1c/internal/praefect/coordinator.go#L33

## Proposal

When a secondary encounters this particular error while syncing, Geo should schedule the next retry at of this repository later than it usually would. For example, we could set a floor of 1 minute. And/or we could double the calculated delay.

Story points: 1.0
[Gitaly] Make Ruby 3.0.x the default Ruby

Part of &5149

Once we have a [Ruby 3 build for gitaly-ruby](https://gitlab.com/gitlab-org/gitaly/-/issues/3715), we actually want to switch the default Ruby over as well, which should happen as a separate step. 

**NOTE**: We had to roll back the Ruby 3 build due to a grpc rollback.

Story points: 4.0
Enable Prometheus by default

Gitaly does not enable Prometheus by default, only when administrators configure the buckets. This issue proposes to have default buckets of:

```
grpc_latency_buckets = [0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]
```

The config value remains to override what the default value is.

See also: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9490#note_495908357

Story points: 1.0