Recurring CPU-bound git-fetch processes on file-praefect-02 gitaly node

While looking into intermittent alerts about high Gitaly latency on one of the nodes behind Praefect (e.g. [here](https://gitlab.pagerduty.com/incidents/PJ6FJVO) and [here](https://gitlab.pagerduty.com/incidents/PFACGWK)), I noticed that this host is almost always running 2 or 3 `git fetch` processes.  These `git fetch` processes are always CPU-bound.  Because they tend to run for several minutes, it was easy to find one and do some profiling on it before it exited.  Repeating this a few times on different `git fetch` processes consistently showed the same pattern -- these particular `git fetch` processes spent a lot of time reading directories via the `getdents` syscall.

These slow CPU-intensive `git fetch` processes may or may not be why the Gitaly latency is high enough to alert -- it's plausible but not proven.

I thought it was worth documenting, mainly because further investigation may lead to tuning the git config of the handful of GitLab-owned repos stored on this host.  These repos (e.g. the repo for gitaly itself) tend to have many refs, which may or may not be related to why `git fetch` is doing so many expensive dentry fetches.

### Observations

#### perf_event profile

This is one of several examples, each giving consistent results even though the `git fetch` was running in at least 3 different git repos.

```shell
msmiley@file-praefect-02-stor-gprd.c.gitlab-production.internal:~$ pgrep -u git -f '/opt/gitlab/embedded/bin/git ' | xargs -r ps uwf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
git      32544 97.1  0.1 224848 207100 ?       R    21:18   0:51 /opt/gitlab/embedded/bin/git fetch tmp-f0c29603f769ef2a5e19823338fa

msmiley@file-praefect-02-stor-gprd.c.gitlab-production.internal:~$ sudo perf record --freq 99 -g --pid 32544 -- sleep 120
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 1.079 MB perf.data (8860 samples) ]
Terminated
```

Complete flamegraph in SVG format: [git-fetch.svg](/uploads/7a5463cb953f0f86279eb111e6d4b59a/git-fetch.svg)

Screenshot without the tall thin tower:

![Screenshot_from_2020-03-15_15-03-44](/uploads/789aac5e826cf7061a109039e41b2ab4/Screenshot_from_2020-03-15_15-03-44.png)


#### Size of dentry cache at the time

The kernel's slab cache for dentries is roughly 600 MB as of time writing.

```shell
msmiley@file-praefect-02-stor-gprd.c.gitlab-production.internal:~$ date -u ; sudo slabtop --once
Sun Mar 15 21:26:54 UTC 2020
 Active / Total Objects (% used)    : 35725531 / 36164607 (98.8%)
 Active / Total Slabs (% used)      : 880109 / 880109 (100.0%)
 Active / Total Caches (% used)     : 78 / 127 (61.4%)
 Active / Total Size (% used)       : 5496363.91K / 5627374.90K (97.7%)
 Minimum / Average / Maximum Object : 0.01K / 0.16K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
24567153 24533027  99%    0.10K 629927       39   2519708K buffer_head            
3320415 3051784  91%    0.19K  79287       42    634296K dentry                 
1887918 1886804  99%    0.04K  18509      102     74036K ext4_extent_status     
1250816 1237169  98%    0.06K  19544       64     78176K kmalloc-64             
1233603 1179390  95%    1.05K  42948       30   1374336K ext4_inode_cache       
...

msmiley@file-praefect-02-stor-gprd.c.gitlab-production.internal:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:         120867        3323       13714        1160      103829      114721
Swap:             0           0           0
```

#### Confirm that `getdents` syscall is both frequent and time-consuming

Profile the top syscalls by count and duration for another example `git fetch` process.

```shell
msmiley@file-praefect-02-stor-gprd.c.gitlab-production.internal:~$ pgrep -u git -f '/opt/gitlab/embedded/bin/git ' | xargs -r ps uwf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
git      11227 68.1  0.0  47364 12140 ?        Sl   21:45   0:06 /opt/gitlab/embedded/bin/git fetch tmp-abdca8edcb52002d5ac34c56dbd7
git      11346 99.4  0.0  42608 27176 ?        R    21:46   0:04 /opt/gitlab/embedded/bin/git fetch tmp-da29fee686ab94a6c8a08bf13797

msmiley@file-praefect-02-stor-gprd.c.gitlab-production.internal:/usr/share/bcc/tools$ sudo ./syscount --pid 11346 --duration 10 --latency
Tracing syscalls, printing top 10... Ctrl+C to quit.
[21:47:00]
SYSCALL                   COUNT        TIME (us)
getdents                  66594      8425669.975
open                      22198        79449.929
write                     11099        79104.089
lstat                     22204        58918.045
fstat                     22198        30487.748
close                     22198        28841.996
brk                           2           16.327
stat                          1           14.969

Detaching...
```

Story points: 2.0