work_mem should be reconsidered on all Postgres nodes

# Current picture: amount of temporary files generated

Currently, `work_mem` is set to `'16MB'` on all Postgres GitLab.com nodes.

Its is insufficient, as we can see from temporary files creation statistics (see [the full postgres-checkup data](https://gitlab.com/gitlab-com/gl-infra/production/snippets/1848486#postgres-checkup_A004)):

master (`patroni-04`):

 Indicator | Value
-----------|-------
Stats Since | 2019-02-13&nbsp;21:05:54+00
Stats Age | 62&nbsp;days&nbsp;11:02:22
Temp Files: total size | 5647&nbsp;GB
Temp Files: total number of files | 483381
Temp Files: total number of files per day | 7796
Temp Files: avg file size | 12&nbsp;MB

– 7482 * 12 MiB = ~87.7 GiB, this is how much temporary files is written every day in average, according last 64 days stats.

A replica (`patroni-01`):

 Indicator | Value
-----------|-------
Stats Since | 2019-03-21&nbsp;01:44:39+00
Stats Age | 27&nbsp;days&nbsp;06:25:29
Temp Files: total size | 11&nbsp;TB
Temp Files: total number of files | 225048
Temp Files: total number of files per day | 8335
Temp Files: avg file size | 52&nbsp;MB

– 7895 * 56 = ~431 GiB (!) of tmp files data daily.

This looks like huge inefficiency, which is relatively easy to fix. Here is how much memory we have and how it is used now:

# Current picture: use of memory

The master (`patroni-04`):

```
MemTotal:       429299392 kB
MemFree:        26501664 kB
MemAvailable:   269576376 kB
Buffers:          116700 kB
Cached:         357627264 kB
SwapCached:            0 kB
Active:         327929036 kB
Inactive:       36116408 kB
Active(anon):   126797136 kB
Inactive(anon):  1784692 kB
Active(file):   201131900 kB
Inactive(file): 34331716 kB
```

A replica (`patroni-01`):

```
MemTotal:       429299392 kB
MemFree:        38309544 kB
MemAvailable:   289822468 kB
Buffers:          111740 kB
Cached:         366551768 kB
SwapCached:            0 kB
Active:         339501400 kB
Inactive:       29624252 kB
Active(anon):   122687420 kB
Inactive(anon):  2000436 kB
Active(file):   216813980 kB
```

As we can see, from all ~400GiB of RAM available, right now ~250 GiB is used for OS file cache.

On the master, average tmp file size is just 12 MiB. On replicas, it's more, 52 MiB.

# Considerations on work_mem increase

`work_mem` is not allocated fully for every session – a session might use a fraction of it. On the other hand, a single session might use multiple `work_mem` portions, if the statement being executed has multiple operations that need memory. So rough calculations of worst case scenario, when all `max_connection` (which is 300 now) sessions are consuming `work_mem` fully two times, is 300 * 16 MiB * 2 = 9600 MiB.  This is a very small fraction of total RAM available.

If we raise `work_mem` to `'64MB'` – 4 times more than the current value – we'll have 300 * 64 MiB * 2 = 38400 MiB. It's still not a problem, we will just take ~38 GiB from those 250 GiB of OS file cache. And this is the *worst* case, which actually will not happen. But `work_mem = '64MB'` will allow us to get rid of most temporary files.

Additionally, we know from [postgres-checkup's K001 report](https://gitlab.com/postgres-ai-team/hc-results-tmp/blob/master/g/md_reports/20190419002_2019_04_19T07_13_42_-0700/0_Full_report.md#postgres-checkup_K001) that reading from OS file cache is very rare -- just hundreds blocks per second. This means, that the most of working data set is present in the buffer pool. This again tells us that we can easily increase `work_mem`, even to values like '100MB' and more.

# Log analysis

Let's see average and stddev for temporary file sizes for one given day (April 23):
```bash
$ sudo journalctl --since '2019-04-23' --until '2019-04-24' | grep "temporary file" | awk '{print $18}' | awk '{for(i=1;i<=NF;i++) {sum[i] += $i; sumsq[i] += ($i)^2}}
          END {for (i=1;i<=NF;i++) {
          printf "%f %f \n", sum[i]/NR, sqrt((sumsq[i]-sum[i]^2/NR)/NR)}
         }'
46059369.520593 10821203.571495
```
-- average size was 46 MB, standard deviation ~11 MB.

This means, that raising to `'64MB'` might in insufficient if we want to decrease number of files generated per day to dozens – additional increase to values like 70-80, up to 100 MB might be needed.

# Proposal

1. Raise `work_mem` to `'100MB'` on all Postgres nodes. Restart is not required, only config reloading.
1. After a few days, analyze tmp files generation and memory consumption again. If needed, reconsider `work_mem` again.

Story points: 3.0