Backup restore hangs/failed

On 2018-09-12, a pipeline run for https://gitlab.com/gitlab-restore/postgres-01.db.prd.gitlab.com was triggered. This spins up a new GCE database instance and pulls the latest backup using `wal-e backup-fetch`.

Timeline:

* Sep 12 15:28 - Instance boot
* Sep 12 15:33 - wal-e starts `backup-fetch`
* Sep 12 15:33 - Encryption errors the may be caused by a permission error for `/var/opt/gitlab/postgresql/data/server.crt` (see log below) - may be harmless
* Sep 13 2:11 - wal-e downloads `part_00001959.tar.lzo` (this is the last partition - so may even have completed restoring the backup) - no more log messages after this.
* Sep 13 8:58 - the `backup-fetch` process still hangs (strace see below)

The backup we were fetching was: `base_00000006000084260000008D_03825496`. 

strace of the stalled backup-fetch process:

```
gitlab-+ 21309 27.1  0.8 363136 65124 ?        Sl   Sep12 283:07 /opt/wal-e/bin/python3 /opt/wal-e/bin/wal-e backup-fetch /var/opt/gitlab/postgresql/data/ base_00000006000084260000008D_03825496
root@restore-postgres-01-prd:/var/opt/gitlab/postgresql# strace -c -p 21309
strace: Process 21309 attached
^Cstrace: Process 21309 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 60.60    0.063848          12      5504           clock_gettime
 29.87    0.031475          20      1570           epoll_wait
  9.52    0.010033          13       791           wait4
------ ----------- ----------- --------- --------- ----------------
100.00    0.105356                  7865           total
```

In the beginning, we were seeing these errors (which may be harmless and caused by permission errors for the certificate):
```
Sep 12 15:37:35 restore-postgres-01-prd startup-script: INFO startup-script: wal_e.worker.s3.s3_worker INFO     MSG: beginning partition download
Sep 12 15:37:35 restore-postgres-01-prd startup-script: INFO startup-script:         DETAIL: The partition being downloaded is part_00000000.tar.lzo.
Sep 12 15:37:35 restore-postgres-01-prd startup-script: INFO startup-script:         HINT: The absolute S3 key is postgres-02/basebackups_005/base_00000006000084260000008D_03825496/tar_partitions/part_00000000.tar.lzo.
Sep 12 15:37:35 restore-postgres-01-prd startup-script: INFO startup-script:         STRUCTURED: time=2018-09-12T15:37:35.702363-00 pid=21309
Sep 12 15:37:35 restore-postgres-01-prd wal_e.worker.s3.s3_worker INFO     MSG: beginning partition download#012        DETAIL: The partition being downloaded is part_00000000.tar.lzo.#012        HINT: The absolute S3 key is postgres-02/basebackups_005/base_00000006000084260000008D_03825496/tar_partitions/part_00000000.tar.lzo.#012        STRUCTURED: time=2018-09-12T15:37:35.702363-00 pid=21309
Sep 12 15:37:38 restore-postgres-01-prd startup-script: INFO startup-script: gpg: block_filter 0x24e83c0: read error (size=15934,a->size=15934)
Sep 12 15:37:38 restore-postgres-01-prd startup-script: INFO startup-script: gpg: block_filter 0x24e8ea0: read error (size=16165,a->size=16165)
Sep 12 15:37:38 restore-postgres-01-prd startup-script: INFO startup-script: gpg: WARNING: encrypted message has been manipulated!
Sep 12 15:37:38 restore-postgres-01-prd startup-script: INFO startup-script: gpg: block_filter: pending bytes!
Sep 12 15:37:38 restore-postgres-01-prd startup-script: INFO startup-script: gpg: block_filter: pending bytes!
Sep 12 15:37:38 restore-postgres-01-prd startup-script: INFO startup-script: lzop: Inappropriate ioctl for device: <stdin>
Sep 12 15:37:41 restore-postgres-01-prd wal_e.retries WARNING  MSG: retrying after encountering exception#012        DETAIL: Exception information dump: #012        Traceback (most recent call last):#012          File "/opt/wal-e/lib/python3.5/site-packages/wal_e/retries.py", line 87, in shim#012            return f(*args, **kwargs)#012          File "/opt/wal-e/lib/python3.5/site-packages/wal_e/worker/s3/s3_worker.py", line 78, in fetch_partition#012            TarPartition.tarfile_extract(pl.stdout, self.local_root)#012          File "/opt/wal-e/lib/python3.5/site-packages/wal_e/tar_partition.py", line 301, in tarfile_extract#012            tar.extract(member, path=dest_path)#012          File "/opt/wal-e/lib/python3.5/tarfile.py", line 2038, in extract#012            numeric_owner=numeric_owner)#012          File "/opt/wal-e/lib/python3.5/tarfile.py", line 2108, in _extract_member#012            self.makefile(tarinfo, targetpath)#012          File "/opt/wal-e/lib/python3.5/tarfile.py", line 2148, in makefile#012            with bltn_open(targetpath, "wb") as target:#012        PermissionError: [Errno 13] Permission denied: '/var/opt/gitlab/postgresql/data/server.crt'#012        #012        HINT: A better error message should be written to handle this exception.  Please report this output and, if possible, the situation under which it arises.#012        STRUCTURED: time=2018-09-12T15:37:41.346707-00 pid=21309
```

Story points: 5.0