Prometheus in the staging environment crashes

On the night of December 27th, the early morning, prometheus kept crashing in the staging environment.   We were alerted twice for both prometheus-01 and for prometheus-02 that we are restarting frequently.

![image](/uploads/7aa266e29f6af94d4c93a8ead215666a/image.png)

Source: https://prometheus.gstg.gitlab.net/graph?g0.range_input=12h&g0.expr=changes(process_start_time_seconds%7Bjob%3D~%22prometheus.*%22%7D%5B30m%5D)&g0.tab=0

Both servers appear to be suffering from a damaged wal file:
```
2018-12-27_13:23:09.53663 level=warn ts=2018-12-27T13:23:09.536566544Z caller=wal.go:116 component=tsdb msg="last page of the wal is torn, filling it with zeros" segment=/opt/prometheus/prometheus/data/wal/00032880
2018-12-27_13:25:36.46930 level=warn ts=2018-12-27T13:25:36.469178923Z caller=head.go:434 component=tsdb msg="unknown series references" count=536346
```

Both servers have go stack traces in their log history:
```
2018-12-27_13:23:21.56483 goroutine 3517 [select]:
2018-12-27_13:23:21.56484 net/http.(*persistConn).writeLoop(0xc16d30afc0)
2018-12-27_13:23:21.56484       /usr/local/go/src/net/http/transport.go:1885 +0x113
2018-12-27_13:23:21.56484 created by net/http.(*Transport).dialConn
2018-12-27_13:23:21.56485       /usr/local/go/src/net/http/transport.go:1339 +0x966
2018-12-27_13:23:21.56485
2018-12-27_13:23:21.56485 goroutine 3391 [select]:
2018-12-27_13:23:21.56486 net/http.(*persistConn).writeLoop(0xc16d2a7c20)
2018-12-27_13:23:21.56486       /usr/local/go/src/net/http/transport.go:1885 +0x113
2018-12-27_13:23:21.56486 created by net/http.(*Transport).dialConn
2018-12-27_13:23:21.56487       /usr/local/go/src/net/http/transport.go:1339 +0x966
2018-12-27_13:23:21.56487
2018-12-27_13:23:21.56487 goroutine 3496 [select]:
2018-12-27_13:23:21.56488 net/http.(*persistConn).writeLoop(0xc16b3af680)
2018-12-27_13:23:21.56488       /usr/local/go/src/net/http/transport.go:1885 +0x113
2018-12-27_13:23:21.56488 created by net/http.(*Transport).dialConn
2018-12-27_13:23:21.56489       /usr/local/go/src/net/http/transport.go:1339 +0x966
2018-12-27_13:23:21.56489
2018-12-27_13:23:21.56489 goroutine 3515 [select]:
2018-12-27_13:23:21.56490 net/http.(*persistConn).writeLoop(0xc16d024360)
2018-12-27_13:23:21.56490       /usr/local/go/src/net/http/transport.go:1885 +0x113
2018-12-27_13:23:21.56490 created by net/http.(*Transport).dialConn
2018-12-27_13:23:21.56490       /usr/local/go/src/net/http/transport.go:1339 +0x966
2018-12-27_13:23:21.56491
2018-12-27_13:23:21.56491 goroutine 3523 [select]:
2018-12-27_13:23:21.56491 net/http.(*persistConn).writeLoop(0xc16d024240)
2018-12-27_13:23:21.56491       /usr/local/go/src/net/http/transport.go:1885 +0x113
2018-12-27_13:23:21.56492 created by net/http.(*Transport).dialConn
2018-12-27_13:23:21.56492       /usr/local/go/src/net/http/transport.go:1339 +0x966
<SNIP>
2018-12-27_13:23:21.56493 goroutine 3488 [IO wait]:
2018-12-27_13:23:21.56493 internal/poll.runtime_pollWait(0x7f73558b98c0, 0x72, 0xc16b498a40)
2018-12-27_13:23:21.56493       /usr/local/go/src/runtime/netpoll.go:173 +0x66
2018-12-27_13:23:21.56494 internal/poll.(*pollDesc).wait(0xc16d448c18, 0x72, 0xffffffffffffff00, 0x1e9d680, 0x2e045a8)
2018-12-27_13:23:21.56495       /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
2018-12-27_13:23:21.56495 internal/poll.(*pollDesc).waitRead(0xc16d448c18, 0xc16d4ba000, 0x1000, 0x1000)
2018-12-27_13:23:21.56495       /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
2018-12-27_13:23:21.56496 internal/poll.(*FD).Read(0xc16d448c00, 0xc16d4ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
2018-12-27_13:23:21.56497       /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
2018-12-27_13:23:21.56497 net.(*netFD).Read(0xc16d448c00, 0xc16d4ba000, 0x1000, 0x1000, 0xc16d4ad200, 0x4, 0x0)
2018-12-27_13:23:21.56498       /usr/local/go/src/net/fd_unix.go:202 +0x4f
2018-12-27_13:23:21.56499 net.(*conn).Read(0xc169185c58, 0xc16d4ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
2018-12-27_13:23:21.56500       /usr/local/go/src/net/net.go:177 +0x68
2018-12-27_13:23:21.56500 net/http.(*persistConn).Read(0xc16d4ad200, 0xc16d4ba000, 0x1000, 0x1000, 0xc16b498c70, 0x404c85, 0xc16d49a360)
2018-12-27_13:23:21.56501       /usr/local/go/src/net/http/transport.go:1497 +0x75
2018-12-27_13:23:21.56502 bufio.(*Reader).fill(0xc16cdee6c0)
2018-12-27_13:23:21.56502       /usr/local/go/src/bufio/bufio.go:100 +0x106
2018-12-27_13:23:21.56503 bufio.(*Reader).Peek(0xc16cdee6c0, 0x1, 0x0, 0x0, 0x1, 0xc093b5ea80, 0x0)
<SNIP>
```

As of the time of this writing, https://prometheus.gstg.gitlab.com throws a 502, despite both servers having prometheus ready to receive web requests...

Due to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5823, currently these pages for the staging environment will wake up the on-call person.

/cc @bjk\-gitlab 
/cc @gitlab\-com/gl\-infra

Story points: 1.0