dashboards.gitlab.com down

I got an alert for dashboards.gitlab.com being down. Upon investigation I can't log into it. Checked the console and lo, `thanos` had OOMed. Below is the log. 

```
[7387773.088506] Out of memory: Kill process 6189 (thanos) score 923 or sacrifice child
[7387773.096489] Killed process 6189 (thanos) total-vm:14551252kB, anon-rss:14190680kB, file-rss:1328kB, shmem-rss:0kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088171] thanos invoked oom-killer: gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088174] thanos cpuset=/ mems_allowed=0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088183] CPU: 2 PID: 6194 Comm: thanos Not tainted 4.15.0-1034-gcp #36~16.04.1-Ubuntu
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088184] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088185] Call Trace:
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088198]  dump_stack+0x85/0xcb
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088203]  dump_header+0x77/0x285
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088206]  oom_kill_process+0x22e/0x450
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088208]  out_of_memory+0x11d/0x4c0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088211]  __alloc_pages_slowpath+0xda2/0xe90
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088214]  __alloc_pages_nodemask+0x265/0x280
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088221]  alloc_pages_vma+0x88/0x1e0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088226]  __handle_mm_fault+0xe26/0x11b0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088228]  handle_mm_fault+0xcc/0x1c0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088234]  __do_page_fault+0x265/0x500
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088236]  do_page_fault+0x2e/0xf0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088242]  ? page_fault+0x2f/0x50
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088244]  page_fault+0x45/0x50
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088248] RIP: 0033:0x45dab3
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088249] RSP: 002b:000000c000095f08 EFLAGS: 00010202
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088252] RAX: 0000000000000000 RBX: 00000000000aa000 RCX: 000000c21d158000
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088253] RDX: 0000000003214d20 RSI: 0000000000000010 RDI: 000000c21d16a000
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088254] RBP: 000000c000095f50 R08: 00007f3ed4d14638 R09: 000000000000005e
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088255] R10: 0000000000000909 R11: 000000000000005d R12: 0000000000001901
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088257] R13: 000000000000000a R14: 0000000000000009 R15: 000000000000000e
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088272] Mem-Info:
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088278] active_anon:3719380 inactive_anon:14957 isolated_anon:0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088278]  active_file:1686 inactive_file:1810 isolated_file:12
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088278]  unevictable:913 dirty:0 writeback:0 unstable:0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088278]  slab_reclaimable:13754 slab_unreclaimable:19825
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088278]  mapped:4934 shmem:37945 pagetables:9802 bounce:0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088278]  free:32774 free_pcp:0 free_cma:0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088283] Node 0 active_anon:14877520kB inactive_anon:59828kB active_file:6744kB inactive_file:7240kB unevictable:3652kB isolated(anon):0kB isolated(file):48kB mapped:19736kB dirty:0kB writeback:0kB shmem:151780kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12288kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088285] Node 0 DMA free:15908kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088290] lowmem_reserve[]: 0 2980 14999 14999 14999
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088294] Node 0 DMA32 free:61196kB min:13412kB low:16764kB high:20116kB active_anon:2976080kB inactive_anon:0kB active_file:1784kB inactive_file:1036kB unevictable:0kB writepending:0kB present:3129332kB managed:3063764kB mlocked:0kB kernel_stack:48kB pagetables:4048kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088298] lowmem_reserve[]: 0 0 12018 12018 12018
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088308] Node 0 Normal free:53992kB min:54096kB low:67620kB high:81144kB active_anon:11900208kB inactive_anon:59828kB active_file:6276kB inactive_file:6744kB unevictable:3652kB writepending:0kB present:12582912kB managed:12313748kB mlocked:3652kB kernel_stack:6400kB pagetables:35160kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088312] lowmem_reserve[]: 0 0 0 0 0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088315] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088328] Node 0 DMA32: 184*4kB (UME) 158*8kB (UME) 179*16kB (UME) 251*32kB (UME) 120*64kB (UME) 133*128kB (UME) 63*256kB (UME) 9*512kB (U) 1*1024kB (U) 1*2048kB (H) 0*4096kB = 61408kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088342] Node 0 Normal: 603*4kB (UME) 532*8kB (UME) 906*16kB (UME) 295*32kB (ME) 246*64kB (ME) 57*128kB (M) 1*256kB (M) 0*512kB 1*1024kB (M) 0*2048kB 0*4096kB = 54924kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088356] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088357] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088358] 40487 total pagecache pages
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088360] 0 pages in swap cache
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088361] Swap cache stats: add 0, delete 0, find 0/0
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088362] Free swap  = 0kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088363] Total swap = 0kB
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088363] 3932059 pages RAM
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088364] 0 pages HighMem/MovableOnly
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088365] 83704 pages reserved
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088365] 0 pages cma reserved
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088366] 0 pages hwpoisoned
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088367] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088373] [  447]     0   447    25742       47    94208        0             0 lvmetad
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088375] [  463]     0   463     3051     1090    61440        0             0 haveged
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088378] [  465]     0   465    15257     4602   159744        0             0 systemd-journal
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088380] [  475]     0   475    10775      525   118784        0         -1000 systemd-udevd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088382] [ 1212]     0  1212     4031      566    73728        0             0 dhclient
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088384] [ 1392]     0  1392     1305       29    61440        0             0 iscsid
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088386] [ 1393]     0  1393     1430      877    65536        0           -17 iscsid
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088388] [ 1403]     0  1403     6932      454   102400        0             0 cron
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088390] [ 1413]     0  1413    95627      312   114688        0             0 lxcfs
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088392] [ 1438]     0  1438     6511      362    94208        0             0 atd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088394] [ 1449]     0  1449     7154      293    98304        0             0 systemd-logind
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088396] [ 1457]     0  1457    69182      813   172032        0             0 accounts-daemon
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088398] [ 1464]     0  1464     1098      140    53248        0             0 runsvdir
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088400] [ 1472]     0  1472     1099      316    57344        0             0 acpid
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088402] [ 1480]   107  1480    10757      516   135168        0          -900 dbus-daemon
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088404] [ 1492]     0  1492     1060      271    53248        0             0 runsv
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088406] [ 1494]     0  1494     1060       69    53248        0             0 runsv
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088408] [ 1495]     0  1495     1060      235    53248        0             0 runsv
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088410] [ 1496]     0  1496     1060      217    49152        0             0 runsv
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088412] [ 1508]     0  1508     1096       29    53248        0             0 svlogd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088414] [ 1511]     0  1511     1096       17    57344        0             0 svlogd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088416] [ 1513]   999  1513    29142     6731   143360        0             0 node_exporter
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088418] [ 1521]     0  1521     1096       16    53248        0             0 svlogd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088420] [ 1526]     0  1526     1096      270    57344        0             0 svlogd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088422] [ 1759]     0  1759    43334     2459   233472        0             0 unattended-upgr
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088424] [ 1760]     0  1760    69272      269   184320        0             0 polkitd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088426] [ 1787]     0  1787     3343       84    69632        0             0 mdadm
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088428] [ 1895]   112  1895    10067      290   110592        0             0 ntpd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088430] [ 1906]     0  1906     3618      392    69632        0             0 agetty
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088432] [ 1909]     0  1909     3664      327    69632        0             0 agetty
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088434] [ 1968]     0  1968     8134      578    98304        0             0 nginx
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088436] [ 2048]     0  2048    16861     3964   180224        0             0 google_network_
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088438] [ 2049]     0  2049    16808     3710   172032        0             0 google_clock_sk
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088441] [ 2167]     0  2167    16378      350   167936        0         -1000 sshd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088442] [ 2173]     0  2173    16352      156   110592        0             0 master
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088445] [ 2177]   113  2177    17308      777   126976        0             0 qmgr
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088447] [ 6399]   999  6399   337760    27793  2686976        0             0 trickster
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088449] [ 8803]   998  8803    45310     5678   249856        0             0 consul
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088451] [23096]  1002 23096    11322      204   118784        0             0 systemd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088453] [23097]  1002 23097    52180      500   163840        0             0 (sd-pam)
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088455] [23217]  1002 23217     6479      525    94208        0             0 screen
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088457] [23218]  1002 23218     4995      513    77824        0             0 bash
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088459] [23223]     0 23223    12855      527   147456        0             0 sudo
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088461] [23224]     0 23224    12751      494   143360        0             0 su
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088463] [23225]     0 23225     5361      920    86016        0             0 bash
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088465] [29388]     0 29388    43355    10443   364544        0             0 fluentd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088467] [29393]     0 29393   193329    38773  1748992        0             0 ruby
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088469] [ 6189]   999  6189  3637813  3548002 29052928        0             0 thanos
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088471] [10135]     0 10135    41314    12584   360448        0             0 chef-client
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088473] [  831]   115   831    15848      119   106496        0             0 memcached
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088475] [ 8839]   116  8839   650365    16297   761856        0             0 grafana-server
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088477] [16643]   104 16643    64098      243   135168        0             0 rsyslogd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088479] [12654]   113 12654    16869      299   126976        0             0 pickup
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088482] [21777]    33 21777     8314      406    98304        0             0 nginx
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088484] [21778]    33 21778     8313      496    98304        0             0 nginx
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088485] [21779]    33 21779     8316      549    98304        0             0 nginx
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088487] [21780]    33 21780     8314      516    98304        0             0 nginx
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088490] [22344]     0 22344     1094      161    57344        0             0 sleep
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088492] [22348]     0 22348     1094      156    53248        0             0 sleep
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088494] [22350]     0 22350    12235      402   143360        0             0 cron
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088496] [22351]     0 22351     1126      174    53248        0             0 sh
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088498] [22352]     0 22352    35950     2728   184320        0             0 ruby
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088500] [22353]     0 22353    42667    14441   368640        0             0 chef-client
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088502] [22356]     0 22356    16378      608   172032        0             0 sshd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088504] [22361]   110 22361    16378      433   163840        0             0 sshd
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.088506] Out of memory: Kill process 6189 (thanos) score 923 or sacrifice child
Sep 21 04:05:04 dashboards-com-01-inf-ops kernel: [7387773.096489] Killed process 6189 (thanos) total-vm:14551252kB, anon-rss:14190680kB, file-rss:1328kB, shmem-rss:0kB
Sep 21 04:05:05 dashboards-com-01-inf-ops kernel: [7387774.148113] oom_reaper: reaped process 6189 (thanos), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
```

Story points: 1.0