Alert Investigation: 2018-11-30 13:49 UTC High CPU on git-01

![image](/uploads/2e11ef1c0bedf6269762c8500b0f8fe1/image.png)

During investigation there were 18 `sshd: git@notty` processes tying up the CPU.  I performed an strace on a few and saw very similar output that didn't really point me into any direction as to what I should continue to look for:

```
clock_gettime(CLOCK_BOOTTIME, {4569259, 270293753}) = 0
select(12, [3 5 9 11], [3], NULL, {30, 0}) = 2 (in [9], out [3], left {29, 999993})
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
clock_gettime(CLOCK_BOOTTIME, {4569259, 270591957}) = 0
read(9, "\17_\267e$\370M\340>\277\371 \376\246\244(\203H\213@\v\313_\344\31 \214[\200uPl"..., 16384) = 16384
write(3, "_\275\356\314O\257$\\\\O\\\302\"\303\262\367\250x\205\337BR\364\357\347\n5\227\37\272\4\377"..., 16448) = 16448
clock_gettime(CLOCK_BOOTTIME, {4569259, 367838203}) = 0
select(12, [3 5 9 11], [3], NULL, {30, 0}) = 2 (in [9], out [3], left {29, 999993})
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
clock_gettime(CLOCK_BOOTTIME, {4569259, 368819627}) = 0
read(9, "\17F\377a\276[\255O\34\267\21\177\234T\34qm*:\345h\264\363\372C[%+\312g^\233"..., 16384) = 16384
write(3, "\335\251R\2125\270\330$\303\272Z&2\342 yV\230W\371f\230\371\242\323\6\316\255)#,\375"..., 16448) = 16448
clock_gettime(CLOCK_BOOTTIME, {4569259, 570915822}) = 0
select(12, [3 5 9 11], [3], NULL, {30, 0}) = 2 (in [9], out [3], left {29, 999994})
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
clock_gettime(CLOCK_BOOTTIME, {4569259, 571406231}) = 0
read(9, "\215#t\366\203u\244K\370\346\23\325\334\351\241\314\241\376\301\314\3011\366\205\361d\376\260\320D\344#"..., 16384) = 16384
```

The host was able to process requests just fine.  It was never removed from our load balancers and during this time, I proceeded to force a few git commands through this server, and despite being a tad slower than other git servers, the request worked just fine.  There were no known "stuck" git processes, based on the command provided in our runbook.

At 14:04 UTC I put this server in state DRAIN for all load balancers.  The load immediately started to fall off, but the remaining git processes were apparently long running and left the cpu in a pegged state.

At 14:26 UTC I decided to pull the plug on the 18 processes via `kill`.  The CPU usage immediately dropped.  I waited for other metrics (load) and the alert to clear before proceeding to put this server back into rotation

At 14:32 UTC I killed any remaining git ssh process.  There were roughly 20 of them, some of them as old as November 27th.

At 14:34 UTC the server was put back into rotation.

---

Due to lack of knowledge and visibility into what these processes are doing, I'm not sure what to look at for this type of situation.  If they are long running git commands, it'd be nice if I could tie an ssh session to a process.

If anyone else has better ideas as to how I should've approach this situation please let me know.  I am not marking this as an incident due to not having enough information to prove abuse and for lifetime in which this situation was occurring, the server was still able to serve requests successfully.

/cc @gitlab\-com/gl\-infra

Story points: 1.0