marktriggs/thread-grep — pull request #1 by ato (2014-10-23 06:39:41+00:00)
Title: Add features from clusterstack

Coincidentally I wrote [something similar] (https://github.com/ato/junk/blob/master/cuttrace) to thread-grep (but much dumber) a few months ago. It has a [companion script] (https://github.com/ato/junk/blob/master/clusterstack) with two handy features I'm replicating here.

Firstly, clustering threads with identical stack traces:
* makes it easy to see hotspots (many threads blocking on a slow web service, database, calculation etc)
* makes idle thread pools and the like more less verbose

For example here's a livelock due to an unsynchronized HashMap bug in restlets:

```
"1202479321@qtp-749039839-1245" prio=10 tid=0x00007f4054016800 nid=0x3e43 runnable [0x00007f40bedc6000] (STATE: RUNNABLE)
"213733551@qtp-749039839-1243" prio=10 tid=0x00007f4054006000 nid=0x3e40 runnable [0x00007f40b7ffc000] (STATE: RUNNABLE)
"1390834978@qtp-749039839-1242" prio=10 tid=0x00007f4054029800 nid=0x3e3f runnable [0x00007f40b7bf8000] (STATE: RUNNABLE)
"433169521@qtp-749039839-1241" prio=10 tid=0x00007f405001d000 nid=0x3e3e runnable [0x00007f40b6ce9000] (STATE: RUNNABLE)
"922076105@qtp-749039839-1240" prio=10 tid=0x00007f4050009800 nid=0x3e3d runnable [0x00007f40b70ed000] (STATE: RUNNABLE)
"1114517399@qtp-749039839-1236" prio=10 tid=0x00007f4054027800 nid=0x3d1c runnable [0x00007f40bd362000] (STATE: RUNNABLE)
"1229875577@qtp-749039839-1235" prio=10 tid=0x00007f405001a800 nid=0x3d1b runnable [0x00007f40b77f4000] (STATE: RUNNABLE)
"473895544@qtp-749039839-1234" prio=10 tid=0x00007f405001a000 nid=0x3d14 runnable [0x00007f40b7efb000] (STATE: RUNNABLE)
"1259511661@qtp-749039839-1227" prio=10 tid=0x00007f4050007800 nid=0x3c4d runnable [0x00007f40be2c2000] (STATE: RUNNABLE)
"1325934700@qtp-749039839-1223" prio=10 tid=0x00007f4050015800 nid=0x3b8a runnable [0x00007f40beec7000] (STATE: RUNNABLE)
"272071871@qtp-749039839-1201" prio=10 tid=0x00007f4054005000 nid=0x3587 runnable [0x00007f40bc51b000] (STATE: RUNNABLE)
at java.util.WeakHashMap.getEntry(WeakHashMap.java:498)
at java.util.WeakHashMap.containsKey(WeakHashMap.java:484)
at org.restlet.engine.util.ImmutableDate.valueOf(ImmutableDate.java:66)
at org.restlet.engine.util.DateUtils.unmodifiable(DateUtils.java:255)
at org.restlet.representation.Representation.setExpirationDate(Representation.java:547)
at org.restlet.engine.header.HeaderUtils.extractEntityHeaders(HeaderUtils.java:752)
at org.restlet.ext.jaxrs.internal.util.Util.copyResponseHeaders(Util.java:330)
```

Secondly it colorizes the output by the state of threads (waiting/blocked vs running) and greys out boring standard library stack frames. Can be paged with:

thread-grep -c --color=always foo.log | less -r

Also handy to use with [ansi2html] (http://www.pixelbeat.org/scripts/ansi2html.sh):

jstack $(pgrep -f suckyapp) > stack.txt
thread-grep --cluster --color=always stack.txt | ansi2html > ~/public_html/deadlock.html
echo 'http://example.org/deadlock.html' | mail -s'Fix this please' culprit@example.org