overview/overview-word-cloud — pull request #2 by ethanresnick (2014-10-22 09:07:17+00:00)
Title: Bug Fixes and convergence testing

@adamhooper This PR addresses the bugs you caught last time (e.g. it uses a better font stack, clearer names, etc). Re the most substantive issue—how to stream from the server—it sends a blob of data every time the server processes 100 documents. This is roughly analogous to having the server send new data every x milliseconds. But doing it after a certain number of documents, rather than after a certain number of milliseconds, was more convenient: first, because timers are a pain, but also because it simplified my implementation of convergence testing (by simplifying the logic for predicting when the doc set will converge).

Re the convergence testing: obviously, this isn't statistically valid until the docs are coming back in a random order. But I figured I'd get the code ready anyway. On the Pitchfork doc set @jstray gave me, the wordcloud converged (meaning that the cosine similarity between wordcloud n and wordcloud n+1 was > .9999) after only 2,800 docs. (It's a 13,200 document set.) As a result, the server was able to return the "final" data in ~6 seconds, as opposed to ~17 seconds before. So we got a 3x speedup, even though we're only processing 1/5 as many docs. We didn't get a bigger bump because we're processing the token data (i.e. finding the top tokens and, now, computing computing the cosine similarity) more often; now we're doing it once per 100 docs until convergence, whereas before we were only doing it the 5 times we were responding. It'll be interesting to see how we can set `incrementSize` to minimize total response time (i.e. balance fewer times processing the tokens against converging after reading fewer docs).