ttezel/twit — pull request #150 by mroth (2014-12-15 21:57:01+00:00)
Title: Performance enhancements for streaming

I made a few changes to the way streaming parsing is done to improve performance and throughput.

Here are some ad-hoc benchmarks from my workstation, consuming the streaming API with a query that generates ~1000 tweets per second for 35 seconds.

Before:
```
RESULTS 10.18s user 0.83s system 31% cpu 35.008 total
-> user: 10.18s
-> system: 0.83s
-> cpu: 31%
-> max memory: 68064
-> page faults from disk: 0
-> other page faults: 18426
-> socket msgs received: 82913
-> socket msgs sent: 10
```

After:
```
RESULTS 6.70s user 1.06s system 22% cpu 35.007 total
-> user: 6.70s
-> system: 1.06s
-> cpu: 22%
-> max memory: 69004
-> page faults from disk: 0
-> other page faults: 19171
-> socket msgs received: 104496
-> socket msgs sent: 10
```

In addition, the new streaming parser adds support for sending the [`delimited: length`](https://dev.twitter.com/streaming/overview/request-parameters#delimited) parameter to the Twitter API streaming endpoint, which causes it to prefix each tweet object with its length in bytes. This enables more efficient parsing since you don't have to scan ahead for an EOL.

Using `{delimited: length}`:
```
RESULTS 5.75s user 0.94s system 19% cpu 35.006 total
-> user: 5.75s
-> system: 0.94s
-> cpu: 19%
-> max memory: 69336
-> page faults from disk: 0
-> other page faults: 18433
-> socket msgs received: 103859
-> socket msgs sent: 10
```

I rarely use Javascript or NodeJS so I would strongly suggest a code review on this before merging.