sferik/buftok — pull request #4 by Naomarik (2014-10-23 20:30:56+00:00)
Title: fix issue when delimiter was split and fragment was separate

I ran into an issue using the twitter gem when listening to tweets. The problem is that every ~7-20k tweets listened to, it would buffer \n into its own line when trying to tokenize \r\n

So I would get output like:

```
"..."timestamp_ms\":\"1414093345229\"}\r",
"\n",
"{\"created}..."
```

and this would throw a JSON.parse exception.

![pastedgraphic-1](https://cloud.githubusercontent.com/assets/913889/4760733/61d0bd46-5af2-11e4-8ec5-19947205feff.png)

The CONTRIBUTE.md mentions a few rake tasks that don't seem to apply to this project, unless I'm missing something. I've added a test case and ran test cases and all passing.

This is my first pull request to any open source project, hoping I did this correctly.