sixohsix/twitter — pull request #196 by adonoho (2014-01-15 15:14:00+00:00)
Title: A Simpler Fix to the Streaming Code due to Changes from Twitter on Jan. 13, 2014.

Gentlefolk,

First allow me to applaud and thank RouxRC on his fine work pushing this little library forward. This pull request would not have been possible without his efforts. That said, RouxRC has dropped a quite large pull request that had problems in my Python3 system. When I rolled back to Python2, the system started working but would not stay connected for very long.

As a result of the above, I felt I needed to look into RouxRC's pull request in more detail and use just the parts I needed to get my streams running again. Any errors in this pull request are mine and do not reflect negatively on RouxRC or his code.

RouxRC's streaming patch falls into two main areas. First, it modifies the `api.py` to respect a new property, `gzip`. I have largely copied his work here. I did simplify how he set the headers from 3 lines to one. The second fix occurs in `stream.py`. Once the gzip compression was turned off, I was able to see what Twitter is now sending us. Even though length delimiters are turned off, Twitter nonetheless is inserting a hex encoded number and a `\r\n` between JSON items. RouxRC and I differ in how we chose to address these changes. My fix is focussed upon handling those extra items if they appear in the stream, removing them and then returning to the existing control flow. The important code block is below and my changes occur with the `if` statement:

while True:
try:
utf8_buf = self.buf.decode('utf8').lstrip()
if utf8_buf and utf8_buf[0] != '{': # Remove the hex delimiter length and extra whitespace.
utf8_buf = utf8_buf.lstrip('0123456789abcdefABCDEF')
utf8_buf = utf8_buf.lstrip()
res, ptr = self.decoder.raw_decode(utf8_buf)
self.buf = utf8_buf[ptr:].encode('utf8')
yield wrap_response(res, self.handle.headers)

RouxRC's code does this in a different way. I believe my solution is both shorter and more robust. But you, gentle reader, are the ultimate arbiter. RouxRC did identify in his code some opportunities to raise `TwitterHTTPError`s. I've tried to duplicate those.

I've run this set of fixes under Python 3.3.3 and Python 2.7.6 both on OS X 10.8.5 (Mountain Lion).

In addition to the changes to the library, I modernized a test script, `stream-example` to take the OAuth parameters needed by modern Twitter. As I also use PyCharm, I've added `.idea` to the `.gitignore` file.

Again, this simpler fix would not have been possible without the fine work of RouxRC. Thank you RouxRC.

Anon,
Andrew