sixohsix/twitter — pull request #195 by RouxRC (2013-12-27 11:53:11+00:00)
Title: Fix broken userstream and all other streams supposed to break after Jan 6 + diverses issues

Merry Christmas!... Twitter is moving all of its streaming APIs towards HTTP1.1 chunked encoded transfer and breaking the whole lib for us! https://dev.twitter.com/blog/deprecating-http-1.0-streaming-api

Userstream passed already mid-november and has been broken since, cf #190
I first didn't pay much more attention to it than reading the issue thread since I only use the public statuses.filter stream. But I just found out about this announce and browsing issues it became clear the userstream issue here was coming from this change, which did happen on November 23rd in fact (https://dev.twitter.com/discussions/23224#comment-53758) just as the issues reported here confirm.

The thing is the announce also says that on January 6, Twitter will apply it as well for all the other streaming calls, meaning they probably will all be broken as well.
So this motivated me to fix the issue for userstream and supposedly for the other ones further on.

The solution was to stop asking for gzip encoded data for streaming calls since they already answer not zipped with the HTTP1.0 calls (like statuses/filter for now) and should not be zipped when dealing with chunked transfer. This also required to adjust the parsing of the chunks to separate "delimited" length sent by twitter from actual data.

I've tried it out successfully on a combination of calls on (public stream HTTP1.0 | user stream HTTP1.1chunked) x (not blocking | blocking | with timeout) x (low traffic | high traffic | high traffic with argument 'delimited') with python2.7 only. I don't believe I've modified anything incompatible but any tryout from py3 users would be great.

I also took the occasion to fix a couple other issues, including fixing some hanging when using the timeout with low traffic, providing TwitterHTTPErrors on streaming calls errors, fixing the broken follow create/destroy with cmdline and the irc bot, and completing the POST methods list.

Considering the urge, I will personnally migrate my [IRC bot gazouilleur](https://github.com/RouxRC/gazouilleur) to my branch until this is pulled or handled otherwise.

git+https://github.com/RouxRC/twitter.git@fix-broken-streams

Happy tweeting holidays!

PS: [Twitter also announced an SSL change](https://dev.twitter.com/blog/rest-api-ssl-certificate-updates) for Jan 6, but since all the calls are already using secure by default, we should't be at risk here.