Package: tidehunter
tidehunterHTTP streaming with accurate flow controlMaster branch:NOTE: Not backward compatible with 0.x since 1.x.HighlightsConsumption limits, total control over your stream quota just on the
client side.Instant on/off switch and accurate consumption counter. Best used
withtechies.Queue interface for scalable stream data consumption. Best used withtechies.Core mechanisms based on the solidrequestslibrary,
inherits all its goodness.Installation$pipinstalltidehunter$pipinstalltidehunter--upgradeUsageExample 1 (with limit):NOTE: when no externalQueueorStateCountersupplied,Hunteruses the Python standardQueueand the builtinSimpleStateCounterrespectively, which are usually enough for single
process designs and other simple cases.fromtidehunterimportHunter# The Hunter!h=Hunter(url='https://httpbin.org/stream/20')# Start streamingh.tide_on(limit=5)# Consume the data which should be in the data queue nowwhileh.q.qsize():print(h.q.get())# profit x 5# You can re-use the same Hunter object, with a difference limitr=h.tide_on(limit=1)# this time we only want one recordasserth.q.qsize()==1# or else there's a bug, create an issue!print(h.q.get())# more profit# r is actually just a requests.Response objectprint(r.headers)print(r.status_code)# ... read up on requests library for more informationExample 2 (without limit):NOTE: this example usestechiesand therefore requires Redis
installed.Assume you have a process running the following code:fromtechiesimportStateCounter,QueuefromtidehunterimportHunter# The data queueq=Queue(key='demo_q',host='localhost',port=6379,db=0)# The state machine and record counter (state counter)sc=StateCounter(key='demo_sc',host='localhost',port=6379,db=0)# The Hunter!h=Hunter(url='SOME_ENDLESS_STREAM_LIKE_TWITTER_FIREHOSE',q=q,sc=sc)# Start streaming, FOREVAh.tide_on()Then you delegate the flow control and data consumption to another/many
other processes such as:fromtechiesimportStateCounter,Queue# The key is to have the SAME state countersc=StateCounter(key='demo_sc',host='localhost',port=6379,db=0)# And the SAME data queueq=Queue(key='demo_q',host='localhost',port=6379,db=0)whilesc.started:data=q.get()# dequeue and# ...do something with dataifSHT_HITS_THE_FAN:sc.stop()# instant off switch# end of this loop, as well as the streaming process from above# If neededq.clear()sc.clear()Example 3 (OAuth with Twitter Sample Firehose):NOTE: this example requiresrequests_oauthlibimportosimportjsonfromrequests_oauthlibimportOAuth1fromtidehunterimportHunterurl='https://stream.twitter.com/1.1/statuses/sample.json'auth=OAuth1(os.environ['TWITTER_CONSUMER_KEY'],os.environ['TWITTER_CONSUMER_SECRET'],os.environ['TWITTER_TOKEN_KEY'],os.environ['TWITTER_TOKEN_SECRET'])h=Hunter(url=url,q=q,auth=auth)r=h.tide_on(5)# let's just get 5 for nowprint(r.status_code)print('')whileh.q.qsize():print(json.loads(h.q.get()))print('')You can find other authentications onthis requests
doc.
In short, all you have to do is to pass the desiredauthparameter
toHunter, like what you would do withrequests.Test (Unit Tests)$pipinstall-rrequirements.txt$pipinstall-rtest_requirements.txt$nosetests--with-coverage--cover-package=tidehunterLicenseThe MIT License (MIT). See the fullLICENSE.ContributorsRunzhou Li (Leo)Changelog1.0.1 (2015-04-17)Breaking Change:tidehunter.SimpleStateCounterupdated to
reflect`techies<https://github.com/woozyking/techies>`__ 0.2.0
changes onStateCounter.1.0.0 (2014-01-22)Moved codebase ofQueue,StateCountertotechies. It’s recommended
to usetechiestogether withtidehunter, but not always
required, and therefore not a dependency oftidehunterAddedtidehunter.SimpleStateCounterto be used when no other
state counter provided. It’s a pure in-process implementation and
therefore cannot be accessed by other processesYou can now dofrom tidehunter import Hunterinstead offrom tidehunter.stream import HunterReplacedPycURLwithrequests. Some of the
benefits:Straight Python 2/3 supportMuch cleaner implementationFurther delegation ofvarious authentications
supporttorequestsitself0.1.9 (2013-12-24)PyCurlandRedisPython libraries bumped to the latest
versions.Queuenow isalmostPython Queue compatible (in a complaint
free fashion), with the exception ofQueue.fullwhich always
returnsFalse;Queue.task_doneandQueue.joindo nothing.NEW: BothQueueandStateCounternow have aclearmethod
which performs a RedisDELcommand on the said key and
reinitialize based on each class’sinitializemethod.0.1.8 (2013-10-02)Added alias methodsput_nowait()andget_nowait()and other
place holders to map the Python built-in Queue interfaces.Addedrstgenshell script for Markdown to reStructuredText. To
use, run$ source rstgenin the root folder.Credentials involved in unit tests and demo are now using environment
variables.0.1.7 (2013-07-22)Massive update to README.rstFixed PyPi rendering of long description.0.1.5 (2013-07-22)NEW:Hunter.tide_on()now accepts an optional limit parameter for
on the fly limit adjustment. The adjustment is not permanent, meaning
if you want to reuse the same Hunter object, the old limit (or
default None) is in effect.Fixed a potential issue of Hunter puts in more records than desired
limit.Added temp Basic Auth test case (no stream, need to find a better
source).0.1.3 (2013-07-13)Use the great httpbin.org (by Kenneth Reitz) for unit test now.Auth (oauth or basic) is no longer required, as long as the target
stream server supports access without auth.0.1.2 (2013-07-12)Include CHANGES (changelog) to be shown on PyPi.Use with statement to open files for setup.py.Added the firstdemo.0.1.1 (2013-07-12)Clean up setup.py to ensure requirements are installed/updated.0.1.0 (2013-07-12)Initial release