Package: riko
IndexIntroduction|Requirements|Word Count|Motivation|Usage|Installation|Design Principles|Scripts|Command-line Interface|Contributing|Credits|More Info|Project Structure|LicenseIntroductionrikois a pure Pythonlibraryfor analyzing and processingstreamsof
structured data.rikohassynchronousandasynchronousAPIs, supportsparallel
execution, and is well suited for processing RSS feeds[1].rikoalso supplies
acommand-line interfacefor executingflows, i.e., stream processors akaworkflows.Withriko, you canRead csv/xml/json/html filesCreate text and data basedflowsvia modularpipesParse, extract, and process RSS/Atom feedsCreate awesome mashups[2], APIs, and mapsPerformparallel processingvia cpus/processors or threadsand much more…Notes[1]Really Simple Syndication[2]Mashup (web application hybrid)Requirementsrikohas been tested and is known to work on Python 3.7, 3.8, and 3.9; and PyPy3.7.Optional DependenciesFeatureDependencyInstallationAsync APITwistedpip install riko[async]Accelerated xml parsinglxml[3]pip install riko[xml]Accelerated feed parsingspeedparser[4]pip install riko[xml]Notes[3]Iflxmlisn’t present,rikowill default to the builtin Python xml parser[4]Ifspeedparserisn’t present,rikowill default tofeedparserWord CountIn this example, we use severalpipesto count the words on a webpage.>>>### Create a SyncPipe flow ###>>>#>>># `SyncPipe` is a convenience class that creates chainable flows>>># and allows for parallel processing.>>>fromriko.collectionsimportSyncPipe>>>>>>### Set the pipe configurations ###>>>#>>># Notes:>>>#   1. the `detag` option will strip all html tags from the result>>>#   2. fetch the text contained inside the 'body' tag of the hackernews>>>#      homepage>>>#   3. replace newlines with spaces and assign the result to 'content'>>>#   4. tokenize the resulting text using whitespace as the delimeter>>>#   5. count the number of times each token appears>>>#   6. obtain the raw stream>>>#   7. extract the first word and its count>>>#   8. extract the second word and its count>>>#   9. extract the third word and its count>>>url='https://news.ycombinator.com/'>>>fetch_conf={...'url':url,'start':'<body>','end':'</body>','detag':True}# 1>>>>>>replace_conf={...'rule':[...{'find':'\r\n','replace':' '},...{'find':'\n','replace':' '}]}>>>>>>flow=(...SyncPipe('fetchpage',conf=fetch_conf)# 2....strreplace(conf=replace_conf,assign='content')# 3....tokenizer(conf={'delimiter':' '},emit=True)# 4....count(conf={'count_key':'content'}))# 5>>>>>>stream=flow.output# 6>>>next(stream)# 7{"'sad":1}>>>next(stream)# 8{'(':28}>>>next(stream)# 9{'(1999)':1}MotivationWhy I built rikoYahoo! Pipes[5]was a user friendly web application used toaggregate, manipulate, and mashup content from around the webWanting to create custom pipes, I came acrosspipe2pywhich translated a
Yahoo! Pipe into python code.pipe2pysuited my needs at the time
but was unmaintained and lacked asynchronous or parallel processing.rikoaddresses the shortcomings ofpipe2pybut removed support for
importing Yahoo! Pipes json workflows.rikocontains ~40 built-inmodules, akapipes, that allow you to programatically perform most of the
tasks Yahoo! Pipes allowed.Why you should use rikorikoprovides a number of benefits / differences from other stream processing
applications such as Huginn, Flink, Spark, and Storm[6]. Namely:a small footprint (CPU and memory usage)native RSS/Atom supportsimple installation and usagea pure python library withpypysupportbuiltin modularpipesto filter, sort, and modifystreamsThe subsequent tradeoffsrikomakes are:not distributed (able to run on a cluster of servers)no GUI for creatingflowsdoesn’t continually monitorstreamsfor new datacan’t react to specific eventsiterator (pull) based so streams only support a single consumer[7]The following table summarizes these observations:libraryStream TypeFootprintRSSsimple[8]asyncparallelCEP[9]distributedrikopullsmall√√√√pipe2pypullsmall√√Huginnpushmed√[10]√√Otherspushlarge[11][12][13]√√√For more detailed information, please check-out theFAQ.Notes[5]Yahoo discontinued Yahoo! Pipes in 2015, but you can view whatremains[6]Huginn,Flink,Spark, andStorm[7]You can mitigate this via thesplitmodule[8]Doesn’t depend on outside services like MySQL, Kafka, YARN, ZooKeeper, or Mesos[9]Complex Event Processing[10]Huginn doesn’t appear to makeasync web requests[11]Many libraries can’t parse RSS streams without the use of 3rd party libraries[12]While most libraries offer a local mode, many require integrating with a data ingestor (e.g., Flume/Kafka) to do anything useful[13]I can’t find evidence that these libraries offer an async APIs (and apparentlySpark doesn’t)Usagerikois intended to be used directly as a Python library.Usage IndexFetching feedsSynchronous processingParallel processingAsynchronous processingCookbookFetching feedsrikocan fetch rss feeds from both local and remote filepaths via “source”pipes. Each “source”pipereturns astream, i.e., an iterator of
dictionaries, akaitems.>>>fromriko.modulesimportfetch,fetchsitefeed>>>>>>### Fetch an RSS feed ###>>>stream=fetch.pipe(conf={'url':'https://news.ycombinator.com/rss'})>>>>>>### Fetch the first RSS feed found ###>>>stream=fetchsitefeed.pipe(conf={'url':'http://arstechnica.com/rss-feeds/'})>>>>>>### View the fetched RSS feed(s) ###>>>#>>># Note: regardless of how you fetch an RSS feed, it will have the same>>># structure>>>item=next(stream)>>>item.keys()dict_keys(['title_detail','author.uri','tags','summary_detail','author_detail','author.name','y:published','y:title','content','title','pubDate','guidislink','id','summary','dc:creator','authors','published_parsed','links','y:id','author','link','published'])>>>item['title'],item['author'],item['id']('Gravity doesn’t care about quantum spin','Chris Lee','http://arstechnica.com/?p=924009')Please see theFAQfor a complete list of supportedfile typesandprotocols. Please seeFetching data and feedsfor more examples.Synchronous processingrikocan modifystreamsvia the40 built-inpipes>>>fromriko.collectionsimportSyncPipe>>>>>>### Set the pipe configurations ###>>>fetch_conf={'url':'https://news.ycombinator.com/rss'}>>>filter_rule={'field':'link','op':'contains','value':'.com'}>>>xpath='/html/body/center/table/tr[3]/td/table[2]/tr[1]/td/table/tr/td[3]/span/span'>>>xpath_conf={'url':{'subkey':'comments'},'xpath':xpath}>>>>>>### Create a SyncPipe flow ###>>>#>>># `SyncPipe` is a convenience class that creates chainable flows>>># and allows for parallel processing.>>>#>>># The following flow will:>>>#   1. fetch the hackernews RSS feed>>>#   2. filter for items with '.com' in the link>>>#   3. sort the items ascending by title>>>#   4. fetch the first comment from each item>>>#   5. flatten the result into one raw stream>>>#   6. extract the first item's content>>>#>>># Note: sorting is not lazy so take caution when using this pipe>>>>>>flow=(...SyncPipe('fetch',conf=fetch_conf)# 1....filter(conf={'rule':filter_rule})# 2....sort(conf={'rule':{'sort_key':'title'}})# 3....xpathfetchpage(conf=xpath_conf))# 4>>>>>>stream=flow.output# 5>>>next(stream)['content']# 6'Open Artificial Pancreas home:'Please seealternate workflow creationfor an alternative (function based) method for
creating astream. Please seepipesfor a complete list of availablepipes.Parallel processingAn example usingriko’s parallel API to spawn aThreadPool[14]>>>fromriko.collectionsimportSyncPipe>>>>>>### Set the pipe configurations ###>>>fetch_conf={'url':'https://news.ycombinator.com/rss'}>>>filter_rule={'field':'link','op':'contains','value':'.com'}>>>xpath='/html/body/center/table/tr[3]/td/table[2]/tr[1]/td/table/tr/td[3]/span/span'>>>xpath_conf={'url':{'subkey':'comments'},'xpath':xpath}>>>>>>### Create a parallel SyncPipe flow ###>>>#>>># The following flow will:>>>#   1. fetch the hackernews RSS feed>>>#   2. filter for items with '.com' in the article link>>>#   3. fetch the first comment from all items in parallel (using 4 workers)>>>#   4. flatten the result into one raw stream>>>#   5. extract the first item's content>>>#>>># Note: no point in sorting after the filter since parallel fetching doesn't guarantee>>># order>>>flow=(...SyncPipe('fetch',conf=fetch_conf,parallel=True,workers=4)# 1....filter(conf={'rule':filter_rule})# 2....xpathfetchpage(conf=xpath_conf))# 3>>>>>>stream=flow.output# 4>>>next(stream)['content']# 5'He uses the following example for when to throw your own errors:'Asynchronous processingTo enable asynchronous processing, you must install theasyncmodule.pipinstallriko[async]An example usingriko’s asynchronous API.>>>fromriko.badoimportcoroutine,react>>>fromriko.collectionsimportAsyncPipe>>>>>>### Set the pipe configurations ###>>>fetch_conf={'url':'https://news.ycombinator.com/rss'}>>>filter_rule={'field':'link','op':'contains','value':'.com'}>>>xpath='/html/body/center/table/tr[3]/td/table[2]/tr[1]/td/table/tr/td[3]/span/span'>>>xpath_conf={'url':{'subkey':'comments'},'xpath':xpath}>>>>>>### Create an AsyncPipe flow ###>>>#>>># The following flow will:>>>#   1. fetch the hackernews RSS feed>>>#   2. filter for items with '.com' in the article link>>>#   3. asynchronously fetch the first comment from each item (using 4 connections)>>>#   4. flatten the result into one raw stream>>>#   5. extract the first item's content>>>#>>># Note: no point in sorting after the filter since async fetching doesn't guarantee>>># order>>>@coroutine...defrun(reactor):...stream=yield(...AsyncPipe('fetch',conf=fetch_conf,connections=4)# 1....filter(conf={'rule':filter_rule})# 2....xpathfetchpage(conf=xpath_conf)# 3....output)# 4......print(next(stream)['content'])# 5>>>>>>try:...react(run)...exceptSystemExit:...passHere's how iteration works ():CookbookPlease see thecookbookoripython notebookfor more examples.Notes[14]You can instead enable aProcessPoolby additionally passingthreads=FalsetoSyncPipe, i.e.,SyncPipe('fetch',conf={'url':url}, parallel=True, threads=False).Installation(You are using avirtualenv, right?)At the command line, installrikousing eitherpip(recommended)pipinstallrikooreasy_installeasy_installrikoPlease see theinstallation docfor more details.Design PrinciplesThe primary data structures inrikoare theitemandstream. Anitemis just a python dictionary, and astreamis an iterator ofitems. You can
create astreammanually with something as simple as[{'content':'helloworld'}]. You manipulatestreamsinrikoviapipes. Apipeis simply a function that accepts either astreamoritem, and returns astream.pipesare composable: you
can use the output of onepipeas the input to anotherpipe.rikopipescome in two flavors;operatorsandprocessors.operatorsoperate on an entirestreamat once and are unable to handle
individual items. Exampleoperatorsincludecount,pipefilter,
andreverse.>>>fromriko.modules.reverseimportpipe>>>>>>stream=[{'title':'riko pt. 1'},{'title':'riko pt. 2'}]>>>next(pipe(stream)){'title':'riko pt. 2'}processorsprocess individualitemsand can be parallelized across
threads or processes. Exampleprocessorsincludefetchsitefeed,hash,pipeitembuilder, andpiperegex.>>>fromriko.modules.hashimportpipe>>>>>>item={'title':'riko pt. 1'}>>>stream=pipe(item,field='title')>>>next(stream){'title':'riko pt. 1','hash':2853617420}Someprocessors, e.g.,pipetokenizer, return multiple results.>>>fromriko.modules.tokenizerimportpipe>>>>>>item={'title':'riko pt. 1'}>>>tokenizer_conf={'delimiter':' '}>>>stream=pipe(item,conf=tokenizer_conf,field='title')>>>next(stream){'tokenizer':[{'content':'riko'},{'content':'pt.'},{'content':'1'}],'title':'riko pt. 1'}>>># In this case, if we just want the result, we can `emit` it instead>>>stream=pipe(item,conf=tokenizer_conf,field='title',emit=True)>>>next(stream){'content':'riko'}operatorsare split into sub-types ofaggregatorsandcomposers.aggregators, e.g.,count, combine
allitemsof an inputstreaminto a newstreamwith a singleitem;
whilecomposers, e.g.,filter, create a newstreamcontaining
some or allitemsof an inputstream.>>>fromriko.modules.countimportpipe>>>>>>stream=[{'title':'riko pt. 1'},{'title':'riko pt. 2'}]>>>next(pipe(stream)){'count':2}In case you are confused from the “Word Count” example up top,countcan return
multiple items if you pass in thecount_keyconfig option.>>>counted=pipe(stream,conf={'count_key':'title'})>>>next(counted){'riko pt. 1':1}>>>next(counted){'riko pt. 2':1}processorsare split into sub-types ofsourceandtransformer.sources, e.g.,itembuilder, can create astreamwhiletransformers, e.g.hashcan only transform items in astream.>>>fromriko.modules.itembuilderimportpipe>>>>>>attrs={'key':'title','value':'riko pt. 1'}>>>next(pipe(conf={'attrs':attrs})){'title':'riko pt. 1'}The following table summaries these observations:typesub-typeinputoutputparallelizable?creates streams?operatoraggregatorstreamstream[15]composerstreamstreamprocessorsourceitemstream√√transformeritemstream√If you are unsure of the type ofpipeyou have, check its metadata.>>>fromriko.modulesimportfetchpage,count>>>>>>fetchpage.async_pipe.__dict__{'type':'processor','name':'fetchpage','sub_type':'source'}>>>count.pipe.__dict__{'type':'operator','name':'count','sub_type':'aggregator'}TheSyncPipeandAsyncPipeclasses (among other things) perform this
check for you to allow for convenient method chaining and transparent
parallelization.>>>fromriko.collectionsimportSyncPipe>>>>>>attrs=[...{'key':'title','value':'riko pt. 1'},...{'key':'content','value':"Let's talk about riko!"}]>>>flow=SyncPipe('itembuilder',conf={'attrs':attrs}).hash()>>>flow.list[0]{'title':'riko pt. 1','content':"Let's talk about riko!",'hash':1346301218}Please see thecookbookfor advanced examples including how to wire in
vales from other pipes or accept user input.Notes[15]the outputstreamof anaggregatoris an iterator of only 1item.Command-line Interfacerikoprovides a command,runpipe, to executeworkflows. Aworkflowis simply a file containing a function namedpipethat creates
aflowand processes the resultingstream.CLI Usageusage: runpipe [pipeid]description: Runs a riko pipepositional arguments:pipeid       The pipe to run (default: reads from stdin).optional arguments:-h,--helpshow this help message and exit-a,--asyncLoad async pipe.-t,--testRun in test mode (uses default inputs).CLI Setupflow.pyfrom__future__importprint_functionfromriko.collectionsimportSyncPipeconf1={'attrs':[{'value':'https://google.com','key':'content'}]}conf2={'rule':[{'find':'com','replace':'co.uk'}]}defpipe(test=False):kwargs={'conf':conf1,'test':test}flow=SyncPipe('itembuilder',**kwargs).strreplace(conf=conf2)stream=flow.outputforiinstream:print(i)CLI ExamplesNow to executeflow.py, type the commandrunpipe flow. You should
then see the following output in your terminal:https://google.co.ukrunpipewill also search theexamplesdirectory forworkflows. Typerunpipe demoand you should see the following output:Deadlinetoclearuphealthlaweligibilitynear682Scriptsrikocomes with a built in task managermanage.Setuppipinstallriko[develop]ExamplesRun python linter and nose testsmanagelintmanagetestContributingPlease mimic the coding style/conventions used in this repo.
If you add new classes or functions, please add the appropriate doc blocks with
examples. Also, make sure the python linter and nose tests pass.Please see thecontributing docfor more details.CreditsShoutout topipe2pyfor heavily inspiringriko.rikostarted out as a fork
ofpipe2py, but has since diverged so much that little (if any) of the original
code-base remains.More InfoFAQCookbookiPython NotebookStep-by-Step Intro. TutorialProject Structure┌──benchmarks│├──__init__.py│└──parallel.py├──bin│└──run├──data/*├──docs│├──AUTHORS.rst│├──CHANGES.rst│├──COOKBOOK.rst│├──FAQ.rst│├──INSTALLATION.rst│└──TODO.rst├──examples/*├──helpers/*├──riko│├──__init__.py│├──lib││├──__init__.py││├──autorss.py││├──collections.py││├──dotdict.py││├──log.py││├──tags.py││└──py│├──modules/*│└──twisted│├──__init__.py│├──collections.py│└──py├──tests│├──__init__.py│├──standard.rc│└──test_examples.py├──CONTRIBUTING.rst├──dev-requirements.txt├──LICENSE├──Makefile├──manage.py├──MANIFEST.in├──optional-requirements.txt├──py2-requirements.txt├──README.rst├──requirements.txt├──setup.cfg├──setup.py└──tox.iniLicenserikois distributed under theMIT License.