Package: aiopype
aiopypePython asynchronous data pipelinesaiopypeallows running continuous data pipelines reliably with a
plain simple approach to their development.Aiopypecreates a centralized message handler to allow every
processor to work as an independent non-blocking message
producer/consumer.Aiopypehas 4 main concepts:FlowManagerProcessorMessage HandlerFlowThe Flow isaiopype’s main component. A flow is the entrypoint for
reliability running pipeline managers.Flowis responsible for:Starting all registered managersHandling manager failuresReporting errorsRestarting failed managersManagerThe manager is responsible for registering a data pipeline from top to
bottom. This means it must register a source and connect it with it’s
consumers, until the pipeline finally outputs.ProcessorA processor is a message consumer/producer.SourcesSources are special cases of processors. Their special characteristic is
that they can run forever, and are the starting point of any pipeline.Examples of sources may be:AREST APIpollerAnWebsocketclientACronjobMessage handlerThe message handler is the central piece that allowsaiopypeto
scale.A Flow will start one or more Sources as the starting point for each
registered Manager. Once a Source produces an event, a message will be
triggered and the handler will identify and fire the corresponding
handlers.There are two available message handlers:SyncProtocolAsyncProtocolSyncProtocolThe synchronous event handler is, as its name suggests, synchronous,
meaning that once the source emits a message, it must be handled until
the end of the pipeline and the source can proceed with it’s normal
behavior. This is good for development purposes but fails to meet the
asynchronous event driven pattern required to allowing component
isolation.AsyncProtocolThe main difference between SyncProtocol and AsyncProtocol is that the
latter uses a decoupled event loop to assess if there are new messages
in the queue for processing, whilst the first simply starts processing
received messages instantaneously. This allows total isolation of
processors.ExampleApple stock processor.SourceOur source will beYahoo Financefor gathering data fromAAPLticker price. We’ll useaiopypeRestSourceas a base class.fromaiopype.sourcesimportRestSourceclassYahooRestSource(RestSource):"""
  Yahoo REST API source.
  """def__init__(self,name,handler,symbol):super().__init__(name,handler,'http://finance.yahoo.com/webservice/v1/symbols/{}/quote?format=json&view=detail'.format(symbol),{'exception_threshold':10,'request_interval':30})ProcessorOur sample processor will simply extract the price from the returned
json.fromaiopypeimportProcessorclassHandleRawData(Processor):defhandle(self,data,time):self.emit('price',time,data['list']['resources'][0]['resource']['fields']['price'])OutputOur output processor will write price data onto a CSV File.importcsvclassCSVOutput(Processor):def__init__(self,name,handler,filename):super().__init__(name,handler)self.filename=filenamewithopen(self.filename,'w',newline='')ascsvfile:writer=csv.writer(csvfile,delimiter=';')writer.writerow(['time','price'])defwrite(self,time,price):withopen(self.filename,'w',newline='')ascsvfile:writer=csv.writer(csvfile,delimiter=';')writer.writerow([time,price])ManagerThe manager will instantiateSource,ProcessorandOutput.
It will connectSource’sdataevent toProcessor.handlehandler andProcessor’spriceevent toOutput.writehandler.
This will be our data pipeline.fromaiopypeimportManagerclassYahooManager(Manager):name='yahoo_apple'def__init__(self,handler):super().__init__(handler)self.processor=HandleRawData(self.build_processor_name('processor'),self.handler)self.source=YahooRestSource(self.build_processor_name('source'),self.handler,'AAPL')self.writer=CSVOutput(self.build_processor_name('writer'),self.handler,'yahoo_appl.csv')self.source.on('data',self.processor.handle)self.processor.on('price',self.writer.write)FlowOur flow config will have theyahoo_applemanager only.fromaiopypeimportAsyncFlowclassFlowConfig(object):FLOWS=['yahoo_apple']dataflow=AsyncFlow(FlowConfig())Main method:Will simply start the dataflow.if__name__=="__main__":dataflow.start()Running the exampleCompile all the above code in a file calledexample.pyand run:pythonexample.pyClustersWIP:This decentralized mechanism makes distributed pipelines a possibility,
if we have coordination between nodes.Changelog0.1.4 / 2016-07-14#10Avoid unfinished
flows (@jAlpedrinha)0.1.3 / 2016-07-11#8Fix AsyncProtocol
termination condition (@jAlpedrinha)0.1.2 / 2016-07-06#6Handle exceptions
from async protocol listener (@jAlpedrinha)0.1.1 / 2016-07-05#4Avoid failure on
pusherclient disconnection (@jAlpedrinha)0.1.0 / 2016-07-05#1Add flow manager
and processors (@jAlpedrinha)