Package: minibatch
Dependencies:a running MongoDB accessible to minibatchPython 3.xsee extras & optional dependencies below for specific requirementsminibatch provides a straight-forward, Python-native approach to mini-batch streaming and complex-event
processing that is easily scalable. Streaming primarily consists ofa producer, which is some function inserting data into the streama consumer, which is some function retrieving data from the streamtransform and windowing functions to process the data in small batches and in parallelminibatch is an integral part ofomega|ml, however also works independently. omega|ml is the Python DataOps and MLOps
platform for humans.Featuresnative Python producers and consumersincludes three basic Window strategies: CountWindow, FixedTimeWindow, RelaxedTimeWindowextensible Window strategies by subclassing and overriding a few methodsscalable, persistent streams - parallel inserts, parallel processing of windowsA few hightlightscreating a stream and appending data is just 2 lines of codeproducer and consumer stream code runs anywhereno dependencies other than mongoengine, pymongoextensible sources and sinks (already available: Kafka, MQTT, MongoDB collections, omega|ml datasets)a fully functional streaming web app can be built in less than 15 lines of code (using Flask)Why is it called minibatch? Because it focuses on getting things done by using existing
technology, and making it easy to use this techonlogy. It may be minimalistic in approach, but maximises results.Quick startInstall and setup$pipinstallminibatch$dockerrun-d-p27017:27017mongoSee extras & optional dependencies below to select specific packages according
to your deployment needs, e.g. for MQTT, Kafka, omega|mlCreate a stream producer or attach to a sourceimportminibatchasmbstream=mb.stream('test')foriinrange(100):stream.append({'date':datetime.datetime.utcnow().isoformat()})sleep(.5)Currently there is support for Kafka and MQTT sources. However
arbitrary other sources can be added.fromminibatch.contrib.kafkaimportKafkaSourcesource=KafkaSource('topic',urls=['kafka:port'])stream.attach(source)Consume the streamfromminibatchimportstreaming@streaming('test',size=2,keep=True)defmyprocess(window):print(window.data)returnwindow=>[{'date':'2018-04-30T20:18:22.918060'},{'date':'2018-04-30T20:18:23.481320'}][{'date':'2018-04-30T20:18:24.041337'},{'date':'2018-04-30T20:18:24.593545'}...myprocessis called for every N-tuple of items (size=2)  appended to the stream by the producer(s).
The frequency is determined by the emitter strategy. This can be configured or changed for a custom
emitter strategy, as shown in the next step.Configure the emitter strategyNote the@streamingdecorator. It implements a blocking consumer that delivers batches
of data according to some strategy implemented by a WindowEmitter. Currently@streamingprovides the following interface:size=N- uses theCountWindowemitterinterval=SECONDS- uses theRelaxedTimeWindowemitterinterval=SECONDS, relaxed=False- uses theFixedTimeWindowemitteremitter=CLASS:WindowEmitter- uses the given subclass of aWindowEmitterworkers=N- set the number of workers to process the decorated function, defaults to number of CPUsexecutor=CLASS:Executor- the asynchronous executor to use, defaults toconcurrent.futures.ProcessPoolExecutorWrite a flask app as a streaming sourceThis is a simple helloworld-style streaming application that is fully
functional and distributable.# app.pydefconsumer(url):@streaming('test-stream',url=url)defprocessing(window):...# whatever processing you need to doif__name__=='__main__':app=StreamingApp()app.start_streaming(consumer)app.run()# run the app (check status at http://localhost:5000/status)$pythonapp.py# in an other process, stream data$python[]importminibatchasmbstream=mb.stream('test-stream')stream.append(dict(data='foobar')NotethereisnoUIinthisexample,howeverthedataisprocessedasitcomesin.ToaddaUI,specifyusing@app.route,asforanyflaskapp,writetheprocesseddataintoasinkthattheUIcanaccess.Forafullexampleseehelp(minibatch.contrib.apps.omegaml.StreamingApp)Stream sourcesCurrently provided inminibatch.contrib:KafkaSource - attach a stream to a Apache Kafka topicMQTTSource - attach to an MQTT brokerMongoSource - attach to a MongoDB collectionDatasetSource - attach to a omega|ml datasetCeleryEventSource - attach to a Celery app event dispatcherStream sources are arbitrary objects that support thestream()method, as follows.classSomeSource:...defstream(self,stream):fordatainsource:stream.append(data)Stream SinksThe result of a stream can be forwarded to a sink. Currently
provided sinks inminibatch.contribare:KafkaSink - forward messagess to a Apache Kafka topicMQTTSink  - forward messages to an MQTT brokerMongoSink - forward messages to a MongoDB collectionDatasetSink - write to a omega|ml datasetStream sinks are arbitrary objects that support theput()method, as follows.classSomeSink:...defput(self,message):sink.send(message)Window emittersminibatch provides the following window emitters out of the box:CountWindow- emit fixed-sized windows. Waits until at leastnmessages areavailable before emitting a new windowFixedTimeWindow- emit all messages retrieved within specific, time-fixed windows ofa given interval ofnseconds. This guarantees that messages were received in the specific
window.RelaxedTimeWindow- every interval ofnseconds emit all messages retrieved sincethe last window was created. This does not guarantee that messages were received in a given
window.Implementing a custom WindowEmitterCustom emitter strategies are implemented as a subclass toWindowEmitter. The main methods
to implement arewindow_ready- returns the tuple(ready, data), where ready is True if there is datato emitquery- returns the data for the new window. This function retrieves thedatapartof the return value ofwindow_readySee the API reference for more details.classSortedWindow(WindowEmitter):"""
    sort all data by value and output only multiples of 2 in batches of interval size
    """defwindow_ready(self):qs=Buffer.objects.no_cache().filter(processed=False)data=[]forobjinsorted(qs,key=lambdaobj:obj.data['value']):ifobj.data['value']%2==0:data.append(obj)iflen(data)>=self.interval:breakself._data=datareturnlen(self._data)==self.interval,()defquery(self,*args):returnself._dataWhat is streaming and how does minibatch implement it?ConceptsInstead of directly connection producers and consumers, a producer sends messages to a stream. Think
of a stream as an endless buffer, or a pipeline, that takes input from many producers on one end, and
outputs messages to a consumer on the other end. This transfer of messages happens asynchronously, that
is the producer can send messages to the stream independent of whether the consumer is ready to receive, and the  consumer can take messages from the stream independent of whether the producer is ready to send.Unlike usual asynchronous messaging, however, we want the consumer to receive messages in small batches to optimize throughput. That is, we want the pipeline toemitmessages only subject to some criteria
of grouping messages, where each group is called amini-batch. The function that determines whether the
batching criteria is met (e.g. time elapsed, number of messages in the pipeline) is calledemitter strategy,
and the output it produces is calledwindow.Thus in order to connect producers and consumers we need the following parts to our streaming system:aStream, keeping metadata for the stream such as its name and when it was created, last read etc.aBufferacting as the buffer where messages sent by producers are stored until the emittingaWindowEmitterimplementing the emitter strategyaWindowrepresenting the output produced by the emitter strategyNoteThe producer accepts input from some external system, say an MQTT end point. The producer’s responsibility is to enter the data into the streaming buffer.
The consumer uses an emitter strategy to produce a Window of data that is then forwarded to the user’s processing code.Implementationminibatch uses MongoDB to implement Streams, Buffers and Windows. Specifically, the following collections are used:stream- represents instances ofStream, each document is a stream with a unique namebuffer- a virtually endless buffer for all streams in the system, each document contains one message of a streamwindow- each document represents the data as emitted by the particular emitter strategyBy default messages go through the following statesupon append by a producer: message is inserted intobuffer, with flagprocessed = Falseupon being seen by an emitter: message is marked asprocessed = Trueupon being emitted: message is copied towindow, markedprocessed = False(in Window)upon emit success (no exceptions raised by the emit function): message is deleted frombufferand markedprocessed = TrueinwindowNotes:emitters typically act on a collection of messages, that is steps 2 - 4 are applied to more
than one message at a timeto avoid deleting messages from the buffer, pass@streaming(…, keep=True)custom emitters can modify the behavior of both creating windows and handling the buffer by
overriding theprocess(),emit()andcommit()methods for each of the above steps
2/3/4, respectively.Extras & optional dependenciesminibatch provides the following pip install extras, which come with some
additional dependencies. Extras are installed by running$pipinstallminibatch[<extra>|all]Available extras are:apps- adds StreamingApp for easy development & deployment of producers & consumerskafka- to work with Kafka as a source or a sinkmqtt- to work with an MQTT broker as a source or a sinkmongodb- to work with MongoDB as a source or a sinkomegaml- to work with omega|ml datasets as a source or a sinkall- all of the abovedev- all of the above plus a few development packagesFurther developmentHere are a couple of ideas to extend minibatch. Contributions are welcome.more examples, following typical streaming examples like word count, filteringmore emitter strategies, e.g. for sliding windowsperformance testing, benchmarkingdistributed processing of windows via distributed framework such as celery, ray, daskextend emitters by typical stream operations e.g. to support operations like count, filter, map, groupby, merge, joinadd other storage backends (e.g. Redis, or some Python-native in-memory db that provides network access and an easy to use ORM layer, like mongoengine does for MongoDB)ContributingWe welcome any contributions - examples, issues, bug reports, documentation, code. Please seeCONTRIBUTING.mdfor details.LicenseApache 2.0 licensed with “No Sell, Consulting Yes” clause.
See LICENSE and LICENSE-NOSELLCLAUSE files.