Package: bspump
Documentationhttps://bitswanpump.readthedocs.io/en/latest/index.htmlPrinciplesWrite once, use many timesEverything is a streamSchema-lessKappa architectureReal-TimeHigh performanceSimple to use and well documented, so anyone can write their own stream processorAsynchronous via Python 3.5+async/awaitandasyncioEvent driven Architecture/Reactor patternSingle-threaded core but compatible with threadsCompatible withpypy, Just-In-Time compiler capable of boosting Python code performace more then 5x timesGood citizen of the Python ecosystemModularizedStream processor example#!/usr/bin/env python3importbspumpimportbspump.socketimportbspump.commonimportbspump.elasticsearchclassMyPipeline(bspump.Pipeline):def__init__(self,app):super().__init__(app)self.build(bspump.socket.TCPStreamSource(app,self),bspump.common.JSONParserProcessor(app,self),bspump.elasticsearch.ElasticSearchSink(app,self,"ESConnection"))if__name__=='__main__':app=bspump.BSPumpApplication()svc=app.get_service("bspump.PumpService")svc.add_connection(bspump.elasticsearch.ElasticSearchConnection(app,"ESConnection"))svc.add_pipeline(MyPipeline(app))app.run()Video tutorialBuildDocker buildDockerfile and instructions are inseparate repository.PyPI releaseReleases are happening from a git tag (format:vYY.MM)git tag-av19.07Following thePyPI packaging, generatedistribution packageandupload itusing following commandpython-mtwine upload dist/*Blank application setupYou can clone blank application from it’sown repository.Available technologiesbspump.amqpAMQP/RabbitMQ connection, source and sinkbspump.avroApache Avro file source and sinkbspump.commonCommon processors and parsersbspump.elasticsearchElasticSearch connection, source and sinkbspump.fileFile sources and sinks (plain files, JSON, CSV)bspump.filterContent, Attribute and TimeDrift filtersbspump.http.clientHTTP client source, WebSocket client sinkbspump.http.webHTTP server source and sink, WebSocket server sourcebspump.influxdbInfluxDB connection and sinkbspump.kafkaKafka connection, source and sinkbspump.mailSMTP connection and sinkbspump.mongodbMongoDB connection and lookupbspump.mysqlMySQL connection, source and sinkbspump.parquetApache Parquet file sinkbspump.postgresqlPostgreSQL connection and sinkbspump.slackSlack connection and sinkbspump.socketTCP source, UDP sourcebspump.triggerOpportunistic, PubSub and Periodic triggersbspump.cryptoCryptographybspump.declarativeDeclarative processors and expressionsHashing: SHA224, SHA256, SHA384, SHA512, SHA1, MD5, BLAKE2b, BLAKE2sSymmetric Encryption: AES 128, AES 192, AES 256bspump.analyzerTime Window analyzerSession analyzerGeographical analyzerTime Drift analyzerbspump.lookupGeoIP Lookupbspump.unittestInterface for testing Processors / Pipelinesbspump.webPump API endpoints for pipelines, lookups etc.Google Sheet with technological compatiblity matrix:https://docs.google.com/spreadsheets/d/1L1DvSuHuhKUyZ3FEFxqEKNpSoamPH2Z1ZaFuHyageoI/edit?usp=sharingHigh-level architectureUnit testfromunittest.mockimportMagicMockfrombspump.unittestimportProcessorTestCaseclassMyProcessorTestCase(ProcessorTestCase):deftest_my_processor(self):# setup processor for testself.set_up_processor(my_project.processor.MyProcessor,"proc-arg",proc="key_arg")# mock methods to suit your needs on pipeline ..self.Pipeline.method=MagicMock()# .. or instance of processorself.Pipeline.Processor.method=MagicMock()output=self.execute([(None,{'foo':'bar'})]# Context, event)# assert outputself.assertEqual([eventforcontext,eventinoutput],[{'FOO':'BAR'}])# asssert expected calls on `self.Pipeline.method` or `self.Pipeline.Processor.method`self.Pipeline.Processor.method.assert_called_with(**expected)Running of unit testspython3-munittest testYou can replacetestwith a location of your unit test module.LicenceBSPump is an open-source software, available under BSD 3-Clause License.