Package: scramjet-framework-py
Scramjet in Python⭐ Star us on GitHub — it motivates us a lot! 🚀Scramjet is a simple reactive stream programming framework. The code is written
by chaining functions that transform the streamed data, including well known
map, filter and reduce.The main advantage of Scramjet is running asynchronous operations on your data
streams concurrently. It allows you to perform the transformations both
synchronously and asynchronously by using the same API - so now you can "map"
your stream from whatever source and call any number of API's consecutively.Originally writtenon top of node.js
object streams, Scramjet is now being ported into Python. This is what is
happening in this repository.Tested with Python 3.8.10 and Ubuntu 20.04.Table of contentsInstallationQuick startUsageRequesting featuresReporting bugsContributingDevelopment SetupInstallationScramjet Framework is available on PyPI, You can install it with simple pip command:pipinstallscramjet-framework-pyQuick startLet's say we have afruits.csvfile like this:orange,sweet,1
lemon,sour,2
pigface,salty,5
banana,sweet,3
cranberries,bitter,6and we want to write the names of the sweet fruits to a separate file.
To do this, write an async function like this:fromscramjetimportstreamsimportasyncioasyncdefsweet_stream():withopen("fruits.csv")asfile_in,open("sweet.txt","w")asfile_out:await(streams.Stream.read_from(file_in).map(lambdaline:line.split(',')).filter(lambdarecord:record[1]=="sweet").map(lambdarecord:f"{record[0]}\n").write_to(file_out))asyncio.run(sweet_stream())output saved in sweet.txt:orange
bananaand that's it!UsageBasic building block of Scramjet is theStreamclass. It reads input in
chunks, performs operations on these chunks and produces an iterable output
that can be collected and written somewhere.Creating a streamis done usingread_fromclass method. It accepts
any iterable or an object implementing .read() method as the input, and returns
aStreaminstance.Transforming a stream:map- transform each chunk in a stream using specified function.filter- keep only chunks for which specified function evaluates toTrue.flatmap- run specified function on each chunk, and return all of its results as separate chunks.batch- convert a stream of chunks into a stream of lists of chunks.Each of these methods return the modified stream, so they can be chained like
this:some_stream.map(...).filter(...).batch(...)Collecting datafrom the stream (asynchronous):write_to- write all resulting stream chunks into a target.to_list- return a list with all stream chunks.reduce- combine all chunks using specified function.Examples :books:You can find more examples inhello_datastream.pyfile. They don't require any additional dependencies, just the standard library,
so you can run them simply with:pythonhello_datastream.pyRequesting FeaturesAnything missing? Or maybe there is something which would make using Scramjet Framework much easier or efficient? Don't hesitate to fill up anew feature request! We really appreciate all feedback.Reporting bugsIf you have found a bug, inconsistent or confusing behavior please fill up anew bug report.ContributingYou can contribute to this project by giving us feedback (reporting bugsandrequesting features) and also by writing code yourself!The easiest way is tocreate a forkof this repository and thencreate a pull requestwith all your changes. In most cases, you should branch from and targetmainbranch.Please refer toDevelopment Setupsection on how to setup this project.Development SetupInstall Python3 interpreter on your computer. Refer toofficial docs.Installgitversion control system. Refer toofficial docs.Clone this repository:gitclonegit@github.com:scramjetorg/framework-python.gitCreate and activate a virtualenv:sudoaptinstallpython3-virtualenv
virtualenv-ppython3venv
.venv/bin/activateCheck Python version:$python--version
Python3.8.10Install dependencies:pipinstall-rdev-requirements.txtRun test cases (with activated virtualenv):pytest:bulb:HINT:add a filename if you want to limit which tests are runIf you want to enable detailed debug logging, set one of the following env variables:PYFCA_DEBUG=1# debug pyfcaDATASTREAM_DEBUG=1# debug datastreamSCRAMJET_DEBUG=1# debug both