Package: tubing
Tubing is a Python I/O library.  What makes tubing so freakin’ cool is the
gross abuse of the bit-wise OR operator (|). Have you ever been writing python
code and thought to yourself, “Man, this is great, but I really wish it was a
little more like bash.” Whelp, we’ve made python a little more like bash.If you
are a super lame nerd-kid, you can replace any of the bit-wise ORs with the
tube() function and pray we don’t overload any other operators in future
versions. Here’s how you install tubing:$ pip install tubingTubing is pretty bare-bones at the moment. We’ve tried to make it easy to add
your own functionality. Hopefully you find it not all that unpleasant. There
are three sections below for adding sources, tubes and sink. If you do make
some additions, think about committing them back upstream. We’d love to have
a full suite of tools.Now, witness the power of this fully operational I/O library.fromtubingimportsources,tubes,sinksobjs=[dict(name="Bob Corsaro",birthdate="08/03/1977",alignment="evil",),dict(name="Tom Brady",birthdate="08/03/1977",alignment="good",),]sources.Objects(objs)\|tubes.JSONDumps()\|tubes.Joined(by=b"\n")\|tubes.Gzip()\|sinks.File("output.gz","wb")Then in our old friend bash.$zcatoutput.gz{"alignment":"evil","birthdate":"08/03/1977","name":"Bob Corsaro"}{"alignment":"good","birthdate":"08/03/1977","name":"Tom Brady"}$You can find more documentation onreadthedocsCatalogSourcesObjectsTakes alistof python objects.FileCreates a stream from a file.BytesTakes a byte string.IOTakes an object with a read function.SocketTakes an addr, port and socket() args.              .HTTPTakes an method, url and any args that can be passed
to requests library.TubesGunzipUnzips a binary stream.GzipZips a binary stream.JSONLoadsParses a byte string stream of raw JSON objects. Will
try to use ujson, then built-in json.JSONDumpsSerializes an object stream usingjson.dumps. Will
try to use ujson, then built-in json.SplitSplits a stream that supports thesplitmethod.JoinedJoins a stream of the same type as thebyargument.TeeTakes a sink and passes chunks along apparatus.MapTakes a transformer function for single items in
stream.FilterTakes a filter test callback and only forwards items
that pass.ChunkMapTakes a transformer function for batch of stream
items.SinksObjectsA list that stores all passed items to self.BytesSaves each chunk self.results.FileWrites each chunk to a file.HTTPPostWrites data via HTTPPost.HashTakes algorithm name, updates hash with contents.DebuggerWrites each chunk to the tubing.tubes debugger with level DEBUG.Extensionss3.S3SourceCreate stream from an S3 object.s3.MultipartUploaderStream data to S3 object.elasticsearch.BulkSinkStreamelasticsearch.DocUpdateobjects to the
elasticsearch _bulk endpoint.SourcesTo make your own source, create a Reader class with the following interface.classMyReader(object):"""
    MyReader returns count instances of data.
    """def__init__(self,data="hello world\n",count=10):self.data=dataself.count=countdefread(self,amt):"""
        read(amt) returns $amt of data and a boolean indicating EOF.
        """ifnotamt:amt=self.countr=self.data*min(amt,self.count)self.count-=amtreturnr,self.count<=0The important thing to remember is that your read function should return an
iterable of units of data, not a single piece of data. Then wrap your reader in
the loving embrace of MakeSourceFactory.fromtubingimportsourcesMySource=sources.MakeSourceFactory(MyReader)Now it can be used in a apparatus!from__future__importprint_functionfromtubingimporttubessink=MySource(data="goodbye cruel world!",count=1)\|tubes.Joined(by=b"\n")\|sinks.Bytes()print(sinks.result)# Output: goodbye cruel world!TubesMaking your own tube is a lot more fun, trust me. First make a Transformer.classOptimusPrime(object):deftransform(self,chunk):returnlist(reversed(chunk))chunkis an iterable with a len() of whatever type of data the stream is
working with. In Transformers, you don’t need to worry about buffer size or
closing or exception, just transform an iterable to another iterable. There are
lots of examples in tubes.py.Next give Optimus Prime a hug.fromtubingimporttubesAllMixedUp=tubes.MakeTranformerTubeFactory(OptimusPrime)Ready to mix up some data?from__future__importprint_functionimportjsonfromtubingimportsources,sinksobjs=[{"number":i}foriinrange(0,10)]sink=sources.Objects(objs)\|AllMixedUp(chunk_size=2)\|sinks.Objects()print(json.dumps(sink))# Output: [{"number": 1}, {"number": 0}, {"number": 3}, {"number": 2}, {"number": 5}, {"number": 4}, {"number": 7}, {"number": 6}, {"number": 9}, {"number": 8}]SinksReally getting tired of making documentation… Maybe I’ll finish later. I have real work to do.Well.. I’m this far, let’s just push through.from__future__importprint_functionfromtubingimportsources,tubes,sinksclassStdoutWriter(object):defwrite(self,chunk):forpartinchunk:print(part)defclose(self):# this function is optionalprint("That's all folks!")defabort(self):# this is also optionalprint("Something terrible has occurred.")Debugger=sinks.MakeSinkFactory(StdoutWriter)objs=[{"number":i}foriinrange(0,10)]sink=sources.Objects(objs)\|AllMixedUp(chunk_size=2)\|tubes.JSONDumps()\|tubes.Joined(by=b"\n")\|Debugger()# Output:#{"number": 1}#{"number": 0}#{"number": 3}#{"number": 2}#{"number": 5}#{"number": 4}#{"number": 7}#{"number": 6}#{"number": 9}#{"number": 8}#That's all folks!