Package: iotoolz
iotoolziotoolzis an improvement overe2fyi-utilsand is inspired partly bytoolz.iotoolzis a lib to help provide a consistent dev-x for interacting with any IO resources.
It provides an abstract classiotoolz.AbcStreamwhich mimics python's nativeopenvery closely (with some additional parameters and methods such assave).API documentation can be found athttps://iotoolz.readthedocs.io/en/latest/.Change logs are available inCHANGELOG.md.Python 3.6.1 and aboveLicensed underApache-2.0.Supported streamsCurrent the following streams are supported:iotoolz.FileStream: wrapper over built-inopenfunction (file://)iotoolz.TempStream: in-memory stream that will rollover to disk (tmp://,temp://)iotoolz.HttpStream: http or https stream implemented withrequests(http://,https://)iotoolz.extensions.S3Stream: s3 stream implemented withboto3(s3://,s3a://,s3n://)iotoolz.extensions.MinioStream: s3 stream implemented withminio(minio://)Installation# install the default packages only (most lite-weight)pipinstalliotoolz# install dependencies for specific extensionpipinstalliotoolz[boto3]# install all the extraspipinstalliotoolz[all]Available extras:all: All the optional dependenciesboto3:boto3foriotoolz.extensions.S3Streamminio:minioforiotoolz.extensions.MinioStreamchardet: install LGPLchardetfor guessing encoding. Otherwisecharset_normalizerwill be used.QuickstartThe helper objectiotoolz.streams.stream_factoryis a default singleton ofiotoolz.streams.Streamsprovided to support most of the common use cases.iotoolz.streams.open_stream(aliasiotoolz.streams.Stream) and is a util method
provided by the singleton helper to create a stream object. This method accepts the same
arguments as python'sopenmethod with the following additional parameters:data: optional str or bytes that will be passed into the streamfileobj: optional file-like object which will be copied into the streamcontent_type: optional mime type information to describe the stream (e.g. application/json)inmem_size: determines how much memory to allocate to the stream before rolling over to local file system. Defaults to no limits (may result in MemoryError).schema_kwargs: optional mapping of schemas to their default kwargs.Basic Setupfromiotoolz.streamsimport(set_schema_kwargs,set_buffer_rollover_size,)# set params to pass to the Stream obj handling https# i.e. HttpStream (implemented with requests)set_schema_kwargs("https",verify=False,# do not verify ssl certuse_post=True# use POST instead of PUT when writing to https)# use a custom client for S3Stream (via boto3)set_schema_kwargs("s3",client=boto3.client("s3",aws_access_key_id=ACCESS_KEY,aws_secret_access_key=SECRET_KEY,aws_session_token=SESSION_TOKEN,))# use a custom credentials for MinioStreamset_schema_kwargs("minio",access_key=ACCESS_KEY,secret_key=SECRET_KEY,secure=True,)# buffer will rollover to disk if the data is more than 100 MB# (default is everything is in-memory - may result in memory error)set_buffer_rollover_size(10**8)Opening streamsYou can open any stream just like python's built-inopenmethod.importpandasfromiotoolzimportopen_stream# print line by line some data in from a https endpoint# and do not verify the ssl cert of the https endpointwithopen_stream("https://foo/bar/data.txt",mode="r",schema_kwargs={"https":{"verify":False}})asstream:forlineinstream:print(line)# POST some binary content to a http endpoint (default is PUT)withopen_stream("https://foo.bar/api/data","wb",use_post=True)asstream:stream.write(b"hello world")# Copying a local file to s3withopen_stream("path/to/data.csv","r")ascsv_source,open_stream("s3://bucket/foobar.txt?StorageClass=STANDARD","w")ass3_sink:# pipe content in csv_source to tmpsinkcsv_source.pipe(s3_sink)# load to pandas dataframe from s3 fileobjwithopen_stream("s3://bucket/foobar.csv","r")ascsv:df=pd.read_csv(csv)TempStreamTempStreamis a stream can functions like a virtual file system in memory.importgcfromiotoolzimportStream,exists,glob,iter_dir# this stream can be garbage collectedStream("tmp://foo/bar/data.txt",data="foobar")# True if not gc yet, False if already gcexists("tmp://foo/bar/data.txt")# force gcgc.collect()# will not existexists("tmp://foo/bar/data.txt")# create temp stream with strong ref (hence will not be gc)s1=Stream("tmp://foo/bar/data.txt",data="foobar")s2=Stream("tmp://foo/example.txt",data="...")# returns s1 and s2iter_dir("tmp://foo/")# returns s1 onlyglob("tmp://foo/bar/*.txt")Stream-like operationsStreamis an alias ofopen_stream, both methods return a concreteAbcStreamobject.
You can treat the object as both a "file-like" and "stream-like" object - i.e. you can
read, write, seek, flush, close the object.NOTEBy default, the underlying buffer is in-memory. You can enable rollover to disk by
passing theinmem_sizearg to the method, or update the defaultinmem_sizevalue
with theiotoolz.streams.set_buffer_rollover_sizemethod.fromiotoolzimportopen_stream,Stream,set_buffer_rollover_size# `Stream` is an alias of `open_stream`assertopen_stream==Stream# rollover to disk if data is over 100 MBset_buffer_rollover_size(10**8)# you can overwrite the default kwargs here alsostream=Stream("path/to/data.txt",mode="rw",# you can both read and write to a stream)# stream is lazily evaluated, nothing will be buffered until you call some methods# that requires the datadata=stream.read()# will attempt to provide encoding and content_type if not provided when opening the streamprint(stream.encoding)print(stream.content_type)# stream has the same interface as an IO object - i.e. u can seek, flush, close, etcstream.seek(5)# go to offset 5 from start of bufferstream.write("replace with this text")stream.seek(0,whence=2)# go to end of bufferstream.write("additional text after original eof")# continue writing to the end of the bufferstream.save()# flush save the entire buffer to the same dst locationstream.close()# close the streamPath-like operationsexists,mkdir,iter_dirandglobare path-like methods that are available to the
stream object. These methods mimics their equivalent inpathlib.Pathwhen appropriate.methodsupported streamsdescstatsAll Streamsreturn the StreamInfo for an existing resourceunlink,delete,removeAll StreamsDelete and remove the stream (except forTempStreamwhere the buffer is cleared instead)existsAll Streamscheck if a stream points to an existing resource.mkdirFileStreamcreate a directory.rmdirFileStream,TempStream, andS3Stream,remove recursively everything in the directory.iter_dirFileStream,TempStream, andS3Streamiterate thru the streams in the directory.globFileStream,TempStream, andS3Streamiterate thru the streams in the directory that match a pattern.importitertoolsfromiotoolzimportStream,mkdir,iter_dir,glob,exists# similar to 'mkdir -p'mkdir("path/to/folder",parents=True,exist_ok=True)Stream("path/to/folder").mkdir(parents=True,exist_ok=True)# list object in an s3 bucketiter_dir("s3://bucket/prefix/")forstreaminStream("s3://bucket/prefix/").iter_dir():print(stream.uri)# find s3 objects with a specific patternglob("s3://bucket/prefix/*txt")forstreaminStream("s3://bucket/prefix/").glob("*.txt"):print(stream.uri)# existsexists("s3://bucket/prefix/foo.txt")# statsinfo=stats("s3://bucket/prefix/foo.txt")print(info.name)print(info.content_type)print(info.encoding)print(info.last_modified)print(info.etag)print(info.extras)# delete resourceunlink("s3://bucket/prefix/foo.txt")# rm all key with prefixrmdir("s3://bucket/prefix/")Piping streamspipeis method to push data to a sink (similar to NodeJS stream except it has no
watermark or buffering).fromiotoolz.streamsimportopen_streamlocal_file=open_stream("path/to/google.html",content_type="text/html",mode="w")temp_file=open_stream("tmp://google.html",content_type="text/html",mode="wb")# when source is closed, all sinks will be closed alsowithopen_stream("https://google.com")assource:# writes to a temp file then to a local file in sequencesource.pipe(temp_file).pipe(local_file)local_file2=open_stream("path/to/google1.html",content_type="text/html",mode="w")local_file3=open_stream("path/to/google2.html",content_type="text/html",mode="w")# when source is closed, all sinks will be closed alsowithopen_stream("tmp://foo_src",mode="w")assource:# writes in a fan shape mannersource.pipe(local_file2)source.pipe(local_file3)source.write("hello world")TODO support transform streams so that pipe can be more usefulLicense