Package: to-file-like-obj
to-file-like-objPython utility function to convert an iterable ofbytesorstrto a readable file-like object.It can be seen as the inverse of thetwo-argument iter function. The iter function allows conversion of file-like objects to iterables, but the function here converts from iterables to file-like objects. This allows you to bridge the gap between incompatible streaming APIs - passing data from sources that offer data as iterables to destinations that only accept file-like objects.FeaturesInherits fromIOBase- some APIs require thisThe resulting file-like object is well-behaved - it does not return more data than requestedIt evaluates the iterable lazily - avoiding loading all its data into memoryUnder the hood copying is avoided as much as possibleConverts iterables ofbytesto bytes-based file-like objects, which can be passed toboto3's upload_fileobjor toio.TextIOWrapperwhich is useful in stream CSV parsing.Converts iterables ofstrto text-based file-like objects, which can be passed topsycopg2's copy_expertInstallationpipinstallto-file-like-objUsageIf you have an iterable ofbytesinstances, you can pass them to theto_file_like_objfunction, and it will return the corresponding file-like object.fromto_file_like_objimportto_file_like_objf=to_file_like_obj((b'one',b'two',b'three',))print(f.read(5))# b'onetw'print(f.read(6))# b'othree'If you have an iterable ofstrinstances, you can pass them to theto_file_like_obj, along withbase=stras a named argument, and it will return the corresponding file-like object.fromto_file_like_objimportto_file_like_objf=to_file_like_obj(('one','two','three',),base=str)print(f.read(5))# 'onetw'print(f.read(6))# 'othree'These examples have the iterables hard coded and so loaded all into memory. However,to_file_like_objworks equally well with iterables that are generated dynamically, and without loading them all into memory.Recipe: parsing a CSV file while downloading itUsinghttpxit's possible to use theto_file_like_objfunction to parse a CSV file while downloading it.importcsvimportioimporthttpxfromto_file_like_objimportto_file_like_objwithhttpx.stream("GET","https://www.example.com/my.csv")asr:bytes_iter=r.iter_bytes()f=to_file_like_obj(bytes_iter)lines_iter=io.TextIOWrapper(f,newline='',encoding='utf=8')rows_iter=csv.reader(lines):forrowinrows_iter:print(row)Recipe: parsing a zipped CSV file while downloading itSimilarly, usinghttpxandstream-unzip, it's possible to use theto_file_like_objfunction to robustly parse a zipped CSV file while downloading it.importcsvimportioimporthttpxfromstream_unzipimportstream_unzipfromto_file_like_objimportto_file_like_objwithhttpx.stream("GET","https://www.example.com/my.zip")asr:zipped_bytes_iter=r.iter_bytes()# Assumes a single CSV file in the ZIP (in the case of more, this will concatanate them together)unzipped_bytes_iter=(chunkfor_,_,chunksinstream_unzip(zipped_bytes_iter)forchunkinchunks)f=to_file_like_obj(unzipped_bytes_iter)lines_iter=io.TextIOWrapper(f,newline='',encoding='utf=8')rows_iter=csv.reader(lines):forrowinrows_iter:print(row)Recipe: uploading large objects to S3 while receiving their contentsboto3's upload_fileobjis a powerful function, but it's not obvious that it can be used with iterables of bytes that are returned from various APIs, such as those inhttpx.importhttpxfromto_file_like_objimportto_file_like_objs3=boto3.client('s3')withhttpx.stream("GET","https://www.example.com/my.zip")asr:bytes_iter=r.iter_bytes()f=to_file_like_obj(bytes_iter)s3.upload_fileobj(f,'my-bucket','my.zip')Recipe: stream-zipping while uploading to S3stream-zipcan be used with boto3 and this package to upload objects to S3 while zipping them.importdatetimeimporthttpxfromstatimportS_IFREGfromto_file_like_objimportto_file_like_objfromstream_zipimportZIP_32,stream_zips3=boto3.client('s3')withhttpx.stream("GET","https://www.example.com/my.txt")asr:unzipped_bytes_iter=r.iter_bytes()member_files=(('my.txt',datetime.now(),S_IFREG|0o600,ZIP_32,unzipped_bytes_iter,),)zipped_bytes_iter=stream_zip(member_files)f=to_file_like_obj(zipped_bytes_iter)s3.upload_fileobj(f,'my-bucket','my.zip')