Package: e2fyi-utils
e2fyi-utilse2fyi-utilsis ane2fyinamespaced python package withutilssubpackage
(i.e.e2fyi.utils) which holds a collections of useful helper classes and
functions to interact with various cloud resources.API documentation can be found athttps://e2fyi-utils.readthedocs.io/en/latest/.Change logs are available inCHANGELOG.md.Python 3.6 and aboveLicensed underApache-2.0.Quickstartpipinstalle2fyi-utils>=0.2S3StreamS3Streamrepresents the data stream of a S3 resource, and provides static
methods to convert any python objects into a stream. This is generally used withS3Resourceto upload or download resource from S3 buckets.NOTE:str,float,int, andboolwill be saved as txt files with mime type "text/plain".pydanticmodels,dictorlistwill be saved as json files with mime type "application/json" (fallback to pickle if unable to serialize into json string).pandasdataframe or series can be saved as either a csv ("application/csv") or json format ("application/json").path to files will be read withopenand mime type will be inferred (fallback to "application/octet-stream").all other python objects will be pickled withjoblib.importioimportpandasaspdfrome2fyi.utils.awsimportS3StreamfrompydanticimportBaseModel# create a s3 streamstream=S3Stream(io.StringIO("random text"),"text/plain")print(stream.read())# prints "random text"print(stream.content_type)# prints "text/plain"# stringstream=S3Stream.from_any("hello world")print(stream.read())# prints "hello world"print(stream.content_type)# prints "text/plain"# dictstream=S3Stream.from_any({"foo":"bar"})print(stream.read())# prints "{"foo": "bar"}"print(stream.content_type)# prints "application/json"# pandas dataframe as csvdf=pd.DataFrame([{"key":"a","value":1},{"key":"b","value":2}])stream=S3Stream.from_any(df,index=False)# do not include index columnprint(stream.read())# prints string as csv format for the dataframeprint(stream.content_type)# prints "application/csv"# pandas dataframe as jsonstream=S3Stream.from_any(df,orient="records")# orient dataframe as recordsprint(stream.read())# prints string as json list for the dataframeprint(stream.content_type)# prints "application/json"# pydantic modelclassPerson(BaseModel):name:strage:intstream=S3Stream.from_any(Person(name="william",age=21))print(stream.read())# prints "{"name": "william", "age": 21}"print(stream.content_type)# prints "application/json"# any other python objectsclassPet:name:strage:intstream=S3Stream.from_any(Pet(name="kopi",age=1))print(stream.read())# prints some binary bytesprint(stream.content_type)# prints "application/octet-stream"S3ResourceS3Resourcerepresents a resource in S3 currently or a local resource that will
be uploaded to S3.S3Resourceconstructor will automatically attempts to convert
any inputs into aS3Stream, but for more granular controlS3Stream.from_anyshould be used instead to create theS3Stream.S3Resourceis a readable stream - i.e. it hasread,seek, andclose.NOTE:Seehttps://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.htmlfor additional keyword arguments that can be passed to S3Resource.Example: Creating S3Resource from local python object or file.importboto3frome2fyi.utils.awsimportS3Resource,S3Stream# create custom s3 clients3client=boto3.client('s3',aws_access_key_id=ACCESS_KEY,aws_secret_access_key=SECRET_KEY)# creates a local copy of s3 resource with S3Stream from a local fileobj=S3Resource(# full path shld be "prefix/some_file.json"filename="some_file.json",prefix="prefix/",# bucket to download from or upload tobucketname="some_bucket",# or "s3n://" or "s3://"protocol="s3a://",# uses default client if not provideds3client=s3client,# attempts to convert to S3Stream if input is not a S3Streamstream=S3Stream.from_file("./some_path/some_file.json"),# addition kwarg to pass to `s3.upload_fileobj` or `s3.download_fileobj` methodsMetadata={"label":"foo"})print(obj.key)# prints "prefix/some_file.json"print(obj.uri)# prints "s3a://some_bucket/prefix/some_file.json"# will attempt to fix prefix and filename if incorrect filename is providedobj=S3Resource(filename="subfolder/some_file.json",prefix="prefix/")print(obj.filename)# prints "some_file.json"print(obj.prefix)# prints "prefix/subfolder/"Example: Upload S3Resource to S3# creates a local copy of s3 resource with some python objectobj=S3Resource(filename="some_file.txt",prefix="prefix/",bucketname="some_bucket",stream={"some":"dict"},)# upload obj to s3 bucket "some_bucket" with the key "prefix/some_file.json"# with the json string content.obj.save()# upload to s3 bucket "another_bucket" instead with a metadata tag.obj.save("another_bucket",MetaData={"label":"foo"})Example: Read S3Resource from S3frompydanticimportBaseModel# do not provide a stream input to the S3Resource constructorobj=S3Resource(filename="some_file.json",prefix="prefix/",bucketname="some_bucket",content_type="application/json")# read the resource like a normal file object from S3data=obj.read()print(type(data))# prints <class 'str'># read and load json string into a dict or list# for content_type == "application/json" onlydata_obj=obj.load()print(type(data_obj))# prints <class 'dict'> or <class 'list'># read and convert into a pydantic modelclassPerson(BaseModel):name:strage:int# automatically unpack the dictdata_obj=obj.load(lambdaname,age:Person(name=name,age=age))# alternatively, do not unpackdata_obj=obj.load(lambdadata:Person(**data),unpack=False)print(type(data_obj))# prints <class 'Person'>S3BucketS3Bucketis an abstraction of the actual S3 bucket with methods to interact
with the actual S3 bucket (e.g. list objects inside the bucket), and some utility
methods.Prefix rules can also be set during the creation of theS3Bucketobject - i.e.
enforce a particular prefix rules for a particular bucket.Quickstartfrome2fyi.utils.awsimportS3Bucket# prints key for all resources with prefix "some_folder/"forresourceinS3Bucket("some_bucket").list("some_folder/"):print(resource.key)# prints key for the first 2,000 resources with prefix "some_folder/"forresourceinS3Bucket("some_bucket").list("some_folder/",max_objects=2000):print(resource.key)# creates a s3 bucket with prefix ruleprj_bucket=S3Bucket("some_bucket",get_prefix=lambdaprefix:"prj-a/%s"%prefix)forresourceinprj_bucket.list("some_folder/"):print(resource.key)# prints "prj-a/some_folder/<resource_name>"print(resource.stats)# prints metadata for the resource (e.g. size, etag)# get obj key in the bucketprint(prj_bucket.create_resource_key("foo.json"))# prints "prj-a/foo.json"# get obj uri in the bucket# prints "s3a://some_bucket/prj-a/foo.json"print(prj_bucket.create_resource_uri("foo.json","s3a://"))# create S3Resource in bucket to read infoo=prj_bucket.create_resource("foo.json","application/json")# read "s3a://some_bucket/prj-a/foo.json" and load as a dict (or list)foo_dict=foo.load()# create S3Resource in bucket and save to "s3a://some_bucket/prj-a/foo.json"prj_bucket.create_resource("foo.json",obj={"foo":"bar"}).save()e2fyi.utils.core.MaybeMayberepresents an uncertain object (exception might be raised so no value
will be returned). This is generally used inside a function where all exceptions
will be caught.NOTE:Maybe.valueis the actual returned value.Maybe.exceptionis the exception caught (if any).Maybe.with_defaultmethod can be used to provide a default value if no value
is returned.Maybe.is_okmethod can be used to check if any value is returned.importloggingfrome2fyi.utils.coreimportMaybedefload_from_file(filepath:str)->Maybe[string]:"""loads the content of a file."""try:withopen(filepath,"r")asfp:returnMaybe(fp.read())exceptIOErroraserr:returnMaybe(exception=err)data=load_from_file("some_file.json")# print with a default value fallbackprint(data.with_default("default value"))# print data if ok, else log exceptionifdata.is_ok:print(data)else:logging.exception(data.exception)