Package: pbu
Python Basic UtilitiespbuAvailable onPyPiTable of ContentsInstallationUsageClassesJSON- a JavaScript-like dictionary access helperLogger- a wrapper around the Python logging frameworkTimeSeries- powerful helper class to organise time seriesBasicMonitor- monitor class orchestrating regular operationsConstantListing- a parent class allowing to fetch attribute values from a constant classPerformanceLogger- a utility class to log runtime performance of processesPerformanceTracker- a utility class to track performance of a repeated processBasicConfig- application utility class managing access to environment variablesJsonDocument- a class that can serialise/deserialise a dictionary into a class instanceFunctionslist_to_jsonjson_to_listdefault_optionsdefault_valuelist_find_onelist_map_filterlist_joinnot_noneDatetime Functionsweighted_meannormaliseInstallationInstall via pip:pipinstallpbuUsageOptional: If you have a requirement.txt file, you can addpbu:pbuThen, simply import the class / module you need:frompbuimportJSON# and start using itobj=JSON({"my":{"obj":"content"}})print(obj.my.obj)ClassesJSONThis is an adaptation of the nativedictclass, providing Javascript-like dictionary access using the "dot-notation"
(e.g.person.relations[0].address.street) rather than the Python-native bracket notation (e.g.person["relations"][0]["address"]["street"]). It overrides the basic__getattr__and__setattr__methods as a
shortcut to manage the dictionary content.ExamplefrompbuimportJSONmy_obj=JSON({"initial":"content"})print(my_obj.initial)# prints out "content"my_obj.initial={"a":5,"b":3}print(my_obj.initial.a+my_obj.initial.b)# prints out 8my_obj.initial.b=13print(my_obj.initial.a+my_obj.initial.b)# prints out 18my_obj.extension=10print(my_obj.extension)# prints out 10LoggerThis is a basic logger allowing to write log files, forlogger.infoit writes a debug.log and forlogger.errororlogger.exceptionit writes an error.log file.ExamplefrompbuimportLoggerlogger=Logger(name="logger-name")logger.debug("Some debug message goes here")logger.error("Error executing something")logger=Logger(name="logger-name",log_folder="./logs")logger.debug("This will create the debug.log and error.log in the ./logs folder")TimeSeriesThe time series class is a helper utility, that allows to compile complex time-series, offering functionality to add
time series, remove time series and most importantly align time series with timestamps to a previously defined
resolution by interpolating missing values and re-aligning measurements within the tolerance of the provided time
series.It supports 2 different structures:List of Dictionary Itemsfromdatetimeimportdatetime,timedeltalist_of_dict=[{"date_time":datetime.now(),"measurement_1":12,"measurement_2":15},{"date_time":datetime.now()+timedelta(hours=1),"measurement_1":10,"measurement_2":16},{"date_time":datetime.now()+timedelta(hours=2),"measurement_1":9,"measurement_2":12},]Dictionary of Listsfromdatetimeimportdatetime,timedeltadict_of_list={"date_time":[datetime.now(),datetime.now()+timedelta(hours=1),datetime+timedelta(hours=2)],"measurement_1":[12,10,16],"measurement_2":[15,16,12],}ExamplefrompbuimportTimeSeriesfromdatetimeimportdatetime,timedelta# initial time series base data (you can add measurements as well or provide as list of dictionariesdict_of_list={"date_time":TimeSeries.create_date_range(datetime.now(),datetime.now()+timedelta(days=1),timedelta(hours=3)),}# init time seriests=TimeSeries(input_data=dict_of_list,date_time_key="date_time")# add values (ensure same length as date_time series)ts.add_values("measurement_1",[12,10,16,10,5,8,12,9])# you can translate into a list of dictionary items (keys are maintained)list_of_dict=ts.translate_to_list_of_dicts()# extract data series from the time seriesmeasurement_1=ts.get_values("measurement_1")# create new series that provides same value for all timestampsts.fill_values("constant_series",5)# remove a series from the total data structurets.remove_series("constant_series")# re-sample data to 5 minute resolution, interpolating values, also pre-pending another day in front of the time seriests.align_to_resolution(resolution=timedelta(minutes=5),start_date=datetime.now()-timedelta(days=1))# this will result in "interpolated" values for the first day, using the first value (12) to fill missing valuesprint(len(ts.translate_to_list_of_dicts()))# 12 an hour, 2 days, 48 * 12 = ~576 items# the same can also be achieved by:ts.set_resolution(timedelta(minutes=5))# no need to provide resolution nowts.align_to_resolution(start_date=datetime.now()-timedelta(days=1))BasicMonitorAn abstract class providing base-functionality for running monitors - threads that run a specific routine in a regular
interval. This can be an executor waiting for new tasks to be processed (and checking every 5 seconds) or a thread that
monitors some readout in a regular interval. The monitor is wrapped to re-start itself, in case of errors.ExamplefrompbuimportBasicMonitorclassMyOwnMonitor(BasicMonitor):def__init__(self,data):super().__init__(monitor_id="my_id",wait_time=5)# waits 5 seconds between each execution loopself.data=datadefrunning(self):whileself.active:# your code goes here (example):# result = fetch_data(self.data)# store_result(result)self.wait()If you want to run in a regular interval, therunningmethod needs to be slightly modified:fromtimeimporttimefrompbuimportBasicMonitorclassMyRegularOwnMonitor(BasicMonitor):def__init__(self,data):super().__init__(monitor_id="another_id",wait_time=60,run_interval=True)# execute every 60 secondsself.data=datadefrunning(self):whileself.active:start_ts=time()# capture start of loop# your code goes here (example):# result = do_something(self.data)# store_result(result)self.wait(exec_duration=round(time()-start_ts))# include the execution durationOptional constructor parametersYou can also pass a custom logger ascustom_loggerargument to the constructor. By default it will use thepbu.Loggerand log major events such as start/stop/restart and errors.Passing aping_intervalparameter allows you to check for overdue jobs more often than the wait time. For example
you could have await_timeof 1800s (30 min) and aping_intervalof 60s, which allows you to not miss out on an
execution if your machine running the monitor should sleep (e.g. on a laptop when you put it on standby, the sleep
timer stops). By default this is60seconds (or thewait_time, if thewait_timeis lower than 60s)Manage and run monitorimportthreadingdefstart_monitor_thread(monitor):"""Thread function to be run by the new thread.:param monitor: BasicMonitor - an instance of sub-class of BasicMonitor"""# start the monitormonitor.start()# create monitor instance of your own class that implements BasicMonitorregular_monitor=MyRegularOwnMonitor(data={"some":"data"})# create thread with start-up function and start itt=threading.Thread(target=start_monitor_thread,args=(regular_monitor,),daemon=True)t.start()# in a separate piece of code (e.g. REST handler or timer) you can stop the monitor instanceregular_monitor.stop()Stopping a monitor doesn't interrupt the current thread. If the monitor is for example in a wait period and you send thestopsignal, the thread will still run until the wait period passes.In an API scenario, I recommend using adictorlistto cache monitors and retrieve them via the API using theto_json()method for identification. This then allows you to signal starting / stopping of monitors by providing the
monitor ID and lookup the monitor instance in the monitor cache.BasicMonitorMethodsstart()- starts the monitorstop()- stops the monitorto_json()- returns a dictionary with basic monitor technical information (id, state, wait behaviour, etc)wait_till_midnight()- waits till the next midnight in your machines time zonewait(exec_duration=0)- waits for the time specified in the constructor and in case ofrun_interval=Truefor the
optionalexec_duration, if provided.ConstantListingManaging constants is good practice for avoiding typos. Imagine the following class:classTags:GEO="GEO"EQUIPMENT="EQUIPMENT"This allows you to just do:Tags.GEOallowing you to use your IDEs auto-complete, avoiding typos. But if you want to
programmatically getallpossible values forTags, you can usepbu'sConstantListingclass:frompbuimportConstantListingclassTags(ConstantListing):GEO="GEO"EQUIPMENT="EQUIPMENT"list_of_values=Tags().get_all()# will return ['GEO', 'EQUIPMENT']PerformanceLoggerThis utility class allows to print out or log runtime performance expressed as time delta between a start time and an
end time.Basic usage:frompbuimportPerformanceLoggerperf=PerformanceLogger()perf.start()# this is optional and will reset the start-time# do something useful...perf.checkpoint(message="Step 1")# will print "Step 1 took <timedelta># some some more useful stuff...perf.finish(message="Something useful")# will print out the whole duration from start to finishYou can omit the message of acheckpointcall if you don't need an output for an operation, but want to print out the
duration of the step that follows.You can also use a PythonLoggerobject (orpbu.Logger) instead of the message being printed out onto the console.frompbuimportLogger,PerformanceLoggerlogger=Logger("my-logger-name")perf=PerformanceLogger()# do something...perf.checkpoint()# next output will print the duration between this point and the next checkpoint call# do some more stuff...perf.checkpoint(message="Some More Stuff",logger=logger)# and even more ...perf.finish(message="Total operation",logger=logger)Methodsstart()- will reset the start time of the performance loggercheckpoint(message=None, logger=None)- creates a new checkpoint and optionally logs a messagefinish(message=None, logger=None)- prints out the total runtime sincestart()was called or the class was
initialisedPerformanceTrackerA utility class that allows to track the runtime of a repeated process and print out performance stats everynrepetitions.Basic usage:frompbuimportPerformanceTrackertracker=PerformanceTracker(operation_name="compute",print_interval=20)foriinrange(0,100):# starting the operation is thread-safe and can be executed in parallel, unique keys are getting returnedtrack_key=tracker.start_operation()# perform your operationa=i*i*itracker.end_operation(track_key)Every 20 executions, this will print out a line line this:Performance for operation 'compute' (20): Avg: 5.960464477539062e-07s | Min: 2.384185791015625e-07 | Max: 1.1920928955078125e-06with the operation name, followed by the number of executions and then avg, min and max performance in seconds.BasicConfigThis class can be used in applications to simplify access to environment variables. It is recommended to write your own
sub-class of this class, where you can provide even more convenient access. However, the class can also be used
standalone.Basic usage:importosfrompbuimportBasicConfigclassConfig(BasicConfig):def__init__(self):super().__init__(default_values={"PORT":5000,"IS_DEBUG":1,"DATA_DIRECTORY":None,},directory_keys=["DATA_DIRECTORY"],required=["DATA_DIRECTORY"])defget_port(self)->int:returnint(self.get_config_value("port"))defis_debug(self)->bool:returnint(self.get_config_value("is_debug"))==1defget_data_directory(self)->str:returnself.get_config_value("DATA_DIRECTORY")cfg=Config()# BasicConfig will ensure the directory existsresult=os.path.exists(cfg.get_data_directory())# result is TrueMethodsget_config_value(config_key, default_value=None)- retrieves a config value, the default value override is optional
as it should already be provided in thedefault_valuesof the constructor. If aconfig_keyhasn't been provided by
thedefault_valuesof the constructor, this will trigger reading the value fresh from the environment and storing it
within this class.__init__(default_values={}, directory_keys=[], required=[], env_file=".env")- super constructor, which will be used
to load the initial environment.Thedefault_valuesprovide the keys that will be extracted from the OS environment.Thedirectory_keysare config keys that will be used to run a directory check. If the provided environment value
refers to a directory that doesn't exist yet, the class will attempt to create it.Therequiredparameter provides environment keys that have to be provided by the OS environment. If they are not
available in the environment, anEnvironmentErrorwill be raised.JsonDocumentMethodsto_json()- call this to return a dict representation of the instance. This will serialise theidanddata_model_versionattributes and any attributes provided in theget_attribute_mapping()method.get_attribute_mapping()- provides a dict mapping between class attributes and JSON keys that will be used in thedictrepresentation.extract_system_fields(json: dict)- this will deserialise adictand map the_idfield to theidattribute,dataModelVersionfield todata_model_versionattribute and any field defined in theget_attribute_mapping()method.apply_updates(update, attributes = [])- overwrites attributes of the current instance with theupdate. The list
of attributes has to be specified and is empty by default. Theupdatemust be of the same type as the current
instance. If anattributeis listed that does not exist, a warning will be issued.Static Methods.from_json(json)- this method has to be implemented by any sub-class and is responsible for
deserialising a JSON document into an instance of your sub-class. The instance methodextract_system_fields(json)can be used to map most simple attributes - i.e. any attributes provided in theget_attribute_mapping()method.Functionslist_to_jsonfrompbuimportlist_to_json# assuming we have `my_store` as an instance of MongoDB store or MySQL store, you can:list_of_dictionaries=list_to_json(item_list=my_store.get_all())# output is a list of dictionariesThis function operates on lists of objects inheriting fromJsonDocumentand converts them into dictionaries using theto_json()method of any object passed into the function. Objects passed into the functionrequiretheto_json()method and need to return the dictionary representation of the object. This function is just a mapping shortcut.list_from_jsonfrompbuimportlist_from_json# assuming we have a class `MyClass` that inherits from `JsonDocument` and implements the `from_json()` methodlist_from_json(item_list=[{"a":1,"b":2},{"a":3,"b":4}],class_type=MyClass)default_optionsfrompbuimportdefault_optionsDEFAULTS={"a":1,"b":2,"c":3,}result=default_options(default=DEFAULTS,override={"b":4,"d":5})# result is: {"a": 1, "b": 4, "c": 3, "d": 5}If you want to avoid additional keys other than the keys in DEFAULTS, you can provide a third argument:frompbuimportdefault_optionsDEFAULTS={"a":1,"b":2,}result=default_options(default=DEFAULTS,override={"b":4,"d":5},allow_unknown_keys=False)# result is: {"a": 1, "b": 4}default_valuefrompbuimportdefault_valueresult=default_value(value=None,fallback=5)# None is by default disallowed# result is 5result=default_value(value=0,fallback=5,disallowed=[None,0])# either 0 or None would return the fallback# result is 5result=default_value(0,5)# value will be used, as it doesn't match None# result is 0list_find_oneFinds the first item in a list that matches the filter function - this is a shortcut for runningfilter(..)on a list,
then checking its length and if the length is > 0 fetching the first item.frompbuimportlist_find_onemy_list=["a","b","c"]result=list_find_one(lambdax:x=="c",my_list)# result is "c"result=list_find_one(lambdax:x=="d",my_list)# result is Nonelist_map_filterA shorthand for filtering and mapping a lsit of items. The function allows to pass both lambdas (filterandmap)
into one function call. A boolean flag (filter_first=True) decides whether the filter or map operation is called
first.frompbuimportlist_map_filtermy_list=[{"name":"a","count":5},{"name":"b","count":100},{"name":"b","count":32},]result=list_map_filter(my_list,filter_func=lambdax:x["count"]%5==0,map_func=lambdax:x["name"])# result is ["a", "b"]result=list_map_filter(my_list,filter_func=lambdax:x>50,map_func=lambdax:x["count"],filter_first=False)# result is [100]list_joinA helper function that joins a list with a given token. The Python default way for joining a list of items uses the join
token (e.g. ",") and then calls.joinon that string, passing the list of items as parameter. However, unfortunately
this only accepts a list of strings and throws an error, if other types are passed (e.g. a list of numbers).This helper casts all items tostrbefore joining.frompbuimportlist_joinmy_list=["a",0,4.5,False]result=list_join(my_list,"-")# result is "a-0-4.5-False"result="-".join(my_list)# throws Error because my_list contains items of type other than `str`not_noneA helper function to filter outNonevalues from a list.frompbuimportnot_nonemy_list=["a",None,"b",None,"c"]result=not_none(my_list)# result is ["a", "b", "c"]Datetime FunctionsPBU provides some utilities to help deal with timezones and datetime objects. All timezone specifications can be made
either as a string (i.e. the name of the timezone, like"Australia/Melbourne") or aspytz.timezoneobject.combine_date_time(date, time, tz)Combines the provided date and time values.fromdatetimeimportdate,timefrompbuimportcombine_date_timeresult=combine_date_time(date(year=2021,month=12,day=25),time(hour=15,minute=12,second=6),"Australia/Perth")to_timezone(local_datetime, target_tz)Translates a datetime to the provided target timezone.fromdatetimeimportdatetimefrompytzimportutcfrompbuimportto_timezoneutc_dt=datetime(year=2021,month=12,day=25,hour=3,minute=0,tzinfo=utc)# 3:00am @ 2021-12-25perth_dt=to_timezone(utc_dt,"Australia/Perth")# > Result: 11:00am @ 2021-12-25 (+0800)to_utc(local_datetime)Shorthand forto_timezone(dt, pytz.utc)set_timezone(datetime, target_timezone)Simply replaces the timezone information without changing any of the time values of the datetime.fromdatetimeimportdatetimefrompytzimportutc,timezonefrompbuimportset_timezoneutc_dt=datetime(year=2021,month=12,day=25,hour=3,minute=0,tzinfo=utc)# 3:00am @ 2021-12-25perth_dt=set_timezone(utc_dt,timezone("Australia/Perth"))# > Result: 3:00am @ 2021-12-25 (+0800)weighted_meanProvides the mean (average) of a list of values, where the values are weighted by the provided weights (in the same
order as the value are provided). For missing weights, the default weight is 1frompbuimportweighted_meanweights=[5,3,1]values=[10,5,5,4,3]# ((10 * 5) + (3 * 5) + (1 * 5) + 4 + 3) / (5 + 3 + 1) = 7.0wm=weighted_mean(values,weights)# 7.0normaliseNormalises a numeric value between a lower and an upper boundary. The result is a value between 0.0 and 1.0. If the
provided value exceeds any of the boundaries, the boundary value will automatically be chosen (defaults to 1.0 or 0.0).It is possible to provide a smaller upper bound than lower bound, which will invert the function and provide the negated
value. As an example, if we normalise 4 between 0 and 10, we get 0.4. If we invert the boundaries to normalise 4 between
10 and 0, we get 0.6 (1.0 - 0.4).Any invalid input (None) will result in 0.0 being returned.frompbuimportnormalise# the "standard" casenorm1=normalise(value=4,min_val=0,max_val=10)# 0.4# inverted normalisationnorm2=normalise(value=4,min_val=10,max_val=0)# 0.6# exceeding the boundariesnorm3=normalise(value=11,min_val=5,max_val=10)# 1.0# float works as well as integernorm4=normalise(value=-5.0,min_val=2.3,max_val=199.0)# 0.0# inverted exceeding boundariesnorm5=normalise(value=-5,min_val=100,max_val=0.5)# 1.0# invalid inputs will return 0.0norm6=normalise(value=None,min_val=0,max_val=10)# 0.0norm7=normalise(value=5,min_val=0,max_val=None)# 0.0Since version 1.0.1 a new parameter can be passed to the function that normalises the value, but can exceed the
boundaries provided bymin_valandmax_val.frompbuimportnormalisenorm1=normalise(value=12,min_val=0,max_val=10,limit=False)# 1.2discretiseDiscretises a numeric value into a number of buckets determined by the provided precision and boolean flag indicating
whether to use the lower, upper or middle value of the bucket as the value for the bucket.frompbuimportdiscretisedisc1=discretise(value=4.5,precision=1.0,floor=True)# 4.0disc2=discretise(value=4.5,precision=0.4,ceil=True)# 4.8disc3=discretise(value=4.5,precision=0.4)# 4.6 (assumes mid-point if neither floor nor ceil is set)