Package: pelutils
pelutilsVarious utilities useful for Python projects. Features includeA simple and powerful logger with colourful printing and stacktrace loggingParsing for combining config files and command-line arguments - especially useful for algorithms with several parametersA timer inspired by Matlab'sticandtocSimple code profilerAn extension to the built-indataclassfor saving and loading dataTable formattingMiscellaneous standalone functions - seepelutils/__init__.pyData-science submodule with extra utilities for statistics, plotting withmatplotlib, and machine learning usingPyTorchLinear timeuniquefunction in the style ofnumpy.uniquepelutilssupports Python 3.9+.To install, simply runpip install pelutils.
A small subset of the functionality requiresPyTorch, which has to be installed separately.Timing and Code ProfilingSimple time taker inspired by Matlab Tic, Toc, which also has profiling tooling.# Time a taskTT.tick()<sometask>seconds_used=TT.tock()# Profile a for loopforiinrange(100):TT.profile("Repeated code")<sometask>TT.profile("Subtask")<somesubtask>TT.end_profile()TT.end_profile()print(TT)# Prints a table view of profiled code sections# Alternative syntax using with statementwithTT.profile("The best task"):<sometask># When using multiprocessing, it can be useful to simulate multiple hits of the same profilewithmp.Pool()asp,TT.profile("Processing 100 items on multiple threads",hits=100):p.map(100items)# Similar for very quick loopsa=0withTT.profile("Adding 1 to a",hits=100):for_inrange(100):a+=1# Examples so far use a global TickTock instance, which is convenient,# but it can also be desirable to use for multiple different timers, e.g.tt1=TickTock()tt2=TickTock()t1_interval=1# Do task 1 every secondt2_interval=2# Do task 2 every other secondtt1.tick()tt2.tick()whileTrue:iftt1.tock()>t1_interval:<task1>tt1.tick()iftt2.tock()>t2_interval:<task2>tt2.tick()time.sleep(0.01)Data StorageThe DataStorage class is an augmentation of the dataclass that incluces save and load functionality.
This simplifies saving data, as only save command has to be issued for all data, and it keeps type
hinting when loading data compared to e.g. a dictionary.Data is in general preserved exactly as-is when saved data is loaded into memory with few exceptions.
Notably, tuples are considered json-serializble, and so will be saved to the json file and will be
loaded as lists.Usage example:@dataclassclassResultData(DataStorage):shots:intgoalscorers:listdists:np.ndarrayrdata=ResultData(shots=1,goalscorers=["Max Fenger"],dists=np.ones(22)*10)rdata.save("max")# Now shots and goalscorers are saved in <pwd>/max/ResultData.json and dists in <pwd>/max/ResultData.pkl# Then to loadrdata=ResultData.load("max")print(rdata.goalscorers)# ["Max Fenger"]ParsingA parsing tool for combining command-line and config file arguments.
Useful for parametric methods such as machine learning.
The first argument must always be a path. This can for instance be used to put log files, results, plots etc.Consider the execution of a filemain.pywith the command line callpython main.py path/to/put/results -c path/to/config/file.ini --data-path path/to/dataThe config file could contain[DEFAULT]
fp16
learning-rate=1e-4

[LOWLR]
learning-rate=1e-5

[NOFP16]
fp16=Falsewheremain.pycontainsoptions=[# Mandatory argument with set abbreviation -pArgument("data-path",help="Path to where data is located",abbrv"-p"),# Optional argument with auto-generated abbreviation -lOption("learning-rate",default=1e-5,help="Learning rate to use for gradient descent steps"),# Boolean flag with auto-generated abbreviation -fFlag("fp16",help="Use mixed precision for training"),]parser=Parser(*options,multiple_jobs=True)# Two jobs are specified in the config file, so multiple_jobs=Truelocation=parser.location# Experiments are stored here. In this case path/to/put/resultsjob_descriptions=parser.parse_args()# Run each experimentforjobinjob_descriptions:# Get the job as a dictionaryjob_dict=job.todict()# Clear directory where job is located and put a documentation file therejob.prepare_directory()# Get location of this job as job.locationrun_experiment(job)This could then by run bypython main.py data/my-big-experiment --learning-rate 1e-5or bypython main.py data/my-big-experiment --config cfg.inior using a combination where CLI args takes precedence:python main.py data/my-big-experiment --config cfg.ini --learning-rate 1e-5wherecfg.inicould containLoggingThe logging submodule contains a simple yet feature-rich logger which fits common needs. Can be imported frompelutilsdirectly, e.g.from pelutils import log.# Configure logger for the scriptlog.configure("path/to/save/log.log")# Start loggingforiinrange(70):# Nicelog("Execution%i"%i)# Sectionslog.section("New section in the logfile")# Adjust logging levelslog.warning("Will be logged")withlog.level(LogLevels.ERROR):# Only log at ERROR level or abovelog.warning("Will not be logged")withlog.no_log:log.section("I will not be logged")# Error handling# The zero-division error and stacktrace is loggedwithlog.log_errors:0/0# Entire chained stacktrace is loggedwithlog.log_errors:try:0/0exceptZeroDivisionErrorase:raiseValueError("Denominator must be non-zero")frome# User input - acts like built-in input but logs both prompt and user inputinp=log.input("Continue [Y/n]? ")# Parse yes/no user inputcont=log.parse_bool_input(inp,default=True)# Log all logs from a function at the same time# This is especially useful when using multiple threads so logging does not get mixed updeffun():withlog.collect:log("Hello there")log("General Kenobi!")withmp.Pool()asp:p.map(fun,args)# It is also possible to create multiple loggers by importing the Logger class, e.g.log2=Logger()log2.configure("path/to/save/log2.log")Data ScienceThis submodule contains various utility functions for data science, statistics, plotting, and machine learning.Deep LearningStatisticsIncludes various commonly used statistical functions.# Get one sided z value for exponential(lambda=2) distribution with a significance level of 1 %zval=z(alpha=0.01,two_sided=False,distribution=scipy.stats.expon(loc=1/2))# Get correlation, confidence interval, and p value for two vectorsa,b=np.random.randn(100),np.random.randn(100)r,lower_r,upper_r,p=corr_ci(a,b,alpha=0.01)Plottingpelutilsprovides plotting utilities based onmatplotlib.
Most notable is theFigurecontext class, which attempts to remedy some of the common grievances withmatplotlib,
e.g. having to remember the correctkwargsandrcParamsfor setting font sizes, legend edge colour etc.frompelutils.ds.plotsimportFigure# The following makes a plot and saves it to `plot.png`.# The seaborn is style is used for demonstration, but if the `style` argument# is not given, the default matplotlib style is used.# The figure and font size are also given for demonstration, but their default# values are increased compared to matplotlib's default, as these are generally# too small for finished plots.withFigure("plot.png",figsize=(20,10),style="seaborn",fontsize=20):plt.scatter(x,y,label="Data")plt.grid()plt.title("Very nice plot")# The figure is automatically saved to `plot.png` and closed, such that# plt.plot can be used again from here.# Figure changes `matplotlib.rcParams`, but these changes are also undone# after the end of the `with statement`.The plotting utilies also include binning functions for creating nice histograms.
Thehistogramfunction produces bins based on a binning function, of which three are provided:linear_binning: Bins are spaced evenly from the lowest to the largest value of the data.log_binning: Bins are log-spaced from the lowest to the largest value of the data, which is assumed to be positive.normal_binning: Bins are distributed according to the distribution of the data, such there are more bins closer to the center of the data. This is useful if the data somewhat resembles a normal distribution, as the resolution will be the greatest where there is the most data.It is also possible to provide custom binning functions.histogramprovide bothxandycoordinates, making it simple to use with argument unpacking:importmatplotlib.pyplotaspltimportnumpyasnpfrompelutils.ds.plotsimporthistogram,normal_binning# Generate normally distributed datax=np.random.randn(100)# Plot distributionplt.plot(*histogram(x,binning_fn=normal_binning))Finally, different smoothing functions are provided.
The two most common aremoving_avgandexponential_avgwhich smooth the data using a moving average and exponential smoothing, respectively.Thedouble_moving_avgis special in that the number of smoothed data points do not depend on the number of given data points but is instead based on a given number of samples, which allows the resulting smoothed curve to not by jagged as happens with the other smoothing functions.
It also has two smoothness parameters, which allows a large degree of smoothness control.Apart from smoothness parameters, all smoothness functions have the same call signature:frompelutils.ds.plotsimportdouble_moving_avg# Generate noisy datan=100x=np.linspace(-1,1,n)y=np.random.randn(n)# Plot data along with smoothed curveplt.plot(*double_moving_avg(x,y))# If x is not given, it is assumed to go from 0 to n-1 in steps of 1plt.plot(*double_moving_avg(y))Examples of all the plotting utilities are shown in theexamplesdirectory.Supported platformsPrecompiled wheels are provided for most common platforms.
Notably, they are not provided for 32-bit systems.
If no wheel is provided,pipshould attempt a source install.
If all else fails, it is possible to install from source by pointingpipto Github directly:pip install git+https://github.com/peleiden/pelutils.git@release#egg=pelutilsIt is also possible to install from source usingpip's--no-binaryoption.