Package: popyrous
Popyrous(Pouya's Python Routines) A collection of useful and frequently encountered Python routines for (data) science, research, development, and everyday life.Author: Pouya P. Niaz (pniaz20@ku.edu.tr,pouya.p.niaz@gmail.com)Version: 0.0.12Last Update: February 15, 2024This is a collection of Python routines for the following purposes:Checking for packages and installing missing ones iwithin scripts without the need for Jupyter and symbols like "!" and "%".Reading and writing.matfiles coming to/from MathWorks MATLAB software.Building and manipulating time series data using sliding windows, low-pass filtering, etc.Building flexible and easy-to-use datasets for data analysis or machine learning out of structured time series experiments (multiple subjects, conditions, repetitions, etc.).Downloading data/files from the internet and Google Drive, using simple functions.Compressing or extracting Zip files with LZMA, etc., using simple functions.Install with:pipinstallpopyrous1- IntroThis package is a collection of routines I have widely used in my scientific, academic and engineering life.
It holds functionality for data and file manipulation, some tools for manipulating time series data,
some tools for extracting machine-learning-ready time series datasets from tabular timeseries data of structured experiments,
i.e., experiments performed with multiple subjects, under multiple conditions, with many repetitions, and so forth.The contents and applications of this package are described briefly below. However, extensive documentation is provided in the docstrings of
all functions and classes in the code, which is where you should look for further information.2- Contents and Submodules2-1- matlabThis submodule contains functions for reading and writing data to and from.matfiles.type_compatible(typ): Determining whether or not a Python data type is compatible for writing into.matfiles.save_workspace(filename, masterdict): Save dictionary holding variables and data into.matfile.load_workspace(filename, dictname): Load contents of.matfile into an (existing or new) dictionary.2-2- packagesThis submodule contains functions for checking which packages are installed in the environment without having to be in a notebook and running commands with!or%.
Also, you can check for a list of required packages (with or without required versions) and install missing packages, or wrong-versioned packages at the same time.get_package_list(): Get list (dictionary with keys being packages and values being versions) of packages in the (conda) environment.check_packages(pkglst, install_missing, **kwargs): Get a list of required packages and see if they are all installed, installing the missing ones in the process.Example:frompopyrous.packagesimportcheck_packagescheck_packages(["numpy","scipy","pandas==1.5.2"],install_missing=True,reinstall_wrong_versions=True)2-3- timeseriesThis submodule contains some classes and functions for working easily and efficiently with time series data.
You can filter data, pass it through sliding window and extract data for machine/deep learning, etc.
Also, given the dataframe of a tructured time series experiment where multiple subjects repeated an experiment
multiple times under various conditions, you can get their data, preprocess, post-process, filter, extract sliding window, etc.
and then keep some subjects, conditions, or trials for training and the rest for testing (for data analysis or machine learning), and so forth.2-3-1- sliding_windowThesliding_windowfunction gets tabular timeseries data, extracts sliding windows from it, then downsamples or inverts them, etc. then returns them.
Sliding windows of time series data is used for time series modeling, prediction, classification, regression and forecasting problems.2-3-2- datasetsTabularDataset: A class for reading time series data from an array, downsampling, preprocessing, and extracting sliding windows from it.make_squeezed_dataset(hparams, inputs, outputs, **kwargs): Gets inputs/outputs, returns squeezed (2D) sliding window dataset ready to be fed to, e.g., an ANN model.make_unsqueezed_dataset(hparams, inputs, outputs, **kwargs): Gets inputs/outputs, returns unsqueezed (3D) sliding window dataset ready to be fed to, e.g., an LSTM model.2-3-3- experimentTimeseriesExperiment: A class that gets a single dataframe containing the time series data of a series of structured experiments where
there are multiple subjects, repetitions and trials. The data can then be processed such that data of each trial is separated and processed individually,
some subjects, conditions or trials are kept for training/testing, there is preprocessing before extracting sliding windows, and postprocessing after it,
and so on. This class comes in handy when the data of such a structured series of experiments needs to be processed and fed to a machine learning model, for instance.generate_cell_array: A function, which is a more concise version of the above class, doing everything in one shot and returning everything together.2-3-4- filtSome functions for low-pass filtering time series data.butter_lowpass_filter_forwardfilters input data with a digital Butterworth low-pass filter gvien sampling and cutoff frequncies, and filter order.
This filter is causal, and only goes forward in time. It does not see its future. It is used for real-time implementations.
Because this filter is causal, it induces a phase shift, so the filtered signal will have a delay relative to the real signal.
The lower the cutoff frequency, the longer the delay. This function in turn uses thesosfiltfiltutility of SciPy.butter_lowpass_filter_back_to_backfilters input data similarly, but usessosfiltfiltto go back to back, so it looks both to past and future.
It can only smooth the data offline, since it has access to the future as well. Unlike the previous causal filter, it has no phase shift.butter_highpass_filter_forwardandbutter_highpass_filter_back_to_backcan be used for similar purposes.The newbutter_filterfunction encompasses virtually any kind of digital Butterworth filter, including all of the above.2-3-5- metricsSome metrics used for time-series classification, etc.tsc_metrics: Time-series classification metrics, including accuracy, f1 score, concurrency (transitioning on time) and consistency (not changing prediction in consistent non-transitioning portions of the data)2-3-6- cwtContinmuous Wavelet Transformcwt_for_batch: gets a numpy array of shape, e.g., (batchsize, channels, seqlen) [could be any shape, as long as time is the last dimension]
and returns an array of its CWT coefficients.
Additionally, it can downsample it and remove the last row and column. Returns a (batchsize, channels, coefs, seqlen) dataset of 2D images.cwt_for_tensor: gets a data tensor of any shape and simply performs CWT on it. Takes the last dimension as time, and adds a dimension
to the beginning, containing coefficients.2-4- webThis submodule contains some web-related functions for downloading files from the internet or Google Drive, storing them, reading their contents, etc.download_google_drive_file(shareable_link, output_file): Gets shareable link of a Google Drive file, and downlaods it.download(url, filename, **kwargs): Downlaods material from the internet, and reads its content or stores in a file.2-5- zipfilesThis submodule contains some functions for compressing/decompressing zip files.extract_files(fileName): Extracts everything in the zip file.compress_files(file_name, **kwargs): Compresses files into a zip file. Options for compression method, etc. are provided.2-6- mlThis submodule contains some machine-learning-related code. For now, it just contains a function for pretty plotting confusion matrices (see credits).make_confusion_matrixgets a confusion matrix and some parameters, and pretty plots it.3- LicenseThis package is built with MIT license.4- CreditsPretty plotting confusion matrix:Dennis Thttps://github.com/DTrimarchi10/confusion_matrixhttps://medium.com/@dtuk81/confusion-matrix-visualization-fc31e3f30fea