Package: commonpy
CommonPyThis is a collection of common utility functions and classes that we at the Caltech Library have found useful in our other Python projects.Table of contentsIntroductionInstallationUsageGetting helpContributingLicenseAuthors and historyAcknowledgmentsIntroductionThis repository does not constitute a single program; instead, it contains a collection of modules with utility functions and classes that we have found ourselves using repeatedly in other Python projects.InstallationThe instructions below assume you have a Python interpreter installed on your computer; if that's not the case, please firstinstall Python version 3and familiarize yourself with running Python programs on your system.OnLinux,macOS, andWindowsoperating systems, you should be able to installcommonpywithpip.  To installcommonpyfrom thePython package repository (PyPI), run the following command:python3 -m pip install commonpyAs an alternative to getting it fromPyPI, you can usepipto installcommonpydirectly from GitHub, like this:python3-mpipinstallgit+https://github.com/caltechlibrary/commonpy.gitUsageThe basic approach to using this package is to import the modules and functions you need.  For example:fromcommonpy.file_utilsimportreadableifreadable('/path/to/some/file'):# do somethingThe following subsections describe the different modules available.Data structuresThedata_structuresmodule provides miscellaneous data classes.ClassPurposeCaseFoldDictA version ofdictthat compares keys in a case-insensitive mannerCaseFoldSetA version ofsetthat compares keys in a case-insensitive mannerData utilitiesThedata_utilsmodule provides a number of miscellaneous simple functions for some common operations on data of various kinds.FunctionPurposeexpanded_range(string)Given a string of the form "X-Y", returns the list of integers it representsflattened(thing)Takes a list or dictionary and returns a recursively flattened versionordinal(integer)Returns a string with the number followed by "st", "nd, "rd", or "th"parsed_datetime(string)Returns a date object representing the given date stringpluralized(word, n, include_num)Returns a plural version ofwordifn > 1sliced(list, n)Yieldsnnumber of slices from thelisttimestamp()Returns a string for an easily-readable form of the current time and dateunique(list)Takes a list and return a version without duplicatesFile utilitiesThefile_utilsmodule provides a number of miscellaneous simple functions for some common operations on files and directories.FunctionPurposealt_extension(file, ext)Returnsfilewith the extension replaced byextcopy_file(src, dst)Copies file fromsrctodstdelete_existing(file)Deletes the givenfilefilename_basename(file)Returnsfilewithout any extensionsfilename_extension(file)Returns the extension of filenamefilefiles_in_directory(dir, ext, recursive)filtered_by_extensions(list, endings)nonempty(file)ReturnsTrueif filefileis not emptyopen_file(file)Opens thefileby calling the equivalent of "open" on this systemopen_url(url)Opens theurlin the user's default web browserreadable(dest)ReturnsTrueif file or directorydestis accessible and readablerelative(file)Returns a path string forfilerelative to the current directoryrename_existing(file)Renamesfiletofile.bakwritable(dest)ReturnsTrueif file or directorydestcan be writtenInterruptible wait and interruption handling utilitiesTheinterruptmodule includeswait(...), a replacement forsleep(...)that is interruptible and works with multiple threads.  It also provides methods to cause an interruption (including doing it by issuing a^Cto the program), check whether an interruption occurred, and other related operations.FunctionPurposeconfig_interrupt(callback, raise_ex, signal)Sets up a callback functioninterrupt()Interrupts anywaitin progressinterrupted()ReturnsTrueif an interruption has been calledraise_for_interrupts()Raises an exception ifinterrupt()has been invokedreset_interrupts()Resets the interruption flagwait(duration)Waits fordurationin an interruptible fashionModule utilitiesThemodule_utilscollection of functions is useful for working with paths related to a running module, for example to find internal data files that might be needed for normal operation.FunctionPurposeconfig_path(module_name)Returns the path to local config data directory for the moduledatadir_path(module_name)Returns the path to the/datasubdirectory of the moduledesktop_path()Returns the path to the user's Desktop directory on this systeminstallation_path(module_name)Returns the path to module's installation directorymodule_path(module_name)Returns the path to the installed moduleFunctionconfig_path(...)is useful to use in conjunction with Python'sconfigparsermodule.  It returns~/.config/modulename/on Unix-like systems.Network utilitiesThenetwork_utilsmodule provides several functions that are useful when performing network operations.FunctionPurposedownload(url, local_dest)Download a filedownload_file(url, local_dest)Download a file without raising exceptionshostname(url)Returns the hostname portion of a URLnet(...)See belownetlock(url)Returns the hostname, port number (if any), and login info (if any)network_available()ReturnsTrueif external hosts are reacheable over the networkon_localhost(url)ReturnsTrueif the address ofurlpoints to the local hostscheme(url)Returns the protocol portion of the url; e.g., "https"netThenetfunction in thenetwork_utilsmodule implements a fairly high-level network operation interface that internally handles timeouts, rate limits, polling, HTTP/2, and more. The function signature is:net(method,url,client=None,handle_rate=True,polling=False,recursing=0,**kwargs)Themethodparameter should have a value such as'get','post','head', or similar.  The function returns a tuple of (response, exception), where the first element is the response from the get or post HTTP call, and the second element is an exception object if an exception occurred.  If no exception occurred, the second element will beNone.  This allows the caller to inspect the response even in cases where exceptions are raised.If keywordclientis notNone, it's assumed to be aPython HTTPX Clientobject to use for the network call.  Settings such as timeouts should be done by the caller creating appropriately-configuredClientobjects.If keywordhandle_rateisTrue, this function will automatically pause and retry if it receives an HTTP code 429 ("too many requests") from the server.  IfFalse, it will return the exceptionCommonPy.RateLimitExceededinstead.If keywordpollingisTrue, certain statuses like 404 are ignored and the response is returned; otherwise, they are considered errors.  The behavior whenTrueis useful in situations where a URL does not exist until something is ready at the server, and the caller is repeatedly checking the URL.  It is up to the caller to implement the polling schedule and call this function (withpolling = True) as needed.This method always passes the argumentallow_redirects = Trueto the underlying Python HTTPX library network calls.downloadanddownload_fileThe functionsdownload(url, local_destination)anddownload_file(url, local_destination)download a file at the givenurl, writing it to the file specified by the parameterlocal_destination. The former version of the function will raise exceptions in case of problems; the latter version simply returnTrueorFalsedepending on the success of the download.String utilitiesFunctionPurposeantiformat(s)Quote instances of{and}insso it can be passed to format.print_boxed(msg)Print a message with a box around it using pure ASCII characters.System utilitiesFunctionPurposesystem_profile()Returns a string describing the system running this Python program.ExceptionsThe CommonPy module defines a number of exceptions that it may return.  (Most of the exceptions are potentially thrown bynet, discussed above.)ExceptionMeaningCommonPyExceptionBase class for CommonPy exceptionsArgumentErrorThe function call was given invalid or unexpected argumentsAuthenticationFailureProblem obtaining or using authentication credentialsInternalErrorUnrecoverable problem involving CommonPy itselfInterruptedThe user elected to cancel/quit the programNetworkFailureUnrecoverable problem involving netNoContentNo content found at the given locationRateLimitExceededThe service flagged reports that its rate limits have been exceededServiceFailureUnrecoverable problem involving a remote serviceGetting helpIf you find an issue, please submit it inthe GitHub issue trackerfor this repository.ContributingWe would be happy to receive your help and participation with enhancing CommonPy!  Please visit theguidelines for contributingfor some tips on getting started.If you plan on doing any development on CommonPy, you may want to install the package dependencies listed inrequirements-dev.txt, e.g., using a command such as the following. This will install dependencies necessary to runpytest.python3 -m pip install -r requirements-dev.txtLicenseSoftware produced by the Caltech Library is Copyright (C) 2020-2023, Caltech.  This software is freely distributed under a BSD/MIT type license.  Please see theLICENSEfile for more information.Authors and historyMike Hucka started this collection of utilities sometime in 2018.AcknowledgmentsThis work was funded by the California Institute of Technology Library.Thevector artworkof a toolbox, used as the icon for this repository, was created bypriyankafrom the Noun Project.  It is licensed under the Creative CommonsCC-BY 3.0license.CommonPy makes use of numerous open-source packages, without which it would have been effectively impossible to develop CommonPy with the resources we had.  I want to acknowledge this debt.  In alphabetical order, the packages are:boltons– package of miscellaneous Python utilitiesdateparser– parse dates in almost any string formatdeprecation– a library to handle deprecation of code in your Python packagesh2– HTTP/2 support library used byHTTPXhttpx– Python HTTP client library that supports HTTP/2humanize– make numbers more easily readable by humansipdb– the IPython debuggerpytest– testing framework for Pythonpytest-mock– wrapper around themockpackage for use withpytestpython_dateutil– date utilitiesPyYAML– Python YAML parserpywin32– Windows APIs for Pythonsidetrack– simple debug logging/tracing packagetldextract– module to parse domains from URLstwine– package for publishing Python packages to PyPIvalidator-collection– collection of Python functions for validating data