Package: ubelt
Ubelt is a utility library for Python with a stdlib like feel.Elevator Pitch:Is the Python standard library good?  Yes.  Could it’s conciseness be improved?  Yes.  Ubelt aims to provide a quicker way to express things you can do in the standard library.  Progress?ub.ProgIter.  Hashing?ub.hash_data/ub.hash_file.  Caching?ub.Cacher/ub.CacheStamp.  Shell commands?ub.cmd. There are similar functions for downloading data, futures-based parallel (or serial) job execution, pretty reprs, path management, iteration, and one of my favorites: set operation enriched dictionaries:ub.udict.There are 120ish functions and classes to help make your code shorter and easier to express concisely.  The library is fast to install and import, all dependencies are optional.  As of 2023 it is 6 years old, regularly maintained, and mature.  It is well tested and has moderate usage.To learn more, the function usefulness chart is a good place to start.  This shows how often I use particular functions, and while some of the less used ones are candidates for removal, some of them still worth checking out. For a slightly slower start, read the introduction:Introduction:Ubelt is a lightweight library of robust, tested, documented, and simple functions
that extend the Python standard library. It has a flat API that all behaves
similarly on Windows, Mac, and Linux (up to some small unavoidable
differences).  Almost every function inubeltwas written with a doctest.
This provides helpful documentation and example usage as well as helping
achieve 100% test coverage (with minor exceptions on Windows).Goal: provide simple functions that accomplish common tasks not yet addressed by the python standard library.Constraints: Must be low-impact pure python; it should be easy to install and use.Method: All functions are written with docstrings and doctests to ensure that a baseline level of documentation and testing always exists (even if functions are copy/pasted into other libraries)Motto: Good utilities lift all codes.Read the docs here:http://ubelt.readthedocs.io/en/latest/These are some of the tasks that ubelt’s API enables:extended pathlib with expand, ensuredir, endswith, augment, delete (ub.Path)get paths to cross platform data/cache/config directories  (ub.Path.appdir, …)perform set operations on dictionaries (SetDict)a dictionary with extended helper methods like subdict, take, peek_value, invert, sorted_keys, sorted_vals (UDict)hash common data structures like list, dict, int, str, etc. (hash_data)hash files (hash_file)cache a block of code (Cacher, CacheStamp)time a block of code (Timer)show loop progress with less overhead than tqdm (ProgIter)download a file with optional caching and hash verification (download, grabdata)run shell commands (cmd)find a file or directory in candidate locations (find_path, find_exe)string-repr for nested data structures (urepr)color text with ANSI tags (color_text)horizontally concatenate multiline strings (hzcat)create cross platform symlinks (symlink)import a module using the path to that module (import_module_from_path)check if a particular flag or value is on the command line (argflag, argval)memoize functions (memoize, memoize_method, memoize_property)build ordered sets (oset)argmax/min/sort on lists and dictionaries (argmin, argsort,)get a histogram of items or find duplicates in a list (dict_hist, find_duplicates)group a sequence of items by some criterion (group_items)Ubelt is small. Its top-level API is defined using roughly 40 lines:fromubelt.util_argimport(argflag,argval,)fromubelt.util_cacheimport(CacheStamp,Cacher,)fromubelt.util_colorsimport(NO_COLOR,color_text,highlight_code,)fromubelt.util_constimport(NoParam,)fromubelt.util_cmdimport(cmd,)fromubelt.util_dictimport(AutoDict,AutoOrderedDict,SetDict,UDict,ddict,dict_diff,dict_hist,dict_isect,dict_subset,dict_union,dzip,find_duplicates,group_items,invert_dict,map_keys,map_vals,map_values,named_product,odict,sdict,sorted_keys,sorted_vals,sorted_values,udict,varied_values,)fromubelt.util_deprecateimport(schedule_deprecation,)fromubelt.util_downloadimport(download,grabdata,)fromubelt.util_download_managerimport(DownloadManager,)fromubelt.util_funcimport(compatible,identity,inject_method,)fromubelt.util_reprimport(ReprExtensions,urepr,)fromubelt.util_futuresimport(Executor,JobPool,)fromubelt.util_ioimport(delete,touch,)fromubelt.util_linksimport(symlink,)fromubelt.util_listimport(allsame,argmax,argmin,argsort,argunique,boolmask,chunks,compress,flatten,iter_window,iterable,peek,take,unique,unique_flags,)fromubelt.util_hashimport(hash_data,hash_file,)fromubelt.util_importimport(import_module_from_name,import_module_from_path,modname_to_modpath,modpath_to_modname,split_modpath,)fromubelt.util_indexableimport(IndexableWalker,indexable_allclose,)fromubelt.util_memoizeimport(memoize,memoize_method,memoize_property,)fromubelt.util_mixinsimport(NiceRepr,)fromubelt.util_pathimport(Path,TempDir,augpath,ensuredir,expandpath,shrinkuser,userhome,)fromubelt.util_platformimport(DARWIN,LINUX,POSIX,WIN32,find_exe,find_path,platform_cache_dir,platform_config_dir,platform_data_dir,)fromubelt.util_strimport(codeblock,hzcat,indent,paragraph,)fromubelt.util_streamimport(CaptureStdout,CaptureStream,TeeStringIO,)fromubelt.util_timeimport(Timer,timeparse,timestamp,)fromubelt.util_zipimport(split_archive,zopen,)fromubelt.orderedsetimport(OrderedSet,oset,)fromubelt.progiterimport(ProgIter,)Installation:Ubelt is distributed on pypi as a universal wheel and can be pip installed on
Python 3.6+. Installations are tested on CPython and PyPy implementations. For
Python 2.7 and 3.5, the last supported version was 0.11.1.pip install ubeltNote that our distributions on pypi are signed with GPG. The signing public key
isD297D757; this should agree with the value indev/public_gpg_key.Function UsefulnessWhen I had to hand pick a set of functions that I thought were the most useful
I chose these and provided some comment on why:importubeltasubub.Path# inherits from pathlib.Path with quality of life improvementsub.UDict# inherits from dict with keywise set operations and quality of life improvementsub.Cacher# configuration based on-disk cachineub.CacheStamp# indirect caching with corruption detectionub.hash_data# hash mutable python containers, useful with Cacher to config stringsub.cmd# combines the best of subprocess.Popen and os.systemub.download# download a file with a single command. Also see grabdata for the same thing, but caching from CacheStamp.ub.JobPool# easy multi-threading / multi-procesing / or single-threaded processingub.ProgIter# a minimal progress iterator. It's single threaded, informative, and faster than tqdm.ub.memoize# like ``functools.cache``, but uses ub.hash_data if the args are not hashable.ub.urepr# readable representations of nested data structuresBut a better way might to objectively measure the frequency of usage and built
a histogram of usefulness. I generated this histogram usingpython dev/maintain/gen_api_for_docs.py,
which roughly counts the number of times I’ve used a ubelt function in another
project. Note: this measure is biased towards older functions.Function nameUsefulnessubelt.urepr2893ubelt.Path992ubelt.ProgIter544ubelt.paragraph482ubelt.take387ubelt.codeblock358ubelt.expandpath331ubelt.cmd302ubelt.udict271ubelt.ensuredir256ubelt.odict253ubelt.iterable252ubelt.ddict238ubelt.NiceRepr221ubelt.NoParam216ubelt.map_vals215ubelt.flatten214ubelt.dzip200ubelt.oset198ubelt.peek196ubelt.argflag177ubelt.group_items171ubelt.hash_data165ubelt.grabdata131ubelt.argval125ubelt.Timer120ubelt.dict_isect113ubelt.dict_hist111ubelt.augpath106ubelt.identity106ubelt.ensure_app_cache_dir105ubelt.allsame102ubelt.memoize99ubelt.color_text98ubelt.dict_diff95ubelt.delete89ubelt.hzcat88ubelt.schedule_deprecation87ubelt.named_product85ubelt.compress85ubelt.IndexableWalker74ubelt.indent68ubelt.JobPool67ubelt.unique63ubelt.dict_union57ubelt.map_keys49ubelt.invert_dict48ubelt.iter_window46ubelt.timestamp46ubelt.argsort44ubelt.Cacher43ubelt.find_exe41ubelt.symlink41ubelt.dict_subset41ubelt.writeto40ubelt.find_duplicates39ubelt.chunks38ubelt.hash_file37ubelt.modname_to_modpath37ubelt.ensure_unicode33ubelt.memoize_property33ubelt.highlight_code33ubelt.sorted_vals32ubelt.CacheStamp30ubelt.WIN3228ubelt.import_module_from_name27ubelt.argmax27ubelt.readfrom23ubelt.import_module_from_path22ubelt.compatible17ubelt.touch17ubelt.Executor16ubelt.memoize_method16ubelt.sorted_keys14ubelt.AutoDict11ubelt.shrinkuser11ubelt.inject_method10ubelt.varied_values9ubelt.split_modpath8ubelt.modpath_to_modname8ubelt.get_app_cache_dir8ubelt.zopen7ubelt.LINUX7ubelt.CaptureStdout6ubelt.download5ubelt.timeparse5ubelt.DARWIN5ubelt.argmin5ubelt.find_path4ubelt.indexable_allclose4ubelt.boolmask3ubelt.map_values2ubelt.AutoOrderedDict2ubelt.argunique2ubelt.NO_COLOR2ubelt.UDict1ubelt.unique_flags1ubelt.userhome0ubelt.split_archive0ubelt.sorted_values0ubelt.sdict0ubelt.platform_data_dir0ubelt.platform_config_dir0ubelt.platform_cache_dir0ubelt.get_app_data_dir0ubelt.get_app_config_dir0ubelt.ensure_app_data_dir0ubelt.ensure_app_config_dir0ubelt.TempDir0ubelt.TeeStringIO0ubelt.SetDict0ubelt.ReprExtensions0ubelt.POSIX0ubelt.OrderedSet0ubelt.DownloadManager0ubelt.CaptureStream0ExamplesThe most up to date examples are the doctests.
We also have a Jupyter notebook:https://github.com/Erotemic/ubelt/blob/main/docs/notebooks/Ubelt%20Demo.ipynbHere are some examples of some features insideubeltPathsUbelt extendspathlib.Pathby adding several new (often chainable) methods.
Namely,augment,delete,expand,ensuredir,shrinkuser. It
also modifies behavior oftouchto be chainable. (New in 1.0.0)>>># Ubelt extends pathlib functionality>>>importubeltasub>>>dpath=ub.Path('~/.cache/ubelt/demo_path').expand().ensuredir()>>>fpath=dpath/'text_file.txt'>>>aug_fpath=fpath.augment(suffix='.aux',ext='.jpg').touch()>>>aug_dpath=dpath.augment('demo_path2')>>>assertaug_fpath.read_text()==''>>>fpath.write_text('text data')>>>assertaug_fpath.exists()>>>assertnotaug_fpath.delete().exists()>>>assertdpath.exists()>>>assertnotdpath.delete().exists()>>>print(f'{fpath.shrinkuser()}')>>>print(f'{dpath.shrinkuser()}')>>>print(f'{aug_fpath.shrinkuser()}')>>>print(f'{aug_dpath.shrinkuser()}')~/.cache/ubelt/demo_path/text_file.txt~/.cache/ubelt/demo_path~/.cache/ubelt/demo_path/text_file.aux.jpg~/.cache/ubelt/demo_pathdemo_path2HashingTheub.hash_dataconstructs a hash for common Python nested data
structures. Extensions to allow it to hash custom types can be registered.  By
default it handles lists, dicts, sets, slices, uuids, and numpy arrays.>>>importubeltasub>>>data=[('arg1',5),('lr',.01),('augmenters',['flip','translate'])]>>>ub.hash_data(data,hasher='sha256')0d95771ff684756d7be7895b5594b8f8484adecef03b46002f97ebeb1155fb15Support for torch tensors and pandas data frames are also included, but needs to
be explicitly enabled.  There also exists an non-public plugin architecture to
extend this function to arbitrary types. While not officially supported, it is
usable and will become better integrated in the future. Seeubelt/util_hash.pyfor details.CachingCache intermediate results from blocks of code inside a script with minimal
boilerplate or modification to the original code.For direct caching of data, use theCacherclass.  By default results will
be written to the ubelt’s appdir cache, but the exact location can be specified
viadpathor theappnamearguments.  Additionally, process dependencies
can be specified via thedependsargument, which allows for implicit cache
invalidation.  As far as I can tell, this is the most concise way (4 lines of
boilerplate) to cache a block of code with existing Python syntax (as of
2022-06-03).>>>importubeltasub>>>depends=['config',{'of':'params'},'that-uniquely-determine-the-process']>>>cacher=ub.Cacher('test_process',depends=depends,appname='myapp')>>># start fresh>>>cacher.clear()>>>for_inrange(2):>>>data=cacher.tryload()>>>ifdataisNone:>>>myvar1='result of expensive process'>>>myvar2='another result'>>>data=myvar1,myvar2>>>cacher.save(data)>>>myvar1,myvar2=dataFor indirect caching, use theCacheStampclass. This simply writes a
“stamp” file that marks that a process has completed. Additionally you can
specify criteria for when the stamp should expire. If you letCacheStampknow about the expected “product”, it will expire the stamp if that file has
changed, which can be useful in situations where caches might becomes corrupt
or need invalidation.>>>importubeltasub>>>dpath=ub.Path.appdir('ubelt/demo/cache').delete().ensuredir()>>>params={'params1':1,'param2':2}>>>expected_fpath=dpath/'file.txt'>>>stamp=ub.CacheStamp('name',dpath=dpath,depends=params,>>>hasher='sha256',product=expected_fpath,>>>expires='2101-01-01T000000Z',verbose=3)>>># Start fresh>>>stamp.clear()>>>>>>for_inrange(2):>>>ifstamp.expired():>>>expected_fpath.write_text('expensive process')>>>stamp.renew()Seehttps://ubelt.readthedocs.io/en/latest/ubelt.util_cache.htmlfor more
details aboutCacherandCacheStamp.Loop ProgressProgIteris a no-threads attached Progress meter that writes to stdout.  It
is a mostly drop-in alternative totqdm.The advantage of ``ProgIter`` is that it does not use any python threading,
and therefore can be safer with code that makes heavy use of multiprocessing.Note:ProgIteris also defined in a standalone module:pip install progiter)>>>importubeltasub>>>defis_prime(n):...returnn>=2andnotany(n%i==0foriinrange(2,n))>>>forninub.ProgIter(range(1000),verbose=2):>>># do some work>>>is_prime(n)0/1000...rate=0.00Hz,eta=?,total=0:00:00,wall=14:05EST1/1000...rate=82241.25Hz,eta=0:00:00,total=0:00:00,wall=14:05EST257/1000...rate=177204.69Hz,eta=0:00:00,total=0:00:00,wall=14:05EST642/1000...rate=94099.22Hz,eta=0:00:00,total=0:00:00,wall=14:05EST1000/1000...rate=71886.74Hz,eta=0:00:00,total=0:00:00,wall=14:05ESTCommand Line InteractionThe builtin Pythonsubprocess.Popenmodule is great, but it can be a
bit clunky at times. Theos.systemcommand is easy to use, but it
doesn’t have much flexibility. Theub.cmdfunction aims to fix this.
It is as simple to run asos.system, but it returns a dictionary
containing the return code, standard out, standard error, and thePopenobject used under the hood.This utility is designed to provide as consistent as possible behavior across
different platforms.  We aim to support Windows, Linux, and OSX.>>>importubeltasub>>>info=ub.cmd('gcc --version')>>>print(ub.urepr(info)){'command':'gcc --version','err':'','out':'gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609\nCopyright (C) 2015 Free Software Foundation, Inc.\nThis is free software; see the source for copying conditions.  There is NO\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n\n','proc':<subprocess.Popenobjectat0x7ff98b310390>,'ret':0,}Also note the use ofub.urepr(previouslyub.repr2) to nicely format the output
dictionary.Additionally, if you specifyverbose=True,ub.cmdwill
simultaneously capture the standard output and display it in real time (i.e. it
will “tee” the output).>>>importubeltasub>>>info=ub.cmd('gcc --version',verbose=True)gcc(Ubuntu5.4.0-6ubuntu1~16.04.9)5.4.020160609Copyright(C)2015FreeSoftwareFoundation,Inc.Thisisfreesoftware;seethesourceforcopyingconditions.ThereisNOwarranty;notevenforMERCHANTABILITYorFITNESSFORAPARTICULARPURPOSE.A common use case forub.cmdis parsing version numbers of programs>>>importubeltasub>>>cmake_version=ub.cmd('cmake --version')['out'].splitlines()[0].split()[-1]>>>print('cmake_version ={!r}'.format(cmake_version))cmake_version=3.11.0-rc2This allows you to easily run a command line executable as part of a
python process, see what it is doing, and then do something based on its
output, just as you would if you were interacting with the command line
itself.The idea is thatub.cmdremoves the need to think about if you need to pass
a list of args, or a string. Both will work.New in1.0.0, a third variant with different consequences for executing
shell commands. Using thesystem=Truekwarg will directly useos.systeminstead ofPopenentirely. In this mode it is not possible toteethe
output because the program is executing directly in the foreground. This is
useful for doing things like spawning a vim session and returning if the user
manages to quit vim.Downloading FilesThe functionub.downloadprovides a simple interface to download a
URL and save its data to a file.>>>importubeltasub>>>url='http://i.imgur.com/rqwaDag.png'>>>fpath=ub.download(url,verbose=0)>>>print(ub.shrinkuser(fpath))~/.cache/ubelt/rqwaDag.pngThe functionub.grabdataworks similarly toub.download, but
whereasub.downloadwill always re-download the file,ub.grabdatawill check if the file exists and only re-download it if
it needs to.>>>importubeltasub>>>url='http://i.imgur.com/rqwaDag.png'>>>fpath=ub.grabdata(url,verbose=0,hash_prefix='944389a39')>>>print(ub.shrinkuser(fpath))~/.cache/ubelt/rqwaDag.pngNew in version 0.4.0: both functions now accepts thehash_prefixkeyword
argument, which if specified will check that the hash of the file matches the
provided value. Thehasherkeyword argument can be used to change which
hashing algorithm is used (it defaults to"sha512").Dictionary Set OperationsDictionary operations that are analogous to set operations.
See each funtions documentation for more details on the behavior of the values.
Typically the last seen value is given priority.I hope Python decides to add these to the stdlib someday.ubelt.dict_unioncorresponds toset.union.ubelt.dict_isectcorresponds toset.intersection.ubelt.dict_diffcorresponds toset.difference.>>>d1={'a':1,'b':2,'c':3}>>>d2={'c':10,'e':20,'f':30}>>>d3={'e':10,'f':20,'g':30,'a':40}>>>ub.dict_union(d1,d2,d3){'a':40,'b':2,'c':10,'e':10,'f':20,'g':30}>>>ub.dict_isect(d1,d2){'c':3}>>>ub.dict_diff(d1,d2){'a':1,'b':2}New in Version 1.2.0: Ubelt now contains a dictionary subclass with set
operations that can be invoked asubelt.SetDictorub.sdict.
Note that n-ary operations are supported.>>>d1=ub.sdict({'a':1,'b':2,'c':3})>>>d2={'c':10,'e':20,'f':30}>>>d3={'e':10,'f':20,'g':30,'a':40}>>>d1|d2|d3{'a':40,'b':2,'c':10,'e':10,'f':20,'g':30}>>>d1&d2{'c':3}>>>d1-d2{'a':1,'b':2}>>>ub.sdict.intersection({'a':1,'b':2,'c':3},['b','c'],['c','e']){'c':3}Note this functionality and more is available inubelt.UDictorub.udict.Grouping ItemsGiven a list of items and corresponding ids, create a dictionary mapping each
id to a list of its corresponding items.  In other words, group a sequence of
items of typeVTand corresponding keys of typeKTgiven by a function
or corresponding list, group them into aDict[KT, List[VT]such that each
key maps to a list of the values associated with the key.  This is similar topandas.DataFrame.groupby.Group ids can be specified by a second list containing the id for
each corresponding item.>>>importubeltasub>>># Group via a corresonding list>>>item_list=['ham','jam','spam','eggs','cheese','bannana']>>>groupid_list=['protein','fruit','protein','protein','dairy','fruit']>>>dict(ub.group_items(item_list,groupid_list)){'dairy':['cheese'],'fruit':['jam','bannana'],'protein':['ham','spam','eggs']}They can also be given by a function that is executed on each item in the list>>>importubeltasub>>># Group via a function>>>item_list=['ham','jam','spam','eggs','cheese','bannana']>>>defgrouper(item):...returnitem.count('a')>>>dict(ub.group_items(item_list,grouper)){1:['ham','jam','spam'],0:['eggs','cheese'],3:['bannana']}Dictionary HistogramFind the frequency of items in a sequence.
Given a list or sequence of items, this returns a dictionary mapping each
unique value in the sequence to the number of times it appeared.
This is similar topandas.DataFrame.value_counts.>>>importubeltasub>>>item_list=[1,2,39,900,1232,900,1232,2,2,2,900]>>>ub.dict_hist(item_list){1232:2,1:1,2:4,900:3,39:1}Each item can also be given a weight>>>importubeltasub>>>item_list=[1,2,39,900,1232,900,1232,2,2,2,900]>>>weights=[1,1,0,0,0,0,0.5,0,1,1,0.3]>>>ub.dict_hist(item_list,weights=weights){1:1,2:3,39:0,900:0.3,1232:0.5}Dictionary ManipulationMap functions across dictionarys to transform the keys or values in a
dictionary.  Theubelt.map_keysfunction applies a function to each key in
a dictionary and returns this transformed copy of the dictionary. Key conflict
behavior currently raises and error, but may be configurable in the future. Theubelt.map_valsfunction is the same except the function is applied to each
value instead.  I these functions are useful enough to be ported to Python
itself.>>>importubeltasub>>>dict_={'a':[1,2,3],'bb':[],'ccc':[2,]}>>>dict_keymod=ub.map_keys(len,dict_)>>>dict_valmod=ub.map_vals(len,dict_)>>>print(dict_keymod)>>>print(dict_valmod){1:[1,2,3],2:[],3:[2]}{'a':3,'bb':0,'ccc':1}Take a subset of a dictionary. Note this is similar toub.dict_isect,
except this will raise an error if the given keys are not in the dictionary.>>>importubeltasub>>>dict_={'K':3,'dcvs_clip_max':0.2,'p':0.1}>>>subdict_=ub.dict_subset(dict_,['K','dcvs_clip_max'])>>>print(subdict_){'K':3,'dcvs_clip_max':0.2}Theubelt.takefunction works on dictionarys (and lists). It is similar toubelt.dict_subset, except that it returns just a list of the values, and
discards information about the keys. It is also possible to specify a default
value.>>>importubeltasub>>>dict_={1:'a',2:'b',3:'c'}>>>print(list(ub.take(dict_,[1,3,4,5],default=None)))['a','c',None,None]Invert the mapping defined by a dictionary. By defaultinvert_dictassumes that all dictionary values are distinct (i.e. the mapping is
one-to-one / injective).>>>importubeltasub>>>mapping={0:'a',1:'b',2:'c',3:'d'}>>>ub.invert_dict(mapping){'a':0,'b':1,'c':2,'d':3}However, by specifyingunique_vals=Falsethe inverted dictionary
builds a set of keys that were associated with each value.>>>importubeltasub>>>mapping={'a':0,'A':0,'b':1,'c':2,'C':2,'d':3}>>>ub.invert_dict(mapping,unique_vals=False){0:{'A','a'},1:{'b'},2:{'C','c'},3:{'d'}}New in Version 1.2.0: Ubelt now contains a dictionary subclassubelt.UDictwith these quality of life operations (and also inherits fromubelt.SetDict). The aliasubelt.udictcan be used for quicker access.>>>importubeltasub>>>d1=ub.udict({'a':1,'b':2,'c':3})>>>d1&{'a','c'}{'a':1,'c':3}>>>d1.map_keys(ord){97:1,98:2,99:3}>>>d1.invert(){1:'a',2:'b',3:'c'}>>>d1.subdict(['b','c','e'],default=None){'b':2,'c':3,'e':None}>>>d1.sorted_keys()OrderedDict([('a',1),('b',2),('c',3)])>>>d1.peek_key()'a'>>>d1.peek_value()1Next time you have a default configuration dictionary like and you allow the
developer to pass keyword arguments to modify these behaviors, consider using
dictionary intersection (&) to separate out only the relevant parts and
dictionary union (|) to update those relevant parts.  You can also use
dictionary differences (-) if you need to check for unused arguments.importubeltasubdefrun_multiple_algos(**kwargs):algo1_defaults={'opt1':10,'opt2':11}algo2_defaults={'src':'./here/','dst':'./there'}kwargs=ub.udict(kwargs)algo1_specified=kwargs&algo1_defaultsalgo2_specified=kwargs&algo2_defaultsalgo1_config=algo1_defaults|algo1_specifiedalgo2_config=algo2_defaults|algo2_specifiedunused_kwargs=kwargs-(algo1_defaults|algo2_defaults)print('algo1_specified ={}'.format(ub.urepr(algo1_specified,nl=1)))print('algo2_specified ={}'.format(ub.urepr(algo2_specified,nl=1)))print(f'algo1_config={algo1_config}')print(f'algo2_config={algo2_config}')print(f'The following kwargs were unused{unused_kwargs}')print(chr(10))print('-- Run with some specified --')run_multiple_algos(src='box',opt2='fox')print(chr(10))print('-- Run with extra unspecified --')run_multiple_algos(a=1,b=2)Produces:-- Run with some specified --
algo1_specified = {
    'opt2': 'fox',
}
algo2_specified = {
    'src': 'box',
}
algo1_config={'opt1': 10, 'opt2': 'fox'}
algo2_config={'src': 'box', 'dst': './there'}
The following kwargs were unused {}


-- Run with extra unspecified --
algo1_specified = {}
algo2_specified = {}
algo1_config={'opt1': 10, 'opt2': 11}
algo2_config={'src': './here/', 'dst': './there'}
The following kwargs were unused {'a': 1, 'b': 2}Find DuplicatesFind all duplicate items in a list. More specifically,ub.find_duplicatessearches for items that appear more thanktimes, and returns a mapping from each duplicate item to the positions
it appeared in.>>>importubeltasub>>>items=[0,0,1,2,3,3,0,12,2,9]>>>ub.find_duplicates(items,k=2){0:[0,1,6],2:[3,8],3:[4,5]}Cross-Platform Config and Cache DirectoriesIf you have an application which writes configuration or cache files,
the standard place to dump those files differs depending if you are on
Windows, Linux, or Mac. Ubelt offers a unified functions for determining
what these paths are.New in version 1.0.0: theub.Path.appdirclassmethod provides a way to
achieve the above with a chainable object oriented interface.Theub.Path.appdir(...,type='cache'),ub.Path.appdir(...,type='config'), andub.Path.appdir(...,type='data')functions find the correct platform-specific location for these files and
callingensuredirensures that the directories exist.The config root directory is~/AppData/Roamingon Windows,~/.configon Linux and~/Library/Application Supporton Mac. The
cache root directory is~/AppData/Localon Windows,~/.configon
Linux and~/Library/Cacheson Mac.Example usage on Linux might look like this:>>>importubeltasub>>>print(ub.Path.appdir('my_app').ensuredir().shrinkuser())# default is cache~/.cache/my_app>>>print(ub.Path.appdir('my_app',type='config').ensuredir().shrinkuser())~/.config/my_appSymlinksTheub.symlinkfunction will create a symlink similar toos.symlink. The main differences are that 1) it will not error if
the symlink exists and already points to the correct location. 2) it
works* on Windows (*hard links and junctions are used if real symlinks
are not available)>>>importubeltasub>>>dpath=ub.Path('ubelt','demo_symlink')>>>real_path=dpath/'real_file.txt'>>>link_path=dpath/'link_file.txt'>>>real_path.write_text('foo')>>>ub.symlink(real_path,link_path)AutoDict - AutovivificationWhile thecollections.defaultdictis nice, it is sometimes more
convenient to have an infinitely nested dictionary of dictionaries.>>>importubeltasub>>>auto=ub.AutoDict()>>>print('auto ={!r}'.format(auto))auto={}>>>auto[0][10][100]=None>>>print('auto ={!r}'.format(auto))auto={0:{10:{100:None}}}>>>auto[0][1]='hello'>>>print('auto ={!r}'.format(auto))auto={0:{1:'hello',10:{100:None}}}String-based importsUbelt contains functions to import modules dynamically without using the
pythonimportstatement. Whileimportlibexists, theubeltimplementation is simpler to user and does not have the disadvantage of
breakingpytest.Noteubeltsimply provides an interface to this functionality, the
core implementation is inxdoctest(over as of version0.7.0,
the code is statically copied into an autogenerated file such thatubeltdoes not actually depend onxdoctestduring runtime).>>>importubeltasub>>>try:>>># This is where I keep ubelt on my machine, so it is not expected to work elsewhere.>>>module=ub.import_module_from_path(ub.expandpath('~/code/ubelt/ubelt'))>>>print('module ={!r}'.format(module))>>>exceptOSError:>>>pass>>>>>>module=ub.import_module_from_name('ubelt')>>>print('module ={!r}'.format(module))>>>#>>>try:>>>module=ub.import_module_from_name('does-not-exist')>>>raiseAssertionError>>>exceptModuleNotFoundError:>>>pass>>>#>>>modpath=ub.Path(ub.util_import.__file__)>>>print(ub.modpath_to_modname(modpath))>>>modname=ub.util_import.__name__>>>assertub.Path(ub.modname_to_modpath(modname)).resolve()==modpath.resolve()module=<module'ubelt'from'/home/joncrall/code/ubelt/ubelt/__init__.py'>>>>module=ub.import_module_from_name('ubelt')>>>print('module ={!r}'.format(module))module=<module'ubelt'from'/home/joncrall/code/ubelt/ubelt/__init__.py'>Related to this functionality are the functionsub.modpath_to_modnameandub.modname_to_modpath, whichstaticallytransform (i.e. no code in the target modules is imported
or executed) between module names (e.g.ubelt.util_import) and
module paths (e.g.~/.local/conda/envs/cenv3/lib/python3.5/site-packages/ubelt/util_import.py).>>>importubeltasub>>>modpath=ub.util_import.__file__>>>print(ub.modpath_to_modname(modpath))ubelt.util_import>>>modname=ub.util_import.__name__>>>assertub.modname_to_modpath(modname)==modpathHorizontal String ConcatenationSometimes its just prettier to horizontally concatenate two blocks of
text.>>>importubeltasub>>>B=ub.urepr([[1,2],[3,4]],nl=1,cbr=True,trailsep=False)>>>C=ub.urepr([[5,6],[7,8]],nl=1,cbr=True,trailsep=False)>>>print(ub.hzcat(['A = ',B,' * ',C]))A=[[1,2],*[[5,6],[3,4]][7,8]]TimingQuickly time a single line.>>>importmath>>>importubeltasub>>>timer=ub.Timer('Timer demo!',verbose=1)>>>withtimer:>>>math.factorial(100000)tic('Timer demo!')...toc('Timer demo!')=0.1453sExternal toolsSome of the tools inubeltalso exist as standalone modules. I haven’t
decided if its best to statically copy them into ubelt or require on pypi to
satisfy the dependency. There are some tools that are not used by default
unless you explicitly allow for them.Code that is currently statically included (vendored):ProgIter -https://github.com/Erotemic/progiterOrderedSet -https://github.com/LuminosoInsight/ordered-setCode that is completely optional, and only used in specific cases:Numpy -ub.ureprwill format a numpy array nicely by defaultxxhash - this can be specified as a hasher toub.hash_dataPygments - used by theutil_colormodule.dateutil - used by theutil_timemodule.Similar ToolsUBelt is one of many Python utility libraries. A selection of similar libraries
are listed here.Libraries that contain a broad scope of utilities:Boltons:https://github.com/mahmoud/boltonsToolz:https://github.com/pytoolz/toolzCyToolz:https://github.com/pytoolz/cytoolz/UnStdLib:https://github.com/shazow/unstdlib.pyLibraries that contain a specific scope of utilities:More-Itertools: iteration tools:https://pypi.org/project/more-itertools/Funcy: functional tools:https://github.com/Suor/funcyRich: pretty CLI displays -https://github.com/willmcgugan/richtempora: time related tools -https://github.com/jaraco/temporaLibraries that contain one specific data structure or utility:Benedict: dictionary tools -https://pypi.org/project/python-benedict/tqdm: progress bars -https://pypi.org/project/tqdm/pooch: data downloading -https://pypi.org/project/pooch/timerit: snippet timing for benchmarks -https://github.com/Erotemic/timeritJaraco (i.e. Jason R. Coombs) has an extensive library of utilities:jaraco.classes -https://github.com/jaraco/jaraco.classesjaraco.collections -https://github.com/jaraco/jaraco.collectionsjaraco.context -https://github.com/jaraco/jaraco.contextjaraco.crypto -https://github.com/jaraco/jaraco.cryptojaraco.functools -https://github.com/jaraco/jaraco.functoolsjaraco.geo -https://github.com/jaraco/jaraco.geojaraco.imaging -https://github.com/jaraco/jaraco.imagingjaraco.itertools -https://github.com/jaraco/jaraco.itertoolsjaraco.logging -https://github.com/jaraco/jaraco.loggingjaraco.media -https://github.com/jaraco/jaraco.mediajaraco.path -https://github.com/jaraco/jaraco.pathjaraco.text -https://github.com/jaraco/jaraco.textjaraco.util -https://github.com/jaraco/jaraco.utiljaraco.windows -https://github.com/jaraco/jaraco.windowsand many others not listed here. See:https://github.com/jaraco?tab=repositories&q=jaraco.Ubelt is included in the the [bestof-python list](https://github.com/ml-tooling/best-of-python),
which contains many other tools that you should check out.History:Ubelt is a migration of the most useful parts ofutool(https://github.com/Erotemic/utool) into a standalone module
with minimal dependencies.Theutoollibrary contains a number of useful utility functions, but it
also contained non-useful functions, as well as the kitchen sink. A number of
the functions were too specific or not well documented. Theubeltis a port
of the simplest and most useful parts ofutool.Note that there are other cool things inutoolthat are not inubelt.
Notably, the doctest harness ultimately becamexdoctest.
Code introspection and dynamic analysis tools were ported toxinspect.
The more IPython-y tools were ported toxdev.
Parts of it made their way intoscriptconfig.
The init-file generation was moved tomkinit.
Some vim and system-y things can be found invimtk.Development on ubelt started 2017-01-30 and development of utool mostly stopped
on utool was stopped later that year, but received patches until about 2020.
Ubelt achieved 1.0.0 and removed support for Python 2.7 and 3.5 on 2022-01-07.Notes.PRs are welcome.Also check out my other projects which are powered by ubelt:xinspecthttps://github.com/Erotemic/xinspectxdevhttps://github.com/Erotemic/xdevvimtkhttps://github.com/Erotemic/vimtkgraphidhttps://github.com/Erotemic/graphidibeishttps://github.com/Erotemic/ibeiskwarrayhttps://github.com/Kitware/kwarraykwimagehttps://github.com/Kitware/kwimagekwcocohttps://github.com/Kitware/kwcocoAnd my projects related to ubelt:ProgIterhttps://github.com/Erotemic/progiterTimerithttps://github.com/Erotemic/timeritmkinithttps://github.com/Erotemic/mkinitxdoctesthttps://github.com/Erotemic/xdoctest