Package: carabiner-tools
🪨 carabinerUseful python utilities.InstallationFast and flexible reading and random access of very large filesReading tabular dataUtilities to simplify building command-line appsReservoir samplingMultikey dictionariesDecoratorsVectorized functionsReturnNoneinstead of errorDecorators with parametersColorblind paletteGrids with sensible defaults in MatplotlibFast indicator matrix x dense matrix multiplication in TensorflowIssues, problems, suggestionsDocumentationInstallationThe easy wayInstall the pre-compiled version from GitHub:$pipinstallcarabiner-toolsIf you want to use thetensorflow,pandas, ormatplotlibutilities, these must be installed separately
or together:$pipinstallcarabiner-tools[deep]# or$pipinstallcarabiner-tools[pd]# or$pipinstallcarabiner-tools[mpl]# or$pipinstallcarabiner-tools[all]From sourceClone the repository, thencdinto it. Then run:$pipinstall-e.Fast and flexible reading and random access of very large filesSubsets of lines from very large, optionally compressed, files can be read quickly
into memory. for example, we can read the first 10,000 lines of an arbitrarily large
file:>>>fromcarabiner.ioimportget_lines>>>get_lines("big-table.tsv.gz",lines=10_000)Or random access of specific lines. Hundreds of millions of lines can be
parsed per minute.>>>get_lines("big-table.tsv.gz",lines=[999999,10000000,100000001])This pattern will allow sampling a random subset:>>>fromrandomimportsample>>>fromcarabiner.ioimportcount_lines,get_lines>>>number_of_lines=count_lines("big-table.tsv.gz")>>>line_sample=sample(range(number_of_lines),k=1000)>>>get_lines("big-table.tsv.gz",lines=line_sample)Reading tabular dataWith this backend, we can read subsets of very large files more quickly
and flexibly than plainpandas.read_csv. Formats (delimiters) including Excel
are inferred from file extensions, but can also be over-ridden with theformatparameter.>>>fromcarabiner.pdimportread_table>>>read_table("big-table.tsv.gz",lines=10_000)The same fast random access is availavble as for reading lines. Hundreds of
millions of records can be looped through per minute.>>>fromrandomimportsample>>>fromcarabiner.ioimportcount_lines,get_lines>>>number_of_lines=count_lines("big-table.tsv.gz")>>>line_sample=sample(range(number_of_lines),k=1000)>>>read_table("big-table.tsv.gz",lines=line_sample)Utilities to simplify building command-line appsThe standard libraryargparseis robust but verbose when building command-line apps with several sub-commands, each with many options.carabiner.cliutilssmooths this process. Apps are built by definingCLIOptionswhich are then assigned toCLICommandsdirecting the functions to run when called, which then form part of aCLIApp.First define the options:inputs=CLIOption('inputs',type=str,default=[],nargs='*',help='')output=CLIOption('--output','-o',type=FileType('w'),default=sys.stdout,help='Output file. Default: STDOUT')formatting=CLIOption('--format','-f',type=str,default='TSV',choices=['TSV','CSV','tsv','csv'],help='Format of files. Default:%(default)s')Then the commands:test=CLICommand("test",description="Test CLI subcommand using Carabiner utilities.",options=[inputs,output,formatting],main=_main)The same options can be assigned to multiple commands if necessary.Fianlly, define the app and run it:app=CLIApp("Carabiner",version=__version__,description="Test CLI app using Carabiner utilities.",commands=[test])app.run()Reservoir samplingIf you need to sample a random subset from an iterator of unknown length by looping through only once, you can use this pure python implementation ofreservoir sampling.An important limitation is that while the population to be sampled is not necessarily in memory, the sampled population must fit in memory.Originally written inPython Bugs.Based onthis GitHub Gist.>>>fromcarabiner.randomimportsample_iter>>>fromstringimportascii_letters>>>fromitertoolsimportchain>>>fromrandomimportseed>>>seed(1)>>>sample_iter(chain.from_iterable(ascii_lettersfor_inrange(1000000)),10)['X','c','w','q','T','e','u','w','E','h']>>>seed(1)>>>sample_iter(chain.from_iterable(ascii_lettersfor_inrange(1000000)),10,shuffle_output=False)['T','h','u','X','E','e','w','q','c','w']Multikey dictionariesConveniently return the values of multiple keys from a dictionary without manually looping.>>>fromcarabiner.collectionsimportMultiKeydict>>>d=MultiKeyDict(a=1,b=2,c=3)>>>d{'a':1,'b':2,'c':3}>>>d['c']{'c':3}>>>d['a','b']{'a':1,'b':2}Decoratorscarabinerprovides several decorators to facilitate functional programming.Vectorized functionsIn scientific programming frameworks likenumpywe are used to functions which take a scalar or vector and apply to every element. It is occasionally useful to convert functions from arbitrary packages to behave in a vectorized manner on Python iterables.Scalar functions can be converted to a vectorized form easily using@vectorize.>>>@vectorize...defvector_adder(x):returnx+1...>>>list(vector_adder(range(3)))[1,2,3]>>>list(vector_adder((4,5,6)))[5,6,7]>>>vector_adder([10])11>>>vector_adder(10)11ReturnNoneinstead of errorWhen it is useful for a function to not fail, but have a testable indicator of success, you can wrap in@return_none_on_error.>>>deferror_maker(x):raiseKeyError...>>>@return_none_on_error...deferror_maker2(x):raiseKeyError...>>>@return_none_on_error(exception=ValueError)...deferror_maker3(x):raiseKeyError...>>>error_maker('a')# Causes an errorTraceback(mostrecentcalllast):File"<stdin>",line1,in<module>File"<stdin>",line1,inerror_makerKeyError>>>error_maker2('a')# Wrapped returns None>>>error_maker3('a')# Only catches ValueErrorTraceback(mostrecentcalllast):File"<stdin>",line1,in<module>File".../carabiner/decorators.py",line59,inwrapped_functionFile"<stdin>",line2,inerror_maker3KeyErrorDecorators with parametersSometimes a decorator has optional parameters to control its behavior. It's convenient to use it in the form@decoratorwhen you want the default behavior, or@decorator(*kwargs)when you want to custmize the behavior. Usually this requires some convoluted code, but this has been packed up into@decorator_with_params, to decorate your decorator definitions!>>>defdecor(f,suffix="World"):...returnlambdax:f(x+suffix)...>>>@decor...defprinter(x):...print(x)...# doesn't work, raises an error!>>>@decor(suffix="everyone")...defprinter2(x):...print(x)...Traceback(mostrecentcalllast):File"<stdin>",line1,in<module>TypeError:decor()missing1requiredpositionalargument:'f'# decorate the decorator!>>>@decorator_with_params...defdecor2(f,suffix="World"):...returnlambdax:f(x+suffix)...# Now it works!>>>@decor2(suffix="everyone")...defprinter3(x):...print(x)...>>>printer("Hello ")HelloWorld>>>printer3("Hello ")HelloeveryoneColorblind paletteHere's a qualitative palette that's colorblind friendly.>>>fromcarabinerimportcolorblind_palette>>>colorblind_palette()('#EE7733','#0077BB','#33BBEE','#EE3377','#CC3311','#009988','#BBBBBB','#000000')# subsets>>>colorblind_palette(range(2))('#EE7733','#0077BB')>>>colorblind_palette(slice(3,6))('#EE3377','#CC3311','#009988')Grids with sensible defaults in MatplotlibWhileplt.subplots()is very flexible, it requires many defaults to be defined. Instead,carabiner.mpl.grid()generates thefig, axtuple with sensible defaults of a 1x1 grid with panel size 3 and aconstrainedlayout.fromcarabiner.mplimportgridfig,ax=grid()# 1x1 gridfig,ax=grid(ncol=3)# 1x3 grid; figsize expands appropriatelyfig,ax=grid(ncol=3,nrow=2,sharex=True)#additional parameters are passed to `plt.subplots()`Fast indicator matrix x dense matrix multiplication in TensorflowIf you want to multiply an indicator matrix, i.e. a sparse matrix of zeros and ones with the same number of non-zero entries per row (as in linear models), as part of a Tensorflow model, this pattern will be faster than usingtensorflow.SparseMatrixif you convert the indicator matrix to a[n x 1]matrix providing the index of the non-zero element per row.Issues, problems, suggestionsAdd to theissue tracker.DocumentationAvailable atReadTheDocs.