Package: big
Copyright 2022-2023 by Larry Hastingsbigis a Python package of small functions and classes
that aren't big enough to get a package of their own.
It's zillions of useful little bits of
Python code I always want to have handy.For years, I've copied-and-pasted all my little
helper functions between projects--we've all done it.
But now I've finally taken the time to consolidate all those
useful little functions into onebigpackage--no more
copy-and-paste, I just install one package and I'm ready to go.
And, since it's a public package, you can use 'em too!Not only that, but I've taken my time and re-thought and
retooled a lot of this code.  All the difficult-to-use,
overspecialized, cheap hacks I've lived with for years have
been upgraded with elegant, intuitive APIs and dazzling
functionality.bigis chock full of the sort of little functions and classes
we've all hacked together a million times--only with all the
API gotchas fixed, and thoroughly tested with 100% coverage.
It's the code youwouldhave written... if only you had the time.
It's a real pleasure to use!bigrequires Python 3.6 or newer.  Its only dependency
ispython-dateutil, and that's optional.Think big!Using bigTo usebig, just install thebigpackage (and its dependencies)
from PyPI using your favorite Python package manager.Oncebigis installed, you can simply import it.  However, the
top-levelbigpackage doesn't contain anything but a version number.
Internallybigis broken up into submodules, aggregated together
loosely by problem domain, and you can selectively import just the
functions you want.  For example, if you only want to use the text functions,
just import thetextsubmodule:importbig.textIf you'd prefer to import everything all at once, simply import thebig.allmodule.  This one module imports all the other modules,
and imports all their symbols too.  So, one convenient way to work
withbigis this:importbig.allasbigThat will make every symbol defined inbigaccessible from thebigobject. For example, if you want to usemultisplit,you can access it with justbig.multisplit.You can also usebig.allwithimport *:frombig.allimport*but that's up to you.bigis licensed using theMIT license.You're free to use it and even ship it in your own programs,
as long as you leave my copyright notice on the source code.The best of bigAlthoughbigiscrammed fullof fabulous code, a few of
its subsystems rise above the rest.  If you're curious whatbigmight do for you, here are the five things inbigI'm proudest of:Themulti-family of string functionsbig.stateBound inner classesEnhancedTopologicalSorterlinesand lines modifier functionsAnd here are five little functions/classes I use all the time:gently_titleLogpushdre_partitiontouchIndexFunctions, classes, values, and modulesaccessor(attribute='state', state_manager='state_manager')ascii_linebreaksascii_linebreaks_without_crlfascii_whitespaceascii_whitespace_without_crlfbig.allbig.boundinnerclassbig.builtinbig.filebig.graphbig.heapbig.itertoolsbig.logbig.schedulerbig.statebig.textbig.timeBoundInnerClassbytes_linebreaksbytes_linebreaks_without_crlfbytes_whitespacebytes_whitespace_without_crlfCycleError()datetime_ensure_timezone(d, timezone)datetime_set_timezone(d, timezone)default_clock()Delimiter(open, close, *, backslash=False, nested=True)dispatch(state_manager='state_manager', *, prefix='', suffix='')encode_strings(o, *, encoding='ascii')Event(scheduler, event, time, priority, sequence)Event.cancel()fgrep(path, text, *, encoding=None, enumerate=False, case_insensitive=False)file_mtime(path)file_mtime_ns(path)file_size(path)gently_title(s, *, apostrophes=None, double_quotes=None)get_float(o, default=_sentinel)get_int(o, default=_sentinel)get_int_or_float(o, default=_sentinel)grep(path, pattern, *, encoding=None, enumerate=False, flags=0)Heap(i=None)Heap.append(o)Heap.clear()Heap.copy()Heap.extend(i)Heap.remove(o)Heap.popleft()Heap.append_and_popleft(o)Heap.popleft_and_append(o)Heap.queueint_to_words(i, *, flowery=True, ordinal=False)linebreakslinebreaks_without_crlflines(s, separators=None, *, line_number=1, column_number=1, tab_width=8, **kwargs)lines_convert_tabs_to_spaces(li)lines_filter_comment_lines(li, comment_separators)lines_containing(li, s, *, invert=False)lines_grep(li, pattern, *, invert=False, flags=0)lines_rstrip(li)lines_sort(li, *, reverse=False)lines_strip(li)lines_strip_comments(li, comment_separators, *, quotes=('"', "'"), backslash='\\', rstrip=True, triple_quotes=True)lines_strip_indent(li)Log(clock=None)Log.enter(subsystem)Log.exit()Log.print(*, print=None, title="[event log]", headings=True, indent=2, seconds_width=2, fractional_width=9)Log.reset()merge_columns(*columns, column_separator=" ", overflow_response=OverflowResponse.RAISE, overflow_before=0, overflow_after=0)multipartition(s, separators, count=1, *, reverse=False, separate=True)multisplit(s, separators, *, keep=False, maxsplit=-1, reverse=False, separate=False, strip=False)multistrip(s, separators, left=True, right=True)normalize_whitespace(s, separators=None, replacement=None)parse_delimiters(s, delimiters=None)parse_timestamp_3339Z(s, *, timezone=None)pure_virtual()PushbackIterator(iterable=None)PushbackIterator.next(default=None)PushbackIterator.push(o)pushd(directory)re_partition(text, pattern, count=1, *, flags=0, reverse=False)re_rpartition(text, pattern, count=1, *, flags=0)Regulator()Regulator.now()Regulator.sleep(t)Regulator.wake()reversed_re_finditer(pattern, string, flags=0)safe_mkdir(path)safe_unlink(path)Scheduler(regulator=default_regulator)Scheduler.schedule(o, time, *, absolute=False, priority=DEFAULT_PRIORITY)Scheduler.cancel(event)Scheduler.queueScheduler.non_blocking()SingleThreadedRegulator()split_quoted_strings(s, quotes=('"', "'"), *, triple_quotes=True, backslash='\\')split_text_with_code(s, *, tab_width=8, allow_code=True, code_indent=4, convert_tabs_to_spaces=True)State()StateManager(state, *, on_enter='on_enter', on_exit='on_exit', state_class=None)str_linebreaksstr_linebreaks_without_crlfstr_whitespacestr_whitespace_without_crlftimestamp_3339Z(t=None, want_microseconds=None)timestamp_human(t=None, want_microseconds=None)ThreadSafeRegulator()TopologicalSorter(graph=None)TopologicalSorter.copy()TopologicalSorter.cycle()TopologicalSorter.print()TopologicalSorter.remove(node)TopologicalSorter.reset()TopologicalSorter.ViewTopologicalSorter.view()TopologicalSorter.View.close()TopologicalSorter.View.copy()TopologicalSorter.View.done(*nodes)TopologicalSorter.View.print(print=print)TopologicalSorter.View.ready()TopologicalSorter.View.reset()touch(path)TransitionErrortranslate_filename_to_exfat(s)translate_filename_to_unix(s)try_float(o)try_int(o)UnboundInnerClassunicode_linebreaksunicode_linebreaks_without_crlfunicode_whitespaceunicode_whitespace_without_crlfwhitespacewhitespace_without_crlfwrap_words(words, margin=79, *, two_spaces=True)Topic deep-divesThemulti-family of string functionsWhitespace and line-breaking characters in Python and biglinesand lines modifier functionsWord wrapping and formattingBound inner classesEnhancedTopologicalSorterAPI Reference, By Modulebig.allThis submodule doesn't define any of its own symbols.  Instead, it
imports every other submodule inbig, and usesimport *to
import every symbol from every other submodule, too.  Every
public symbol inbigis available inbig.all.big.boundinnerclassClass decorators that implement bound inner classes.  See theBound inner classesdeep-dive for more information.BoundInnerClass(cls)Class decorator for an inner class.  When accessing the inner class
through an instance of the outer class, "binds" the inner class to
the instance.  This changes the signature of the inner class's__init__fromdef__init__(self,*args,**kwargs):`todef__init__(self,outer,*args,**kwargs):whereouteris the instance of the outer class.Compare this to functions:If you put afunctioninside a class,
and access it through an instanceIof that class,
thefunctionbecomes amethod.When you call
themethod,Iis automatically passed in as the
first argument.If you put aclassinside a class, and access
and access it through an instance of that class,
theclassbecomes abound inner class.When
you call thebound inner class,Iis automatically
passed in as the second argument to__init__,
afterself.Note that this has an implication for all subclasses.
If classBis decorated withBoundInnerClass,
and classSis a subclass ofB, such thatissubclass(S,B),
classSmustbe decorated with eitherBoundInnerClassorUnboundInnerClass.UnboundInnerClass(cls)Class decorator for an inner class that prevents binding
the inner class to an instance of the outer class.If classBis decorated withBoundInnerClass,
and classSis a subclass ofB, such thatissubclass(S,B)returnsTrue,
classSmustbe decorated with eitherBoundInnerClassorUnboundInnerClass.big.builtinFunctions for working with builtins.  (Namedbuiltinto avoid a
name collision with thebuiltinsmodule.)In general, the idea with these functions is a principle I first
read about in eitherCode CompleteorWriting Solid Code:Don't associate with losers.The intent here is, try to design APIs where it's impossible to call them
the wrong way.  Restrict the inputs to your functions to values you can
always handle, and you won't ever have to return an error.The functions in this sub-module are designed to always work.  None of
them should ever raise an exception--no matterwhatnonsense you pass in.
(But don't take that as a challenge!)get_float(o, default=_sentinel)Returnsfloat(o), unless that conversion fails,
in which case returns the default value.  If
you don't pass in an explicit default value,
the default value iso.get_int(o, default=_sentinel)Returnsint(o), unless that conversion fails,
in which case returns the default value.  If
you don't pass in an explicit default value,
the default value iso.get_int_or_float(o, default=_sentinel)Convertsointo a number, preferring an int to a float.Ifois already an int or float, returnsounchanged.  Otherwise,
triesint(o).  If that conversion succeeds, returns the result.
Otherwise, triesfloat(o).  If that conversion succeeds, returns
the result.  Otherwise returns the default value.  If you don't
pass in an explicit default value, the default value iso.pure_virtual()A decorator for class methods.  When you have a method in
a base class that's "pure virtual"--that must not be called,
but must be overridden in child classes--decorate it with@pure_virtual().  Calling that method will throw aNotImplementedError.Note that the body of any function decorated with@pure_virtual()is ignored.  By convention the body
of these methods should contain only a single ellipsis,
literally like this:classBaseClass:@big.pure_virtual()defon_reset(self):...try_float(o)ReturnsTrueifocan be converted into afloat,
andFalseif it can't.try_int(o)ReturnsTrueifocan be converted into anint,
andFalseif it can't.big.fileFunctions for working with files, directories, and I/O.fgrep(path, text, *, encoding=None, enumerate=False, case_insensitive=False)Find the lines of a file that match some text, like the UNIXfgreputility
program.pathshould be an object representing a path to an existing file, one of:a string,a bytes object, orapathlib.Pathobject.textshould be either string or bytes.encodingis used as the file encoding when opening the file.Iftextis a str, the file is opened in text mode.Iftextis a bytes object, the file is opened in binary mode.encodingmust beNonewhen the file is opened in binary mode.Ifcase_insensitiveis true, perform the search in a case-insensitive
manner.Returns a list of lines in the file containingtext.  The lines are either
strings or bytes objects, depending on the type ofpattern.  The lines
have their newlines stripped but preserve all other whitespace.Ifenumerateis true, returns a list of tuples of (line_number, line).
The first line of the file is line number 1.For simplicity of implementation, the entire file is read in to memory
at one time.  Ifcase_insensitiveis true,fgrepalso makes a lowercased
copy.file_mtime(path)Returns the modification time ofpath, in seconds since the epoch.
Note that seconds is a float, indicating the sub-second with some
precision.file_mtime_ns(path)Returns the modification time ofpath, in nanoseconds since the epoch.file_size(path)Returns the size of the file atpath, as an integer representing the
number of bytes.grep(path, pattern, *, encoding=None, enumerate=False, flags=0)Look for matches to a regular expression pattern in the lines of a file,
similarly to the UNIXgreputility program.pathshould be an object representing a path to an existing file, one of:a string,a bytes object, orapathlib.Pathobject.patternshould be an object containing a regular expression, one of:a string,a bytes object, oranre.Pattern, initialized with eitherstrorbytes.encodingis used as the file encoding when opening the file.Ifpatternuses astr, the file is opened in text mode.
Ifpatternuses a bytes object, the file is opened in binary mode.encodingmust beNonewhen the file is opened in binary mode.flagsis passed in as theflagsargument tore.compileifpatternis a string or bytes.  (It's ignored ifpatternis anre.Patternobject.)Returns a list of lines in the file matching the pattern.  The lines
are either strings or bytes objects, depending on the type oftext.
The lines have their newlines stripped but preserve all other whitespace.Ifenumerateis true, returns a list of tuples of(line_number, line).
The first line of the file is line number 1.For simplicity of implementation, the entire file is read in to memory
at one time.Tip: to perform a case-insensitive pattern match, pass in there.IGNORECASEflag into flags for this function (if pattern is a string
or bytes) or when creating your regular expression object (if pattern is
anre.Patternobject.(In older versions of Python,re.Patternwas a private type calledre._pattern_type.)pushd(directory)A context manager that temporarily changes the directory.
Example:withbig.pushd('x'):passThis would change into the'x'subdirectory before
executing the nested block, then change back to
the original directory after the nested block.You can change directories in the nested block;
this won't affect pushd restoring the original current
working directory upon exiting the nested block.You can safely nestwith pushdblocks.safe_mkdir(path)Ensures that a directory exists atpath.
If this function returns and doesn't raise,
it guarantees that a directory exists atpath.If a directory already exists atpath,safe_mkdirdoes nothing.If a file exists atpath,safe_mkdirunlinkspaththen creates the directory.If the parent directory doesn't exist,safe_mkdircreates that directory,
then createspath.This function can still fail:pathcould be on a read-only filesystem.You might lack the permissions to createpath.You could ask to create the directoryx/yandxis a file (not a directory).safe_unlink(path)Unlinkspath, ifpathexists and is a file.touch(path)Ensures thatpathexists, and its modification time is the current time.Ifpathdoes not exist, creates an empty file.Ifpathexists, updates its modification time to the current time.translate_filename_to_exfat(s)Ensures that all characters in s are legal for a FAT filesystem.Returns a copy ofswhere every character not allowed in a FAT
filesystem filename has been replaced with a character (or characters)
that are permitted.translate_filename_to_unix(s)Ensures that all characters in s are legal for a UNIX filesystem.Returns a copy ofswhere every character not allowed in a UNIX
filesystem filename has been replaced with a character (or characters)
that are permitted.big.graphA drop-in replacement for Python'sgraphlib.TopologicalSorterwith an enhanced API.  This version ofTopologicalSorterallows modifying the
graph at any time, and supports multiple simultaneousviews,allowing
iteration over the graph more than once.See theEnhancedTopologicalSorterdeep-dive for more information.CycleError()Exception thrown byTopologicalSorterwhen it detects a cycle.TopologicalSorter(graph=None)An object representing a directed graph of nodes.  See Python'sgraphlib.TopologicalSorterfor concepts and the basic API.New methods onTopologicalSorter:TopologicalSorter.copy()Returns a shallow copy of the graph.  The copy also duplicates
the state ofget_readyanddone.TopologicalSorter.cycle()Checks the graph for cycles.  If no cycles exist, returns None.
If at least one cycle exists, returns a tuple containing nodes
that constitute a cycle.TopologicalSorter.print(print=print)Prints the internal state of the graph.  Used for debugging.printis the function used for printing;
it should behave identically to the builtinprintfunction.TopologicalSorter.remove(node)Removesnodefrom the graph.If any nodePdepends on a nodeN, andNis removed,
this dependency is also removed, butPis not
removed from the graph.Note that, whileremove()works, it's slow. (It's O(N).)TopologicalSorteris optimized for fast adds and fast views.TopologicalSorter.reset()Resetsget_readyanddoneto their initial state.TopologicalSorter.view()Returns a newViewobject on this graph.TopologicalSorter.ViewA view on aTopologicalSortergraph object.
Allows iterating over the nodes of the graph
in dependency order.Methods on aViewobject:TopologicalSorter.View.__bool__()ReturnsTrueif more work can be done in the
view--if there are nodes waiting to be yielded byget_ready, or waiting to be returned bydone.Aliased toTopologicalSorter.is_activefor compatibility
with graphlib.TopologicalSorter.View.close()Closes the view.  A closed view can no longer be used.TopologicalSorter.View.copy()Returns a shallow copy of the view, duplicating its current state.TopologicalSorter.View.done(*nodes)Marks nodes returned byreadyas "done",
possibly allowing additional nodes to be available
fromready.TopologicalSorter.View.print(print=print)Prints the internal state of the view, and its graph.
Used for debugging.printis the function used for printing;
it should behave identically to the builtinprintfunction.TopologicalSorter.View.ready()Returns a tuple of "ready" nodes--nodes with no
predecessors, or nodes whose predecessors have all
been marked "done".Aliased toTopologicalSorter.get_readyfor
compatibility withgraphlib.TopologicalSorter.View.reset()Resets the view to its initial state,
forgetting all "ready" and "done" state.big.heapFunctions for working with heap objects.
Well, just one heap object really.Heap(i=None)An object-oriented wrapper around theheapqlibrary, designed to be
easy to use--and easy to remember how to use.  Theheapqlibrary
implements abinary heap,
a data structure used for sorting;
you add objects to the heap, and you can then remove
objects in sorted order.  Heaps are useful because they have are efficient
both in space and in time; they're also inflexible, in that iterating over
the sorted items is destructive.TheHeapAPI inbigmimics thelistandcollections.dequeobjects;
this way, all you need to remember is "it works kinda like alistobject".
Youappendnew items to the heap, thenpopleftthem off in sorted order.By defaultHeapcreates an empty heap.  If you pass in an iterableito the constructor, this is equivalent to calling theextend(i)on the
freshly-constructedHeap.In addition to the below methods,Heapobjects support iteration,len, theinoperator, and use as a boolean expression.  You can
also index or slice into aHeapobject, which behaves as if the
heap is a list of objects in sorted order.  Getting the first item
(Heap[0], akapeek) is cheap, the other operations can get very
expensive.Methods on aHeapobject:Heap.append(o)Adds objectoto the heap.Heap.clear()Removes all objects from the heap, resetting it to empty.Heap.copy()Returns a shallow copy of the heap.
Only duplicates the heap data structures itself;
does not duplicate the objectsinthe heap.Heap.extend(i)Adds all the objects from the iterableito the heap.Heap.remove(o)If objectois in the heap, removes it.  Ifois not
in the heap, raisesValueError.Heap.popleft()If the heap is not empty, returns the first item in the
heap in sorted order.  If the heap is empty, raisesIndexError.Heap.append_and_popleft(o)Equivalent to callingHeap.append(o)immediately followed
byHeap.popleft().  Ifois smaller than any other object
in the heap at the time it's added, this will returno.Heap.popleft_and_append(o)Equivalent to callingHeap.popleft()immediately followed
byHeap.append(o).  This method willneverreturno,
unlessowas already in the heap before the method was called.Heap.queueNot a method, a property.  Returns a copy of the contents
of the heap, in sorted order.big.itertoolsFunctions and classes for working with iteration.
Only one entry so far.PushbackIterator(iterable=None)Wraps any iterator, allowing you to push items back on the iterator.
This allows you to "peek" at the next item (or items); you can get the
next item, examine it, and then push it back.  If any objects have
been pushed onto the iterator, they are yielded first, before attempting
to yield from the wrapped iterator.The constructor accepts one argument, aniterable, with a default ofNone.
IfiterableisNone, thePushbackIteratoris created in an exhausted state.When the wrappediterableis exhausted (or if you passed inNoneto the constructor) you can still call thepushmethod to add items,
at which point thePushBackIteratorcan be iterated over again.In addition to the following methods,PushbackIteratorsupports
the iterator protocol and testing for truth.  APushbackIteratoris true if iterating over it will yield at least one value.PushbackIterator.next(default=None)Equivalent tonext(PushbackIterator), but won't raiseStopIteration.
If the iterator is exhausted, returns thedefaultargument.PushbackIterator.push(o)Pushes a value into the iterator's internal stack.
When aPushbackIteratoris iterated over, and there are
any pushed values, the top value on the stack will be popped
and yielded.PushbackIteratoronly yields from the
iterator it wraps when this internal stack is empty.big.logA simple and lightweight logging class, useful for performance analysis.
Not intended as a full-fledged logging facility like Python'sloggingmodule.default_clock()The default clock function used by theLogclass.
This function returns elapsed time in nanoseconds,
expressed as an integer.In Python 3.7+, this istime.monotonic_ns;
in Python 3.6 this is
a custom function that callstime.perf_counter,
then converts that time to an integer number of nanoseconds.Log(*, clock=None)A simple and lightweight logging class, useful for performance analysis.
Not intended as a full-fledged logging facility like Python'sloggingmodule.Allows nesting, which is literally just a presentation thing.Theclocknamed parameter specifies the function theLogobject
should call to get the time.  This function should return anint,
representing elapsed time in nanoseconds.To use: first, create yourLogobject.log=Log()Then log events by calling yourLogobject, passing in
a string describing the event.log('text')Enter a nested subsystem containing events withlog.enter:log.enter('subsystem')Then later exit that subsystem withlog.exit:log.exit()And finally print the log:log.print()You can also iterate over the log events usingiter(log).
This yields 4-tuples:(start_time, elapsed_time, event, depth)start_timeandelapsed_timeare times, expressed as
an integer number of nanoseconds.  The first event
is atstart_time0, and all subsequent start times are
relative to that time.eventis the event string you
passed in tolog()(or"<subsystem> start"or"<subsystem> end").depthis an integer indicating how many subsystems
the event is nested in; larger numbers indicate deeper nesting.Log.enter(subsystem)Notifies the log that you've entered a subsystem.
Thesubsystemparameter should be a string describing the subsystem.This is really just a presentation
thing; all subsequent logged entries will be indented
until you make the correspondinglog.exit()call.You may nest subsystems as deeply as you like.Log.exit()Exits a logged subsystem.  SeeLog.enter.Log.print(*, print=None, title="[event log]", headings=True, indent=2, seconds_width=2, fractional_width=9)Prints the log.Keyword-only parameters:printspecifies the print function to use, default isbuiltins.print.titlespecifies the title to print at the beginning.
Default is"[event log]".  To suppress, pass inNone.headingsis a boolean; ifTrue(the default),
prints column headings for the log.indentis the number of spaces to indent in front of log entries,
and also how many spaces to indent each time we enter a subsystem.seconds_widthis how wide to make the seconds column, default 2.fractional_widthis how wide to make the fractional column, default 9.Log.reset()Resets the log to its initial state.big.schedulerA replacement for Python'ssched.schedulerobject,
adding full threading support and a modern Python interface.Python'ssched.schedulerobject was a clever idea for the
time.  It abstracted away the concept of time from its interface,
allowing it to be adapted to new schemes of measuring time--including
mock time used for testing.  Very nice!But unfortunately,sched.schedulerwas designed in 1991--long
before multithreading was common, years before threading support
was added to Python.  Sadly its API isn't flexible enough to
correctly handle some scenarios:If one thread has calledsched.scheduler.run,
and the next scheduled event will occur at timeT,
and a second thread schedules a new event which
occurs at a time <T,sched.scheduler.runwon't
return any events to the first thread until timeT.If one thread has calledsched.scheduler.run,
and the next scheduled event will occur at timeT,
and a second thread cancels all events,sched.scheduler.runwon't exit until timeT.Also,sched.scheduleris thirty years behind the times in
Python API design--its design predates many common modern
Python conventions.  Its events are callbacks, which it
calls directly.Schedulerfixes this: its events are
objects, and you iterate over theSchedulerobject to receive
events as they become due.Scheduleralso benefits from thirty years of improvements
tosched.scheduler.  In particular,bigreimplements the
relevant parts of thesched.schedulertest suite, to ensure thatSchedulernever repeats the historical problems discovered
over the lifetime ofsched.scheduler.Event(scheduler, event, time, priority, sequence)An object representing a scheduled event in aScheduler.
You shouldn't need to create them manually;Eventobjects
are created automatically when you add events to aScheduler.Supports one method:Event.cancel()Cancels this event.  If this event has already been canceled,
raisesValueError.Regulator()An abstract base class forSchedulerregulators.A "regulator" handles all the details about time
for aScheduler.Schedulerobjects don't actually
understand time; it's all abstracted away by theRegulator.You can implement your ownRegulatorand use it
withScheduler.  YourRegulatorsubclass needs to
implement a minimum of three methods:now,sleep, andwake.  It must also provide an
attribute called 'lock'.  The lock must implement
the context manager protocol, and should ensure thread
safety for theRegulator.  (Schedulerwill only
request theRegulator's lock if it's not already
holding it.  Put another way, theRegulatordoesn't
need to be a "reentrant" or "recursive" lock.)Normally aRegulatorrepresents time using
a floating-point number, representing a fractional
number of seconds since some epoch.  But this
isn't strictly necessary.  Any Python object that
fulfills these requirements will work:The time class must implement__le__,__eq__,__add__,
and__sub__, and these operations must be consistent in the
same way they are for number objects.Ifaandbare instances of the time class,
anda.__le__(b)is true, thenamust either be
an earlier time, or a smaller interval of time.The time class must also implement rich comparison
with numbers (integers and floats), and0must
represent both the earliest time and a zero-length
interval of time.Regulator.now()Returns the current time in local units.
Must be monotonically increasing; for any
two calls to now during the course of the
program, the later call mustneverhave a lower value than the earlier call.ASchedulerwill only call this method while
holding this regulator's lock.Regulator.sleep(t)Sleeps for some amount of time, in local units.
Must support an interval of0, which should
represent not sleeping.  (Though it's preferable
that an interval of0yields the rest of the
current thread's remaining time slice back to
the operating system.)Ifwakeis called on thisRegulatorobject while a
different thread has called this function to sleep,sleepmust abandon the rest of the sleep interval
and return immediately.ASchedulerwill only call this method whilenotholding this regulator's lock.Regulator.wake()Aborts all current calls tosleepon thisRegulator, across all threads.ASchedulerwill only call this method while
holding this regulator's lock.Scheduler(regulator=default_regulator)Implements a scheduler.  The only argument is the
"regulator" object to use; the regulator abstracts away all
time-related details for the scheduler.  By defaultScheduleruses an instance ofSingleThreadedRegulator,
which is not thread-safe.(If you need the scheduler to be thread-safe, pass in an
instance of a thread-safeRegulatorclass likeThreadSafeRegulator.)In addition to the below methods,Schedulerobjects support
being evaluated in a boolean context (they are true if they
contain any events), and they support being iterated over.
Iterating over aSchedulerobject blocks until the next
event comes due, at which point theScheduleryields that
event.  An emptySchedulerthat is iterated over raisesStopIteration.  You can reuseSchedulerobjects, iterating
over them until empty, then adding more objects and iterating
over them again.Scheduler.schedule(o, time, *, absolute=False, priority=DEFAULT_PRIORITY)Schedules an objectoto be yielded as an event by thisscheduleobject
at some time in the future.By default thetimevalue is a relative time value,
and is added to the current time; using atimevalue of 0
should schedule this event to be yielded immediately.Ifabsoluteis true,timeis regarded as an absolute time value.If multiple events are scheduled for the same time, they will
be yielded by order ofpriority.  Lowever values ofpriorityrepresent higher priorities.  The default value
isScheduler.DEFAULT_PRIORITY, which is 100.  If two events
are scheduled for the same time, and have the same priority,Schedulerwill yield the events in the order they were added.Returns anEventobject, which can be used to cancel the event.Scheduler.cancel(event)Cancels a scheduled event.eventmust be an object
returned by thisSchedulerobject.  Ifeventis not
currently scheduled in thisSchedulerobject,
raisesValueError.Scheduler.queueA list of the currently scheduledEventobjects,
in the order they will be yielded.Scheduler.non_blocking()Returns an iterator for the events in theSchedulerthat only yields the events that
are currently due.  Never blocks; if the next
event is not due yet, raisesStopIteration.SingleThreadedRegulator()An implementation ofRegulatordesigned for
use in single-threaded programs.  It doesn't support
multiple threads, and in particular is not thread-safe.
But it's much higher performance
than thread-safeRegulatorimplementations.ThisRegulatorisn't guaranteed to be safe
for use while in a signal-handler callback.ThreadSafeRegulator()A thread-safe implementation ofRegulatordesigned for use in multithreaded programs.ThisRegulatorisn't guaranteed to be safe
for use while in a signal-handler callback.big.stateLibrary code for working with simple state machines.There are lots of popular Python libraries for implementing
state machines.  But they all seem to be designed forlarge-scalestate machines.  These libraries are
sophisticated and data-driven, with expansive APIs.
And, as a rule, they require the state to be
a passive object (e.g. anEnum), and require you to explicitly
describe every possible state transition.This approach is great for massive, super-complex state
machines--you need a sophisticated library to manage such
complex state machines.  This approach also permits these
libraries to offer clever features like automatically generating
diagrams of your state machine.However, most of the time, this level of sophistication is
unnecessary.  There are lots of use cases for small scale,simplestate machines, where theis complex data-driven approach
only gets in the way.  I prefer writing my state machines
with active objects--where states are implemented as classes,
events are implemented as method calls on those classes,
and you transition to a new state by simply overwriting astateattribute with a new instance of one of these classes.big.stateis designed to make it easy to write this style of
state machine.  It has
a deliberately minimal, simple interface--the constructor for
the mainStateManagerclass only has four parameters,
and it only exposes three attributes.  The module also has
two decorators to make your life easier.  And that's it!
With that small API surface area, you can effortlessly write
large scale state machines.(But you can also write tiny data-driven state machines too.
Althoughbig.statemakes state machines with active states
easy to write, it's agnostic about how you actually implement
your state machine.big.statemakes it easy to write any
kind of state machine you like!)big.stateprovides features like:method calls that get called when entering and exiting a state,observer objects that get called each time you transition
to a new state, andsafety mechanisms to catch bugs and prevent design mistakes.Recommended best practicesThe main class inbig.stateisStateManager.  This class
maintains the current "state" of your state machine, and
manages transitions to new states.  It takes one required
parameter, which is the initial state.Here are my recommended best practices for working withStateManagerfor medium-sized and larger state machines:Your state machine should be implemented as aclass.You should storeStateManageras an an attribute of that class,
preferably calledstate_manager.  (Your state machine
should have a "has-a" relationship withStateManager,
not an "is-a" relationship where it inherits fromStateManager.)Decorating your state machine class with theaccessordecorator will save you a lot of boilerplate.
If your state machine is stored ino, decorating withaccessorlets you can access the current state usingo.stateinstead ofo.state_manager.state.Your states should be implemented asclasses.You should have a base class for your state classes,
containing whatever functionality they have in common.You're encouraged to define these state classesinsideyour state machine class, and useBoundInnerClassso they automatically get references to the state machine
they're a part of.Events should bemethod callsmade on your state machine object.As a rule, events should be dispatched from the state machine
to a method call on the current state with the same name.If all the code to handle a particular event lives in the
states, use thedispatchdecorator to handle dispatching the call.  This will write
a new method for you, that calls the equivalent method on
the current state, passing in all the arguments it received.Your state base class should have a method for every
event, decorated with`pure_virtual'.Example codeHere's a simple example demonstrating all this functionality.
It's a state machine with two states,OnandOff, and
one event methodtoggle.  Callingtoggletransitions
the state machine from theOffstate to theOnstate,
and vice-versa.frombig.allimportaccessor,BoundInnerClass,dispatch,pure_virtual,StateManager@accessor()classStateMachine:def__init__(self):self.state_manager=StateManager(self.Off())@dispatch()deftoggle(self):...@BoundInnerClassclassState:def__init__(self,state_machine):self.state_machine=state_machinedef__repr__(self):returnf"<{type(self).__name__}>"@pure_virtual()deftoggle(self):...@BoundInnerClassclassOff(State.cls):defon_enter(self):print("off!")deftoggle(self):sm=self.state_machinesm.state=sm.On()# sm.state is the accessor@BoundInnerClassclassOn(State.cls):defon_enter(self):print("on!")deftoggle(self):sm=self.state_machinesm.state=sm.Off()sm=StateMachine()print(sm.state)for_inrange(3):sm.toggle()print(sm.state)For another, more complete example of working withStateManager,
see thetest_vending_machinetest code intests/test_state.pyin thebigsource tree.accessor(attribute='state', state_manager='state_manager')Class decorator.  Adds a convenient state accessor attribute to your class.When you have a state machine class containing aStateManagerobject, it can be wordy and inconvenient to access the state through
the state machine attribute:classStateMachine:def__init__(self):self.state_manager=StateManager(self.InitialState)...sm=StateMachine()# vvvvvvvvvvvvvvvvvvvv that's a lot!sm.state_manager.state=NextState()Theaccessorclass decorator creates a property for you, a
short-cut that directly accesses thestateattribute of
your state manager.  Just decorate with@accessor():@accessor()classStateMachine:def__init__(self):self.state_manager=StateManager(self.InitialState)...sm=StateMachine()# vvvvvv that's a lot shorter!sm.state=NextState()Thestateattribute evaluates to the same value:sm.state==sm.state_manager.stateAnd setting it sets the state on yourStateManagerinstance.
These two statements now do the same thing:sm.state_manager.state=new_statesm.state=new_stateBy default, this decorator assumes yourStateManagerinstance
is in thestate_managerattribute, and you want to name the new
accessor attributestate.  You can override these defaults;
the decorator's first parameter,attribute, should be the string used
for the new accessor attribute, and the second parameter,state_manager, should be the name of the attribute where yourStateManagerinstance is stored.For example, if your state manager is stored in an attribute calledsm,
and you want the short-cut to be calledst, you'd decorate your
class with@accessor(attribute='st',state_manager='sm')dispatch(state_manager='state_manager', *, prefix='', suffix='')Decorator for state machine event methods,
dispatching the event from the state machine object
to its current state.dispatchhelps with the following scenario:You have your own state machine class which contains
aStateManagerobject.You want your state machine class to have methods
representing events.Rather than handle those events in your state machine
object itself, you want to dispatch them to the current
state.Simply create a method in your state machine class
with the correct name and parameters but a no-op body,
and decorate it with@dispatch.  Thedispatchdecorator will rewrite your method so it calls the
equivalent method on the current state, passing through
all the arguments.For example, instead of writing this:classStateMachine:def__init__(self):self.state_manager=StateManager(self.InitialState)defon_sunrise(self,time,*,verbose=False):returnself.state_manager.state.on_sunrise(time,verbose=verbose)you can literally write this, which does the same thing:classStateMachine:def__init__(self):self.state_manager=StateManager(self.InitialState)@dispatch()defon_sunrise(self,time,*,verbose=False):...Here, theon_sunrisefunction you wrote is actually thrown away.
(That's why the body is simply one"..."statement.)  Your function
is replaced with a function that gets thestate_managerattribute
fromself, then gets thestateattribute from thatStateManagerinstance, then calls a method with the same name as the decorated
function, passing in using*argsand**kwargs.Note that, as a stylistic convention, you're encouraged to literally
use a single ellipsis as the body of these functions, like in the
example above.  This is a visual cue to readers that the body of the
function doesn't matter.Thestate_managerargument to the decorator should be the name of
the attribute where theStateManagerinstance is stored inself.
The default is'state_manager', but you can specify a different
string if you've stored yourStateManagerin another attribute.
For example, if your state manager is in the attributesmedley,
you'd decorate with:@dispatch('smedley')Theprefixandsuffixarguments are strings added to the
beginning and end of the method call we call on the current state.
For example, if you want the method you call to have an active verb
form (e.g.reset), but you want it to directly call an event
handler that starts withon_by convention (e.g.on_reset),
you could do this:@dispatch(prefix='on_')defreset(self):...This is equivalent to:defreset(self):returnself.state_manager.state.on_reset()If you have more than one event method, instead of decorating
every event method with the same copy-and-pasteddispatchcall, it's better to calldispatchonce, cache the
function it returns, and decorate with that.  Like so:my_dispatch=dispatch('smedley',prefix='on_')@my_dispatchdefreset(self):...@my_dispatchdefsunrise(self):...State()Base class for state machine state implementation classes.
Use of this base class is optional; states can be instances
of any type excepttypes.NoneType.StateManager(state, *, on_enter='on_enter', on_exit='on_exit', state_class=None)Simple, Pythonic state machine manager.Has three public attributes:stateThe current state.  You transition from
one state to another by assigning to this attribute.nextThe state theStateManageris transitioning to,
if it's currently in the process of transitioning to a
new state.  If theStateManagerisn't currently
transitioning to a new state, itsnextattribute isNone.
While the manager is currently transitioning to a new
state, it's illegal to start a second transition.  (In
other words: you can't assign tostatewhilenextis notNone.)observersA list of callables that get called
during every state transition.  It's initially empty;
you should add and remove observers to the list
as needed.The callables will be called with one positional argument,
the state manager object.Since observers are called during the state transition,
they aren't permitted to initiate state transitions.You're permitted to modify the list of observers
at any time.  If you modify the list of observers
during an observer call,StateManagerwill finish
the current observer callbacks using a copy of the
old list.The constructor takes the following parameters:stateThe initial state.  It can be any valid state object;
by default, any Python value can be a state exceptNone.
(But also see thestate_classparameter below.)on_enteron_enterrepresents a method call on states called when
entering that state.  The value itself is a string used
to look up an attribute on state objects; by defaulton_enteris the string'on_enter', but it can be any legal
Python identifier string, or any false value.Ifon_enteris a valid identifier string, and thisStateManagerobject transitions to a state objectO, andOhas an attribute
with this name,StateManagerwill call that attribute (with no
arguments) immediately after transitioning to that state.
Passing in a false value foron_enterdisables this behavior.on_enteris called immediately after the transition is complete,
which means you're expressly permitted to make a state transition
inside anon_entercall.If defined,on_exitwill be called on
the initial state object, from inside theStateManagerconstructor.on_exiton_exitis similar toon_enter, except the attribute is
called when transitioningawayfrom a state object.
Its default value is `'on_exit'``.on_exitis calledduringthe state transition, which means you're expressly
forbidden from making a state transition inside anon_exitcall.state_classstate_classis used to enforce that thisStateManageronly ever transitions to valid state objects.
It should be eitherNoneor a class.  If it's a class,
theStateManagerobject will require every value assigned
to itsstateattribute to be an instance of that class.
If it'sNone, states can be any object (exceptNone).State transitionsTo transition to a new state, simply assign to the 'state'
attribute.Ifstate_classisNone, you may useanyvalue as a state
exceptNone.It's illegal to assign tostatewhile currently
transitioning to a new state.  (Or, in other words,
at any timeself.nextis notNone.)If the current state object has anon_exitmethod,
it will be called (with zero arguments)duringthe the transition to the next state.  This means it's
illegal to initiate a state transition inside anon_exitcall.If you assign an object tostatethat has anon_enterattribute, that method will be called (with zero
arguments) immediatelyafterwe have transitioned to that
state.  This means it's permitted to initiate a state
transition inside anon_entercall.It's illegal to attempt to transition to the current
state.  Ifstate_manager.stateis alreadyfoo,state_manager.state = foowill raise an exception.Sequence of events during a state transitionIf you have anStateManagerinstance calledstate_manager,
and you transition it tonew_state:state_manager.state=new_stateStateManagerwill execute the following sequence of events:Setstate_manager.nexttonew_state.At of this momentstate_manageris "transitioning"
to the new state.Ifstate_manager.statehas anon_exitattribute,
callstate_manager.state.on_exit().For every objectoin thestate_manager.observerlist,
callo(self).Setstate_manager.nexttoNone.Setstate_manager.statetonew_state.As of this moment, the transition is complete, andstate_manageris now "in" the new state.Ifstate_manager.statehas anon_enterattribute,
callstate_manager.state.on_enter().TransitionError()Exception raised when attempting to execute an illegal state transition.There are only two types of illegal state transitions:An attempted state transition while we're in the process
of transitioning to another state.  In other words,
ifstate_manageris yourStateManagerobject, you can't
setstate_manager.statewhenstate_manager.nextis notNone.An attempt to transition to the current state.
This is illegal:state_manager=StateManager()state_manager.state=foostate_manager.state=foo# <-- this statement raises TransitionErrorNote that transitioning to a different butidenticalobject
is expressly permitted.big.textFunctions for working with text strings.  There are
several families of functions inside thetextmodule;
for a higher-level view of those families, read the
following deep-dives:Themulti-family of string functionslinesand lines modifier functionsWord wrapping and formattingAll the functions inbig.textwill work with eitherstrorbytesobjects, except the threeWord wrapping and formattingfunctions.  When working withbytes,
by default the functions will only work with ASCII
characters.Support for bytes and strThebigtext functions all support bothstrandbytes.
The functions all automatically detect whether you passed instrorbytesusing an
intentionally simple and predictable process, as follows:At the start of each function, it'll test its first "string"
argument to see if it's abytesobject.is_bytes=isinstance(<argument>,bytes)IfisinstancereturnsTrue, the function assumes all arguments arebytesobjects.  Otherwise the function assumes all arguments
arestrobjects.As a rule, no further further testing, casting, or catching exceptions
is done.Functions that take multiple string-like parameters require all
such arguments to be the same type.
These functions will check that all such arguments
are of the same type.Subclasses ofstrandbyteswill also work; anywhere you
should pass in astr, you can also pass in a subclass ofstr, and likewise forbytes.ascii_linebreaksA tuple ofstrobjects, representing every line-breaking whitespace
whitespace character defined by ASCII.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.
If you don't want to include this string, useascii_linebreaks_without_crlfinstead.
See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.ascii_linebreaks_without_crlfEquivalent toascii_linebreakswithout'\r\n'.ascii_whitespaceA tuple ofstrobjects, representing every whitespace
whitespace character defined by ASCII.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.
If you don't want to include this string, useascii_whitespace_without_crlfinstead.
See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.ascii_whitespace_without_crlfEquivalent toascii_whitespacewithout'\r\n'.bytes_linebreaksA tuple ofbytesobjects, representing every line-breaking whitespace
whitespace character recognized by the Pythonbytesobject.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also containsb'\r\n'.
If you don't want to include this string, usebytes_linebreaks_without_crlfinstead.
See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.bytes_linebreaks_without_crlfEquivalent tobytes_linebreakswithout'\r\n'.bytes_whitespaceA tuple ofbytesobjects, representing every line-breaking whitespace
whitespace character recognized by the Pythonbytesobject.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also containsb'\r\n'.
If you don't want to include this string, usebytes_whitespace_without_crlfinstead.
See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.bytes_whitespace_without_crlfEquivalent tobytes_whitespacewithout'\r\n'.Delimiter(open, close, *, backslash=False, nested=True)Class representing a delimiter forparse_delimiters.openis the opening delimiter character, can bestrorbytes, must be length 1.closeis the closing delimiter character, must be the same type asopen, and length 1.backslashis a boolean: when inside this delimiter, can you escape delimiters
with a backslash?  (You usually can inside single or double quotes.)nestedis a boolean: must other delimiters nest in this delimiter?
(Delimiters don't usually need to be nested inside single and double quotes.)encode_strings(o, *, encoding='ascii')Accepts a container objectocontainingstrobjects; returns an equivalent object
with the strings encoded tobytes.omust be eitherdict,list, ortuple,
or a subclass of one of those.Encodes everystrinside using the encoding
specified in theencodingparameter, default
is'ascii'.  Handles nested containers.Ifois astr, raisesTypeError.
Ifois a container and contains an object that
isn't astr,dict,list, ortuple, raisesTypeError.gently_title(s, *, apostrophes=None, double_quotes=None)Uppercases the first character of every word ins,
leaving the other letters alone.sshould bestrorbytes.(For the purposes of this algorithm, words are
any contiguous run of non-whitespace characters.)This function will also capitalize the letter after an apostrophe
if the apostrophe:is immediately after whitespace, oris immediately after a left parenthesis character ('('), oris the first letter of the string, oris immediately after a letter O or D, when that O or Dis after whitespace, oris the first letter of the string.In this last case, the O or D will also be capitalized.Finally, this function will capitalize the letter
after a quote mark if the quote mark:is after whitespace, oris the first letter of a string.(A run of consecutive apostrophes and/or
quote marks is considered one quote mark for
the purposes of capitalization.)All these rules meangently_titlecorrectly handles
internally quoted strings:He Said 'No I Did Not'and contractions that start with an apostrophe:'Twas The Night Before Christmasas well as certain Irish, French, and Italian names:Peter O'Toole
    Lord D'ArcyIf specified,apostrophesshould be astrorbytesobject containing characters that
should be considered apostrophes.  Ifapostrophesis false, andsisbytes,apostrophesis set to
a bytes object containing the only ASCII apostrophe character:'Ifapostrophesis false and s isstr,apostrophesis set to a string containing these Unicode apostrophe code points:'‘’‚‛Note that neither of these strings contains the "back-tick"
character:`This is a diacritical used for modifying letters, and isn't
used as an apostrophe.If specified,double_quotesshould be astrorbytesobject containing characters that
should be considered double-quote characters.
Ifdouble_quotesis false, andsisbytes,double_quotesis set to a bytes object containing
the only ASCII double-quote character:"Ifdouble_quotesis false andsisstr, double_quotes
is set to a string containing these Unicode double-quote code points:"“”„‟«»‹›int_to_words(i, *, flowery=True, ordinal=False)Converts an integer into the equivalent English string.int_to_words(2)->"two"int_to_words(35)->"thirty-five"If the keyword-only parameterfloweryis true (the default),
you also get commas and the wordandwhere you'd expect them.
(Whenfloweryis true,int_to_words(i)produces identical
output toinflect.engine().number_to_words(i), except for
negative numbers:inflectstarts negative numbers with
"minus",bigstarts them with "negative".)If the keyword-only parameterordinalis true,
the string produced describes thatordinalnumber
(instead of thatcardinalnumber).  Ordinal numbers
describe position, e.g. where a competitor placed in
a competition.  In other words,int_to_words(1)returns the string'one', butint_to_words(1, ordinal=True)returns the
string'first'.Numbers >=10**66(one thousand vigintillion)
are only converted usingstr(i).  Sorry!linebreaksA tuple ofstrobjects, representing every line-breaking
whitespace character recognized by the Pythonstrobject.
Identical tostr_linebreaks.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.  See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.linebreaks_without_crlfEquivalent tolinebreakswithout'\r\n'.lines(s, separators=None, *, line_number=1, column_number=1, tab_width=8, **kwargs)A "lines iterator" object.  Splits s into lines, and iterates yielding those lines.scan bestr,bytes, or any iterable ofstrorbytes.Ifsis neitherstrnorbytes,smust be an iterable;linesyields successive elements ofsas lines.  All objects
yielded by this iterable should be homogeneous, eitherstrorbytes.Ifsisstrorbytes, andseparatorsisNone,lineswill splitsat line boundaries and yield those lines, including
empty lines.  Ifseparatorsis notNone, it must be an iterable
of strings of the same type ass;lineswill splitsusingmultisplit.When iterated over, yields 2-tuples:(info, line)infois aLineInfoobject, which contains three fields by
default:line- the original line, never modifiedline_number- the line number of this line, starting at theline_numberpassed in and adding 1 for each successive linecolumn_number- the column this line starts on,
starting at thecolumn_numberpassed in, and adjusted when
characters are removed from the beginning oflineThetab_widthkeyword-only parameter is an integer, representing
how many spaces wide a tab character should be.  It isn't used bylinesitself; instead, it's stored internally, and may be used by
lines modifier functions (e.g.lines_convert_tabs_to_spaces,lines_strip_indent).  Similarly, all keyword arguments passed
in viakwargsare stored internally and can be accessed by
user-defined lines modifier functions.For more information, see the deep-dive onlinesand lines modifier functions.LineInfo(line, line_number, column_number, **kwargs)The second object yielded by alinesiterator, containing metadata about the line.
You can add your own fields by passing them in
via**kwargs; you can also add new attributes
or modify existing attributes as needed from
inside a "lines modifier" function.For more information, see the deep-dive onlinesand lines modifier functions.lines_convert_tabs_to_spaces(li)A lines modifier function.  Converts tabs to spaces for the lines
of a "lines iterator", using thetab_widthpassed in tolines.For more information, see the deep-dive onlinesand lines modifier functions.lines_filter_comment_lines(li, comment_separators)A lines modifier function.  Filters out comment lines from the
lines of a "lines iterator".  Comment lines are lines whose
first non-whitespace characters appear in the iterable ofcomment_separatorsstrings passed in.What's the difference betweenlines_strip_commentsandlines_filter_comment_lines?lines_filter_comment_linesonly recognizes lines thatstartwith a comment separator
(ignoring leading whitespace).  Also, it filters out those
lines completely, rather than modifying the line.lines_strip_commentshandles comment characters anywhere in the line, although it can ignore
comments inside quoted strings.  It truncates the line but still always
yields the line.For more information, see the deep-dive onlinesand lines modifier functions.lines_containing(li, s, *, invert=False)A lines modifier function.  Only yields lines
that contains.  (Filters out lines that
don't contains.)Ifinvertis true, returns the opposite--filters
out lines that contains.For more information, see the deep-dive onlinesand lines modifier functions.lines_grep(li, pattern, *, invert=False, flags=0)A lines modifier function.  Only yields lines
that match the regular expressionpattern.
(Filters out lines that don't matchpattern.)patterncan bestr,bytes, or anre.Patternobject.
Ifpatternis not anre.Patternobject, it's compiled
withre.compile(pattern, flags=flags).Ifinvertis true, returns the opposite--filters
out lines that matchpattern.For more information, see the deep-dive onlinesand lines modifier functions.(In older versions of Python,re.Patternwas a private type calledre._pattern_type.)lines_rstrip(li)A lines modifier function.  Strips trailing whitespace from the
lines of a "lines iterator".For more information, see the deep-dive onlinesand lines modifier functions.lines_sort(li, *, reverse=False)A lines modifier function.  Sorts all input lines before
yielding them.Lines are sorted lexicographically, from lowest to highest.
Ifreverseis true, lines are sorted from highest to lowest.For more information, see the deep-dive onlinesand lines modifier functions.lines_strip(li)A lines modifier function.  Strips leading and trailing whitespace
from the lines of a "lines iterator".Iflines_stripremoves leading whitespace from a line,
it updatesLineInfo.column_numberwith the new starting
column number, and also adds a field to theLinesInfoobject:leading- the leading whitespace string that was removedFor more information, see the deep-dive onlinesand lines modifier functions.lines_strip_comments(li, comment_separators, *, quotes=('"', "'"), backslash='\\', rstrip=True, triple_quotes=True)A lines modifier function.  Strips comments from the lines
of a "lines iterator".  Comments are substrings that indicate
the rest of the line should be ignored;lines_strip_commentstruncates the line at the beginning of the leftmost comment
separator.Ifrstripis true (the default),lines_strip_commentscalls
therstrip()method onlineafter it truncates the line.Ifquotesis true, it must be an iterable of quote characters.
(Each quote charactermustbe a single character.)lines_strip_commentswill parse the line and ignore comment
characters inside quoted strings.  Ifquotesis false,
quote characters are ignored andline_strip_commentswill truncate
anywhere in the line.backslashandtriple_quotesare passed in tosplit_quoted_string, which is used internally to detect the quoted
strings in the line.Sets a new field on the associatedLineInfoobject for every line:comment- the comment stripped from the line, if any.
If no comment was found,commentwill be an empty string.What's the difference betweenlines_strip_commentsandlines_filter_comment_lines?lines_filter_comment_linesonly recognizes lines thatstartwith a comment separator
(ignoring leading whitespace).  Also, it filters out those lines
completely, rather than modifying the line.lines_strip_commentshandles comment characters anywhere in the line, although it can ignore
comments inside quoted strings.  It truncates the line but still always
yields the line.For more information, see the deep-dive onlinesand lines modifier functions.lines_strip_indent(li)A lines modifier function.  Automatically measures and strips indents.Sets two new fields on the associatedLineInfoobject for every line:indent- an integer indicating how many indents it's observedleading- the leading whitespace string that was removedAlso updates LineInfo.column_number as needed.Uses an intentionally simple algorithm.
Only understands tab and space characters as indent characters.
Internally detabs to spaces first for consistency, using thetab_widthpassed in to lines.You can only dedent out to a previous indent.
RaisesIndentationErrorif there's an illegal dedent.For more information, see the deep-dive onlinesand lines modifier functions.merge_columns(*columns, column_separator=" ", overflow_response=OverflowResponse.RAISE, overflow_before=0, overflow_after=0)Merge an arbitrary number of separate text strings into
columns.  Returns a single formatted string.columnsshould be an iterable of "column tuples".
Each column tuple should contain three items:(text,min_width,max_width)textshould be a single string, eitherstrorbytes,
with newline characters separating lines.min_widthandmax_widthare the minimum and maximum permissible widths
for that column, not including the column separator (if any).Note that this function does not text-wrap the text of the
columns.  The text in the columns should already be broken
into lines and separated by newline characters.  (Lines in
that are longer than that column tuple'smax_widthare
handled with theoverflow_strategy, below.)column_separatoris printed between every column.overflow_strategytells merge_columns how to handle a column
with one or more lines that are wider than that column'smax_width.
The supported values are:OverflowStrategy.RAISE: Raise an OverflowError.  The default.OverflowStrategy.INTRUDE_ALL: Intrude into all subsequent columns
on all lines where the overflowed column is wider than itsmax_width.OverflowStrategy.DELAY_ALL: Delay all columns after the overflowed
column, not beginning any until after the last overflowed line
in the overflowed column.Whenoverflow_strategyisINTRUDE_ALLorDELAY_ALL, and
eitheroverflow_beforeoroverflow_afteris nonzero, these
specify the number of extra lines before or after
the overflowed lines in a column.For more information, see the deep-dive onWord wrapping and formatting.multipartition(s, separators, count=1, *, reverse=False, separate=True)Likestr.partition, but supports partitioning based on multiple
separator strings, and can partition more than once.scan be eitherstrorbytes.separatorsshould be an iterable of objects of the same type ass.By default, if any of the strings inseparatorsare found ins,
returns a tuple of three strings: the portion ofsleading up to
the earliest separator, the separator, and the portion ofsafter
that separator.  Example:multipartition('aXbYz', ('X', 'Y')) => ('a', 'X', 'bYz')If none of the separators are found in the string, returns
a tuple containingsunchanged followed by two empty strings.multipartitionisgreedy:if two or more separators appear at
the leftmost location ins,multipartitionpartitions using
the longest matching separator.  For example:multipartition('wxabcyz', ('a', 'abc')) => `('wx', 'abc', 'yz')`Passing in an explicitcountlets you control how many timesmultipartitionpartitions the string.multipartitionwill always
return a tuple containing(2*count)+1elements.
Passing in acountof 0 will always return a tuple containings.Ifseparateis true, multiple adjacent separator strings behave
like one separator.  Example:big.text.multipartition('aXYbYXc', ('X', 'Y',), count=2, separate=False) => ('a', 'XY', 'b', 'YX', 'c')
big.text.multipartition('aXYbYXc', ('X', 'Y',), count=2, separate=True ) => ('a', 'X', '', 'Y', 'bYXc')Ifreverseis true, multipartition behaves likestr.rpartition.
It partitions starting on the right, scanning backwards through s
looking for separators.For more information, see the deep-dive onThemulti-family of string functions.multisplit(s, separators=None, *, keep=False, maxsplit=-1, reverse=False, separate=False, strip=False)Splits strings likestr.split, but with multiple separators and options.scan bestrorbytes.separatorsshould either beNone(the default),
or an iterable ofstrorbytes, matchings.IfseparatorsisNoneandsisstr,multisplitwill usebig.whitespaceasseparators.
IfseparatorsisNoneandsisbytes,multisplitwill usebig.ascii_whitespaceasseparators.Returns an iterator yielding the strings split froms.  Ifkeepis true (orALTERNATING), andstripis false, joining these strings
together will recreates.multisplitisgreedy:if two or more separators start at the same
location ins,multisplitsplits using the longest matching separator.
For example:big.multisplit('wxabcyz',('a','abc'))yields'wx'then'yz'.keepindicates whether or not multisplit should preserve the separator
strings in the strings it yields.  It supports four values:false (the default)Discard the separators.true (apart fromALTERNATINGandAS_PAIRS)Append the separators to the end of the split strings.
You can recreate the original string by using"".jointo join the strings yielded bymultisplit.ALTERNATINGYield alternating strings in the output: strings consisting
of separators, alternating with strings consisting of
non-separators.  If "separate" is true, separator strings
will contain exactly one separator, and non-separator strings
may be empty; if "separate" is false, separator strings will
contain one or more separators, and non-separator strings
will never be empty, unless "s" was empty.
You can recreate the original string by using"".jointo join the strings yielded bymultisplit.AS_PAIRSYield 2-tuples containing a non-separator string and its
subsequent separator string.  Either string may be empty;
the separator string in the last 2-tuple will always be
empty, and if "s" ends with a separator string,bothstrings in the final 2-tuple will be empty.separateindicates whether multisplit should consider adjacent
separator strings insas one separator or as multiple separators
each separated by a zero-length string.  It supports two values:false (the default)Group separators together.  Multiple adjacent separators
behave as if they're one big separator.trueDon't group separators together.  Each separator should
split the string individually, even if there are no
characters between two separators.  (multisplitwill
behave as if there's a zero-character-wide string between
adjacent separators.)stripindicates whether multisplit should strip separators from
the beginning and/or end ofs.  It supports five values:false (the default)Don't strip separators from the beginning or end of "s".true (apart from LEFT, RIGHT, and PROGRESSIVE)Strip separators from the beginning and end of "s"
(similarly to `str.strip`).LEFTStrip separators only from the beginning of "s"
(similarly to `str.lstrip`).RIGHTStrip separators only from the end of "s"
(similarly to `str.rstrip`).PROGRESSIVEStrip from the beginning and end of "s", unless "maxsplit"
is nonzero and the entire string is not split.  If
splitting stops due to "maxsplit" before the entire string
is split, and "reverse" is false, don't strip the end of
the string. If splitting stops due to "maxsplit" before
the entire string is split, and "reverse" is true, don't
strip the beginning of the string.  (This is how `str.strip`
and `str.rstrip` behave when you pass in `sep=None`.)maxsplitshould be either an integer orNone.  Ifmaxsplitis an
integer greater than -1, multisplit will splittextno more thanmaxsplittimes.reversechanges wheremultisplitstarts splitting the string, and
what direction it moves through the string when parsing.false (the default)Start splitting from the beginning of the string
and parse moving right (towards the end).trueStart splitting from the end of the string and
parse moving left (towards the beginning).Splitting starting from the end of the string and parsing moving
left has two effects.  First, ifmaxsplitis a number
greater than 0, the splits will start at the end of the string
rather than the beginning.  Second, if there are overlapping
instances of separators in the string,multisplitwill prefer
therightmostseparator rather than theleftmost.Consider this
example, wherereverseis false:multisplit("A x x Z", (" x ",), keep=big.ALTERNATING) => "A", " x ", "x Z"If you pass in a true value forreverse,multisplitwill prefer
the rightmost overlapping separator:multisplit("A x x Z", (" x ",), keep=big.ALTERNATING, reverse=True) => "A x", " x ", "Z"For more information, see the deep-dive onThemulti-family of string functions.multistrip(s, separators, left=True, right=True)Likestr.strip, but supports stripping multiple substrings froms.Strips from the stringsall leading and trailing instances of strings
found inseparators.sshould bestrorbytes.separatorsshould be an iterable of eitherstrorbytesobjects matching the type ofs.Ifleftis a true value, strips all leading separators
froms.Ifrightis a true value, strips all trailing separators
froms.Processing always stops at the first character that
doesn't match one of the separators.Returns a copy ofswith the leading and/or trailing
separators stripped.  (Ifleftandrightare both
false, returnssunchanged.)For more information, see the deep-dive onThemulti-family of string functions.normalize_whitespace(s, separators=None, replacement=None)Returnss, but with every run of consecutive
separator characters turned into a replacement string.
By default turns all runs of consecutive whitespace
characters into a single space character.smay bestrorbytes.separatorsshould be an iterable of eitherstrorbytesobjects, matchings.replacementshould be either astrorbytesobject,
also matchings, orNone(the default).
IfreplacementisNone,normalize_whitespacewill use
a replacement string consisting of a single space character.Leading or trailing runs of separator characters will
be replaced with the replacement string, e.g.:normalize_whitespace("   a    b   c") == " a b c"parse_delimiters(s, delimiters=None)Parses a string containing nesting delimiters.
Raises an exception if mismatched delimiters are detected.smay bestrorbytes.delimitersmay be eitherNoneor an iterable containing
eitherDelimiterobjects or objects matchings(strorbytes).
Entries in thedelimitersiterable which arestrorbytesshould be exactly two characters long; these will be used
as theopenandclosearguments for a newDelimiterobject.IfdelimitersisNone,parse_delimitersuses a default
value matching these pairs of delimiters:() [] {} "" ''The quote mark delimiters enable backslash quoting and disable nesting.Yields 3-tuples containing strings:(text, open, close)wheretextis the text before the next opening or closing delimiter,openis the trailing opening delimiter,
andcloseis the trailing closing delimiter.
At least one of these three strings will always be non-empty.
Ifopenis non-empty,closewill be empty, and vice-versa.
Ifsdoes not end with a closing delimiter, in the final tuple
yielded, bothopenandclosewill be empty strings.(Concatenating every string yielded byparse_delimiterstogether
produces a new string identical tos.)You can only specify a particular character as an opening delimiter
once, though you may reuse a particular character as a closing
delimiter multiple times.re_partition(text, pattern, count=1, *, flags=0, reverse=False)Likestr.partition, butpatternis matched as a regular expression.textcan be a string or a bytes object.patterncan be a string, bytes, orre.Patternobject.textandpattern(orpattern.pattern) must be the same type.Ifpatternis found in text, returns a tuple(before,match,after)wherebeforeis the text before the matched text,matchis there.Matchobject resulting from the match, andafteris the text after the matched text.Ifpatternappears intextmultiple times,re_partitionwill match against the first (leftmost)
appearance.Ifpatternis not found intext, returns a tuple(text,None,'')where the empty string isstrorbytesas appropriate.Passing in an explicitcountlets you control how many timesre_partitionpartitions the string.re_partitionwill always
return a tuple containing(2*count)+1elements, and
odd-numbered elements will be eitherre.Matchobjects orNone.
Passing in acountof 0 will always return a tuple containings.Ifpatternis a string or bytes object,flagsis passed in
as theflagsargument tore.compile.Ifreverseis true, partitions starting at the right,
likere_rpartition.(In older versions of Python,re.Patternwas a private type calledre._pattern_type.)re_rpartition(text, pattern, count=1, *, flags=0)Likestr.rpartition, butpatternis matched as a regular expression.textcan be astrorbytesobject.patterncan be astr,bytes, orre.Patternobject.textandpattern(orpattern.pattern) must be the same type.Ifpatternis found intext, returns a tuple(before,match,after)wherebeforeis the text before the matched text,matchis the re.Match object resulting from the match, andafteris the text after the matched text.Ifpatternappears intextmultiple times,re_partitionwill match against the last (rightmost)
appearance.Ifpatternis not found intext, returns a tuple('',None,text)where the empty string isstrorbytesas appropriate.Passing in an explicitcountlets you control how many timesre_rpartitionpartitions the string.re_rpartitionwill always
return a tuple containing(2*count)+1elements, and
odd-numbered elements will be eitherre.Matchobjects orNone.
Passing in acountof 0 will always return a tuple containings.Ifpatternis a string,flagsis passed in
as theflagsargument tore.compile.(In older versions of Python,re.Patternwas a private type calledre._pattern_type.)reversed_re_finditer(pattern, string, flags=0)An iterator.  Behaves almost identically to the Python
standard library functionre.finditer, yielding
non-overlapping matches ofpatterninstring.  The difference
is,reversed_re_finditersearchesstringfrom right to left.patterncan bestr,bytes, or a precompiledre.Patternobject.
If it'sstrorbytes, it'll be compiled
withre.compileusing theflagsyou passed in.stringshould be the same type aspattern(orpattern.pattern).split_quoted_strings(s, quotes=('"', "'"), *, triple_quotes=True, backslash='\\')Splitssinto quoted and unquoted segments.scan be eitherstrorbytes.quotesis an iterable of quote separators, eitherstrorbytesmatchings.  Note thatsplit_quoted_stringsonly supports quotecharacters,as in, each quote separator must be exactly
one character long.Returns an iterator yielding 2-tuples:(is_quoted, segment)wheresegmentis a substring ofs, andis_quotedis true if the segment is
quoted.  Joining all the segments together recreatess.Iftriple_quotesis true, supports "triple-quoted" strings like Python.Ifbackslashis a character, this character will quoting characters inside
a quoted string, like the backslash character inside strings in Python.split_text_with_code(s, *, tab_width=8, allow_code=True, code_indent=4, convert_tabs_to_spaces=True)Splitssinto individual words,
suitable for feeding intowrap_words.smay be eitherstrorbytes.Paragraphs indented by less thancode_indentwill be
broken up into individual words.Ifallow_codeis true, paragraphs indented by at leastcode_indentspaces will preserve their whitespace:
internal whitespace is preserved, and the newline is
preserved.  (This will preserve the formatting of code
examples when these words are rejoined into lines bywrap_words.)For more information, see the deep-dive onWord wrapping and formatting.str_linebreaksA tuple ofstrobjects, representing every line-breaking
whitespace character recognized by the Pythonstrobject.
Identical tolinebreaks.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.  See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.str_linebreaks_without_crlfEquivalent tostr_linebreakswithout'\r\n'.str_whitespaceA tuple ofstrobjects, representing every whitespace
character recognized by the Pythonstrobject.
Identical towhitespace.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.  See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.str_whitespace_without_crlfEquivalent tostr_whitespacewithout'\r\n'.unicode_linebreaksA tuple ofstrobjects, representing every line-breaking
whitespace character defined by Unicode.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.  See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.unicode_linebreaks_without_crlfEquivalent tounicode_linebreakswithout'\r\n'.unicode_whitespaceA tuple ofstrobjects, representing every whitespace
whitespace character defined by Unicode.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.  See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.unicode_whitespace_without_crlfEquivalent tounicode_whitespacewithout'\r\n'.whitespaceA tuple ofstrobjects, representing every whitespace
character recognized by the Pythonstrobject.
Identical tostr_whitespace.Useful as aseparatorargument forbigfunctions that accept one,
e.g.thebig"multi-" family of functions,or thelinesand lines modifier functions.Also contains'\r\n'.  See the deep-dive section onThe Unix, Mac, and DOS line-break conventionsfor more.For more information, please see theWhitespace and line-breaking characters in Python and bigdeep-dive.whitespace_without_crlfEquivalent towhitespacewithout'\r\n'.wrap_words(words, margin=79, *, two_spaces=True)Combineswordsinto lines and returns the result as a string.
Similar totextwrap.wrap.wordsshould be an iterator yielding str or bytes strings,
and these strings should already be split at word boundaries.
Here's an example of a valid argument forwords:"this is an example of text split at word boundaries".split()A single'\n'indicates a line break.
If you want a paragraph break, embed two'\n'characters in a row.marginspecifies the maximum length of each line. The length of
every line will be less than or equal tomargin, unless the length
of an individual element insidewordsis greater thanmargin.Iftwo_spacesis true, elements fromwordsthat end in
sentence-ending punctuation ('.','?', and'!')
will be followed by two spaces, not one.Elements inwordsare not modified; any leading or trailing
whitespace will be preserved.  You can use this to preserve
whitespace where necessary, like in code examples.For more information, see the deep-dive onWord wrapping and formatting.big.timeFunctions for working with time.  Currently deals specifically
with timestamps.  The time functions inbigare designed
to make it easy to use best practices.date_ensure_timezone(d, timezone)Ensures that adatetime.dateobject has a timezone set.Ifdhas a timezone set, returnsd.
Otherwise, returns a newdatetime.dateobject equivalent todwith itstzinfoset
totimezone.date_set_timezone(d, timezone)Returns a newdatetime.dateobject identical
todbut with itstzinfoset totimezone.datetime_ensure_timezone(d, timezone)Ensures that adatetime.datetimeobject has
a timezone set.Ifdhas a timezone set, returnsd.
Otherwise, creates a newdatetime.datetimeobject equivalent todwith itstzinfoset
totimezone.datetime_set_timezone(d, timezone)Returns a newdatetime.datetimeobject identical
todbut with itstzinfoset totimezone.parse_timestamp_3339Z(s, *, timezone=None)Parses a timestamp string returned bytimestamp_3339Z.
Returns adatetime.datetimeobject.timezoneis an optional default timezone, and should
be adatetime.tzinfoobject (orNone).  If provided,
and the time represented in the string doesn't specify
a timezone, thetzinfoattribute of the returned object
will be explicitly set totimezone.parse_timestamp_3339Zdepends on thepython-dateutilpackage.  Ifpython-dateutilis unavailable,parse_timestamp_3339Zwill also be unavailable.timestamp_3339Z(t=None, want_microseconds=None)Return a timestamp string in RFC 3339 format, in the UTC
time zone.  This format is intended for computer-parsable
timestamps; for human-readable timestamps, usetimestamp_human().Example timestamp:'2021-05-25T06:46:35.425327Z'tmay be one of several types:Iftis None,timestamp_3339Zuses the current time in UTC.Iftis an int or a float, it's interpreted as seconds
since the epoch in the UTC time zone.Iftis atime.struct_timeobject ordatetime.datetimeobject, and it's not in UTC, it's converted to UTC.
(Technically,time.struct_timeobjects are converted to GMT,
usingtime.gmtime.  Sorry, pedants!)Ifwant_microsecondsis true, the timestamp ends with
microseconds, represented as a period and six digits between
the seconds and the'Z'.  Ifwant_microsecondsisfalse, the timestamp will not include this text.
Ifwant_microsecondsisNone(the default), the timestamp
ends with microseconds if the type oftcan represent
fractional seconds: a float, adatetimeobject, or the
valueNone.timestamp_human(t=None, want_microseconds=None)Return a timestamp string formatted in a pleasing way
using the currently-set local timezone.  This format
is intended for human readability; for computer-parsable
time, usetimestamp_3339Z().Example timestamp:"2021/05/24 23:42:49.099437"tcan be one of several types:IftisNone,timestamp_humanuses the current local time.Iftis an int or float, it's interpreted as seconds since the epoch.Iftis atime.struct_timeordatetime.datetimeobject,
it's converted to the local timezone.Ifwant_microsecondsis true, the timestamp will end with
the microseconds, represented as ".######".  Ifwant_microsecondsis false, the timestamp will not include the microseconds.Ifwant_microsecondsisNone(the default), the timestamp
ends with microseconds if the type oftcan represent
fractional seconds: a float, adatetimeobject, or the
valueNone.Topic deep-divesThemulti-family of string functionsThis family of string functions was inspired by Python'sstr.strip,str.rstrip, andstr.splitlinesmethods.  These string splitting
methods are well-designed and often do what you want.  But they're
surprisingly narrow and opinionated.  What if your use case doesn't
map neatly to one of these functions?str.stripsupports twovery specificmodes of operation--unless you want to split your
string inexactlyone of those two modes, you probably can't usestr.stripto solve your problem.So whatcanyou use?  There'sre.strip, but that can be
hard to use.1Now there's a new answer:multisplit.multisplit'sgoal is to be the be-all end-all string splitting function.
It's designed to replaceeverymode of operation provided bystr.split,str.rstrip, andstr.splitlines, and it
can even replacestr.partitionandstr.rpartition.
(bigusesmultisplitto implementmultipartition.)[multisplit] can do it all!The downside ofmultisplit's'awesome flexibility is that, since itisso
sophisticated and tunable, it can be hard to use.  It takesfive keyword-only parametersafter all.  However, they're
designed to be reasonably memorable, and their default values
are designed to be easy to remember.  The best way to cope withmultisplit's'complexitiy is to use it as a building block for your own
text splitting functions.  For examplebigusesmultisplitto implementmultipartition,normalize_whitespace,lines,and several others functions.UsingmultisplitTo usemultisplit,pass in the string you want to split, the separators you
want to split on, and tweak its behavior with its five
keyword arguments.  It returns an iterator that yields
string segments from the original string in your preferred
format.The cornerstone ofmultisplitis theseparatorsargument.
This is an iterable of strings, of the same type (strorbytes)
as the string you want to split (s).multisplitwill split
the string ateachnon-overlapping instance of any string
specified inseparators.multisplitalso let you fine-tune how it splits, through five keyword-only
parameters:keeplets you include the separator strings in the output,
in a number of different formats.separatelets you specify whether adjacent separator strings
should be grouped together (likestr.stripoperating on
whitespace) or regarded as separate (likestr.stripwhen
you pass in an explicitsepseparator).striplets you strip separator strings from the beginning,
end, or both ends of the string you're splitting.  It also
supports a specialprogressivemode that duplicates the
behavior ofstr.stripwhen you useNoneas the separator.maxsplitlets you specify the maximum number of times to
split the string, exactly like themaxsplitargument tostr.strip.reverselets you applymaxsplitto the end of the string
and splitting backwards, exactly likestr.rstrip.To make it slightly easier to remember, all these keyword-only
parameters default to a false value.  (Well, technically,maxsplitdefaults to the special value-1, for compatibility
withstr.split.  But this is its special "don't do anything"
magic value.  All theotherkeyword-only parameters default
toFalse.)multisplitalso inspiredmultistripandmultipartition,which also take this sameseparatorsarguments.  There are also
otherbigfunctions that take aseparatorsargument; for
consistency's sakes, the parameter name always has the wordseparatorsin it.
(For example,comment_separatorsforlines_filter_comment_lines.)Demonstrations of eachmultisplitkeyword-only parameterTo give you a sense of how the five keyword-only parameters changes the behavior ofmultisplit,here's a breakdown of each of these parameters with examples.maxsplitmaxsplitspecifies the maximum number of times the string should be split.
It behaves the same as themaxsplitparameter tostr.split.The default value of-1means "split as many times as you can".  In our
example here, the string can be split a maximum of three times.  Therefore,
specifying amaxsplitof-1is equivalent to specifying amaxsplitof2or greater:>>>list(big.multisplit('appleXbananaYcookie',('X','Y')))# "maxsplit" defaults to -1['apple','banana','cookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=0))['appleXbananaYcookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=1))['apple','bananaYcookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=2))['apple','banana','cookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=3))['apple','banana','cookie']maxsplithas interactions withreverseandstrip.  For more
information, see the documentation regarding those parameters, below.keepkeepindicates whether or notmultisplitshould preserve the separator
strings in the strings it yields.  It supports four values: false, true,
and the special valuesALTERNATINGandAS_PAIRS.>>>list(big.multisplit('appleXbananaYcookie',('X','Y')))# "keep" defaults to False['apple','banana','cookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),keep=False))['apple','banana','cookie']Whenkeepis true,multisplitkeeps the separators, appending them to
the end of the separated string:>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),keep=True))['appleX','bananaY','cookie']WhenkeepisALTERNATING,multisplitkeeps the separators as separate
strings.  The first string yielded is always a non-separator string, and
from then on it always alternates between a separator string and a non-separator
string.  Put another way, if you store the output ofmultisplitin a list,
entries with an even-numbered index (0, 2, 4, ...) are always non-separator strings,
and entries with an odd-numbered index (1, 3, 5, ...) are always separator strings.>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),keep=big.ALTERNATING))['apple','X','banana','Y','cookie']Note thatALTERNATINGalways emits an odd number of strings; the first and last
strings are always non-separator strings.  Likestr.split, if the string you're
splitting starts or ends with a separator string,multisplitwill emit an empty
string there:>>>list(big.multisplit('1a1z1',('1',),keep=big.ALTERNATING))['','1','a','1','z','1','']Finally, whenkeepisAS_PAIRS,multisplitkeeps the separators as separate
strings.  But instead of yielding strings, it yields 2-tuples of strings.  Every
2-tuple contains a non-separator string followed by a separator string.If the original string starts with a separator, the first 2-tuple will contain
an empty non-separator string and the separator:>>>list(big.multisplit('YappleXbananaYcookie',('X','Y'),keep=big.AS_PAIRS))[('','Y'),('apple','X'),('banana','Y'),('cookie','')]The last 2-tuple willalwayscontain an empty separator string:>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),keep=big.AS_PAIRS))[('apple','X'),('banana','Y'),('cookie','')]>>>list(big.multisplit('appleXbananaYcookieXXX',('X','Y'),keep=big.AS_PAIRS,strip=True))[('apple','X'),('banana','Y'),('cookie','')](This rule means thatAS_PAIRSalways emits anevennumber of strings.
Contrast that withALTERNATING, which always emits anoddnumber of strings,
and the last string it emits is always a non-separator string.  Put another
way: if you ignore the tuples, the list of strings emitted byAS_PAIRSis the
same as those emitted byALTERNATING, exceptAS_PAIRSappends an empty
string.)Because of this rule, if the original string ends with a separator,
andmultisplitdoesn'tstripthe right side, the final tuple
emitted byAS_PAIRSwill be a 2-tuple containing two empty strings:>>>list(big.multisplit('appleXbananaYcookieX',('X','Y'),keep=big.AS_PAIRS))[('apple','X'),('banana','Y'),('cookie','X'),('','')]This looks strange and unnecessary.  But itiswhat you want.
This odd-looking behavior is discussed at length in the section below, titledWhy do you sometimes get empty strings when you split?The behavior ofkeepcan be affected by the value ofseparate.
For more information, see the next section, onseparate.separateseparateindicates whether multisplit should consider adjacent
separator strings insas one separator or as multiple separators
each separated by a zero-length string.  It can be either false or
true.>>>list(big.multisplit('appleXYbananaYXYcookie',('X','Y')))# separate defaults to False['apple','banana','cookie']>>>list(big.multisplit('appleXYbananaYXYcookie',('X','Y'),separate=False))['apple','banana','cookie']>>>list(big.multisplit('appleXYbananaYXYcookie',('X','Y'),separate=True))['apple','','banana','','','cookie']Ifseparateandkeepare both true values, and your string
has multiple adjacent separators,multisplitwill viewsas having zero-length non-separator strings between the
adjacent separators:>>>list(big.multisplit('appleXYbananaYXYcookie',('X','Y'),separate=True,keep=True))['appleX','Y','bananaY','X','Y','cookie']>>>list(big.multisplit('appleXYbananaYXYcookie',('X','Y'),separate=True,keep=big.AS_PAIRS))[('apple','X'),('','Y'),('banana','Y'),('','X'),('','Y'),('cookie','')]stripstripindicates whether multisplit should strip separators from
the beginning and/or end ofs.  It supports five values:
false, true,big.LEFT,big.RIGHT, andbig.PROGRESSIVE.By default,stripis false, which means it doesn't strip any
leading or trailing separators:>>>list(big.multisplit('XYappleXbananaYcookieYXY',('X','Y')))# strip defaults to False['','apple','banana','cookie','']Settingstripto true strips both leading and trailing separators:>>>list(big.multisplit('XYappleXbananaYcookieYXY',('X','Y'),strip=True))['apple','banana','cookie']big.LEFTandbig.RIGHTtellmultistripto only strip on that
side of the string:>>>list(big.multisplit('XYappleXbananaYcookieYXY',('X','Y'),strip=big.LEFT))['apple','banana','cookie','']>>>list(big.multisplit('XYappleXbananaYcookieYXY',('X','Y'),strip=big.RIGHT))['','apple','banana','cookie']big.PROGRESSIVEduplicates a specific behavior ofstr.splitwhen usingmaxsplit.  It always strips on the left, but it only strips on the right
if the string is completely split.  Ifmaxsplitis reached before the entire
string is split, andstripisbig.PROGRESSIVE,multisplitwon'tstrip
the right side of the string.  Note in this example how the trailing separatorYisn't stripped from the input string whenmaxsplitis less than3.>>>list(big.multisplit('XappleXbananaYcookieY',('X','Y'),strip=big.PROGRESSIVE))['apple','banana','cookie']>>>list(big.multisplit('XappleXbananaYcookieY',('X','Y'),maxsplit=0,strip=big.PROGRESSIVE))['appleXbananaYcookieY']>>>list(big.multisplit('XappleXbananaYcookieY',('X','Y'),maxsplit=1,strip=big.PROGRESSIVE))['apple','bananaYcookieY']>>>list(big.multisplit('XappleXbananaYcookieY',('X','Y'),maxsplit=2,strip=big.PROGRESSIVE))['apple','banana','cookieY']>>>list(big.multisplit('XappleXbananaYcookieY',('X','Y'),maxsplit=3,strip=big.PROGRESSIVE))['apple','banana','cookie']>>>list(big.multisplit('XappleXbananaYcookieY',('X','Y'),maxsplit=4,strip=big.PROGRESSIVE))['apple','banana','cookie']reversereversespecifies wheremultisplitstarts parsing the string--from
the beginning, or the end--and in what direction it moves when parsing
the string--towards the end, or towards the beginning.  It only supports
two values: when it's false,multisplitstarts at the beginning of the
string, and parses moving to the right (towards the end of the string).
But whenreverseis true,multisplitstarts at theendof the
string, and parses moving to theleft(towards thebeginningof the string).This has two noticable effects onmultisplit's output.  First, this
changes which splits are kept whenmaxsplitis less than the total number
of splits in the string.  Whenreverseis true, the splits are counted
starting on the right and moving towards the left:>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),reverse=True))# maxsplit defaults to -1['apple','banana','cookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=0,reverse=True))['appleXbananaYcookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=1,reverse=True))['appleXbanana','cookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=2,reverse=True))['apple','banana','cookie']>>>list(big.multisplit('appleXbananaYcookie',('X','Y'),maxsplit=3,reverse=True))['apple','banana','cookie']The second effect is far more subtle.  It's only relevant when splitting strings
containing multipleoverlappingseparators.  Whenreverseis false, and there
are two (or more) overlapping separators, the string is split by theleftmostoverlapping separator.  Whenreverseis true, and there are two (or more)
overlapping separators, the string is split by therightmostoverlapping
separator.Consider these two calls tomultisplit.  The only difference between them is
the value ofreverse.  They produce different results, even though neither
one usesmaxsplit.>>>list(big.multisplit('appleXAYbananaXAYcookie',('XA','AY')))# reverse defaults to False['apple','Ybanana','Ycookie']>>>list(big.multisplit('appleXAYbananaXAYcookie',('XA','AY'),reverse=True))['appleX','bananaX','cookie']Reimplementing library functions usingmultisplitHere are some examples of how you could usemultisplitto replace some common Python string splitting methods.  These exactly duplicate the
behavior of the originals.def_multisplit_to_split(s,sep,maxsplit,reverse):separate=sep!=Noneifseparate:strip=Falseelse:sep=big.ascii_whitespaceifisinstance(s,bytes)elsebig.whitespacestrip=big.PROGRESSIVEresult=list(big.multisplit(s,sep,maxsplit=maxsplit,reverse=reverse,separate=separate,strip=strip))ifnotseparate:# ''.split() == '   '.split() == []ifresultand(notresult[-1]):result.pop()returnresultdefstr_split(s,sep=None,maxsplit=-1):return_multisplit_to_split(s,sep,maxsplit,False)defstr_rsplit(s,sep=None,maxsplit=-1):return_multisplit_to_split(s,sep,maxsplit,True)defstr_splitlines(s,keepends=False):linebreaks=big.ascii_linebreaksifisinstance(s,bytes)elsebig.linebreaksl=list(big.multisplit(s,linebreaks,keep=keepends,separate=True,strip=False))iflandnotl[-1]:# yes, ''.splitlines() returns an empty listl.pop()returnldef_partition_to_multisplit(s,sep,reverse):ifnotsep:raiseValueError("empty separator")l=tuple(big.multisplit(s,(sep,),keep=big.ALTERNATING,maxsplit=1,reverse=reverse,separate=True))iflen(l)==1:empty=b''ifisinstance(s,bytes)else''ifreverse:l=(empty,empty)+lelse:l=l+(empty,empty)returnldefstr_partition(s,sep):return_partition_to_multisplit(s,sep,False)defstr_rpartition(s,sep):return_partition_to_multisplit(s,sep,True)You wouldn't want to use these, of course--Python's built-in
functions are so much faster!Why do you sometimes get empty strings when you split?Sometimes when you split usingmultisplit,
you'll get empty strings in the return value.  This might be unexpected,
violating thePrinciple Of Least Astonishment.But there are excellent reasons for this behavior.Let's start by observing whatstr.splitdoes.str.splitreally has two
major modes of operation: when you don't pass in a separator (or pass inNonefor the
separator), and when you pass in an explicit separator string.  In this latter mode,
the documentation says it regards every instance of a separator string as an individual
separator splitting the string.  What does that mean?  Watch what happens when you have
two adjacent separators in the string you're splitting:>>>'1,2,,3'.split(',')['1','2','','3']What's that empty string doing between'2'and'3'?  Here's how you should think about it:
when you pass in an explicit separator,str.splitsplits ateveryoccurance of that
separator in the string.  Italwayssplits the string into two places, whenever there's
a separator.  And when there are two adjacent separators, conceptually, they have a
zero-length string in between them:>>>'1,2,,3'[4:4]''The empty string in the output ofstr.splitrepresents the fact that there
were two adjacent separators.  Ifstr.splitdidn't add that empty string,
the output would look like this:['1','2','3']But then it'd be indistinguishable from splitting the same stringwithouttwo separators in a row:>>>'1,2,3'.split(',')['1','2','3']This difference is crucial when you want to reconstruct the original string from
the split list.str.splitwith a separator should always be reversable usingstr.join, and with that empty string there it works correctly:>>>','.join(['1','2','3'])'1,2,3'>>>','.join(['1','2','','3'])'1,2,,3'Now take a look at what happens when the string
you're splitting starts or ends with a separator:>>>',1,2,3,'.split(',')['','1','2','3','']This might seem weird.  But, just like with two adjacent separators,
this behavior is important for consistency.  Conceptually there's
a zero-length string between the beginning of the string and the first
comma.  Andstr.joinneeds those empty strings in order to correctly
recreate the original string.>>>','.join(['','1','2','3',''])',1,2,3,'Naturally,multisplitlets you duplicate this behavior.  When you wantmultisplitto behave just likestr.splitdoes with an explicit separator
string, just pass inkeep=False,separate=True, andstrip=False.
That is, ifaandbare strings,big.multisplit(a,(b,),keep=False,separate=True,strip=False)always produces the same output asa.split(b)For example, here'smultisplitsplitting the strings we've been playing with, using these parameters:>>>list(big.multisplit('1,2,,3',(',',),keep=False,separate=True,strip=False))['1','2','','3']>>>list(big.multisplit(',1,2,3,',(',',),keep=False,separate=True,strip=False))['','1','2','3','']This "emit an empty string" behavior also has ramifications whenkeepisn't false.
The behavior ofkeep=Trueis easy to predict;multisplitjust appends the separators
to the previous string segment:>>>list(big.multisplit('1,2,,3',(',',),keep=True,separate=True,strip=False))['1,','2,',',','3']>>>list(big.multisplit(',1,2,3,',(',',),keep=True,separate=True,strip=False))[',','1,','2,','3,','']The principle here is that, when you usekeep=True, you should be able to reconstitute
the original string with''.join:>>>''.join(['1,','2,',',','3'])'1,2,,3'>>>''.join([',','1,','2,','3,',''])',1,2,3,'keep=big.ALTERNATINGis much the same, except we insert the separators as their
own segments, rather than appending each one to the previous segment:>>>list(big.multisplit('1,2,,3',(',',),keep=big.ALTERNATING,separate=True,strip=False))['1',',','2',',','',',','3']>>>list(big.multisplit(',1,2,3,',(',',),keep=big.ALTERNATING,separate=True,strip=False))['',',','1',',','2',',','3',',','']Remember,ALTERNATINGoutput always begins and ends with a non-separator string.
If the string you're splitting begins or ends with a separator, the output
frommultisplitspecifyingkeep=ALTERNATINGwill correspondingly begin or end
with an empty string.And, as withkeep=True, you can also recreate the original string by passing
these arrays in to''.join:>>>''.join(['1',',','2',',','',',','3'])'1,2,,3'>>>''.join(['',',','1',',','2',',','3',',',''])',1,2,3,'Finally there'skeep=big.AS_PAIRS.  The behavior here seemed so strange,
initially I thought it was wrong.  But I've given it alotof thought, and
I've convinced myself that this is correct:>>>list(big.multisplit('1,2,,3',(',',),keep=big.AS_PAIRS,separate=True,strip=False))[('1',','),('2',','),('',','),('3','')]>>>list(big.multisplit(',1,2,3,',(',',),keep=big.AS_PAIRS,separate=True,strip=False))[('',','),('1',','),('2',','),('3',','),('','')]That tuple at the end, just containing two empty strings:('','')It'sso strange.How can that be right?It's the same asstr.split.multisplitmustsplit the string into two pieceseverytime it finds the separator
in the original string.  So itmustemit the empty non-separator string.
And since that zero-length string isn't (cannot!) be followed by a separator,
when usingkeep=AS_PAIRSthe final separator string isalsoempty.Think of it this way: with the tuple of empty strings there, you can easily
convert onekeepformat into any another.  (Provided that you know
what the separators were--either the sourcekeepformat was not false,
or you only used one separator string when callingmultisplit).
Without that tuple of empty strings at the end, you'd also have to have anifstatement to add or remove empty stuff from the end.I'll demonstrate this with a simple example.  Here's the output ofmultisplitsplitting the string'1a1z1'by the separator'1',
in each of the fourkeepformats:>>>list(big.multisplit('1a1z1','1',keep=False))['','a','z','']>>>list(big.multisplit('1a1z1','1',keep=True))['1','a1','z1','']>>>list(big.multisplit('1a1z1','1',keep=big.ALTERNATING))['','1','a','1','z','1','']>>>list(big.multisplit('1a1z1','1',keep=big.AS_PAIRS))[('','1'),('a','1'),('z','1'),('','')]Because theAS_PAIRSoutput ends with that tuple of empty
strings, we can mechanically convert it into any of the other
formats, like so:>>>result=list(big.multisplit('1a1z1','1',keep=big.AS_PAIRS))>>>result[('','1'),('a','1'),('z','1'),('','')]>>>[s[0]forsinresult]# convert to keep=False['','a','z','']>>>[s[0]+s[1]forsinresult]# convert to keep=True['1','a1','z1','']>>>[sfortinresultforsint][:-1]# convert to keep=big.ALTERNATING['','1','a','1','z','1','']If theAS_PAIRSoutputdidn'tend with that tuple of empty strings,
you'd need to add anifstatement to restore the trailing empty
strings as needed.Other differences between multisplit and str.splitstr.splitreturns anempty listwhen you split an
empty string by whitespace:>>>''.split()[]But not when you split by an explicit separator:>>>''.split('x')['']multisplitis consistent here.  If you split an empty string, it always returns an empty string,
as long as the separators are valid:>>>list(big.multisplit(''))['']>>>list(big.multisplit('',('a','b','c')))['']Similarly, when splitting a string that only contains whitespace,str.splitalso
returns an empty list:>>>'     '.split()[]This is really the same as "splitting an empty string", because whenstr.splitsplits on whitespace, the first thing it does is strip leading whitespace.If youmultisplita string that only contains whitespace, and you split on whitespace characters,
it returns two empty strings:>>>list(big.multisplit('     '))['','']This is because the string conceptually starts with a zero-length string,
then has a run of whitespace characters, then ends with another zero-length
string.  So those two empty strings are the leading and trailing zero-length
strings, separated by whitespace.  If you tellmultisplitto also strip the string, you'll get back a single empty string:>>>list(big.multisplit('     ',strip=True))['']Andmultisplitbehaves consistently even when you use different separators:>>>list(big.multisplit('ababa','ab'))['','']>>>list(big.multisplit('ababa','ab',strip=True))['']And I should know--multisplitis implemented usingre.split!Whitespace and line-breaking characters in Python and bigOverviewSeveral functions inbigtake aseparatorsargument, an iterable of separator strings.
Examples of these functions includelinesandmultisplit.Although you can use any iterable of strings
you like, most often you'll be separating on some
form of whitespace.  But what, exactly,iswhitespace?  There's more to this topic than you
might suspect.The good news is, you can almost certainly ignore all the
complexity.  These days the only whitespace characters you're
likely to encounter are spaces, tabs, newlines, and maybe
carriage returns.  Python andbighandle all those easily.With respect tobigand theseseparatorsarguments,bigprovides four values designed for use asseparators.
All four of these are tuples containing whitespace characters:When working withstrobjects, you'll want to use eitherbig.whitespaceorbig.linebreaks.big.whitespacecontains all the whitespace characters,big.linebreakscontains just the line-breaking
whitespace characters.bigalso has equivalents for working withbytesobjects:big.bytes_whitespaceandbig.bytes_linebreaks,respectively.Apart from exceptionally rare occasions, these are all you'll ever need.
And if that's all you need, you can stop reading this
section now.But what about those exceptionally rare occasions?
You'll be pleased to knowbighandles them too.
The rest of this section is
a deep dive into these rare occasions.PythonHere's the list of all characters recognized by
Pythonstrobjects as whitespace characters:# char    decimal   hex      name
##########################################
'\t'    , #     9 - 0x0009 - tab
'\n'    , #    10 - 0x000a - newline
'\v'    , #    11 - 0x000b - vertical tab
'\f'    , #    12 - 0x000c - form feed
'\r'    , #    13 - 0x000d - carriage return
'\x1c'  , #    28 - 0x001c - file separator
'\x1d'  , #    29 - 0x001d - group separator
'\x1e'  , #    30 - 0x001e - record separator
'\x1f'  , #    31 - 0x001f - unit separator
' '     , #    32 - 0x0020 - space
'\x85'  , #   133 - 0x0085 - next line
'\xa0'  , #   160 - 0x00a0 - non-breaking space
'\u1680', #  5760 - 0x1680 - ogham space mark
'\u2000', #  8192 - 0x2000 - en quad
'\u2001', #  8193 - 0x2001 - em quad
'\u2002', #  8194 - 0x2002 - en space
'\u2003', #  8195 - 0x2003 - em space
'\u2004', #  8196 - 0x2004 - three-per-em space
'\u2005', #  8197 - 0x2005 - four-per-em space
'\u2006', #  8198 - 0x2006 - six-per-em space
'\u2007', #  8199 - 0x2007 - figure space
'\u2008', #  8200 - 0x2008 - punctuation space
'\u2009', #  8201 - 0x2009 - thin space
'\u200a', #  8202 - 0x200a - hair space
'\u2028', #  8232 - 0x2028 - line separator
'\u2029', #  8233 - 0x2029 - paragraph separator
'\u202f', #  8239 - 0x202f - narrow no-break space
'\u205f', #  8287 - 0x205f - medium mathematical space
'\u3000', # 12288 - 0x3000 - ideographic spaceThis list was derived by iterating over every character
defined in Unicode, and testing to see if thesplit()method on a Pythonstrobject splits at that character.The first surprise: thisisn'tthe same as the list ofall characters defined by Unicode as whitespace.It'salmostthe same list, except Python adds four extra
characters:'\x1c','\x1d','\x1e', and'\x1f',
which respectively are called "file separator", "group separator",
"record separator", and "unit separator".
I'll refer to these as "the four ASCII separator characters".These characters were defined as part ofthe original ASCII
standard,way back in 1963.
As their names suggest, they were intended to be used as separator
characters for data, the same wayCtrl-Z was used to indicate end-of-file in the CPM and earliest
FAT filesystems.But the four ASCII separator characters were rarely used
even back in the day.  Today they're practically unheard of.As a rule, printing these characters to the screen generally
doesn't do anything--they don't move the cursor, and the
screen doesn't change.
So their behavior is a bit mysterious.  A lot of people (including
early Python programmers it seems!) thought that meant they're
whitespace.  This seems like an odd conclusion to me.  After all,
all the other whitespace characters move the cursor, either right
or down or both; these don't move the cursor at all.The Unicode standardis unambiguous: these characters arenot whitespace.And yet Python's
"Unicode object" behaves as if they are.  So I'd say this is a bug;
Python's Unicode object should implement what the Unicode standard says.
Like many bugs, this one has lingered for a long time.  The behavior
is present in Python 2, there'sa ten-year-old issue on the Python issue tracker about this,and it's not making progress.The second surprise has to do withbytesobjects.
Of course,bytesobjects represent binary data, and don't
necessarily represent characters.  Even if they do, they don't
have any encoding associated with them.  However, for
convenience--and backwards-compatibility with Python 2--Python'sbytesobjects support several method calls that treat the data
as if it were "ASCII-compatible".The surprise: These methods on Pythonbytesobjects recognize
adifferentset of whitespace characters.  Here's the list of
all bytes recognized by Pythonbytesobjects as whitespace:# char  decimal  hex    name
#######################################
'\t'    , #  9 - 0x09 - tab
'\n'    , # 10 - 0x0a - newline
'\v'    , # 11 - 0x0b - vertical tab
'\f'    , # 12 - 0x0c - form feed
'\r'    , # 13 - 0x0d - carriage return
' '     , # 32 - 0x20 - spaceThis list was derived by iterating over every possible
byte value, and testing to see if thesplit()method
on a Pythonbytesobject splits at that byte.The good news is, this list is the same as ASCII's list,
and it agrees with Unicode.
In fact this list is quite familiar to C programmers;
it's the same whitespace characters recognized by the
standard C functionisspace()(inctypes.h).Python has used this function to decide which characters
are and aren't whitespace in 8-bit strings since its very
beginning.Notice that this listdoesn'tcontain the
four ASCII separator characters.  That these two
types in Python don't agree only enhances the mystery.Line-breaking charactersThe situation is slightly worse with line-breaking
characters. Line-breaking characters are a subset of
whitespace characters; they're whitespace characters that
always move the cursor down.  And, as with whitespace
generally, Pythonstrobjects don't agree with Unicode
about what is and is not a line-breaking character,
and Pythonbytesobjects don't agree with either of those.Here's the list of all Unicode characters recognized by
Pythonstrobjects as line-breaking characters:# char    decimal   hex      name
##########################################
'\n'    , #   10 0x000a - newline
'\v'    , #   11 0x000b - vertical tab
'\f'    , #   12 0x000c - form feed
'\r'    , #   13 0x000d - carriage return
'\x1c'  , #   28 0x001c - file separator
'\x1d'  , #   29 0x001d - group separator
'\x1e'  , #   30 0x001e - record separator
'\x85'  , #  133 0x0085 - next line
'\u2028', # 8232 0x2028 - line separator
'\u2029', # 8233 0x2029 - paragraph separatorThis list was derived by iterating over every character
defined in Unicode, and testing to see if thesplitlines()method on a Pythonstrobject splits at that character.Again, this is different fromthe list of characters
defined as line-breaking whitespace in Unicode.And again it's because Python defines some of the four ASCII separator
characters as line-breaking characters.  In this case
it's only the first three; Python doesn't consider
the fourth, "unit separator", as a line-breaking character.
(I don't know why Python draws this distinction...
but then again, I don't know why it considers the
first three to be line-breaking.  It'salla mystery to me.)Here's the list of all characters recognized by
Pythonbytesobjects as line-breaking characters:# char  decimal hex      name
#######################################
'\n'    , #  10 0x000a - newline
'\r'    , #  13 0x000d - carriage returnThis list was derived by iterating over every possible
byte, and testing to see if thesplitlines()method on a Pythonbytesobject splits at that byte.It's here we find our final unpleasant surprise:
the methods on Pythonbytesobjectsdon'tconsider'\v'(vertical tab)
and'\f'(form feed)
to be line-break characters.  I assert this is also a bug.
These are well understood to be line-breaking characters;
"vertical tab" is like a "tab", except it moves the cursor
down instead of to the right.  And "form feed" moves the
cursor to the top left of the next "page", which requires
advancing at least one line.Howbighandles this situationTo be crystal clear: the odds that any of this will cause
a problem for you areextremelylow.  In order for it
to make a difference:you'd have to encounter text using one of these six characters
where Python disagrees with Unicode and ASCII, andyou'd have to process the input based on some definition
of whitespace, andit would have to produce different results than you might
have other wise expected,andthis difference in results would have to be important.It seems extremely unlikely that all of these will be true for you.In case thisdoesaffect you,bighas
a complete set of predefined whitespace tuples that will
handle any of these situations.bigdefines a total oftentuples, sorted into
five categories.In every category there are two values: one that containswhitespace, the other containslinebreaks.  Thewhitespacetuple contains all the possible values of
whitespace--characters that move the cursor either
horizontally, or vertically, or both, but don't
print anything visible to the screen.  Thelinebreakstuple contains the subset of whitespace characters that
move the cursor vertically.The most important two values start withstr_:str_whitespaceandstr_linebreaks.These contain all the whitespace characters
recognized by the Pythonstrobject.Next are two values that start withunicode_:unicode_whitespaceandunicode_linebreaks.These contain all the whitespace characters
defined in the Unicode standard.Third, two values that start withascii_:ascii_whitespaceandascii_linebreaks.These contain all the whitespace characters
defined in ASCII.Fourth, two values that start withbytes_:bytes_whitespaceandbytes_linebreaks.These contain all the whitespace characters
recognized by the Pythonbytesobject.Finally we have the two tuples that lack a prefix:whitespaceandlinebreaks.These are the tuples you should use most of the time,
and severalbigfunctions use them as default values.
These are identical tostr_whitespaceandstr_linebreaksrespectively.(bigactually defines anadditionalten tuples,
as discussed in the very next section.)The Unix, Mac, and DOS line-break conventionsHistorically, different platforms used different
ASCII characters--or sequences of ASCII characters--to
represent "go to the next line" in text files.  Here are the
most popular conventions:\n    - UNIX, Amiga
\r    - Mac OS (before OS X), many 8-bit computers
\r\n  - Windows, DOS(There are a couple more conventions, and a lot more history,
in theWikipedia article on newlines.)Handling these differing conventions was a stumbling block
for a long time--both for computer programs, and in the
daily lives of computer users.  Python went through several
iterations on how to handle this, eventually settling on
the"universal newlines"support added in Python 2.3.
These days the world seems to be converging on the UNIX
standard'\n'; Windows supports it, and it's the default
on every other modern platform.
So in practice you probably don't have end-of-line conversion
problems, either.But just in case,bighas one more trick.  All of
the tuples defined in the previous section--fromwhitespacetoascii_linebreaks--also contain this string:'\r\n'(The twobytes_tuples contain thebytesequivalent,b'\r\n.)This addition means that, when you use one of these tuples
with one of thebigfunctions that take separators,
it'll split on\r\nas if it was one character.  This
means thatbigitself should automatically handle
the DOS and Windows end-of-line character sequence, in
case one happens to creep into your data.If you don't want this behavior, just add the suffix_without_crlfto the end of any of the ten tuples,
e.g.whitespace_without_crlf,bytes_linebreaks_without_crlf.Whitespace and line-breaking characters for other platformsWhat if you need to split text by whitespace, or by lines,
but that text is inbytesformat with an unusual encoding?bigmakes that easy too.  If one of the builtin tuples
won't work for you, you can can make your own tuple from scratch,
or modify an existing tuple to meet your needs.For example, let's say you need to split a document by
whitespace, and the document is encoded incode page 850,
aka "latin-1".Normally the easiest thing would be to decode it astrobject
using the'latin-1'text codec, then operate on it normally.
But you might have reasons why you don't want to decode it--maybe
the document is damaged and doesn't decode properly, and it's
easier to work with the encoded bytes than to fix it.  If you
want to process the text with abigfunction that accepts aseparatorargument, you could make your own custom tuple
of "latin-1" whitespace characters.  "latin-1" has the same
whitespace characters as ASCII, but adds one more, value 255,
which is not line-breaking.  So it's easy to make the appropriate
tuples yourself:latin_1_whitespace = big.bytes_whitespace + (b'\xff',)
latin_1_linebreaks = big.bytes_linebreaksWhat if you want to process abytesobject containing
UTF-8?  That's easy too.  Just convert one of the existing
tuples containingstrobjects usingbig.encode_strings.For example, to split a UTF-8 encoded bytes objectbusing
the Unicode line-breaking characters, you could call:multisplit(b, encode_strings(unicode_linebreaks, encoding='utf-8'))Note that this technique probably won't work correctly for most other
multibyte encodings, for exampleUTF-16.For these encodings, you should decode tostrbefore processing.Why?  It's becausemultisplitcould find matches in multibyte
sequencesstraddlingcharacters.  Consider this example:>>>haystack='\u0101\u0102'>>>needle='\u0201'>>>needleinhaystackFalse>>>>>>encoded_haystack=haystack.encode('utf-16-le')>>>encoded_needle=needle.encode('utf-16-le')>>>encoded_needleinencoded_haystackTrueThe character'\u0201'doesn't appear in the original string,
but theencodedversion appears in theencodedstring,
as thesecondbyte of thefirstcharacter and thefirstbyte of thesecondcharacter:>>>encoded_haystackb'\x01\x01\x02\x01'>>>encoded_needleb'\x01\x02'linesand lines modifier functionslinesis a function that makes it easy to write well-behaved, feature-rich text parsers.linesitself
iterates over a string, returning an iterator that yields individual lines
split from that string.  The iterator yields a 2-tuple:(LinesInfo, line)TheLinesInfoobject provides the line number and starting column number
for each line.  This makes it easy for your parser to provide
line and column information for error messages.This iterator is designed to be modified by "lines modifier"
functions.  These are functions that consume a lines
iterator and re-yield the values, possibly modifying or
discarding them along the way.  For example, passing
alinesiterator intolines_filter_empty_linesresults
in an iterator that skips over the empty lines.
All the lines modifier functions that ship withbigstart with the stringlines_.Most lines modifier function names belong to a category,
encoded as the second word in the function name
(immediately afterlines_).  Some examples:lines_filter_functions conditionally remove
lines from the output.  For example,lines_filter_empty_lineswill only yield a line if it isn't empty.lines_strip_functions may remove one or
more substrings from the line.  For example,lines_strip_indent(li)strips the leading whitespace from a line before yielding
it.  (Whenever a lines modifier removes leading text from a line,
it will add aleadingfield to the accompanyingLineInfoobject
containing the removed substring, and will also update thecolumn_numberof the line to reflect the new starting column.)lines_convert_functions means this lines modifier may change one
or more substrings in the line.  For example,lines_convert_tabs_to_spaceschanges tab characters
to space characters in any lines it processes.(bigisn't strict about these category names though.
For example,lines_containing(li, s, *, invert=False)andlines_grep(li, pattern, *, invert=False, flags=0)are obviously "filter" modifiers, but their names
don't start withlines_filter_.)All lines modifier functions are composable with each
other; you can "stack" them together simply by passing
the output of one into the input of another.  For example,withopen("textfile.txt","rt")asf:forinfo,lineinbig.lines_filter_empty_lines(big.lines_rstrip(lines(f.read()))):...will iterate over the lines oftextfile.txt, skipping
over all empty lines and lines that consist only of
whitespace.When you stack line modifiers in this way, note that theoutermodifiers happenlater.  In the above example,
each line is first "r-stripped", and then discarded
if it's empty.  If you stacked the line modifiers in
the opposite order:withopen("textfile.txt","rt")asf:forinfo,lineinbig.lines_rstrip(big.lines_filter_empty_lines(lines(f.read()))):...then it'd filter out empty lines first, andthen"r-strip" the lines.  So lines in the input that contained
only whitespace would still get yielded as empty lines,
which is probably not what you want.  Ordering is important!Of course, you can write your own lines modifier functions.
Simply accept a lines iterator as an argument, iterate over
it, and yield each line info and line, modifying them
(or not yielding them!) as you see fit.  You could
even write your own lines iterator, a replacement forlines,
if you need functionalitylinesdoesn't provide.Note that if you write your own lines modifier function,
and it removes text from the beginning the line, you must
updatecolumn_numberin theLineInfoobject manually--it
doesn't happen automatically.Word wrapping and formattingbigcontains three functions used to reflow and format text
in a pleasing manner.  In the order you should use them, they aresplit_text_with_code,wrap_words(),,
and optionallymerge_columns.
This trio of functions gives you the following word-wrap superpowers:Paragraphs of text representing embedded "code" don't get word-wrapped.
Instead, their formatting is preserved.Multiple texts can be merged together into multiple columns."text" vs "code"Thebigword wrapping functions also distinguish between
"text" and "code".  The main distinction is, "text" lines can
get word-wrapped, but "code" lines shouldn't.bigconsiders
any line starting with enough whitespace to be a "code" line;
by default, this is four spaces.  Any non-blank line that
starting with four spaces is a "code" line, and any non-blank
line that starts with less than four spaces is a "text" line.In "text" mode:words are separated by whitespace,initial whitespace on the line is discarded,the amount of whitespace between words is irrelevant,individual newline characters are ignored, andmore than two newline characters are converted into exactly
two newlines (aka a "paragraph break").In "code" mode:all whitespace is preserved, except for trailing whitespace on a line, andall newline characters are preserved.Also, wheneversplit_text_with_codeswitches between
"text" and "code" mode, it emits a paragraph break.Split text arrayAsplit text arrayis an intermediary data structure
used bybig.textfunctions to represent text.
It's literally just an array of strings, where the strings
represent individual word-wrappable substrings.split_text_with_codereturns asplit text array,andwrap_words()consumes asplit text array.You'll see four kinds of strings in asplit text array:Individual words, ready to be word-wrapped.Entire lines of "code", preserving their formatting.Line breaks, represented by a single newline:'\n'.Paragraph breaks, represented by two newlines:'\n\n'.ExamplesThis might be clearer with an example or two.  The following text:hello there!
this is text.


this is a second paragraph!would be represented in a Python string as:"hello there!\nthis is text.\n\n\nthis is a second paragraph!"Note the three newlines between the second and third lines.If you then passed this string in tosplit_text_with_code,it'd return thissplit text array:['hello','there!','this','is','text.','\n\n','this','is','a','second','paragraph!']split_text_with_codemerged the first two lines together into
a single paragraph, and collapsed the three newlines separating
the two paragraphs into a "paragraph break" marker
(two newlines in one string).Now let's add an example of text with some "code".  This text:What are the first four squared numbers?

    for i in range(1, 5):


        print(i**2)

Python is just that easy!would be represented in a Python string as (broken up into multiple strings for clarity):"What are the first four squared numbers?\n\n"+"    for i in range(1, 5):\n\n\n"+"        print(i**2)\n\nPython is just that easy!"split_text_with_codeconsiders the two lines with initial whitespace as "code" lines,
and so the text is split into the followingsplit text array:['What','are','the','first','four','squared','numbers?','\n\n','    for i in range(1, 5):','\n','\n','\n','        print(i**2)','\n\n','Python','is','just','that','easy!']Here we have a "text" paragraph, followed by a "code" paragraph, followed
by a second "text" paragraph.  The "code" paragraph preserves the internal
newlines, though they are represented as individual "line break" markers
(strings containing a single newline).  Every paragraph is separated by
a "paragraph marker".Here's a simple algorithm for joining asplit text arrayback into a
single string:prev=Nonea=[]forwordinsplit_text_array:ifnot(prevandprev.isspace()andword.isspace()):a.append(' ')a.append(word)text="".join(a)Of course, this algorithm is too simple to do word wrapping.
Nor does it handle adding two spaces after sentence-ending
punctuation.  In practice, you shouldn't do this by hand;
you should usewrap_words.Merging columnsmerge_columnsmerges multiple strings into columns on the same line.For example, it could merge these three Python strings:["Here's the first\ncolumn of text.","More text over here!\nIt's the second\ncolumn!  How\nexciting!","And here's a\nthird column.",]into the following text:Here's the first    More text over here!   And here's a
column of text.     It's the second        third column.
                    column!  How
                    exciting!(Note thatmerge_columnsdoesn't do its own word-wrapping;
instead, it's designed to consume the output ofwrap_words.)Each column is passed in tomerge_columnsas a "column tuple":(s,min_width,max_width)sis the string,min_widthis the minimum width of the column, andmax_widthis the minimum width of the column.As you saw above,scan contain newline characters,
andmerge_columnsobeys those when formatting each column.For each column,merge_columnsmeasures the longest
line of each column.  The width of the column is determined
as follows:If the longest line is less thanmin_widthcharacters long,
the column will bemin_widthcharacters wide.If the longest line is less than or equal tomin_widthcharacters long, and less than or equal tomax_widthcharacters long, the column will be as wide as the longest line.If the longest line is greater thanmax_widthcharacters long,
the column will bemax_widthcharacters wide, and lines that
are longer thanmax_widthcharacters will "overflow".OverflowWhat is "overflow"?  It's a conditionmerge_columnsmay encounter when the text in a column is wider than that
column'smax_width.merge_columnsneeds to consider both "overflow lines",
lines that are longer thanmax_width, and "overflow columns",
columns that contain one or more overflow lines.What doesmerge_columnsdo when it encounters overflow?merge_columnssupports three "strategies" to deal with this condition, and you can specify
which one you want using itsoverflow_strategyparameter.  The three
strategies are:OverflowStrategy.RAISE: Raise anOverflowErrorexception.  The default.OverflowStrategy.INTRUDE_ALL: Intrude into all subsequent columns on
all lines where the overflowed column is wider than itsmax_width.
The subsequent columns "make space" for the overflow text by not adding
text on those overflowed lines; this is called "pausing" their output.OverflowStrategy.DELAY_ALL:  Delay all columns after the overflowed
column, not beginning any until after the last overflowed line
in the overflowed column.  This is like theINTRUDE_ALLstrategy,
except that the columns "make space" by pausing their output until
the last overflowed line.Whenoverflow_strategyisINTRUDE_ALLorDELAY_ALL, and
eitheroverflow_beforeoroverflow_afteris nonzero, these
specify the number of extra lines before or after
the overflowed lines in a column where the subsequent columns
"pause".EnhancedTopologicalSorterOverviewbig'sTopologicalSorteris a drop-in replacement forgraphlib.TopologicalSorterin the Python standard library (new in 3.9).
However, the version inbighas been greatly upgraded:prepareis now optional, though it still performs a cycle check.You can add nodes and edges to a graph at any time, even while
iterating over the graph.  Adding nodes and edges always succeeds.You can remove nodes from graphgwith the new methodg.remove(node).
Again, you can do this at any time, even while iterating over the graph.
Removing a node from the graph always succeeds, assuming the node is in the graph.The functionality for iterating over a graph now lives in its own object called
aview.  View objects implement theget_ready,done, and__bool__methods.  There's a default view built in to the graph object;
theget_ready,done, and__bool__methods on a graph just call
into the graph's default view.  You can create a new view at any time
by calling the newviewmethod.Note that if you're using a view to iterate over the graph, and you modify the graph,
and the view now represents a state that isn'tcoherentwith the graph,
attempting to use that view raises aRuntimeError.  More on what I mean
by "coherence" in a minute.This implementation also fixes some minor warts with the existing API:In Python's implementation,static_orderandget_ready/doneare mutually exclusive.  If you ever callget_readyon a graph,  you can never callstatic_order, and vice-versa.  The implementaiton inbigdoesn't have this restriction, because its implementation ofstatic_ordercreates and uses a new view object
every time it's called..In Python's implementation, you can only iterate over the graph once, or callstatic_orderonce.
The implementation inbigsolves this in several ways: it allows you to create as many views as you
want, and you can call the newresetmethod on a view to reset it to its initial state.Graph / view coherenceSo what does it mean for a view to no longer be coherent with the graph?
Consider the following code:g=big.TopologicalSorter()g.add('B','A')g.add('C','A')g.add('D','B','C')g.add('B','A')v=g.view()g.ready()# returns ('A',)g.add('A','Q')First this code creates a graphgwith a classic "diamond"
dependency pattern.  Then it creates a new viewv, and gets
the currently "ready" nodes, which consists just of the node'A'.  Finally it adds a new dependency:'A'depends on'Q'.At this moment, viewvis no longercoherent.'A'has been
marked as "ready", but'Q'has not.  And yet'A'depends on'Q'.
All those statements can't be true at the same time!
So viewvis no longercoherent,and any attempt to interact
withvraises an exception.To state it more precisely: if viewvis a view on graphg,
and you callg.add('Z', 'Y'),
and neither of these statements is true in viewv:'Y'has been marked asdone.'Z'has not yet been yielded byget_ready.thenvis no longer "coherent".(If'Y'has been marked asdone, then it's okay to make'Z'dependent on'Y'regardless of what state'Z'is in.  Likewise, if'Z'hasn't been yielded
byget_readyyet, then it's okay to make'Z'dependent on'Y'regardless
of what state'Y'is in.)Note that you can restore a view to coherence.  In this case,
removing eitherYorZfromgwould resolve the incoherence
betweenvandg, andvwould start working again.Also note that you can have multiple views, in various states of iteration,
and by modifying the graph you may cause some to become incoherent but not
others.  Views are completely independent from each other.Bound inner classesOverviewOne minor complaint I have about Python regards inner classes.
An "inner class" is a class defined inside another class.  And,
well, inner classes seem kind of half-baked.  Unlike functions,
inner classes don't get bound to the object.Consider this Python code:classOuter(object):defmethod(self):passclassInner(object):def__init__(self):passo=Outer()o.method()i=o.Inner()Wheno.methodis called, Python automatically passes in theoobject as the first parameter
(generally calledself).  In object-oriented lingo,oisboundtomethod, and indeed
Python calls this object abound method:>>> o.method
    <bound method Outer.method of <__main__.Outer object at 0x########>>But that doesn't happen wheno.Inneris called.  (Itdoespass in
aself, but in this case it's the newly-createdInnerobject.)
There's just no built-in way for theo.Innerobject being constructed
toautomaticallyget a reference too.  If you need one, you must
explicitly pass one in, like so:classOuter(object):defmethod(self):passclassInner(object):def__init__(self,outer):self.outer=outero=Outer()o.method()i=o.Inner(o)This seems redundant.  You don't have to pass inoexplicitly to method calls,
why should you have to pass it in explicitly to inner classes?Well--now you don't have to!
You just decorate the inner class with@big.BoundInnerClass,
andBoundInnerClasstakes care of the rest!Using bound inner classesLet's modify the above example to use ourBoundInnerClassdecorator:frombigimportBoundInnerClassclassOuter(object):defmethod(self):pass@BoundInnerClassclassInner(object):def__init__(self,outer):self.outer=outero=Outer()o.method()i=o.Inner()Notice thatInner.__init__now requires anouterparameter,
even though you didn't pass in any arguments too.Inner.
When it's called,ois magically passed in toouter!
Thanks,BoundInnerClass!  You've saved the day!Decorating an inner class like this always adds a second positional
parameter, afterself.  And, likeself, you don't have
to use the nameouter, you can use any name you like.
(Although it's probably a good idea, for consistency's sakes.)InheritanceBound inner classes get slightly complicated when mixed with inheritance.
It's not all that difficult, you merely need to obey the following rules:A bound inner class can inherit normally from any unbound class.To subclass from a bound inner class while still inside the outer
class scope, or when referencing the inner class from the outer class
(as opposed to an instance of the outer class), you must actually
subclass or referenceclassname.cls.This is because inside the
outer class, the "class" you see is actually an instance of aBoundInnerClassobject.All classes that inherit from a bound inner class must always call the
superclass's__init__. You don't need to pass in theouterparameter;
it'll be automatically passed in to the superclass's__init__as before.An inner class that inherits from a bound inner class, and which also
wants to be bound to the outer object, should be decorated withBoundInnerClass.An inner class that inherits from a bound inner class, but doesn't
want to be bound to the outer object, should be decorated withUnboundInnerClass.Restating the last two rules: every class that descends from anyBoundInnerClassshould be decorated with eitherBoundInnerClassorUnboundInnerClass.
Which one you use depends on what behavior you want--whether or
not you want your inner subclass to automatically get theouterinstance passed in to its__init__.Here's a simple example using inheritance with bound inner classes:frombigimportBoundInnerClass,UnboundInnerClassclassOuter(object):@BoundInnerClassclassInner(object):def__init__(self,outer):self.outer=outer@UnboundInnerClassclassChildOfInner(Inner.cls):def__init__(self):super().__init__()o=Outer()i=o.ChildOfInner()We followed the rules:Innerinherits from object; since object isn't a bound inner class,
there are no special rules about inheritanceInnerneeds to obey.ChildOfInnerinherits fromInner.cls, notInner.SinceChildOfInnerinherits from aBoundInnerClass,
it must be
decorated with eitherBoundInnerClassorUnboundInnerClass.
It doesn't want the outer object passed in, so it's decorated
withUnboundInnerClass.ChildOfInner.__init__callssuper().__init__.Note that, becauseChildOfInneris decorated withUnboundInnerClass,
it doesn't take anouterparameter.  Nor does it pass in anouterargument when it callssuper().__init__.  But when the constructor forInneris called, the correctouterparameter is passed in--like magic!
Thanks again,BoundInnerClass!If you wantedChildOfInnerto also get the outer argument passed in to
its__init__, just decorate it withBoundInnerClassinstead ofUnboundInnerClass,
like so:frombigimportBoundInnerClassclassOuter(object):@BoundInnerClassclassInner(object):def__init__(self,outer):self.outer=outer@BoundInnerClassclassChildOfInner(Inner.cls):def__init__(self,outer):super().__init__()assertself.outer==outero=Outer()i=o.ChildOfInner()Again,ChildOfInner.__init__doesn't need to explicitly
pass inouterwhen callingsuper.__init__.You can see more complex examples of using inheritance withBoundInnerClass(andUnboundInnerClass)
in thebigtest suite.Miscellaneous notesIf you refer to a bound inner class directly from the outerclass,rather than using the outerinstance,you get the original class.
This ensures that references toOuter.Innerare consistent; this
class is also a base class of all the bound inner classes. Additionally,
if you attempt to construct an instance of an unboundOuter.Innerclass without referencing it via an instance, you must pass in the
outer parameter by hand--just like you'd have to pass in theselfparameter by hand when calling a method on theclass itselfrather
than on aninstanceof the class.If you refer to a bound inner class from an outer instance,
you get a subclass of the original class.Bound classes are cached in the outer object, which both provides
a small speedup and ensures thatisinstancerelationships are
consistent.You must not rename inner classes decorated with eitherBoundInnerClassorUnboundInnerClass!
The implementation ofBoundInnerClasslooks up
the bound inner class in the outer object by name in several places.
Adding aliases to bound inner classes is harmless, but the original
attribute name must always work.Bound inner classes from different objects are different classes.
This is symmetric with bound methods; if you have two objectsaandbthat are instances of the same class,a.BoundInnerClass != b.BoundInnerClass, just asa.method != b.method.The binding only goes one level deep; if you had an inner classCinside another inner classBinside a classA, the constructor
forCwould be called with theBobject, not theAobject.Similarly, if you have a bound inner classBinside a classA,
and another bound inner classDinside a classC, andDinherits fromB, the constructor forDwill be called with
theBobject but not theAobject. WhenDcallssuper().__init__it'll have to fill in theouterparameter by hand.There's a race condition in the implementation: if you access a
bound inner class through an outer instance from two separate threads,
and the bound inner class was not previously cached, the two threads
may get different (but equivalent) bound inner class objects, and only
one of those instances will get cached on the outer object.  This could
lead to confusion and possibly cause bugs. For example, you could have
two objects that would be considered equal if they were instances of
the same bound inner class, but would not be considered equal if
instantiated by different instances of that same bound inner class.
There's an easy workaround for this problem: access the bound inner
class from the__init__of the outer class, which should allow
the code to cache the bound inner class instance before a second
thread could ever get a reference to the outer object.Release history0.112023/09/19Breaking change: renamed almost all the oldwhitespaceandnewlinestuples.
Worse yet, one symbol has the same name but adifferent value:ascii_whitespace!
I've also changed the suffix_without_dosto the more accurate and intuitive_without_crlf, and similarly changednewlinestolinebreaks.
Sorry for all the confusion.  This resulted from a lot of research into whitespace
and newline characters, in Python, Unicode, and ASCII; please see the new deep-diveWhitespace and line-breaking characters in Python and bigto see what all the fuss is about.  Here's a summary of all the
changes to the whitespace tuples:RENAMED TUPLES (old name -> new name)
    ascii_newlines               -> bytes_linebreaks
    ascii_whitespace             -> bytes_whitespace
    newlines                     -> linebreaks

    ascii_newlines_without_dos   -> bytes_linebreaks_without_crlf
    ascii_whitespace_without_dos -> bytes_whitespace_without_crlf
    newlines_without_dos         -> linebreaks_without_crlf
    whitespace_without_dos       -> whitespace_without_crlf

  REMOVED TUPLES
    utf8_newlines
    utf8_whitespace

    utf8_newlines_without_dos
    utf8_whitespace_without_dos

  UNCHANGED TUPLES (same name, same meaning)
    whitespace

  NEW TUPLES
    ascii_linebreaks
    ascii_whitespace
    str_linebreaks
    str_whitespace
    unicode_linebreaks
    unicode_whitespace

    ascii_linebreaks_without_crlf
    ascii_whitespace_without_crlf
    str_linebreaks_without_crlf
    str_whitespace_without_crlf
    unicode_linebreaks_without_crlf
    unicode_whitespace_without_crlfNew function in thebig.textmodule:encode_strings,which takes a container object containingstrobjects and returns an equivalent object
containing encoded versions of those strings asbytes.Changedsplit_text_with_codeimplementation to useStateManager.
(No API or semantic changes, just an change to the internal implementation.)When you callmultisplitwith a type mismatch
between 's' and 'separators', the exception it raises
now includes the values of 's' and 'separators'.Added more tests forbig.stateto exercise all the string arguments
ofaccessoranddispatch.The exhaustivemultisplittester now lets you
specify test cases as cohesive strings, rather
than forcing you to split the string manually.The exhaustivemultisplittester is better at
internally verifying that it's doing the right
thing.  (There are some internal sanity checks,
and those are more accurate now.)Whoops!  The name of the main class inbig.stateisStateManager.I accidentally wroteStateMachineinstead in the docs... several times.Originally themultisplitparameter 'separators'
was required.  I changed it to optional a while ago,
with a default ofNone.  (If you pass inNoneit usesbig.str_whitespaceorbig.bytes_whitespace,
depending on the type ofs.)  But the documentation
didn't reflect this change until... now.Improved the prose inThemulti-family of string functionsdeep-dive.Hopefully now it does a better job of sellingmultisplitto the reader.The usual smattering of small doc fixes and improvements.My thanks again to Eric V. Smith for his willingness to consider and discuss these
issues.  Eric is now officially a contributor tobig,increasing the project'sbus factorto two.  Thanks, Eric!0.10released 2023/09/04Added the newbig.statemodule, with its excitingStateManagerclass!int_to_wordsnow supports the newordinalkeyword-only parameter, to produceordinalstrings instead ofcardinalstrings.  (The number 1
as acardinalstring is'one', but as anordinalstring is'first').Added thepure_virtualdecorator tobig.builtin.The documentation is now much prettier!  I finally discovered a syntax
I can use to achieve a proper indent in Markdown, supported by both
GitHub and PyPI. You simply nest the text you want indented inside
an HTML description list as the description text, and skip the
description item (<dl><dd>).  Note that you need a blank
line after the<dl><dd>line, or else Markdown will ignore the
markup in the following paragraph.  Thanks to Hugo van Kemenade
for his help confirming this!  Oh, and, Hugo also fixed the image markup
so thebigbanner displays properly on PyPI.  Thanks, Hugo!0.9.2released 2023/07/22Extremely minor release.  No new features or bug fixes.Fixed coverage, now back to the usual 100%.
(This just required changing the tests, whichdidn'tfind any new bugs.)Made the tests forLogdeterministic.  They now use a fake clock
that always returns the same values.Added GitHub Actions integration.  Tests and
coverage are run in the cloud after every checkin.
Thanks toDan Popefor gently walking me through this!Fixed metadata in thepyproject.tomlfile.Added badges for testing, coverage,
and supported Python versions.0.9.1released 2023/06/28Added the newbig.logmodule, with its newLogclass!
I wrote this for another project--but it turned out so nice
I just had to add it tobig!0.9released 2023/06/15Bugfix!  If an outer classOuterhad an inner classInnerdecorated with@BoundInnerClass, andois an instance ofOuter, andoevaluated to false in a boolean context,o.Innerwould be theunboundversion ofInner.  Now
it's the bound version, as is proper.Modifiedtests/test_boundinnerclasses.py:Added regression test for the above bugfix (of course!).It now takes advantage of that newfangled "zero-argumentsuper".Added testing of an unbound subclass of an unbound subclass.0.8.3released 2023/06/11Addedint_to_words.All tests now insert the localbigdirectory
ontosys.path, so you can run the tests on your
local copy without having to install.  Especially
convenient for testing with old versions of Python!Note:tomorrow,bigwill be one year old!0.8.2released 2023/05/19Convert all iterator functions to use my new approach:
instead of checking arguments inside the iterator,
the function you call checks arguments, then has a
nested iterator function which it runs and returns the
result.  This means bad inputs raise their exceptions
at the call site where the iterator is constructed,
rather than when the first value is yielded by the iterator!0.8.1released 2023/05/19Addedparse_delimitersandDelimiter.0.8released 2023/05/18Major retooling ofstrandbytessupport inbig.text.Functions inbig.textnow uniformly acceptstrorbytesor a subclass of either.  See theSupport for bytes and strsection
for how it works.Functions inbig.textare now more consistent about raisingTypeErrorvsValueError.  If you mixbytesandstrobjects together in one call, you'll get aTypeError, but
if you pass in an empty iterable (of a correct type) where
a non-empty iterable is required you'll get aValueError.big.textgenerally tries to give theTypeErrorhigher
priority; if you pass in a value that fails both the type
check and the value check, thebig.textfunction will raiseTypeErrorfirst.Major rewrite ofre_rpartition.I realized it had the same "reverse mode" problem that
I fixed inmultisplitback in version0.6.10: the regular expression should really
search the string in "reverse mode", from right to left.
The difference is whether the regular
expression potentially matches against overlapping strings.
When in forwards mode, the regular expression should prefer
theleftmostoverlapping match, but in reverse mode it
should prefer therightmostoverlapping match.  Most of the
time this produces the same list of matches as you'd
find searching the string forwards--but sometimes the matches come
outverydifferent.
This was way harder to fix withre_rpartitionthan withmultisplit,
because Python'sremodule only supports searching forwards.
I have to emulate reverse-mode searching by manually checking for
overlapping matches and figuring out which one(s) to keep--alotof
work!  Fortunately it's only a minor speed hit if you don't have
overlapping matches.  (And if youdohave overlapping matches,
you're probably just happyre_rpartitionnow produces correct
results--though I did my best to make it performant anyway.)
In the future,bigwill probably add support for the
PyPI packageregex, which reimplements Python'sremodule
but adds many features... including reverse mode!New function:reversed_re_finditer.Behaves almost identically to the Python
standard library functionre.finditer, yielding
non-overlapping matches ofpatterninstring.  The difference
is,reversed_re_finditersearchesstringfrom right to left.
(Written as part of there_rpartitionrewrite mentioned above.)Addedapostrophes,double_quotes,ascii_apostrophes,ascii_double_quotes,utf8_apostrophes, andutf8_double_quotesto thebig.textmodule.  Previously the first
four of these were hard-coded strings insidegently_title.(And the last two didn't exist!)Code cleanup insplit_text_with_code, removed redundant code.
I think it has about the same number ofifstatements; if anything
it might be slightly faster.Retooledre_partitionandre_rpartitionslightly, should now be very-slightly faster.  (Well,re_rpartitionwill be slower if your pattern finds overlapping matches.  But at
least now it's correct!)Lots and lots of doc improvements, as usual.0.7.1released 2023/03/13Tweaked the implementation ofmultisplit.Internally, it does the
string splitting usingre.split, which returns alist.  It used
to iterate over the list and yield each element.  But that meant keeping
the entire list around in memory untilmultisplitexited.  Now,multisplitreverses the list, pops off the final element, and yields
that.  This meansmultisplitdrops all references to the split strings
as it iterates over the string, which may help in low-memory situations.Minor doc fixes.0.7released 2023/03/11Breaking changes to theScheduler:It's no longer thread-safe by default, which means it's much faster
for non-threaded workloads.The lock has been moved out of theSchedulerobject and into theRegulator.  Among other things, this
means that theSchedulerconstructor no longer takes alockargument.Regulatoris now an abstract base class.big.scheduleralso provides two concrete implementations:SingleThreadedRegulatorandThreadSafeRegulator.RegulatorandEventare now defined in thebig.schedulernamespace.  They were
previously defined inside theSchedulerclass.The arguments to theEventconstructor were rearranged.  (You shouldn't care, as you
shouldn't be manually constructingEventobjects anyway.)TheSchedulernow guarantees that it will only callnowandwakeon aRegulatorobject while holding thatRegulator's lock.Minor doc fixes.0.6.18released 2023/03/09Retooledmultisplitandmultistripargument verification code.  Both functions now consistently check all
their inputs, and use consistent error messages when raising an exception.0.6.17released 2023/03/09Fixed a minor crashing bug inmultisplit:
if you passed in alistof separators (orseparatorswas of any non-hashable type), andreversewas true,multisplitwould crash.  It usedseparatorsas a key
into a dict, which meantseparatorshad to be hashable.multisplitnow verifies that thespassed in is eitherstrorbytes.Updated all copyright date notices to 2023.Lots of doc fixes.0.6.16released 2023/02/26Fixed Python 3.6 support! Some equals-signs-in-f-strings and some
other anachronisms had crept in.  0.6.16 has been tested on all
versions from 3.6 to 3.11 (as well as having 100%coverage).Made thedateutilspackage an optional dependency.  Only one function
needs it,parse_timestamp_3339Z().Minor cleanup inPushbackIterator().
It also uses slots now, which should make it a bit faster.0.6.15released 2023/01/07Added the new functionsdatetime_ensure_timezone(d, timezone)anddatetime_set_timezone(d, timezone).
These allow you to ensure or explicitly set a timezone on adatetime.datetimeobject.Added thetimezoneargument toparse_timestamp_3339Z().gently_title()now capitalizes the first letter after a left parenthesis.Changed the secretmultirpartitionfunction slightly.  Itsreverseparameter now means to un-reverse its reversing behavior.  Stated
another way,multipartition(reverse=X)andmultirpartition(reverse=not X)now do the same thing.0.6.14released 2022/12/11Improved the text of theRuntimeErrorraised byTopologicalSorter.Viewwhen the view is incoherent.  Now it tells you exactly what nodes are
conflicting.Expanded the deep dive onmultisplit.0.6.13released 2022/12/11Changedtranslate_filename_to_exfat(s)behavior: when modifying a string with a colon (':')notfollowed by
a space, it used to convert it to a dash ('-').  Now it converts the
colon to a period ('.'), which looks a little more natural.  A colon
followed by a space is still converted to a dash followed by a space.0.6.12tagged 2022/12/04Bugfix: When callingTopologicalSorter.print(),
it sorts the list of nodes, for consistency's sakes and for ease of reading.
But if the node objects don't support<or>comparison,
that throws an exception.TopologicalSorter.print()now catches
that exception and simply skips sorting.  (It's only a presentation thing anyway.)Added a secret (otherwise undocumented!) function:multirpartition,
which is likemultipartitionbut withreverse=True.Added the list of conflicted nodes to the "node is incoherent"
exception text.Note:although version0.6.12was tagged, it was never packaged for release.0.6.11tagged 2022/11/13Changed the import strategy.  The top-levelbigmodule used
to import all its child modules, andimport *all the symbols
from all those modules.  But a friend (hi Mark Shannon!) talked
me out of this.  It's convenient, but if a user doesn't care about
a particular module, why make them import it.  So now the top-levelbigmodule contains nothing but a version number, and you
can either import just the submodules you need, or you can importbig.allto get all the symbols (likebigitself used to do).Note:although version0.6.11was tagged, it was never packaged for release.0.6.10released 2022/10/26All code changes had to do withmultisplit:Fixed a subtle bug.  When splitting with a separator that can overlap
itself, like' x ',multisplitwill prefer theleftmostinstance.
But whenreverse=True, it must prefer therightmostinstance.
Thanks to Eric V. Smith for suggesting the clever "reverse everything,
callre.split, and un-reverse everything" approach.  That let me
fix this bug while still implementing on top ofre.split!ImplementedPROGRESSIVEmode for thestripkeyword.  This behaves
likestr.strip: when splitting, strip on the left, then start splitting.
If we don't exhaustmaxsplit, strip on the right; if wedoexhaustmaxsplit,don'tstrip on the right.  (Similarly forstr.rstripwhenreverse=True.)Changed the default forstriptoFalse.  It used to beNOT_SEPARATE.  But this was too surprising--I'd forget that it
was the default, and turning onkeepwouldn't return everything I
thought I should get, and I'd head off to debugmultisplit, when in
fact it was behaving as specified.  The Principle Of Least Surprise
tells me thatstripdefaulting toFalseis less surprising.
Also, maintaining the invariant that all the keyword-only parameters
tomultisplitdefault toFalseis a helpful mnemonic device in
several ways.RemovedNOT_SEPARATE(and the not-yet-implementedSTR_STRIP)
modes forstrip.  They're easy to implement yourself, and this
removes some surface area from the already-too-bigmultisplitAPI.Modernizedpyproject.tomlmetadata to makeflithappier.  This was
necessary to ensure thatpip install bigalso installs its dependencies.0.6.8released 2022/10/16Renamed two of the three freshly-added lines modifier functions:lines_filter_containsis nowlines_containing,
andlines_filter_grepis nowlines_grep.0.6.7released 2022/10/16Added three new lines modifier functions
to thetextmodule:lines_filter_contains,lines_filter_grep,
andlines_sort.gently_titlenow acceptsstrorbytes.  Also added theapostrophesanddouble_quotesarguments.0.6.6released 2022/10/14Fixed a bug inmultisplit.
I thought when usingkeep=AS_PAIRSthat it shouldn't ever emit a 2-tuple
containing just empty strings--but on further reflection I've realized that
that's correct.  This behavior is now tested and documented, along with
the reasoning behind it.Added thereverseflag tore_partition.whitespace_without_dosandnewlines_without_dosstill had the DOS
end-of-line sequence in them!  Oops!Added a unit test to check that.  The unit test also ensures thatwhitespace,newlines, and all the variants (utf8_,ascii_,
and_with_dos) exactly match the set of characters Python considers
whitespace and newline characters.Lots more documentation and formatting fixes.0.6.5released 2022/10/13Added the newitertoolsmodule, which so far only containsPushbackIterator.Addedlines_strip_commentsandsplit_quoted_stringsto thetextmodule.0.6.1released 2022/10/13I realized thatwhitespaceshould contain the DOS end-of-line
sequence ('\r\n'), as it should be considered a single separator
when splitting etc.  I added that, along withwhitespace_no_dos,
and naturallyutf8_whitespace_no_dosandascii_whitespace_no_dostoo.Minor doc fixes.0.6released 2022/10/13Abigupgrade!Completely retooled and upgradedmultisplit,
and addedmultistripandmultipartition,collectively calledThemulti-family of string functions.(Thanks to Eric Smith for suggestingmultipartition!Well, sort of.)[multisplit](#multisplits-separatorsnone--keepfalse-maxsplit-1-reversefalse-separatefalse-stripfalse)now supports five (!) keyword-only parameters, allowing the caller
to tune its behavior to an amazing degree.Also, the original implementation of[multisplit](#multisplits-separatorsnone--keepfalse-maxsplit-1-reversefalse-separatefalse-stripfalse)got its semantics a bit wrong; it was inconsistent and maybe a little buggy.multistripis likestr.stripbut accepts an iterable of
separator strings.  It can strip from the left, right, both, or
neither (in which case it does nothing).multipartitionis likestr.partition, but accepts an iterable
of separator strings.  It can also partition more than once,
and supportsreverse=Truewhich causes it to partition from the right
(likestr.rpartition).Also added useful predefined lists of separators for use with all
themultifunctions:whitespaceandnewlines,
withascii_andutf8_versions of each, andwithout_dosvariants of
all threenewlinesvariants.Added theSchedulerandHeapclasses.Scheduleris a replacement for Python'ssched.schedulerclass, with a modernized
interface and a major upgrade in functionality.Heapis an object-oriented interface to Python'sheapqmodule, used byScheduler.
These are in their own modules,big.heapandbig.scheduler.Addedlinesand all thelines_modifiers.  These are great for writing little text parsers.
For more information, please see the deep-dive onlinesand lines modifier functions.Removedstripped_linesandrstripped_linesfrom thetextmodule,
as they're superceded by the far superiorlinesfamily.Enhancednormalize_whitespace.
Added theseparatorsandreplacementparameters,
and added support forbytesobjects.Added thecountparameter tore_partitionandre_rpartition.0.5.2released 2022/09/12Addedstripped_linesandrstripped_linesto thetextmodule.Added support forlento theTopologicalSorterobject.0.5.1released 2022/09/04Addedgently_titleandnormalize_whitespaceto thetextmodule.Changedtranslate_filename_to_exfatto handle translating':'in a special way.
If the colon is followed by a space, then the colon is turned into' -'.
This yields a more natural translation when colons are used in text, e.g.'xXx: The Return Of Xander Cage'is translated to'xXx - The Return Of Xander Cage'.
If the colon is not followed by a space, turns the colon into'-'.
This is good for tiresome modern gobbledygook like'Re:code', which
will now be translated to'Re-code'.0.5released 2022/06/12Initial release.