Package: tblib
Serialization library for Exceptions and Tracebacks.Free software: BSD licenseIt allows you to:Pickletracebacks and raise exceptions
with pickled tracebacks in different processes. This allows better error handling when running
code over multiple processes (imagine multiprocessing, billiard, futures, celery etc).Create traceback objects from strings (thefrom_stringmethod).No pickling is used.Serialize tracebacks to/from plain dicts (thefrom_dictandto_dictmethods).No pickling is used.Raise the tracebacks created from the aforementioned sources.Pickle an Exception together with its traceback and exception chain
(raise ... from ...)(Python 3 only)Again, note that using the pickle support is completely optional. You are solely responsible for
security problems should you decide to use the pickle support.Installationpip install tblibDocumentationPickling tracebacksUnpickling tracebacksRaisingPickling Exceptions together with their traceback and chain (Python 3 only)What if we have a local stack, does it show correctly ?It also supports more contrived scenariosReferencetblib.Tracebacktblib.Traceback.to_dicttblib.Traceback.from_dicttblib.Traceback.from_stringtblib.decorators.return_errorWhat if we have a local call stack ?Other weird stuffPickling tracebacksNote: The traceback objects that come out are stripped of some attributes (like variables). But you’ll be able to raise exceptions with
those tracebacks or print them - that should cover 99% of the usecases.>>> from tblib import pickling_support
>>> pickling_support.install()
>>> import pickle, sys
>>> def inner_0():
...     raise Exception('fail')
...
>>> def inner_1():
...     inner_0()
...
>>> def inner_2():
...     inner_1()
...
>>> try:
...     inner_2()
... except:
...     s1 = pickle.dumps(sys.exc_info())
...
>>> len(s1) > 1
True
>>> try:
...     inner_2()
... except:
...     s2 = pickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s2) > 1
True

>>> try:
...     import cPickle
... except ImportError:
...     import pickle as cPickle
>>> try:
...     inner_2()
... except:
...     s3 = cPickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s3) > 1
TrueUnpickling tracebacks>>> pickle.loads(s1)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s2)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s3)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)Raising>>> from six import reraise
>>> reraise(*pickle.loads(s1))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s2))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s3))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: failPickling Exceptions together with their traceback and chain (Python 3 only)>>> try:  # doctest: +SKIP
...     try:
...         1 / 0
...     except Exception as e:
...         raise Exception("foo") from e
... except Exception as e:
...     s = pickle.dumps(e)
>>> raise pickle.loads(s)  # doctest: +SKIP
Traceback (most recent call last):
  File "<doctest README.rst[16]>", line 3, in <module>
    1 / 0
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<doctest README.rst[17]>", line 1, in <module>
    raise pickle.loads(s)
  File "<doctest README.rst[16]>", line 5, in <module>
    raise Exception("foo") from e
Exception: fooBaseException subclasses defined after callingpickling_support.install()willnotretain their traceback and exception chain pickling.
To cover custom Exceptions, there are three options:Use@pickling_support.installas a decorator for each custom Exception>>>fromtblibimportpickling_support>>># Declare all imports of your package's dependencies>>>importnumpy# doctest: +SKIP>>>pickling_support.install()# install for all modules imported so far>>>@pickling_support.install...classCustomError(Exception):...passEventual subclasses ofCustomErrorwill need to be decorated again.Invokepickling_support.install()after all modules have been imported and all
Exception subclasses have been declared>>># Declare all imports of your package's dependencies>>>importnumpy# doctest: +SKIP>>>fromtblibimportpickling_support>>># Declare your own custom Exceptions>>>classCustomError(Exception):...pass>>># Finally, install tblib>>>pickling_support.install()Selectively install tblib for Exception instances just before they are pickledpickling_support.install(<Exceptioninstance>,[Exceptioninstance],...)The above will install tblib pickling for all listed exceptions as well as any other
exceptions in their exception chains.For example, one could write a wrapper to be used withProcessPoolExecutor,Dask.distributed, or similar libraries:>>> from tblib import pickling_support
>>> def wrapper(func, *args, **kwargs):
...     try:
...         return func(*args, **kwargs)
...     except Exception as e:
...         pickling_support.install(e)
...         raiseWhat if we have a local stack, does it show correctly ?Yes it does:>>> exc_info = pickle.loads(s3)
>>> def local_0():
...     reraise(*exc_info)
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> local_2()
Traceback (most recent call last):
  File "...doctest.py", line ..., in __run
    compileflags, 1) in test.globs
  File "<doctest README.rst[24]>", line 1, in <module>
    local_2()
  File "<doctest README.rst[23]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[22]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[21]>", line 2, in local_0
    reraise(*exc_info)
  File "<doctest README.rst[11]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: failIt also supports more contrived scenariosLike tracebacks with syntax errors:>>> from tblib import Traceback
>>> from examples import bad_syntax
>>> try:
...     bad_syntax()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[58]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[57]>", line 2, in <module>
    bad_syntax()
  File "...tests...examples.py", line 18, in bad_syntax
    import badsyntax
  File "...tests...badsyntax.py", line 5
    is very bad
     ^
SyntaxError: invalid syntaxOr other import failures:>>> from examples import bad_module
>>> try:
...     bad_module()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[61]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[60]>", line 2, in <module>
    bad_module()
  File "...tests...examples.py", line 23, in bad_module
    import badmodule
  File "...tests...badmodule.py", line 3, in <module>
    raise Exception("boom!")
Exception: boom!Or a traceback that’s caused by exceeding the recursion limit (here we’re
forcing the type and value to have consistency across platforms):>>> def f(): f()
>>> try:
...    f()
... except RuntimeError:
...    et, ev, tb = sys.exc_info()
...    tb = Traceback(tb)
...
>>> reraise(RuntimeError, RuntimeError("maximum recursion depth exceeded"), tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  ...
RuntimeError: maximum recursion depth exceededReferencetblib.TracebackIt is used by thepickling_support. You can use it too if you want more flexibility:>>> from tblib import Traceback
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: failtblib.Traceback.to_dictYou can use theto_dictmethod and thefrom_dictclassmethod to
convert a Traceback into and from a dictionary serializable by the stdlib
json.JSONDecoder:>>> import json
>>> from pprint import pprint
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...     tb_dict = tb.to_dict()
...     pprint(tb_dict)
{'tb_frame': {'f_code': {'co_filename': '<doctest README.rst[...]>',
                         'co_name': '<module>'},
              'f_globals': {'__name__': '__main__'},
              'f_lineno': 5},
 'tb_lineno': 2,
 'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                     'co_name': 'inner_2'},
                          'f_globals': {'__name__': '__main__'},
                          'f_lineno': 2},
             'tb_lineno': 2,
             'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                 'co_name': 'inner_1'},
                                      'f_globals': {'__name__': '__main__'},
                                      'f_lineno': 2},
                         'tb_lineno': 2,
                         'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                             'co_name': 'inner_0'},
                                                  'f_globals': {'__name__': '__main__'},
                                                  'f_lineno': 2},
                                     'tb_lineno': 2,
                                     'tb_next': None}}}}tblib.Traceback.from_dictBuilding on the previous example:>>> tb_json = json.dumps(tb_dict)
>>> tb = Traceback.from_dict(json.loads(tb_json))
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: failtblib.Traceback.from_string>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: failIf you use thestrict=Falseoption then parsing is a bit more lax:>>> tb = Traceback.from_string("""
... File "bogus.py", line 123, in bogus
... Traceback (most recent call last):
...  File "tests/examples.py", line 2, in func_a
...   func_b()
...    File "tests/examples.py", line 6, in func_b
...     func_c()
...    File "tests/examples.py", line 10, in func_c
...   func_d()
...  File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """, strict=False)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "bogus.py", line 123, in bogus
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: failtblib.decorators.return_error>>> from tblib.decorators import return_error
>>> inner_2r = return_error(inner_2)
>>> e = inner_2r()
>>> e
<tblib.decorators.Error object at ...>
>>> e.reraise()
Traceback (most recent call last):
  ...
  File "<doctest README.rst[26]>", line 1, in <module>
    e.reraise()
  File "...tblib...decorators.py", line 19, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 25, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: failHow’s this useful? Imagine you’re using multiprocessing like this:# Note that Python 3.4 and later will show the remote traceback (but as a string sadly) so we skip testing this.
>>> import traceback
>>> from multiprocessing import Pool
>>> from examples import func_a
>>> pool = Pool()  # doctest: +SKIP
>>> try:  # doctest: +SKIP
...     for i in pool.map(func_a, range(5)):
...         print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    for i in pool.map(func_a, range(5)):
  File "...multiprocessing...pool.py", line ..., in map
    ...
  File "...multiprocessing...pool.py", line ..., in get
    ...
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()  # doctest: +SKIPNot very useful is it? Let’s sort this out:>>> from tblib.decorators import apply_with_return_error, Error
>>> from itertools import repeat
>>> pool = Pool()
>>> try:
...     for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...         if isinstance(i, Error):
...             i.reraise()
...         else:
...             print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 4, in <module>
    i.reraise()
  File "...tblib...decorators.py", line ..., in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line ..., in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line ..., in apply_with_return_error
    return args[0](*args[1:])
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()Much better !What if we have a local call stack ?>>> def local_0():
...     pool = Pool()
...     try:
...         for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...             if isinstance(i, Error):
...                 i.reraise()
...             else:
...                 print(i)
...     finally:
...         pool.close()
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> try:
...     local_2()
... except:
...     print(traceback.format_exc())
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    local_2()
  File "<doctest README.rst[...]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[...]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[...]>", line 6, in local_0
    i.reraise()
  File "...tblib...decorators.py", line 20, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 27, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line 47, in apply_with_return_error
    return args[0](*args[1:])
  File "...tests...examples.py", line 2, in func_a
    func_b()
  File "...tests...examples.py", line 6, in func_b
    func_c()
  File "...tests...examples.py", line 10, in func_c
    func_d()
  File "...tests...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>Other weird stuffClearing traceback works (Python 3.4 and up):>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> import traceback, sys
>>> if sys.version_info > (3, 4):
...     traceback.clear_frames(tb)Creditsmitsuhiko/jinja2for figuring a way to create traceback objects.Changelog3.0.0 (2023-10-22)Added support for__context__,__suppress_context__and__notes__.
Contributed by Tim Maxwell in#72.Added theget_localsargument totblib.pickling_support.install(),tblib.Tracebackandtblib.Frame.
Fixes#41.Dropped support for now-EOL Python 3.7 and added 3.12 in the test grid.2.0.0 (2023-06-22)Removed support for legacy Pythons (2.7 and 3.6) and added Python 3.11 in the test grid.Some cleanups and refactors (mostly from ruff).1.7.0 (2020-07-24)Add more attributes toFrameandCodeobjects for pytest compatibility. Contributed by Ivanq in#58.1.6.0 (2019-12-07)When pickling an Exception, also pickle its traceback and the Exception chain
(raise ... from ...). Contributed by Guido Imperiale in#53.1.5.0 (2019-10-23)Added support for Python 3.8. Contributed by Victor Stinner in#42.Removed support for end of life Python 3.4.Few CI improvements and fixes.1.4.0 (2019-05-02)Removed support for end of life Python 3.3.Fixed tests for Python 3.7. Contributed by Elliott Sales de Andrade in#36.Fixed compatibility issue with Twised (twisted.python.failure.Failureexpected aco_codeattribute).1.3.2 (2017-04-09)Add support for PyPy3.5-5.7.1-beta. PreviouslyAttributeError: 'Frame' object has no attribute 'clear'could be raised. See PyPy
issue#2532.1.3.1 (2017-03-27)Fixed handling for tracebacks due to exceeding the recursion limit.
Fixes#15.1.3.0 (2016-03-08)AddedTraceback.from_string.1.2.0 (2015-12-18)Fixed handling for tracebacks from generators and other internal improvements
and optimizations. Contributed by DRayX in#10and#11.1.1.0 (2015-07-27)Added support for Python 2.6. Contributed by Arcadiy Ivanov in#8.1.0.0 (2015-03-30)Addedto_dictmethod andfrom_dictclassmethod on Tracebacks.
Contributed by beckjake in#5.