Package: pygolang
Packagegolangprovides Go-like features for Python:gpythonis Python interpreter with support for lightweight threads.gospawns lightweight thread.chanandselectprovide channels with Go semantic.funcallows to define methods separate from class.deferallows to schedule a cleanup from the main control flow.errorand packageerrorsprovide error chaining.banduprovide way to make sure an object is either bytes or unicode.gimportallows to import python modules by full path in a Go workspace.Packagegolang.pyxprovidessimilar features for Cython/nogil.Additional packages and utilities are alsoprovidedto close other gaps
between Python/Cython and Go environments.ContentsGPythonGoroutines and channelsMethodsDefer / recover / panicErrorsStringsImportCython/nogil APIAdditional packages and utilitiesGPython optionsPygolang change historyGPythonCommandgpythonprovides Python interpreter that supports lightweight threads
via tight integration withgevent. The standard library of GPython is API
compatible with Python standard library, but inplace of OS threads lightweight
coroutines are provided, and IO is internally organized vialibuv/libev-based IO scheduler. Consequently programs can spawn lots of
coroutines cheaply, and modules liketime,socket,ssl,subprocessetc -
all could be used from all coroutines simultaneously, and in the same blocking way
as if every coroutine was a full OS thread. This gives ability to scale programs
without changing concurrency model and existing code.Additionally GPython sets UTF-8 to be default encoding always, and putsgo,chan,selectetc into builtin namespace.NoteGPython is optional and the rest of Pygolang can be used from under standard Python too.
However without gevent integrationgospawns full - not lightweight - OS thread.
GPython can be also used with threads - not gevent - runtime. Please seeGPython optionsfor details.Goroutines and channelsgospawns a coroutine, or thread if gevent was not activated. It is possible to
exchange data in between either threads or coroutines via channels.chancreates a new channel with Go semantic - either synchronous or buffered. Usechan.recv,chan.sendandchan.closefor communication.nilchanstands for nil channel.selectcan be used to multiplex on several
channels. For example:ch1 = chan()    # synchronous channel
ch2 = chan(3)   # channel with buffer of size 3

def _():
    ch1.send('a')
    ch2.send('b')
go(_)

ch1.recv()      # will give 'a'
ch2.recv_()     # will give ('b', True)

ch2 = nilchan   # rebind ch2 to nil channel
_, _rx = select(
    ch1.recv,           # 0
    ch1.recv_,          # 1
    (ch1.send, obj),    # 2
    ch2.recv,           # 3
    default,            # 4
)
if _ == 0:
    # _rx is what was received from ch1
    ...
if _ == 1:
    # _rx is (rx, ok) of what was received from ch1
    ...
if _ == 2:
    # we know obj was sent to ch1
    ...
if _ == 3:
    # this case will be never selected because
    # send/recv on nil channel block forever.
    ...
if _ == 4:
    # default case
    ...By defaultchancreates new channel that can carry arbitrary Python objects.
However type of channel elements can be specified viachan(dtype=X)- for
examplechan(dtype=’C.int’)creates new channel whose elements are C
integers.chan.nil(X)creates typed nil channel.Cython/nogil APIexplains how channels with non-Python dtypes, besides in-Python usage, can be
additionally used for interaction in between Python and nogil worlds.Methodsfuncdecorator allows to define methods separate from class.For example:@func(MyClass)
def my_method(self, ...):
    ...will defineMyClass.my_method().funccan be also used on just functions, for example:@func
def my_function(...):
    ...Defer / recover / panicdeferallows to schedule a cleanup to be executed when current function
returns. It is similar totry/finallybut does not force the cleanup part
to be far away in the end. For example:wc = wcfs.join(zurl)    │     wc = wcfs.join(zurl)
defer(wc.close)         │     try:
                        │        ...
...                     │        ...
...                     │        ...
...                     │     finally:
                        │        wc.close()If deferred cleanup fails, previously unhandled exception, if any, won’t be
lost - it will be chained with (PEP 3134) and included into traceback dump
even on Python2.For completeness there isrecoverandpanicthat allow to program with
Go-style error handling, for example:def _():
   r = recover()
   if r is not None:
      print("recovered. error was: %s" % (r,))
defer(_)

...

panic("aaa")Butrecoverandpanicare probably of less utility since they can be
practically natively modelled withtry/except.Ifdeferis used, the function that uses it must be wrapped with@funcdecorator.ErrorsIn concurrent systems operational stack generally differs from execution code
flow, which makes code stack traces significantly less useful to understand an
error. Pygolang provides support for error chaining that gives ability to build
operational error stack and to inspect resulting errors:erroris error type that can be used by itself or subclassed. By
providing.Unwrap()method, an error can optionally wrap another error this
way forming an error chain.errors.Isreports whether an item in error chain
matches target.fmt.Errorfprovides handy way to build wrapping errors.
For example:e1 = error("problem")
e2 = fmt.Errorf("doing something for %s: %w", "joe", e1)
print(e2)         # prints "doing something for joe: problem"
errors.Is(e2, e1) # gives True

# OpError is example class to represents an error of operation op(path).
class OpError(error):
   def __init__(e, op, path, err):
      e.op   = op
      e.path = path
      e.err  = err

   # .Error() should be used to define what error's string is.
   # it is automatically used by error to also provide both .__str__ and .__repr__.
   def Error(e):
      return "%s %s: %s" % (e.op, e.path, e.err)

   # provided .Unwrap() indicates that this error is chained.
   def Unwrap(e):
      return e.err

mye = OpError("read", "file.txt", io.ErrUnexpectedEOF)
print(mye)                          # prints "read file.txt: unexpected EOF"
errors.Is(mye, io.EOF)              # gives False
errors.Is(mye. io.ErrUnexpectedEOF) # gives TrueBoth wrapped and wrapping error can be of arbitrary Python type - not
necessarily oferroror its subclass.erroris also used to represent at Python level an error returned by
Cython/nogil call (seeCython/nogil API) and preserves Cython/nogil error
chain for inspection at Python level.Pygolang error chaining integrates with Python error chaining and takes.__cause__attribute into account for exception created viaraise X from Y(PEP 3134).Stringsbanduprovide way to make sure an object is either bytes or unicode.b(obj)converts str/unicode/bytes obj to UTF-8 encoded bytestring, whileu(obj)converts str/unicode/bytes obj to unicode string. For example:b("привет мир")   # -> gives bytes corresponding to UTF-8 encoding of "привет мир".

def f(s):
   s = u(s)       # make sure s is unicode, decoding as UTF-8(*) if it was bytes.
   ...            # (*) but see below about lack of decode errors.The conversion in both encoding and decoding never fails and never looses
information:b(u(·))andu(b(·))are always identity for bytes and unicode
correspondingly, even if bytes input is not valid UTF-8.Importgimportprovides way to import python modules by full path in a Go workspace.For examplelonet = gimport('lab.nexedi.com/kirr/go123/xnet/lonet')will import eitherlab.nexedi.com/kirr/go123/xnet/lonet.py, orlab.nexedi.com/kirr/go123/xnet/lonet/__init__.pylocated insrc/under$GOPATH.Cython/nogil APICython packagegolangprovidesnogilAPI with goroutines, channels and
other features that mirror corresponding Python package. Cython API is not only
faster compared to Python version, but also, due tonogilproperty, allows to
build concurrent systems without limitations imposed by Python’s GIL. All that
while still programming in Python-like language. Brief description of
Cython/nogil API follows:gospawns new task - a coroutine, or thread, depending on activated runtime.chan[T]represents a channel with Go semantic and elements of typeT.
Usemakechan[T]to create new channel, andchan[T].recv,chan[T].send,chan[T].closefor communication.nilstands for nil channel.selectcan be used to multiplex on several channels. For example:cdef nogil:
   struct Point:
      int x
      int y

   void worker(chan[int] chi, chan[Point] chp):
      chi.send(1)

      cdef Point p
      p.x = 3
      p.y = 4
      chp.send(p)

   void myfunc():
      cdef chan[int]   chi = makechan[int]()       # synchronous channel of integers
      cdef chan[Point] chp = makechan[Point](3)    # channel with buffer of size 3 and Point elements

      go(worker, chi, chp)

      i = chi.recv()    # will give 1
      p = chp.recv()    # will give Point(3,4)

      chp = nil         # rebind chp to nil channel
      cdef cbool ok
      cdef int j = 33
      _ = select([
          chi.recvs(&i),         # 0
          chi.recvs(&i, &ok),    # 1
          chi.sends(&j),         # 2
          chp.recvs(&p),         # 3
          default,               # 4
      ])
      if _ == 0:
          # i is what was received from chi
          ...
      if _ == 1:
          # (i, ok) is what was received from chi
          ...
      if _ == 2:
          # we know j was sent to chi
          ...
      if _ == 3:
          # this case will be never selected because
          # send/recv on nil channel block forever.
          ...
      if _ == 4:
          # default case
          ...Python channels are represented bypychancdef class. Python
channels that carry non-Python elements (pychan.dtype != DTYPE_PYOBJECT) can
be converted to Cython/nogilchan[T]viapychan.chan_*().
Similarly Cython/nogilchan[T]can be wrapped intopychanviapychan.from_chan_*(). This provides interaction mechanism
in betweennogiland Python worlds. For example:def myfunc(pychan pych):
   if pych.dtype != DTYPE_INT:
      raise TypeError("expected chan[int]")

   cdef chan[int] ch = pych.chan_int()  # pychan -> chan[int]
   with nogil:
      # use ch in nogil code. Both Python and nogil parts can
      # send/receive on the channel simultaneously.
      ...

def mytick(): # -> pychan
   cdef chan[int] ch
   with nogil:
      # create a channel that is connected to some nogil task of the program
      ch = ...

   # wrap the channel into pychan. Both Python and nogil parts can
   # send/receive on the channel simultaneously.
   cdef pychan pych = pychan.from_chan_int(ch)  # pychan <- chan[int]
   return pycherroris the interface that represents errors.errors.Newandfmt.errorfprovide way to build errors from text. An error can optionally wrap another
error by implementingerrorWrapperinterface and providing.Unwrap()method.errors.Isreports whether an item in error chain matches target.fmt.errorfwith%wspecifier provide handy way to build wrapping errors. For example:e1 = errors.New("problem")
e2 = fmt.errorf("doing something for %s: %w", "joe", e1)
e2.Error()        # gives "doing something for joe: problem"
errors.Is(e2, e1) # gives TrueAnerrorcan be exposed to Python viapyerrorcdef class wrapper
instantiated bypyerror.from_error().pyerrorpreserves Cython/nogil error
chain for inspection by Python-levelerror.Is.panicstops normal execution of current goroutine by throwing a C-level
exception. On Python/C boundaries C-level exceptions have to be converted to
Python-level exceptions withtopyexc. For example:cdef void _do_something() nogil:
   ...
   panic("bug")   # hit a bug

# do_something is called by Python code - it is thus on Python/C boundary
cdef void do_something() nogil except +topyexc:
   _do_something()

def pydo_something():
   with nogil:
      do_something()Seelibgolang.handgolang.pxdfor details of the API.
See alsotestprog/golang_pyx_user/for demo project that uses Pygolang in
Cython/nogil mode.Additional packages and utilitiesThe following additional packages and utilities are also provided to close gaps
between Python/Cython and Go environments:ConcurrencyString conversionBenchmarking and testingConcurrencyIn addition togoand channels, the following packages are provided to help
handle concurrency in structured ways:golang.context(py,pyx) provides contexts to propagate deadlines, cancellation and
task-scoped values among spawned goroutines[*].golang.sync(py,pyx) providessync.WorkGroupto spawn group of goroutines working
on a common task. It also provides low-level primitives - for examplesync.Once,sync.WaitGroup,sync.Mutexandsync.RWMutex- that are
sometimes useful too.golang.time(py,pyx) provides timers integrated with channels.golang.os.signal(py,pyx) provides signal handling via channels.[*]SeeGo Concurrency Patterns: Contextfor overview.String conversionqq(import fromgolang.gcompat) provides%qfunctionality that quotes as
Go would do. For example the following code will print name quoted in“without escaping printable UTF-8 characters:print('hello %s' % qq(name))qqaccepts bothstrandbytes(unicodeandstron Python2)
and also any other type that can be converted tostr.Packagegolang.strconvprovides direct access to conversion routines, for
examplestrconv.quoteandstrconv.unquote.Benchmarking and testingpy.benchallows to benchmark python code similarly togo test -benchandpy.test.
For example, runningpy.benchon the following code:def bench_add(b):
    x, y = 1, 2
    for i in xrange(b.N):
        x + ygives something like:$ py.bench --count=3 x.py
...
pymod: bench_add.py
Benchmarkadd    50000000        0.020 µs/op
Benchmarkadd    50000000        0.020 µs/op
Benchmarkadd    50000000        0.020 µs/opPackagegolang.testingprovides corresponding runtime bits, e.g.testing.B.py.benchproduces output inGo benchmark format, and so benchmark results
can be analyzed and compared with standard Go tools, for example withbenchstat.
Additionally packagegolang.x.perf.benchlibcan be used to load and process
such benchmarking data in Python.GPython optionsGPython mimics and supports most of Python command-line options, likegpython
-c <commands>to run Python statements from command line, orgpython -m
<module>to execute a module. Such options have the same meaning as in
standard Python and are not documented here.GPython-specific options and environment variables are listed below:-X gpython.runtime=(gevent|threads)Specify which runtime GPython should use.geventprovides lightweight
coroutines, while withthreadsgospawns full OS thread.geventis
default. The runtime to use can be also specified via$GPYTHON_RUNTIMEenvironment variable.Pygolang change history0.1 (2022-01-26)Addos.signalpackage that provides signal handing via nogil channels. This
allows to build concurrent systems without limitation of Python’s standardsignalmodule: signal delivery is not delayed, potentially indefinitely, if
main thread is blocked or is busy doing any non-Python work (commit 1,2,3,example usage).Add C++ API for IO. (commit 1,2,3,4,5,6).Fix segmentation-fault crashes on unhandled panic with gevent backend (commit,greenlet bug).Fixprint(qq(·))crash on Python2 (commit).0.0.9 (2021-12-08)Fix deadlock when new context is created from already-canceled parent (commit 1,2).Add support for“with”statement insync.WorkGroup.
This is sometimes handy and is referred to as“structured concurrency”in Python world (commit,discussion).Fixstrconv.unqouteto handle all input that Gostrconv.Qoutemight produce (commit).More fixes forgpythonto be compatible with CPython in how it handles
program on stdin, interactive session and__main__module setup (commit 1,2,3,4,5).0.0.8 (2020-12-02)Add support for SlapOS (commit 1,2,3,4).Add way to run tests underNexedi testing infrastructure(commit).Fixgpythoncrash when invoked via relative path as e.g../bin/gpython(commit).More fixes forgpythonto be compatible with CPython on command line
handling (commit 1,2,3,4,5,6,7,8).0.0.7 (2020-09-22)Add way to rungpythonwith either gevent or threads runtime. This allowsgpythonusage without forcing projects to switch from threads to greenlets
(commit 1,2,3).Fixgpythonto be more compatible with CPython on command line handling
(commit 1,2,3,4,5,6,7).Teachqqto be usable with bothbytesandstrformat whatever typeqq’s argument is (commit 1,2).Teachrecoverto always return exception with.__traceback__set even on
Python2 (commit).Fixpyx.buildfor develop install (commit).Fixpyx.buildon macOS (commit).Add tests for IPython and Pytest integration patches (commit 1,2,3,4,5,6,7).Add support for Python38 (commit 1,2).Fix ThreadSanitizer/AddressSanitizer support on upcoming Debian 11 (commit).By this release Pygolang was included intoNexedi Software Stack.0.0.6 (2020-02-28)Provide support for error chaining. In concurrent systems
operational stack generally differs from execution code flow, which makes
code stack traces significantly less useful to understand an error.
Error chaining gives ability to build operational
error stack and to inspect resulting errors.
(commit 1,2,3,4,5,6,overview 1,overview 2).Provideunicode↔bytesconversion:b(obj)converts str/unicode/bytes obj to UTF-8 encoded bytestring, whileu(obj)converts str/unicode/bytes obj to unicode string. The conversion in
both encoding and decoding never fails and never looses information:b(u(·))andu(b(·))are always identity for bytes and unicode
correspondingly, even if bytes input is not valid UTF-8.
(commit 1,2,3,4,5,6,7).Providesync.RWMutex(commit 1,2).Providenilas alias fornullptrand NULL (commit 1,2,3,4).Addiopackage withio.EOFandio.ErrUnexpectedEOF(commit).Correctcxx.dictAPI to follow libgolang comma-ok style (commit).Providepyx.build.DSOfor projects to build dynamic libraries that
use/link-to libgolang (commit 1,2).Fixpyx.build.build_extto allow customization (commit).This release is driven bywendelin.corev2 needs.0.0.5 (2019-11-27)Add support for typed Python channels. For
examplechan(dtype=’C.int’)creates channel whose elements type is Cintinstead of Python object. Besides providing runtime type-safety, this allows
to build interaction in between Python and nogil worlds (commit 1,2,3,4,5,6,7,8,9,10,11).Provide automatic memory management for C++/Cython/nogil classes.
Used approach complements“Automatic multithreaded-safe memory managed
classes in Cython”(Gwenaël Samain et al. 2019,blog post) (commit 1,2,3,4,5,6,7).Provide minimal support for interfaces with empty anderrorinterfaces
provided by base library (commit 1,2).Providesync.Mutexandsync.Semaas part of both Python and Cython/nogil
API (commit 1,2,3,4,5,6).Provide C++/Cython/nogil API fortimepackage. Python-leveltimebecomes a
small wrapper around Cython/nogil one (commit 1,2,3,4,5,6,7,8,9,10,11,12,13,14).Provide C++/Cython/nogil API forcontextpackage. Python-levelcontextbecomes a small wrapper around Cython/nogil one (commit 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15).Provide C++/Cython/nogil API forsyncpackage. Python-levelsyncbecomes a
small wrapper around Cython/nogil one (commit 1,2,3,4,5,6,7,8,9).Adderrorspackage witherrors.Newto create new error with provided text (commit).Addfmtpackage withfmt.sprintfandfmt.errorfto format text into
strings and errors (commit).Addstringspackage with utilities likestrings.has_prefix,strings.splitand similar (commit).Addcxxpackage withcxx.dictandcxx.setproviding ergonomic interface
over STL hash map and set (commit).Teachdeferto chain exceptions (PEP 3134) and adjust traceback dumps to
include exception cause/context even on Python2 (commit 1,2,3,4,5).Providedeferas part of C++ API too (commit 1,2,3).Providebuild_extas part ofpyx.buildpackage API. This allows projects
to customize the way their Pygolang-based extensions are built (commit 1,2).Fixrecoverto clean current exception (commit 1,2).Fixselectto not leak object reference on error path (commit).Fix gevent runtime to preserve Python exception state during runtime calls
(commit 1,2).This release is driven bywendelin.corev2 needs.This release is dedicated to the memory ofБася.0.0.4 (2019-09-17)Add ThreadSanitizer, AddressSanitizer and Python debug builds to testing coverage (commit).Fix race bugs inclose,recvandselect(commit 1,2,3,4,5,6).
A 25-years old race condition in Python was also discovered while doing
quality assurance on concurrency (commit 7,Python bug,PyPy bug).If C-level panic causes termination, its argument is now printed (commit).0.0.3 (2019-08-29)Provide Cython/nogil API with goroutines and channels. Cython API is not only
faster compared to Python version, but also, due tonogilproperty, allows to
build concurrent systems without limitations imposed by Python’s GIL.
This work was motivated bywendelin.corev2, which, due to its design,
would deadlock if it tries to take the GIL in its pinner thread.
Implementation of Python-level goroutines and channels becomes tiny wrapper
around Cython/nogil API. This brings in ~5x speedup to Python-levelgolangpackage along the way (commit 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27).Provide way to install Pygolang with extra requirements in the form ofpygolang[<package>]. For examplepygolang[x.perf.benchlib]additionally
selects NumPy,pygolang[pyx.build]- everything needed by build system, andpygolang[all]selects everything (commit).Improve tests to exercise the implementation more thoroughly in many
places (commit 1,2,3,4,5,6).Fix race bugs in buffered channel send and receive (commit 1,2).Fix deadlock insync.WorkGrouptests (commit).Fix@func(cls) def namenot to overridenamein calling context (commit).Fixsync.WorkGroupto propagate all exception types, not only those derived
fromException(commit).Replacethreading.Eventwithchaninsync.WorkGroupimplementation.
This removes reliance on outside semaphore+waitlist code and speeds upsync.WorkGroupalong the way (commit).Speedupsync.WorkGroupby not using@funcat runtime (commit).Add benchmarks forchan,select,@funcanddefer(commit).This release is dedicated to the memory ofВера Павловна Супрун.0.0.2 (2019-05-16)Addtimepackage withtime.Timerandtime.Ticker(commit 1,2,3).Add support for deadlines and timeouts tocontextpackage (commit 1,2,3,4).0.0.1 (2019-05-09)Add support for nil channels (commit).Addcontextpackage to propagate cancellation and task-scoped values among
spawned goroutines (commit,overview).Addsyncpackage withsync.WorkGroupto spawn group of goroutines working
on a common task (commit 1,2).Remove deprecated@method(commit).0.0.0.dev8 (2019-03-24)Fixgpythonto properly initializesys.path(commit).Fix channel tests to pass irregardless of surround OS load (commit).Deprecate@method(cls)in favour of@func(cls)(commit).Support bothPyPy2andPyPy3(commit 1,2,3).0.0.0.dev7 (2019-01-16)Providegpythoninterpreter, that sets UTF-8 as default encoding, integrates
gevent and putsgo,chan,selectetc into builtin namespace (commit).0.0.0.dev6 (2018-12-13)Addstrconvpackage withquoteandunquote(commit 1,2).SupportPyPyas well (commit).0.0.0.dev5 (2018-10-30)Fixselectbug that was causing several cases to be potentially executed
at the same time (commit 1,2,3).Adddeferandrecover(commit).
The implementation is partly inspired by work of Denis Kolodin (1,2).Fix@methodon Python3 (commit).A leaked goroutine no longer prevents whole program to exit (commit 1,2).0.0.0.dev4 (2018-07-04)Addpy.benchprogram andgolang.testingpackage with corresponding bits (commit).0.0.0.dev3 (2018-07-02)Support both Python2 and Python3;qqnow does not escape printable UTF-8
characters. (commit 1,2,3).golang/x/perf/benchlib:New module to load & work with data in Go benchmark
format (commit).0.0.0.dev2 (2018-06-20)Turn into full pygolang:go,chan,select,methodandgcompat.qqare provided in addition togimport(commit). The implementation is
not very fast, but should be working correctly includingselect-selectsends for synchronous channels.0.0.0.dev1 (2018-05-21)Initial release;gimportfunctionality only (commit).