Package: ringbuf
ringbufA lock-free, single-producer, single-consumer, ring buffer for Python and Cython.InstallationOS X:brew install boostUbuntu:apt-get install libboost-all-devWindows: Install the latest version ofBoostthen set theBOOST_ROOTenvironment variable to point to its folder.Then:pipinstallringbufMotivationWhen working with realtime DSP in Python, we might be wrapping some external C/C++ library (for instance, PortAudio) which runs some user-provided callback function in realtime. The callback function shouldn't allocate/deallocate memory, shouldn't contain any critical sections (mutexes), and so forth, to prevent priority inversion. If the callback were to contain Python objects, we'd likely be allocating and deallocating, and at the very least, acquiring and releasing the GIL. So, the callback cannot interact with Python objects if we expect realtime performance. As such, there's a need for buffering data in a non-locking way between a C/C++ callback and Python.Enter ringbuf, Cython wrappers forboost::lockfree::spsc_queue. Our Python code can read from and write to aringbuf.RingBufferobject, and our C++ code can read from and write to that buffer's underlyingspsc_queue, no GIL required.UsageAny Python object which supports thebuffer protocolcan be stored inringbuf.RingBuffer. This includes, but is not limited to:bytes,bytearray,array.array, andnumpy.ndarray.NumPyimportnumpyasnpfromringbufimportRingBufferbuffer=RingBuffer(format='f',capacity=100)data=np.linspace(-1,1,num=100,dtype='f')buffer.push(data)popped=buffer.pop(100)assertnp.array_equal(data,popped)bytesfromringbufimportRingBufferbuffer=RingBuffer(format='B',capacity=11)buffer.push(b'hello world')popped=buffer.pop(11)assertbytes(popped)==b'hello world'Interfacing with C/C++mymodule.pxd:# distutils: language = c++cdefvoidcallback(void*q)mymodule.pyx:# distutils: language = c++fromarrayimportarrayfromringbuf.boostcimportspsc_queue,void_ptr_to_spsc_queue_char_ptrfromringbuf.ringbufcycimportRingBufferfromsome_c_librarycimportsome_c_functioncdefvoidcallback(void*q):cdef:# Cast the void* back to an spsc_queue.# The underlying queue always holds chars.spsc_queue[char]*queue=void_ptr_to_spsc_queue_char_ptr(q)double[5]to_push=[1.0,2.0,3.0,4.0,5.0]# Since the queue holds chars, you'll have to cast and adjust size accordingly.queue.push(<char*>to_push,sizeof(double)*5)defdo_stuff():cdef:RingBufferbuffer=RingBuffer(format='d',capacity=100)void*queue=buffer.queue_void_ptr()# Pass our callback and a void pointer to the buffer's queue to some third party library.# Presumably, the C library schedules the callback and passes it the queue's void pointer.some_c_function(callback,queue)sleep(1)assertarray.array('d',buffer.pop(5))==array.array('d',range(1,6))Handling overflow & underflowWhenRingBuffer.push()overflows, it returns the data that couldn't be pushed (or None, if all was pushed):fromringbufimportRingBufferbuffer=RingBuffer(format='B',capacity=10)overflowed=buffer.push(b'spam eggs ham')assertoverflowed==b'ham'WhenRingBuffer.pop()underflows, it returns whatever data could be popped:fromringbufimportRingBufferbuffer=RingBuffer(format='B',capacity=13)buffer.push(b'spam eggs ham')popped=buffer.pop(buffer.capacity*100)assertbytes(popped)==b'spam eggs ham'For additional usage see thetests.Supported platformsGitHub Actions tests the following matrix:Linux:CPython 3.7CPython 3.8CPython 3.9CPython 3.10macOS:CPython 3.10Windows:CPython 3.10Any platform with a C++11 compiler and boost installed should work.ContributingPull requests are welcome, please file any issues you encounter.The code is linted withlintball. There is a pre-commit hook to lint, configured by running:npminstall-glintball
gitconfig--localcore.hooksPath.githooksChangelogv2.6.0 2022-09-27Move CI to GitHub Actions.Lint codebase with lintballImprove project structurev2.5.0 2020-04-17Added experimental support for Windows.v2.4.0 2020-03-23AddedRingBuffer.reset()method to clear the buffer.v2.3.0 2020-03-22Addedconcatenatefunction for joining multiple arbitrary Python objects that support the buffer protocol.