Package: concurrentbuffer
Concurrent BufferThe available number of central processing units (CPUs) in home and server computers can help to parallelize and speed up specific tasks. The python programming language comes with amultiprocessing packagethat allows running tasks on multiple CPUs. In addition, python 3.8 introducedshared memory, allowing for sharing data between processes. A potential use case for this shared memory is to compensate for possible speed variance when transferring data via a buffer.This package aims to be a general solution for a concurrent buffer, .i.e., a buffer filled with data using parallel running 'producer' processes.
A commander process controls the produced data via user-defined dictionary messages. At the same time, the main process can consume the data using a BufferIterator in a fast and user-friendly way.Please see below theinstallation instructionsand anexampleon how to use this package as well as how tocreate your own commanderand how tocreate your own producer. For more information please see thedocs. Feel free to open an issue if you have any questions or remarks.Installation and DependenciesThis package requirespython>=3.8andnumpyA binary installer for the latest version is available at thePython Package Index (PyPI)pipinstallconcurrentbufferExample usage:Important note:'spawn' multiprocessing context will not work in a jupyter notebook/lab, use fork instead when working in a jupyter notebook / jupyter labEasy Usage:fromconcurrentbuffer.iteratorimportbuffer_iterator_factory# the number of cpus/producerscpus=8# the buffershape in the shared memorybuffer_shapes=((64,256,256,3),)# the context of multiprocess (spawn or fork)context='spawn'# if the messages from the commander and the produced data are first in first outdeterministic=True# You will have to create your own Commander class, please see instructions below# a user defined commander, should subclass the Commander classcommander=IndexCommander(max_index=10)# You will have to create your own Producer class, please see instructions below# a user defined producer, should subclass the Producer classproducer=DataProducer(data_shapes=buffer_shapes)# create a buffer iteratorbuffer_iterator=buffer_iterator_factory(cpus=cpus,buffer_shapes=buffer_shapes,commander=commander,producer=producer,context=context,deterministic=deterministic,)# loop through the buffer that is filled concurrentlyforindexinrange(10):data=next(buffer_iterator)# always stop the iterator to close all processes and free the shared memorybuffer_iterator.stop()Advanced Usage:frommultiprocessing.contextimportForkContext,SpawnContextfromconcurrentbuffer.factoryimportBufferFactoryfromconcurrentbuffer.infoimportBufferInfofromconcurrentbuffer.iteratorimportBufferIteratorfromconcurrentbuffer.stateimportBufferStatefromconcurrentbuffer.systemimportBufferSystem# the number of cpus/producerscpus=8# the buffershape in the shared memorybuffer_shapes=((64,256,256,3),)# the context of multiprocess (spawn or fork)context=SpawnContext()# if the messages from the commander and the produced data are first in first outdeterministic=True# the number of buffers each with shape of buffer_shapecount=cpus*len(BufferState)# buffer system contains the information of the systembuffer_system=BufferSystem(cpus=cpus,context=context,deterministic=deterministic)# buffer info contains the information of the buffersbuffer_info=BufferInfo(count=count,shapes=buffer_shapes)# You will have to create your own Commander class, please see instructions below# a user defined commander, should subclass the Commander classcommander=IndexCommander(max_index=10)# You will have to create your own Producer class, please see instructions below# a user defined producer, should subclass the Producer classproducer=DataProducer(data_shapes=buffer_shapes)# a factor class that builds the buffer componentsbuffer_factory=BufferFactory(buffer_system=buffer_system,buffer_info=buffer_info,commander=commander,producer=producer,)# a buffer iterator created with the buffer factory that allows iterating throught the 'concurrent' buffer.withBufferIterator(buffer_factory=buffer_factory)asdata_buffer_iterator:forindexinrange(10):data=next(data_buffer_iterator)Creating a Commanderfrom concurrentbuffer.commander import Commander
class IndexCommander(Commander):
    def __init__(self, max_index: int):
        self._max_index = max_index
        self._index = 0

    def create_message(self) -> dict:
        message = {"index": self._index}
        self._index = (self._index + 1) % self._max_index
        return messageCreating a Producerimport numpy as np
from concurrentbuffer.producer import Producer

class DataProducer(Producer):
    def __init__(self, data_shapes: tuple):
        self._data_shapes = data_shapes

    def create_data(self, message: dict) -> np.ndarray:
        index = message['index']
        return self._time_consuming_processing(index)

    def _time_consuming_processing(self, index) -> np.ndarray:
        ...
        #TODO use index and self._data_shape to create and process a numpy array