Package: faster-fifo
faster-fifoFaster alternative to Python's standard multiprocessing.Queue (IPC FIFO queue). Up to 30x faster in some configurations.Implemented in C++ using POSIX mutexes with PTHREAD_PROCESS_SHARED attribute. Based on a circular buffer, low footprint, brokerless.
Completely mimics the interface of the standard multiprocessing.Queue, so can be used as a drop-in replacement.Addsget_many()andput_many()methods to receive/send multiple messages at once for the price of a single lock.RequirementsLinux or MacOSPython 3.6 or newerGCC 4.9.0 or newerInstallationpip install faster-fifo(on a fresh Linux installation you might need some basic compiling toolssudo apt install --reinstall build-essential)Manual build instructionspip install Cython
python setup.py build_ext --inplace
pip install -e .Usage examplefromfaster_fifoimportQueuefromqueueimportFull,Emptyq=Queue(1000*1000)# specify the size of the circular buffer in the ctor# any pickle-able Python object can be added to the queuepy_obj=dict(a=42,b=33,c=(1,2,3),d=[1,2,3],e='123',f=b'kkk')q.put(py_obj)assertq.qsize()==1retrieved=q.get()assertq.empty()assertpy_obj==retrievedforiinrange(100):try:q.put(py_obj,timeout=0.1)exceptFull:log.debug('Queue is full!')num_received=0whilenum_received<100:# get multiple messages at once, returns a list of messages for better performance in many-to-few scenarios# get_many does not guarantee that all max_messages_to_get will be received on the first call, in fact# no such guarantee can be made in multiprocessing systems.# get_many() will retrieve as many messages as there are available AND can fit in the pre-allocated memory# buffer. The size of the buffer is increased gradually to match demand.messages=q.get_many(max_messages_to_get=100)num_received+=len(messages)try:q.get(timeout=0.1)assertTrue,'This won\'t be called'exceptEmpty:log.debug('Queue is empty')Performance comparison (faster-fifo vs multiprocessing.Queue)System #1 (Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz, 10 cores, Ubuntu 18.04)(measured execution times in seconds)multiprocessing.Queuefaster-fifo, get()faster-fifo, get_many()1 producer 1 consumer (200K msgs per producer)2.540.860.921 producer 10 consumers (200K msgs per producer)4.001.391.3610 producers 1 consumer (100K msgs per producer)13.196.740.943 producers 20 consumers (100K msgs per producer)9.302.222.1720 producers 3 consumers (50K msgs per producer)18.627.410.6420 producers 20 consumers (50K msgs per producer)36.511.323.79System #2 (Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz, 2 cores, Ubuntu 18.04)(measured execution times in seconds)multiprocessing.Queuefaster-fifo, get()faster-fifo, get_many()1 producer 1 consumer (200K msgs per producer)7.862.092.21 producer 10 consumers (200K msgs per producer)11.684.013.8810 producers 1 consumer (100K msgs per producer)44.4816.685.983 producers 20 consumers (100K msgs per producer)22.597.837.4920 producers 3 consumers (50K msgs per producer)66.322.36.3520 producers 20 consumers (50K msgs per producer)78.7514.3915.78Run testspython -m unittest(there are also C++ unit tests, should run them if C++ code was altered)Recent PyPI releasesv1.4.5Added methoddata_size()to query the total size of the messages in queue (in bytes). Thank you@LucaNicosia!v1.4.4Fixed an obscure issue with the TLSBuffer ctor being called without arguments (guessing it's Cython's weirdness)v1.4.3Simplified usage with "spawn" multiprocessing context. No need to usefaster_fifo_reductionanymore. Thank you@MosBas!v1.4.2Fixed an issue with the custom Queue picklerv1.4.1Fixed multithreading issues using threading.local for message recv buffer (huge thanks to@brianmacy!)Better error reporting in Cython and C++Added threading testsv1.4.0Increase default receive buffer size from 10 bytes to 5000 bytes.v1.3.1Minor change: better debugging messages + improved C++ testsv1.3.0Now support custom serializers and deserializers instead of Pickle (thank you@beasteers!):q=Queue(max_size_bytes=100000,loads=custom_deserializer,dumps=custom_serializer)FootnoteOriginally designed for SampleFactory, a high-throughput asynchronous RL codebasehttps://github.com/alex-petrenko/sample-factory.Programmed byAleksei Petrenkoand Tushar Kumar atUSC RESL.Developed under MIT License, feel free to use for any purpose, commercial or not, at your own risk.If you wish to cite this repository:@misc{faster-fifo,
    author={Petrenko, Aleksei and Kumar, Tushar},
    title={A Faster Alternative to Python's multiprocessing.Queue},
    publisher={GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/alex-petrenko/faster-fifo}},
    year={2020},
}