github.com/grandecola/bigqueue (go): Package bigqueue provides embedded, fast and persistent queue written in pure Go using memory mapped file. bigqueue is currently not thread safe. To use bigqueue in parallel context, a Write lock needs to be acquired (even for Read APIs).  Create or open a bigqueue:  bigqueue persists the data of the queue in multiple Arenas. Each Arena is a file on disk that is mapped into memory (RAM) using mmap syscall. Default size of each Arena is set to 128MB. It is possible to create a bigqueue with custom Arena size:  bigqueue also allows setting up the maximum possible memory that it can use. By default, the maximum memory is set to [3 x Arena Size].  In this case, bigqueue will never allocate more memory than `4KB*10=40KB`. This memory is above and beyond the memory used in buffers for copying data.  Bigqueue allows to set periodic flush based on either elapsed time or number of mutate (enqueue/dequeue) operations. Flush syncs the in memory changes of all memory mapped files with disk. *This is a best effort flush*. Elapsed time and number of mutate operations are only checked upon an enqueue/dequeue.  This is how we can set these options:  In this case, a flush is done after every two mutate operations.  In this case, a flush is done after one minute elapses and an Enqueue/Dequeue is called.  Write to bigqueue:  bigqueue allows writing string data directly, avoiding conversion to `[]byte`:  Read from bigqueue:  we can also read string data from bigqueue:  Check whether bigqueue has non zero elements:  bigqueue allows reading data from bigqueue using consumers similar to Kafka. This allows multiple consumers from reading data at different offsets (not in thread safe manner yet). The offsets of each consumer are persisted on disk and can be retrieved by creating a consumer with the same name. Data will be read from the same offset where it was left off.  We can create a new consumer as follows. The offsets of a new consumer are set at the start of the queue wherever the first non-deleted element is.  We can also copy an existing consumer. This will create a consumer that will have the same offsets into the queue as that of the existing consumer.  Now, read operations can be performed on the consumer:
Keywords: bigqueue|embedded|golang|persistence|queue