Package: ocycle
ocycleStream buffering and event triggering for unbalanced frame sizes. Data goes in with a certain frame size and comes out with a different size.It supports:threaded/multiprocessing callbacks(in addition to serial execution)a process/thread pool is created that callbacks are sent to.always available, recycled bufferswhen a buffer is passed to an asynchronous callback, a new one is created to be written to while the other is being used.once the callback function is done, it is added back to the stack, ready to be reused.temporal sampling- you can pass a function or scalar which will define how long to wait to start writing again after emitting a callback.So for example, if you wanted 50% coverage, you could passsampler=size / fswhich will wait forsize / fsseconds before writing again.Exampleimportocycle# this is some random data that represents our# incoming data bufferSOME_DATA='asdfghjkl;'# how full should the buffer be before emitting dataCONSUME_SIZE=29# a data consumerdefcreate_consumer(size):# this is called when the buffer reaches/exceeds size.defcallback(value,t0):assertlen(value)>=CONSUME_SIZEprint('Emitting:',t0,value)returnocycle.BufferEmit(callback,size)buff=create_consumer(CONSUME_SIZE)# a generator then writes to the buffer.# once it is full enough it will call `callback(buff, t0)`whileTrue:print('Writing to buffer:',SOME_DATA)buff.write(SOME_DATA)Example Use CaseAudio ProcessingYou want to stream audio from a microphone and perform multiple independent tasks on that stream. Each of these feature extractors have a different audio chunk size required.Say you want to:extract SPL (every 0.5s)write encrypted flac files (every 10s)run a tensorflow model (every 2s)Instead of collecting the data with a single chunk size and then manually tracking and spawning a new process every time each individual buffer fills up,BufferEmitwill handle all of that for you.UsageCallbackYou need to pass in a callback function,callback, which will be called once the buffer exceeds sizesize. This will receive the filled buffer and the
timestamp of when the first buffer frame arrived.You can also pass an additional callbackon_donewhich will be called back on the main process and is useful if you need to store a result from the callback on some object in memory.Execution MethodYou can choose how your callback is executed:mode='serial': the callback will be run in the executing thread/processmode='thread': the callback will be run in a new threadmode='process': the callback will be run in a new processYou can pass e.g.npool=10to use a 10 thread/process worker pool.Value TypeValue TypeBy default, the callback will receive abytesand timestamp object. If you passasbuffer=True, it will
passio.BytesIOwithout callingbuff.getvalue().ClippingBy default, the callback will receive the entire buffer after the buffer has exceededsize. This is useful if you don't want to clip partial messages, but this also means thatlen(buff) >= size. If you passclip=True, it will return buff in exact chunks of sizesize.SamplerIn some cases, you may not want all of the data and you just want to take some subsample over time.For example, say your machine learning model is not quite real time. You've tried really hard, but you can only get it down to 1.7x. You can define a sampler which will say after you emit a buffer, waitxseconds before you start writing to the buffer again. So you could setsampler=1.0and it will pause for 1 second before it begins writing to the buffer again.A sampler can also be any callable that needs no arguments and returns an int/float. This is useful if you want to implement stochastic/dynamic sampling strategies. For example, you could set:sampler=lambda: (np.random.randn() * 1.3 + 4)which will delay using a normal distribution of mean 4 and std dev of 1.3.Configuration ExamplesThis shows more of the different configurations thatBufferEmitcan exist as.TODO: it would be more illustrative with different buffer sizes.... idk what I was thinking lolimporttimeimportocycle# build a list of receiversbuffers=[]SIZE=15defon_emit(label):# callbackreturnlambdabuff,t0:print(label,buff.getvalue(),t0)# default operationbuffers.append(ocycle.BufferEmit(on_emit('default'),size=SIZE))# execute the callback in a threadbuffers.append(ocycle.BufferEmit(on_emit('im in a thread!'),size=SIZE,mode='thread'))# the callback for multiprocessing must be pickleabledefon_process_emit(buff,t0):# callbackx=buff.getvalue()print('Im in a process!',x,t0)returnx[::-1]# execute the callback in a separate process# execute `on_done` in the main processbuffers.append(ocycle.BufferEmit(on_process_emit,on_done=lambdares:print('The process returned:',res),size=SIZE,mode='process'))# send data to each receiverdefgenerate_data(length=4,n=1):for_inrange(n):foriinrange(10):yieldstr(i).encode('utf-8')*lengthforxingenerate_data(4):print('writing',x)forbinbuffers:b.write(x)time.sleep(1)Outputs:writing b'0000'
writing b'1111'
writing b'2222'
writing b'3333'
default b'0000111122223333' 1585253921.2961438
im in a thread! b'0000111122223333' 1585253921.296198
gets value, not buffer object b'0000111122223333' 1585253921.2967098
value clipped to size b'000011112222333' 1585253921.296758
Im in a process! b'0000111122223333' 1585253921.296656
The process returned: b'3333222211110000'
writing b'4444'
writing b'5555'
writing b'6666'
writing b'7777'
default b'4444555566667777' 1585253924.3013031
im in a thread! b'4444555566667777' 1585253924.3013911
gets value, not buffer object b'4444555566667777' 1585253924.3263922
value clipped to size b'344445555666677' 1585253924.326594
Im in a process! b'4444555566667777' 1585253924.301939
The process returned: b'7777666655554444'
writing b'8888'
writing b'9999'
writing b'0000'
writing b'1111'
default b'8888999900001111' 1585253928.3309472
im in a thread! b'8888999900001111' 1585253928.331071
gets value, not buffer object b'8888999900001111' 1585253928.3323038
value clipped to size b'778888999900001' 1585253928.33305
Im in a process! b'8888999900001111' 1585253928.332133
The process returned: b'1111000099998888'
writing b'2222'
writing b'3333'