Package: ragged-buffer
ENN Ragged BufferThis Python package implements an efficientRaggedBufferdatatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use inenn-trainerand currently only supports a small selection of the numpy array methods.User GuideInstall the package withpip install ragged-buffer.
The package currently supports threeRaggedBuffervariants,RaggedBufferF32,RaggedBufferI64, andRaggedBufferBool.Creating a RaggedBufferGet sizeConvert to numpy arrayIndexingAdditionConcatenationClearCreating a RaggedBufferThere are three ways to create aRaggedBuffer:RaggedBufferF32(features: int)creates an emptyRaggedBufferwith the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)creates aRaggedBufferfrom a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_arraycreates aRaggedBuffer(with equal sequence lenghts) from a 3D numpy array.Creating an empty buffer and pushing each row:importnumpyasnpfromragged_bufferimportRaggedBufferF32# Create an empty RaggedBuffer with a feature size of 3buffer=RaggedBufferF32(3)# Push sequences with 3, 5, 0, and 1 elementsbuffer.push(np.array([[1,2,3],[4,5,6],[7,8,9]],dtype=np.float32))buffer.push(np.array([[10,11,12],[13,14,15],[16,17,18],[19,20,21],[22,23,24]],dtype=np.float32))buffer.push(np.array([],dtype=np.float32))# Alternative: `buffer.push_empty()`buffer.push(np.array([[25,25,27]],dtype=np.float32))Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension,
and an array of sequence lengths:importnumpyasnpfromragged_bufferimportRaggedBufferF32buffer=RaggedBufferF32.from_flattened(np.array([[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15],[16,17,18],[19,20,21],[22,23,24],[25,25,27]],dtype=np.float32),np.array([3,5,0,1],dtype=np.int64)))Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):importnumpyasnpfromragged_bufferimportRaggedBufferF32buffer=RaggedBufferF32.from_array(np.zeros((4,5,3),dtype=np.float32))Get sizeThesize0,size1, andsize2methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.importnumpyasnpfromragged_bufferimportRaggedBufferF32buffer=RaggedBufferF32.from_flattened(np.zeros((9,64),dtype=np.float32),np.array([3,5,0,1],dtype=np.int64)))# Get size of the first/batch dimension.assertbuffer.size0()==10# Get size of individual sequences.assertbuffer.size1(1)==5assertbuffer.size1(2)==0# Get size of the last/feature dimension.assertbuffer.size2()==64Convert to numpy arrayas_arayconverts aRaggedBufferto a flat 2D numpy array that combines the first and second dimension.importnumpyasnpfromragged_bufferimportRaggedBufferI64buffer=RaggedBufferI64(1)buffer.push(np.array([[1],[1],[1]],dtype=np.int64))buffer.push(np.array([[2],[2]],dtype=np.int64))assertnp.all(buffer.as_array(),np.array([[1],[1],[1],[2],[2]],dtype=np.int64))IndexingYou can index aRaggedBufferwith a single integer (returning aRaggedBufferwith a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.importnumpyasnpfromragged_bufferimportRaggedBufferF32# Create a new `RaggedBufferF32`buffer=RaggedBufferF32.from_flattened(np.arange(0,40,dtype=np.float32).reshape(10,4),np.array([3,5,0,1],dtype=np.int64))# Retrieve the first sequence.assertnp.all(buffer[0].as_array()==np.array([[0,1,2,3],[4,5,6,7],[8,9,10,11]],dtype=np.float32))# Get a RaggedBatch with 2 randomly selected sequences.buffer[np.random.permutation(4)[:2]]AdditionYou can add twoRaggedBuffers with the+operator if they have the same number of sequences, sequence lengths, and features. You can also add aRaggedBufferwhere all sequences have a length of 1 to aRaggedBufferwith variable length sequences, broadcasting along each sequence.importnumpyasnpfromragged_bufferimportRaggedBufferF32# Create ragged buffer with dimensions (3, [1, 3, 2], 1)rb3=RaggedBufferI64(1)rb3.push(np.array([[0]],dtype=np.int64))rb3.push(np.array([[0],[1],[2]],dtype=np.int64))rb3.push(np.array([[0],[5]],dtype=np.int64))# Create ragged buffer with dimensions (3, [1, 1, 1], 1)rb4=RaggedBufferI64.from_array(np.array([0,3,10],dtype=np.int64).reshape(3,1,1))# Add rb3 and rb4, broadcasting along the sequence dimension.rb5=rb3+rb4assertnp.all(rb5.as_array()==np.array([[0],[3],[4],[5],[10],[15]],dtype=np.int64))ConcatenationTheextendmethod can be used to mutate aRaggedBufferby appending anotherRaggedBufferto it.importnumpyasnpfromragged_bufferimportRaggedBufferF32rb1=RaggedBufferF32.from_array(np.zeros((4,5,3),dtype=np.float32))rb2=RaggedBufferF32.from_array(np.zeros((2,5,3),dtype=np.float32))rb1.extend(r2)assertrb1.size0()==6ClearTheclearmethod removes all elements from aRaggedBufferwithout deallocating the underlying memory.importnumpyasnpfromragged_bufferimportRaggedBufferF32rb=RaggedBufferF32.from_array(np.zeros((4,5,3),dtype=np.float32))rb.clear()assertrb.size0()==0LicenseENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.