Package: shared-ndarray2
shared-ndarray2SharedNDArrayencapsulates a NumPyndarrayinterface for using shared memory in multiprocessing, usingmultiprocessing.shared_memoryin Python 3.8+.Quick StartRequirementsSimilar ProjectsUsageCreationSharedNDArray()SharedNDArray.from_shape()SharedNDArray.from_array()Using likenp.ndarrayReleasing Shared Memory.lockattributeTyped SharedNDArrayQuick Startimportmultiprocessingfrommultiprocessing.managersimportSharedMemoryManagerimportnumpyasnpfromshared_ndarray2importSharedNDArraydefprocess_data(arr:SharedNDArray):# Work with dataarr[:]+=1withSharedMemoryManager()asmem_mgr:arr=SharedNDArray.from_array(mem_mgr,np.arange(1024))p=multiprocessing.Process(target=process_data,args=(arr,))p.start()p.join()assertnp.all(arr[:]==np.arange(1,1025))RequirementsPython 3.8+NumPy 1.21+Similar ProjectsSharedArray- POSIX-only. Quite a different paradigm, uses
pre-Python 3.8 memory-sharing constructs, requires building a C module withgcc.shared-ndarray- POSIX-only. Similar (uses NumPy ndarraybufferarg), uses pre-Python 3.8 memory-sharing constructs (requiresposix_ipc).UsageCreationThere are three methods for constructing aSharedNDArray.SharedNDArray()To create aSharedNDArrayobject from existing shared memory that represents a NumPy array, use the regular
constructor providingshapeanddtype, either with an existingmultiprocessing.SharedMemoryobject or
the name of one:shm=SharedMemory(create=True,size=1024)arr=SharedNDArray(shm,(1024,),np.uint8)# -or-arr=SharedNDArray(shm.name,(1024,),np.uint8)SharedNDArray.from_shape()This method allocates shared memory managed by a SharedMemoryManager to represent a NumPyndarraywith someshapeanddtype.withSharedMemoryManagerasmem_mgr:arr=SharedNDArray.from_shape(mem_mgr,(3,1024,1024),dtype=np.uint16)# ... Use arr with e.g. multiprocessing.Pool or multiprocess.Process# ... Be sure process instances join/terminate before exiting SharedMemoryManager context managerSharedNDArray.from_array()This method allocates shared memory managed by a SharedMemoryManager to represent a some provided NumPyndarrayand copies that ndarray into the shared memoryx=np.arange(100.0).reshape(2,2,25)withSharedMemoryManagerasmem_mgr:arr=SharedNDArray.from_array(mem_mgr,x)assertnp.all(arr[:]==x[:])# ... Use arr as above...Using likenp.ndarrayThe point ofSharedNDArrayis to remove the boilerplate of creating shared memory, passing around shapes
and dtypes and reconstructingnp.ndarrayobjects.SharedNDArraydoes this last step with its.get()method, which creates anp.ndarrayon-the-fly using the shared memory buffer. The__getitem__()and__setitem__()methods use the.get()method to get the np.ndarray to access the data, so
multi-dimensional indexing and slicing work the same as with anndarray. Othernp.ndarraymethods are not
directly implemented but may be accessed by first calling.get(), e.g.arr.get().mean().Releasing Shared MemorySharedNDArrayimplements a__del__()method that calls the.close()method on theSharedMemorywhen the instance is destroyed (i.e. at process exit). When the shared memory is
unlinked in the parent process (either manually withshm.unlink()or by exiting aSharedMemoryManagercontext manager) the shared_memory is properly released. However if a
sub-process is not joined or terminated before the shared memory is unlinked a warning will be emitted about
"leaked shared_memory objects to clean up at shutdown"..lockattributeThe__init__(),from_shape(), andfrom_array()methods may be given alock=Trueargument that will
also create amultiprocessing.Lockobject and include it in theSharedNDArray, accesible as the.lockattribute. It should be noted, however, that it doesn't work well to pass amultiprocessing.Lockas an
argument to amultiprocessing.Poolfunction, for reasons describedhere.
Thus by default.lockis set toNone.Typed SharedNDArraySharedNDArrayis able to be typed with NumPy types. When using thefrom_array()constructor, it is also
able to inherit the type of thendarrayif it is typed usingnumpy.typing.NDArray(new in NumPy 1.21).
Typing information does not pass through with slicing (__getitem__), however.x:npt.NDArray[np.int_]=np.arange(1024)arr=SharedNDArray(mem_mgr,x)# type of x is SharedNDArray[int_]arr2=arr[:]# arr2 is typing.Any