Package: npy-append-array
NpyAppendArrayCreate Numpy.npyfiles by appending on the growth axis (0 for C order, -1
for Fortran order). It behaves likenumpy.concatenatewith the difference
that the result is stored out-of-memory in a.npyfile and can be reused for
further appending. After creation, the file can then be read with memory
mapping (e.g. by addingmmap_mode="r") which altogether allows to create and
read files (optionally) larger than the machine's main memory.Some possible applications:efficiently create large.npy(optionally database-like) filesHandling of offsets not included, can be done in an extra arrayLarge legacy files can be made appendable by callingensure_appendablecan (optionally) be performed in-place to minimize disk space usagecreate binary log files (optionally on low-memory embedded devices)Check the optionrewrite_header_on_append=Falsefor extra efficiencyBinary log files can be accessed very efficiently without parsingIncomplete files can be recovered efficiently by callingrecoverAnother feature of this library is the (above mentioned)recoverfunction,
which makes incomplete.npyfiles readable bynumpy.loadagain, no matter
whether they should be appended to or not.Incomplete files can be the result of broken downloads or unfinished writes.
Recovery works by rewriting the header and inferring the growth axis (see
above) by the file size. As the data length may not be evenly divisible by the
non-append-axis shape, incomplete entries can either be ignored
(zerofill_incomplete=False), which probably makes sense in most scenarios.
Alternatively, to squeeze out the as much information from the file as
possible,zerofill_incomplete=Truecan be used, which fills the incomplete
last append axis item with zeros.RaisesValueErrorinstead ofTypeErrorsince version 0.9.14 to be more
consistent with Numpy.NpyAppendArray can be used in multithreaded environments.Installationcondainstall-cconda-forgenpy-append-arrayorpipinstallnpy-append-arrayUsagefromnpy_append_arrayimportNpyAppendArrayimportnumpyasnparr1=np.array([[1,2],[3,4]])arr2=np.array([[1,2],[3,4],[5,6]])filename='out.npy'withNpyAppendArray(filename)asnpaa:npaa.append(arr1)npaa.append(arr2)npaa.append(arr2)data=np.load(filename,mmap_mode="r")print(data)ConcurrencyConcurrency can be achieved by multithreading: A singleNpyAppendArrayobject (per file) needs to be created. Then,appendcan be called from
multiple threads and locks will ensure that file writes do not happen in
parallel. When using with awithstatement, make sure thejoinhappens
within it, comparetest.py.Multithreaded writes are not the pinnacle of what is technically possible with
modern operating systems. It would be highly desirable to useasyncfile
writes. However, although modules likeaiofileexist, this is currently not
supported natively by Python or Numpy, comparehttps://github.com/python/cpython/issues/76742Implementation DetailsNpyAppendArray contains a modified, partial version offormat.pyfrom the
Numpy package. It ensures that array headers are created with 21
(=len(str(8*2**64-1))) bytes of spare space. This allows to fit an array of
maxed out dimensions (for a 64 bit machine) without increasing the array
header size. This allows to simply rewrite the header as we append data to the
end of the.npyfile.Supported SystemsTestes with Ubuntu Linux, macOS and Windows.