Package: wendelin.core
Wendelin.core allows you to work with arrays bigger than RAM and local disk.
Bigarrays are persisted to storage, and can be changed in transactional manner.In other words bigarrays are something likenumpy.memmapfor numpy.ndarray
and OS files, but support transactions and files bigger than disk. The whole
bigarray cannot generally be used as a drop-in replacement for numpy arrays,
but bigarrayslicesare real ndarrays and can be used everywhere ndarray can
be used, including in C/Cython/Fortran code. Slice size is limited by
virtual address-space size, which is ~ max 127TB on Linux/amd64.The main class to work with isZBigArrayand is used likendarrayfromNumPy:create array:from wendelin.bigarray.array_zodb import ZBigArray
import transaction

# root is connected to opened database
root['A'] = A = ZBigArray(shape=..., dtype=...)
transaction.commit()view array as a real ndarray:a = A[:]        # view which covers all array, if it fits into address-space
b = A[10:100]data for views will be loaded lazily on memory access.work with views, including using C/Cython/Fortran functions from NumPy
and other libraries to read/modify data:a[2] = 1
a[10:20] = numpy.arange(10)
numpy.mean(a)the amount of modifications in one transaction should be less than available RAM.the amount of data read is limited only by virtual address-space size.data can be appended to array in O(δ) time:values                  # ndarray to append of shape  (δ,)
A.append(values)and array itself can be resized in O(1) time:A.resize(newshape)changes to array data can be either discarded or saved back to DB:transaction.abort()     # discard all made changes
transaction.commit()    # atomically save all changesWhen usingNEOorZEOas a database, bigarrays can be simultaneously used by
several nodes in a cluster.Please seedemo/demo_zbigarray.pyfor a complete example.Current state and RoadmapWendelin.core works in real life for workloadsNexediis using in production,
including 24/7 projects. We are, however, aware of the following
limitations and things that need to be improved:wendelin.core is currently not very fastthere are big - proportional to input in size - temporary array allocations
in third-party libraries (NumPy,scikit-learn, …) which might practically
prevent processing out-of-core arrays depending on the functionality used.Thuswe are currently working on improved wendelin.core design and implementation,
which will use kernel virtual memory manager (instead of oneimplementedinuserspace) with arrays backend presented to kernel via FUSE as virtual
filesystem implemented in Go.In parallel we will also:try wendelin.core 1.0 on large data setsidentify and incrementally fix big-temporaries allocation issues in NumPy and
scikit-learnWe are open to community help with the above.Additional materialsWendelin.coretutorialSlides(pdf) from presentation about wendelin.core in PyData Paris 2015Wendelin.core change history0.13 (2019-06-18)Add support for Python 3.7 (commit).AddRAMArrayclass which is compatible toZBigArrayin API and semantic,
but stores its data in RAM only (commit 1,2).Addlib.xnumpy.structured- utility to create structured view of an array (commit 1,2).Fix logic to keepZBigFileHin sync with ZODB connection (commit).Fix crash on PyVMA deallocation (commit).Movepy.benchtopygolangso that it can be used not only in
Wendelin.core (commit).Enhancet/qemu-runlinux- utility that is used to work on combined
kernel/user-space workloads (commit 1,2,3,4,5,6).
This was in particular useful to develop Linux kernel fixes that are needed
for Wendelin.core 2.0 (kernel commit 1,2,3,4,5,6,7).Various bugfixes.0.12 (2018-04-16)Update licensing to be in line with whole Nexedi stack (commit). Please
seehttps://www.nexedi.com/licensingfor details, rationale and options.AddArrayRefutility to find out for a NumPy array its top-level root
parent and how to recreate the array as some view of the root;
this builds the foundation for e.g. sending arrays as references without copy
in CMFActivity joblib backend
(commit 1,2,3).Don’t crash if duringloadblk()garbage collection was run twice at tricky
times (commit 1,2).Don’t crash on writeout if previouslystoreblk()resulted in error
(commit).Fixpy.benchand rework it to produce output in Go benchmarking format
(commit 1,2,3,4,5); add benchmarks for handling pagefaults
(commit).Use zodbtools/zodburi, if available, to open database by URL
(commit).Start to make sure it works with ZODB5 too (commit 1,2).Various bugfixes.0.11 (2017-03-28)Switch back to using ZBlk0 format by default (commit)0.10 (2017-03-16)Tell the worlddtype=objectis not supported (commit)0.9 (2017-01-17)Avoid deadlocks via doingstoreblk()calls with virtmem lock released
(commit 1,2)Don’t crash if inloadblk()implementation an exception is internally
raised & caught
(commit 1,2,3)0.8 (2016-09-28)Do not leak memory when loading data in ZBlk1 format (commit).0.7 (2016-07-14)Add support for Python 3.5 (commit 1,2)Fix bug in pagemap code which could lead to crashes and other issues (commit)Various bugfixes0.6 (2016-06-13)Add support for FORTRAN ordering (commit 1,2)Avoid deadlocks via doingloadblk()calls with virtmem lock released
(commit 1,2)Various bugfixes0.5 (2015-10-02)Introduce another storage format, which is optimized for small changes, and
make it the default.
(commit 1,2)Various bugfixes and documentation improvements0.4 (2015-08-19)Add support for O(δ) in-place BigArray.append() (commit)Implement proper multithreading support (commit)Implement proper RAM pages invalidation when backing ZODB objects are changed
from outside (commit 1,2)Fix all kind of failures that could happen when ZODB connection changes
worker thread in-between handling requests (commit 1,2)Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages
(commit 1,2)Various bugfixes0.3 (2015-06-12)Add support for automatic BigArray -> ndarray conversion, so that e.g. the
following:A = BigArray(...)
numpy.mean(A)       # passing BigArray to plain NumPy functioneither succeeds, or raises MemoryError if not enough address space is
available to cover whole A. (current limitation is ~ 127TB on linux/amd64)(commit)Various bugfixes (build-fixes, crashes, overflows, etc)0.2 (2015-05-25)Add support for O(1) in-place BigArray.resize() (commit)Various build bugfixes (older systems, non-std python, etc)0.1 (2015-04-03)Initial release