Dazo-org/zerovm — Issue #3: Pagination for input and output data
Author: camuel
Created: 2011-11-25 20:18:06+00:00
State: open
Labels: CRITICAL, FEATURE

zerovm is limited to 4GB address space and swift objects could be 5GB each, hence the need to some mechanism for streaming data through zerovm. Some processing workloads will need simultaneous access to different sections of input and/or output file and more than one pass. For example merge-sort needs three cursors and multiple passes. So instead of simple streaming interface we need a paging mechanism where user can map regions of hos 4GB address space to different places in input and/or output file.

Requirements:

1. The rate and the number of remapping initiated by NEXE must be capped and the cap strictly enforced. Let's say just few re-mappings per second per ZeroVM should be enough for any workload and any well behaving NEXE. After all there are 3GB+ that could be mapped right away and it will take some time to process all this data before another remapping will be needed.

2. There should be one and only one strictly controlled hole in the sandbox/zerovm through which NEXE communicates with ZeroVM TCB (trusted code base). All communication of NEXE with outer world is carried either through this hole or via initial data loaded into memory, or data left in the memory by NEXE after its execution. As part of this feature such communication channel must be designed and implemented. A nice convenient user library must be created ton top of this mechanism to hide its boring implementation details from the user.

3. The pagination mechanism must allow NEXE to supply an optimization hint that is similar to fadvise. For example, NEXE may ask for unmap page but indicate that it will need it again soon or it can indicate that this page will not be needed anymore (hinting that cache may be dropped). For output data it can also indicate when data is ready and could be started to be sent over network even before all result data is generated. ZeroVM must enforce hints meaning if NEXE indicated output page as finished he should not be able to map it again, same regarding hints of input data pages.

4. Pagination must allow only read-only mapping of input data file, actively preventing NEXE from modifying it. Output data is read-write while mapped. Unmapped pages of both input and output must be totally inaccessible for NEXE ofcourse.

5. Page size must be configurable.

6. It should be considered whether using huge pages makes sense here...