Repository: numpy/numpy

Issue:
memmap docstring doesn't say how a new mapping is filled
Is `np.memmap` like `np.zeros`, or like `np.empty` wrt. the initial values in the array?



Discussion:
I'm pretty sure it's implemented like np.zeros, I.e. numpy itself doesn't
touch the allocated mapping. (Consider the case where you're mapping a file
that has some preexisting contents!) However, on most (all?) OSes, new
uninitialized mappings are filled with binary zero. (This is a security
issue - leaving the memory uninitialised would leak data from other
processes.)
On 6 Oct 2014 12:24, "Lars Buitinck" notifications@github.com wrote:

> Is np.memmap like np.zeros, or like np.empty wrt. the initial values in
> the array?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/5157.

The question mark in "all?" is exactly what I'm worried about. I'm aware of the security risk, but that can also be solved by writing `0xDEADBEEF` to all words in the mapping. [OpenBSD](http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/mmap.2?query=mmap) doesn't promise anything about the contents, and I was wondering what NumPy promises. If it just gives the user whatever the OS gave it, then I think that should be documented.

I'd lay substantial odds that openbsd zero-initializes newly map mapped
memory, whether it documents this or not.

Want to submit a pull request with whatever doc changes you think would
help?
On 6 Oct 2014 12:48, "Lars Buitinck" notifications@github.com wrote:

> The question mark in "all?" is exactly what I'm worried about. I'm aware
> of the security risk, but that can also be solved by writing 0xDEADBEEF
> to all words in the mapping. OpenBSD
> http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/mmap.2?query=mmap
> doesn't promise anything about the contents, and I was wondering what NumPy
> promises. If it just gives the user whatever the OS gave it, then I think
> that should be documented.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/numpy/numpy/issues/5157#issuecomment-58006070.

The documentation probably should just state that Numpy just returns the
memory as it received it from the OS. AFAIK posix-compatible mmap zeroes
the memory.

More careful reading of the POSIX standard
http://pubs.opengroup.org/onlinepubs/9699919799/functions/mmap.html
shows that access beyond end-of-file should result to SIGBUS. Numpy
tries to avoid that by explicity zero-padding the file for modes 'w+'
and 'r+', if the length of the mmap is bigger than the file:
https://github.com/numpy/numpy/blob/master/numpy/core/memmap.py#L242
.
The zero-padding relies on POSIX behavior on write-past-end. Python does
not seem to emulate this on Windows, and I don't know if fwrite() is
posix-compatible in this respect on windows or not...
.
For modes other than 'w+' and 'r+', Python's mmap() raises a ValueError
if the file is too short. If the file is truncated behind Numpy's back,
accessing the mmapped array indeed starts to raise SIGBUS errors.

> I don't know if fwrite() is posix-compatible in this respect on windows or not...

Nor me. What I find on [MSDN](http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537%28v=vs.85%29.aspx) about `CreateFileMapping` (not sure if that's used under the hood) is

> If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access [...], then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the file and the new end of the file are **not guaranteed to be zero**; the behavior is defined by the file system.

Emphasis added. Looks like the write-zero trick won't work on Windows as it does on Unix.


Resolving patch:
diff --git a/numpy/core/memmap.py b/numpy/core/memmap.py
--- a/numpy/core/memmap.py
+++ b/numpy/core/memmap.py
@@ -111,6 +111,11 @@ class memmap(ndarray):
     certain size depending on the platform. This size is always < 2GB
     even on 64-bit systems.
 
+    When a memmap causes a file to be created or extended beyond its
+    current size in the filesystem, the contents of the new part are
+    unspecified. On systems with POSIX filesystem semantics, the extended
+    part will be filled with zero bytes.
+
     Examples
     --------
     >>> data = np.arange(12, dtype='float32')


Test patch: