# Hyperspectral Python: HypPy
## 2 Design and Implementation
### Image Formats
#### 2.2.2 Memory Map  
HypPy uses Numpy's memory map, 'numpy.memmap', to map binary data files onto arrays. Memory-mapped files are used to access large files on a disk without having to read the entire file into memory first. NumPy memmap objects are array-like objects; they can be manipulated in the same way as numpy arrays. Because, in HypPy the choice was made to have the BIP structure for the internal array, the axes of the two other formats, BSQ and BIL, must be swapped to look like BIP in the returned memory map. This is achieved by calling the transpose function of the memory map. BSQ is mapped using a transpose(1, 2, 0). BIL is mapped using a transpose(0, 2, 1). The BIP format does not need to be transposed.  
One would think that using the transpose function would create a copy of the data with the axes of the data swapped in a different way. However, this is not the case because, for certain operations, numpy supports what is called a 'view' on the data. A view creates a reference to an existing array, with the required reorganization of the data added to the metadata. This has two advantages: first, the view does not create a copy of the data in memory, and second, when the view is opened in read-write, any modifications to the view will be reflected immediately in the original data.  
Furthermore, endianness and byte order can be changed by creating a view. The three formats (BSQ, BIP, and BIL), endianness (little endian and big endian), and data type (from 8-bit unsigned integers to 64-bit floating points, etc.) can be handled by creating a view of the data on file. For the input and output image data, no copy needs to reside in the memory of the computer. In such a way, HypPy can handle large files that would normally not fit in memory.  
The following functions are applied to create views:  
* Creating a memory map: numpy.memmap(fname, mode = 'r', shape = shape, dtype = data_type);
* Changing the byte order: method newbyteorder();
* Swapping the axes from BSQ to BIP: method transpose(1, 2, 0);
* Swapping the axes from BIL to BIP: method transpose(0, 2, 1).  
Unfortunately, changing the data type of an input file creates a copy in memory, not a view:  
* Change the data type of the input: method astype(dtype).  
However, there are a few caveats in the use of numpy memory maps. Not all operations and functions that can be used on numpy arrays can be used on memory maps, and only a limited number of operations or functions on a memory map are capable of returning a view.  
Additionally, the use of a memory map itself has disadvantages as well. Setting up a memory map uses a bit of extra memory, and all the reorganization of a view must be executed on-the-fly whenever the data on file are accessed. In this way, the memory map is less efficient than making an array copy in memory.  
The advantages of a memory map are obvious, it enables the handling of large datasets that cannot possibly fit in the physical memory of the computer. Furthermore, internally, the data are mapped on the same BIP structure all the time; in a program, there is no need to keep track of the real interleave format of the data file. All this is handled transparently by the memory map and the views thereof.