jacobhinkle/imp — Issue #3: data format
Author: cscheid
Created: 2012-02-23 01:37:43+00:00
State: open

We need to decide on the data format. Some terminology first:

imp will be a (key, value) store, where a (key,value) pair is called a *datum*, a key is a /-separated string of path delimiters, where each path delimiter is an identifier string of the following characters: [A-Za-z0-9_\-.]

We now need to decide on the supported value types. Jacob indicated he would rather have simple formats, for example, that value be one the following:

- an array of single-precision IEEE 754 floating point numbers
- a dense matrix of single-precision IEEE 754 floating point numbers
- an array of double-precision IEEE 854 floating point numbers
- a dense matrix of double-precision IEEE 854 floating point numbers

I think that if we're going to use a BSON variant for the transport, there might be advantages in supporting a more complicated format as well. For example, why not support BSON itself? The main advantage is that we're easily future-proofing the protocol:

- R data frames are easily represented in the following JSON object: { key1:array1, key2:array2, key3:array3 }
- Sparse matrices are easily represented as a record which respects a certain convention. CSC matrices would, for example, be { val: array1, row_ind: array2, col_ptr: array3 }

The argument _against_ a complicated file format is that we might want to store complicated objects as hierarchical objects within the naming system itself. The advantage of forcing such one such convention is that we would be able to change individual pieces of the larger object separately (by storing new files in "sublocations")

I guess if the typical use case is always going to be relatively shallow objects, then this discussion is not that important. If we expect deeper objects, then it could be trouble.