JuliaLang/julia — pull request #9363 by mrocklin (2014-12-15 03:03:14+00:00)
Title: Missing values by Sentinels

Here is a draft for missing value support by sentinels. This is my first real go at Julia so I suspect I'm doing many things badly. Still, hopefully the general design is sound. More on missing bits at the end of this comment. Work was done with @dchudz.

Sentinels represent missing values (NA) by rarely used values within the domain of a datatype. In particular we use the following values for the following types:

* `Bool`: The integer 2
* `Float32/64`: The exponent signalling `NaN` and the mantissa 1954 (the birth year of an R creator)
* `Int32/64`: The minimum integer -2**31/63 + 1
* `Complex`: The float definition of `NA` for both real and imaginary parts

We extend arithmetic and mathematical operations to respect the new semantics of these special values. Each operation is now accompanied by an `isNA` check.

The choice of sentinel values here agrees with the choices made by the R language and subsequently by the DyND project in Python (cc @mwiebe) ([article on R sentinels](http://www.markvanderloo.eu/yaRb/2012/07/08/representation-of-numerical-nas-in-r-and-the-1954-enigma/).)

Adopting this as a cross-language standard for missing values might be of some use.

Example
-------
```julia

julia> x = Option(1)
1

julia> x + 3.14
4.140000000000001

julia> typeof(x + 3.14)
Option{Float64}

julia> x = Array{Option{Int32}}([1, 2, NA, 4])
4-element Array{Option{Int32},1}:
1
2
NA
4

julia> x[4] = NA
NA

julia> x
4-element Array{Option{Int32},1}:
1
2
NA
NA

julia> mean(x)
NA

julia> mean(nonnull(x))
1.5
```

TODO
----

I have no idea if this is an appropriate solution for missing values in Julia. `DataArray` (maked arrays) and `Nullable` (explicitly boxed) are two other fine solutions in various contexts. This is just a third.

If this is an appropriate solution then certainly my implementation could use a lot of work. In particular

1. Option should be a subtype of Number. When I do this I run into a large number of dispatch ambiguities.
2. We need to extend all elementary mathematical functions. Is there a list in Base somewhere? DataArray has such a list. It'd be nice if that were moved somewhere closer to core.
3. I haven't yet invested in learning how to properly test while developing in Julia. As a result there are no tests.
4. In many cases when we use `Number` we really mean `{Int32, Int64, Float32, Float64, ...}` all of the Optionable types.
5. The singleton `NA` could use better interaction with promotion rules and such.

Also, presumably people have discussed this before. I'm coming into this without much context. My apologies.

Also, presumably people have discussed this before so I'm really coming into this without much context. My