Repository: vega/altair

Issue:
numpy scalar in Dataframe not JSON serializable
Is it possible to allow numpy scalars in a dataframe?

The following code is not working, because a np.array is not json serializable:

```
df_numpy = pd.DataFrame([dict(
        x=np.array(1),
        y=np.array(2),
        )])
Chart(df_numpy).encode(x='x', y='y')
# TypeError: array(1) is not JSON serializable
```

Full Example: [
https://github.com/boeddeker/ipynb/blob/master/bug_altair_numpy.ipynb](https://github.com/boeddeker/ipynb/blob/master/bug_altair_numpy.ipynb)



Discussion:
Thanks for the report!

I think the issue here is that you've created a dataframe containing object-types, rather than a dataframe containing numeric types. This is easier to see if you pass multi-valued arrays to your dataframe:

``` python
df_numpy = pd.DataFrame([dict(
        x=np.array([1, 2, 3]),
        y=np.array([2, 3, 4, 5]),
        )])
>>> df_numpy
           x          y
0  [1, 2, 3]  [2, 3, 4, 5]
```

It's not entirely clear to me what you might expect Altair to do in this case: at each index, the column contains an array, which cannot be meaningfully serialized either as a numerical value or as a string.

Now it's true that the zero-dimensional array is perhaps a corner case in which we could specially convert the array-object-type to a single numerical scalar type, but that is a case we'd have to explicitly handle (and explicitly test _each_ value in a type-object column to see if it's (1) an array and (2) has zero dimensions)

I think better would be to have the user give the data in a more standard form, e.g.

``` python
df_numpy = pd.DataFrame(dict(
        x=np.array([1]),
        y=np.array([2]),
    ))
```

in which case the chart works as you would expect.

I'm curious how you came across this – do you think it's important for Altair to check for and correct such a corner-case for malformed inputs?

Thinking about it, one solution might be to preprocess the dataframes with something like this:

``` python
for col in df_numpy.columns:
    try:
        df_numpy[col] = df_numpy[col].astype(float)
    except ValueError:
        pass
```

then you end up with a serializable array in the above case. The problem is, though, that if you use this on an array of strings containing only digits, these digits will be converted to a numeric type (which probably is not what the user wants). In both cases, the dtype of the pandas column is `object`. So before doing this, you'd have to individually check the Python type of each entry in any column with dtype object and make sure it's a zero-dimensional numpy array.

That strikes me as the kind of corner case that we should leave to the user to correct – but let me know if you disagree.

The problem is, that I use a library, which forces each value to be a numpy array. Because of that, also scalars are numpy array's.

I currently try to replace matplotlib with Altair, but in my opinion this error should not appear, since other plotting library's can handle this. 

Since this example works

```
df_numpy = pd.DataFrame([dict(
        x=np.array(1).tolist(),
        y=np.array(2).tolist(),
        )])
```

I would implement something like this (maybe slow, because of the iteration over all elements)

```
for index, series in df_numpy.iterrows():
    for k, s in series.items():
        if isinstance(s, np.ndarray):
            df_numpy.loc[index][k] = s.tolist()
```

this would not break anything what currently is working.

An alternate is to use an JSONEncoder as described in http://stackoverflow.com/a/27050186 .

```
class MyEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, numpy.integer):
            return int(obj)
        elif isinstance(obj, numpy.floating):
            return float(obj)
        elif isinstance(obj, numpy.ndarray):
            return obj.tolist()
        else:
            return super(MyEncoder, self).default(obj)

json.dumps(numpy.float32(1.2), cls=MyEncoder)
json.dumps(numpy.arange(12), cls=MyEncoder)
json.dumps({'a': numpy.int32(42)})
```

> I'm curious how you came across this – do you think it's important for Altair to check for and correct such a corner-case for malformed inputs?

I know that this is a special case, but the problem here is, that json can not handle numpy arrays and it seems, that the people form json don't want to solve this (http://bugs.python.org/issue18303, https://bugs.python.org/issue24313)

I see – yes I think we could add something along these lines to handle it. @ellisonbg – I'm curious about your opinion here: should this be handled via explicit conversion like we already do for some cases, or via a custom JSON encoder?


Resolving patch:
diff --git a/altair/utils/core.py b/altair/utils/core.py
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -142,6 +142,12 @@ def sanitize_dataframe(df):
     if isinstance(df.columns, pd.core.index.MultiIndex):
         raise ValueError('Hierarchical indices not supported')
 
+    def to_list_if_array(val):
+        if isinstance(val, np.ndarray):
+            return val.tolist()
+        else:
+            return val
+
     for col_name, dtype in df.dtypes.iteritems():
         if str(dtype) == 'category':
             # XXXX: work around bug in to_json for categorical types
@@ -158,4 +164,9 @@ def sanitize_dataframe(df):
             # Convert datetimes to strings
             # astype(str) will choose the appropriate resolution
             df[col_name] = df[col_name].astype(str).replace('NaT', '')
+        elif dtype == object:
+            # Convert numpy arrays saved as objects to lists
+            # Arrays are not JSON serializable
+            col = df[col_name].apply(to_list_if_array, convert_dtype=False)
+            df[col_name] = col.where(col.notnull(), None)
     return df


Test patch:
diff --git a/altair/utils/tests/test_utils.py b/altair/utils/tests/test_utils.py
--- a/altair/utils/tests/test_utils.py
+++ b/altair/utils/tests/test_utils.py
@@ -85,12 +85,14 @@ def test_sanitize_dataframe():
                        'f': np.arange(5, dtype=float),
                        'i': np.arange(5, dtype=int),
                        'd': pd.date_range('2012-01-01', periods=5, freq='H'),
-                       'c': pd.Series(list('ababc'), dtype='category')})
+                       'c': pd.Series(list('ababc'), dtype='category'),
+                       'o': pd.Series([np.array(i) for i in range(5)])})
 
     # add some nulls
     df.ix[0, 's'] = None
     df.ix[0, 'f'] = np.nan
     df.ix[0, 'd'] = pd.NaT
+    df.ix[0, 'o'] = np.array(np.nan)
 
     # JSON serialize. This will fail on non-sanitized dataframes
     df_clean = sanitize_dataframe(df)
@@ -108,4 +110,6 @@ def test_sanitize_dataframe():
         else:
             df2[col] = df2[col].astype(df[col].dtype)
 
+    # pandas doesn't properly recognize np.array(np.nan), so change it here
+    df.ix[0, 'o'] = np.nan
     assert df.equals(df2)