Repository: vega/altair

Issue:
pyarrow datetime is not supported using altair 5.0
I created an altair chart from a duckdb dataframe exported as an arrow table,  but I get this error 

TypeError: Object of type datetime is not JSON serializable

when I cast the field to string everything works fine.



Discussion:
Thanks for the report! I can reproduce it with the following:
```python
from datetime import datetime
import pyarrow as pa
import altair as alt

data = {    
    'date': [datetime(2004, 8, 1), datetime(2004, 9, 1)],
    'value': [102, 129]
 }
pa_table = pa.table(data)

alt.Chart(pa_table).mark_line().encode(
    x='date',
    y='value'
)
```

> ```cmd
> TypeError: Object of type datetime is not JSON serializable
> ```

Trying to narrow it down a bit further. The `pa_table` is a `pyarrow.Table` that contains two columns of type `timestamp[us]` and `int64`:
```
pa_table
```` 

> ```cmd
> pyarrow.Table
> date: timestamp[us]
> value: int64
> ----
> date: [[2004-08-01 00:00:00.000000,2004-09-01 00:00:00.000000]]
> value: [[102,129]]
> ```

The serialization of the arrow table is done using the dataframe interchange protocol available in `pyarrow`. 
```python
import pyarrow.interchange as pi

pi_table = pi.from_dataframe(pa_table)
pi_table
```

> ```cmd
> pyarrow.Table
> date: timestamp[us]
> price: int64
> ----
> date: [[2004-08-01 00:00:00.000000,2004-09-01 00:00:00.000000]]
> price: [[102,129]]
> ```

Where this object is converted into a pylist before inserted as values within the generated vega-lite specification:
```python
pi_table.to_pylist()
```

> ```cmd
> [{'date': datetime.datetime(2004, 8, 1, 0, 0), 'price': 102},
>  {'date': datetime.datetime(2004, 9, 1, 0, 0), 'price': 129}]
> ```

As can be seen, the `date` has become a `datetime.datetime()` timestamp. And this is problematic as this is not JSON serializable.

In Altair this happens around here: https://github.com/altair-viz/altair/blob/master/altair/utils/data.py#L168-L170

```python
elif hasattr(data, "__dataframe__"):
    # experimental interchange dataframe support
    pi = import_pyarrow_interchange()
    pa_table = pi.from_dataframe(data)
    return {"values": pa_table.to_pylist()}
```
Currently we do not run the `pi_table` through [`sanitize_dataframe`](https://github.com/altair-viz/altair/blob/master/altair/utils/core.py#L287), something that does happen for pandas.DataFrames, In this process of sanitation we serialize type [`datetime`](https://github.com/altair-viz/altair/blob/master/altair/utils/core.py#L347-L356) as such:
```python
elif str(dtype).startswith("datetime"):
    # Convert datetimes to strings. This needs to be a full ISO string
    # with time, which is why we cannot use ``col.astype(str)``.
    # This is because Javascript parses date-only times in UTC, but
    # parses full ISO-8601 dates as local time, and dates in Vega and
    # Vega-Lite are displayed in local time by default.
    # (see https://github.com/altair-viz/altair/issues/1027)
    df[col_name] = (
        df[col_name].apply(lambda x: x.isoformat()).replace("NaT", "")
    )
```
Not sure if we should sanitize interchange dataframes on the Python side or if it is better to push the `pyarrow.Table` using Arrow IPC serialization to the JavaScript side and construct valid objects there. Probably easier to include sanitation on the Python side. 
I think you can also explore the idea of extending the JSON serialiser with support for datetimes. For example that's what we have done in TurboGears ( https://github.com/TurboGears/tg2/blob/development/tg/jsonify.py#L94-L98 ).

That way you don't have to care about sanitising types explicitly because the json dumper will take care of it for you independently from where that value came from. 
If you decide to sanitize interchange dataframes on the Python side, you could use `cast` function (as @djouallah mentioned) and cast Timestamp to string to get the correct ISO format, for example:

```python
>>> from datetime import datetime
>>> import pyarrow as pa
>>> arr = pa.array([datetime(2010, 1, 1), datetime(2015, 1, 1)])
>>> arr
<pyarrow.lib.TimestampArray object at 0x12f37a020>
[
  2010-01-01 00:00:00.000000,
  2015-01-01 00:00:00.000000
]
>>> import pyarrow.compute as pc
>>> pc.cast(arr, pa.string())
<pyarrow.lib.StringArray object at 0x12f37a0e0>
[
  "2010-01-01 00:00:00.000000",
  "2015-01-01 00:00:00.000000"
]
```
Note: in case of duration, the error should be raised.

Resolving patch:
diff --git a/altair/utils/core.py b/altair/utils/core.py
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -397,6 +397,31 @@ def to_list_if_array(val):
     return df
 
 
+def sanitize_arrow_table(pa_table):
+    """Sanitize arrow table for JSON serialization"""
+    import pyarrow as pa
+    import pyarrow.compute as pc
+
+    arrays = []
+    schema = pa_table.schema
+    for name in schema.names:
+        array = pa_table[name]
+        dtype = schema.field(name).type
+        if str(dtype).startswith("timestamp"):
+            arrays.append(pc.strftime(array))
+        elif str(dtype).startswith("duration"):
+            raise ValueError(
+                'Field "{col_name}" has type "{dtype}" which is '
+                "not supported by Altair. Please convert to "
+                "either a timestamp or a numerical value."
+                "".format(col_name=name, dtype=dtype)
+            )
+        else:
+            arrays.append(array)
+
+    return pa.Table.from_arrays(arrays, names=schema.names)
+
+
 def parse_shorthand(
     shorthand,
     data=None,
diff --git a/altair/utils/data.py b/altair/utils/data.py
--- a/altair/utils/data.py
+++ b/altair/utils/data.py
@@ -8,7 +8,7 @@
 from toolz import curried
 from typing import Callable
 
-from .core import sanitize_dataframe
+from .core import sanitize_dataframe, sanitize_arrow_table
 from .core import sanitize_geo_interface
 from .deprecation import AltairDeprecationWarning
 from .plugin_registry import PluginRegistry
@@ -166,7 +166,7 @@ def to_values(data):
     elif hasattr(data, "__dataframe__"):
         # experimental interchange dataframe support
         pi = import_pyarrow_interchange()
-        pa_table = pi.from_dataframe(data)
+        pa_table = sanitize_arrow_table(pi.from_dataframe(data))
         return {"values": pa_table.to_pylist()}
 
 
@@ -185,8 +185,6 @@ def check_data_type(data):
 # ==============================================================================
 # Private utilities
 # ==============================================================================
-
-
 def _compute_data_hash(data_str):
     return hashlib.md5(data_str.encode()).hexdigest()
 


Test patch:
diff --git a/tests/utils/test_dataframe_interchange.py b/tests/utils/test_dataframe_interchange.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_dataframe_interchange.py
@@ -0,0 +1,57 @@
+from datetime import datetime
+import pyarrow as pa
+import pandas as pd
+import pytest
+import sys
+import os
+
+from altair.utils.data import to_values
+
+
+def windows_has_tzdata():
+    """
+    From PyArrow: python/pyarrow/tests/util.py
+
+    This is the default location where tz.cpp will look for (until we make
+    this configurable at run-time)
+    """
+    tzdata_path = os.path.expandvars(r"%USERPROFILE%\Downloads\tzdata")
+    return os.path.exists(tzdata_path)
+
+
+# Skip test on Windows when the tz database is not configured.
+# See https://github.com/altair-viz/altair/issues/3050.
+@pytest.mark.skipif(
+    sys.platform == "win32" and not windows_has_tzdata(),
+    reason="Timezone database is not installed on Windows",
+)
+def test_arrow_timestamp_conversion():
+    """Test that arrow timestamp values are converted to ISO-8601 strings"""
+    data = {
+        "date": [datetime(2004, 8, 1), datetime(2004, 9, 1), None],
+        "value": [102, 129, 139],
+    }
+    pa_table = pa.table(data)
+
+    values = to_values(pa_table)
+    expected_values = {
+        "values": [
+            {"date": "2004-08-01T00:00:00.000000", "value": 102},
+            {"date": "2004-09-01T00:00:00.000000", "value": 129},
+            {"date": None, "value": 139},
+        ]
+    }
+    assert values == expected_values
+
+
+def test_duration_raises():
+    td = pd.timedelta_range(0, periods=3, freq="h")
+    df = pd.DataFrame(td).reset_index()
+    df.columns = ["id", "timedelta"]
+    pa_table = pa.table(df)
+    with pytest.raises(ValueError) as e:
+        to_values(pa_table)
+
+    # Check that exception mentions the duration[ns] type,
+    # which is what the pandas timedelta is converted into
+    assert "duration[ns]" in e.value.args[0]