Repository: streamlit/streamlit

Issue:
Altair bindings do not work with numerical data types on Streamlit
### Summary

When using Altair binding select/range controls to filter columns with **integer** data types, no data matches the `transform_filter`. The binding controls work in Jupyter Notebook but not in Streamlit, which leads me to believe there might be an issue casting the data received from the Streamlit frontend to the appropriate type. Changing the filtering field to a string data type works for `binding_select`, but the same issue is also present for `binding_range`, which cannot be used with string data types. **Edit:** Range controls do appear to work for float fields, but they should also work for integer fields.

### Steps to reproduce

The below minimum reproducing example works in Jupyter notebook but not on Streamlit:

```python
import altair as alt
import streamlit as st
from vega_datasets import data

cars = data.cars()

control = alt.binding_select(options=sorted(cars['Cylinders'].unique()), name="Cylinders")
selection = alt.selection_single(name="cylinders", fields=['Cylinders'], bind=control, init={'Cylinders': 4})

chart = alt.Chart(cars).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
).interactive().add_selection(selection).transform_filter(selection)
st.altair_chart(chart)
```

However, if you change Cylinders to a string datatype, the following code works in both environments:

```python
cars['Cylinders_str'] = cars['Cylinders'].astype(str)
control = alt.binding_select(options=sorted(cars['Cylinders_str'].unique()), name="Cylinders")
selection = alt.selection_single(name="cylinders", fields=['Cylinders_str'], bind=control, init={'Cylinders_str': "4"})

chart = alt.Chart(cars).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
).interactive().add_selection(selection).transform_filter(selection)
st.altair_chart(chart)
```

<img width="534" alt="Screen Shot 2022-02-26 at 11 43 28 AM" src="https://user-images.githubusercontent.com/13065028/155851423-3e9a88c5-0ca8-406c-b79c-e88e9d0b25c7.png">

**Expected behavior:**

The data for the selected year should show up in the scatter plot, and the select control should have an effect.

**Actual behavior:**

No data appears in the scatter plot, and the bound control has no effect. The same behavior appears when using `alt.binding_range`.

### Is this a regression?

Unknown

### Debug info

- Streamlit version: 1.5.0
- Python version: 3.9.7
- Conda
- OS version: macOS 12.1
- Browser version: Safari 15.2

### Additional information

Reporting because a few students in the class I am TA'ing for have been experiencing this issue. It'd be great to have the `binding_range` fixed so that we can implement Gapminder-like charts without using Streamlit components, because this would require reloading the chart component on each change.



Discussion:
Thanks for reporting this issue. I was able to reproduce the problem. There is also this warning in the frontend associated with this problem:

<img width="518" alt="image" src="https://user-images.githubusercontent.com/2852129/155994464-fe50bfa3-25ff-4f81-a787-b0a4f5c0a0d3.png">

Also, this seems to be related to the arrow version of `st.altair_chart` since `st._legacy_altair_chart` seems to work fine for this example (this can be used as a temporary workaround).




Resolving patch:
diff --git a/lib/streamlit/elements/arrow_altair.py b/lib/streamlit/elements/arrow_altair.py
--- a/lib/streamlit/elements/arrow_altair.py
+++ b/lib/streamlit/elements/arrow_altair.py
@@ -380,7 +380,7 @@ def _melt_data(
 
     # Arrow has problems with object types after melting two different dtypes
     # pyarrow.lib.ArrowTypeError: "Expected a <TYPE> object, got a object"
-    data_df = type_util.convert_mixed_columns_to_string(
+    data_df = type_util.fix_arrow_incompatible_column_types(
         data_df, selected_columns=[x_column, color_column, y_column]
     )
 
diff --git a/lib/streamlit/errors.py b/lib/streamlit/errors.py
--- a/lib/streamlit/errors.py
+++ b/lib/streamlit/errors.py
@@ -73,10 +73,6 @@ class DuplicateWidgetID(StreamlitAPIException):
     pass
 
 
-class NumpyDtypeException(StreamlitAPIException):
-    pass
-
-
 class StreamlitAPIWarning(StreamlitAPIException, Warning):
     """Used to display a warning.
 
diff --git a/lib/streamlit/type_util.py b/lib/streamlit/type_util.py
--- a/lib/streamlit/type_util.py
+++ b/lib/streamlit/type_util.py
@@ -33,6 +33,7 @@
 )
 
 import pyarrow as pa
+from pandas import MultiIndex
 from pandas.api.types import infer_dtype
 from typing_extensions import Final, Literal, Protocol, TypeAlias, TypeGuard, get_args
 
@@ -531,13 +532,59 @@ def pyarrow_table_to_bytes(table: pa.Table) -> bytes:
     return cast(bytes, sink.getvalue().to_pybytes())
 
 
-def convert_mixed_columns_to_string(
+def _is_colum_type_arrow_incompatible(column: Union[Series, Index]) -> bool:
+    """Return True if the column type is known to cause issues during Arrow conversion."""
+
+    # Check all columns for mixed types and complex128 type
+    # The dtype of mixed type columns is always object, the actual type of the column
+    # values can be determined via the infer_dtype function:
+    # https://pandas.pydata.org/docs/reference/api/pandas.api.types.infer_dtype.html
+
+    return (
+        column.dtype == "object" and infer_dtype(column) in ["mixed", "mixed-integer"]
+    ) or column.dtype == "complex128"
+
+
+def fix_arrow_incompatible_column_types(
     df: DataFrame, selected_columns: Optional[List[str]] = None
 ) -> DataFrame:
+    """Fix column types that are not supported by Arrow table.
+
+    This includes mixed types (e.g. mix of integers and strings)
+    as well as complex numbers (complex128 type). These types will cause
+    errors during conversion of the dataframe to an Arrow table.
+    It is fixed by converting all values of the column to strings
+    This is sufficient for displaying the data on the frontend.
+
+    Parameters
+    ----------
+    df : pandas.DataFrame
+        A dataframe to fix.
+
+    selected_columns: Optional[List[str]]
+        A list of columns to fix. If None, all columns are evaluated.
+
+    Returns
+    -------
+    The fixed dataframe.
+    """
+
     for col in selected_columns or df.columns:
-        column = df[col]
-        if column.dtype == "object" and "mixed" in infer_dtype(column):
-            df[col] = column.astype(str)
+        if _is_colum_type_arrow_incompatible(df[col]):
+            df[col] = df[col].astype(str)
+
+    # The index can also contain mixed types
+    # causing Arrow issues during conversion.
+    # Skipping multi-indices since they won't return
+    # the correct value from infer_dtype
+    if not selected_columns and (
+        not isinstance(
+            df.index,
+            MultiIndex,
+        )
+        and _is_colum_type_arrow_incompatible(df.index)
+    ):
+        df.index = df.index.astype(str)
     return df
 
 
@@ -551,25 +598,14 @@ def data_frame_to_bytes(df: DataFrame) -> bytes:
 
     """
     try:
-        try:
-            table = pa.Table.from_pandas(df)
-        except pa.ArrowTypeError:
-            # Fallback: Convert all object types to string
-            df = convert_mixed_columns_to_string(df)
-            table = pa.Table.from_pandas(df)
-        return pyarrow_table_to_bytes(table)
-    except Exception as e:
-        _NUMPY_DTYPE_ERROR_MESSAGE = "Could not convert dtype"
-        if _NUMPY_DTYPE_ERROR_MESSAGE in str(e):
-            raise errors.NumpyDtypeException(
-                """
-Unable to convert `numpy.dtype` to `pyarrow.DataType`.
-This is likely due to a bug in Arrow (see https://issues.apache.org/jira/browse/ARROW-14087).
-As a temporary workaround, you can convert the DataFrame cells to strings with `df.astype(str)`.
-"""
-            )
-        else:
-            raise errors.StreamlitAPIException(e)
+        table = pa.Table.from_pandas(df)
+    except (pa.ArrowTypeError, pa.ArrowInvalid, pa.ArrowNotImplementedError):
+        _LOGGER.info(
+            "Applying automatic fixes for column types to make the dataframe Arrow-compatible."
+        )
+        df = fix_arrow_incompatible_column_types(df)
+        table = pa.Table.from_pandas(df)
+    return pyarrow_table_to_bytes(table)
 
 
 def bytes_to_data_frame(source: bytes) -> DataFrame:


Test patch:
diff --git a/lib/tests/streamlit/type_util_test.py b/lib/tests/streamlit/type_util_test.py
--- a/lib/tests/streamlit/type_util_test.py
+++ b/lib/tests/streamlit/type_util_test.py
@@ -18,10 +18,15 @@
 
 import pandas as pd
 import plotly.graph_objs as go
+from pandas.api.types import infer_dtype
 
 from streamlit import type_util
-from streamlit.errors import NumpyDtypeException
-from streamlit.type_util import data_frame_to_bytes, is_bytes_like, to_bytes
+from streamlit.type_util import (
+    data_frame_to_bytes,
+    fix_arrow_incompatible_column_types,
+    is_bytes_like,
+    to_bytes,
+)
 
 
 class TypeUtilTest(unittest.TestCase):
@@ -91,9 +96,86 @@ def test_to_bytes(self):
         with self.assertRaises(RuntimeError):
             to_bytes(string_obj)
 
-    def test_data_frame_to_bytes_numpy_dtype_exception(self):
+    def test_data_frame_with_dtype_values_to_bytes(self):
         df1 = pd.DataFrame(["foo", "bar"])
         df2 = pd.DataFrame(df1.dtypes)
 
-        with self.assertRaises(NumpyDtypeException):
+        try:
             data_frame_to_bytes(df2)
+        except Exception as ex:
+            self.fail(f"Converting dtype dataframes to Arrow should not fail: {ex}")
+
+    def test_fix_complex_column_type(self):
+        """Test that `fix_unsupported_column_types` correctly fixes
+        columns containing complex types by converting them to string.
+        """
+        df = pd.DataFrame(
+            {
+                "complex": [1 + 2j, 3 + 4j, 5 + 6 * 1j],
+                "integer": [1, 2, 3],
+                "string": ["foo", "bar", None],
+            }
+        )
+
+        self.assertEqual(infer_dtype(df["complex"]), "complex")
+
+        fixed_df = fix_arrow_incompatible_column_types(df)
+        self.assertEqual(infer_dtype(fixed_df["complex"]), "string")
+
+    def test_fix_mixed_column_types(self):
+        """Test that `fix_arrow_incompatible_column_types` correctly fixes
+        columns containing mixed types by converting them to string.
+        """
+        df = pd.DataFrame(
+            {
+                "mixed-integer": [1, "foo", 3],
+                "mixed": [1.0, "foo", 3],
+                "integer": [1, 2, 3],
+                "float": [1.0, 2.1, 3.2],
+                "string": ["foo", "bar", None],
+            },
+            index=[1.0, "foo", 3],
+        )
+
+        fixed_df = fix_arrow_incompatible_column_types(df)
+
+        self.assertEqual(infer_dtype(fixed_df["mixed-integer"]), "string")
+        self.assertEqual(infer_dtype(fixed_df["mixed"]), "string")
+        self.assertEqual(infer_dtype(fixed_df["integer"]), "integer")
+        self.assertEqual(infer_dtype(fixed_df["float"]), "floating")
+        self.assertEqual(infer_dtype(fixed_df["string"]), "string")
+        self.assertEqual(infer_dtype(fixed_df.index), "string")
+
+        self.assertEqual(
+            str(fixed_df.dtypes),
+            """mixed-integer     object
+mixed             object
+integer            int64
+float            float64
+string            object
+dtype: object""",
+        )
+
+    def test_data_frame_with_unsupported_column_types(self):
+        """Test that `data_frame_to_bytes` correctly handles dataframes
+        with unsupported column types by converting those types to string.
+        """
+        df = pd.DataFrame(
+            {
+                "mixed-integer": [1, "foo", 3],
+                "mixed": [1.0, "foo", 3],
+                "complex": [1 + 2j, 3 + 4j, 5 + 6 * 1j],
+                "integer": [1, 2, 3],
+                "float": [1.0, 2.1, 3.2],
+                "string": ["foo", "bar", None],
+            },
+            index=[1.0, "foo", 3],
+        )
+
+        try:
+            data_frame_to_bytes(df)
+        except Exception as ex:
+            self.fail(
+                "No exception should have been thrown here. "
+                f"Unsupported types of this dataframe should have been automatically fixed: {ex}"
+            )