Repository: altair-viz/altair

Issue:
Support of ordinal based on pandas' ordered Categorical type?
I've just started to play with altair, using the [diamonds](http://vincentarelbundock.github.io/Rdatasets/datasets.html) dataset. Here is the notebook to clarify what I did https://gist.github.com/pierre-haessig/09fa9268aa0a0e7d91356f681f96ca18

Since, I'm not familiar with altair, I maybe missed something, but I've got the feeling that ordered Categorical types from pandas are not supported.

Indeed, if I use a color='cut' encoding, when cut is a pandas Series with an _ordered_ category dtype, I get by default a nominal type of coloring (with "unordered" colors).

On the other hand, if I force the use of ordered with color='cut:O', I indeed get the ordered colored (the shades of green), but the _order is wrong_! (I get Fair, Good, Ideal, Premium, Very Good, while the correct order is 'Fair', 'Good', 'Very Good', 'Premium', 'Ideal', as manually defined in pandas' category)


Discussion:
joelostblom: Sorry I noticed that I forgot to click "Request review" here initially.
nkrishnaswami: Would love to see support for pandas ordinals! @jakevdp, ptal?
mattijn: I noticed that the changes to the tests are aiming vegalite/v4, you probably want to change this to vegalite/v5 if you think this PR should be merged @joelostblom.
joelostblom: @mattijn I moved the tests to v5 instead of v4 and also updated the docs. Feel free to give this a review if you have the time.
mattijn: I looked to the code changes and looked to the related issues PRs, but I still feel I miss a bit of context. Would it be possible to include a comment with a very simple example of the behavior before this PR and the behavior incorporating the changes of this PR?
binste: # Example
Based on [this comment](https://github.com/altair-viz/altair/issues/245#issuecomment-748443434).

Data and imports:

```python
import altair as alt
import pandas as pd
from vega_datasets import data

source = data.barley()
```
If a field is passed in as an ordered `pd.Categorical` such as
```python
site_lst = ['Crookston', 'Morris', 'University Farm','Duluth', 'Grand Rapids', 'Waseca']
source.site = pd.Categorical(source.site, site_lst, ordered = True)
```
Altair so far ignored this ordering and just treated it as a nominal field as if it would be a column of strings.

This means that in the current master branch, the colors are not sorted according to `site_lst` and the type of `site` is inferred as `nominal`:

```python
alt.Chart(source).mark_bar().encode(
    x='sum(yield)',
    y='variety',
    color="site",
)
```
<img width="511" alt="image" src="https://user-images.githubusercontent.com/23366411/214492592-6f4e4a4f-042c-41f3-897e-d0704aed9030.png">

Running the same code again on the branch of this PR, Altair will choose `ordinal` as the type for `site` and set the `sort` argument to the order defined in `site_lst`:
<img width="607" alt="image" src="https://user-images.githubusercontent.com/23366411/214493011-8484e057-0195-415c-9188-5cbdc1f9e83f.png">

So this PR basically expands `color='site` to `color=alt.Color('site:O', sort=alt.Sort(site_lst))` (note the `:O` as well as `sort`).

To sort the stacked bar charts as well, one still needs to use the trick documented by @mattijn [here](https://github.com/altair-viz/altair/issues/245#issuecomment-748443434). Note that this is maybe not the best example to show the use of ordinal data as the data is probably best represented as `nominal` to get a qualitative color scheme but with a `sort` attribute on `Color` to get the desired sorting. But I hope this helps in reviewing.
binste: A better example is probably the one in [this comment](https://github.com/altair-viz/altair/issues/245#issuecomment-792434474).

```python
import altair as alt
import pandas as pd

dfdict = {
    "Questions": {
        0: "Question Text",
        1: "Question Text",
        2: "Question Text",
        3: "Question Text",
        4: "Question Text",
        5: "Question Text",
    },
    "level": {
        0: "1 - Strongly Disagree",
        1: "2 - Disagree",
        2: "3 - Neutral",
        3: "4 - Agree",
        4: "5 - Strongly Agree",
        5: "N/A",
    },
    "value": {0: 1.4, 1: 5.7, 2: 10.0, 3: 32.9, 4: 47.1, 5: 2.9},
}

df = pd.DataFrame(dfdict)

sort_order = [
    "N/A",
    "5 - Strongly Agree",
    "4 - Agree",
    "3 - Neutral",
    "2 - Disagree",
    "1 - Strongly Disagree",
]

# This doesn't seem to be needed
df["level"] = pd.Categorical(df["level"], sort_order, ordered=True)

chart = (
    alt.Chart(df)
    .mark_bar()
    .encode(
        x=alt.X("value"),
        y=alt.Y("Questions", title=""),
        color="level", 
        order=alt.Order("color_site_sort_index:Q"),
    )
)

chart
```

Behaviour with this PR:
<img width="602" alt="image" src="https://user-images.githubusercontent.com/23366411/214494566-92bb4184-44f3-4ff9-b726-0ccdb708737b.png">

You can see that color is represented as ordinal and with the colors sorted according to `sort_order`.

binste: If a different data type is specified such as `:N` then `sort` is not set and the behaviour is the same as on the master branch. This happens here https://github.com/altair-viz/altair/pull/2522/files#diff-25906c2570b127f0bb9cff4550bcc153006b2fdb36aea55ae804ca9a74eb4404R367. Just tested and verified this with an example.

I guess it can be surprising to people who are used to that only the datatypes are inferred. But converting a column in pandas to an ordered categorical one is usually a rather conscious decision so I'd expect them to either be happy about the choice that Vega-Lite then makes in terms of color scale and ordering or that they at least figure out how to fix it. Certainly a feature which would be great to have in a release candidate.

Patch:
diff --git a/altair/utils/core.py b/altair/utils/core.py
index 53785b17..c47b9a04 100644
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -193,8 +193,6 @@ def infer_vegalite_type(data):
     # Otherwise, infer based on the dtype of the input
     typ = infer_dtype(data)
 
-    # TODO: Once this returns 'O', please update test_select_x and test_select_y in test_api.py
-
     if typ in [
         "floating",
         "mixed-integer-float",
@@ -203,6 +201,8 @@ def infer_vegalite_type(data):
         "complex",
     ]:
         return "quantitative"
+    elif typ == "categorical" and data.cat.ordered:
+        return ("ordinal", data.cat.categories.tolist())
     elif typ in ["string", "bytes", "categorical", "boolean", "mixed", "unicode"]:
         return "nominal"
     elif typ in [
@@ -316,8 +316,9 @@ def sanitize_dataframe(df):  # noqa: C901
 
     for col_name, dtype in df.dtypes.items():
         if str(dtype) == "category":
-            # XXXX: work around bug in to_json for categorical types
+            # Work around bug in to_json for categorical types in older versions of pandas
             # https://github.com/pydata/pandas/issues/10778
+            # https://github.com/altair-viz/altair/pull/2170
             col = df[col_name].astype(object)
             df[col_name] = col.where(col.notnull(), None)
         elif str(dtype) == "string":
@@ -527,6 +528,10 @@ def parse_shorthand(
     if isinstance(data, pd.DataFrame) and "type" not in attrs:
         if "field" in attrs and attrs["field"] in data.columns:
             attrs["type"] = infer_vegalite_type(data[attrs["field"]])
+            # ordered categorical dataframe columns return the type and sort order as a tuple
+            if isinstance(attrs["type"], tuple):
+                attrs["sort"] = attrs["type"][1]
+                attrs["type"] = attrs["type"][0]
     return attrs
 
 
diff --git a/altair/utils/schemapi.py b/altair/utils/schemapi.py
index b9f8bae9..fab24a4a 100644
--- a/altair/utils/schemapi.py
+++ b/altair/utils/schemapi.py
@@ -364,6 +364,14 @@ class SchemaBase(object):
             # parsed_shorthand is removed from context if it exists so that it is
             # not passed to child to_dict function calls
             parsed_shorthand = context.pop("parsed_shorthand", {})
+            # Prevent that pandas categorical data is automatically sorted
+            # when a non-ordinal data type is specifed manually
+            if "sort" in parsed_shorthand and kwds["type"] not in [
+                "ordinal",
+                Undefined,
+            ]:
+                parsed_shorthand.pop("sort")
+
             kwds.update(
                 {
                     k: v
diff --git a/doc/user_guide/encodings/channels.rst b/doc/user_guide/encodings/channels.rst
index 057db9c7..073580b2 100644
--- a/doc/user_guide/encodings/channels.rst
+++ b/doc/user_guide/encodings/channels.rst
@@ -110,7 +110,7 @@ We map the ``symbol`` variable to ``detail`` to use them to group lines.
 Order
 ~~~~~
 
-The `order` option and :class:`Order` channel can sort how marks are drawn on the chart.
+The ``order`` option and :class:`Order` channel can sort how marks are drawn on the chart.
 
 For stacked marks, this controls the order of components of the stack. Here, the elements of each bar are sorted alphabetically by the name of the nominal data in the color channel.
 
@@ -144,9 +144,6 @@ The order can be reversed by changing the sort option to `descending`.
         order=alt.Order("site", sort="descending")
     )
 
-If we want to sort stacked segments in a custom order, we can `follow the approach in this issue comment <https://github.com/altair-viz/altair/issues/245#issuecomment-748443434>`_, although there might be edge cases where this is not fully supported. This also makes the order of the segments align with the order colors shows up in a legend that uses custom sorting for the color domain.
-
-
 The same approach works for other mark types, like stacked areas charts.
 
 .. altair-plot::
@@ -163,7 +160,12 @@ The same approach works for other mark types, like stacked areas charts.
         order=alt.Order("site", sort="ascending")
     )
 
-For line marks, the `order` channel encodes the order in which data points are connected. This can be useful for creating a scatter plot that draws lines between the dots using a different field than the x and y axes.
+Note that unlike the ``sort`` parameter to positional encoding channels,
+the :class:`Order` channel cannot take a list of values to sort by
+and is not automatically sorted when an ordered pandas categorical column is passed.
+If we want to sort stacked segments in a custom order, we can `follow the approach in this issue comment <https://github.com/altair-viz/altair/issues/245#issuecomment-748443434>`_, although there might be edge cases where this is not fully supported. This workaround also makes the order of the segments align with the order that the colors shows up in a legend that uses custom sorting for the color domain.
+
+For line marks, the :class:`Order` channel encodes the order in which data points are connected. This can be useful for creating a scatter plot that draws lines between the dots using a different field than the x and y axes.
 
 .. altair-plot::
 
diff --git a/doc/user_guide/encodings/index.rst b/doc/user_guide/encodings/index.rst
index b46acf6b..32073259 100644
--- a/doc/user_guide/encodings/index.rst
+++ b/doc/user_guide/encodings/index.rst
@@ -318,18 +318,22 @@ Sort Option
 ~~~~~~~~~~~
 
 Some channels accept a  :class:`sort` option which determines the
-order of the scale being used for the channel. There are a number of different
-sort options available:
+order of the scale being used for the channel.
+By default the scale is sorted in ascending alphabetical order,
+unless an `ordered pandas categorical column <https://pandas.pydata.org/docs/user_guide/categorical.html?highlight=categorical#sorting-and-order>`_ is passed (without an explicit type specification)
+in which case Altair will use the column's inherent order to sort the scale.
+There are a number of different
+options available to change the sort order:
 
 - ``sort='ascending'`` (Default) will sort the field's value in ascending order.
-  for string data, this uses standard alphabetical order.
+  For string data, this uses standard alphabetical order.
 - ``sort='descending'`` will sort the field's value in descending order
-- passing the name of an encoding channel to ``sort``, such as ``"x"`` or ``"y"``, allows for
+- Passing the name of an encoding channel to ``sort``, such as ``"x"`` or ``"y"``, allows for
   sorting by that channel. An optional minus prefix can be used for a descending
   sort. For example ``sort='-x'`` would sort by the x channel in descending order.
-- passing a list to ``sort`` allows you to explicitly set the order in which
+- Passing a list to ``sort`` allows you to explicitly set the order in which
   you would like the encoding to appear
-- passing a :class:`EncodingSortField` class to ``sort`` allows you to sort
+- Passing a :class:`EncodingSortField` class to ``sort`` allows you to sort
   an axis by the value of some other field in the dataset.
 
 Here is an example of applying these five different sort approaches on the
diff --git a/tools/schemapi/schemapi.py b/tools/schemapi/schemapi.py
index aeaf156c..d04bdf2d 100644
--- a/tools/schemapi/schemapi.py
+++ b/tools/schemapi/schemapi.py
@@ -362,6 +362,14 @@ class SchemaBase(object):
             # parsed_shorthand is removed from context if it exists so that it is
             # not passed to child to_dict function calls
             parsed_shorthand = context.pop("parsed_shorthand", {})
+            # Prevent that pandas categorical data is automatically sorted
+            # when a non-ordinal data type is specifed manually
+            if "sort" in parsed_shorthand and kwds["type"] not in [
+                "ordinal",
+                Undefined,
+            ]:
+                parsed_shorthand.pop("sort")
+
             kwds.update(
                 {
                     k: v


Test patch:
diff --git a/tests/vegalite/v5/tests/test_api.py b/tests/vegalite/v5/tests/test_api.py
index b2e5aa15..12e8b666 100644
--- a/tests/vegalite/v5/tests/test_api.py
+++ b/tests/vegalite/v5/tests/test_api.py
@@ -123,6 +123,7 @@ def test_chart_infer_types():
             "x": pd.date_range("2012", periods=10, freq="Y"),
             "y": range(10),
             "c": list("abcabcabca"),
+            "s": pd.Categorical([1, 2] * 5, categories=[2, 1], ordered=True),
         }
     )
 
@@ -134,32 +135,45 @@ def test_chart_infer_types():
         assert dct["encoding"]["y"]["field"] == "y"
         assert dct["encoding"]["color"]["type"] == "nominal"
         assert dct["encoding"]["color"]["field"] == "c"
+        assert dct["encoding"]["size"]["type"] == "ordinal"
+        assert dct["encoding"]["size"]["field"] == "s"
+        assert dct["encoding"]["size"]["sort"] == [2, 1]
 
     # Pass field names by keyword
-    chart = alt.Chart(data).mark_point().encode(x="x", y="y", color="c")
+    chart = alt.Chart(data).mark_point().encode(x="x", y="y", color="c", size="s")
     _check_encodings(chart)
 
     # pass Channel objects by keyword
     chart = (
         alt.Chart(data)
         .mark_point()
-        .encode(x=alt.X("x"), y=alt.Y("y"), color=alt.Color("c"))
+        .encode(x=alt.X("x"), y=alt.Y("y"), color=alt.Color("c"), size=alt.Size("s"))
     )
     _check_encodings(chart)
 
     # pass Channel objects by value
-    chart = alt.Chart(data).mark_point().encode(alt.X("x"), alt.Y("y"), alt.Color("c"))
+    chart = (
+        alt.Chart(data)
+        .mark_point()
+        .encode(alt.X("x"), alt.Y("y"), alt.Color("c"), alt.Size("s"))
+    )
     _check_encodings(chart)
 
     # override default types
     chart = (
         alt.Chart(data)
         .mark_point()
-        .encode(alt.X("x", type="nominal"), alt.Y("y", type="ordinal"))
+        .encode(
+            alt.X("x", type="nominal"),
+            alt.Y("y", type="ordinal"),
+            alt.Size("s", type="nominal"),
+        )
     )
     dct = chart.to_dict()
     assert dct["encoding"]["x"]["type"] == "nominal"
     assert dct["encoding"]["y"]["type"] == "ordinal"
+    assert dct["encoding"]["size"]["type"] == "nominal"
+    assert "sort" not in dct["encoding"]["size"]
 
 
 @pytest.mark.parametrize(