Repository: vega/altair

Issue:
Support of ordinal based on pandas' ordered Categorical type?
I've just started to play with altair, using the [diamonds](http://vincentarelbundock.github.io/Rdatasets/datasets.html) dataset. Here is the notebook to clarify what I did https://gist.github.com/pierre-haessig/09fa9268aa0a0e7d91356f681f96ca18

Since, I'm not familiar with altair, I maybe missed something, but I've got the feeling that ordered Categorical types from pandas are not supported.

Indeed, if I use a color='cut' encoding, when cut is a pandas Series with an _ordered_ category dtype, I get by default a nominal type of coloring (with "unordered" colors).

On the other hand, if I force the use of ordered with color='cut:O', I indeed get the ordered colored (the shades of green), but the _order is wrong_! (I get Fair, Good, Ideal, Premium, Very Good, while the correct order is 'Fair', 'Good', 'Very Good', 'Premium', 'Ideal', as manually defined in pandas' category)



Discussion:
Hi, thanks for the report!

Currently, because of a [bug](https://github.com/pydata/pandas/issues/10778) in behavior for categoricals in Pandas' `to_json` function, altair first [converts categorical types to strings](https://github.com/altair-viz/altair/blob/master/altair/utils/core.py#L152) before serializing a dataframe as JSON. Thus Altair only knows about alphabetical ordering.

This is something that we should figure out how to address. In the meantime, you can specify the order manually within the encoding with, e.g. `x=X('cut:O', scale=Scale(domain=['Fair', 'Good', 'Very Good', 'Premium', 'Ideal'])` I haven't tested that, but I think it should work.

@jakevdp thanks for the feedback. I had to adapt your suggestion, since it's the color I wants, rather than x.

this is what I end up with:

``` python
c = Chart(data_samp)
cut_cat = ['Fair', 'Good', 'Very Good', 'Premium', 'Ideal']
cut_scale = Scale(domain=cut_cat, type='ordinal')
c.mark_circle().encode(x='carat', y='price',
                       color=Color('cut:O', scale=cut_scale))
```

so there are two remaining issues:
1. I shouldn't have to specify that the scale is ordinal if I already say it in the data encoding. However, if I remove `type='ordinal'` from the def of Scale, I get black marks...
2. After specifying `type='ordinal'`, I get the shades of green, but the order is really messy (i.e.: order in the legend is back to alphabetical, and color order is mysterious), see plot below.

![vega](https://cloud.githubusercontent.com/assets/1010404/19677881/abd3a8b4-9a9b-11e6-8dee-a9e4d8170258.png)
!

Huh... that's not great. I'm not certain why Vega-Lite requires you to specify 'ordinal' in both places, or why the color order is so strange. Maybe @kanitw would have ideas?

Having the same issue. See [this notebook](https://github.com/dhimmel/biorxiv-licenses/blob/8af02ba1cbdc3dd1c8ca179b5576b341fdd47b52/1.analysis.ipynb) where I use nominal ordering as a workaround to the [color issue](https://github.com/altair-viz/altair/issues/245#issuecomment-255968124):

![vega](https://cloud.githubusercontent.com/assets/1117703/20689191/05107662-b592-11e6-9075-60739ae18fd2.png)

Another issue arises where the legend ordering doesn't match the ordering in the plot. The legend is in the right order, but the area marks are incorrectly ordered.
I think this would be worth posting as a bug to Vega-Lite itself.
@pierre-haessig -- Thanks for reporting. This is definitely a bug. 

I just created https://github.com/vega/vega-lite/issues/1732.

The issue contains a workaround for this: using nominal type like @dhimmel suggests and set custom color range manually. (You might find [colorbrewer](http://colorbrewer2.org) useful.) 

We will make sure to fix this for the 2.0 release.  (We probably won't fix this in 1.x since it should be very easy to fix this in Vega 3, but quite complicated to do this in Vega 2.  Since we have a temporary workaround, we will focus our efforts on 2.0 development.) 
@kanitw I posted https://github.com/vega/vega-lite/issues/1732#issuecomment-263454554 before I saw the previous comment.

> The issue contains a workaround for this: using nominal type like @dhimmel suggests

It's not quite a workaround because the marks (bands) are not in the right order? Is there a way to fix that?
For other people following this issue, here is [a workaround](https://github.com/vega/vega-lite/issues/1734#issuecomment-263481177) for the following question.

> It's not quite a workaround because the marks (bands) are not in the right order? Is there a way to fix that?
FWIW the to_json segfault issue for Categorical dtype appears to be fixed in pandas: https://github.com/pandas-dev/pandas/pull/12802
@dsaxton Did you PR work for using pandas categoricals? I would love this functionality i Altair and it seems promising that the pandas json bug has been fixed!

> @dsaxton Did you PR work for using pandas categoricals? I would love this functionality i Altair and it seems promising that the pandas json bug has been fixed!

It seemed to be working, although I didn't do any testing outside of Altair's CI
Just to add another example where this issue is limiting.

The [Stacked Bar Chart example with Sorted Segments ](https://altair-viz.github.io/gallery/stacked_bar_chart_sorted_segments.html) in the Example's gallery doesn't sort if one uses custom ordering on a pandas categorical data type.

```python
from vega_datasets import data

source = data.barley()

# custom ordering of categorical data type
site_lst = ['Crookston', 'Morris', 'University Farm','Duluth', 'Grand Rapids', 'Waseca']
source.site = pd.Categorical(source.site, site_lst, ordered = True)

# The stacks are not ordered according to the 'site' variable ordering
# They use alphabetical sort by default and I have no idea how to alter this to work.
alt.Chart(source).mark_bar().encode(
    x='sum(yield)',
    y='variety',
    color='site',
    order=alt.Order(
      # Sort the segments of the bars by this field
      'site',
      sort= 'ascending'  
    )
)
```

![image](https://user-images.githubusercontent.com/76084890/102465487-b8285600-4073-11eb-85d2-f0db60c0f0b6.png)

Things I have tried to no avail:
1. I tried using `sort = None` as an argument for `alt.Color` but that doesn't work for correcting the stack either.
2. `alt.Order` doesn't accept a list; only accepts `"ascending"` or `"descending"`
3. using the sort argument with the list `site_lst` in `alt.X` doesn't work either

@sadzart Custom sorting is possible, by accessing undocumented created new fields during compilation.. probably not recommended. (related to https://github.com/vega/vega-lite/issues/1734#issuecomment-533223530)
```python
import altair as alt
import pandas as pd
from vega_datasets import data

source = data.barley()

# custom ordering of categorical data type
site_lst = ['Crookston', 'Morris', 'University Farm','Duluth', 'Grand Rapids', 'Waseca']
source.site = pd.Categorical(source.site, site_lst, ordered = True)

alt.Chart(source).mark_bar().encode(
    x='sum(yield)',
    y='variety',
    color=alt.Color('site', sort=alt.Sort(site_lst)),
    order=alt.Order('color_site_sort_index:Q',
      sort='ascending'  
    )
)
```
<img width="616" alt="image" src="https://user-images.githubusercontent.com/5186265/102685040-e9bd3080-41dd-11eb-8910-88c61ac0f0bb.png">

By introducing a `sort` for the `color` channel, the `color_site_sort_index` is added as a new field/column in the dataset. You'll have to use `:Q` to assign type manually in the `order` channel, since Altair cannot defer the type of a yet unexisting field. 

To observe what happens to your chart you can inspect the data-viewer in the Vega editor.
[Open the Chart in the Vega Editor](https://vega.github.io/editor/#/url/vega-lite/N4Igxg9gdgZglgcxALlANzgUwO4tJKAFzigFcJSBnAdTgBNCALFAFgAY2AacaYsiygAlMiRoRQBmDgF9p3OgENCCvCCgKAtphQhFygLRsAnBMxhMADjASArACM7AJjosAjMZZgYdgGyuJFmYW7nYgciAaCgBOANY6dtEg3JhQkHQkSKg8ADYQUaqEAJ4ADtrIahAaJArZSSDwmNl0OpRwhNrclHniyADaIADCURAQMZSE0HUAsnlRcJR1AKpQcGiYUa1FAAQAYtEadQAipNmkTHUA4lEKUHRbAEoKxfQL3NQKlGYqALrheXTrAolMoVKrqWrcBpNFptDogLpRHogD7mW4ZMLcAAeQNKOgAjqQbsRlMQ1nUFAgEFFMAglCDKKQDpCsNDyoUWc1woUcSCoJVqhD6hydGholgimFwgASShgRiYSI6MSEYqUZAAenVa1pADoEG1GKQ7Dq4BB1bL5ZEtTSFPpsrCtSwdRYda4dQArLpQOp6D6YQgLLK+wwmMyWaz2JwudxGTzePwBIIhFC9UDsxrNZCOADs3FFc393PKUxuctIcxU3EKmESyFcJlcnVhOmWq3Wm0Ku32GLTwuQLBdFh8w9zIHz4qLIBLqUNFbq1dr9YkjfhzfK70+YBUcl7GZQOZ1LAkx5YebFhZ007Lc6rNfydYbTfal9m8x7IHTrIkRh1JmPEjPAsJWLUtZzgSsPzvFAlxXTYQSGEYxgmb0dw-PsJEcX9hx8UdxwvECZ3LcD5ygh9lyfEErhuO5HmeOgFlQz9M0cF0jGw3Dz2AqdQKIiCF3vGCKJ0Y5TnORi+yPHU2GwnxAInHQLmyBQ7GGb1b0XR9V2fcpWzWDY2k7PYogOcS92QGwbB1Rw5PwkBFOU1SSI08itJBDcvnfJj9xdbN2Jsrj7JUyZ1IEzS4JfKI5gYzhdy-F1-H-fzJ0CxyQugsK10GYZRnGSZTNZRwfwS48koUpSgrUyDnNgzKqNuB4nheTy+0KnUZI4oDkvK1KqtClzwvKESzmYfLM1sV1-wAsdOMnABlUUoC6ZQnL6mrtJAXT2wMrtjOasyWGzHVJqmvCuPmm4lr40jBNcnR3K3PaCss3yR1K8pzsWiBlrSsi1pBGZIrfUbWDYQ8-OmzqdA+y6VvS-rMoQnLkMe5if3at6QGhr6ruqoTyjqmjGvolH92ejGse+3q4b+4STmGknkG-X8MYANUaNZxB+m6Bo2lY9I7HaTJitCzJsUHHEm1n2f9WHfrxkB7u3YWvKzHwJsSiH5PKNnsg52XucygGooZtwjslzXbJ1vWuYy9bEaQvLlZaiQpPN07JytmWbfh9aCYaujoti5i1ZOmadE9zmqbl27BrpsSnbMiQ1ZsGSMYAFWpOw4H122QU2-TtiMoWg5QHwXYsDX3Z0DPMCznOfbcv0HuB-sXZenD08z7PvZp4tX0DkXWRYNW-xKi2uJruue-l+3cpQhOCp-duOq1kBJ+7qODd9656topqW+PX83bD8p1-r3uQCG+OS9bqzwar8oADkIB1LYWBsbNz-l-OBaLhmbHihjZ+r936f2njHBWTclY3xYjqFetlgFvw-l-CBRsgYL0zO-Nq98T4gEQaAlBPNZ7IwPphVwODIZPxfkgsBm9c4KR3oTAODND6yXHpOfByDwE8yviNDBkg1ZsIfng6hw9HCEMyj-baf8W4+Geq9dhOh8E+HEdwzKisWGgyTgo4RyjVF0IbhFY2LcBw6hTthIBoiVESLttlB288YFOlHqeRRVCQHWLUdvai-t978KzIA1xIj3H6P4tTeWvCWGYWXhjAACjWQgSlbg2LznzLahduwmJdOY4csT4mJOaJ4xum5oGD1Rs6Chq84lKHyckox6Cb6myEbgqpCTqK1PKMQx2N8JBOngVxFpNTCkMO8XvYmpDcnVLaUM2Ook+GOOwTo3B+Dsw2HabzNsBdDIZL8cPOBFSEGiJWWsjRLdHCYSaZQoJb8jnTKnP3E2ToVy6MOas25nSHGlPSnAyxICbkGIvn7UZA8VaH2khYwJyzXn-PCXHOZnzGY-lDpc2gspoCtCgFsRBAQ1lSPSbtFuADnQYxRQQdFmLqHYtuScvxrVh6LORfMUlJByWv0pdC1B9yTE-iHPS1eJK0XMqxRYNZ7yWFOjpR3QJ-LFqCopcK25gKibAvQm6C5fLGUCoxUKtZETTkjwxleMCOM+rWQgbirZ+K-GHwlX0ychreL11NTzal3SnTaJyYE+1N4DFOsNpyq1mE2K8tsl64i3tfW2MQnPBmZz1Zj2EaG41cMI2UUYT4sZNLMLZNtZeHi3rQlkRTbTWZLC1YWAxilYKPrv6pM2YLFhh1szH0uZWyqBalxFvXFAlhllipItXq2x18s0HKrMo4NW5Dg0BW6lW9tJhO1ZSjSQvxrhxXNoHTOtt1153y0Vcw05lkeWSuEYO8NMKS2nMOkeE85MFowzPWa2tv9tndKybei62Mh0QJdfC8abB122Qpkmwtw7-UwNBmXSuuCgNfqIXY6NLdyFtQA2dO9n6H08z3b4mBmEJZQcuTBjDmVdU0rLVLXWXtq2Po2c+y13Sr3ketlR513aD6DhQx7aWkc50YVA4DUdX5MJgo9cIiOsGEbweXd03DHHw5cfE143eSqY2YRtYxyjPGF0kZgT+YTObT5dwUykmj0iX3wpYD+PD-bbJnyI+tH9KtdmQfjbg2zzG-X8ZNm6Cu+HV5uc0zPSTXT4WOAg7JgztcN4BYgVhjN3SIOd0i0Z4t9M9WHn-C455ICuHufWua+tJjcM-JoclrtxSTYuxsBl4rBC7P-TA7+t14Wrm1dy-BILHyVb1jBlOjhoicvRcw2moFKnvkQv67QwbxHYUxsstmmrHi2stifaZuj5nekLZCdu3j37WM7P0y1xbU31ojs0WY5rejSuLqRsFrrP4jCbau7FgTzEs2Pbqyl6+IWXR6Yma0pJH2dIrbxcXX9at-2+dsgMqZS2yseRMS7U1wjocA9h3czzB6rLNZRwUtHoq0vRMCTjp7w3lMHzdO6g7xPAeXxmwfLg43flQuO8Z-mq3QeObdOO3rSiXlXYcxJJ0VnrNcUhVd07pC40i760zq7+OV2WWFzVv5LPhlKf3TSw6lPlfM+21punVry1So1TKrVcqrv5ZkX4wlk7j24OlWS7VNOBf7UOrbg7DvZWsvlWjiXVrLIPeN6i03LKthstVx0jrMaIP7K4p7s33uScjLJzSxFzX4+h-D3r89qXfjSCAA)
@mattijn Worked like a charm. Thanks
That's a really neat solution @mattijn - but I'm having trouble with my dataset. I can change the legend order, but the order on the actual plot doesn't change.

Here's a minimal example:

```
import altair as alt
import pandas as pd

dfdict = {
    "Questions": {
        0: "Question Text",
        1: "Question Text",
        2: "Question Text",
        3: "Question Text",
        4: "Question Text",
        5: "Question Text",
    },
    "level": {
        0: "1 - Strongly Disagree",
        1: "2 - Disagree",
        2: "3 - Neutral",
        3: "4 - Agree",
        4: "5 - Strongly Agree",
        5: "N/A",
    },
    "value": {0: 1.4, 1: 5.7, 2: 10.0, 3: 32.9, 4: 47.1, 5: 2.9},
}

df = pd.DataFrame(dfdict)

sort_order = [
    "N/A",
    "5 - Strongly Agree",
    "4 - Agree",
    "3 - Neutral",
    "2 - Disagree",
    "1 - Strongly Disagree",
]

# This doesn't seem to be needed
df["level"] = pd.Categorical(df["level"], sort_order, ordered=True)

chart = (
    alt.Chart(df)
    .mark_bar()
    .encode(
        x=alt.X("value"),
        y=alt.Y("Questions", title=""),
        color=alt.Color("level", sort=alt.Sort(sort_order)),
        order=alt.Order("color_site_sort_index:Q"),
    )
)

chart
```
<img width="764" alt="Screen Shot 2021-03-07 at 5 32 25 PM" src="https://user-images.githubusercontent.com/2507459/110263290-1e35c280-7f6b-11eb-9aad-176a7cb30fb2.png">

Ideal order would be - from left to right - N/A, 5 - Strongly Agree, 4 - Agree, etc... What I am able to accomplish is my ideal order from right to left. Inverting the sort_order list just changes the legend order, but the order the data is plotted.

FYI: I know this is an issue with my plot because I can reproduce your example just fine.

Thanks in advance

The syntax of the undocumented feature is `"color_<column-name>_sort_index"`, so if you change your example to `"color_level_sort_index:O"`, it works.
Whoops!! Good catch! 

Thanks that works. 

Full code for someone looking to reproduce:

```
import altair as alt
import pandas as pd

dfdict = {
    "Questions": {
        0: "Question Text",
        1: "Question Text",
        2: "Question Text",
        3: "Question Text",
        4: "Question Text",
        5: "Question Text",
    },
    "level": {
        0: "1 - Strongly Disagree",
        1: "2 - Disagree",
        2: "3 - Neutral",
        3: "4 - Agree",
        4: "5 - Strongly Agree",
        5: "N/A",
    },
    "value": {0: 1.4, 1: 5.7, 2: 10.0, 3: 32.9, 4: 47.1, 5: 2.9},
}

df = pd.DataFrame(dfdict)

sort_order = [
    "N/A",
    "5 - Strongly Agree",
    "4 - Agree",
    "3 - Neutral",
    "2 - Disagree",
    "1 - Strongly Disagree",
]

# This doesn't seem to be needed
df["level"] = pd.Categorical(df["level"], sort_order, ordered=True)

chart = (
    alt.Chart(df)
    .mark_bar()
    .encode(
        x=alt.X("value"),
        y=alt.Y("Questions", title=""),
        color=alt.Color("level", sort=alt.Sort(sort_order)),
        order=alt.Order("color_level_sort_index:Q"),
    )
)

chart
```
<img width="839" alt="Screen Shot 2021-03-07 at 7 24 42 PM" src="https://user-images.githubusercontent.com/2507459/110270487-c606bc80-7f7a-11eb-9af6-7f64f3f18277.png">


Resolving patch:
diff --git a/altair/utils/core.py b/altair/utils/core.py
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -193,8 +193,6 @@ def infer_vegalite_type(data):
     # Otherwise, infer based on the dtype of the input
     typ = infer_dtype(data)
 
-    # TODO: Once this returns 'O', please update test_select_x and test_select_y in test_api.py
-
     if typ in [
         "floating",
         "mixed-integer-float",
@@ -203,6 +201,8 @@ def infer_vegalite_type(data):
         "complex",
     ]:
         return "quantitative"
+    elif typ == "categorical" and data.cat.ordered:
+        return ("ordinal", data.cat.categories.tolist())
     elif typ in ["string", "bytes", "categorical", "boolean", "mixed", "unicode"]:
         return "nominal"
     elif typ in [
@@ -316,8 +316,9 @@ def to_list_if_array(val):
 
     for col_name, dtype in df.dtypes.items():
         if str(dtype) == "category":
-            # XXXX: work around bug in to_json for categorical types
+            # Work around bug in to_json for categorical types in older versions of pandas
             # https://github.com/pydata/pandas/issues/10778
+            # https://github.com/altair-viz/altair/pull/2170
             col = df[col_name].astype(object)
             df[col_name] = col.where(col.notnull(), None)
         elif str(dtype) == "string":
@@ -527,6 +528,10 @@ def parse_shorthand(
     if isinstance(data, pd.DataFrame) and "type" not in attrs:
         if "field" in attrs and attrs["field"] in data.columns:
             attrs["type"] = infer_vegalite_type(data[attrs["field"]])
+            # ordered categorical dataframe columns return the type and sort order as a tuple
+            if isinstance(attrs["type"], tuple):
+                attrs["sort"] = attrs["type"][1]
+                attrs["type"] = attrs["type"][0]
     return attrs
 
 
diff --git a/altair/utils/schemapi.py b/altair/utils/schemapi.py
--- a/altair/utils/schemapi.py
+++ b/altair/utils/schemapi.py
@@ -364,6 +364,14 @@ def to_dict(self, validate=True, ignore=None, context=None):
             # parsed_shorthand is removed from context if it exists so that it is
             # not passed to child to_dict function calls
             parsed_shorthand = context.pop("parsed_shorthand", {})
+            # Prevent that pandas categorical data is automatically sorted
+            # when a non-ordinal data type is specifed manually
+            if "sort" in parsed_shorthand and kwds["type"] not in [
+                "ordinal",
+                Undefined,
+            ]:
+                parsed_shorthand.pop("sort")
+
             kwds.update(
                 {
                     k: v
diff --git a/tools/schemapi/schemapi.py b/tools/schemapi/schemapi.py
--- a/tools/schemapi/schemapi.py
+++ b/tools/schemapi/schemapi.py
@@ -362,6 +362,14 @@ def to_dict(self, validate=True, ignore=None, context=None):
             # parsed_shorthand is removed from context if it exists so that it is
             # not passed to child to_dict function calls
             parsed_shorthand = context.pop("parsed_shorthand", {})
+            # Prevent that pandas categorical data is automatically sorted
+            # when a non-ordinal data type is specifed manually
+            if "sort" in parsed_shorthand and kwds["type"] not in [
+                "ordinal",
+                Undefined,
+            ]:
+                parsed_shorthand.pop("sort")
+
             kwds.update(
                 {
                     k: v


Test patch:
diff --git a/tests/vegalite/v5/tests/test_api.py b/tests/vegalite/v5/tests/test_api.py
--- a/tests/vegalite/v5/tests/test_api.py
+++ b/tests/vegalite/v5/tests/test_api.py
@@ -123,6 +123,7 @@ def test_chart_infer_types():
             "x": pd.date_range("2012", periods=10, freq="Y"),
             "y": range(10),
             "c": list("abcabcabca"),
+            "s": pd.Categorical([1, 2] * 5, categories=[2, 1], ordered=True),
         }
     )
 
@@ -134,32 +135,45 @@ def _check_encodings(chart):
         assert dct["encoding"]["y"]["field"] == "y"
         assert dct["encoding"]["color"]["type"] == "nominal"
         assert dct["encoding"]["color"]["field"] == "c"
+        assert dct["encoding"]["size"]["type"] == "ordinal"
+        assert dct["encoding"]["size"]["field"] == "s"
+        assert dct["encoding"]["size"]["sort"] == [2, 1]
 
     # Pass field names by keyword
-    chart = alt.Chart(data).mark_point().encode(x="x", y="y", color="c")
+    chart = alt.Chart(data).mark_point().encode(x="x", y="y", color="c", size="s")
     _check_encodings(chart)
 
     # pass Channel objects by keyword
     chart = (
         alt.Chart(data)
         .mark_point()
-        .encode(x=alt.X("x"), y=alt.Y("y"), color=alt.Color("c"))
+        .encode(x=alt.X("x"), y=alt.Y("y"), color=alt.Color("c"), size=alt.Size("s"))
     )
     _check_encodings(chart)
 
     # pass Channel objects by value
-    chart = alt.Chart(data).mark_point().encode(alt.X("x"), alt.Y("y"), alt.Color("c"))
+    chart = (
+        alt.Chart(data)
+        .mark_point()
+        .encode(alt.X("x"), alt.Y("y"), alt.Color("c"), alt.Size("s"))
+    )
     _check_encodings(chart)
 
     # override default types
     chart = (
         alt.Chart(data)
         .mark_point()
-        .encode(alt.X("x", type="nominal"), alt.Y("y", type="ordinal"))
+        .encode(
+            alt.X("x", type="nominal"),
+            alt.Y("y", type="ordinal"),
+            alt.Size("s", type="nominal"),
+        )
     )
     dct = chart.to_dict()
     assert dct["encoding"]["x"]["type"] == "nominal"
     assert dct["encoding"]["y"]["type"] == "ordinal"
+    assert dct["encoding"]["size"]["type"] == "nominal"
+    assert "sort" not in dct["encoding"]["size"]
 
 
 @pytest.mark.parametrize(