I think Categorical was the thing that throws.

Is there a neat way to just include only numerics from a Series?

Maybe call `numeric_only` like groupby: https://github.com/pydata/pandas/commit/dda23636354136f753561ead07bdbe88b21fa5ae. Not sure it should be default though.