The problem is that we don't use the matplotlibs legend code but build our own: https://github.com/yhat/ggplot/blob/master/ggplot/components/legend.py