BUG: Correct issue if patsy handles missing. Closes #2083.

why does this raise?
I thought this would just break somewhere with the nans, if `missing` is default (not `'drop'`)