MAINT: Remove old pandas compat code.

I think there may be more simplifications that are possible in the time series code. Opening this so I don't forget about it.