I've opened https://github.com/pydata/pandas/pull/5283 to get started on this.