For the record, `astropy.table` can't handle this.  Maybe I'll create an issue to get @taldcroft's thoughts about adding it.