I am not suggesting we move to astropy table just for the want to make pandas an optional dependency, although that is a big plus in my eyes.

IMO, astropy table provides a couple of key benefits, the largest is the fact that it is a very lightweight wrapper around a numpy array, meaning it should be faster than initialising a DataFrame as well as easier to understand.

The Input/Output routines designed for astropy table are much more suited for our needs.

And finally I see no loss of functionality to the end user, we will make it just as easy as it currently is to get at a pandas data frame, and if in the future there is something that we want to use pandas for we can always go fully dual-mode where we keep the pandas dependency.