MAINT: in setup.py, only catch ImportError for pandas.

See for example gh-617.