Remove previous version pandas testing from travis matrix

As sent out to the mailing list for opinions this would implement that change.