ENH: Speed up legendre tests

Some progress toward #1227.