Implement a build matrix for travis to test with/without numpy