Allow build from index

Unfortunately, this isn't a great test because it is pointing at wheelhouses that contain wheels for the matplotlib dependencies. 

Later today can setup a wheelhouse without them and give it a try...