Stop repeated rebuilds on Travis

Fixes #2114, on Travis at least.