Allow build exclusion from incomplete specifications

Fixes travis-ci/travis-ci#1492