Attempt to skip unnecessary builds on pull request

This is an attempt to skip some builds for pull requests to speed up the testing.