Don't build upstream master branches

They're often not set up right for our build system