Tweak how we determine build status

Presence of the build matrix when can confuse build status
under some circumstances