[JENKINS-8597] Deal with matrix builds better

Fixes [JENKINS-8597] Deal with matrix builds better