[FIXED JENKINS-24020] Fail-fast Matrix builds, as soon some configuration fail

I'm not sure a public method is required. If yes, IOException should be documented.

Please also make sure that a call of this method from parallel threads is safe