Clean up matrix state check when the build matrix contains allowed failures

We've had quite a few exceptions when an allowed_failure job updates the matrix state.

When a job finishes, it checks if we are waiting for the matrix to finish, but this was always `true` if the build had `fast_finish` on. As a result, we would be checking the build statuses of *all* builds (which is wrong), which leads to `Build::InvalidMatrixStateException`.