[FIXED JENKINS-24914] Don't block indefinitely every other month