[JENKINS-14575] Fix regression and add tests as well