dmwm/WMCore — pull request #5491 by mmascher (2014-11-26 21:19:24+00:00)
Title: Review lumiCorrection function in the Lumibased Splitting

Do the lumi correction on the fly. Keep track of the already processed
lumis in a dict, and skip them it they were already encountered. Also,
keep track of the input files of the duplicated lumis, and add these input
files to the "original" job.

In the tests I did with my CRAB3 instance on the problematic datasets/splitting parameters, things have improved dramaticly. The previous algorithm was taking hours to complete and not it finishes pretty quickly (< 1 minute).

One other improvement is that if you want 50 lumis per job you will really have 50 lumis per job, while before you were removing lumis and ending up with jobs analyzing less than 50 lumis.

Finally, I have also checked that the issue raised by Andrew Levin (removing all the lumis in one job and ending up analyzing the whole dataset) is not there anymore. I added a unit test for it.

I made the check optional, and I am using the default as False, so unless explicitly enabled WMAgent will not make the check about duplicated lumis.