anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: dmwm/wmcore
Issue: ReqMgr2: OutputDatasets is not updated during assignment

in the reqmgr_workload_cache doc. It seems to keep the same OutputDatasets built during request creation and that's it. See example:
https://cmsweb-testbed.cern.ch/couchdb/_utils/document.html?reqmgr_workload_cache/pdmvserv_HIG-RunIISummer15GS-01331_00274_v2__160308_111346_296


Changed files:
src/python/WMCore/ReqMgr/Service/Request.py

diff --git a/src/python/WMCore/ReqMgr/Service/Request.py b/src/python/WMCore/ReqMgr/Service/Request.py
index 2da1fdee2..c65b898ea 100644
--- a/src/python/WMCore/ReqMgr/Service/Request.py
+++ b/src/python/WMCore/ReqMgr/Service/Request.py
@@ -411,7 +411,8 @@ class Request(RESTEntity):
         
         # legacy update schema to support ops script
         loadRequestSchema(workload, request_args)
-        # trailing / is needed for the savecouchUrl function
+        #update OutputDatasets after ProcessingString and AcquisionEra is updated
+        request_args['OutputDatasets'] = workload.listOutputDatasets()
         report = self.reqmgr_db_service.updateRequestProperty(workload.name(), request_args, dn)
         workload.saveCouch(self.config.couch_host, self.config.couch_reqmgr_db)
         return report
Repository: dmwm/wmcore
Issue: time.sleep delaying Agent components

The polling cycle of some components in the WMAgent is taking too long. The cause is this line in the Dashboard Reporter:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/Dashboard/DashboardAPI.py#L85
It is called every time that a job report is sent to Dashboard. Hundreds of jobs are reported per polling cycle when the agent has a big load, causing the delays that we observed. 


Changed files:
src/python/WMCore/Services/Dashboard/DashboardAPI.py

diff --git a/src/python/WMCore/Services/Dashboard/DashboardAPI.py b/src/python/WMCore/Services/Dashboard/DashboardAPI.py
index 6ce3561ba..92790f725 100644
--- a/src/python/WMCore/Services/Dashboard/DashboardAPI.py
+++ b/src/python/WMCore/Services/Dashboard/DashboardAPI.py
@@ -82,7 +82,6 @@ def apmonFree():
     global APMONINSTANCE
     global APMONINIT
     if APMONINSTANCE is not None:
-        time.sleep(1)
         try:
             APMONINSTANCE.free()
         except Exception:
Repository: dmwm/wmcore
Issue: dbsReader listFilesInBlockWithParents crash on RAW : dataset with no parents

Whilte trying to bring https://github.com/CMSCompOps/WmAgentScripts/blob/master/recover/recoverMissingLumis.py to have parent info

Calling 

blockFileParents = dbsReader.listFilesInBlockWithParents(blockName)

one gets 

/DoubleEG/Run2015D-v1/RAW#fceb85c4-5ce3-11e5-824b-001e67ac06a0

Traceback (most recent call last):
  File "recover/recoverMissingLumis.py", line 666, in <module>
    sys.exit(main())
  File "recover/recoverMissingLumis.py", line 663, in main
    val.group, val.dbsUrl, val.fake, datasetInformation)
  File "recover/recoverMissingLumis.py", line 410, in defineRequests
    blockBlacklist, blockWhitelist, dbsUrl, fakeLocation=fakeLocation)
  File "recover/recoverMissingLumis.py", line 246, in getFiles
    blockFileParents = dbsReader.listFilesInBlockWithParents(blockName)
  File "/data/srv/wmagent/v1.0.17.pre4/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.pre4/lib/python2.7/site-packages/WMCore/Services/DBS/DBS3Reader.py", line 528, in listFilesInBlockWithParents
    parentLumis = self._getLumiList(lfns = parentsLFNs)
  File "/data/srv/wmagent/v1.0.17.pre4/sw/slc6_amd64_gcc493/cms/wmagent/1.0.17.pre4/lib/python2.7/site-packages/WMCore/Services/DBS/DBS3Reader.py", line 82, in _getLumiList
    for lumisItem in lumiLists:
UnboundLocalError: local variable 'lumiLists' referenced before assignment

admittedly, the block has no parent, it might not be a reason to crash.


Changed files:
src/python/WMCore/Services/DBS/DBS3Reader.py

diff --git a/src/python/WMCore/Services/DBS/DBS3Reader.py b/src/python/WMCore/Services/DBS/DBS3Reader.py
index 9b82361c1..6e372a889 100644
--- a/src/python/WMCore/Services/DBS/DBS3Reader.py
+++ b/src/python/WMCore/Services/DBS/DBS3Reader.py
@@ -74,6 +74,10 @@ class DBS3Reader:
                 lumiLists = []
                 for slfn in grouper(lfns, 50):
                     lumiLists.extend(self.dbs.listFileLumiArray(logical_file_name = slfn))
+            else:
+                # shouldn't call this with both blockName and lfns empty
+                # but still returns empty dict for that case
+                return {}
         except dbsClientException as ex:
             msg = "Error in "
             msg += "DBSReader.listFileLumiArray(%s)\\n" % lfns
@@ -531,7 +535,14 @@ class DBS3Reader:
             # Probably a child can have more than 1 parent file
             for fp in f['parent_logical_file_name']:
                 childByParents[fp].append(f['logical_file_name'])
+        
         parentsLFNs = childByParents.keys()
+        
+        if len(parentsLFNs) == 0:
+            msg = "Error in "
+            msg += "DBSReader.listFilesInBlockWithParents(%s)\\n There is no parents files" % (
+                fileBlockName)
+            raise DBSReaderError(msg)
 
         parentFilesDetail = []
         #TODO: slicing parentLFNs util DBS api is handling that.
Repository: dmwm/wmcore
Issue: wrong total lumi calculation in task chain.

Review the code below.

https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WorkQueue/Policy/Start/MonteCarlo.py#L58

Following gives double total lumis than what it supposed to be.
https://cmsweb.cern.ch/couchdb/reqmgr_workload_cache/pdmvserv_task_TOP-RunIIWinter15wmLHE-00015__v1_T_150404_183555_8973


Changed files:
src/python/WMCore/WorkQueue/Policy/Start/MonteCarlo.py

diff --git a/src/python/WMCore/WorkQueue/Policy/Start/MonteCarlo.py b/src/python/WMCore/WorkQueue/Policy/Start/MonteCarlo.py
index 2f7eb8e6b..3774e7f48 100644
--- a/src/python/WMCore/WorkQueue/Policy/Start/MonteCarlo.py
+++ b/src/python/WMCore/WorkQueue/Policy/Start/MonteCarlo.py
@@ -5,7 +5,7 @@ WorkQueue splitting by block
 """
 __all__ = []
 
-
+from __future__ import division
 
 from WMCore.WorkQueue.Policy.Start.StartPolicyInterface import StartPolicyInterface
 from WMCore.WorkQueue.WorkQueueExceptions import WorkQueueWMSpecError, WorkQueueNoWorkError
@@ -51,15 +51,11 @@ class MonteCarlo(StartPolicyInterface):
 
             #Calculate the job splitting without actually doing it
             nEvents = mask['LastEvent'] - mask['FirstEvent'] + 1
-            lumis_per_job = ceil(self.args['SliceSize'] /
-                                 float(self.args['SubSliceSize']))
-            remainingEvents = nEvents % self.args['SliceSize']
-            lumis = (nEvents / self.args['SliceSize']) * lumis_per_job
-            lumis += ceil(remainingEvents / float(self.args['SubSliceSize']))
-            jobs = ceil(lumis/lumis_per_job)
-
-            mask['LastLumi'] = mask['FirstLumi'] + int(lumis) - 1 # inclusive range
-            nLumis = mask['LastLumi'] - mask['FirstLumi'] + 1
+            lumis_per_job = ceil(self.args['SliceSize'] / self.args['SubSliceSize'])
+            nLumis = ceil(nEvents / self.args['SubSliceSize'])
+            jobs = ceil(nEvents/self.args['SliceSize'])
+
+            mask['LastLumi'] = mask['FirstLumi'] + int(nLumis) - 1 # inclusive range
             self.newQueueElement(WMSpec = self.wmspec,
                                  NumberOfLumis = nLumis,
                                  NumberOfEvents = nEvents,
Repository: dmwm/wmcore
Issue: Resubmission crashes on "old" requests with unknown key args

As resubmission is handled now, it will break when the original request has arguments that it "should not", because we simply copy those over to the new request:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/ReqMgr/DataStructs/Request.py#L74

issue reported here:
https://github.com/CMSCompOps/WmAgentScripts/issues/222

Changed files:
src/python/WMCore/WMSpec/StdSpecs/StdBase.py

diff --git a/src/python/WMCore/WMSpec/StdSpecs/StdBase.py b/src/python/WMCore/WMSpec/StdSpecs/StdBase.py
index 1d3d12034..a6fb7d540 100644
--- a/src/python/WMCore/WMSpec/StdSpecs/StdBase.py
+++ b/src/python/WMCore/WMSpec/StdSpecs/StdBase.py
@@ -843,11 +843,7 @@ class StdBase(object):
         if 'OriginalRequestName' not in arguments:
             self.masterValidation(schema=arguments)
             self.validateSchema(schema=arguments)
-        elif arguments.get("RequestType") == "Resubmission":
-            # Don't validate schema for cloned workflow
-            # But Resubmission workflow has to be validated
-            self.validateSchema(schema=arguments)
-        
+
         workload = self.__call__(workloadName=workloadName, arguments=arguments)
         self.validateWorkload(workload)
Repository: dmwm/wmcore
Issue: Multicore at assignment time for ACDC

I thought Multicore could be set during assignment and we went ahead like this for 

https://cmsweb.cern.ch/reqmgr2/fetch?rid=mcremone_recovery-3-fabozzi_Run2016H-v1-ZeroBiasIsolatedBunch0-09Nov2016_8023__161211_012840_7645

but it did not work. This was assigned with 

{'AcquisitionEra': u'Run2016H',
 'CustodialSites': [],
 'Dashboard': 'reprocessing',
 'MaxMergeEvents': 50000,
 'MaxMergeSize': 4294967296,
 'MaxVSize': 4294967296,
 'Memory': 50000,
 'MergedLFNBase': u'/store/data',
 'MinMergeSize': 2147483648,
 'Multicore': 24,
 'ProcessingString': u'09Nov2016',
 'ProcessingVersion': 1,
 'SiteBlacklist': [],
 'SiteWhitelist': ['T2_US_Nebraska'],
 'SoftTimeout': 159600,
 'Teamproduction': 'checked',
 'TrustSitelists': True,
 'UnmergedLFNBase': '/store/unmerged',
 'action': 'Assign',
 'checkboxmcremone_recovery-3-fabozzi_Run2016H-v1-ZeroBiasIsolatedBunch0-09Nov2016_8023__161211_012840_7645': 'checked',
 'execute': True}

so, is this supposed to work ?
FYI @mcremone

Changed files:
src/python/WMCore/WMSpec/WMWorkload.py

diff --git a/src/python/WMCore/WMSpec/WMWorkload.py b/src/python/WMCore/WMSpec/WMWorkload.py
index 5b89dbf90..6ce0d03c1 100644
--- a/src/python/WMCore/WMSpec/WMWorkload.py
+++ b/src/python/WMCore/WMSpec/WMWorkload.py
@@ -628,17 +628,31 @@ class WMWorkloadHelper(PersistencyHelper):
 
         return
 
-    def setCoresAndStreams(self, cores, nStreams):
+    def setCoresAndStreams(self, cores, nStreams, initialTask=None):
         """
-        _setCores_
+        _setCoresAndStreams_
 
-        Update number of cores and event streams for each task in the spec
+        Update the number of cores and event streams for each task in the spec.
+
+        One can update only the number of cores, which will set the number of streams to 0 (default).
+        However, updating only the number of streams is not allowed, it's coupled to # of cores.
+
+        :param cores: number of cores. Can be either an integer or a dict key'ed by taskname
+        :param nStreams: number of streams. Can be either an integer or a dict key'ed by taskname
+        :param initialTask: parent task object
         """
         if not cores:
             return
 
-        for task in self.taskIterator():
+        if initialTask:
+            taskIterator = initialTask.childTaskIterator()
+        else:
+            taskIterator = self.taskIterator()
+
+        for task in taskIterator:
             task.setNumberOfCores(cores, nStreams)
+            self.setCoresAndStreams(cores, nStreams, task)
+
         return
 
     def setMemory(self, memory):
Repository: dmwm/wmcore
Issue: Merge jobs skipping ALL input files - exit code 60450

As reported by @prozober in this Jira: https://its.cern.ch/jira/browse/CMSCOMPPR-328

merge jobs are failing quite often in T2_AT_Vienna with exit code 60450, which basically means there was no output files in the report file (besides logArch). Taking a look at the condor logs, not all of them have errors, so these jobs should succeed. However, you can see there were no files merged and no merged file stage out (only the logArch).

I managed to find the following log in vocms0130
install/wmagent/JobArchiver/logDir/p/pdmvserv_EXO-RunIISummer15wmLHEGS-01970_00249_v0__161123_003233_2902/JobCluster_4127/Job_4127624.tar.bz2

which was not copied over here:
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7467/

I also unpickled the job config (job.pkl) and one of the reports (Report.3.pkl).

Looking at the job configuration, we can see the list of `input_files`, though I don't know if the structure/values are correct (especially the `'merged': True,`).

Then looking at the report file, we can see that all the unmerged files were skipped ?!?

Changed files:
src/python/WMCore/FwkJobReport/Report.py

diff --git a/src/python/WMCore/FwkJobReport/Report.py b/src/python/WMCore/FwkJobReport/Report.py
index d37e9ff43..72a0986e3 100644
--- a/src/python/WMCore/FwkJobReport/Report.py
+++ b/src/python/WMCore/FwkJobReport/Report.py
@@ -98,12 +98,12 @@ def addAttributesToFile(fileSection, **attributes):
 
     Add attributes to a file in the FWJR.
     """
-    for attName in attributes.keys():
-        setattr(fileSection, attName, attributes[attName])
+    for attName, attValue in attributes.items():
+        setattr(fileSection, attName, attValue)
     return
 
 
-class Report:
+class Report(object):
     """
     The base class for the new jobReport
 
@@ -531,7 +531,8 @@ class Report:
         newFile = getattr(analysisFiles, label)
         newFile.fileName = filename
 
-        [setattr(newFile, x, y) for x, y in attrs.items()]
+        for x, y in attrs.items():
+            setattr(newFile, x, y)
 
         analysisFiles.fileCount += 1
         return
@@ -549,7 +550,8 @@ class Report:
         removedFiles.section_(label)
         newFile = getattr(removedFiles, label)
 
-        [setattr(newFile, x, y) for x, y in attrs.items()]
+        for x, y in attrs.items():
+            setattr(newFile, x, y)
 
         self.report.cleanup.removed.fileCount += 1
         return
@@ -1503,6 +1505,9 @@ class Report:
         analysisFiles = self.getAnalysisFilesFromStep(step=stepName)
         if len(files) == 0 and len(analysisFiles) == 0:
             msg = WM_JOB_ERROR_CODES[60450]
+            msg += "\\nList of skipped files is:\\n"
+            for skipF in self.getSkippedFilesFromStep(stepName=stepName):
+                msg += "  %s\\n" % skipF
             self.addError(stepName, 60450, "NoOutput", msg)
             self.setStepStatus(stepName=stepName, status=60450)
         return
Repository: dmwm/wmcore
Issue: Changing priority in assignment-approved fails

https://cmsweb.cern.ch/reqmgr2/fetch?rid=pdmvserv_task_PPD-RunIISummer17PrePremix-00006__v1_T_180212_090017_6410 trying to set up to 111000 gives people

Server error message
Invalid spec parameter value: There are invalid arguments for no-status update: {u'SoftTimeout': 129600, u'BlockCloseMaxEvents': 25000000, u'MaxMergeEvents': 100000000, u'CustodialSites': u'', u'Ma... 


please advise, and prioritize a fix

Changed files:
src/python/WMCore/ReqMgr/Service/Request.py

diff --git a/src/python/WMCore/ReqMgr/Service/Request.py b/src/python/WMCore/ReqMgr/Service/Request.py
index bdec7be7d..bc2a3d85e 100644
--- a/src/python/WMCore/ReqMgr/Service/Request.py
+++ b/src/python/WMCore/ReqMgr/Service/Request.py
@@ -379,7 +379,9 @@ class Request(RESTEntity):
          1. RequestPriority
          2. Global workqueue statistics, while acquiring a workflow
         """
-        if 'RequestPriority' in request_args and len(request_args) == 1:
+        if 'RequestPriority' in request_args:
+            # Yes, we completely ignore any other arguments posted by the user (web UI case)
+            request_args = {'RequestPriority': request_args['RequestPriority']}
             # must update three places: GQ elements, workload_cache and workload spec
             self.gq_service.updatePriority(workload.name(), request_args['RequestPriority'])
             report = self.reqmgr_db_service.updateRequestProperty(workload.name(), request_args)
Repository: dmwm/wmcore
Issue: reqmgr_workload_cache db and wmstats reporting wrong # events

It looks like reqmgr_workload_cache db documents sometimes report an incorrect number of TotalInputEvents, TotalInputLumis and I guess also TotalEstimatedJobs, as can be seen in this example:
https://cmsweb.cern.ch/couchdb/reqmgr_workload_cache/pdmvserv_TRK-TP2023HGCALGS-00001_00001_v0__141113_160840_557

I've seen this in testbed already but I guess I did not manage to easily reproduce it.
Taking a quick look at the code, it seems it's coming from here:
https://github.com/dmwm/WMCore/blob/master/src/couchapps/ReqMgr/updates/totalstats.js#L17

but to be honest, I have no idea where/what "req.query.input_events" goes here.


Changed files:
src/python/WMCore/WorkQueue/WorkQueue.py

diff --git a/src/python/WMCore/WorkQueue/WorkQueue.py b/src/python/WMCore/WorkQueue/WorkQueue.py
index d0dd02866..4a0f915fe 100644
--- a/src/python/WMCore/WorkQueue/WorkQueue.py
+++ b/src/python/WMCore/WorkQueue/WorkQueue.py
@@ -1044,11 +1044,6 @@ class WorkQueue(WorkQueueBase):
                                                                      mask=inbound['Mask'], inbound=inbound,
                                                                      continuous=continuous)
 
-                    # if global queue, then update workflow stats to request mgr couch doc
-                    # remove the "UnittestFlag" - need to create the reqmgrSvc emulator
-                    if not self.params.get('LocalQueueFlag') and not self.params.get("UnittestFlag", False):
-                        self.reqmgrSvc.updateRequestStats(inbound['WMSpec'].name(), totalStats)
-
                     # save inbound work to signal we have completed queueing
                     self.backend.insertElements(work, parent=inbound)  # if this fails, rerunning will pick up here
 
@@ -1074,6 +1069,10 @@ class WorkQueue(WorkQueueBase):
                                 rejectedWork = rejectedWork[chunkSize:]
                                 chunkProcessed = processedInputs[:chunkSize]
                                 chunkRejected = rejectedWork[:chunkSize]
+                        # if global queue, then update workflow stats to request mgr couch doc
+                        # remove the "UnittestFlag" - need to create the reqmgrSvc emulator
+                        if not self.params.get("UnittestFlag", False):
+                            self.reqmgrSvc.updateRequestStats(inbound['WMSpec'].name(), totalStats)
 
             except TERMINAL_EXCEPTIONS as ex:
                 if not continuous:
Repository: dmwm/wmcore
Issue: JobSubmitter crashes when job files aren't found

On Line 278 of JobSubmitterPoller:
    badJobs[601103].append(newJob)
Makes JobSubmitter crash when a file is not found. This is because 601103 is not a valid code in the range of badJbs (see line 248).
The correct code should be:
 badJobs[61103].append(newJob)
So it report the missing file instead of crashing.


Changed files:
src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py

diff --git a/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py b/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py
index fafee9108..96e915c7b 100644
--- a/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py
+++ b/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py
@@ -275,7 +275,7 @@ class JobSubmitterPoller(BaseWorkerThread):
             if not os.path.isfile(pickledJobPath):
                 # Then we have a problem - there's no file
                 logging.error("Could not find pickled jobObject %s" % pickledJobPath)
-                badJobs[601103].append(newJob)
+                badJobs[61103].append(newJob)
                 continue
             try:
                 jobHandle = open(pickledJobPath, "r")