anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: dmwm/wmcore
Issue: Check for DBS3 duplicate insertions

Sometimes DBS3 takes longer than what the FrontEnd expects before giving back a response, this results in broken connection that the DBSUpload sees as a Proxy Error 502. However, the block injection may have been successful and afterwards the WMAgent is stuck trying to inject the block again over and over.

We need to check for errors like:

400  DBSBlockInsert/insertBlock. Block already exists.

And trust DBS3 that if we get that message then the block is safely there and we can stop uploading.

Option 2: Whenever we get an upload error, verify with DBS3 if the block is actually there or not.


Changed files:
src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py

diff --git a/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py b/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
index aedc8ea95..e22646c5f 100644
--- a/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
+++ b/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
@@ -137,13 +137,12 @@ def uploadWorker(input, results, dbsUrl):
             results.put({'name': name, 'success': True})
         except Exception, ex:
             exString = str(ex)
-            if 'Duplicate entry' in exString:
+            if 'Block already exists.' in exString:
                 # Then this is probably a duplicate
                 # Ignore this for now
-                logging.error("Had duplicate entry for block %s\\n" % name)
-                logging.error("Ignoring for now.\\n")
-                logging.error("Exception: %s\\n" % exString)
-                logging.error("Traceback: %s\\n" % str(traceback.format_exc()))
+                logging.error("Had duplicate entry for block %s. Ignoring for now." % name)
+                logging.debug("Exception: %s" % exString)
+                logging.debug("Traceback: %s" % str(traceback.format_exc()))
                 results.put({'name': name, 'success': True})
             else:
                 msg =  "Error trying to process block %s through DBS.\\n" % name
Repository: dmwm/wmcore
Issue: Cannot abort workflow in assigned status

Also, -archived status shouldn't move to another status.
Hi,

for the record, while validating HG1407 in testbed (ReqMgr 0.9.95e), I noticed we cannot abort workflows in "assigned" status (even having the status available in the select box).
It returns a 400 Bad Request with the following message:
"Cannot find ProdMgr for request amaltaro_700_pre11_TTbar_140702_203614_3642 Request may not be known to WorkQueue. If aborted immediately after assignment, ignore this."

If this transition is not allowed, then we should remove the "abort" option from the select box when it's in "assigned" status.

Thanks, Alan.


Changed files:
src/python/WMCore/HTTPFrontEnd/RequestManager/ReqMgrWebTools.py

diff --git a/src/python/WMCore/HTTPFrontEnd/RequestManager/ReqMgrWebTools.py b/src/python/WMCore/HTTPFrontEnd/RequestManager/ReqMgrWebTools.py
index 72da9174d..608f6f07f 100644
--- a/src/python/WMCore/HTTPFrontEnd/RequestManager/ReqMgrWebTools.py
+++ b/src/python/WMCore/HTTPFrontEnd/RequestManager/ReqMgrWebTools.py
@@ -193,9 +193,10 @@ def abortRequest(requestName):
     if response == [] or response[0] == None or response[0] == "":
         msg =  "Cannot find ProdMgr for request %s\\n " % requestName
         msg += "Request may not be known to WorkQueue.  If aborted immediately after assignment, ignore this."
-        raise cherrypy.HTTPError(400, msg)
-    workqueue = WorkQueue.WorkQueue(response[0])
-    workqueue.cancelWorkflow(requestName)
+        logging.warning(msg)
+    else:
+        workqueue = WorkQueue.WorkQueue(response[0])
+        workqueue.cancelWorkflow(requestName)
     return
 
 def insecure():
Repository: dmwm/wmcore
Issue: 'KeepOutput': False affects the job splitting

Hi guys,

it looks like when we have 2 'KeepOutput': False in cascaded, e.g., for Task1 and Task2, as in this testbed request:
amaltaro_RVCMSSW_7_0_0_pre11ProdTTbar_140228_160041_5739

the job splitting field/algorithm is empty for Task2, even when we provide this job splitting at injection time, as one can see in the spec file [1]

Looking into details in WMStats, I noticed
Task1 - ProdTTbar ran 45 jobs
Task2 - DIGIPROD1 ran 1 job (so it took all the input from Task1 in one shot)
Task3 - RECOPROD1 ran 5 jobs (that's correct, each of them with 10lumis)

I noticed this behavior is there since like 2 or 3 months ago, but I'm reporting it only now, sorry :-)

Thanks, Alan.

[1]
request.schema.Task2 = {'KeepOutput': False, 'GlobalTag': 'START70_V4::All', 'InputFromOutputModule': 'RAWSIMoutput', 'ProcessingString': 'START70_V4', 'SplittingAlgo': 'LumiBased', 'InputTask': 'ProdTTbar', 'ConfigCacheID': 'b6db80a24b9858e6cb9ee6c49891c6d6', 'LumisPerJob': 10, 'TaskName': 'DIGIPROD1', 'AcquisitionEra': 'CMSSW_7_0_0_pre11'}


Changed files:
src/python/WMCore/WMSpec/StdSpecs/TaskChain.py

diff --git a/src/python/WMCore/WMSpec/StdSpecs/TaskChain.py b/src/python/WMCore/WMSpec/StdSpecs/TaskChain.py
index 38df540b8..45308578c 100644
--- a/src/python/WMCore/WMSpec/StdSpecs/TaskChain.py
+++ b/src/python/WMCore/WMSpec/StdSpecs/TaskChain.py
@@ -356,13 +356,14 @@ class TaskChainWorkloadFactory(StdBase):
             if not inputTaskConf["KeepOutput"] or len(inputTaskConf["TransientOutputModules"]) > 0:
                 inpMod = taskConf["InputFromOutputModule"]
                 # Check if the splitting has to be changed
-                if inputTaskConf["SplittingAlgo"] == 'EventBased' \\
-                   and (('InputDataset' in inputTaskConf) or ('InputTask' in inputTaskConf)):
-                    splitAlgorithm = 'WMBSMergeBySize'
-                    splitArguments = {'max_merge_size'   : self.maxMergeSize,
-                                      'min_merge_size'   : self.minMergeSize,
-                                      'max_merge_events' : self.maxMergeEvents,
-                                      'max_wait_time'    : self.maxWaitTime}
+                # Lines below commented out to fix #5007 (by Alan M. on 03/Nov/2014 
+                #if inputTaskConf["SplittingAlgo"] == 'EventBased' \\
+                #   and (('InputDataset' in inputTaskConf) or ('InputTask' in inputTaskConf)):
+                #    splitAlgorithm = 'WMBSMergeBySize'
+                #    splitArguments = {'max_merge_size'   : self.maxMergeSize,
+                #                      'min_merge_size'   : self.minMergeSize,
+                #                      'max_merge_events' : self.maxMergeEvents,
+                #                      'max_wait_time'    : self.maxWaitTime}
             else:
                 inpMod = "Merged"
Repository: dmwm/wmcore
Issue: Another unicode encoding failure

It is in

https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Storage/Execute.py#L70

output contains "\\x95" which cause encoding error
  
and

'utf8' codec can't decode byte 0xb2 in position 24451


Changed files:
src/python/WMCore/FwkJobReport/Report.py

diff --git a/src/python/WMCore/FwkJobReport/Report.py b/src/python/WMCore/FwkJobReport/Report.py
index 35241435a..9bb30b3e7 100644
--- a/src/python/WMCore/FwkJobReport/Report.py
+++ b/src/python/WMCore/FwkJobReport/Report.py
@@ -560,7 +560,13 @@ class Report(object):
         errDetails = getattr(stepSection.errors, errEntry)
         errDetails.exitCode = exitCode
         errDetails.type = str(errorType)
-        errDetails.details = errorDetails
+
+        if hasattr(errorDetails, "decode"):
+            # Fix for the unicode encoding issue, #8043
+            # interprets this string using utf-8 codec and ignoring any errors
+            errDetails.details = errorDetails.decode('utf-8', 'ignore')
+        else:
+            errDetails.details = errorDetails
 
         setattr(stepSection.errors, "errorCount", errorCount + 1)
         self.setStepStatus(stepName=stepName, status=exitCode)
Repository: dmwm/wmcore
Issue: ACDC workflows failing to be acquired in GQ

Testing agent 1.0.11.pre8 in testbed, all the ACDC workflows I assigned yesterday went directly to failed in GQ, this is the log for one of them (*). It seems the faulty code is this one:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/SiteDB/SiteDB.py#L308

which is called in ResubmitBlock. In principle that extend is getting a string, and then each char is splitted.

I'll fix it on Monday.

(*)
2015-11-12 18:05:09,910 INFO     WorkQueueReqMgrInterface: Processing request amaltaro_ACDC_StepChain_LumiMask_Agent1011_Validation_v2_151112_175531_917 at https://cmsweb-testbed.cern.ch/couchdb/reqmgr_workload_cache/amaltaro_ACDC_StepChain_LumiMask_Agent1011_Validation_v2_151112_175531_917/spec
2015-11-12 18:05:09,911 INFO     WorkQueue: queueWork() begin queueing "https://cmsweb-testbed.cern.ch/couchdb/reqmgr_workload_cache/amaltaro_ACDC_StepChain_LumiMask_Agent1011_Validation_v2_151112_175531_917/spec"
2015-11-12 18:05:10,593 INFO     WorkQueue: Splitting /amaltaro_ACDC_StepChain_LumiMask_Agent1011_Validation_v2_151112_175531_917/DIGI with policy ResubmitBlock params = {'ResubmitBlock': {'args': {}, 'name': 'ResubmitBlock'}, 'MonteCarlo': {'args': {}, 'name': 'MonteCarlo'}, 'Dataset': {'args': {}, 'name': 'Dataset'}, 'Block': {'args': {}, 'name': 'Block'}, 'DatasetBlock': {'args': {}, 'name': 'Dataset'}}
2015-11-12 18:05:11,324 WARNING  SiteDB: No PSNs for PNN: T
2015-11-12 18:05:11,325 WARNING  SiteDB: No PSNs for PNN: 1
2015-11-12 18:05:11,326 WARNING  SiteDB: No PSNs for PNN: _
2015-11-12 18:05:11,327 WARNING  SiteDB: No PSNs for PNN: I
2015-11-12 18:05:11,328 WARNING  SiteDB: No PSNs for PNN: T
2015-11-12 18:05:11,329 WARNING  SiteDB: No PSNs for PNN: _
2015-11-12 18:05:11,329 WARNING  SiteDB: No PSNs for PNN: C
2015-11-12 18:05:11,330 WARNING  SiteDB: No PSNs for PNN: N
2015-11-12 18:05:11,331 WARNING  SiteDB: No PSNs for PNN: A
2015-11-12 18:05:11,332 WARNING  SiteDB: No PSNs for PNN: F
2015-11-12 18:05:11,333 WARNING  SiteDB: No PSNs for PNN: _
2015-11-12 18:05:11,333 WARNING  SiteDB: No PSNs for PNN: D
2015-11-12 18:05:11,334 WARNING  SiteDB: No PSNs for PNN: i
2015-11-12 18:05:11,335 WARNING  SiteDB: No PSNs for PNN: s
2015-11-12 18:05:11,336 WARNING  SiteDB: No PSNs for PNN: k


Changed files:
src/python/WMCore/Services/SiteDB/SiteDB.py

diff --git a/src/python/WMCore/Services/SiteDB/SiteDB.py b/src/python/WMCore/Services/SiteDB/SiteDB.py
index 44da50118..9f972bdf6 100644
--- a/src/python/WMCore/Services/SiteDB/SiteDB.py
+++ b/src/python/WMCore/Services/SiteDB/SiteDB.py
@@ -311,7 +311,7 @@ class SiteDBJSON(Service):
         if it is convert to PNN
         if not just return argument
         """
-        if isinstance(seNameOrPNN, str):
+        if isinstance(seNameOrPNN, basestring):
             seNameOrPNN = [seNameOrPNN]
         
         newList = []
@@ -319,5 +319,5 @@ class SiteDBJSON(Service):
             if not pnn_regex.match(se):
                 newList.extend(self.seToPNNs(se))
             else:
-                newList.extend(se)
+                newList.append(se)
         return newList
Repository: dmwm/wmcore
Issue: Campaign configuration couch document with incorrect doc type

The CampaignConfig class is suppose to create couch documents with this doc type:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/ReqMgr/Service/Auxiliary.py#L280

but instead it defaults to `WMAGENT_CONFIG`, as reported here:
https://github.com/dmwm/WMCore/issues/9142#issuecomment-482697437

Changed files:
src/python/WMCore/ReqMgr/Service/RestApiHub.py

diff --git a/src/python/WMCore/ReqMgr/Service/RestApiHub.py b/src/python/WMCore/ReqMgr/Service/RestApiHub.py
index 04ebd20f3..4962a7e8a 100644
--- a/src/python/WMCore/ReqMgr/Service/RestApiHub.py
+++ b/src/python/WMCore/ReqMgr/Service/RestApiHub.py
@@ -55,7 +55,7 @@ class RestApiHub(RESTApi):
                    "cmsswversions": CMSSWVersions(app, IndividualCouchManager(config), config, mount),
                    "wmagentconfig": WMAgentConfig(app, IndividualCouchManager(config), config, mount),
                    "permissions": PermissionsConfig(app, IndividualCouchManager(config), config, mount),
-                   "campaignconfig": WMAgentConfig(app, IndividualCouchManager(config), config, mount),
+                   "campaignconfig": CampaignConfig(app, IndividualCouchManager(config), config, mount),
                    "unifiedconfig": UnifiedConfig(app, IndividualCouchManager(config), config, mount),
                    "status": RequestStatus(app, IndividualCouchManager(config), config, mount),
                    "type": RequestType(app, IndividualCouchManager(config), config, mount),
Repository: dmwm/wmcore
Issue: Log for the ReqMgr2 parentageFixTask thread

Looking at the logs on the backend, I see it uses the date from when the process was started, e.g.
vocms0132/reqmgr2/parentageFixTask-20180831.log 

and it never rotates and/or create a new log file when it gets into a new day. Even more interesting, looking at this file now, I can see entries only for today (e.g. 2018-09-03 ...), while the rest is gone.
If one look at vocms055, the file name is the same but the content is from "2018-08-31"...

Weird thing is, it uses the same base class as statusChangeTasks or auxCacheUpdateTasks, and those are rotating just fine (see vocms0731)...

Changed files:
src/python/WMCore/WMLogging.py

diff --git a/src/python/WMCore/WMLogging.py b/src/python/WMCore/WMLogging.py
index ac9696a5d..1ff66890e 100644
--- a/src/python/WMCore/WMLogging.py
+++ b/src/python/WMCore/WMLogging.py
@@ -4,9 +4,9 @@ _WMLogging_
 
 Logging facilities used in WMCore.
 """
-import time
 import logging
 import codecs
+from datetime import date, timedelta
 from logging.handlers import HTTPHandler, RotatingFileHandler, TimedRotatingFileHandler
 
 # a new log level which is lower than debug
@@ -75,8 +75,9 @@ class MyTimedRotatingFileHandler(TimedRotatingFileHandler):
         """
         self.stream.close()
         # replace yesterday's date by today
-        yesterdayStr = time.strftime("%Y%m%d", time.localtime(time.time() - 7200))
-        self.baseFilename = self.baseFilename.replace(yesterdayStr, time.strftime("%Y%m%d"))
+        yesterdayStr = (date.today() - timedelta(1)).strftime("%Y%m%d")
+        todayStr = date.today().strftime("%Y%m%d")
+        self.baseFilename = self.baseFilename.replace(yesterdayStr, todayStr)
         if self.encoding:
             self.stream = codecs.open(self.baseFilename, 'w', self.encoding)
         else:
Repository: dmwm/wmcore
Issue: One of WorkQueue methods is called in the TaskArchiverPoller in Tier-0 WMAgent

In a replay testing the 2.1.5 T0 release (WMAgent 1.2.4.patch3 release) I observed the following error in the TaskArchiver logs:
```
2019-07-25 17:02:08,297:140059098547968:INFO:BaseWorkerThread:TaskArchiverPoller took 0.035 secs to execute
2019-07-25 17:02:08,300:140059098547968:DEBUG:LogDB:LogDB delete request, res=None
2019-07-25 17:06:08,595:140059098547968:DEBUG:LogDB:LogDB delete request, res=None
2019-07-25 17:06:08,596:140059098547968:INFO:TaskArchiverPoller:Polling for finished subscriptions
2019-07-25 17:06:08,600:140059098547968:INFO:TaskArchiverPoller:Finished subscriptions updated
2019-07-25 17:06:08,625:140059098547968:INFO:TaskArchiverPoller:Found 15 candidate workflows for completing:
2019-07-25 17:06:08,629:140059098547968:INFO:TaskArchiverPoller:There are 0 requests in ['force-complete', 'aborted'] status in central couch.
2019-07-25 17:06:08,630:140059098547968:ERROR:TaskArchiverPoller:we will try again when remote couch server comes back
'NoneType' object has no attribute 'killWMBSWorkflows'
2019-07-25 17:06:08,630:140059098547968:INFO:BaseWorkerThread:TaskArchiverPoller took 0.034 secs to execute
```
Refactoring the TaskArchiverPoller.py caused the `self.workQueue.killWMBSWorkflows(reqNames)` to be called:
https://github.com/dmwm/WMCore/commit/0f531f6826d1115e03267f4f55360fa7ef5e0911#diff-748e5dcfaf8a48d0e9774b3066780e42L197
even though T0 config.py configures workqueue = false.

Changed files:
src/python/WMComponent/TaskArchiver/TaskArchiverPoller.py

diff --git a/src/python/WMComponent/TaskArchiver/TaskArchiverPoller.py b/src/python/WMComponent/TaskArchiver/TaskArchiverPoller.py
index efecfb925..0a623808e 100644
--- a/src/python/WMComponent/TaskArchiver/TaskArchiverPoller.py
+++ b/src/python/WMComponent/TaskArchiver/TaskArchiverPoller.py
@@ -193,7 +193,8 @@ class TaskArchiverPoller(BaseWorkerThread):
             statusList = [statusList]
         reqNames = self.centralCouchDBWriter.getRequestByStatus(statusList)
         logging.info("There are %d requests in %s status in central couch.", len(reqNames), statusList)
-        self.workQueue.killWMBSWorkflows(reqNames)
+        if self.workQueue is not None:
+            self.workQueue.killWMBSWorkflows(reqNames)
         return reqNames
 
     def completeTasks(self, finishedwfs):