anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: dmwm/wmcore
Issue: DBS3Upload crashes during startup in WMAgent 2.1.6

**Impact of the bug**
WMAgent

**Describe the bug**
This PR
https://github.com/dmwm/WMCore/pull/11173

injected a syntax error to the source code, which causes DBS3Upload to crash during startup. See traceback in [1]

**How to reproduce it**
Deploy WMAgent 2.1.6 and start DBS3Upload component

**Expected behavior**
The component needs to successfully startup and perform its expected actions :-)

**Additional context and error message**
[1]
```
Starting : DBS3Upload
Traceback (most recent call last):
  File "/data/srv/wmagent/v2.1.6/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6/bin/wmcoreD", line 348, in <module>
    startup(config)
  File "/data/srv/wmagent/v2.1.6/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6/bin/wmcoreD", line 221, in startup
    componentObject = factory.loadObject(classname = namespace, args = config)
  File "/data/srv/wmagent/v2.1.6/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6/lib/python3.8/site-packages/WMCore/WMFactory.py", line 58, in loadObject
    module = __import__(module, globals(), locals(), [classname])
  File "/data/srv/wmagent/v2.1.6/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6/lib/python3.8/site-packages/WMComponent/DBS3Buffer/DBS3Upload.py", line 23, in <module>
    from WMComponent.DBS3Buffer.DBSUploadPoller import DBSUploadPoller
  File "/data/srv/wmagent/v2.1.6/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6/lib/python3.8/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py", line 117
    msg = f"Hit a general exception while inserting block {name. Error: {str(ex}}"
          ^
SyntaxError: f-string: closing parenthesis '}' does not match opening parenthesis '('
```

Changed files:
src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py

diff --git a/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py b/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
index 66cebddcd..5b0e2a764 100644
--- a/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
+++ b/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
@@ -114,7 +114,7 @@ def uploadWorker(workInput, results, dbsUrl, gzipEncoding=False):
                 logging.error(msg)
                 results.put({'name': name, 'success': "error", 'error': msg})
         except Exception as ex:
-            msg = f"Hit a general exception while inserting block {name. Error: {str(ex}}"
+            msg = f"Hit a general exception while inserting block {name}. Error: {str(ex)}"
             logging.exception(msg)
             results.put({'name': name, 'success': "error", 'error': msg})
     return
Repository: dmwm/wmcore
Issue: Fix validation function for TrustSitelists and TrustPUSitelists

**Impact of the bug**
ReqMgr2

**Describe the bug**
Today Hasan mentioned that Unified mistakenly assigned hundreds of workfllows with `TrustSitelists=1`, instead of providing a boolean flag. From the StdBase spec, I see nothing wrong with the parameter signature:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/StdSpecs/StdBase.py#L1160

which basically performs this validate function:
https://github.com/dmwm/WMCore/blob/master/src/python/Utils/Utilities.py#L64-L76

**How to reproduce it**
Testing that `strToBool` function interactively, here is the problem:
```
In [2]: 1 in [False, True]
Out[2]: True
```

**Expected behavior**
Fix the `strToBool` validate function and fail the action if a boolean value is not provided, or one of those variation strings that are supported ("FALSE", "false", "False").

**Additional context and error message**
Here is one such request that went through with that value equals 1 (Hasan mentioned to have assigned around 200 of such workflows):
https://cmsweb.cern.ch/reqmgr2/fetch?rid=haozturk_ACDC0_task_JME-RunIISummer20UL18wmLHEGEN-01157__v1_T_220822_115827_2847


Changed files:
test/python/Utils_t/Utilities_t.py, src/python/Utils/Utilities.py

diff --git a/src/python/Utils/Utilities.py b/src/python/Utils/Utilities.py
index c2cf85c8d..1821f6063 100644
--- a/src/python/Utils/Utilities.py
+++ b/src/python/Utils/Utilities.py
@@ -63,11 +63,16 @@ def makeNonEmptyList(stringList):
 
 def strToBool(string):
     """
-    _strToBool_
-
-    Try to convert a string to boolean. i.e. "True" to python True
+    Try to convert different variations of True or False (including a string
+    type object) to a boolean value.
+    In short:
+     * True gets mapped from: True, "True", "true", "TRUE".
+     * False gets mapped from: False, "False", "false", "FALSE"
+     * anything else will fail
+    :param string: expects a boolean or a string, but it could be anything else
+    :return: a boolean value, or raise an exception if value passed in is not supported
     """
-    if string in [False, True]:
+    if string is False or string is True:
         return string
     elif string in ["True", "true", "TRUE"]:
         return True
diff --git a/test/python/Utils_t/Utilities_t.py b/test/python/Utils_t/Utilities_t.py
index 309886ab0..a6df88526 100644
--- a/test/python/Utils_t/Utilities_t.py
+++ b/test/python/Utils_t/Utilities_t.py
@@ -75,7 +75,7 @@ class UtilitiesTests(unittest.TestCase):
         for v in [False, "False", "FALSE", "false"]:
             self.assertFalse(strToBool(v))
 
-        for v in ["", "alan", [], [''], {'a': 123}]:
+        for v in [1, 0, "", "alan", [], [''], {'a': 123}]:
             self.assertRaises(ValueError, strToBool, v)
 
     def testSafeStr(self):
Repository: dmwm/wmcore
Issue: Missing `readTotalSecs` metric at  Monit

**Impact of the bug**
Monitoring 

**Describe the bug**
It was reported by the Monitoring Team that we fail to upload some metrics from the Frame work job report to monit. And the eventual reason we've been pointed to, must be a typo in the following line at the FwkJobReport XMLParser: https://github.com/dmwm/WMCore/blob/02f70c36892c146546a7eeec5d3e4fda4b278682/src/python/WMCore/FwkJobReport/XMLParser.py#L458 

i.e.  `totalMSecs` should be changed to `totalMsecs` (lower case `s`).

**How to reproduce it**
Try to parse a CMSRun XMl output with the current parser.

**Expected behavior**
The parser not to miss  a field.

**Additional context and error message**
This bug was found in the context of preparing better `CPUEfficiency` plots: 
https://cms-monitoring.cern.ch/web/cpueff/
https://cms-monitoring.cern.ch/web/cpueff/condor-detailed

And here is some useful documentation about all the aggregations used:
https://github.com/dmwm/CMSMonitoring/blob/master/cpueff-goweb/docs/aggregations.md

FYI @mrceyhun  @Dr15Jones 

Changed files:
src/python/WMCore/FwkJobReport/XMLParser.py

diff --git a/src/python/WMCore/FwkJobReport/XMLParser.py b/src/python/WMCore/FwkJobReport/XMLParser.py
index 6873d342d..b7d448172 100644
--- a/src/python/WMCore/FwkJobReport/XMLParser.py
+++ b/src/python/WMCore/FwkJobReport/XMLParser.py
@@ -455,7 +455,7 @@ def perfStoreHandler():
                           storageValues.get("Timing-tstoragefile-read-numOperations", 0)
             readCachOps = storageValues.get("Timing-tstoragefile-readViaCache-numSuccessfulOperations", 0) / \\
                           storageValues.get("Timing-tstoragefile-read-numOperations", 0)
-            readTotalT = storageValues.get("Timing-tstoragefile-read-totalMSecs", 0) / 1000
+            readTotalT = storageValues.get("Timing-tstoragefile-read-totalMsecs", 0) / 1000
             readNOps = storageValues.get("Timing-tstoragefile-read-numOperations", 0)
             writeTime = storageValues.get("Timing-tstoragefile-write-totalMsecs", 0) / 1000
             writeTotMB = storageValues.get("Timing-%s-write-totalMegabytes" % writeMethod, 0) \\
Repository: dmwm/wmcore
Issue: [MS] Premature deletion of pileup documents in the cleanup task

**Impact of the bug**
MSPileup

**Describe the bug**
While testing MSPileup/MSPileupTasks with real workload against Rucio Prod (in dryRun mode), I observed that pileup documents matched the constraints for document deletion, thus getting deleted from MongoDB prematurely.
Further investigation is required to identify why that happen.

**How to reproduce it**
Inject pileup documents with `active=false`, using WMCore `2.2.0` and wait for MSPileupTasks cycle to run.

**Expected behavior**
When a pileup object is created, it is supposed to be set with `deactivatedOn=current GMT time`. This is then evaluated by `cleanupTask`, and disabled pileup documents older than X days (cleanupDaysThreshold) are supposed to be automatically deleted by MSPileup.

**Additional context and error message**
I think the error is with this code:
```
            if not doc['ruleIds'] and not doc['currentRSEs'] and \\
                    doc['deactivatedOn'] + seconds > time.time():
```

which should actually be:
```
            if not doc['ruleIds'] and not doc['currentRSEs'] and time.time() > doc['deactivatedOn'] + seconds:
```


Changed files:
src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py

diff --git a/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py b/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py
index 309391fc4..cca610ec4 100644
--- a/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py
+++ b/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py
@@ -91,7 +91,7 @@ class MSPileupTasks():
         seconds = cleanupDaysThreshold * 24 * 60 * 60  # convert to second
         for doc in docs:
             if not doc['ruleIds'] and not doc['currentRSEs'] and \\
-                    doc['deactivatedOn'] + seconds > time.time():
+                    time.time() > doc['deactivatedOn'] + seconds:
                 spec = {'pileupName': doc['pileupName']}
                 if not self.dryRun:
                     self.logger.info("Cleanup task deleting pileup %s", doc['pileupName'])
Repository: creare-com/podpac
Issue: chunked Median does not work

**Description**
The stats median node reduced _chunk method does not work.

**Steps to Reproduce**
```
coords = Coordinate(
    lat=(0, 1, 10),
    lon=(0, 1, 10),
    time=('2018-01-01', '2018-01-10', '1,D'),
    order=['lat', 'lon', 'time'])

a = np.random.random(coords.shape)
source = NumpyArray(source=a, native_coordinates=coords)
data = source.execute(coords)
node = Median(source=source)

output = node.execute(coords)
output_chunked = node.execute(coords, {'iter_chunk_size': 100})
```

**Expected Behavior**
The reduce and reduce_chunked should both work and should return the same output.

**Observed Behavior**
The chunked execution raises an Exception.

**Additional Notes**
There is a failing testing for this that is currently flagged to skip.


Changed files:
podpac/core/algorithm/test/test_stats.py, podpac/core/algorithm/stats.py

diff --git a/podpac/core/algorithm/stats.py b/podpac/core/algorithm/stats.py
index d0df1be1..580d6540 100644
--- a/podpac/core/algorithm/stats.py
+++ b/podpac/core/algorithm/stats.py
@@ -869,9 +869,12 @@ class Reduce2(Reduce):
         UnitsDataArray
             Reduced output.
         """
-        I = [self.input_coordinates.dims.index(dim)
-             for dim in
-             self.evaluated_coordinates.dims]
+        # special case for full reduce
+        if not self.evaluated_coordinates.dims:
+            xslc, x = next(xs)
+            return self.reduce(x)
+
+        I = [self.input_coordinates.dims.index(dim) for dim in self.evaluated_coordinates.dims]
         y = xr.full_like(self.output, np.nan)
         for xslc, x in xs:
             yslc = [xslc[i] for i in I]
diff --git a/podpac/core/algorithm/test/test_stats.py b/podpac/core/algorithm/test/test_stats.py
index 27ebdd0a..7845380f 100644
--- a/podpac/core/algorithm/test/test_stats.py
+++ b/podpac/core/algorithm/test/test_stats.py
@@ -167,7 +167,6 @@ class TestKurtosis(BaseTests):
     def setup_class(cls):
         cls.node = Kurtosis(source=source)
 
-@pytest.mark.skip("not-yet-working")
 class TestMedian(BaseTests):
     @classmethod
     def setup_class(cls):
Repository: dmwm/wmcore
Issue: Exit code reporting of stepchain job failures to MONIT requires improvement

**Impact of the bug**
Monitoring

**Describe the bug**
When a stepchain job fails, different cmsRuns generate different exit codes. While one of these exit codes indicate the real error, the others indicate that the particular cmsRun couldn't run because the previous one failed. Our MONIT dashboards use ` Chirp_WMCore_cmsRun_ExitCode, ExitCode` classAds to demonstrate the error and in many cases, these fields don't include the main exit code, but rather the `Skipping this step due to a failure in a previous one. (Exit Code: 99108)`


**How to reproduce it**

Please check this JSON file to see an example. 
```
"Chirp_WMCore_cmsRun3_ExitCode": 8021,
```
is the main problem whereas `Chirp_WMCore_cmsRun_ExitCode`, `Chirp_WMCore_cmsRun[4-6]_ExitCode` and `ExitCode` shows `99108`. 

[99108.json.zip](https://github.com/dmwm/WMCore/files/9839338/99108.json.zip)

**Expected behavior**
IMO, MONIT dashboards should use `ExitCode` for exit code monitoring (`Chirp_WMCore_cmsRun_ExitCode` has precedence over `ExitCode`, don't know why) and the agent should report the real exit code in this classAd, not 99108. 

It's also possible to implement this logic in MONIT, but I don't think it's the proper way to do it. IMO, this info should be propagated from central services.

**Additional context and error message**
None


Changed files:
src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py

diff --git a/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py b/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py
index 2e4d045fc..17aeebeef 100644
--- a/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py
+++ b/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py
@@ -50,14 +50,18 @@ class CMSSW(Executor):
         self.failedPreviousStep = None
 
     def _setStatus(self, returnCode, returnMessage):
-        """
-        Set return code.
-        """
-        self.setCondorChirpAttrDelayed('Chirp_WMCore_cmsRun_ExitCode', returnCode)
+        """Chirp Job/step exit code through HTCondor"""
+        # we will set Chirp_WMCore_cmsRun_ExitCode codes only if previous step was clean
+        if not self.failedPreviousStep:
+            self.setCondorChirpAttrDelayed('Chirp_WMCore_cmsRun_ExitCode', returnCode)
+            logging.info("Step %s: Chirp_WMCore_cmsRun_ExitCode %s", self.stepName, returnCode)
+            if returnMessage and returnCode != 0:
+                self.setCondorChirpAttrDelayed('Chirp_WMCore_cmsRun_Exception_Message', returnMessage, compress=True)
         self.setCondorChirpAttrDelayed('Chirp_WMCore_%s_ExitCode' % self.stepName, returnCode)
+        logging.info("Step %s: Chirp_WMCore_%s_ExitCode %s", self.stepName, self.stepName, returnCode)
         if returnMessage and returnCode != 0:
-            self.setCondorChirpAttrDelayed('Chirp_WMCore_cmsRun_Exception_Message', returnMessage, compress=True)
             self.setCondorChirpAttrDelayed('Chirp_WMCore_%s_Exception_Message' % self.stepName, returnMessage, compress=True)
+
         self.step.execution.exitStatus = returnCode
 
     def pre(self, emulator=None):
Repository: dmwm/wmcore
Issue: MSPileup monitoringTask fail to track containers without any wmcore_transferor rules

**Impact of the bug**
MSPileup

**Describe the bug**
Hasan pointed out to this RelVal pileup:
```
/RelValMinBias_14TeV/CMSSW_13_1_0_pre2-131X_mcRun4_realistic_v2_2026D95noPU-v2/GEN-SIM
```

which is supposed to have 1 `wmcore_transferor` rule locking it at T2_CH_CERN, accessible through (but also copied in [1]):
https://cmsweb.cern.ch/ms-pileup/data/pileup?pileupName=/RelValMinBias_14TeV/CMSSW_13_1_0_pre2-131X_mcRun4_realistic_v2_2026D95noPU-v2/GEN-SIM

It turns out that that rule `4650a485fe3f44258f5aedb60139a5cb` does not even exist in Rucio; and MSPileup is not taking any action on the pileup either.

Problem seems to be in the `monitoringTask` which iterates through all the `wmcore_transferor`, however, it does not account for the case where no rules are found in Rucio:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py#L259

**How to reproduce it**
Have a pileup configuration with existent rules, then delete all wmcore_transferor rules.

**Expected behavior**
If a pileup is active (active=True), it has expected RSEs in the configuration, but no rules in Rucio are found; it is expected that MSPileup will:
* cleanup the `currentRSEs` property
* and cleanup the `ruleIds` property.

With that, the `activeTask` will pick it up and make the relevant data placement.

**Additional context and error message**
[1]
```
{"result": [
 {
  "pileupName": "/RelValMinBias_14TeV/CMSSW_13_1_0_pre2-131X_mcRun4_realistic_v2_2026D95noPU-v2/GEN-SIM",
  "pileupType": "premix",
  "insertTime": 1681914426,
  "lastUpdateTime": 1682596591,
  "expectedRSEs": [
    "T2_CH_CERN"
  ],
  "currentRSEs": [
    "T2_CH_CERN"
  ],
  "fullReplicas": 1,
  "campaigns": [],
  "containerFraction": 1.0,
  "replicationGrouping": "ALL",
  "activatedOn": 1682596591,
  "deactivatedOn": 1681914426,
  "active": true,
  "pileupSize": 33060714720,
  "ruleIds": [
    "4650a485fe3f44258f5aedb60139a5cb"
  ]
}]}
```

Changed files:
src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py

diff --git a/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py b/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py
index cca610ec4..7a6a6174e 100644
--- a/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py
+++ b/src/python/WMCore/MicroService/MSPileup/MSPileupTasks.py
@@ -256,6 +256,15 @@ def monitoringTask(doc, spec):
     rules = rucioClient.listDataRules(pname, **kwargs)
     modify = False
 
+    if not rules:
+        logger.info(f"Did not find any wmcore_transferor rules for container: {pname}.")
+        if not doc['expectedRSEs']:
+            logger.warning(f"Container: {pname} is active but has no expected RSEs.")
+        elif doc['currentRSEs'] or doc['ruleIds']:
+            doc['currentRSEs'].clear()
+            doc['ruleIds'].clear()
+            modify = True
+
     for rdoc in rules:
         rses = rucioClient.evaluateRSEExpression(rdoc['rse_expression'])
         rid = rdoc['id']
Repository: dmwm/wmcore
Issue: Bug on maxSecondsUntilRampdown missing a parenthesis during pset tweaking

**Impact of the bug**
maxSecondsUntilRampdown is missing a parenthesis in the json created for the pset tweak. Tweak will fail

**Describe the bug**
Missing parenthesis will make pset tweak for that parameter fail

**How to reproduce it**
Tweak that parameter

**Expected behavior**
Tweak should happen

**Additional context and error message**
Bug noticed by @belforte while looking at the code, thanks a lot! 

Changed files:
src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py

diff --git a/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py b/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py
index e339ceeac..4bac97d20 100644
--- a/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py
+++ b/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py
@@ -815,7 +815,7 @@ class SetupCMSSWPset(ScriptInterface):
         # limit run time if desired
         if hasattr(self.step.data.application.configuration, "maxSecondsUntilRampdown"):
             self.tweak.addParameter("process.maxSecondsUntilRampdown.input",
-                "customTypeCms.untracked.PSet(input=cms.untracked.int32(%s)" % self.step.data.application.configuration.maxSecondsUntilRampdown)
+                "customTypeCms.untracked.PSet(input=cms.untracked.int32(%s))" % self.step.data.application.configuration.maxSecondsUntilRampdown)
 
         # accept an overridden TFC from the step
         if hasattr(self.step.data.application, 'overrideCatalog'):
Repository: dmwm/wmcore
Issue: [MS] UnboundLocalError exception in MSTransferor

**Impact of the bug**
MSTransferor

**Describe the bug**
A somewhat harmless error in the service execution, but we better get it fixed.
Checking the latest tag in testbed, I see this exception has been raised once [1], so crashing the cycle with:
```
UnboundLocalError: local variable 'requestRecords' referenced before assignment
```

**How to reproduce it**
Hit an exception while retrieving workflows from ReqMgr2

**Expected behavior**
Fix the code such that this exception never happens again (1-line fix).

**Additional context and error message**
[1]
```
2021-07-31 12:04:52,395:ERROR:MSManager: MS daemon error: local variable 'requestRecords' referenced before assignment
Traceback (most recent call last):
  File "/data/srv/HG2108g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre5/lib/python3.8/site-packages/WMCore/MicroService/MSManager.py", line 42, in daemon
    func(reqStatus)
  File "/data/srv/HG2108g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre5/lib/python3.8/site-packages/WMCore/MicroService/MSManager.py", line 184, in transferor
    res = self.msTransferor.execute(reqStatus)
  File "/data/srv/HG2108g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre5/lib/python3.8/site-packages/WMCore/MicroService/MSTransferor/MSTransferor.py", line 184, in execute
    for reqSlice in grouper(requestRecords, 100):
UnboundLocalError: local variable 'requestRecords' referenced before assignment
```

Changed files:
src/python/WMCore/MicroService/MSTransferor/MSTransferor.py

diff --git a/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py b/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py
index 2b9b488d3..c0c838093 100644
--- a/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py
+++ b/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py
@@ -160,6 +160,7 @@ class MSTransferor(MSCore):
             msg += "Service set to process up to %s requests per cycle." % self.msConfig["limitRequestsPerCycle"]
             self.logger.info(msg)
         except Exception as err:  # general error
+            requestRecords = []
             msg = "Unknown exception while fetching requests from ReqMgr2. Error: %s", str(err)
             self.logger.exception(msg)
             self.updateReportDict(summary, "error", msg)
Repository: dmwm/wmcore
Issue: Fix initial job priority at job submission

**Impact of the bug**
Job priority of jobs at submission time are wrong. This affects the priority of job types, setting them to values that are too small until they get edited in the condor pool, once a new request priority takes place.

**Describe the bug**
Job priority at submission time is not taking into account the `maxTaskPriority`.

UPDATE: This issue has been introduced with this PR: https://github.com/dmwm/WMCore/pull/9133

**How to reproduce it**
Just submit any workflow

**Expected behavior**
Job priority should be: `(priority + taskPriority * maxTaskPriority)`
where `maxTaskPriority` is a big constant number.


Changed files:
src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py

diff --git a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
index e4c7d34c4..e7201c5df 100644
--- a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
+++ b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
@@ -599,7 +599,7 @@ class SimpleCondorPlugin(BasePlugin):
             ad['My.WMCore_ResizeJob'] = str(job.get('resizeJob', False))
             taskPriority = int(job.get('taskPriority', 1))
             priority = int(job.get('wf_priority', 0))
-            ad['My.JobPrio'] = str(int(priority + taskPriority * 1))
+            ad['My.JobPrio'] = str(int(priority + taskPriority * self.maxTaskPriority))
             ad['My.PostJobPrio1'] = str(int(-1 * len(job.get('potentialSites', []))))
             ad['My.PostJobPrio2'] = str(int(-1 * job['task_id']))
             # Add OS requirements for jobs
Repository: dmwm/wmcore
Issue: Fix initial job priority at job submission

**Impact of the bug**
Job priority of jobs at submission time are wrong. This affects the priority of job types, setting them to values that are too small until they get edited in the condor pool, once a new request priority takes place.

**Describe the bug**
Job priority at submission time is not taking into account the `maxTaskPriority`.

UPDATE: This issue has been introduced with this PR: https://github.com/dmwm/WMCore/pull/9133

**How to reproduce it**
Just submit any workflow

**Expected behavior**
Job priority should be: `(priority + taskPriority * maxTaskPriority)`
where `maxTaskPriority` is a big constant number.


Changed files:
src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py

diff --git a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
index bc4d24c3e..b4183dd88 100644
--- a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
+++ b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
@@ -599,7 +599,7 @@ class SimpleCondorPlugin(BasePlugin):
             ad['My.WMCore_ResizeJob'] = str(job.get('resizeJob', False))
             taskPriority = int(job.get('taskPriority', 1))
             priority = int(job.get('wf_priority', 0))
-            ad['My.JobPrio'] = str(int(priority + taskPriority * 1))
+            ad['My.JobPrio'] = str(int(priority + taskPriority * self.maxTaskPriority))
             ad['My.PostJobPrio1'] = str(int(-1 * len(job.get('potentialSites', []))))
             ad['My.PostJobPrio2'] = str(int(-1 * job['task_id']))
             # Add OS requirements for jobs
Repository: dmwm/wmcore
Issue: [py2py3] Python3 list sort built-in takes no positional arguments

**Impact of the bug**
WMCore in general

**Describe the bug**
Another popular issue while integrating python3 unit tests in our jenkins CI.
Python3 list `sort` built-in does not take any positional arguments, where many of our MySQL bind handling fails, see [1]

**How to reproduce it**
Run the DBSBufferFile unit test provided in the details

**Expected behavior**
List sorting functionality should remain intact and work in both python2 and python3 :-D

**Additional context and error message**
Documentation: https://docs.python.org/3.8/library/stdtypes.html#list.sort

[1]
```
======================================================================
ERROR: _testProperties_
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dmwm/wmcore_unittest/WMCore/test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py", line 41, in setUp
    locationAction.execute(siteName="se1.cern.ch")
  File "/home/dmwm/wmcore_unittest/WMCore/src/python/WMComponent/DBS3Buffer/MySQL/DBSBufferFiles/AddLocation.py", line 23, in execute
    self.dbi.processData(self.sql, binds, conn = conn,
  File "/home/dmwm/wmcore_unittest/WMCore/src/python/WMCore/Database/DBCore.py", line 172, in processData
    r = self.executebinds(s, b, connection=connection,
  File "/home/dmwm/wmcore_unittest/WMCore/src/python/WMCore/Database/MySQLCore.py", line 126, in executebinds
    s, b = self.substitute(s, b)
  File "/home/dmwm/wmcore_unittest/WMCore/src/python/WMCore/Database/MySQLCore.py", line 77, in substitute
    bindVarNames.sort(stringLengthCompare)
TypeError: sort() takes no positional arguments
-------------------- >> begin captured logging << --------------------
root: DEBUG: Tables for WMComponent.DBS3Buffer created
--------------------- >> end captured logging << ---------------------
```

Changed files:
src/python/WMCore/REST/Server.py

diff --git a/src/python/WMCore/REST/Server.py b/src/python/WMCore/REST/Server.py
index e7e856fa2..f9f12a1ab 100644
--- a/src/python/WMCore/REST/Server.py
+++ b/src/python/WMCore/REST/Server.py
@@ -188,7 +188,7 @@ class RESTFrontPage(object):
         if instances:
             instances = [dict(id=k, title=v[".title"], order=v[".order"])
                          for k, v in viewitems(instances())]
-            instances.sort(lambda a, b: a["order"] - b["order"])
+            instances.sort(key=lambda x: x["order"])
             self._preamble += (", REST_INSTANCES = %s" % json.dumps(instances))
 
         self._preamble += ";\\n%s" % (preamble or "")
Repository: dmwm/wmcore
Issue: [py2py3] PY3 runtime code failing to load workload pickle with a TypeError

**Impact of the bug**
Py3 WMAgent

**Describe the bug**
Testing Py3 WMAgent 1.5.1.pre6 (plus a bunch of patches), raises a TypeError exception when loading the workload.pkl file.
Error is well known to us by now. as can be seen at [1] 

**How to reproduce it**
See above.

**Expected behavior**
Runtime code should not fail to load the workload pickle object, regardless of the python version. In this case, it's only the python3 version of the agent that is failing

**Additional context and error message**
[1]
```
INFO:root:Loading task
Traceback (most recent call last):
  File "Startup.py", line 32, in <module>
    task = Bootstrap.loadTask(job)
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/Bootstrap.py", line 185, in loadTask
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/Bootstrap.py", line 172, in loadWorkload
TypeError: a bytes-like object is required, not 'str'
+ cp 'WMTaskSpace/Report*.pkl' ../
```

Changed files:
src/python/WMCore/WMRuntime/Startup.py, src/python/WMCore/WMRuntime/Bootstrap.py

diff --git a/src/python/WMCore/WMRuntime/Bootstrap.py b/src/python/WMCore/WMRuntime/Bootstrap.py
index e2555cc8a..2c2e50a13 100644
--- a/src/python/WMCore/WMRuntime/Bootstrap.py
+++ b/src/python/WMCore/WMRuntime/Bootstrap.py
@@ -168,7 +168,7 @@ def loadWorkload():
     """
     sandboxLoc = locateWMSandbox()
     workloadPcl = "%s/WMWorkload.pkl" % sandboxLoc
-    with open(workloadPcl, 'r') as handle:
+    with open(workloadPcl, 'rb') as handle:
         wmWorkload = pickle.load(handle)
 
     return WMWorkloadHelper(wmWorkload)
diff --git a/src/python/WMCore/WMRuntime/Startup.py b/src/python/WMCore/WMRuntime/Startup.py
index ba6f41dfe..83e1badce 100644
--- a/src/python/WMCore/WMRuntime/Startup.py
+++ b/src/python/WMCore/WMRuntime/Startup.py
@@ -21,6 +21,8 @@ if __name__ == '__main__':
     # WMAgent log has to be written to the pilot area in order to be transferred back
     Bootstrap.setupLogging(os.path.join(os.getcwd(), '../'))
     logging.info("Process id: %s\\tCurrent working directory: %s", os.getpid(), os.getcwd())
+    logging.info("Python version: %s", sys.version)
+    logging.info("Python path: %s", sys.path)
 
     logging.info("Loading job definition")
     job = Bootstrap.loadJobDefinition()
Repository: dmwm/wmcore
Issue: Cleanup jobs requesting an empty OS at the job classad

**Impact of the bug**
WMAgent

**Describe the bug**
While validating these job wrapper changes:
https://github.com/dmwm/WMCore/pull/11077

I noticed that Cleanup jobs actually define an empty string for the required OS:
```
  CMS_JobType = "Cleanup"
  REQUIRED_OS = ""
```

**How to reproduce it**
Create a workflow and check the job classads for a Cleanup job.

**Expected behavior**
Cleanup jobs don't rely on any CMSSW release, so ScramArch is actually not defined for this step.
When converting the job/step requirements into job classads, we should make sure that `REQUIRED_OS` has a correct value, which is either `any` or the `rhelX` string.

**Additional context and error message**
None

Changed files:
src/python/WMCore/BossAir/Plugins/BasePlugin.py, test/python/WMCore_t/BossAir_t/BasePlugin_t.py

diff --git a/src/python/WMCore/BossAir/Plugins/BasePlugin.py b/src/python/WMCore/BossAir/Plugins/BasePlugin.py
index 4eedec0d8..f6981b2d2 100644
--- a/src/python/WMCore/BossAir/Plugins/BasePlugin.py
+++ b/src/python/WMCore/BossAir/Plugins/BasePlugin.py
@@ -138,7 +138,7 @@ class BasePlugin(object):
             string to be matched for OS requirements for job
         """
         requiredOSes = set()
-        if scramArch is None:
+        if not scramArch:
             requiredOSes.add('any')
         elif isinstance(scramArch, (str, bytes)):
             for arch, validOSes in viewitems(ARCH_TO_OS):
diff --git a/test/python/WMCore_t/BossAir_t/BasePlugin_t.py b/test/python/WMCore_t/BossAir_t/BasePlugin_t.py
index 4a90cadf1..eb167b4f3 100644
--- a/test/python/WMCore_t/BossAir_t/BasePlugin_t.py
+++ b/test/python/WMCore_t/BossAir_t/BasePlugin_t.py
@@ -33,8 +33,14 @@ class BasePluginTest(BossAirTest):
         self.assertEqual(bp.scramArchtoRequiredOS('slc5_blah_blah'), 'rhel6')
         self.assertEqual(bp.scramArchtoRequiredOS('slc6_blah_blah'), 'rhel6')
         self.assertEqual(bp.scramArchtoRequiredOS('slc7_blah_blah'), 'rhel7')
+        self.assertEqual(bp.scramArchtoRequiredOS('cc8_blah_blah'), 'rhel8')
+        self.assertEqual(bp.scramArchtoRequiredOS('cs8_blah_blah'), 'rhel8')
+        self.assertEqual(bp.scramArchtoRequiredOS('alma8_blah_blah'), 'rhel8')
+        self.assertEqual(bp.scramArchtoRequiredOS('el88_blah_blah'), 'rhel8')
 
         self.assertEqual(bp.scramArchtoRequiredOS(None), 'any')
+        self.assertEqual(bp.scramArchtoRequiredOS(""), 'any')
+        self.assertEqual(bp.scramArchtoRequiredOS([]), 'any')
 
         self.assertEqual(bp.scramArchtoRequiredOS(['slc6_blah_blah', 'slc7_blah_blah']), 'rhel6,rhel7')
Repository: dmwm/wmcore
Issue: Wrong activity for subscriptions to Tape_Test sites

**Impact of the bug**
T0 Agent and probably production WMAgent

**Describe the bug**
During the transfers test, subscribing data to `T0_CH_CERN_Tape_Test` we spotted that a wrong activity was assigned to multiple rules.. For example the following:
```
rucio rule-info 9fadb5aa83114e9485fd7e8a43c6d1b1
Id:                         9fadb5aa83114e9485fd7e8a43c6d1b1
Account:                    tier0_prod
Scope:                      cms
Name:                       /D16/TransferTest2022-v3/RAW
RSE Expression:             T0_CH_CERN_Tape_Test
Copies:                     1
State:                      OK
Locks OK/REPLICATING/STUCK: 3488/0/0
Grouping:                   ALL
Expires at:                 None
Locked:                     True
Weight:                     None
Created at:                 2022-02-21 14:45:38
Updated at:                 2022-02-23 01:51:27
Error:                      None
Subscription Id:            None
Source replica expression:  None
Activity:                   T0 Export
Comment:                    T0 WMAgent automatic container rule
Ignore Quota:               False
Ignore Availability:        False
Purge replicas:             False
Notification:               NO
End of life:                None
Child Rule Id:              None
```

**How to reproduce it**
Steps to reproduce the behavior:
Replay subscribing data to any Tape_Test site

**Expected behavior**
Activity to all Tape sites even for the test sites should be `T0 Tape` 

**Additional context and error message**
None


Changed files:
src/python/WMComponent/RucioInjector/RucioInjectorPoller.py

diff --git a/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py b/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
index 08ac0901b..e9a5f0f60 100644
--- a/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
+++ b/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
@@ -465,7 +465,7 @@ class RucioInjectorPoller(BaseWorkerThread):
                               grouping="ALL",
                               comment=ruleComment,
                               meta=self.metaData)
-            if not rseName.endswith("_Tape"):
+            if not rseName.endswith(("_Tape", "_Tape_Test")):
                 # add extra parameters to the Disk rule as defined in the component configuration
                 ruleKwargs.update(self.containerDiskRuleParams)
 
@@ -529,7 +529,7 @@ class RucioInjectorPoller(BaseWorkerThread):
         """
         if not self.isT0agent and not rseName.endswith("_Tape"):
             return "Production Output"
-        elif self.isT0agent and rseName.endswith("_Tape"):
+        elif self.isT0agent and rseName.endswith(("_Tape", "_Tape_Test")):
             return "T0 Tape"
         elif self.isT0agent:
             return "T0 Export"
Repository: dmwm/wmcore
Issue: Data location inconsistency between job logs and wmstats

@vlimant was asking me about a very weird case where wmstats reports that a job was running at T2_US_Nebraska (failed job) while the condor logs in the agent vocms0303 reports that this job ran at T1_US_FNAL (notice it's the first attemp, so retry 0).

I forgot to add a link to a wmstats screenshot and a tarball of this job, from the JobCreator:
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7217/

@ticoann do you have any idea of what can have happened?


Changed files:
src/python/WMCore/JobStateMachine/ChangeState.py

diff --git a/src/python/WMCore/JobStateMachine/ChangeState.py b/src/python/WMCore/JobStateMachine/ChangeState.py
index 6460f5bbe..77fd8531a 100644
--- a/src/python/WMCore/JobStateMachine/ChangeState.py
+++ b/src/python/WMCore/JobStateMachine/ChangeState.py
@@ -318,7 +318,10 @@ class ChangeState(WMObject, WMConnectionBase):
                     jobState = "jobfailed"
                 else:
                     jobState = newstate
-                    
+                
+                # there is race condition updating couch record location and job is completed.
+                # for the fast fail job, it could miss the location update
+                job["location"] = job["fwjr"].getSiteName() or job.get("location", "Unknown")     
                 # complete fwjr document
                 job["fwjr"].setTaskName(job["task"])
                 jsonFWJR = job["fwjr"].__to_json__(None)