Repository: dmwm/wmcore
Issue: Some CMS@Home jobs end up with the wrong HTCondor requirements

**Impact of the bug**
This bug impacts CMS@Home jobs.  They end up with a requirement that
(AuthenticatedIdentity =!= "volunteer-node@cern.ch") which prevents them from being sent to Volunteer machines.  These jobs sit in the "pending" queue for five days until a time-out leads to their resubmission, usually with the desired requirements.  We use cmsweb-testbed.cern.ch and the agent is vocms0267.cern.ch.

**Describe the bug**
CMS@Home jobs should have the HTCondor requirements
( TARGET.Arch == "X86_64" ) && ( TARGET.OpSys == "LINUX" ) && ( TARGET.Disk >= RequestDisk ) && ( TARGET.Memory >= RequestMemory ) && ( TARGET.Cpus >= RequestCpus ) && ( TARGET.HasFileTransfer )

However, some end up with
( ( ( REQUIRED_OS is "any" ) || ( GLIDEIN_REQUIRED_OS is "any" ) || stringListMember(GLIDEIN_REQUIRED_OS,REQUIRED_OS) ) && ( AuthenticatedIdentity isnt "volunteer-node@cern.ch" ) ) &&    ( TARGET.Arch == "X86_64" ) && ( TARGET.OpSys == "LINUX" ) && ( TARGET.Disk >= RequestDisk ) && ( TARGET.Memory >= RequestMemory ) && ( TARGET.Cpus >= RequestCpus ) && ( TARGET.HasFileTransfer )

The added requirements would appear to come from
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py#L135

Initially, we thought that jobs with the added requirements were those which had failed the condor limit on retries and been passed back to WMAgent, which resubmitted them with the addition, but now we have evidence that in fact some of the initial job submissions already have them.  Jobs with the added requirements stay in the pending queue, and are not distributed on request to Volunteer machines running BOINC tasks.  If these are the only jobs in the queue, the request fails and the BOINC task terminates with an error.  This then has consequences for the Volunteer (failed tasks generate no credit, and repeated failures lead to an automatic decrease in the daily quota of tasks the machine can run).

There is a five-day timeout in the pending queue, after which WMAgent resubmits the job.  As far as we can tell these resubmissions usually have the correct job requirements, as after five days jobs in the particular workflow start running again, and the pending queue decreases.  This five-day delay is somewhat excessive for our requirements (we aim to have jobs that run for 1-2 hours), and puts constraints on our workflow submissions.

**How to reproduce it**
Submit a workflow to T3_CH_Volunteer and observe its behaviour with WMStats:
python inject-test-wfs.py -u https://cmsweb-testbed.cern.ch -m DMWM -f TC_SLC7.json  -c IDR_CMS_Home -r IDR_CMS_Home -t cms-home -a DMWM_Test -p CMS_Home_IDRv5d -s "T3_CH_Volunteer"

**Expected behavior**
We expect that production jobs submitted to T3_CH_Volunteer will all be able to run on Volunteer machines.  Note that post-production jobs (merge, log-collect, etc.) run on a VM farm on T3_CH_CMSAtHome; even if these jobs had the added requirement they would still run as T3_CH_CMSAtHome is not "volunteer-node@cern.ch".



Changed files:
src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py

diff --git a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
index 2cb0fb092..6819c8e95 100644
--- a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
+++ b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
@@ -127,17 +127,11 @@ class SimpleCondorPlugin(BasePlugin):
         # Required for global pool accounting
         self.acctGroup = getattr(config.BossAir, 'acctGroup', "production")
         self.acctGroupUser = getattr(config.BossAir, 'acctGroupUser', "cmsdataops")
-
-        # Build a requirement string.  All CMS resources match DESIRED_Sites on the START
-        # expression side; however, there are currently some resources (T2_CH_CERN_HLT)
-        # that are missing the REQUIRED_OS logic.  Hence, we duplicate it here.
-        # TODO(bbockelm): Remove reqStr once HLT has upgraded.
-        self.reqStr = ('((REQUIRED_OS=?="any") || '
-                       '(GLIDEIN_REQUIRED_OS =?= "any") || '
-                       'stringListMember(GLIDEIN_REQUIRED_OS, REQUIRED_OS)) && '
-                       '(AuthenticatedIdentity =!= "volunteer-node@cern.ch")')
+ 
         if hasattr(config.BossAir, 'condorRequirementsString'):
             self.reqStr = config.BossAir.condorRequirementsString
+        else:
+            self.reqStr = None
 
         # x509 proxy handling
         proxy = Proxy({'logger': myThread.logger})
@@ -515,8 +509,13 @@ class SimpleCondorPlugin(BasePlugin):
             ad['Arguments'] = "%s %i %s" % (os.path.basename(job['sandbox']), job['id'], job["retry_count"])
             ad['transfer_output_files'] = "Report.%i.pkl,wmagentJob.log" % job["retry_count"]
 
-            # Do not define Requirements and X509 ads for Volunteer resources
-            if self.reqStr and "T3_CH_Volunteer" not in job.get('possibleSites'):
+            # Dictionary keys need to be consistent across all jobs within the same 
+            # clusterId when working with queue_with_itemdata()
+            # Initialize 'Requirements' to an empty string for all jobs.
+            # See issue: https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=7715 
+            ad['Requirements'] = ''
+            # Do not define custom Requirements for Volunteer resources
+            if self.reqStr is not None:
                 ad['Requirements'] = self.reqStr
 
             ad['My.x509userproxy'] = classad.quote(self.x509userproxy)