Repository: dmwm/wmcore
Issue: Cannot inject work against PNNless resources

**Impact of the bug**
WMAgent (either getting stuck in WorkQueueManager, or completely skipping such opportunistic resources)

**Describe the bug**
The CMS@Home agent got redeployed yesterday, and with that a new issue got exposed.
It's related to the fix made in this PR:
https://github.com/dmwm/WMCore/pull/9659/files#diff-7841c008d28dc37b9c039eef99e13610R434

which is supposed to only send top level jobs where there is a common data location (primary + secondary + parent).
The side effect is, it validates the `commonSites` against the information in CRIC (thus mapping PSN to PNN), which of course will fail for most of the opportunistic resources, given that they do not have a storage endpoint and instead rely on other sites remote stage out.

**How to reproduce it**
Assign a MC workflow against one opportunistic resource (T3_CH_Volunteer, T3_US_NERSC, etc). Of course, the agent needs to have those resources in the resource-control.

**Expected behavior**
We need to revisit which CRIC APIs to use, such that:
* this work would still go through to the opportunistic resources
* and that the agent would automatically add such resources to the local database (thus, also considering PNNless resources)

**Additional context and error message**
I have manually applied changes to the vocms0267 (CMS@Home agent) such that work can go through..

Traceback from WorkQueueManager:
```
2020-05-05 20:33:08,791:140536444503808:INFO:WorkQueueBackend:Getting up to 100 available work from http://vocms0267.cern.ch:5984
2020-05-05 20:33:08,807:140536444503808:INFO:WorkQueue:Got 1 elements matching the constraints
2020-05-05 20:33:08,908:140536444503808:INFO:WorkQueue:Running WMBS preparation for ireid_TC_SLC7_IDR_CMS_Home_200505_201537_9548 with ParentQueueId 63dbbb9f098d5119fc8bf38123f3ecba,
  with common location []
2020-05-05 20:33:08,962:140536444503808:INFO:Fileset:Fileset created: ireid_TC_SLC7_IDR_CMS_Home_200505_201537_9548-SinglePiE50HCAL_pythia8_2018_GenSimFull-50cb422cbd94edda4fec6dd36fe0d621
...
2020-05-05 20:33:12,022:140536444503808:ERROR:WorkQueue:Failed to create subscription for ireid_TC_SLC7_IDR_CMS_Home_200505_201537_9548 with block name None
Error: <@========== WMException Start ==========@>
Exception Class: WorkQueueWMBSException
Message: No locations to inject Monte Carlo work to, unable to proceed
        ModuleName : WMCore.WorkQueue.WMBSHelper
        MethodName : addMCFakeFile
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py
        ClassName : None
        LineNumber : 362
        ErrorNr : 0

Traceback: 

<@---------- WMException End ----------@>
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WorkQueue.py", line 352, in getWork
    dbsBlock)
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WorkQueue.py", line 442, in _wmbsPreparation
    sub, match['NumOfFilesAdded'] = wmbsHelper.createSubscriptionAndAddFiles(block=dbsBlock)
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 420, in createSubscriptionAndAddFiles
    addedFiles = self.addMCFakeFile()
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 362, in addMCFakeFile
    raise WorkQueueWMBSException(msg)
WorkQueueWMBSException: <@========== WMException Start ==========@>
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WorkQueue.py", line 352, in getWork
    dbsBlock)
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WorkQueue.py", line 442, in _wmbsPreparation
    sub, match['NumOfFilesAdded'] = wmbsHelper.createSubscriptionAndAddFiles(block=dbsBlock)
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 420, in createSubscriptionAndAddFiles
    addedFiles = self.addMCFakeFile()
  File "/data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 362, in addMCFakeFile
    raise WorkQueueWMBSException(msg)
WorkQueueWMBSException: <@========== WMException Start ==========@>
Exception Class: WorkQueueWMBSException
Message: No locations to inject Monte Carlo work to, unable to proceed
        ModuleName : WMCore.WorkQueue.WMBSHelper
        MethodName : addMCFakeFile
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.3.3.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.3.3.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py
        ClassName : None
        LineNumber : 362
        ErrorNr : 0

Traceback: 

<@---------- WMException End ----------@>
```

Changed files:
src/python/WMCore/Services/CRIC/CRIC.py, src/python/WMCore/WorkQueue/WorkQueue.py, src/python/WMQuality/Emulators/CRICClient/MockCRICApi.py

diff --git a/src/python/WMCore/Services/CRIC/CRIC.py b/src/python/WMCore/Services/CRIC/CRIC.py
index 2f9b98840..538591509 100644
--- a/src/python/WMCore/Services/CRIC/CRIC.py
+++ b/src/python/WMCore/Services/CRIC/CRIC.py
@@ -168,11 +168,12 @@ class CRIC(Service):
                 self["logger"].debug("No PSNs for PNN: %s" % pnn)
         return list(psns)
 
-    def PSNstoPNNs(self, psns):
+    def PSNstoPNNs(self, psns, allowPNNLess=False):
         """
         Given a list of PSNs, return all their PNNs
 
         :param psns: a string or a list of PSNs
+        :param allowPNNLess: flag to return the PSN as a PNN if no match
         :return: a list with unique PNNs matching those PSNs
         """
         mapping = self._CRICSiteQuery(callname='data-processing')
@@ -188,6 +189,9 @@ class CRIC(Service):
                     pnnSet.add(item['phedex_name'])
             if pnnSet:
                 pnns.update(pnnSet)
+            elif allowPNNLess:
+                pnns.add(psn)
+                self["logger"].debug("PSN %s has no PNNs. PNNLess flag enabled though.", psn)
             else:
                 self["logger"].debug("No PNNs for PSN: %s" % psn)
         return list(pnns)
diff --git a/src/python/WMCore/WorkQueue/WorkQueue.py b/src/python/WMCore/WorkQueue/WorkQueue.py
index 71c4ad32d..90e5e64f5 100644
--- a/src/python/WMCore/WorkQueue/WorkQueue.py
+++ b/src/python/WMCore/WorkQueue/WorkQueue.py
@@ -431,7 +431,7 @@ class WorkQueue(WorkQueueBase):
 
         # Keep in mind that WQE contains sites, wmbs location contains pnns
         commonSites = possibleSites(match)
-        commonLocation = self.cric.PSNstoPNNs(commonSites)
+        commonLocation = self.cric.PSNstoPNNs(commonSites, allowPNNLess=True)
         msg = "Running WMBS preparation for %s with ParentQueueId %s,\\n  with common location %s"
         self.logger.info(msg, match['RequestName'], match['ParentQueueId'], commonLocation)
 
diff --git a/src/python/WMQuality/Emulators/CRICClient/MockCRICApi.py b/src/python/WMQuality/Emulators/CRICClient/MockCRICApi.py
index 793530b89..a6a984110 100644
--- a/src/python/WMQuality/Emulators/CRICClient/MockCRICApi.py
+++ b/src/python/WMQuality/Emulators/CRICClient/MockCRICApi.py
@@ -86,7 +86,7 @@ class MockCRICApi(object):
                 psns.update(psnSet)
         return list(psns)
 
-    def PSNstoPNNs(self, psns):
+    def PSNstoPNNs(self, psns, allowPNNLess=False):
         callname = 'data-processing'
         mapping = self.genericLookup(callname)
         if isinstance(psns, basestring):
@@ -100,6 +100,8 @@ class MockCRICApi(object):
                     pnnSet.add(item['phedex_name'])
             if pnnSet:
                 pnns.update(pnnSet)
+            elif allowPNNLess:
+                pnns.add(psn)
         return list(pnns)
 
     def PSNtoPNNMap(self, psnPattern=''):
@@ -114,4 +116,4 @@ class MockCRICApi(object):
         for entry in results:
             if psnPattern.match(entry['psn_name']):
                 mapping.setdefault(entry['psn_name'], set()).add(entry['phedex_name'])
-        return mapping
\\ No newline at end of file
+        return mapping