Repository: dmwm/wmcore
Issue: fix adding job location in DBSBuffer when more than one se is set for given job

It seems when there is more se names involved for a given job which goes in the same site, not all the files in a job set at the same se.
In that case dbs file and location association  won't set correctly sometimes.
Update:
Actually it might not be from the same job. 
following line contains files from multiple jobs.
https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/JobAccountant/AccountantWorker.py#L620
The question is why this never happened before?

Anyway, following line is set inside the loop. it will always record the last se only.
https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/JobAccountant/AccountantWorker.py#L678
and insertion happens outside the loop.
https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/JobAccountant/AccountantWorker.py#L698

This can be fixed by collecting all the se names outside the loop and insert.

IMHO, we need to put something else than se name for location for dbs original_site_name. maybe site name or PhEDEx name
But this need to be discussed in other issue.


Changed files:
src/python/WMComponent/JobAccountant/AccountantWorker.py

diff --git a/src/python/WMComponent/JobAccountant/AccountantWorker.py b/src/python/WMComponent/JobAccountant/AccountantWorker.py
index bbfc2b4c4..d46385a30 100644
--- a/src/python/WMComponent/JobAccountant/AccountantWorker.py
+++ b/src/python/WMComponent/JobAccountant/AccountantWorker.py
@@ -124,7 +124,7 @@ class AccountantWorker(WMConnectionBase):
         self.count = 0
         self.datasetAlgoID     = collections.deque(maxlen = 1000)
         self.datasetAlgoPaths  = collections.deque(maxlen = 1000)
-        self.dbsLocations      = collections.deque(maxlen = 1000)
+        self.dbsLocations      = set()
         self.workflowIDs       = collections.deque(maxlen = 1000)
         self.workflowPaths     = collections.deque(maxlen = 1000)
 
@@ -611,11 +611,11 @@ class AccountantWorker(WMConnectionBase):
             return
 
         dbsFileTuples = []
-        dbsLocations  = []
         dbsFileLoc    = []
         dbsCksumBinds = []
         runLumiBinds  = []
         selfChecksums = None
+        jobLocations  = set()
 
         for dbsFile in self.dbsFilesToCreate:
             # Append a tuple in the format specified by DBSBufferFiles.Add
@@ -676,7 +676,7 @@ class AccountantWorker(WMConnectionBase):
             lfn           = dbsFile['lfn']
             selfChecksums = dbsFile['checksums']
             jobLocation   = dbsFile.getLocations()[0]
-
+            jobLocations.add(jobLocation)
             dbsFileTuples.append((lfn, dbsFile['size'],
                                   dbsFile['events'], assocID,
                                   dbsFile['status'], workflowID))
@@ -693,12 +693,14 @@ class AccountantWorker(WMConnectionBase):
                                           'cktype' : entry})
 
         try:
+            
+            diffLocation = jobLocations.difference(self.dbsLocations)
 
-            if not jobLocation in self.dbsLocations:
+            for jobLocation in diffLocation:
                 self.dbsInsertLocation.execute(siteName = jobLocation,
                                                conn = self.getDBConn(),
                                                transaction = self.existingTransaction())
-                self.dbsLocations.append(jobLocation)
+                self.dbsLocations.add(jobLocation)
 
             self.dbsCreateFiles.execute(files = dbsFileTuples,
                                         conn = self.getDBConn(),