Repository: dmwm/wmcore
Issue: JobAccountant still crashes on files produced without any location info

**Impact of the bug**
WMAgent, JobAccountant crash

**Describe the bug**
The root cause of the problem is still to be found. However, there has been files produced and staged out which are written to the job report without any location, which makes JobAccountant crash when inserting those files into WMBS.

Attempt to fix it was made here:
https://github.com/dmwm/WMCore/pull/9354
but now it crashes a bit later in the chain.

**How to reproduce it**
TBI

**Expected behavior**
First, JobAccountant cannot crash. Second - if possible to be fixed on the same PR, files **should** always get back with a valid location.

**Additional context and error message**
First fix made here: https://github.com/dmwm/WMCore/pull/9354

Current traceback is:
```
2019-10-18 13:47:30,409:140648760407808:INFO:AccountantWorker:Handling /storage/local/data1/cmsdataops/srv/wmagent/v1.2.6.patch1/install/wmagent/JobCreator/JobCache/cmsunified_task_BTV-RunIISummer19UL17SIM-00036__v1_T_191011_04110
0_4240/BTV-RunIISummer19UL17SIM-00036_0/JobCollection_104358_1/job_1250755/Report.0.pkl
2019-10-18 13:47:30,419:140648760407808:WARNING:AccountantWorker:Job 1250755 , list of expected outputModules does not match job report, accepted for multi-step CMSSW job
2019-10-18 13:47:30,419:140648760407808:WARNING:AccountantWorker:The following file doesn't have any location: {'branch_hash': 'ababfbfb3c3e13090c093288b9316164', 'lfn': '/store/unmerged/RunIISummer19UL17RECO/QCD_Pt_300to470_TuneC
P5_13TeV_pythia8/AODSIM/106X_mc2017_realistic_v6-v2/240001/BAF05E4C-CB43-ED4C-9E3F-779E695B931B.root', 'locations': set([]), 'dataset': {'applicationName': 'cmsRun', 'processedDataset': 'RunIISummer19UL17RECO-106X_mc2017_realistic
_v6-v2', 'primaryDataset': 'QCD_Pt_300to470_TuneCP5_13TeV_pythia8', 'dataTier': 'AODSIM', 'applicationVersion': 'CMSSW_10_6_2'}, 'first_event': 0, 'checksums': {'adler32': '9a123f9d', 'cksum': '3707539574'}, 'acquisitionEra': 'Run
IISummer19UL17RECO', 'size': 1110973320, 'guid': 'BAF05E4C-CB43-ED4C-9E3F-779E695B931B', 'outputModule': 'AODSIMoutput', 'configURL': 'https://cmsweb.cern.ch/couchdb;;reqmgr_config_cache;;e9125943b5c87bbdd91187d7641a6ca3', 'parent
s': set([]), 'input': ['../cmsRun3/RAWSIMoutput.root'], 'events': 2700, 'merged': False, 'validStatus': 'PRODUCTION', 'globalTag': '106X_mc2017_realistic_v6', 'inputpfns': ['../cmsRun3/RAWSIMoutput.root'], 'pfn': '/srv/job/WMTaskS
pace/cmsRun4/AODSIMoutput.root', 'catalog': '', 'module_label': 'AODSIMoutput', 'inputPath': '/QCD_Pt_300to470_TuneCP5_13TeV_pythia8/RunIISummer19UL17GEN-106X_mc2017_realistic_v6-v1/GEN', 'runs': set([<WMCore.DataStructs.Run.Run o
bject at 0x7feb563ab510>]), 'branches': [], 'last_event': 0, 'processingVer': 2, 'fileRef': <WMCore.Configuration.ConfigSection object at 0x7feb563e3810>, 'prep_id': 'BTV-RunIISummer19UL17RECO-00003'}
2019-10-18 13:47:30,419:140648760407808:WARNING:AccountantWorker:Job 1250755 , bad jobReport, failing job
2019-10-18 13:47:30,422:140648760407808:ERROR:JobAccountantPoller:Hit general exception in JobAccountantPoller while using worker.
list index out of range
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 237, in __call__
    fwkJobReport=fwkJobReport)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 555, in handleJob
    jobID=jobID, task=fwkJobReport.getTaskName())
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 427, in addFileToWMBS
    wmbsFile = self.createFileFromDataStructsFile(fname=fwjrFile, jobID=jobID)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 879, in createFileFromDataStructsFile
    pnn = list(fname["locations"])[0]
IndexError: list index out of range
2019-10-18 13:47:30,495:140648760407808:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7feb578ec3d0> <@========== WMException Start ==========@>
Exception Class: JobAccountantPollerException
Message: Hit general exception in JobAccountantPoller while using worker.
list index out of range
        ModuleName : WMComponent.JobAccountant.JobAccountantPoller
        MethodName : algorithm
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py
        ClassName : None
        LineNumber : 90
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 237, in __call__
    fwkJobReport=fwkJobReport)

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 555, in handleJob
    jobID=jobID, task=fwkJobReport.getTaskName())

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 427, in addFileToWMBS
    wmbsFile = self.createFileFromDataStructsFile(fname=fwjrFile, jobID=jobID)

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 879, in createFileFromDataStructsFile
    pnn = list(fname["locations"])[0]

<@---------- WMException End ----------@>  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 182, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMCore/Database/DBExceptionHandler.py", line 39, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/Utils/Timers.py", line 24, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 90, in algorithm
    raise JobAccountantPollerException(msg)

2019-10-18 13:47:30,496:140648760407808:INFO:Harness:>>>Terminating worker threads
2019-10-18 13:47:30,500:140648760407808:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7feb578ec3d0> <@========== WMException Start ==========@>
Exception Class: JobAccountantPollerException
```


Changed files:
src/python/WMCore/FwkJobReport/Report.py, src/python/WMComponent/JobAccountant/AccountantWorker.py

diff --git a/src/python/WMComponent/JobAccountant/AccountantWorker.py b/src/python/WMComponent/JobAccountant/AccountantWorker.py
index 671f62e37..dd7fb6a24 100644
--- a/src/python/WMComponent/JobAccountant/AccountantWorker.py
+++ b/src/python/WMComponent/JobAccountant/AccountantWorker.py
@@ -496,7 +496,9 @@ class AccountantWorker(WMConnectionBase):
             # Make sure every file has a valid location
             # see https://github.com/dmwm/WMCore/issues/9353
             for fwjrFile in fileList:
-                if not fwjrFile.get("locations"):
+                # T0 has analysis file without any location, see:
+                # https://github.com/dmwm/WMCore/issues/9497
+                if not fwjrFile.get("locations") and fwjrFile.get("lfn", "").endswith(".root"):
                     logging.warning("The following file doesn't have any location: %s", fwjrFile)
                     jobSuccess = False
                     break
@@ -871,9 +873,9 @@ class AccountantWorker(WMConnectionBase):
         wmbsFile = File()
         wmbsFile.update(fname)
 
-        if isinstance(fname["locations"], set):
+        if fname["locations"] and isinstance(fname["locations"], set):
             pnn = list(fname["locations"])[0]
-        elif isinstance(fname["locations"], list):
+        elif fname["locations"] and isinstance(fname["locations"], list):
             if len(fname['locations']) > 1:
                 logging.error("Have more then one location for a file in job %i", jobID)
                 logging.error("Choosing location %s", fname['locations'][0])
diff --git a/src/python/WMCore/FwkJobReport/Report.py b/src/python/WMCore/FwkJobReport/Report.py
index 79b3ed119..928632ceb 100644
--- a/src/python/WMCore/FwkJobReport/Report.py
+++ b/src/python/WMCore/FwkJobReport/Report.py
@@ -431,7 +431,7 @@ class Report(object):
 
         Add an output file to the outputModule provided.
         """
-
+        logging.info("addOutputFile method called with outputModule: %s, aFile: %s", outputModule, aFile)
         aFile = aFile or {}
 
         # Now load the output module and create the file object
@@ -442,6 +442,7 @@ class Report(object):
         fileSection = "file%s" % count
         outMod.files.section_(fileSection)
         fileRef = getattr(outMod.files, fileSection)
+        logging.info("addOutputFile method fileRef: %s, whole tree: %s", fileRef, fileRef.dictionary_whole_tree_())
         outMod.files.fileCount += 1
 
         # Now we need to eliminate the optional and non-primitives: