Repository: dmwm/wmcore
Issue: JobAccountant still crashes on files produced without any location info

**Impact of the bug**
WMAgent, JobAccountant crash

**Describe the bug**
The root cause of the problem is still to be found. However, there has been files produced and staged out which are written to the job report without any location, which makes JobAccountant crash when inserting those files into WMBS.

Attempt to fix it was made here:
https://github.com/dmwm/WMCore/pull/9354
but now it crashes a bit later in the chain.

**How to reproduce it**
TBI

**Expected behavior**
First, JobAccountant cannot crash. Second - if possible to be fixed on the same PR, files **should** always get back with a valid location.

**Additional context and error message**
First fix made here: https://github.com/dmwm/WMCore/pull/9354

Current traceback is:
```
2019-10-18 13:47:30,409:140648760407808:INFO:AccountantWorker:Handling /storage/local/data1/cmsdataops/srv/wmagent/v1.2.6.patch1/install/wmagent/JobCreator/JobCache/cmsunified_task_BTV-RunIISummer19UL17SIM-00036__v1_T_191011_04110
0_4240/BTV-RunIISummer19UL17SIM-00036_0/JobCollection_104358_1/job_1250755/Report.0.pkl
2019-10-18 13:47:30,419:140648760407808:WARNING:AccountantWorker:Job 1250755 , list of expected outputModules does not match job report, accepted for multi-step CMSSW job
2019-10-18 13:47:30,419:140648760407808:WARNING:AccountantWorker:The following file doesn't have any location: {'branch_hash': 'ababfbfb3c3e13090c093288b9316164', 'lfn': '/store/unmerged/RunIISummer19UL17RECO/QCD_Pt_300to470_TuneC
P5_13TeV_pythia8/AODSIM/106X_mc2017_realistic_v6-v2/240001/BAF05E4C-CB43-ED4C-9E3F-779E695B931B.root', 'locations': set([]), 'dataset': {'applicationName': 'cmsRun', 'processedDataset': 'RunIISummer19UL17RECO-106X_mc2017_realistic
_v6-v2', 'primaryDataset': 'QCD_Pt_300to470_TuneCP5_13TeV_pythia8', 'dataTier': 'AODSIM', 'applicationVersion': 'CMSSW_10_6_2'}, 'first_event': 0, 'checksums': {'adler32': '9a123f9d', 'cksum': '3707539574'}, 'acquisitionEra': 'Run
IISummer19UL17RECO', 'size': 1110973320, 'guid': 'BAF05E4C-CB43-ED4C-9E3F-779E695B931B', 'outputModule': 'AODSIMoutput', 'configURL': 'https://cmsweb.cern.ch/couchdb;;reqmgr_config_cache;;e9125943b5c87bbdd91187d7641a6ca3', 'parent
s': set([]), 'input': ['../cmsRun3/RAWSIMoutput.root'], 'events': 2700, 'merged': False, 'validStatus': 'PRODUCTION', 'globalTag': '106X_mc2017_realistic_v6', 'inputpfns': ['../cmsRun3/RAWSIMoutput.root'], 'pfn': '/srv/job/WMTaskS
pace/cmsRun4/AODSIMoutput.root', 'catalog': '', 'module_label': 'AODSIMoutput', 'inputPath': '/QCD_Pt_300to470_TuneCP5_13TeV_pythia8/RunIISummer19UL17GEN-106X_mc2017_realistic_v6-v1/GEN', 'runs': set([<WMCore.DataStructs.Run.Run o
bject at 0x7feb563ab510>]), 'branches': [], 'last_event': 0, 'processingVer': 2, 'fileRef': <WMCore.Configuration.ConfigSection object at 0x7feb563e3810>, 'prep_id': 'BTV-RunIISummer19UL17RECO-00003'}
2019-10-18 13:47:30,419:140648760407808:WARNING:AccountantWorker:Job 1250755 , bad jobReport, failing job
2019-10-18 13:47:30,422:140648760407808:ERROR:JobAccountantPoller:Hit general exception in JobAccountantPoller while using worker.
list index out of range
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 237, in __call__
    fwkJobReport=fwkJobReport)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 555, in handleJob
    jobID=jobID, task=fwkJobReport.getTaskName())
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 427, in addFileToWMBS
    wmbsFile = self.createFileFromDataStructsFile(fname=fwjrFile, jobID=jobID)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 879, in createFileFromDataStructsFile
    pnn = list(fname["locations"])[0]
IndexError: list index out of range
2019-10-18 13:47:30,495:140648760407808:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7feb578ec3d0> <@========== WMException Start ==========@>
Exception Class: JobAccountantPollerException
Message: Hit general exception in JobAccountantPoller while using worker.
list index out of range
        ModuleName : WMComponent.JobAccountant.JobAccountantPoller
        MethodName : algorithm
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py
        ClassName : None
        LineNumber : 90
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 237, in __call__
    fwkJobReport=fwkJobReport)

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 555, in handleJob
    jobID=jobID, task=fwkJobReport.getTaskName())

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 427, in addFileToWMBS
    wmbsFile = self.createFileFromDataStructsFile(fname=fwjrFile, jobID=jobID)

  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 879, in createFileFromDataStructsFile
    pnn = list(fname["locations"])[0]

<@---------- WMException End ----------@>  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 182, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMCore/Database/DBExceptionHandler.py", line 39, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/Utils/Timers.py", line 24, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v1.2.6.patch1/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 90, in algorithm
    raise JobAccountantPollerException(msg)

2019-10-18 13:47:30,496:140648760407808:INFO:Harness:>>>Terminating worker threads
2019-10-18 13:47:30,500:140648760407808:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7feb578ec3d0> <@========== WMException Start ==========@>
Exception Class: JobAccountantPollerException
```


Changed files:
src/python/WMComponent/JobAccountant/AccountantWorker.py

diff --git a/src/python/WMComponent/JobAccountant/AccountantWorker.py b/src/python/WMComponent/JobAccountant/AccountantWorker.py
index 671f62e37..19084f0f9 100644
--- a/src/python/WMComponent/JobAccountant/AccountantWorker.py
+++ b/src/python/WMComponent/JobAccountant/AccountantWorker.py
@@ -542,6 +542,8 @@ class AccountantWorker(WMConnectionBase):
                 wmbsJob["outcome"] = "success"
             else:
                 wmbsJob["outcome"] = "failure"
+                self.listOfJobsToFail.append(wmbsJob)
+                return jobSuccess
 
             for fwjrFile in fileList:
 
@@ -571,15 +573,11 @@ class AccountantWorker(WMConnectionBase):
 
             # Check if the job had any skipped files, put them in ACDC containers
             # We assume full file processing (no job masks)
-            if jobSuccess:
-                skippedFiles = fwkJobReport.getAllSkippedFiles()
-                if skippedFiles and jobType not in ['LogCollect', 'Cleanup']:
-                    self.jobsWithSkippedFiles[jobID] = skippedFiles
+            skippedFiles = fwkJobReport.getAllSkippedFiles()
+            if skippedFiles and jobType not in ['LogCollect', 'Cleanup']:
+                self.jobsWithSkippedFiles[jobID] = skippedFiles
 
-            if jobSuccess:
-                self.listOfJobsToSave.append(wmbsJob)
-            else:
-                self.listOfJobsToFail.append(wmbsJob)
+            self.listOfJobsToSave.append(wmbsJob)
 
         return jobSuccess