Repository: dmwm/wmcore
Issue: JobSubmitter crashing while loading pickle files

**Impact of the bug**
WMAgent (vocms0283)

**Describe the bug**
This node had to be rebooted this morning, after having a load of >>1000 and becoming unresponsive.
It looks like it has also been affected by the CEPH outage, and some (at least 1 so far) job pickle files are empty. 

The agent crashes when loading such empty pickle file, e.g.:
/data/srv/wmagent/v1.2.8/install/wmagent/JobCreator/JobCache/pgunnell_Run2018D-v1-SingleMuon-12Nov2019_UL2018_1064p1_200113_212951_2532/DataProcessing/DataProcessingMergeSKIMStreamMuTau/JobCollection_146408_0/job_1271395/job.pkl

**How to reproduce it**
none

**Expected behavior**
Ideally, we should recreate that job pickle file. Problem is that the whole job creation logic happens in the JobCreator, when handling a WQE/block/fileset and performing the job splitting algorithm.

I could say that we should not crash the component as well, but in that case, we would need to bypass the pickle file, which would cause incomplete statistics in the output...

I'm creating this issue to keep track of this problem and how we deal with it.

**Additional context and error message**
Component traceback:
```
2020-01-28 11:15:09,178:140481657870080:INFO:JobSubmitterPoller:Determining possible sites for new jobs...
2020-01-28 11:15:14,570:140481657870080:INFO:JobSubmitterPoller:Processed 5000/50000 new jobs.
2020-01-28 11:15:23,534:140481657870080:INFO:JobSubmitterPoller:Processed 10000/50000 new jobs.
2020-01-28 11:15:28,142:140481657870080:ERROR:JobSubmitterPoller:Error while loading pickled job object /data/srv/wmagent/v1.2.8/install/wmagent/JobCreator/JobCache/pgunnell_Run2018D-v1-SingleMuon-12Nov2019_UL2018_1064p1_200113_212951_253
2/DataProcessing/DataProcessingMergeSKIMStreamMuTau/JobCollection_146408_0/job_1271395/job.pkl

2020-01-28 11:15:28,203:140481657870080:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.JobSubmitter.JobSubmitterPoller.JobSubmitterPoller object at 0x7fc476ae07d0> <@========== WMException Start ==========@>
Exception Class: JobSubmitterPollerException
Message: Error while loading pickled job object /data/srv/wmagent/v1.2.8/install/wmagent/JobCreator/JobCache/pgunnell_Run2018D-v1-SingleMuon-12Nov2019_UL2018_1064p1_200113_212951_2532/DataProcessing/DataProcessingMergeSKIMStreamMuTau/JobC
ollection_146408_0/job_1271395/job.pkl

        ModuleName : WMComponent.JobSubmitter.JobSubmitterPoller
        MethodName : refreshCache
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/JobSubmitter/JobSubmitterPoller.py
        ClassName : None
        LineNumber : 322
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/JobSubmitter/JobSubmitterPoller.py", line 317, in refreshCache
    loadedJob = pickle.load(jobHandle)

<@---------- WMException End ----------@>  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 182, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Database/DBExceptionHandler.py", line 39, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/Utils/Timers.py", line 24, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/JobSubmitter/JobSubmitterPoller.py", line 817, in algorithm
    self.refreshCache()
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/JobSubmitter/JobSubmitterPoller.py", line 322, in refreshCache
    raise JobSubmitterPollerException(msg)

2020-01-28 11:15:28,203:140481657870080:INFO:Harness:>>>Terminating worker threads
2020-01-28 11:15:28,275:140481657870080:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.JobSubmitter.JobSubmitterPoller.JobSubmitterPoller object at 0x7fc476ae07d0> <@========== WMException Start ==========@>
Exception Class: JobSubmitterPollerException
Message: Error while loading pickled job object /data/srv/wmagent/v1.2.8/install/wmagent/JobCreator/JobCache/pgunnell_Run2018D-v1-SingleMuon-12Nov2019_UL2018_1064p1_200113_212951_2532/DataProcessing/DataProcessingMergeSKIMStreamMuTau/JobCollection_146408_0/job_1271395/job.pkl

        ModuleName : WMComponent.JobSubmitter.JobSubmitterPoller
        MethodName : refreshCache
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/JobSubmitter/JobSubmitterPoller.py
        ClassName : None
        LineNumber : 322
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/JobSubmitter/JobSubmitterPoller.py", line 317, in refreshCache
    loadedJob = pickle.load(jobHandle)

<@---------- WMException End ----------@>
Backtrace:
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 208, in __call__
    raise ex

2020-01-28 11:15:28,891:140481657870080:INFO:BaseWorkerThread:Worker thread <WMComponent.JobSubmitter.JobSubmitterPoller.JobSubmitterPoller object at 0x7fc476ae07d0> terminated
```


Changed files:
src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py, src/python/WMCore/WMExceptions.py, src/python/WMCore/Agent/DefaultConfig.py

diff --git a/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py b/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py
index 3fc7820ea..585c2bd85 100644
--- a/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py
+++ b/src/python/WMComponent/JobSubmitter/JobSubmitterPoller.py
@@ -271,7 +271,7 @@ class JobSubmitterPoller(BaseWorkerThread):
         # make a counter for jobs pending to sites in drain mode within the grace period
         countDrainingJobs = 0
         timeNow = int(time.time())
-        badJobs = dict([(x, []) for x in range(71101, 71105)])
+        badJobs = dict([(x, []) for x in range(71101, 71106)])
         newJobIds = set()
 
         logging.info("Refreshing priority cache with currently %i jobs", len(self.jobDataCache))
@@ -309,17 +309,16 @@ class JobSubmitterPoller(BaseWorkerThread):
 
             if not os.path.isfile(pickledJobPath):
                 # Then we have a problem - there's no file
-                logging.error("Could not find pickled jobObject %s", pickledJobPath)
-                badJobs[71103].append(newJob)
+                logging.warning("Could not find pickled jobObject %s", pickledJobPath)
+                badJobs[71104].append(newJob)
                 continue
             try:
                 with open(pickledJobPath, 'r') as jobHandle:
                     loadedJob = pickle.load(jobHandle)
             except Exception as ex:
-                msg = "Error while loading pickled job object %s\\n" % pickledJobPath
-                msg += str(ex)
-                logging.error(msg)
-                raise JobSubmitterPollerException(msg)
+                logging.warning("Failed to load job pickle object %s", pickledJobPath)
+                badJobs[71105].append(newJob)
+                continue
 
             # figure out possible locations for job
             possibleLocations = loadedJob["possiblePSN"]
@@ -360,7 +359,7 @@ class JobSubmitterPoller(BaseWorkerThread):
                     elif self.failJobDrain(timeNow, possibleLocations):
                         newJob['possibleSites'] = possibleLocations
                         logging.warning("Job id %s can only run at a sites in Draining state", jobID)
-                        badJobs[71104].append(newJob)
+                        badJobs[71103].append(newJob)
                         continue
                     else:
                         countDrainingJobs += 1
@@ -413,8 +412,13 @@ class JobSubmitterPoller(BaseWorkerThread):
 
         # Register failures in submission
         for errorCode in badJobs:
-            if badJobs[errorCode]:
-                logging.debug("The following jobs could not be submitted: %s, error code : %d", badJobs, errorCode)
+            if badJobs[errorCode] and errorCode in [71101, 71102, 71103]:
+                msg = "%d jobs failed to be submitted due to location constraints (error code: %s)"
+                logging.warning(msg, len(badJobs[errorCode]), errorCode)
+                self._handleSubmitFailedJobs(badJobs[errorCode], errorCode)
+            elif badJobs[errorCode] and errorCode in [71104, 71105]:
+                msg = "%d jobs failed to be submitted with unrecoverable job pickle problems (error code: %s)"
+                logging.warning(msg, len(badJobs[errorCode]), errorCode)
                 self._handleSubmitFailedJobs(badJobs[errorCode], errorCode)
 
         # Persist remaining job packages to disk
@@ -481,7 +485,7 @@ class JobSubmitterPoller(BaseWorkerThread):
         for job in badJobs:
             job['couch_record'] = None
             job['fwjr'] = Report()
-            if exitCode in [71102, 71104]:
+            if exitCode in [71102, 71103]:
                 job['fwjr'].addError("JobSubmit", exitCode, "SubmitFailed",
                                      WM_JOB_ERROR_CODES[exitCode] + ', '.join(job['possibleSites']),
                                      ', '.join(job['possibleSites']))
diff --git a/src/python/WMCore/Agent/DefaultConfig.py b/src/python/WMCore/Agent/DefaultConfig.py
index 57fac8dc1..dbc1e5439 100644
--- a/src/python/WMCore/Agent/DefaultConfig.py
+++ b/src/python/WMCore/Agent/DefaultConfig.py
@@ -16,7 +16,7 @@ DEFAULT_AGENT_CONFIG = {
     # In JobSumitter, submit jobs over the threshold if the priority is higher than current pending/running jobs
     "CondorOverflowFraction": 0.2,  # default 20% over threshold
     # ExitCodes returned by jobs which doesn't need to be retried
-    "NoRetryExitCodes": [70, 73, 8001, 8006, 8009, 8023, 8026, 8501, 50660, 50661, 50664, 71102],
+    "NoRetryExitCodes": [70, 73, 8001, 8006, 8009, 8023, 8026, 8501, 50660, 50661, 50664, 71102, 71104, 71105],
     # Number of times a job will be retried
     "MaxRetries" : {'default': 3, 'Merge': 4, 'Cleanup': 2, 'LogCollect': 1, 'Harvesting': 4},
     # Speed drain thresholds - a draining agent's config is updated when # of remaining condor jobs is below a threshold
diff --git a/src/python/WMCore/WMExceptions.py b/src/python/WMCore/WMExceptions.py
index 6b822448e..dedd5b131 100644
--- a/src/python/WMCore/WMExceptions.py
+++ b/src/python/WMCore/WMExceptions.py
@@ -185,8 +185,9 @@ WM_JOB_ERROR_CODES = {-1: "Error return without specification.",
                       70452: "No run/lumi information in file (WMAgent).",
                       71101: "No sites are available to submit the job because the location of its input(s) do not pass the site whitelist/blacklist restrictions (WMAgent).",
                       71102: "The job can only run at a site that is currently in Aborted state (WMAgent).",
-                      71103: "The JobSubmitter component could not load the job pickle (WMAgent).",
-                      71104: "The job can run only at a site that is currently in Draining state (WMAgent).",
+                      71103: "The job can run only at a site that is currently in Draining state (WMAgent).",
+                      71104: "JobSubmitter component could not find a job pickle object.",
+                      71105: "JobSubmitter component loaded an empty job pickle object.",
                       71300: "The job was killed by the WMAgent, reason is unknown (WMAgent).",
                       71301: "The job was killed by the WMAgent because the site it was running at was set to Aborted (WMAgent).",
                       71302: "The job was killed by the WMAgent because the site it was running at was set to Draining (WMAgent).",