Repository: dmwm/wmcore
Issue: JobAccountant crashing with UnicodeDecodeError exception

**Impact of the bug**
WMAgent

**Describe the bug**
This is a similar problem to: https://github.com/dmwm/WMCore/issues/9926
and I expected to have it resolved with: https://github.com/dmwm/WMCore/pull/9312

Anyhow, JobAccountant finds a byte (or more than it) in the job FJR that it cannot decode with the `ascii` codec. Thus crashing the whole component.

**How to reproduce it**
Can be done by loading the respect job report, which we still have to find out which one was that.

**Expected behavior**
We should be using `utf-8` standard for encoding/decoding these strings. In addition to that, it's expected that any runtime information that can't be parsed with utf-8, should be dropped before it gets persisted into the job FJR (usually worker node logs from the stage out step).

**Additional context and error message**
JobAccountant log from vocms0252:
```
2020-12-03 22:55:40,929:140503924397824:WARNING:AccountantWorker:Job 1736833 , list of expected outputModules does not match job report, accepted for multi-step CMSSW job
2020-12-03 22:55:40,929:140503924397824:INFO:AccountantWorker:Job 1736833 , handle successful job
2020-12-03 22:55:40,935:140503924397824:INFO:AccountantWorker:Handling /data/srv/wmagent/v1.4.1.patch3/install/wmagent/JobCreator/JobCache/cmsunified_task_TOP-RunIISummer20UL16wmLHEGEN-00090__v1_T_201130_132025_4004/TOP-RunIISummer20UL16w
mLHEGEN-00090_0/JobCollection_193528_0/job_1736834/Report.0.pkl
2020-12-03 22:55:40,959:140503924397824:ERROR:JobAccountantPoller:Hit general exception in JobAccountantPoller while using worker.
'ascii' codec can't decode byte 0xff in position 1: ordinal not in range(128)
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)
  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 229, in __call__
    fwkJobReport = self.loadJobReport(job["fwjr_path"])
  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 181, in loadJobReport
    msg += str(ex)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1: ordinal not in range(128)
2020-12-03 22:55:41,084:140503924397824:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.JobAccountant.JobAccountantPoller.JobAccountantPoller object at 0x7fc9a6a11490> <@========== WMException Start ==========@>
Exception Class: JobAccountantPollerException
Message: Hit general exception in JobAccountantPoller while using worker.
'ascii' codec can't decode byte 0xff in position 1: ordinal not in range(128)
        ModuleName : WMComponent.JobAccountant.JobAccountantPoller
        MethodName : algorithm
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py
        ClassName : None
        LineNumber : 90
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 70, in algorithm
    self.accountantWorker(jobsSlice)

  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 229, in __call__
    fwkJobReport = self.loadJobReport(job["fwjr_path"])

  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/AccountantWorker.py", line 181, in loadJobReport
    msg += str(ex)

<@---------- WMException End ----------@>  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 182, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMCore/Database/DBExceptionHandler.py", line 39, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/Utils/Timers.py", line 25, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v1.4.1.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.1.patch3/lib/python2.7/site-packages/WMComponent/JobAccountant/JobAccountantPoller.py", line 90, in algorithm
    raise JobAccountantPollerException(msg)

2020-12-03 22:55:41,084:140503924397824:INFO:Harness:>>>Terminating worker threads
```


Changed files:
src/python/WMComponent/JobAccountant/AccountantWorker.py

diff --git a/src/python/WMComponent/JobAccountant/AccountantWorker.py b/src/python/WMComponent/JobAccountant/AccountantWorker.py
index bea536400..04c4c50d4 100644
--- a/src/python/WMComponent/JobAccountant/AccountantWorker.py
+++ b/src/python/WMComponent/JobAccountant/AccountantWorker.py
@@ -176,9 +176,11 @@ class AccountantWorker(WMConnectionBase):
 
         try:
             jobReport.load(jobReportPath)
+        except UnicodeDecodeError:
+            logging.error("Hit UnicodeDecodeError exception while loading jobReport: %s", jobReportPath)
+            return self.createMissingFWKJR(99997, 'Found undecodable data in jobReport: {}'.format(jobReportPath))
         except Exception as ex:
-            msg = "Error loading jobReport %s\\n" % jobReportPath
-            msg += str(ex)
+            msg = "Error loading jobReport: {}\\nDetails: {}".format(jobReportPath, str(ex))
             logging.error(msg)
             return self.createMissingFWKJR(99997, 'Cannot load jobReport')