Repository: dmwm/wmcore
Issue: JobStatusLite crash with codec can't encode char 

Error message is
```
'ascii' codec can't encode character u'\\u2018' in position 1424: ordinal not in range(128)
```
even though the source code is defining the `utf-8` codec. In addition to that, it has the `ignore` value, which I'd expect to skip anything that can't be decoded...

Full traceback as follows:
```
2019-04-15 04:45:52,072:139744026318592:ERROR:SimpleCondorPlugin:No job report for job with id 5058715 and gridid 95267.128
2019-04-15 04:45:52,152:139744026318592:ERROR:SimpleCondorPlugin:No job report for job with id 5182715 and gridid 95964.25
2019-04-15 04:45:52,156:139744026318592:ERROR:BossAirAPI:Exception while completing jobs!
'ascii' codec can't encode character u'\\u2018' in position 1424: ordinal not in range(128)
2019-04-15 04:45:58,932:139744026318592:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMCore.BossAir.StatusPoller.StatusPoller object at 0x7f18b1a0a490> <@========== WMException Start ==========@>
Exception Class: BossAirException
Message: Exception while completing jobs!
'ascii' codec can't encode character u'\\u2018' in position 1424: ordinal not in range(128)
        ModuleName : WMCore.BossAir.BossAirAPI
        MethodName : _complete
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py
        ClassName : None
        LineNumber : 534
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py", line 526, in _complete
    self.plugins[plugin].complete(jobsToComplete[plugin])

  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/BossAir/Plugins/SimpleCondorPlugin.py", line 324, in complete
    condorReport.addError(exitType, exitCode, exitType, logOutput)

  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/FwkJobReport/Report.py", line 568, in addError
    errDetails.details = errorDetails.decode('utf-8', 'ignore')

  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)


<@---------- WMException End ----------@>  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 182, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/Database/DBExceptionHandler.py", line 35, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/Utils/Timers.py", line 24, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/BossAir/StatusPoller.py", line 68, in algorithm
    self.checkStatus()
  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/BossAir/StatusPoller.py", line 92, in checkStatus
    runningJobs = self.bossAir.track()
  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py", line 494, in track
    self._complete(jobs=jobsToComplete)
  File "/data/srv/wmagent/v1.1.20.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.patch3/lib/python2.7/site-packages/WMCore/BossAir/BossAirAPI.py", line 534, in _complete
    raise BossAirException(msg)

```

Changed files:
src/python/WMCore/FwkJobReport/Report.py

diff --git a/src/python/WMCore/FwkJobReport/Report.py b/src/python/WMCore/FwkJobReport/Report.py
index c46c5d0ba..694747ce6 100644
--- a/src/python/WMCore/FwkJobReport/Report.py
+++ b/src/python/WMCore/FwkJobReport/Report.py
@@ -598,14 +598,21 @@ class Report(object):
         errDetails.exitCode = exitCode
         errDetails.type = str(errorType)
 
-        if hasattr(errorDetails, "decode"):
-            # Fix for the unicode encoding issue, #8043
-            # interprets this string using utf-8 codec and ignoring any errors
-            errDetails.details = errorDetails.decode('utf-8', 'ignore')
-        else:
-            # Then cast it to string and decode it
-            errorDetails = str(errorDetails)
-            errDetails.details = errorDetails.decode('utf-8', 'ignore')
+        try:
+            if hasattr(errorDetails, "decode"):
+                # Fix for the unicode encoding issue, #8043
+                # interprets this string using utf-8 codec and ignoring any errors
+                errDetails.details = errorDetails.decode('utf-8', 'ignore')
+            else:
+                # Then cast it to string and decode it
+                errorDetails = str(errorDetails)
+                errDetails.details = errorDetails.decode('utf-8', 'ignore')
+        except UnicodeEncodeError as ex:
+            msg = "Failed to decode the job error details for job ID: %s." % self.getJobID()
+            msg += "\\nException message: %s\\nOriginal error details: %s" % (str(ex), errorDetails)
+            logging.error(msg)
+            msg = "DEFAULT ERROR MESSAGE, because it failed to UTF-8 decode the original message."
+            errDetails.details = msg
 
         setattr(stepSection.errors, "errorCount", errorCount + 1)
         self.setStepStatus(stepName=stepName, status=exitCode)