Repository: dmwm/wmcore
Issue: Report message encoding error in JobCreator

**Impact of the bug**
WMAgents

**Describe the bug**
I've seen an issue with JobCreator - while creating a job area - and it looks like WMException class is unable to encode all the error message raised from JobCreator, which actually masks the original exception/error in the component.

**How to reproduce it**
not sure

**Expected behavior**
Error messages should be properly encoded/decoded using utf-8 codec. If needed we can ignore and bypass when a byte/word can't be encoded/decoded, it's only for error reporting anyways.

**Additional context and error message**
Exception from submit8, running the latest WMAgent tag 1.3.6:
```
2020-08-08 01:43:29,680:140258839922432:INFO:JobCreatorPoller:Found end in iteration over subscription 12124
2020-08-08 01:43:29,755:140258839922432:INFO:JobCreatorPoller:Retrieved 1 jobGroups from jobSplitter
2020-08-08 01:43:29,783:140258839922432:ERROR:CreateWorkArea:Error in creating directories: mkdir: cannot create directory ‘/storage/local/data1/cmsdataops/srv/wmagent/v1.3.6.patch4/install/wmagent/JobCreator/JobCache/cmsunified_ACDC0_task_JME-RunIISummer19UL16GENAPV-00004__v1_T_200808_062137_1106/JME-RunIISummer19UL16GENAPV-00004_0MergeRAWSIMoutput/JobCollection_78499_0/job_1124238’: File exists


2020-08-08 01:43:29,784:140258839922432:ERROR:JobCreatorPoller:Exception in processing wmbsJobGroup 78499
. Error: 'ascii' codec can't encode character u'\\u2018' in position 62: ordinal not in range(128)
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/JobCreatorPoller.py", line 208, in creatorProcess
    cache=False)
  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/CreateWorkArea.py", line 170, in processJobs
    self.createWorkArea(cache=cache)
  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/CreateWorkArea.py", line 273, in createWorkArea
    createDirectories(nameList)
  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/CreateWorkArea.py", line 42, in createDirectories
    raise CreateWorkAreaException(msg)
  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMCore/WMException.py", line 33, in __init__
    message = message.decode('utf-8', 'ignore')
  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2018' in position 62: ordinal not in range(128)
2020-08-08 01:43:29,884:140258839922432:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.JobCreator.JobCreatorPoller.JobCreatorPoller object at 0x7f908e8688d0> <@========== WMException Start ==========@>
Exception Class: JobCreatorException
Message: Exception in processing wmbsJobGroup 78499
. Error: 'ascii' codec can't encode character u'\\u2018' in position 62: ordinal not in range(128)
        ModuleName : WMComponent.JobCreator.JobCreatorPoller
        MethodName : creatorProcess
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/JobCreatorPoller.py
        ClassName : None
        LineNumber : 233
        ErrorNr : 0
Traceback: 
  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/JobCreatorPoller.py", line 208, in creatorProcess
    cache=False)

  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/CreateWorkArea.py", line 170, in processJobs
    self.createWorkArea(cache=cache)

  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/CreateWorkArea.py", line 273, in createWorkArea
    createDirectories(nameList)

  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMComponent/JobCreator/CreateWorkArea.py", line 42, in createDirectories
    raise CreateWorkAreaException(msg)

  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/cms/wmagent/1.3.6.patch4/lib/python2.7/site-packages/WMCore/WMException.py", line 33, in __init__
    message = message.decode('utf-8', 'ignore')

  File "/data/srv/wmagent/v1.3.6.patch4/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
```

Changed files:
src/python/WMCore/WMException.py, test/python/WMCore_t/WMException_t.py

diff --git a/src/python/WMCore/WMException.py b/src/python/WMCore/WMException.py
index ebf51fbd3..bdf3a9fb8 100644
--- a/src/python/WMCore/WMException.py
+++ b/src/python/WMCore/WMException.py
@@ -6,7 +6,7 @@ General Exception class for WM modules
 
 """
 
-from builtins import str
+from builtins import bytes
 from future.utils import viewitems
 
 import inspect
@@ -29,7 +29,7 @@ class WMException(Exception):
 
     def __init__(self, message, errorNo=None, **data):
         self.name = str(self.__class__.__name__)
-        if hasattr(message, "decode"):
+        if type(message) == bytes:
             # Fix for the unicode encoding issue, see #8056 and #8403
             # interprets this string using utf-8 codec and ignoring any errors
             message = message.decode('utf-8', 'ignore')
@@ -153,7 +153,7 @@ class WMException(Exception):
         strg += self.traceback
         strg += '\\n'
         strg += WMEXCEPTION_END_STR
-        if hasattr(strg, "decode"):
+        if type(strg) == bytes:
             # Fix for the unicode encoding issue, #8043
             strg = strg.decode('utf-8', 'ignore')
         return strg
diff --git a/test/python/WMCore_t/WMException_t.py b/test/python/WMCore_t/WMException_t.py
index c3483ebab..08e5918ca 100644
--- a/test/python/WMCore_t/WMException_t.py
+++ b/test/python/WMCore_t/WMException_t.py
@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 """
 _WMException_t_
 
@@ -50,5 +51,21 @@ class WMExceptionTest(unittest.TestCase):
         exception.addInfo(**data)
         self.logger.debug("String version of exception: " + str(exception))
 
+    def testExceptionUnicode(self):
+        """
+        create an exception with non-ascii characters and do some tests.
+        """
+
+        exception = WMException("an exception message with nr. 100 and some non-ascii characters: ₩♏ℭ☺яε", 100)
+        self.logger.debug("String version of exception: " + str(exception))
+        self.logger.debug("XML version of exception: " + exception.xml())
+        self.logger.debug("Adding data")
+        data = {}
+        data['key1'] = 'value1'
+        data['key2'] = 'data2'
+        exception.addInfo(**data)
+        self.logger.debug("String version of exception: " + str(exception))
+
+
 if __name__ == "__main__":
     unittest.main()