Repository: dmwm/wmcore
Issue: [py2py3] More runtime errors for Py3 agent on rhel7 OS

**Impact of the bug**
Py3 WMAgent

**Describe the bug**
I'm going to try to report all the problems I happen to see while testing Py3 WMAgent during job runtime, on rhel7 setup (thus, using scramArch slc7_* and python 3.8.2).

First issue is also common to the rhel6 jobs and already reported in the other issue linked from this one. For completeness, here's the error message section [1].

Second issue is related to StepChain workflows - and likely the failure of the previous step(s) - when passing an object that is not json serializable, see [2].

A suspicious line - and possibly another problem - can be seen in the error block [2], which is:
```
2021-08-06 19:20:06,264:INFO:ExecuteMaster:StepName: cmsRun1, StepType: CMSSW, with result: 'logArch1'
``` 
the `cmsRun1` process should have an integer result(!). It's the first time I see that "Executor Task Change" code though, so perhaps it's expected...

**How to reproduce it**
Py3 WMAgent 1.5.2.pre1 + 10726, run an slc7_* StepChain workflow.

**Expected behavior**
This issue needs to address all the job runtime related errors seen while commissioning Py3 WMAgent and slc7_* workload.

**Additional context and error message**
Issue tightly coupled to its rhel6 equivalent: https://github.com/dmwm/WMCore/issues/10734

[1]
```
2021-08-06 19:20:06,053:INFO:CMSSW:Running PRE scripts
2021-08-06 19:20:06,099:ERROR:ExecuteMaster:Exception occured when executing step
2021-08-06 19:20:06,099:ERROR:ExecuteMaster:Exception is a bytes-like object is required, not 'str'
2021-08-06 19:20:06,100:ERROR:ExecuteMaster:Traceback:
2021-08-06 19:20:06,101:ERROR:ExecuteMaster:Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/ExecuteMaster.py", line 141, in doExecution
    executionObject.execute()
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/CMSSW.py", line 197, in execute
    scriptProcess.stdin.write("export LD_LIBRARY_PATH=$LD_LIBRARY_PATH\\n")
TypeError: a bytes-like object is required, not 'str'

2021-08-06 19:20:06,101:CRITICAL:CMSSW:CMSDefaultHandler Diagnostic Handler invoked
```

[2]
```
2021-08-06 19:20:06,102:WARNING:Report:Exit code: 99109 has been already added to the job report
2021-08-06 19:20:06,103:INFO:CMSSW:Steps.Executors.CMSSW.post called
2021-08-06 19:20:06,103:INFO:ExecuteMaster:Post Executor Task Change: logArch1
2021-08-06 19:20:06,264:INFO:ExecuteMaster:StepName: cmsRun1, StepType: CMSSW, with result: 'logArch1'
2021-08-06 19:20:06,269:INFO:CMSSW:Steps.Executors.CMSSW.pre called
2021-08-06 19:20:06,357:INFO:CMSSW:Steps.Executors.CMSSW.execute called
2021-08-06 19:20:06,358:CRITICAL:CMSSW:Skipping this step due to a failure in a previous one.
2021-08-06 19:20:06,456:ERROR:ExecuteMaster:Exception occured when executing step
2021-08-06 19:20:06,456:ERROR:ExecuteMaster:Exception is Object of type bytes is not JSON serializable
2021-08-06 19:20:06,457:ERROR:ExecuteMaster:Traceback:
2021-08-06 19:20:06,463:ERROR:ExecuteMaster:Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/ExecuteMaster.py", line 141, in doExecution
    executionObject.execute()
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/CMSSW.py", line 111, in execute
    self._setStatus(99108, msg)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/CMSSW.py", line 57, in _setStatus
    self.setCondorChirpAttrDelayed('Chirp_WMCore_cmsRun_Exception_Message', returnMessage, compress=True)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executor.py", line 197, in setCondorChirpAttrDelayed
    args = [condor_chirp_bin, 'set_job_attr_delayed', key, json.dumps(value)]
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

2021-08-06 19:20:06,463:CRITICAL:CMSSW:CMSDefaultHandler Diagnostic Handler invoked
2021-08-06 19:20:06,464:WARNING:Report:Exit code: 99109 has been already added to the job report
2021-08-06 19:20:06,465:INFO:CMSSW:Steps.Executors.CMSSW.post called
2021-08-06 19:20:06,553:INFO:ExecuteMaster:StepName: cmsRun4, StepType: CMSSW, with result: 99109
2021-08-06 19:20:06,560:INFO:CMSSW:Steps.Executors.CMSSW.pre called
2021-08-06 19:20:06,674:INFO:CMSSW:Steps.Executors.CMSSW.execute called
2021-08-06 19:20:06,675:CRITICAL:CMSSW:Skipping this step due to a failure in a previous one.
2021-08-06 19:20:06,772:ERROR:ExecuteMaster:Exception occured when executing step
2021-08-06 19:20:06,773:ERROR:ExecuteMaster:Exception is Object of type bytes is not JSON serializable
2021-08-06 19:20:06,773:ERROR:ExecuteMaster:Traceback:
2021-08-06 19:20:06,774:ERROR:ExecuteMaster:Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/ExecuteMaster.py", line 141, in doExecution
    executionObject.execute()
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/CMSSW.py", line 111, in execute
    self._setStatus(99108, msg)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/CMSSW.py", line 57, in _setStatus
    self.setCondorChirpAttrDelayed('Chirp_WMCore_cmsRun_Exception_Message', returnMessage, compress=True)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executor.py", line 197, in setCondorChirpAttrDelayed
    args = [condor_chirp_bin, 'set_job_attr_delayed', key, json.dumps(value)]
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/cvmfs/cms.cern.ch/COMP/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
```

Changed files:
src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py, src/python/WMCore/WMSpec/Steps/Executor.py

diff --git a/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py b/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py
index 348c4ca4a..546035c1e 100644
--- a/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py
+++ b/src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py
@@ -15,6 +15,7 @@ import socket
 
 from PSetTweaks.PSetTweak import PSetTweak
 from PSetTweaks.WMTweak import  makeJobTweak, makeOutputTweak, makeTaskTweak, resizeResources
+from Utils.Utilities import decodeBytesToUnicode
 from WMCore.Storage.SiteLocalConfig import loadSiteLocalConfig
 from WMCore.Storage.TrivialFileCatalog import TrivialFileCatalog
 from WMCore.WMRuntime.ScriptInterface import ScriptInterface
@@ -133,10 +134,12 @@ class SetupCMSSWPset(ScriptInterface):
             funcArgs['scenario'] = scenario
 
         try:
-            with open(funcArgsJson, 'wb') as f:
+            with open(funcArgsJson, 'w') as f:
                 json.dump(funcArgs, f)
         except Exception as ex:
-            self.logger.exception("Error writing out process funcArgs json")
+            msg = "Error writing out process funcArgs json."
+            msg += "Type: {} and content: {}".format(type(funcArgs), funcArgs)
+            self.logger.exception(msg)
             raise ex
 
         cmd = "%s --output_pkl %s --funcname %s --funcargs %s" % (
@@ -404,6 +407,7 @@ class SetupCMSSWPset(ScriptInterface):
         fileList = []
         eventsAvailable = 0
         for pileupType in self.step.data.pileup.listSections_():
+            pileupType = decodeBytesToUnicode(pileupType)
             useAAA = True if getattr(self.jobBag, 'trustPUSitelists', False) else False
             self.logger.info("Pileup set to read data remotely: %s", useAAA)
             for blockName in sorted(pileupDict[pileupType].keys()):
@@ -411,12 +415,13 @@ class SetupCMSSWPset(ScriptInterface):
                 if PhEDExNodeName in blockDict["PhEDExNodeNames"] or useAAA:
                     eventsAvailable += int(blockDict.get('NumberOfEvents', 0))
                     for fileLFN in blockDict["FileList"]:
-                        fileList.append(str(fileLFN))
+                        fileList.append(decodeBytesToUnicode(fileLFN))
             newPileupDict[pileupType] = {"eventsAvailable": eventsAvailable, "FileList": fileList}
         newJsonPileupConfig = os.path.join(self.stepSpace.location, "CMSSWPileupConfig.json")
         self.logger.info("Generating json for CMSSW pileup script")
         try:
-            with open(newJsonPileupConfig, 'wb') as f:
+            # If it's a python2 unicode, cmssw_handle_pileup will cast it to str
+            with open(newJsonPileupConfig, 'w') as f:
                 json.dump(newPileupDict, f)
         except Exception as ex:
             self.logger.exception("Error writing out process filelist json:")
diff --git a/src/python/WMCore/WMSpec/Steps/Executor.py b/src/python/WMCore/WMSpec/Steps/Executor.py
index 6ae0d21bc..f387b582c 100644
--- a/src/python/WMCore/WMSpec/Steps/Executor.py
+++ b/src/python/WMCore/WMSpec/Steps/Executor.py
@@ -16,7 +16,8 @@ import subprocess
 import sys
 
 from Utils.FileTools import getFullPath
-from Utils.Utilities import zipEncodeStr
+from Utils.PythonVersion import PY3
+from Utils.Utilities import zipEncodeStr, decodeBytesToUnicodeConditional
 from WMCore.FwkJobReport.Report import Report
 from WMCore.WMSpec.Steps.StepFactory import getStepEmulator
 from WMCore.WMSpec.WMStep import WMStepHelper
@@ -177,9 +178,12 @@ class Executor(object):
         Util to call condor_chirp and publish the key/value pair
 
         """
+
         if compress:
             value = zipEncodeStr(value, maxLen=maxLen)
 
+        # bytes object is not JSON serializable in Python3
+        value = decodeBytesToUnicodeConditional(value, condition=PY3)
         # construct condor_chirp binary location from CONDOR_CONFIG
         # Note: This works when we do not use containers.
         condor_chirp_bin = None