Repository: dmwm/wmcore
Issue: [py2py3] Use pickle protocol=4 for job package in Python3 agents

**Impact of the bug**
WMAgent

**Describe the bug**
As we commission WMAgent in python3, it looks like we are not going to have a consistent python3 version between `rhel6` and `rhel7` OS, for:
* `rhel6`, thus slc6_*: we are planning to actually use the CMSSW python3.6.4
* `rhel7`, thus slc7_*: we are going to use our own COMP python3.8.2

Python 3.6.4 does not support pickle protocol=5, so we cannot simply create pickle files with the standard protocol in python 3.8.2. Otherwise it fails to be loaded during job runtime with:
```
Failed to load JobPackage:/srv/job/WMSandbox/JobPackage.pcl
unsupported pickle protocol: 5
```

**How to reproduce it**
Run a test workflow with Py3 WMAgent >= 1.5.1.pre6 (plus a few extra patches)

**Expected behavior**
My suggestion is to evaluate the python version when creating the job package in JobCreator
* if it's a python2, then simply create it with the highest protocol available
* if it's a python3 env, then create the pickle files with protocol 4

**Additional context and error message**
This PR contains a lot of information: https://github.com/dmwm/WMCore/pull/10619#issuecomment-894079860
And here is the job wrapper changes: https://github.com/dmwm/WMCore/pull/10726

Changed files:
src/python/WMCore/DataStructs/JobPackage.py, src/python/Utils/PythonVersion.py, src/python/WMComponent/JobCreator/JobCreatorPoller.py

diff --git a/src/python/Utils/PythonVersion.py b/src/python/Utils/PythonVersion.py
index f31281e88..18e68c66b 100644
--- a/src/python/Utils/PythonVersion.py
+++ b/src/python/Utils/PythonVersion.py
@@ -5,6 +5,13 @@ Easily get the version of the python interpreter at runtime
 from __future__ import division # Jenkins CI
 
 import sys
+import pickle
 
 PY3 = sys.version_info[0] == 3
 PY2 = sys.version_info[0] == 2
+
+# We need to keep compatibility between multiple python versions
+# Further details at: https://github.com/dmwm/WMCore/pull/10726
+# For PY2: set highest protocol to 2
+# For PY3: set highest protocol to 4 (compatible with python3.6 and python3.8)
+HIGHEST_PICKLE_PROTOCOL = pickle.HIGHEST_PROTOCOL if PY2 else 4
\\ No newline at end of file
diff --git a/src/python/WMComponent/JobCreator/JobCreatorPoller.py b/src/python/WMComponent/JobCreator/JobCreatorPoller.py
index 61895783d..a6f512326 100644
--- a/src/python/WMComponent/JobCreator/JobCreatorPoller.py
+++ b/src/python/WMComponent/JobCreator/JobCreatorPoller.py
@@ -12,12 +12,14 @@ import os
 import os.path
 import threading
 
+
 try:
     import cPickle as pickle
 except ImportError:
     import pickle
 
 from Utils.Timers import timeFunction
+from Utils.PythonVersion import HIGHEST_PICKLE_PROTOCOL
 from Utils.MathUtils import quantize
 from WMComponent.JobCreator.CreateWorkArea import CreateWorkArea
 from WMCore.WorkerThreads.BaseWorkerThread import BaseWorkerThread
@@ -156,7 +158,7 @@ def saveJob(job, workflow, sandbox, wmTask=None, jobNumber=0,
     job['allowOpportunistic'] = allowOpportunistic
 
     with open(os.path.join(cacheDir, 'job.pkl'), 'wb') as output:
-        pickle.dump(job, output, pickle.HIGHEST_PROTOCOL)
+        pickle.dump(job, output, HIGHEST_PICKLE_PROTOCOL)
 
     return
 
diff --git a/src/python/WMCore/DataStructs/JobPackage.py b/src/python/WMCore/DataStructs/JobPackage.py
index 063b226bb..c943f2508 100644
--- a/src/python/WMCore/DataStructs/JobPackage.py
+++ b/src/python/WMCore/DataStructs/JobPackage.py
@@ -4,12 +4,11 @@ _JobPackage_
 
 Data structure for storing and retreiving multiple job objects.
 """
-
 try:
     import cPickle as pickle
 except ImportError:
     import pickle
-
+from Utils.PythonVersion import HIGHEST_PICKLE_PROTOCOL
 from WMCore.DataStructs.WMObject import WMObject
 
 
@@ -37,7 +36,7 @@ class JobPackage(WMObject, dict):
         Pickle this object and save it to disk.
         """
         with open(fileName, 'wb') as fileHandle:
-            pickle.dump(self, fileHandle, protocol=pickle.HIGHEST_PROTOCOL)
+            pickle.dump(self, fileHandle, protocol=HIGHEST_PICKLE_PROTOCOL)
         return
 
     def load(self, fileName):