Repository: dmwm/wmcore
Issue: WMAgent fails to load spec pickle file created in Python3

**Impact of the bug**
ReqMgr2/WorkQueue and WMAgent

**Describe the bug**
Workflows created in Python3 WMCore central services version `1.5.0.pre3`, running on Python2 WMAgent 1.4.7.patch3, fail in the agent's WorkQueueManager because the agent isn't able to load the pickle file retrieved from ReqMgr2 (actually it comes from the workqueue database, but it should be exactly the same!). Error message can be seen in [1]

**How to reproduce it**
See scenario and versions in the comment above

**Expected behavior**
We need to adopt a common pickle protocol that will work and provide the same behaviour in both python2 and python3 version. Which means, WorkQueueManager should not fail to load the workflow spec file when a workflow is being acquired.

NOTE: even though the problem happens in WMAgent, we should make sure this issue is addressed before Python3 central services are deployed. Such that we can avoid spec creation incompatibilities.

**Additional context and error message**
Related to: https://github.com/dmwm/WMCore/issues/9128

[1]
```
2021-06-21 22:44:45,141:140647810840320:INFO:WorkQueueManagerWorkPoller:Obtained 3 unit(s) of work
2021-06-21 22:44:45,250:140647810840320:INFO:WorkQueueManagerWorkPoller:Splitting new work
2021-06-21 22:44:45,250:140647810840320:INFO:WorkQueue:fixing conflict...
2021-06-21 22:44:45,340:140647810840320:ERROR:WorkQueueManagerWorkPoller:Error during split: unsupported pickle protocol: 4
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.4.7.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMComponent/WorkQueueManager/WorkQueueManagerWorkPoller.py", line 114, in processWork
    self.queue.processInboundWork()
  File "/data/srv/wmagent/v1.4.7.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WorkQueue.py", line 1138, in processInboundWork
    inbound_work = self.backend.getElementsForSplitting()
  File "/data/srv/wmagent/v1.4.7.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/WorkQueue/WorkQueueBackend.py", line 111, in getElementsForSplitting
    wmspec.load(self.parentCouchUrlWithAuth + "/%s/spec" % ele['RequestName'])
  File "/data/srv/wmagent/v1.4.7.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.4.7.patch3/lib/python2.7/site-packages/WMCore/WMSpec/Persistency.py", line 78, in load
    self.data = pickle.loads(data[0])
ValueError: unsupported pickle protocol: 4
2021-06-21 22:44:45,341:140647810840320:INFO:WorkQueueManagerWorkPoller:Splitting finished
2021-06-21 22:44:45,341:140647810840320:INFO:BaseWorkerThread:WorkQueueManagerWorkPoller took 0.501 secs to execute
```

Changed files:
src/python/WMCore/WMSpec/Persistency.py, src/python/WMCore/DataStructs/JobPackage.py, src/python/WMCore/WMSpec/Steps/Templates/CMSSW.py

diff --git a/src/python/WMCore/DataStructs/JobPackage.py b/src/python/WMCore/DataStructs/JobPackage.py
index b39c4553a..063b226bb 100644
--- a/src/python/WMCore/DataStructs/JobPackage.py
+++ b/src/python/WMCore/DataStructs/JobPackage.py
@@ -37,7 +37,7 @@ class JobPackage(WMObject, dict):
         Pickle this object and save it to disk.
         """
         with open(fileName, 'wb') as fileHandle:
-            pickle.dump(self, fileHandle, -1)
+            pickle.dump(self, fileHandle, protocol=pickle.HIGHEST_PROTOCOL)
         return
 
     def load(self, fileName):
diff --git a/src/python/WMCore/WMSpec/Persistency.py b/src/python/WMCore/WMSpec/Persistency.py
index 581ad73d4..2320411c1 100644
--- a/src/python/WMCore/WMSpec/Persistency.py
+++ b/src/python/WMCore/WMSpec/Persistency.py
@@ -46,7 +46,8 @@ class PersistencyHelper(object):
         with open(filename, 'wb') as handle:
             # TODO: use different encoding scheme for different extension
             # extension = filename.split(".")[-1].lower()
-            pickle.dump(self.data, handle)
+            # FIXME: once both central services and WMAgent are in Py3, we can remove protocol=0
+            pickle.dump(self.data, handle, protocol=0)
         return
 
     def load(self, filename):
@@ -99,7 +100,8 @@ class PersistencyHelper(object):
             rev = doc['_rev']
 
         # specuriwrev = specuri + '?rev=%s' % rev
-        workloadString = pickle.dumps(self.data)
+        # FIXME: once both central services and WMAgent are in Py3, we can remove protocol=0
+        workloadString = pickle.dumps(self.data, protocol=0)
         # result = database.put(specuriwrev, workloadString, contentType='application/text')
         retval = database.addAttachment(name, rev, workloadString, 'spec')
         if retval.get('ok', False) is not True:
diff --git a/src/python/WMCore/WMSpec/Steps/Templates/CMSSW.py b/src/python/WMCore/WMSpec/Steps/Templates/CMSSW.py
index 5fff88a42..3daac77f3 100644
--- a/src/python/WMCore/WMSpec/Steps/Templates/CMSSW.py
+++ b/src/python/WMCore/WMSpec/Steps/Templates/CMSSW.py
@@ -156,7 +156,8 @@ class CMSSWStepHelper(CoreHelper):
             [setattr(self.data.application.configuration.arguments, k, v) for k, v in viewitems(args)]
         except Exception:
             pass
-        self.data.application.configuration.pickledarguments = pickle.dumps(args)
+        # FIXME: once both central services and WMAgent are in Py3, we can remove protocol=0
+        self.data.application.configuration.pickledarguments = pickle.dumps(args, protocol=0)
         return
 
     def cmsswSetup(self, cmsswVersion, **options):
@@ -209,7 +210,8 @@ class CMSSWStepHelper(CoreHelper):
         if hasattr(self.data.application.configuration, "pickledarguments"):
             args = pickle.loads(self.data.application.configuration.pickledarguments)
         args['globalTag'] = globalTag
-        self.data.application.configuration.pickledarguments = pickle.dumps(args)
+        # FIXME: once both central services and WMAgent are in Py3, we can remove protocol=0
+        self.data.application.configuration.pickledarguments = pickle.dumps(args, protocol=0)
 
         return
 
@@ -238,7 +240,8 @@ class CMSSWStepHelper(CoreHelper):
         if hasattr(self.data.application.configuration, "pickledarguments"):
             args = pickle.loads(self.data.application.configuration.pickledarguments)
         args['datasetName'] = datasetName
-        self.data.application.configuration.pickledarguments = pickle.dumps(args)
+        # FIXME: once both central services and WMAgent are in Py3, we can remove protocol=0
+        self.data.application.configuration.pickledarguments = pickle.dumps(args, protocol=0)
 
         return