Repository: dmwm/wmcore
Issue: MSTransferor: dictionary changed size during iteration

**Impact of the bug**
MSTransferor

**Describe the bug**
Initial tests of the latest WMCore tag raise this exception in MSTransferor [1], where we update the dictionary while iterating over it.
Issue has been inserted with the slice 15 changes, in: https://github.com/dmwm/WMCore/commit/69b820f60a1c20e918614f5b9c5cccc987d302a7#diff-f8b8abf5646a54dcb250a3e23cc5d660442e254a587585d6173c82542fe200ddR432

**How to reproduce it**
Use `viewitems` to iterate over a dict that will change during iteration.

**Expected behavior**
We need to iterate over that dictionary using one of the methods that will expand all the keys/values in memory, instead of yield them on the fly. One of the options seem to be:
```
from future.utils import listitems
```
which will allow the dictionary to change during the for loop iterating over the dictionary items.

**Additional context and error message**
[1]
```
2021-03-08 10:03:52,435:INFO:MSTransferor: secondary dataset: /RelValMinBias_14TeV/CMSSW_10_6_1-106X_mcRun3_2021_realistic_v1_rsb-v1/GEN-SIM already in place. Common locations with site white/black list is: set([u'T1_US_FNAL_Disk'])
2021-03-08 10:03:52,435:ERROR:MSManager: MS daemon error: dictionary changed size during iteration
Traceback (most recent call last):
  File "/data/srv/HG2103g-comp/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre2/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSManager.py", line 46, in daemon
    func(reqStatus)
  File "/data/srv/HG2103g-comp/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre2/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSManager.py", line 162, in transferor
    res = self.msTransferor.execute(reqStatus)
  File "/data/srv/HG2103g-comp/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre2/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 197, in execute
    self.checkPUDataLocation(wflow)
  File "/data/srv/HG2103g-comp/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre2/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 432, in checkPUDataLocation
    for dset, dsetDict in viewitems(pileupInput):
RuntimeError: dictionary changed size during iteration
```

Changed files:
src/python/WMCore/MicroService/Unified/MSTransferor.py

diff --git a/src/python/WMCore/MicroService/Unified/MSTransferor.py b/src/python/WMCore/MicroService/Unified/MSTransferor.py
index e19040d7e..80c8b84c2 100644
--- a/src/python/WMCore/MicroService/Unified/MSTransferor.py
+++ b/src/python/WMCore/MicroService/Unified/MSTransferor.py
@@ -10,7 +10,7 @@ tasks might be extended to multi-threading in the future.
 """
 # futures
 from __future__ import division, print_function
-from future.utils import viewitems, listvalues
+from future.utils import viewitems, listvalues, listitems
 from future import standard_library
 standard_library.install_aliases()
 
@@ -326,7 +326,7 @@ class MSTransferor(MSCore):
             self.logger.info("Request %s has %d initial blocks from %s",
                              wflow.getName(), len(inputBlocks), methodName)
 
-            for block, blockDict in viewitems(inputBlocks):
+            for block, blockDict in listitems(inputBlocks):  # dict can change size here
                 blockLocation = self._diskPNNs(blockDict['locations'])
                 if primaryAAA and blockLocation:
                     msg = "Primary/parent block %s already in place (via AAA): %s" % (block, blockLocation)
@@ -417,7 +417,7 @@ class MSTransferor(MSCore):
 
         if secondaryAAA:
             # what matters is to have pileup dataset(s) available in ANY disk storage
-            for dset, dsetDict in viewitems(pileupInput):
+            for dset, dsetDict in listitems(pileupInput):  # dict can change size here
                 datasetLocation = self._diskPNNs(dsetDict['locations'])
                 msg = "it has secondary: %s, total size: %s GB, disk locations: %s"
                 self.logger.info(msg, dset, gigaBytes(dsetDict['dsetSize']), datasetLocation)
@@ -429,7 +429,7 @@ class MSTransferor(MSCore):
                     self.logger.info("secondary dataset %s not available even through AAA", dset)
         else:
             if len(pileupInput) == 1:
-                for dset, dsetDict in viewitems(pileupInput):
+                for dset, dsetDict in listitems(pileupInput):  # dict can change size here
                     datasetLocation = self._diskPNNs(dsetDict['locations'])
                     msg = "it has secondary: %s, total size: %s GB, current disk locations: %s"
                     self.logger.info(msg, dset, gigaBytes(dsetDict['dsetSize']), datasetLocation)
@@ -464,7 +464,7 @@ class MSTransferor(MSCore):
                 else:
                     self.logger.info("Largest secondary dataset %s not available in a common location. This is BAD!")
                 # now iterate normally through the pileup datasets
-                for dset, dsetDict in viewitems(pileupInput):
+                for dset, dsetDict in listitems(pileupInput):  # dict can change size here
                     datasetLocation = self._diskPNNs(dsetDict['locations'])
                     msg = "it has secondary: %s, total size: %s GB, current disk locations: %s"
                     self.logger.info(msg, dset, gigaBytes(dsetDict['dsetSize']), datasetLocation)