Repository: dmwm/wmcore
Issue: [MS] MSUnmerged has been restarting in the t2t3 replica

**Impact of the bug**
MSUnmerged

**Describe the bug**
While checking the MSUnmerged logs, Todor noticed that the `t2t3` replica has been having some random service restarts, note that it's not the POD getting recreated, but the service restarts inside the same POD.

Reason is still unclear, and the only thing we can extract from the service logs is [1]

NOTE: this causes that MSUnmerged replica to keep cleaning up the same RSEs over and over, since the service cache gets wiped out during the restart process.

UPDATE: the `t1` POD service also gets restarted when FNAL is enabled.

**How to reproduce it**
Not clear yet! Note that 2 replicas work with no issues, only this t2t3 is having these problems.

**Expected behavior**
Investigate the reason for these service/CherryPy restarts and fix the MSUnmerged code; or whatever is making that replica unstable.

One comment that I made to Todor though, is that it would be interesting to log the amount of files retrieved from the Rucio ConMon, just so we know how many files need to be parsed in advanced. 

**Additional context and error message**
[1]
```
2021-08-02 02:19:42,617:WARNING:MSUnmerged: Error while cleaning RSE: T2_BR_UERJ. Will retry in the next cycle. Err: globus_ftp_client: the server responded with an error 550 550-GlobusError: v=1 c=PATH_NOT_FOUND  550-GridFTP-Errno: 2  55
0-GridFTP-Reason: System error in stat  550-GridFTP-Error-String: No such file or directory  550 End.   
2021-08-02 02:19:42,632:DEBUG:MSUnmerged: 
RSE: T2_BR_UERJ 
DELETING: /store/unmerged/SAM/testSRM/SAM-gridftp.hepgrid.uerj.br.
PFN list with: 15415 entries: 
    gsiftp://gridftp.hepgrid.uerj.br:2811/mnt/hadoop/cms/store/unmerged/SAM/testSRM/SAM-gridftp.hepgrid.uerj.br
    gsiftp://gridftp.hepgrid.uerj.br:2811/mnt/hadoop/cms/store/unmerged/SAM/testSRM/SAM-gridftp.hepgrid.uerj.br/lcg-util/testfile-put-nospacetoken-1557824813-6d2024e5235c.txt
    gsiftp://gridftp.hepgrid.uerj.br:2811/mnt/hadoop/cms/store/unmerged/SAM/testSRM/SAM-gridftp.hepgrid.uerj.br/lcg-util/testfile-put-nospacetoken-1557865488-9e5949b3c90b.txt
    gsiftp://gridftp.hepgrid.uerj.br:2811/mnt/hadoop/cms/store/unmerged/SAM/testSRM/SAM-gridftp.hepgrid.uerj.br/lcg-util/testfile-put-nospacetoken-1557867358-4acd14b37e4a.txt
    ...

[02/Aug/2021:02:19:44]  WATCHDOG: server exited with exit code signal 11 (core dumped)... restarting
[02/Aug/2021:02:19:44]  INFO: final CherryPy configuration: {'engine.autoreload.on': False,
...

### and another restart
2021-08-02 04:05:47,963:WARNING:MSUnmerged: Error while cleaning RSE: T2_CN_Beijing. Will retry in the next cycle. Err: globus_ftp_client: the server responded with an error 550 550-GlobusError: v=1 c=PATH_NOT_FOUND  550-GridFTP-Errno: 2 
 550-GridFTP-Reason: System error in name  550-GridFTP-Error-String: No such file or directory  550 End.   
2021-08-02 04:05:47,981:DEBUG:MSUnmerged: 
RSE: T2_CN_Beijing 
DELETING: /store/unmerged/SAM/testSRM/SAM-srm.ihep.ac.cn/lcg-util.
PFN list with: 15296 entries: 
    gsiftp://ccsrm.ihep.ac.cn:2811/dpm/ihep.ac.cn/home/cms/store/unmerged/SAM/testSRM/SAM-srm.ihep.ac.cn/lcg-util/testfile-put-nospacetoken-1603812057-d43e8b99c21d.txt
    gsiftp://ccsrm.ihep.ac.cn:2811/dpm/ihep.ac.cn/home/cms/store/unmerged/SAM/testSRM/SAM-srm.ihep.ac.cn/lcg-util/testfile-put-nospacetoken-1604034358-5dbc61d69427.txt
    gsiftp://ccsrm.ihep.ac.cn:2811/dpm/ihep.ac.cn/home/cms/store/unmerged/SAM/testSRM/SAM-srm.ihep.ac.cn/lcg-util/testfile-put-nospacetoken-1605627798-1b14ee8202a1.txt
    gsiftp://ccsrm.ihep.ac.cn:2811/dpm/ihep.ac.cn/home/cms/store/unmerged/SAM/testSRM/SAM-srm.ihep.ac.cn/lcg-util/testfile-put-nospacetoken-1607445255-faf41de51a14.txt
    ...

[02/Aug/2021:04:05:49]  WATCHDOG: server exited with exit code signal 11 (core dumped)... restarting
[02/Aug/2021:04:05:50]  INFO: final CherryPy configuration: {'engine.autoreload.on': False,
```

Changed files:
src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py

diff --git a/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py b/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
index e39c5a231..469955859 100644
--- a/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
+++ b/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
@@ -37,6 +37,7 @@ from WMCore.Services.WMStatsServer.WMStatsServer import WMStatsServer
 from WMCore.WMException import WMException
 from Utils.Pipeline import Pipeline, Functor
 from Utils.TwPrint import twFormat
+from Utils.IteratorTools import grouper
 
 # from memory_profiler import profile
 
@@ -99,6 +100,7 @@ class MSUnmerged(MSCore):
         self.msConfig.setdefault("dirFilterIncl", [])
         self.msConfig.setdefault("dirFilterExcl", [])
         self.msConfig.setdefault("emulateGfal2", False)
+        self.msConfig.setdefault("filesToDeleteSliceSize", 100)
         if self.msConfig['emulateGfal2'] is False and gfal2 is None:
             msg = "Failed to import gfal2 library while it's not "
             msg += "set to emulate it. Crashing the service!"
@@ -303,15 +305,17 @@ class MSUnmerged(MSCore):
                 if self.msConfig['enableRealMode']:
                     try:
                         # execute the actual deletion in bulk - full list of files per directory
-                        delResult = []
-                        delResult = ctx.unlink(pfnList)
 
-                        # Count all the successfully deleted files (if a deletion was
-                        # successful a value of None is put in the delResult list):
-                        deletedSuccess = [pfnStatus for pfnStatus in delResult if pfnStatus is None]
+                        deletedSuccess = 0
+                        for pfnSlice in list(grouper(pfnList, self.msConfig["filesToDeleteSliceSize"])):
+                            delResult = ctx.unlink(pfnSlice)
+                            # Count all the successfully deleted files (if a deletion was
+                            # successful a value of None is put in the delResult list):
+                            deletedSuccess += sum([1 for pfnStatus in delResult if pfnStatus is None])
+
                         self.logger.debug("RSE: %s, Dir: %s, deletedSuccess: %s",
                                           rse['name'], dirLfn, deletedSuccess)
-                        rse['counters']['deletedSuccess'] += len(deletedSuccess)
+                        rse['counters']['deletedSuccess'] = deletedSuccess
 
                         # Now clean the whole branch
                         self.logger.debug("Purging dirEntry: %s:\\n", dirPfn)