Repository: dmwm/wmcore
Issue: MSUnmerged: Unordered checks for Rucio Consistency Monitor record age

**Impact of the bug**
MSUnmerged

**Describe the bug**
While looking at the service logs today, I have found a strange case: [1]. 

The explanation is quite simple. The RSE is not in status `done` at Rucio Consistency Monitor and hence the field at  `self.rseConsStats[rseName]['end_time'] ` is `None`: [2]. While this is foreseen to happen and we do handle the situation [few lines bellow](https://github.com/dmwm/WMCore/blob/80212e6765ed460ac45c5f8de0fb2d672a51f735/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L501-L508) the problematic one, it is the early creation of the boolean flag `isConsNewer` at line [483](https://github.com/dmwm/WMCore/blob/80212e6765ed460ac45c5f8de0fb2d672a51f735/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L483) that triggers the exception. The end result is not fatal, though. The RSE is still skipped as expected, but it is only that it exits the pipeline with a general exception instead of the normal way as it should be by triggering a `MSUnmergedPlineExit` exception [3], and the timestamps of the RSE are not updated at MongoDB. Since this is an RSE with already stale record, once its status is changed to `done` in Rucio Consistency Monitor it will be safely retried in MSUnmerged. So no harm is done so far. But we better fix that behavior.     

**How to reproduce it**
Try to run the MSUnmerged in standalone mode against an RSE which is not in status `done` at Rucio Consistency Monitor.

**Expected behavior**
Exit the pipeline through the normal mechanism and log the latest timestamps for this RSE in MongoDB.

**Additional context and error message**


[1]
```
2022-03-10 10:05:32,698:ERROR:MSUnmerged:_execute(): plineUnmerged: General error from pipeline. RSE: T2_US_Vanderbilt. Error: '>' not supported between instances of 'NoneType' and 'float' Will retry again in the next cycle.
Traceback (most recent call last):
  File "/afs/cern.ch/user/t/tivanov/WMCoreDev.d/WMCore/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py", line 269, in _execute
    pline.run(MSUnmergedRSE(rseName))
  File "/afs/cern.ch/user/t/tivanov/WMCoreDev.d/WMCore/src/python/Utils/Pipeline.py", line 140, in run
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/afs/cern.ch/user/t/tivanov/WMCoreDev.d/WMCore/src/python/Utils/Pipeline.py", line 140, in <lambda>
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/afs/cern.ch/user/t/tivanov/WMCoreDev.d/WMCore/src/python/Utils/Pipeline.py", line 72, in __call__
    return self.run(obj)
  File "/afs/cern.ch/user/t/tivanov/WMCoreDev.d/WMCore/src/python/Utils/Pipeline.py", line 75, in run
    return self.func(obj, *self.args, **self.kwargs)
  File "/afs/cern.ch/user/t/tivanov/WMCoreDev.d/WMCore/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py", line 487, in consRecordAge
    isConsNewer = self.rseConsStats[rseName]['end_time'] > self.rseTimestamps[rseName]['prevStartTime']
TypeError: '>' not supported between instances of 'NoneType' and 'float'

```

[2]
```
2022-03-10 10:05:32,698:INFO:MSUnmerged:consRecordAge(): rseConsStats[T2_US_Vanderbilt]: {'end_time': None,
 'files': None,
 'roots': [],
 'rse': 'T2_US_Vanderbilt',
 'run': '2022_03_07_08_19',
 'scanner': {'type': 'xrootd', 'version': '2.0'},
 'scanning': {'ignore_subdirectories': ['logs', 'SAM/testSRM'],
              'max_scanners': 8,
              'recursive_threshold': 1,
              'root': 'unmerged',
              'start_time': 1646641162.9798942,
              'timeout': 180},
 'server': 'xrootd-vanderbilt.sites.opensciencegrid.org',
 'server_root': '/store',
 'start_time': 1646641162.9397197,
 'status': 'started',
 'total_size_gb': None}
2022-03-10 10:05:32,698:INFO:MSUnmerged:consRecordAge(): rseTimestamps[T2_US_Vanderbilt]: {'endTime': 0.0,
 'prevEndTime': 0.0,
 'prevStartTime': 0.0,
 'rseConsStatTime': 0.0,
 'startTime': 1646903132.698025}
```

[3]
```
2022-03-10 10:11:45,262:WARNING:MSUnmerged:_execute(): plineUnmerged: Run on RSE: T2_US_Vanderbilt was interrupted due to: MSUnmergedException: MSUnmergedPlineExit: RSE: T2_US_Vanderbilt In non-final state in Rucio Consistency Monitor. Skipping it in the current run. Will retry again in the next cycle.
```

Changed files:
src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py

diff --git a/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py b/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
index f28ca1a5c..ef5d56198 100644
--- a/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
+++ b/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
@@ -480,8 +480,16 @@ class MSUnmerged(MSCore):
             raise MSUnmergedPlineExit(msg)
 
         isConsDone = self.rseConsStats[rseName]['status'] == 'done'
-        isConsNewer = self.rseConsStats[rseName]['end_time'] > self.rseTimestamps[rseName]['prevStartTime']
+        if not isConsDone:
+            msg = "RSE: %s has a non-final Rucio ConMon status: %s. " % (rseName, self.rseConsStats[rseName]['status'])
+            msg += "Skipping it in the current run."
+            self.logger.warning(msg)
+            rse['rucioConMonStatus'] = self.rseConsStats[rseName]['status']
+            self.updateRSETimestamps(rse, start=False, end=True)
+            self.uploadRSEToMongoDB(rse)
+            raise MSUnmergedPlineExit(msg)
 
+        isConsNewer = self.rseConsStats[rseName]['end_time'] > self.rseTimestamps[rseName]['prevStartTime']
         if not isConsNewer:
             msg = "RSE: %s With old consistency record in Rucio Consistency Monitor. " % rseName
             if 'isClean' in rse and rse['isClean']:
@@ -498,15 +506,6 @@ class MSUnmerged(MSCore):
                 self.logger.info(msg)
                 rse['rucioConMonStatus'] = self.rseConsStats[rseName]['status']
 
-        if not isConsDone:
-            msg = "RSE: %s In non-final state in Rucio Consistency Monitor. " % rseName
-            msg += "Skipping it in the current run."
-            self.logger.warning(msg)
-            rse['rucioConMonStatus'] = self.rseConsStats[rseName]['status']
-            self.updateRSETimestamps(rse, start=False, end=True)
-            self.uploadRSEToMongoDB(rse)
-            raise MSUnmergedPlineExit(msg)
-
         if isConsNewer and isConsDone:
             # NOTE: If we've got to this point then we have a brand new record for
             #       the RSE in RucioConMOn and we are then about to start a new RSE cleanup