Repository: rucio/rucio

Issue:
When multiple Reaper instances are running on the same node, the RSE configuration somehow gets mixed
For example, on a VM where a Reaper instance is configured like so with Supervisor:
```
[program:reaper]
command=/usr/bin/rucio-reaper2 --threads 50  --include-rses "(type=CALIBDISK|type=DATADISK|type=GROUPDISK|type=LOCALGROUPDISK|type=SCRATCHDISK|type=TEST|CERN-PROD_TZDISK)\quarantinereaper=True"
…
```
Starting an instance on a particular RSE:
```
# rucio-reaper2 --include-rses CERN-PROD_TEST-CTA --run-once
2020-10-13 14:17:58,861	21869	INFO	main: starting processes
2020-10-13 14:17:58,891	21869	INFO	Reaper: This instance will work on RSEs: AUSTRALIA-ATLAS_DATADISK, AUSTRALIA-ATLAS_LOCALGROUPDISK, AUSTRALIA-ATLAS_SCRATCHDISK, AUSTRALIA-ATLAS_T2ATLASLOCALGROUPDISK, BEIJING-LCG2_DATADISK, BEIJING-LCG2_LOCALGROUPDISK, BNL-OSG2_DATADISK, BNL-OSG2_DET-SLHC, BNL-OSG2_LOCALGROUPDISK, BNL-OSG2_PHYS-HI, BNL-OSG2_SCRATCHDISK, BNLHPC_SCRATCHDISK, CA-SFU-T2_DATADISK, CA-SFU-T2_LOCALGROUPDISK, CA-SFU-T2_SCRATCHDISK, CA-TRIUMF-DYNAFED_DATADISK, CA-TRIUMF-DYNAFED_SCRATCHDISK, CA-UVIC-CLOUD_DATADISK, CA-UVIC-CLOUD_SCRATCHDISK, CA-VICTORIA-WESTGRID-T2_DATADISK, CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK, CA-VICTORIA-WESTGRID-T2_SCRATCHDISK, CA-WATERLOO-T2_DATADISK, CA-WATERLOO-T2_LOCALGROUPDISK, CA-WATERLOO-T2_SCRATCHDISK, CERN-EXTENSION_SCRATCHDISK, CERN-PROD_DATADISK, CERN-PROD_DATAPREP, CERN-PROD_DET-FWD, CERN-PROD_DET-IBL, CERN-PROD_DET-INDET, CERN-PROD_DET-LARG, CERN-PROD_DET-MUON, CERN-PROD_DET-SLHC, CERN-PROD_DET-TILE, CERN-PROD_LOCALGROUPDISK, CERN-PROD_PERF-EGAMMA, CERN-PROD_PERF-FLAVTAG, CERN-PROD_PERF-IDTRACKING, CERN-PROD_PERF-JETS, CERN-PROD_PERF-MUONS, CERN-PROD_PERF-TAU, CERN-PROD_PHYS-BEAUTY, CERN-PROD_PHYS-EXOTICS, CERN-PROD_PHYS-GENER, CERN-PROD_PHYS-HDBS, CERN-PROD_PHYS-HI, CERN-PROD_PHYS-HIGGS, CERN-PROD_PHYS-SM, CERN-PROD_PHYS-SUSY, CERN-PROD_PHYS-TOP, CERN-PROD_PPSTAPE, CERN-PROD_PROJ-SIT, CERN-PROD_SCRATCHDISK, CERN-PROD_SOFT-SIMUL, CERN-PROD_TEST-CTA, CERN-PROD_TRIG-DAQ, CERN-PROD_TRIG-HLT, CERN-PROD_TZDISK, CSCS-LCG2_DATADISK, CSCS-LCG2_LOCALGROUPDISK, CSCS-LCG2_SCRATCHDISK, CYFRONET-LCG2_DATADISK, CYFRONET-LCG2_LOCALGROUPDISK, CYFRONET-LCG2_PHYS-HI, CYFRONET-LCG2_SCRATCHDISK, DESY-HH_DATADISK, DESY-HH_LOCALGROUPDISK, DESY-HH_SCRATCHDISK, DESY-ZN_DATADISK, DESY-ZN_LOCALGROUPDISK, DESY-ZN_PHYS-TOP, DESY-ZN_SCRATCHDISK, EELA-UTFSM_DATADISK, EELA-UTFSM_LOCALGROUPDISK, EELA-UTFSM_SCRATCHDISK, FMPHI-UNIBA_DATADISK, FMPHI-UNIBA_LOCALGROUPDISK, FMPHI-UNIBA_SCRATCHDISK, FR-ALPES_DATADISK_DATALAKES, FR-ALPES_SCRATCHDISK_DATALAKES, FZK-LCG2_DATADISK, FZK-LCG2_PERF-EGAMMA, FZK-LCG2_PERF-IDTRACKING, FZK-LCG2_PERF-TAU, FZK-LCG2_SCRATCHDISK, GOEGRID_DATADISK, GOEGRID_DET-INDET, GOEGRID_LOCALGROUPDISK, GOEGRID_SCRATCHDISK, GRIF-IRFU_DATADISK, GRIF-IRFU_LOCALGROUPDISK, GRIF-IRFU_SCRATCHDISK, GRIF-LAL_DATADISK, GRIF-LAL_LOCALGROUPDISK, GRIF-LAL_SCRATCHDISK, GRIF-LPNHE_DATADISK, GRIF-LPNHE_LOCALGROUPDISK, GRIF-LPNHE_SCRATCHDISK, HK-LCG2_DATADISK, HK-LCG2_SCRATCHDISK, IEPSAS-KOSICE_DATADISK, IEPSAS-KOSICE_LOCALGROUPDISK, IEPSAS-KOSICE_SCRATCHDISK, IFAE_DATADISK, IFAE_LOCALGROUPDISK, IFAE_SCRATCHDISK, IFIC-LCG2_CALIBDISK, IFIC-LCG2_DATADISK, IFIC-LCG2_LOCALGROUPDISK, IFIC-LCG2_SCRATCHDISK, IL-TAU-HEP_DATADISK, IN2P3-CC_DATADISK, IN2P3-CC_DATALAKE_DATADISK, IN2P3-CC_LOCALGROUPDISK, IN2P3-CC_LOCALGROUPDISK-MW, IN2P3-CC_PERF-FLAVTAG, IN2P3-CC_PHYS-HIGGS, IN2P3-CC_PHYS-SM, IN2P3-CC_SCRATCHDISK, IN2P3-CPPM_DATADISK, IN2P3-CPPM_LOCALGROUPDISK, IN2P3-CPPM_SCRATCHDISK, IN2P3-LAPP_DATADISK, IN2P3-LAPP_LOCALGROUPDISK, IN2P3-LAPP_SCRATCHDISK, IN2P3-LPC_DATADISK, IN2P3-LPC_LOCALGROUPDISK, IN2P3-LPC_SCRATCHDISK, IN2P3-LPSC_DATADISK, IN2P3-LPSC_LOCALGROUPDISK, IN2P3-LPSC_SCRATCHDISK, INFN-BOLOGNA-T3_LOCALGROUPDISK, INFN-COSENZA_DATADISK, INFN-COSENZA_LOCALGROUPDISK, INFN-COSENZA_SCRATCHDISK, INFN-FRASCATI_DATADISK, INFN-FRASCATI_LOCALGROUPDISK, INFN-FRASCATI_SCRATCHDISK, INFN-GENOVA_DATADISK, INFN-GENOVA_LOCALGROUPDISK, INFN-GENOVA_SCRATCHDISK, INFN-LECCE_LOCALGROUPDISK, INFN-LECCE_SCRATCHDISK, INFN-MILANO-ATLASC_DATADISK, INFN-MILANO-ATLASC_LOCALGROUPDISK, INFN-MILANO-ATLASC_SCRATCHDISK, INFN-NAPOLI-ATLAS_CALIBDISK, INFN-NAPOLI-ATLAS_DATADISK, INFN-NAPOLI-ATLAS_LOCALGROUPDISK, INFN-NAPOLI-ATLAS_SCRATCHDISK, INFN-NAPOLI-ATLAS_TRIG-DAQ, INFN-ROMA1_CALIBDISK, INFN-ROMA1_DATADISK, INFN-ROMA1_LOCALGROUPDISK, INFN-ROMA1_SCRATCHDISK, INFN-ROMA2_LOCALGROUPDISK, INFN-ROMA3_DATADISK, INFN-ROMA3_LOCALGROUPDISK, INFN-ROMA3_SCRATCHDISK, INFN-T1_DATADISK, INFN-T1_SCRATCHDISK, INFN-TRIESTE_LOCALGROUPDISK, JINR-LCG2_DATADISK, JINR-LCG2_LOCALGROUPDISK, JINR-LCG2_SCRATCHDISK, LRZ-LMU_CALIBDISK, LRZ-LMU_DATADISK, LRZ-LMU_LOCALGROUPDISK, LRZ-LMU_SCRATCHDISK, MAINZ_LOCALGROUPDISK, MPPMU_CALIBDISK, MPPMU_DATADISK, MPPMU_LOCALGROUPDISK, MPPMU_PERF-JETS, MPPMU_PERF-MUONS, MPPMU_SCRATCHDISK, MWT2_DATADISK, MWT2_UC_LOCALGROUPDISK, MWT2_UC_SCRATCHDISK, NCG-INGRID-PT_DATADISK, NCG-INGRID-PT_LOCALGROUPDISK, NCG-INGRID-PT_SCRATCHDISK, NDGF-T1_DATADISK, NDGF-T1_DATADISK-DATAFJORD, NDGF-T1_PHYS-HIGGS, NDGF-T1_PHYS-SUSY, NDGF-T1_SCRATCHDISK, NDGF-T1_TRIG-DAQ, NDGF-T1_TRIG-L1CALO, NIKHEF_DATADISK, NIKHEF_LOCALGROUPDISK, NIKHEF_SCRATCHDISK, NO-NORGRID-T2_LOCALGROUPDISK, OLCF_SCRATCHDISK, PIC_DATADISK, PIC_SCRATCHDISK, PRAGUELCG2_DATADISK, PRAGUELCG2_LOCALGROUPDISK, PRAGUELCG2_PHYS-HI, PRAGUELCG2_SCRATCHDISK, PSNC_DATADISK, PSNC_SCRATCHDISK, RO-02-NIPNE_LOCALGROUPDISK, RO-07-NIPNE_DATADISK, RO-07-NIPNE_LOCALGROUPDISK, RO-07-NIPNE_SCRATCHDISK, RRC-KI-T1_DATADISK, RRC-KI-T1_LOCALGROUPDISK, RRC-KI-T1_SCRATCHDISK, RU-PROTVINO-IHEP_DATADISK, RU-PROTVINO-IHEP_LOCALGROUPDISK, RU-PROTVINO-IHEP_SCRATCHDISK, SARA-MATRIX_DATADISK, SARA-MATRIX_SCRATCHDISK, SE-SNIC-T2_LUND_LOCALGROUPDISK, SE-SNIC-T2_STHLM_LOCALGROUPDISK, SE-SNIC-T2_UPPSALA_LOCALGROUPDISK, SIGNET_LOCALGROUPDISK, TAIWAN-LCG2_CALIBDISK, TAIWAN-LCG2_DATADISK, TAIWAN-LCG2_LOCALGROUPDISK, TAIWAN-LCG2_SCRATCHDISK, TECHNION-HEP_DATADISK, TECHNION-HEP_LOCALGROUPDISK, TECHNION-HEP_SCRATCHDISK, TOKYO-LCG2_DATADISK, TOKYO-LCG2_DET-MUON, TOKYO-LCG2_LOCALGROUPDISK, TOKYO-LCG2_PERF-MUONS, TOKYO-LCG2_PHYS-EXOTICS, TOKYO-LCG2_SCRATCHDISK, TR-10-ULAKBIM_DATADISK, TR-10-ULAKBIM_LOCALGROUPDISK, TR-10-ULAKBIM_SCRATCHDISK, TRIUMF-LCG2_DATADISK, TRIUMF-LCG2_LOCALGROUPDISK, TRIUMF-LCG2_PERF-JETS, TRIUMF-LCG2_PERF-TAU, TRIUMF-LCG2_SCRATCHDISK, TUDRESDEN-ZIH_LOCALGROUPDISK, UAM-LCG2_DATADISK, UAM-LCG2_LOCALGROUPDISK, UAM-LCG2_SCRATCHDISK, UKI-LT2-BRUNEL_DATADISK, UKI-LT2-QMUL_DATADISK, UKI-LT2-QMUL_LOCALGROUPDISK, UKI-LT2-QMUL_SCRATCHDISK, UKI-LT2-RHUL_DATADISK, UKI-LT2-RHUL_LOCALGROUPDISK, UKI-LT2-RHUL_SCRATCHDISK, UKI-NORTHGRID-LANCS-HEP_DATADISK, UKI-NORTHGRID-LANCS-HEP_LOCALGROUPDISK, UKI-NORTHGRID-LANCS-HEP_PHYS-BEAUTY, UKI-NORTHGRID-LANCS-HEP_SCRATCHDISK, UKI-NORTHGRID-LIV-HEP_DATADISK, UKI-NORTHGRID-LIV-HEP_LOCALGROUPDISK, UKI-NORTHGRID-LIV-HEP_SCRATCHDISK, UKI-NORTHGRID-MAN-HEP_DATADISK, UKI-NORTHGRID-MAN-HEP_LOCALGROUPDISK, UKI-NORTHGRID-MAN-HEP_SCRATCHDISK, UKI-SCOTGRID-DURHAM_DATADISK, UKI-SCOTGRID-ECDF_DATADISK, UKI-SCOTGRID-ECDF_LOCALGROUPDISK, UKI-SCOTGRID-ECDF_SCRATCHDISK, UKI-SCOTGRID-GLASGOW_DATADISK, UKI-SCOTGRID-GLASGOW_LOCALGROUPDISK, UKI-SCOTGRID-GLASGOW_SCRATCHDISK, UKI-SOUTHGRID-OX-HEP_DATADISK, UKI-SOUTHGRID-OX-HEP_IPV6TEST, UKI-SOUTHGRID-OX-HEP_LOCALGROUPDISK, UKI-SOUTHGRID-OX-HEP_SCRATCHDISK, UKI-SOUTHGRID-RALPP_DATADISK, UKI-SOUTHGRID-RALPP_LOCALGROUPDISK, UKI-SOUTHGRID-RALPP_SCRATCHDISK, UNI-BONN_LOCALGROUPDISK, UNI-FREIBURG_DATADISK, UNI-FREIBURG_LOCALGROUPDISK, UNI-FREIBURG_SCRATCHDISK, UNI-SIEGEN-HEP_LOCALGROUPDISK, UNIBE-LHEP_DATADISK, UNIBE-LHEP_LOCALGROUPDISK, UNIBE-LHEP_SCRATCHDISK, UNICPH-NBI_LOCALGROUPDISK, USTC-T3_LOCALGROUPDISK, WEIZMANN-LCG2_LOCALGROUPDISK, WUPPERTALPROD_DATADISK, WUPPERTALPROD_LOCALGROUPDISK, WUPPERTALPROD_SCRATCHDISK, ZA-WITS-CORE_DATADISK, ZA-WITS-CORE_LOCALGROUPDISK, ru-PNPI_XCACHE_NODES
…
```


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/reaper/reaper2.py b/lib/rucio/daemons/reaper/reaper2.py
--- a/lib/rucio/daemons/reaper/reaper2.py
+++ b/lib/rucio/daemons/reaper/reaper2.py
@@ -109,7 +109,8 @@ def get_rses_to_process(rses, include_rses, exclude_rses, vos):
             vos = [v['vo'] for v in list_vos()]
         logging.log(logging.INFO, 'Reaper: This instance will work on VO%s: %s' % ('s' if len(vos) > 1 else '', ', '.join([v for v in vos])))
 
-    cache_key = 'rses_to_process'
+    pid = os.getpid()
+    cache_key = 'rses_to_process_%s' % pid
     if multi_vo:
         cache_key += '@%s' % '-'.join(vo for vo in vos)
 


Test patch: