Repository: rucio/rucio

Issue:
Protocol mismatch for multisource transfers
Motivation
----------
There seems to be a bug with protocol mismatch which somehow skips the existing Rucio protocol mapping checks and submits mismatched protocols. 


Modification
------------
Possibly an issue in the core.transfer loop with caching.



Discussion:
Hi,

here are some example from the "davs" sites

davs -> srm

FZK -> AGLT2 <- both davs sites
https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/118c516e-3065-51f3-9885-fcc7c3ddc542 

CNAF -> AGLT2
https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/a8f10a40-be19-5a91-92a3-df76f3e11b25

SFU -> DESY-HH
https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/2c4b19f1-2f6b-58f5-8c7c-90d1b4c19065

NIKHEF -> PIC
https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/16b5a98c-f328-574a-ba9d-f823d796d11a

davs -> gsiftp

PIC -> UKI-NORTHGRID-MAN-HEP  <-- both davs sites
https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/67f4a280-fef8-5d59-9b16-9b25e7461d62

RRC-KI-T1 -> PRAGUE
https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/job/216a84c3-c2a1-5674-86f2-a85a436ec893

the last one has a mixture of mixed protocols and sites
Here some more SARA and IN2P3-CC consistently copying from webdav to srm. If I select only the davs sites this happens only between SARA and IN2P3-CC in both directions.

https://monit-grafana.cern.ch/d/FtSFfwdmk/ddm-transfers?orgId=17&from=now-12h&to=now&var-binning=$__auto_interval_binning&var-groupby=protocol&var-activity=Analysis%20Input&var-activity=Data%20Consolidation&var-activity=Data%20rebalancing&var-activity=Express&var-activity=Functional%20Test&var-activity=Production%20Input&var-activity=Production%20Output&var-activity=Recovery&var-activity=T0%20Export&var-activity=T0%20Tape&var-activity=User%20Subscriptions&var-activity=default&var-protocol=All&var-src_tier=0&var-src_tier=1&var-src_tier=2&var-src_country=All&var-src_cloud=All&var-src_site=AGLT2&var-src_site=DESY-HH&var-src_site=FZK-LCG2&var-src_site=IN2P3-CC&var-src_site=UKI-NORTHGRID-MAN-HEP&var-src_site=ifae&var-src_site=pic&var-src_site=praguelcg2&var-src_endpoint=All&var-src_token=All&var-columns=src_experiment_site&var-dst_tier=0&var-dst_tier=1&var-dst_tier=2&var-dst_tier=3&var-dst_country=All&var-dst_cloud=All&var-dst_site=SARA-MATRIX&var-dst_endpoint=All&var-dst_token=All&var-rows=dst_experiment_site&var-measurement=ddm_transfer&var-retention_policy=raw&var-include=&var-exclude=TEST%7CCEPH%7CPPS%7CGRIDFTP%7CLAKE%7CAWS&var-exclude_es=All&var-include_es_dst=All&var-include_es_src=All 

https://monit-grafana.cern.ch/d/FtSFfwdmk/ddm-transfers?viewPanel=56&orgId=17&from=now-12h&to=now&var-binning=$__auto_interval_binning&var-groupby=protocol&var-activity=Analysis%20Input&var-activity=Data%20Consolidation&var-activity=Data%20rebalancing&var-activity=Express&var-activity=Functional%20Test&var-activity=Production%20Input&var-activity=Production%20Output&var-activity=Recovery&var-activity=T0%20Export&var-activity=T0%20Tape&var-activity=User%20Subscriptions&var-activity=default&var-protocol=All&var-src_tier=0&var-src_tier=1&var-src_tier=2&var-src_country=All&var-src_cloud=All&var-src_site=AGLT2&var-src_site=DESY-HH&var-src_site=FZK-LCG2&var-src_site=IN2P3-CC&var-src_site=UKI-NORTHGRID-MAN-HEP&var-src_site=ifae&var-src_site=pic&var-src_site=praguelcg2&var-src_site=SARA-MATRIX&var-src_endpoint=All&var-src_token=All&var-columns=src_experiment_site&var-dst_tier=0&var-dst_tier=1&var-dst_tier=2&var-dst_tier=3&var-dst_country=All&var-dst_cloud=All&var-dst_site=IN2P3-CC&var-dst_endpoint=All&var-dst_token=All&var-rows=dst_experiment_site&var-measurement=ddm_transfer&var-retention_policy=raw&var-include=&var-exclude=TEST%7CCEPH%7CPPS%7CGRIDFTP%7CLAKE%7CAWS&var-exclude_es=All&var-include_es_dst=All&var-include_es_src=All
 
Any ideas on this?

Hi @ericvaandering 
Sorry the late reply. I highly suspect some kind of caching issue here. We frequently cache protocols and RSE attributes in these loops processing the transfers (core/transfer.py) - My suspicion is that some dictionary/value get's wrongly reused which causes this issue.
Do you see that issue with CMS as well?
Hi, yes we saw this on CMS as well
I am taking the liberty of marking this issue as a high-priority item. We have been notified that this is putting additional strain on FTS operations. Another example of such transfers is [UKI-NORTHGRID-LANCS-HEP→PIC](https://fts3-atlas.cern.ch:8449/fts3/ftsmon/#/jobs?source_se=davs:%2F%2Ffal-pygrid-30.lancs.ac.uk&dest_se=srm:%2F%2Fsrmatlas.pic.es&vo=atlas&with_file=FINISHED&time_window=1).
Just to keep everyone posted, I very likely identified the bug, it's a regression we introduced with the multihop development ~1 year ago. Basically in certain (rare?) cases where for a transfer multiple sources are available a source protocol might be selected which is not compatible with the (previously selected) destination protocol, thus causing the streaming. This doesn't happen every time but depends on the available protocols (and priority) of the involved source RSEs and destination RSE and also potentially on the order these are returned by the database.

We should have a fix for this in the following days, I will build a hotfix release for this then.
Re-Opened, as not fully fixed with `1.23.9.post2` - New hotfix release will come today.

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -660,7 +660,6 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
     multi_hop_dict = {}
 
     rse_mapping = {}
-    current_schemes = SUPPORTED_PROTOCOLS
 
     multihop_rses = []
     try:
@@ -917,7 +916,7 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     file_metadata['previous_attempt_id'] = previous_attempt_id
 
                 transfers[req_id] = {'request_id': req_id,
-                                     'schemes': __add_compatible_schemes(schemes=[destination_protocol], allowed_schemes=current_schemes),
+                                     'schemes': __add_compatible_schemes(schemes=[destination_protocol], allowed_schemes=SUPPORTED_PROTOCOLS),
                                      'account': account,
                                      # 'src_urls': [source_url],
                                      'sources': [(rse, source_url, source_rse_id, ranking if ranking is not None else 0, link_ranking)],


Test patch: