Repository: rucio/rucio

Issue:
Issue with stat/exists and lan/wan protocols
Motivation
----------
With 1.26.8 testing clients the following issue was observed on storm RSEs (e.g. IFIC, INFN-GENOVA)

```
https://bigpanda.cern.ch/job?pandaid=5234466771

021-10-29 02:18:07,303 | DEBUG    | pilot.copytool.rucio             | _upload_item              | The PFN created from the LFN: davs://srmdav.ific.uv.es:8443/lustre/ific.uv.es/grid/atlas/atlasdatadisk/rucio/hc_test/f8/5e/Hits.hc_20226991.IFIC.ca9c591e-cba0-49cb-b0d5-610de1a9cc27_11367.HITS.pool.root
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | *** rucio API upload client failed ***
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | caught exception:
2021-10-29 02:18:07,305 | ERROR    | pilot.copytool.rucio             | _stage_out_api            | Traceback (most recent call last):
   File "/var/lib/condor/execute/dir_35578/atlas_J7y2s02G/pilot2/pilot/copytool/rucio.py", line 610, in _stage_out_api
     result = upload_client.upload([f], summary_file_path=summary_file_path, traces_copy_out=trace_report_out)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 283, in upload
     sign_service=sign_service)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 607, in _upload_item
     if protocol_write.overwrite is False and delete_existing is False and protocol_read.exists(readpfn):
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/rse/protocols/storm.py", line 89, in exists
     raise NotImplementedError
NotImplementedError

ge Genova:
https://bigpanda.cern.ch/job?pandaid=5234466887
```

The underlying issue is that we do 3 different exists/stat checks within a Rucio upload, and the behaviour changed slightly with `1.26`:

1. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L245 here we check if the file already exists. This is done with `WAN` read protocol. 
2. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L625 here we check if the `%.rucio.upload` file exists. This is done with `LAN` read protocol.
3. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L660 here we do a stat on the file. This is done with `LAN` write protocol.

The above issue with storm fails in step 2, since there is no exists() check for storm. In general this is all rather inconsistent to do the checks with different protocols. The third one we HAVE to make with the same protocol which was used to write the file, due to race conditions between the storage doors. I would suggest to make (1) and (2) also with the LAN write protocol in that case.


Discussion:
Just noting this here, probably this is just an issue with the set priorities on these sites and 1.26 clients now correctly honor them. But it's strange why the original PFN is a `davs` one.
@mlassnig @rcarpa I updated the issue text, can you please have a look on the suggestion. To check with 3 different protocols seems unreasonable, I think it would be better to just always do it with the LAN/WAN write protocol, but we need to think if this has any other implications. 
https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/rse/rsemanager.py#L289-L297
We don't have to change (1) for my POV, since it will select a `write` protocol if the `read` protocol does not implement the `exists` function. The `domain` itself is the same as the others...
So this issue is still not fixed when we tested `1.27.2`

```

022-01-13 15:41:12,180 | DEBUG    | pilot.copytool.rucio             | upload                    | lan domain is used for the upload
2022-01-13 15:41:12,180 | INFO     | pilot.copytool.rucio             | preferred_impl            | No preferred protocol impl in rucio.cfg: No section: 'upload'
2022-01-13 15:41:12,180 | DEBUG    | pilot.copytool.rucio             | preferred_impl            | Failed to create protocol "rucio.rse.protocols.gfal.Default", exception: RSE does not support requested protocol.
Details: No protocol for provided settings found : {'availability_delete': True, 'availability_read': True, 'availability_write': True, 'credentials': None, 'deterministic': True, 'domain': ['lan', 'wan'], 'id': '0919372344bb45d28ad964ecfe091e1b', 'lfn2pfn_algorithm': 'hash', 'protocols': [{'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'davs', 'port': 8443, 'prefix': '/webdav/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy': 1}}, 'extended_attributes': None}, {'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'storm', 'port': 8443, 'prefix': '/mnt/lustre_3/storm_3/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.storm.Default', 'domains': {'lan': {'read': 1, 'write': 0, 'delete': 0}, 'wan': {'read': 3, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}, {'hostname': 'xrootd01.esc.qmul.ac.uk', 'scheme': 'root', 'port': 1094, 'prefix': '//atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 0, 'delete': 0}, 'wan': {'read': 2, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}], 'qos_class': None, 'rse': 'UKI-LT2-QMUL_SCRATCHDISK', 'rse_type': 'DISK', 'sign_url': None, 'staging_area': False, 'verify_checksum': True, 'volatile': False, 'read_protocol': 1, 'write_protocol': 1, 'delete_protocol': 1, 'third_party_copy_protocol': 1}.
2022-01-13 15:41:12,180 | INFO     | pilot.copytool.rucio             | preferred_impl            | Preferred protocol impl supported locally and remotely: rucio.rse.protocols.storm.Default
2022-01-13 15:41:12,181 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | *** rucio API upload client failed ***
2022-01-13 15:41:12,181 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | caught exception: RSE does not support requested protocol.
Details: No protocol for provided settings found : {'availability_delete': True, 'availability_read': True, 'availability_write': True, 'credentials': None, 'deterministic': True, 'domain': ['lan', 'wan'], 'id': '0919372344bb45d28ad964ecfe091e1b', 'lfn2pfn_algorithm': 'hash', 'protocols': [{'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'davs', 'port': 8443, 'prefix': '/webdav/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy': 1}}, 'extended_attributes': None}, {'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'storm', 'port': 8443, 'prefix': '/mnt/lustre_3/storm_3/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.storm.Default', 'domains': {'lan': {'read': 1, 'write': 0, 'delete': 0}, 'wan': {'read': 3, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}, {'hostname': 'xrootd01.esc.qmul.ac.uk', 'scheme': 'root', 'port': 1094, 'prefix': '//atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 0, 'delete': 0}, 'wan': {'read': 2, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}], 'qos_class': None, 'rse': 'UKI-LT2-QMUL_SCRATCHDISK', 'rse_type': 'DISK', 'sign_url': None, 'staging_area': False, 'verify_checksum': True, 'volatile': False, 'read_protocol': 1, 'write_protocol': 1, 'delete_protocol': 1, 'third_party_copy_protocol': 1}.
2022-01-13 15:41:12,184 | ERROR    | pilot.copytool.rucio             | _stage_out_api            | Traceback (most recent call last):
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 292, in exists
    protocol.exists(None)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/lib/python3.6/site-packages/rucio/rse/protocols/storm.py", line 89, in exists
    raise NotImplementedError
NotImplementedError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch/lcg/prdatl27/3977702/atlas_ymJDXcWv/pilot2/pilot/copytool/rucio.py", line 610, in _stage_out_api
    result = upload_client.upload([f], summary_file_path=summary_file_path, traces_copy_out=trace_report_out)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 248, in upload
    if rsemgr.exists(rse_settings, pfn if pfn else file_did, domain=domain, scheme=force_scheme, impl=impl, auth_token=self.auth_token, logger=logger):
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 294, in exists
    protocol = create_protocol(rse_settings, 'write', scheme=scheme, impl=impl, domain=domain, auth_token=auth_token, logger=logger)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 201, in create_protocol
    candidate = _get_possible_protocols(rse_settings, operation, scheme, domain, impl=impl)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 148, in _get_possible_protocols
    ' found : %s.' % str(rse_settings))
rucio.common.exception.RSEProtocolNotSupported: RSE does not support requested protocol.
Details: No protocol for provided settings found : {'availability_delete': True, 'availability_read': True, 'availability_write': True, 'credentials': None, 'deterministic': True, 'domain': ['lan', 'wan'], 'id': '0919372344bb45d28ad964ecfe091e1b', 'lfn2pfn_algorithm': 'hash', 'protocols': [{'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'davs', 'port': 8443, 'prefix': '/webdav/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy': 1}}, 'extended_attributes': None}, {'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'storm', 'port': 8443, 'prefix': '/mnt/lustre_3/storm_3/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.storm.Default', 'domains': {'lan': {'read': 1, 'write': 0, 'delete': 0}, 'wan': {'read': 3, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}, {'hostname': 'xrootd01.esc.qmul.ac.uk', 'scheme': 'root', 'port': 1094, 'prefix': '//atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 0, 'delete': 0}, 'wan': {'read': 2, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}], 'qos_class': None, 'rse': 'UKI-LT2-QMUL_SCRATCHDISK', 'rse_type': 'DISK', 'sign_url': None, 'staging_area': False, 'verify_checksum': True, 'volatile': False, 'read_protocol': 1, 'write_protocol': 1, 'delete_protocol': 1, 'third_party_copy_protocol': 1}.
```

In my opinion, the issue here is part (1) we discussed above. I think this still needs a change. In my opinion the workflow is as follows:

1. Implementation `impl` is fixed here
https://github.com/rucio/rucio/blob/d19db9eeb09440ae97477840fff57112f9b25620/lib/rucio/client/uploadclient.py#L222
2. It goes into `rsemgr.exists(...)` with `impl` used here
https://github.com/rucio/rucio/blob/d19db9eeb09440ae97477840fff57112f9b25620/lib/rucio/client/uploadclient.py#L248
3. It fails the `.exists()` check here with the `read` protocol.
https://github.com/rucio/rucio/blob/d19db9eeb09440ae97477840fff57112f9b25620/lib/rucio/rse/rsemanager.py#L292
4. And trys the `write` protocol, but still with the `impl` being picked.
https://github.com/rucio/rucio/blob/d19db9eeb09440ae97477840fff57112f9b25620/lib/rucio/rse/rsemanager.py#L294

But here is where it fails in my opinion. Since the `impl` is already chosen, and storm in aboves settings does not provide a `write` protocol, it will just fail. In my opinion this fallback with the `write` protocol needs to be done without `impl` being chosen.

Resolving patch:
diff --git a/lib/rucio/rse/rsemanager.py b/lib/rucio/rse/rsemanager.py
--- a/lib/rucio/rse/rsemanager.py
+++ b/lib/rucio/rse/rsemanager.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2012-2021 CERN
+# Copyright 2012-2022 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -37,9 +37,9 @@
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2021
+# - Gabriele Fronzé <sucre.91@hotmail.it>, 2021
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
-#
-# PY3K COMPATIBLE
+# - Joel Dierkes <joel.dierkes@cern.ch>, 2022
 
 from __future__ import print_function
 
@@ -291,7 +291,7 @@ def exists(rse_settings, files, domain='wan', scheme=None, impl=None, auth_token
     try:
         protocol.exists(None)
     except NotImplementedError:
-        protocol = create_protocol(rse_settings, 'write', scheme=scheme, impl=impl, domain=domain, auth_token=auth_token, logger=logger)
+        protocol = create_protocol(rse_settings, 'write', scheme=scheme, domain=domain, auth_token=auth_token, logger=logger)
         protocol.connect()
     except:
         pass


Test patch: