Repository: rucio/rucio

Issue:
Issue with stat/exists and lan/wan protocols
Motivation
----------
With 1.26.8 testing clients the following issue was observed on storm RSEs (e.g. IFIC, INFN-GENOVA)

```
https://bigpanda.cern.ch/job?pandaid=5234466771

021-10-29 02:18:07,303 | DEBUG    | pilot.copytool.rucio             | _upload_item              | The PFN created from the LFN: davs://srmdav.ific.uv.es:8443/lustre/ific.uv.es/grid/atlas/atlasdatadisk/rucio/hc_test/f8/5e/Hits.hc_20226991.IFIC.ca9c591e-cba0-49cb-b0d5-610de1a9cc27_11367.HITS.pool.root
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | *** rucio API upload client failed ***
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | caught exception:
2021-10-29 02:18:07,305 | ERROR    | pilot.copytool.rucio             | _stage_out_api            | Traceback (most recent call last):
   File "/var/lib/condor/execute/dir_35578/atlas_J7y2s02G/pilot2/pilot/copytool/rucio.py", line 610, in _stage_out_api
     result = upload_client.upload([f], summary_file_path=summary_file_path, traces_copy_out=trace_report_out)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 283, in upload
     sign_service=sign_service)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 607, in _upload_item
     if protocol_write.overwrite is False and delete_existing is False and protocol_read.exists(readpfn):
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/rse/protocols/storm.py", line 89, in exists
     raise NotImplementedError
NotImplementedError

ge Genova:
https://bigpanda.cern.ch/job?pandaid=5234466887
```

The underlying issue is that we do 3 different exists/stat checks within a Rucio upload, and the behaviour changed slightly with `1.26`:

1. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L245 here we check if the file already exists. This is done with `WAN` read protocol. 
2. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L625 here we check if the `%.rucio.upload` file exists. This is done with `LAN` read protocol.
3. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L660 here we do a stat on the file. This is done with `LAN` write protocol.

The above issue with storm fails in step 2, since there is no exists() check for storm. In general this is all rather inconsistent to do the checks with different protocols. The third one we HAVE to make with the same protocol which was used to write the file, due to race conditions between the storage doors. I would suggest to make (1) and (2) also with the LAN write protocol in that case.


Discussion:
Just noting this here, probably this is just an issue with the set priorities on these sites and 1.26 clients now correctly honor them. But it's strange why the original PFN is a `davs` one.
@mlassnig @rcarpa I updated the issue text, can you please have a look on the suggestion. To check with 3 different protocols seems unreasonable, I think it would be better to just always do it with the LAN/WAN write protocol, but we need to think if this has any other implications. 

Resolving patch:
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -40,6 +40,7 @@
 # - Radu Carpa <radu.carpa@cern.ch>, 2021
 # - David Población Criado <david.poblacion.criado@cern.ch>, 2021
 # - Joel Dierkes <joel.dierkes@cern.ch>, 2021
+# - Nicholas Smith <nick.smith@cern.ch>, 2021
 
 import copy
 import json
@@ -629,7 +630,7 @@ def _upload_item(self, rse_settings, rse_attributes, lfn, source_dir=None, domai
             raise FileReplicaAlreadyExists('File %s in scope %s already exists on storage as PFN %s' % (name, scope, pfn))  # wrong exception ?
 
         # Removing tmp from earlier attempts
-        if protocol_read.exists(readpfn_tmp):
+        if protocol_write.exists(readpfn_tmp):
             logger(logging.DEBUG, 'Removing remains of previous upload attemtps.')
             try:
                 # Construct protocol for delete operation.


Test patch: