Repository: rucio/rucio
Issue: Issue with stat/exists and lan/wan protocols

Motivation
----------
With 1.26.8 testing clients the following issue was observed on storm RSEs (e.g. IFIC, INFN-GENOVA)

```
https://bigpanda.cern.ch/job?pandaid=5234466771

021-10-29 02:18:07,303 | DEBUG    | pilot.copytool.rucio             | _upload_item              | The PFN created from the LFN: davs://srmdav.ific.uv.es:8443/lustre/ific.uv.es/grid/atlas/atlasdatadisk/rucio/hc_test/f8/5e/Hits.hc_20226991.IFIC.ca9c591e-cba0-49cb-b0d5-610de1a9cc27_11367.HITS.pool.root
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | *** rucio API upload client failed ***
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | caught exception:
2021-10-29 02:18:07,305 | ERROR    | pilot.copytool.rucio             | _stage_out_api            | Traceback (most recent call last):
   File "/var/lib/condor/execute/dir_35578/atlas_J7y2s02G/pilot2/pilot/copytool/rucio.py", line 610, in _stage_out_api
     result = upload_client.upload([f], summary_file_path=summary_file_path, traces_copy_out=trace_report_out)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 283, in upload
     sign_service=sign_service)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 607, in _upload_item
     if protocol_write.overwrite is False and delete_existing is False and protocol_read.exists(readpfn):
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/rse/protocols/storm.py", line 89, in exists
     raise NotImplementedError
NotImplementedError

ge Genova:
https://bigpanda.cern.ch/job?pandaid=5234466887
```

The underlying issue is that we do 3 different exists/stat checks within a Rucio upload, and the behaviour changed slightly with `1.26`:

1. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L245 here we check if the file already exists. This is done with `WAN` read protocol. 
2. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L625 here we check if the `%.rucio.upload` file exists. This is done with `LAN` read protocol.
3. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L660 here we do a stat on the file. This is done with `LAN` write protocol.

The above issue with storm fails in step 2, since there is no exists() check for storm. In general this is all rather inconsistent to do the checks with different protocols. The third one we HAVE to make with the same protocol which was used to write the file, due to race conditions between the storage doors. I would suggest to make (1) and (2) also with the LAN write protocol in that case.

Changed files:
lib/rucio/client/uploadclient.py

diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
index d29bfb581..c80121f48 100644
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -40,6 +40,7 @@
 # - Radu Carpa <radu.carpa@cern.ch>, 2021
 # - David Población Criado <david.poblacion.criado@cern.ch>, 2021
 # - Joel Dierkes <joel.dierkes@cern.ch>, 2021
+# - Nicholas Smith <nick.smith@cern.ch>, 2021
 
 import copy
 import json
@@ -629,7 +630,7 @@ class UploadClient:
             raise FileReplicaAlreadyExists('File %s in scope %s already exists on storage as PFN %s' % (name, scope, pfn))  # wrong exception ?
 
         # Removing tmp from earlier attempts
-        if protocol_read.exists(readpfn_tmp):
+        if protocol_write.exists(readpfn_tmp):
             logger(logging.DEBUG, 'Removing remains of previous upload attemtps.')
             try:
                 # Construct protocol for delete operation.