Repository: rucio/rucio
Issue: Issue with stat/exists and lan/wan protocols

Motivation
----------
With 1.26.8 testing clients the following issue was observed on storm RSEs (e.g. IFIC, INFN-GENOVA)

```
https://bigpanda.cern.ch/job?pandaid=5234466771

021-10-29 02:18:07,303 | DEBUG    | pilot.copytool.rucio             | _upload_item              | The PFN created from the LFN: davs://srmdav.ific.uv.es:8443/lustre/ific.uv.es/grid/atlas/atlasdatadisk/rucio/hc_test/f8/5e/Hits.hc_20226991.IFIC.ca9c591e-cba0-49cb-b0d5-610de1a9cc27_11367.HITS.pool.root
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | *** rucio API upload client failed ***
2021-10-29 02:18:07,303 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | caught exception:
2021-10-29 02:18:07,305 | ERROR    | pilot.copytool.rucio             | _stage_out_api            | Traceback (most recent call last):
   File "/var/lib/condor/execute/dir_35578/atlas_J7y2s02G/pilot2/pilot/copytool/rucio.py", line 610, in _stage_out_api
     result = upload_client.upload([f], summary_file_path=summary_file_path, traces_copy_out=trace_report_out)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 283, in upload
     sign_service=sign_service)
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 607, in _upload_item
     if protocol_write.overwrite is False and delete_existing is False and protocol_read.exists(readpfn):
   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.26.8/lib/python3.6/site-packages/rucio/rse/protocols/storm.py", line 89, in exists
     raise NotImplementedError
NotImplementedError

ge Genova:
https://bigpanda.cern.ch/job?pandaid=5234466887
```

The underlying issue is that we do 3 different exists/stat checks within a Rucio upload, and the behaviour changed slightly with `1.26`:

1. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L245 here we check if the file already exists. This is done with `WAN` read protocol. 
2. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L625 here we check if the `%.rucio.upload` file exists. This is done with `LAN` read protocol.
3. https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/client/uploadclient.py#L660 here we do a stat on the file. This is done with `LAN` write protocol.

The above issue with storm fails in step 2, since there is no exists() check for storm. In general this is all rather inconsistent to do the checks with different protocols. The third one we HAVE to make with the same protocol which was used to write the file, due to race conditions between the storage doors. I would suggest to make (1) and (2) also with the LAN write protocol in that case.

Changed files:
lib/rucio/rse/rsemanager.py

diff --git a/lib/rucio/rse/rsemanager.py b/lib/rucio/rse/rsemanager.py
index 96b969a47..c62940d36 100644
--- a/lib/rucio/rse/rsemanager.py
+++ b/lib/rucio/rse/rsemanager.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2012-2021 CERN
+# Copyright 2012-2022 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -37,9 +37,9 @@
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2021
+# - Gabriele Fronzé <sucre.91@hotmail.it>, 2021
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
-#
-# PY3K COMPATIBLE
+# - Joel Dierkes <joel.dierkes@cern.ch>, 2022
 
 from __future__ import print_function
 
@@ -291,7 +291,7 @@ def exists(rse_settings, files, domain='wan', scheme=None, impl=None, auth_token
     try:
         protocol.exists(None)
     except NotImplementedError:
-        protocol = create_protocol(rse_settings, 'write', scheme=scheme, impl=impl, domain=domain, auth_token=auth_token, logger=logger)
+        protocol = create_protocol(rse_settings, 'write', scheme=scheme, domain=domain, auth_token=auth_token, logger=logger)
         protocol.connect()
     except:
         pass