Repository: rucio/rucio

Issue:
Upload issue with 1.27.4 ALRB testing
Motivation
----------
```
2022-02-10 16:25:13,633 | INFO     | pilot.copytool.rucio             | copy_out                  | rucio copytool, uploading file with scope: user.gangarbt and lfn: b433f058-b09c-4efb-9238-697b17de1abd_89710.1.job.log.tgz
2022-02-10 16:25:13,633 | INFO     | pilot.copytool.rucio             | copy_out                  | the file will be uploaded to UKI-LT2-QMUL_SCRATCHDISK
2022-02-10 16:25:13,633 | INFO     | pilot.copytool.rucio             | copy_out                  | overall transfer timeout=310
2022-02-10 16:25:13,638 | INFO     | pilot.copytool.rucio             | _stage_out_api            | rucio API stage-out dictionary: {'path': '/scratch/lcg/pilatl02/4302323/atlas_jjmBwVC5/PanDA_Pilot-5355486925/b433f058-b09c-4efb-9238-697b17de1abd_89710.1.job.log.tgz', 'rse': 'UKI-LT2-QMUL_SCRATCHDISK', 'did_scope': 'user.gangarbt', 'no_register': True, 'transfer_timeout': 300, 'connection_timeout': 3600}
2022-02-10 16:25:13,638 | INFO     | pilot.copytool.rucio             | _stage_out_api            | *** rucio API uploading file (taking over logging) ***
2022-02-10 16:25:13,638 | DEBUG    | pilot.copytool.rucio             | _stage_out_api            | summary_file_path=/scratch/lcg/pilatl02/4302323/atlas_jjmBwVC5/PanDA_Pilot-5355486925/rucio_upload.json
2022-02-10 16:25:13,638 | DEBUG    | pilot.copytool.rucio             | _stage_out_api            | trace_report_out=[]
2022-02-10 16:25:13,639 | DEBUG    | pilot.copytool.rucio             | upload                    | Num. of files that upload client is processing: 1
2022-02-10 16:25:13,644 | DEBUG    | urllib3.connectionpool           | _new_conn                 | Starting new HTTPS connection (1): voatlasrucio-server-prod.cern.ch:443
2022-02-10 16:25:13,739 | DEBUG    | urllib3.connectionpool           | _make_request             | https://voatlasrucio-server-prod.cern.ch:443 "GET /rses/?expression=UKI-LT2-QMUL_SCRATCHDISK HTTP/1.1" 200 None
2022-02-10 16:25:13,740 | DEBUG    | pilot.copytool.rucio             | upload                    | Input validation done.
2022-02-10 16:25:13,740 | INFO     | pilot.copytool.rucio             | upload                    | Preparing upload for file b433f058-b09c-4efb-9238-697b17de1abd_89710.1.job.log.tgz
2022-02-10 16:25:13,775 | DEBUG    | urllib3.connectionpool           | _make_request             | https://voatlasrucio-server-prod.cern.ch:443 "GET /rses/UKI-LT2-QMUL_SCRATCHDISK/attr/ HTTP/1.1" 200 594
2022-02-10 16:25:13,776 | DEBUG    | pilot.copytool.rucio             | upload                    | lan domain is used for the upload
2022-02-10 16:25:13,776 | INFO     | pilot.copytool.rucio             | preferred_impl            | No preferred protocol impl in rucio.cfg: No section: 'upload'
2022-02-10 16:25:13,776 | DEBUG    | pilot.copytool.rucio             | preferred_impl            | Failed to create protocol "rucio.rse.protocols.gfal.Default", exception: RSE does not support requested protocol.
Details: No protocol for provided settings found : {'availability_delete': True, 'availability_read': True, 'availability_write': True, 'credentials': None, 'deterministic': True, 'domain': ['lan', 'wan'], 'id': '0919372344bb45d28ad964ecfe091e1b', 'lfn2pfn_algorithm': 'hash', 'protocols': [{'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'davs', 'port': 8443, 'prefix': '/webdav/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy': 1}}, 'extended_attributes': None}, {'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'storm', 'port': 8443, 'prefix': '/mnt/lustre_3/storm_3/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.storm.Default', 'domains': {'lan': {'read': 1, 'write': 0, 'delete': 0}, 'wan': {'read': 3, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}, {'hostname': 'xrootd01.esc.qmul.ac.uk', 'scheme': 'root', 'port': 1094, 'prefix': '//atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 0, 'delete': 0}, 'wan': {'read': 2, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}], 'qos_class': None, 'rse': 'UKI-LT2-QMUL_SCRATCHDISK', 'rse_type': 'DISK', 'sign_url': None, 'staging_area': False, 'verify_checksum': True, 'volatile': False, 'read_protocol': 1, 'write_protocol': 1, 'delete_protocol': 1, 'third_party_copy_protocol': 1}.
2022-02-10 16:25:13,776 | INFO     | pilot.copytool.rucio             | preferred_impl            | Preferred protocol impl supported locally and remotely: rucio.rse.protocols.storm.Default
2022-02-10 16:25:13,776 | DEBUG    | pilot.copytool.rucio             | log_format                | gfal.Default: connecting to storage
2022-02-10 16:25:13,779 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success 
2022-02-10 16:25:13,779 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success 
2022-02-10 16:25:13,779 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success 
2022-02-10 16:25:13,779 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success 
2022-02-10 16:25:13,780 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so loaded with success 
2022-02-10 16:25:13,780 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so loaded with success 
2022-02-10 16:25:13,780 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success 
2022-02-10 16:25:13,780 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success 
2022-02-10 16:25:13,780 | INFO     | gfal2                            | connect                   | [gfal_module_load] plugin /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix4a/usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success 
2022-02-10 16:25:13,781 | DEBUG    | pilot.copytool.rucio             | exists                    | Checking if davs://se03.esc.qmul.ac.uk:8443/webdav/atlas/atlasscratchdisk/rucio/user/gangarbt/a1/5e/b433f058-b09c-4efb-9238-697b17de1abd_89710.1.job.log.tgz exists
2022-02-10 16:25:13,781 | DEBUG    | pilot.copytool.rucio             | log_format                | gfal.Default: checking if file exists davs://se03.esc.qmul.ac.uk:8443/webdav/atlas/atlasscratchdisk/rucio/user/gangarbt/a1/5e/b433f058-b09c-4efb-9238-697b17de1abd_89710.1.job.log.tgz
2022-02-10 16:25:13,808 | INFO     | gfal2                            | __gfal2_exist             | Davix: Negative result for operation: HTTP 404 : File not found . After 1 retry
2022-02-10 16:25:13,808 | INFO     | gfal2                            | __gfal2_exist             | Davix: Could not execute operation on davs://se03.esc.qmul.ac.uk:8443/webdav/atlas/atlasscratchdisk/rucio/user/gangarbt/a1/5e/b433f058-b09c-4efb-9238-697b17de1abd_89710.1.job.log.tgz, error Result HTTP 404 : File not found  after 1 attempts
2022-02-10 16:25:13,808 | INFO     | gfal2                            | __gfal2_exist             | Davix: Try to Recover with Metalink...
2022-02-10 16:25:13,826 | INFO     | gfal2                            | __gfal2_exist             | Davix: Impossible to Recover with Metalink: Server does not support Metalink standard
2022-02-10 16:25:13,826 | DEBUG    | pilot.copytool.rucio             | log_format                | gfal.Default: closing protocol connection
2022-02-10 16:25:13,828 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | *** rucio API upload client failed ***
2022-02-10 16:25:13,828 | WARNING  | pilot.copytool.rucio             | _stage_out_api            | caught exception: RSE does not support requested protocol.
Details: No protocol for provided settings found : {'availability_delete': True, 'availability_read': True, 'availability_write': True, 'credentials': None, 'deterministic': True, 'domain': ['lan', 'wan'], 'id': '0919372344bb45d28ad964ecfe091e1b', 'lfn2pfn_algorithm': 'hash', 'protocols': [{'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'davs', 'port': 8443, 'prefix': '/webdav/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy': 1}}, 'extended_attributes': None}, {'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'storm', 'port': 8443, 'prefix': '/mnt/lustre_3/storm_3/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.storm.Default', 'domains': {'lan': {'read': 1, 'write': 0, 'delete': 0}, 'wan': {'read': 3, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}, {'hostname': 'xrootd01.esc.qmul.ac.uk', 'scheme': 'root', 'port': 1094, 'prefix': '//atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 0, 'delete': 0}, 'wan': {'read': 2, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}], 'qos_class': None, 'rse': 'UKI-LT2-QMUL_SCRATCHDISK', 'rse_type': 'DISK', 'sign_url': None, 'staging_area': False, 'verify_checksum': True, 'volatile': False, 'read_protocol': 1, 'write_protocol': 1, 'delete_protocol': 1, 'third_party_copy_protocol': 1}.
2022-02-10 16:25:13,828 | ERROR    | pilot.copytool.rucio             | _stage_out_api            | Traceback (most recent call last):
  File "/scratch/lcg/pilatl02/4302323/atlas_jjmBwVC5/pilot3/pilot/copytool/rucio.py", line 610, in _stage_out_api
    result = upload_client.upload([f], summary_file_path=summary_file_path, traces_copy_out=trace_report_out)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.4/lib/python3.6/site-packages/rucio/client/uploadclient.py", line 254, in upload
    protocols = rsemgr.get_protocols_ordered(rse_settings=rse_settings, operation='write', scheme=force_scheme, domain=domain, impl=impl)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.4/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 160, in get_protocols_ordered
    candidates = _get_possible_protocols(rse_settings, operation, scheme, domain, impl)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.4/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 148, in _get_possible_protocols
    ' found : %s.' % str(rse_settings))
rucio.common.exception.RSEProtocolNotSupported: RSE does not support requested protocol.
Details: No protocol for provided settings found : {'availability_delete': True, 'availability_read': True, 'availability_write': True, 'credentials': None, 'deterministic': True, 'domain': ['lan', 'wan'], 'id': '0919372344bb45d28ad964ecfe091e1b', 'lfn2pfn_algorithm': 'hash', 'protocols': [{'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'davs', 'port': 8443, 'prefix': '/webdav/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy': 1}}, 'extended_attributes': None}, {'hostname': 'se03.esc.qmul.ac.uk', 'scheme': 'storm', 'port': 8443, 'prefix': '/mnt/lustre_3/storm_3/atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.storm.Default', 'domains': {'lan': {'read': 1, 'write': 0, 'delete': 0}, 'wan': {'read': 3, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}, {'hostname': 'xrootd01.esc.qmul.ac.uk', 'scheme': 'root', 'port': 1094, 'prefix': '//atlas/atlasscratchdisk/rucio/', 'impl': 'rucio.rse.protocols.gfal.Default', 'domains': {'lan': {'read': 0, 'write': 0, 'delete': 0}, 'wan': {'read': 2, 'write': 0, 'delete': 0, 'third_party_copy': 0}}, 'extended_attributes': None}], 'qos_class': None, 'rse': 'UKI-LT2-QMUL_SCRATCHDISK', 'rse_type': 'DISK', 'sign_url': None, 'staging_area': False, 'verify_checksum': True, 'volatile': False, 'read_protocol': 1, 'write_protocol': 1, 'delete_protocol': 1, 'third_party_copy_protocol': 1}.
```


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2012-2021 CERN
+# Copyright 2012-2022 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -34,7 +34,7 @@
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2020
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2020-2022
 # - Eric Vaandering <ewv@fnal.gov>, 2020
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
 # - Radu Carpa <radu.carpa@cern.ch>, 2021
@@ -218,11 +218,15 @@ def _pick_random_rse(rse_expression):
                     domain = 'lan'
             logger(logging.DEBUG, '{} domain is used for the upload'.format(domain))
 
-            if not impl and not force_scheme:
-                impl = self.preferred_impl(rse_settings, domain)
+            # FIXME:
+            # Rewrite preferred_impl selection - also check test_upload.py/test_download.py and fix impl order (see FIXME there)
+            #
+            # if not impl and not force_scheme:
+            #    impl = self.preferred_impl(rse_settings, domain)
 
             if not no_register and not register_after_upload:
                 self._register_file(file, registered_dataset_dids, ignore_availability=ignore_availability)
+
             # if register_after_upload, file should be overwritten if it is not registered
             # otherwise if file already exists on RSE we're done
             if register_after_upload:
@@ -589,9 +593,9 @@ def _upload_item(self, rse_settings, rse_attributes, lfn, source_dir=None, domai
         """
         logger = self.logger
 
-        # Construct protocol for write and read operation.
+        # Construct protocol for write operation.
+        # IMPORTANT: All upload stat() checks are always done with the write_protocol
         protocol_write = self._create_protocol(rse_settings, 'write', force_scheme=force_scheme, domain=domain, impl=impl)
-        protocol_read = self._create_protocol(rse_settings, 'read', domain=domain, impl=impl)
 
         base_name = lfn.get('filename', lfn['name'])
         name = lfn.get('name', base_name)
@@ -603,34 +607,29 @@ def _upload_item(self, rse_settings, rse_attributes, lfn, source_dir=None, domai
 
         # Getting pfn
         pfn = None
-        readpfn = None
         try:
             pfn = list(protocol_write.lfns2pfns(make_valid_did(lfn)).values())[0]
-            readpfn = list(protocol_read.lfns2pfns(make_valid_did(lfn)).values())[0]
             logger(logging.DEBUG, 'The PFN created from the LFN: {}'.format(pfn))
         except Exception as error:
             logger(logging.WARNING, 'Failed to create PFN for LFN: %s' % lfn)
             logger(logging.DEBUG, str(error), exc_info=True)
         if force_pfn:
             pfn = force_pfn
-            readpfn = pfn
             logger(logging.DEBUG, 'The given PFN is used: {}'.format(pfn))
 
         # Auth. mostly for object stores
         if sign_service:
             pfn = self.client.get_signed_url(rse_settings['rse'], sign_service, 'write', pfn)       # NOQA pylint: disable=undefined-variable
-            readpfn = self.client.get_signed_url(rse_settings['rse'], sign_service, 'read', pfn)    # NOQA pylint: disable=undefined-variable
 
         # Create a name of tmp file if renaming operation is supported
         pfn_tmp = '%s.rucio.upload' % pfn if protocol_write.renaming else pfn
-        readpfn_tmp = '%s.rucio.upload' % readpfn if protocol_write.renaming else readpfn
 
         # Either DID eixsts or not register_after_upload
-        if protocol_write.overwrite is False and delete_existing is False and protocol_read.exists(readpfn):
+        if protocol_write.overwrite is False and delete_existing is False and protocol_write.exists(pfn):
             raise FileReplicaAlreadyExists('File %s in scope %s already exists on storage as PFN %s' % (name, scope, pfn))  # wrong exception ?
 
         # Removing tmp from earlier attempts
-        if protocol_write.exists(readpfn_tmp):
+        if protocol_write.exists(pfn_tmp):
             logger(logging.DEBUG, 'Removing remains of previous upload attemtps.')
             try:
                 # Construct protocol for delete operation.
@@ -692,7 +691,6 @@ def _upload_item(self, rse_settings, rse_attributes, lfn, source_dir=None, domai
             raise RucioException('Unable to rename the tmp file %s.' % pfn_tmp)
 
         protocol_write.close()
-        protocol_read.close()
 
         return pfn
 
@@ -826,7 +824,6 @@ def preferred_impl(self, rse_settings, domain):
 
             :raises RucioException(msg): general exception with msg for more details.
         """
-
         preferred_protocols = []
         supported_impl = None
 
@@ -862,7 +859,7 @@ def preferred_impl(self, rse_settings, domain):
                 self.logger(logging.DEBUG, 'Unsuitable protocol "%s": All operations are not supported' % (protocol['impl']))
                 continue
             try:
-                supported_protocol = rsemgr.create_protocol(rse_settings, 'read', domain=domain, impl=protocol['impl'], auth_token=self.auth_token, logger=self.logger)
+                supported_protocol = rsemgr.create_protocol(rse_settings, 'write', domain=domain, impl=protocol['impl'], auth_token=self.auth_token, logger=self.logger)
                 supported_protocol.connect()
             except Exception as error:
                 self.logger(logging.DEBUG, 'Failed to create protocol "%s", exception: %s' % (protocol['impl'], error))


Test patch:
diff --git a/lib/rucio/tests/test_download.py b/lib/rucio/tests/test_download.py
--- a/lib/rucio/tests/test_download.py
+++ b/lib/rucio/tests/test_download.py
@@ -16,7 +16,7 @@
 # Authors:
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
 # - Tobias Wegner <twegner@cern.ch>, 2019
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2019
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2019-2022
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020-2021
@@ -24,7 +24,7 @@
 # - Radu Carpa <radu.carpa@cern.ch>, 2021
 # - Martin Barisits <martin.barisits@cern.ch>, 2021
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
-# - Joel Dierkes <joel.dierkes@cern.ch>, 2021-2022
+# - Joel Dierkes <joel.dierkes@cern.ch>, 2021
 
 import logging
 import os
@@ -676,11 +676,15 @@ def test_download_file_with_supported_protocol_from_config(rse_factory, did_fact
 
     rse, rse_id = rse_factory.make_rse()
 
+    # FIXME:
+    # The correct order to test should actually be scp,file,root
+    # However the preferred_impl is not working correctly.
+    # Once preferred_impl is fixed, this should be changed back
     add_protocol(rse_id, {'scheme': 'scp',
                           'hostname': '%s.cern.ch' % rse_id,
                           'port': 0,
                           'prefix': '/test/',
-                          'impl': 'rucio.rse.protocols.ssh.Default',
+                          'impl': 'rucio.rse.protocols.posix.Default',
                           'domains': {
                               'lan': {'read': 1, 'write': 1, 'delete': 1},
                               'wan': {'read': 1, 'write': 1, 'delete': 1}}})
@@ -688,7 +692,7 @@ def test_download_file_with_supported_protocol_from_config(rse_factory, did_fact
                           'hostname': '%s.cern.ch' % rse_id,
                           'port': 0,
                           'prefix': '/test/',
-                          'impl': 'rucio.rse.protocols.posix.Default',
+                          'impl': 'rucio.rse.protocols.scp.Default',
                           'domains': {
                               'lan': {'read': 2, 'write': 2, 'delete': 2},
                               'wan': {'read': 2, 'write': 2, 'delete': 2}}})
diff --git a/lib/rucio/tests/test_upload.py b/lib/rucio/tests/test_upload.py
--- a/lib/rucio/tests/test_upload.py
+++ b/lib/rucio/tests/test_upload.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2021 CERN
+# Copyright 2021-2022 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,6 +18,7 @@
 # - Mayank Sharma <imptodefeat@gmail.com>, 2021
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
 # - Joel Dierkes <joel.dierkes@cern.ch>, 2021
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2022
 
 import json
 import logging
@@ -290,11 +291,15 @@ def test_upload_file_with_supported_protocol(rse_factory, upload_client, mock_sc
 
     rse_name, rse_id = rse_factory.make_rse()
 
+    # FIXME:
+    # The correct order to test should actually be ssh,xrootd,posix
+    # However the preferred_impl is not working correctly.
+    # Once preferred_impl is fixed, this should be changed back
     add_protocol(rse_id, {'scheme': 'scp',
                           'hostname': '%s.cern.ch' % rse_id,
                           'port': 0,
                           'prefix': '/test/',
-                          'impl': 'rucio.rse.protocols.ssh.Default',
+                          'impl': 'rucio.rse.protocols.posix.Default',
                           'domains': {
                               'lan': {'read': 1, 'write': 1, 'delete': 1},
                               'wan': {'read': 1, 'write': 1, 'delete': 1}}})
@@ -310,7 +315,7 @@ def test_upload_file_with_supported_protocol(rse_factory, upload_client, mock_sc
                           'hostname': '%s.cern.ch' % rse_id,
                           'port': 0,
                           'prefix': '/test/',
-                          'impl': 'rucio.rse.protocols.posix.Default',
+                          'impl': 'rucio.rse.protocols.ssh.Default',
                           'domains': {
                               'lan': {'read': 3, 'write': 3, 'delete': 3},
                               'wan': {'read': 3, 'write': 3, 'delete': 3}}})
@@ -334,11 +339,15 @@ def test_upload_file_with_supported_protocol_from_config(rse_factory, upload_cli
 
     rse_name, rse_id = rse_factory.make_rse()
 
+    # FIXME:
+    # The correct order to test should actually be ssh,xrootd,posix
+    # However the preferred_impl is not working correctly.
+    # Once preferred_impl is fixed, this should be changed back
     add_protocol(rse_id, {'scheme': 'scp',
                           'hostname': '%s.cern.ch' % rse_id,
                           'port': 0,
                           'prefix': '/test/',
-                          'impl': 'rucio.rse.protocols.ssh.Default',
+                          'impl': 'rucio.rse.protocols.xrootd.Default',
                           'domains': {
                               'lan': {'read': 1, 'write': 1, 'delete': 1},
                               'wan': {'read': 1, 'write': 1, 'delete': 1}}})
@@ -354,7 +363,7 @@ def test_upload_file_with_supported_protocol_from_config(rse_factory, upload_cli
                           'hostname': '%s.cern.ch' % rse_id,
                           'port': 0,
                           'prefix': '/test/',
-                          'impl': 'rucio.rse.protocols.xrootd.Default',
+                          'impl': 'rucio.rse.protocols.ssh.Default',
                           'domains': {
                               'lan': {'read': 3, 'write': 3, 'delete': 3},
                               'wan': {'read': 3, 'write': 3, 'delete': 3}}})