Repository: rucio/rucio

Issue:
bin/rucio download with --pfn option raises in AttributeError
Motivation
----------
ATLAS evevnt seersvice jobs still use the CLI to do certain downloads. With `1.27.2` the following download results in an `AttributeError`

```
2022-01-24 11:40:29,071 | INFO     | pilot.util.container             | print_executable          | executing command: export S3_ACCESS_KEY=XXXX; export S3_SECRET_KEY=********; export S3_IS_SECURE=True; /usr/bin/env rucio -v download --no-subdir --dir /home/tmp/atlas_UvQPWwAO/PanDA_Pilot-5337211214 --rse CERN-PROD_ES --pfn s3://s3.cern.ch:443//atlas-eventservice/EventService_premerge_27900612-5337200306-27468905815-10-5.tar transient:EventService_premerge_27900612-5337200306-27468905815-10-5.tar
2022-01-24 11:40:33,697 | WARNING  | pilot.control.data               | transfer                  | failed to transfer_files() using copytool=<module 'pilot.co\
pytool.objectstore' from '/home/tmp/atlas_UvQPWwAO/pilot3/pilot/copytool/objectstore.py'> .. skipped; error=error code: 1151, message: File transfer timed out \
during stage-in
details: copy command timed out: 2022-01-24 06:40:30,756 WARNING [The use of --rse is deprecated in favour of --rses and will be removed in 1.27]
2022-01-24 06:40:30,757 DEBUG [No trace_host passed. Using rucio_host instead]
2022-01-24 06:40:30,757 DEBUG [No creds passed. Trying to get it from the config file.]
2022-01-24 06:40:30,758 DEBUG [HTTPS is required, but no ca_cert was passed. Trying to get it from X509_CERT_DIR.]
2022-01-24 06:40:30,758 DEBUG [No account passed. Trying to get it from the RUCIO_ACCOUNT environment variable or the config file.]
2022-01-24 06:40:30,758 DEBUG [No VO passed. Trying to get it from environment variable RUCIO_VO.]
2022-01-24 06:40:30,758 DEBUG [No VO found. Trying to get it from the config file.]
2022-01-24 06:40:30,758 DEBUG [No VO found. Using default VO.]
2022-01-24 06:40:30,759 DEBUG [get a new token]
2022-01-24 06:40:30,759 DEBUG [HTTP request: GET https://voatlasrucio-auth-prod.cern.ch:443/auth/x509_proxy]
2022-01-24 06:40:30,759 DEBUG [HTTP header:  X-Rucio-Auth-Token: [hidden]]
2022-01-24 06:40:30,759 DEBUG [HTTP header:  X-Rucio-VO: def]
2022-01-24 06:40:30,759 DEBUG [HTTP header:  Connection: Keep-Alive]
2022-01-24 06:40:30,759 DEBUG [HTTP header:  User-Agent: rucio-clients/1.27.2]
2022-01-24 06:40:30,759 DEBUG [HTTP header:  X-Rucio-Script: rucio::-v]
2022-01-24 06:40:30,759 DEBUG [HTTP header:  X-Rucio-Account: pilot]
2022-01-24 06:40:31,231 DEBUG [HTTP Response: 200 OK]
2022-01-24 06:40:31,232 DEBUG [rucio token folder '/tmp/usatlas1/.rucio_usatlas1' not found. Create it.]
2022-01-24 06:40:31,279 WARNING [Download with --pfn doesn't support --transfer-speed-timeout]
2022-01-24 06:40:31,279 INFO [Processing 1 item(s) for input]
2022-01-24 06:40:31,279 DEBUG [Preparing PFN download of transient:EventService_premerge_27900612-5337200306-27468905815-10-5.tar (s3://s3.cern.ch:443//atlas-e\
ventservice/EventService_premerge_27900612-5337200306-27468905815-10-5.tar) from CERN-PROD_ES]
2022-01-24 06:40:31,279 INFO [Using main thread to download 1 file(s)]
2022-01-24 06:40:31,279 DEBUG [Start processing queued downloads]
2022-01-24 06:40:31,279 INFO [Preparing download of transient:EventService_premerge_27900612-5337200306-27468905815-10-5.tar]
2022-01-24 06:40:31,675 INFO [Trying to download with s3 and timeout of 360s from CERN-PROD_ES: transient:EventService_premerge_27900612-5337200306-27468905815\
-10-5.tar ]
2022-01-24 06:40:32,013 INFO [Using PFN: s3://s3.cern.ch:443//atlas-eventservice/EventService_premerge_27900612-5337200306-27468905815-10-5.tar]
2022-01-24 06:40:33,222 DEBUG [renaming '/home/tmp/atlas_UvQPWwAO/PanDA_Pilot-5337211214/EventService_premerge_27900612-5337200306-27468905815-10-5.tar.part' t\
o '/home/tmp/atlas_UvQPWwAO/PanDA_Pilot-5337211214/EventService_premerge_27900612-5337200306-27468905815-10-5.tar']
2022-01-24 06:40:33,605 INFO [File transient:EventService_premerge_27900612-5337200306-27468905815-10-5.tar successfully downloaded in 1.21 seconds]
2022-01-24 06:40:33,608 DEBUG [Traceback (most recent call last):
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/bin/rucio", line 152, in new_funct
    return function(*args, **kwargs)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.27.2/bin/rucio", line 1201, in download
    for did, did_stats in item.get('input_dids').items():
AttributeError: 'NoneType' object has no attribute 'items'
```

Since it is a download with --pfn and `download_client.download_pfns` is used vs. `download_client.download_dids` I assume there is some kind of difference in the result generation which ticks `bin/rucio` off. 


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -29,11 +29,11 @@
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2021
-# - Radu Carpa <radu.carpa@cern.ch>, 2021
+# - Radu Carpa <radu.carpa@cern.ch>, 2021-2022
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
 # - David Población Criado <david.poblacion.criado@cern.ch>, 2021
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2021
-# - Joel Dierkes <joel.dierkes@cern.ch>, 2021-2022
+# - Joel Dierkes <joel.dierkes@cern.ch>, 2021
 
 from __future__ import division
 
@@ -224,6 +224,7 @@ def download_pfns(self, items, num_threads=2, trace_custom_fields={}, traces_cop
             did_str = item.get('did')
             pfn = item.get('pfn')
             rse = item.get('rse')
+            item['input_dids'] = {DIDType(did_str): {}}
 
             if not did_str or not pfn or not rse:
                 logger(logging.DEBUG, item)


Test patch:
diff --git a/lib/rucio/tests/test_bin_rucio.py b/lib/rucio/tests/test_bin_rucio.py
--- a/lib/rucio/tests/test_bin_rucio.py
+++ b/lib/rucio/tests/test_bin_rucio.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2012-2021 CERN
+# Copyright 2012-2022 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,7 +21,7 @@
 # - Joaquín Bogado <jbogado@linti.unlp.edu.ar>, 2014-2018
 # - Cheng-Hsi Chao <cheng-hsi.chao@cern.ch>, 2014
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2015
-# - Martin Barisits <martin.barisits@cern.ch>, 2015-2021
+# - Martin Barisits <martin.barisits@cern.ch>, 2015-2022
 # - Frank Berghaus <frank.berghaus@cern.ch>, 2017-2018
 # - Tobias Wegner <twegner@cern.ch>, 2018
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018-2019
@@ -30,7 +30,7 @@
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Tomas Javurek <tomas.javurek@cern.ch>, 2020
-# - Radu Carpa <radu.carpa@cern.ch>, 2021
+# - Radu Carpa <radu.carpa@cern.ch>, 2021-2022
 # - Rahul Chauhan <omrahulchauhan@gmail.com>, 2021
 # - Simon Fayer <simon.fayer05@imperial.ac.uk>, 2021
 # - David Población Criado <david.poblacion.criado@cern.ch>, 2021
@@ -687,6 +687,30 @@ def test_download_file(self):
         except Exception:
             pass
 
+    def test_download_pfn(self):
+        """CLIENT(USER): Rucio download files"""
+        tmp_file1 = file_generator()
+        name = os.path.basename(tmp_file1)
+        # add files
+        cmd = 'rucio upload --rse {0} --scope {1} {2}'.format(self.def_rse, self.user, tmp_file1)
+        print(self.marker + cmd)
+        exitcode, out, err = execute(cmd)
+        print(out, err)
+
+        # download files
+        replica_pfn = list(self.replica_client.list_replicas([{'scope': self.user, 'name': name}]))[0]['rses'][self.def_rse][0]
+        cmd = 'rucio -v download --rse {0} --pfn {1} {2}:{3}'.format(self.def_rse, replica_pfn, self.user, name)
+        exitcode, out, err = execute(cmd)
+        print(out, err)
+        assert re.search('Total files.*1', out) is not None
+
+        try:
+            for i in listdir('data13_hip'):
+                unlink('data13_hip/%s' % i)
+            rmdir('data13_hip')
+        except Exception:
+            pass
+
     def test_download_file_with_impl(self):
         """CLIENT(USER): Rucio download files with impl parameter assigned 'posix' value"""
         tmp_file1 = file_generator()
diff --git a/lib/rucio/tests/test_conveyor.py b/lib/rucio/tests/test_conveyor.py
--- a/lib/rucio/tests/test_conveyor.py
+++ b/lib/rucio/tests/test_conveyor.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2015-2021 CERN
+# Copyright 2015-2022 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,8 +16,8 @@
 # Authors:
 # - Wen Guan <wen.guan@cern.ch>, 2015-2016
 # - Vincent Garonne <vincent.garonne@cern.ch>, 2016
-# - Martin Barisits <martin.barisits@cern.ch>, 2019-2021
-# - Radu Carpa <radu.carpa@cern.ch>, 2021
+# - Martin Barisits <martin.barisits@cern.ch>, 2019-2022
+# - Radu Carpa <radu.carpa@cern.ch>, 2021-2022
 # - Mayank Sharma <imptodefeat@gmail.com>, 2021
 # - David Población Criado <david.poblacion.criado@cern.ch>, 2021
 # - Joel Dierkes <joel.dierkes@cern.ch>, 2021
@@ -205,6 +205,8 @@ def test_multihop_intermediate_replica_lifecycle(vo, did_factory, root_account,
 
         # FTS fails the second transfer
         request = __wait_for_request_state(dst_rse_id=dst_rse_id, state=RequestState.FAILED, **did)
+        # Call finisher once to update the source rankings
+        finisher(once=True, partition_wait_time=None)
         # ensure tha the ranking was correctly decreased
         assert __get_source(request_id=request['id'], src_rse_id=jump_rse_id, **did).ranking == -1
         # run submitter again to copy from jump rse to destination rse