Repository: rucio/rucio

Issue:
Python 3 bug in core list_replicas
Motivation
----------

Found the following bug while trying to get Rucio server running on Python 3.8 (and this might also affect Python 3 versions below 3.8):

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/rucio/web/rest/flaskapi/v1/common.py", line 64, in dispatch_request
    return super(ErrorHandlingMethodView, self).dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/views.py", line 163, in dispatch_request
    return meth(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rucio/web/rest/flaskapi/v1/common.py", line 137, in decorated
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rucio/web/rest/flaskapi/v1/replicas.py", line 412, in post
    return try_stream(generate(request_id=request.environ.get('request_id'),
  File "/usr/local/lib/python3.8/site-packages/rucio/web/rest/flaskapi/v1/common.py", line 184, in try_stream
    peek = next(it)
  File "/usr/local/lib/python3.8/site-packages/rucio/web/rest/flaskapi/v1/replicas.py", line 340, in generate
    for rfile in list_replicas(dids=dids, schemes=schemes,
  File "/usr/local/lib/python3.8/site-packages/rucio/api/replica.py", line 186, in list_replicas
    for rep in replicas:
  File "/usr/local/lib/python3.8/site-packages/rucio/db/sqla/session.py", line 339, in new_funct
    for row in function(*args, **kwargs):
  File "/usr/local/lib/python3.8/site-packages/rucio/core/replica.py", line 1271, in list_replicas
    for f in _list_replicas(dataset_clause, file_clause, state_clause, pfns,
  File "/usr/local/lib/python3.8/site-packages/rucio/core/replica.py", line 936, in _list_replicas
    for tmp_archive in archive['pfns']:
RuntimeError: dictionary keys changed during iteration
```


Modification
------------

Fix the corresponding code: https://github.com/rucio/rucio/blob/8de8259d74204221b4cce5b4eb4e3dab55525813/lib/rucio/core/replica.py#L936-L953



Discussion:
Adding @rcarpa as well since he looked at that code recently.

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -933,9 +933,7 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
                         archive['md5'] = constituent_meta['md5']
                         archive['bytes'] = constituent_meta['bytes']
 
-                        for tmp_archive in archive['pfns']:
-                            # declare the constituent as being in a zip file
-                            archive['pfns'][tmp_archive]['domain'] = 'zip'
+                        for tmp_archive in list(archive['pfns']):
                             # at this point we don't know the protocol of the parent, so we have to peek
                             if tmp_archive.startswith('root://') and 'xrdcl.unzip' not in tmp_archive:
                                 # use direct addressable path for root
@@ -947,6 +945,9 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
                             if 'xrdcl.unzip' not in tmp_archive:
                                 archive['pfns'][tmp_archive]['client_extract'] = True
 
+                            # declare the constituent as being in a zip file
+                            archive['pfns'][tmp_archive]['domain'] = 'zip'
+
                             pfns.append((tmp_archive, 'zip',
                                          archive['pfns'][tmp_archive]['priority'],
                                          archive['pfns'][tmp_archive]['client_extract'],


Test patch: