Repository: rucio/rucio

Issue:
Attaching files to archive extremely slow
Motivation
----------

```
2019-01-14 18:50:01,944 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 694.900 sec
2019-01-14 19:00:03,802 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 591.847 sec
2019-01-14 19:20:06,086 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 985.620 sec
2019-01-14 19:40:04,927 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 746.443 sec
2019-01-14 20:00:06,481 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 631.958 sec
2019-01-14 20:20:06,564 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 568.809 sec
2019-01-14 20:50:07,622 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 827.529 sec
2019-01-14 21:10:11,076 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 699.798 sec
2019-01-14 21:30:09,410 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 516.796 sec
2019-01-14 22:00:12,660 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 1058.377 sec
2019-01-14 22:20:14,131 panda.log.Adder: DEBUG    4212476111 registraion with backend=rucio for 2 files took 780.243 sec

```




Discussion:
Reason was identified as a problematic query which changes the query plan. Plan stabilisation needed

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -14,7 +14,7 @@
 #
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2013-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2013-2018
+# - Martin Barisits <martin.barisits@cern.ch>, 2013-2019
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2018
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2018
@@ -44,7 +44,7 @@
 
 from rucio.common import exception
 from rucio.common.config import config_get
-from rucio.common.utils import str_to_date, is_archive
+from rucio.common.utils import str_to_date, is_archive, chunks
 from rucio.core import account_counter, rse_counter
 from rucio.core.message import add_message
 from rucio.core.monitor import record_timer_block, record_counter
@@ -311,9 +311,11 @@ def __add_files_to_archive(scope, name, files, account, ignore_duplicate=False,
     try:
         new_files and session.bulk_insert_mappings(models.DataIdentifier, new_files)
         if existing_files_condition:
-            session.query(models.DataIdentifier).\
-                filter(models.DataIdentifier.did_type == DIDType.FILE).\
-                filter(or_(*existing_files_condition)).update({'constituent': True})
+            for chunk in chunks(existing_files_condition, 50):
+                session.query(models.DataIdentifier).\
+                    with_hint(models.DataIdentifier, "INDEX(DIDS DIDS_PK)", 'oracle').\
+                    filter(models.DataIdentifier.did_type == DIDType.FILE).\
+                    filter(or_(*chunk)).update({'constituent': True})
         contents and session.bulk_insert_mappings(models.ConstituentAssociation, contents)
         session.flush()
     except IntegrityError as error:


Test patch: