Repository: rucio/rucio

Issue:
Size of the "extended_attributes" field of the "rse_protocols" table
Motivation
----------
In order to use the CMS TFC lfn2pfn plugin we currently store, in json format, the TFC rules (a stack of regexp of the LFN to PFN translation) in the extended_attributes of the RSE protocol. Like in

```
Protocols:
==========
  srm
    extended_attributes: {u'space_token': u'CMS', u'tfc_proto': u'srmv2', u'web_service_path': u'/srm/managerv2?SFN=', u'tfc': [{u'protocol': u'direct', u'result': u'/dpm/in2p3.fr/home/cms/trivcat/store/PhEDEx_Debug/LoadTest07_GRIFLLR/LoadTest07_GRIFLLR_$1', u'element_name': u'lfn-to-pfn', u'path-match': u'/store/PhEDEx_Debug/LoadTest07Source/GRIF_LLR_(\\S+)'}, {u'protocol': u'direct', u'result': u'/dpm/in2p3.fr/home/cms/trivcat/store/PhEDEx_Debug/LoadTest07_GRIFLLR', u'element_name': u'lfn-to-pfn', u'path-match': u'/store/PhEDEx_Debug/LoadTest07Source'}, {u'protocol': u'direct', u'result': u'/dpm/in2p3.fr/home/cms/trivcat/store/PhEDEx_Debug/LoadTest07_GRIFLLR/LoadTest07_GRIFLLR_$1', u'element_name': u'lfn-to-pfn', u'path-match': u'.*/LoadTest07_GRIF_LLR_(.*)_.*_.*'}, {u'destination-match': u'.*', u'protocol': u'direct', u'result': u'/dpm/in2p3.fr/home/cms/trivcat/$1', u'element_name': u'lfn-to-pfn', u'path-match': u'/+(.*)'}, {u'protocol': u'srmv2', u'chain': u'direct', u'destination-match': u'.*', u'result': u'srm://polgrid4.in2p3.fr:8446/srm/managerv2?SFN=/$1', u'path-match': u'/+(.*)', u'element_name': u'lfn-to-pfn'}]}
    hostname: polgrid4.in2p3.fr
    prefix: /dpm/in2p3.fr/home/cms
    domains: {u'wan': {u'read': 1, u'write': 0, u'third_party_copy': 0, u'delete': 0}, u'lan': {u'read': 0, u'write': 0, u'delete': 0}}
    scheme: srm
    port: 8446
    impl: rucio.rse.protocols.gfalv2.Default
...
```

The size of the field is currently VARCHAR2(1024).
That's enough to contain all the relevant rules of the TFC tested so far (LLR and Nebraska) but bigger TFC may not fit into this.


Modification
------------
Talking with Mario Lassnig: it may be worth to migrate the extended_attribute field to an arbitrary size json encoded field (like for the DID metadata). This can be done over the timescale of a couple of releases.
That will provide us with a definitive solution for entering the TFC rules in the RSE metadata.
In the meanwhile most of the current TFC should fit in to the current size (I'll double-check this).


Discussion:
That should be fine. I think how JSON() is implemented in sqlalchemy is that if the database does not support it, it just creates a blob or big varchar, thus we can switch to this even before oracle12 is available, you just can't use the json functionality. I think it should be fine to have this in the next feature release.

Resolving patch:
diff --git a/lib/rucio/db/sqla/migrate_repo/versions/8523998e2e76_increase_size_of_extended_attributes_.py b/lib/rucio/db/sqla/migrate_repo/versions/8523998e2e76_increase_size_of_extended_attributes_.py
new file mode 100644
--- /dev/null
+++ b/lib/rucio/db/sqla/migrate_repo/versions/8523998e2e76_increase_size_of_extended_attributes_.py
@@ -0,0 +1,44 @@
+# Copyright 2013-2019 CERN for the benefit of the ATLAS collaboration.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+# - Martin Barisits <martin.barisits@cern.ch>, 2019
+#
+# Topic: Increase size of extended_attributes column
+# Revision ID: 8523998e2e76
+# Revises: 3345511706b8
+# Creation Date: 2019-02-15 15:45:17.171346
+
+from alembic.op import (alter_column)
+
+import sqlalchemy as sa
+
+
+# revision identifiers used by alembic
+revision = '8523998e2e76'       # pylint: disable=invalid-name
+down_revision = '3345511706b8'  # pylint: disable=invalid-name
+
+
+def upgrade():
+    '''
+    Upgrade the database to this revision
+    '''
+    alter_column('rse_protocols', 'extended_attributes', existing_type=sa.String(1024), type_=sa.String(4000))
+
+
+def downgrade():
+    '''
+    Downgrade the database to the previous revision
+    '''
+    alter_column('rse_protocols', 'extended_attributes', existing_type=sa.String(4000), type_=sa.String(1024))
diff --git a/lib/rucio/db/sqla/models.py b/lib/rucio/db/sqla/models.py
--- a/lib/rucio/db/sqla/models.py
+++ b/lib/rucio/db/sqla/models.py
@@ -722,7 +722,7 @@ class RSEProtocols(BASE, ModelBase):
     write_wan = Column(Integer, server_default='0')  # if no value is provided, 0 i.e. not supported is assumed as default value
     delete_wan = Column(Integer, server_default='0')  # if no value is provided, 0 i.e. not supported is assumed as default value
     third_party_copy = Column(Integer, server_default='0')  # if no value is provided, 0 i.e. not supported is assumed as default value
-    extended_attributes = Column(String(1024), nullable=True)
+    extended_attributes = Column(String(4000), nullable=True)
     rses = relationship("RSE", backref="rse_protocols")
     _table_args = (PrimaryKeyConstraint('rse_id', 'scheme', 'hostname', 'port', name='RSE_PROTOCOL_PK'),
                    ForeignKeyConstraint(['rse_id'], ['rses.id'], name='RSE_PROTOCOL_RSE_ID_FK'),


Test patch: