Repository: rucio/rucio

Issue:
1.20.4rc3 fts bug
Motivation
----------
In 1.20.4rc3, the conveyor reported a crash with the trace:
```
2019-08-21 15:56:55,991 3229    ERROR   Thread [1/31] :  Failed to submit a job with error mc16_13TeV is not JSON serializable: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/common.py", line 125, in submit_transfer
    user_transfer_job=user_transfer_job)
  File "/usr/lib/python2.7/site-packages/rucio/core/transfer.py", line 90, in submit_bulk_transfers
    transfer_id = FTS3Transfertool(external_host=external_host).submit(files=job_files, job_params=job_params, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/rucio/transfertool/fts3.py", line 186, in submit
    params_str = json.dumps(params_dict)
  File "/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: mc16_13TeV is not JSON serializableh 
```



Modification
------------
Utilise rucio.common.utils.APIEncoder in the fts transfertools files




Discussion:
This patch also includes a fix for an error reported by the Undertaker:
```
2019-08-23 09:40:25,439 6947    CRITICAL        Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/daemons/undertaker/undertaker.py", line 99, in undertaker
    delete_dids(dids=chunk, account=InternalAccount('root'), expire_rules=True)
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/session.py", line 356, in new_funct
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rucio/core/did.py", line 584, in delete_dids
    rucio.core.rule.archive_localgroupdisk_datasets(scope=did['scope'], name=did['name'], session=session)
  File "/usr/lib/python2.7/site-packages/rucio/common/policy.py", line 96, in new_funct
    return function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/session.py", line 372, in new_funct
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rucio/core/rule.py", line 3113, in archive_localgroupdisk_datasets
    rucio.core.did.set_status(scope='archive', name=name, open=False, session=session)
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/session.py", line 372, in new_funct
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rucio/core/did.py", line 1476, in set_status
    values['bytes'], values['length'], values['events'] = __resolve_bytes_length_events_did(scope=scope, name=name, session=session)
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/session.py", line 372, in new_funct
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/rucio/core/did.py", line 1734, in __resolve_bytes_length_events_did
    did = session.query(models.DataIdentifier).filter_by(scope=scope, name=name).one()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3275, in one
    ret = self.one_or_none()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3244, in one_or_none
    ret = list(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    e, util.text_type(statement), parameters, None, None
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1179, in _execute_context
    context = constructor(dialect, self, conn, *args)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 745, in _init_compiled
    for key in compiled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 745, in <genexpr>
    for key in compiled_params
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 1196, in process
    return impl_processor(process_param(value, dialect))
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/types.py", line 191, in process_bind_param
    raise InvalidType('Cannot insert to db. Expected InternalScope, got string type.')
```

Resolving patch:
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.20.4rc3',
-    'branch_nick': 'patch-0-Release__1_20_4rc3_prep',
-    'revision_id': 'fe0518d675a40da8d34fae3c505a3763cc669c12',
-    'revno': 8433
+    'version': '1.20.4',
+    'branch_nick': 'patch-0-Release__1_20_4_preparation',
+    'revision_id': '2724087e1b31fec3740b10e4f03986305e325440',
+    'revno': 8448
 }


Test patch: