Repository: rucio/rucio

Issue:
Crash when trying to find matching scheme
From the CMS instance, I noticed this exception thrown in the `conveyor-transfer-submitter` after creating some minimal RSEs and trying to trigger a transfer
```
2018-01-11 20:59:36,983	4217	CRITICAL	Exception happened when trying to get transfer for request 8a818df9ca4047e3a6b36f95b98a5c7e: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/core/transfer.py", line 567, in get_transfer_requests_and_source_replicas
    scheme=current_schemes)
  File "/usr/lib/python2.7/site-packages/rucio/rse/rsemanager.py", line 640, in find_matching_scheme
    dest_candidates = sorted(dest_candidates, key=lambda k: k['domains'][domain][operation_dest])
  File "/usr/lib/python2.7/site-packages/rucio/rse/rsemanager.py", line 640, in <lambda>
    dest_candidates = sorted(dest_candidates, key=lambda k: k['domains'][domain][operation_dest])
KeyError: None
```



Discussion:
Here's the protocol definition for the available destination RSE:
```
Protocols:
==========
  gsiftp
    extended_attributes: None
    hostname: gftp.t2.ucsd.edu
    prefix: /hadoop/oppo
    domains: {u'wan': {u'read': 1, u'write': 1, u'third_party_copy': 0, u'delete': 1}, u'lan': {u'read': 0, u'write': 0, u'delete': 0}}
    scheme: gsiftp
    port: 0
    impl: rucio.rse.protocols.gfalv2.Default
```
and the source RSE:
```
Protocols:
==========
  gsiftp
    extended_attributes: None
    hostname: red-gridftp.unl.edu
    prefix: /dropfiles/cms/store
    domains: {u'wan': {u'read': 1, u'write': 1, u'third_party_copy': 0, u'delete': 1}, u'lan': {u'read': 0, u'write': 0, u'delete': 0}}
    scheme: gsiftp
    port: 0
    impl: rucio.rse.protocols.gfalv2.Default
```
This is the suspect line of code:

https://github.com/rucio/rucio/blob/master/lib/rucio/core/transfer.py#L566

When I remove the `domain=None` and just let it use the default keyword (`domain='wan'`), everything works.

However, that definitely wouldn't explain why transfers work for other folks though!
This looks like a regression from the latest WAN/LAN domain support patch! We actually have a domain called "third_party_copy" which we would like to use in the future (because it might be different than WAN data access priority), but it's not used yet.

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -7,7 +7,7 @@
 #
 # Authors:
 # - Martin Barisits, <martin.barisits@cern.ch>, 2017
-# - Mario Lassnig, <mario.lassnig@cern.ch>, 2017
+# - Mario Lassnig, <mario.lassnig@cern.ch>, 2017-2018
 
 import datetime
 import json
@@ -558,12 +558,13 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                 allow_tape_source = True
 
                 # Find matching scheme between destination and source
+                # TODO: In the future, move to domain "third_party_copy"
                 try:
                     matching_scheme = rsemgr.find_matching_scheme(rse_settings_dest=rses_info[dest_rse_id],
                                                                   rse_settings_src=rses_info[source_rse_id],
                                                                   operation_src='read',
                                                                   operation_dest='write',
-                                                                  domain=None,
+                                                                  domain='wan',
                                                                   scheme=current_schemes)
                 except RSEProtocolNotSupported:
                     logging.error('Operation "write" not supported by %s with schemes %s' % (rses_info[dest_rse_id]['rse'], current_schemes))


Test patch: