Repository: rucio/rucio

Issue:
Replica sorting does not work
Motivation
----------

We need file replicas to be returned according to distance from client. All the code is there but something is wrong and independently where the client is, rucio returns replicas in the same order. We (ServiceX, VP, xcache) need this ASAP.

To replicate
----------------
from lxplus using lsetup 'rucio testing'.
This gives you client 1.26.3

 rucio list-file-replicas data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1
+--------------+----------------------------------------+------------+-----------+------------------------
| data17_13TeV | DAOD_PHYS.23589107._001299.pool.root.1 | 1.220 GB   | 1f07615c  | FZK-LCG2_DATADISK: root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1 |
| data17_13TeV | DAOD_PHYS.23589107._001299.pool.root.1 | 1.220 GB   | 1f07615c  | MWT2_DATADISK: root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1

gives LRZ first.

if I set the client to be at MWT2:

~ > export RUCIO_LATITUDE=41.7821
~ > export RUCIO_LONGITUDE=-87.6046

it still returns the same order.  

from python it does the same thing
{'latitude': None, 'ip': '2001:1458:d00:2f::100:2db', 'fqdn': 'lxplus734.cern.ch', 'longitude': None, 'site': 'CERN-PROD'}

Setting my lat and long to UChicago I see that it used it:

~ > export RUCIO_LATITUDE=41.7821
~ > export RUCIO_LONGITUDE=-87.6046

>>> from rucio.common.utils import detect_client_location
>>> detect_client_location()
{'latitude': 41.7821, 'ip': '2001:1458:d00:2f::100:2db', 'fqdn': 'lxplus734.cern.ch', 'longitude': -87.6046, 'site': 'CERN-PROD'}

Asking again for file replicas still gives me FZK as a first file replica...

When doing it from python the same happens:

from rucio.common.utils import detect_client_location from rucio.client.didclient import DIDClient from rucio.client.replicaclient import ReplicaClient

did_client = DIDClient()
replica_client = ReplicaClient()
detect_client_location()

scope='data17_13TeV'
ds='data17_13TeV.00339070.physics_Main.deriv.DAOD_PHYS.r10258_p3399_p4356_tid23589107_00'
fl=[]
count=0

res = did_client.list_files(scope, ds)
for f in res:
    fl.append({'scope':scope, 'name':f['name']})
    count+=1
    if count==10:
        break

reps=replica_client.list_replicas(dids=fl, schemes=['root']) for rep in reps:
    print(rep)

{u'adler32': u'4e9a36ae', u'name': u'DAOD_PHYS.23589107._000010.pool.root.1', u'rses': {u'FZK-LCG2_DATADISK': [u'root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1'], u'MWT2_DATADISK': [u'root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1']}, u'bytes': 1508349284, u'states': {u'FZK-LCG2_DATADISK': u'AVAILABLE', u'MWT2_DATADISK': u'AVAILABLE'}, u'pfns': {u'root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1': {u'domain': u'wan', u'rse': u'FZK-LCG2_DATADISK', u'priority': 1, u'volatile': False, u'client_extract': False, u'type': u'DISK', u'rse_id': u'a9dd4f46935d46ffa0962a9150235a67'}, u'root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1': {u'domain': u'wan', u'rse': u'MWT2_DATADISK', u'priority': 2, u'volatile': False, u'client_extract': False, u'type': u'DISK', u'rse_id': u'f4da89c7cef54133a45c22712dda45d5'}}, u'scope': u'data17_13TeV', u'md5': None}



so either is something wrong in calculation ,of  geoip distance or corrdinates  are not found correctly, or sorting is not done at all.  Or something completely different. I can't debug it  as I don't have access to these  k8s  pods to check what values they cache.  

I have checked that both MWT2 and LRZ-LMU ahave  reasonably good coordinates in maxmind database.


Discussion:
Just to add here, what we just discussed via mail

The issue is that the server only does the sorting if the client asks the replicas in METALINK format. Rucio download does that, rucio list-file-replicas does not.

See

https://github.com/rucio/rucio/blob/b6092bcd78f8cfac079e5b3cc25a92b4163960ed/lib/rucio/web/rest/flaskapi/v1/replicas.py#L362

and below line 397 where the sorting happens.

So if you do a list-file-replicas it never gets to the sorting step. It doesn't matter for download though. So for testing, request the metalink with --metalink to get an actual sorted result.

The issue why this was not backported to the JSON type returns is probably some sorting issue for dictionaries and the need of having the return format being stable. This can perhaps only be fixed in a v2 api (Would need to check though)
Unfortunately, this is not the only issue.
it does not work with metalink either.
`~ > rucio list-file-replicas data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1 --metalink --protocol root
<?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">
 <file name="DAOD_PHYS.23589107._001299.pool.root.1">
  <identity>data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1</identity>
  <hash type="adler32">1f07615c</hash>
  <size>1219706502</size>
  <glfn name="/atlas/rucio/data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1"></glfn>
  <url location="FZK-LCG2_DATADISK" domain="wan" priority="1" client_extract="false">root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1</url>
  <url location="MWT2_DATADISK" domain="wan" priority="2" client_extract="false">root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1</url>
 </file>
</metalink>
~ > export RUCIO_LATITUDE=41.7821
~ > export RUCIO_LONGITUDE=-87.6046
~ > rucio list-file-replicas data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1 --metalink --protocol root
<?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">
 <file name="DAOD_PHYS.23589107._001299.pool.root.1">
  <identity>data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1</identity>
  <hash type="adler32">1f07615c</hash>
  <size>1219706502</size>
  <glfn name="/atlas/rucio/data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1"></glfn>
  <url location="FZK-LCG2_DATADISK" domain="wan" priority="1" client_extract="false">root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1</url>
  <url location="MWT2_DATADISK" domain="wan" priority="2" client_extract="false">root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1</url>
 </file>
</metalink>`

also we need this to be default independently how do we ask for the replicas, the first one should be the closest. 
two examples: 
a. pilot asks rucio for a file that is remote. It should get the closest replica.
b. a user asks for the file using gLFN (there are people still doing it). should get closest replica.

Then there is some other issue; This is a bit painful to debug, we should rule out that this is not based on a geoip issue on the source. Easiest way would be to also give LAT/LONG information on the RSE, and if available, not even do the geoip lookup for the source. (Would be more accurate anyway)

For the other cases I don't fully agree that it should always be ordered. For most queries the client does not care if it is ordered or not, it basically only makes sense for a subsequent download. What is not good though is that the client does not really know when the result is ordered and when not (That `--metalink` supposedly orders it, is not really clear) - there is also the `sort=None` option, I think we have to disentangle that. (But these are two different issues and can be done independently)

Resolving patch:
diff --git a/lib/rucio/core/replica_sorter.py b/lib/rucio/core/replica_sorter.py
--- a/lib/rucio/core/replica_sorter.py
+++ b/lib/rucio/core/replica_sorter.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2017-2020 CERN
+# Copyright 2020-2021 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,10 @@
 # limitations under the License.
 #
 # Authors:
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2017-2020
-# - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2018
-# - Vincent Garonne <vgaronne@gmail.com>, 2018
-# - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
-# - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2019
-# - Martin Barisits <martin.barisits@cern.ch>, 2019
-# - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
+# - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020-2021
+# - Ilija Vukotic <ivukotic@cern.ch>, 2021
+# - David Población Criado <david.poblacion.criado@cern.ch>, 2021
+# - Martin Barisits <martin.barisits@cern.ch>, 2021
 
 # This product includes GeoLite data created by MaxMind,
 # available from <a href="http://www.maxmind.com">http://www.maxmind.com</a>
@@ -119,7 +115,8 @@ def __get_distance(se1, client_location, ignore_error):
     :ignore_error: Ignore exception when the GeoLite DB cannot be retrieved
     """
     # does not cache ignore_error, str.lower on hostnames/ips is fine
-    canonical_parties = list(map(lambda x: str(x).lower(), [se1, client_location['ip']])).sort()
+    canonical_parties = list(map(lambda x: str(x).lower(), [se1, client_location['ip'], client_location['latitude'], client_location['longitude']]))
+    canonical_parties.sort()
     cache_key = f'replica_sorter:__get_distance|site_distance|{canonical_parties}'
     cache_val = REGION.get(cache_key)
     if cache_val is NO_VALUE:


Test patch: