Repository: rucio/rucio
Issue: Replica sorting does not work

Motivation
----------

We need file replicas to be returned according to distance from client. All the code is there but something is wrong and independently where the client is, rucio returns replicas in the same order. We (ServiceX, VP, xcache) need this ASAP.

To replicate
----------------
from lxplus using lsetup 'rucio testing'.
This gives you client 1.26.3

 rucio list-file-replicas data17_13TeV:DAOD_PHYS.23589107._001299.pool.root.1
+--------------+----------------------------------------+------------+-----------+------------------------
| data17_13TeV | DAOD_PHYS.23589107._001299.pool.root.1 | 1.220 GB   | 1f07615c  | FZK-LCG2_DATADISK: root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1 |
| data17_13TeV | DAOD_PHYS.23589107._001299.pool.root.1 | 1.220 GB   | 1f07615c  | MWT2_DATADISK: root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/fd/1a/DAOD_PHYS.23589107._001299.pool.root.1

gives LRZ first.

if I set the client to be at MWT2:

~ > export RUCIO_LATITUDE=41.7821
~ > export RUCIO_LONGITUDE=-87.6046

it still returns the same order.  

from python it does the same thing
{'latitude': None, 'ip': '2001:1458:d00:2f::100:2db', 'fqdn': 'lxplus734.cern.ch', 'longitude': None, 'site': 'CERN-PROD'}

Setting my lat and long to UChicago I see that it used it:

~ > export RUCIO_LATITUDE=41.7821
~ > export RUCIO_LONGITUDE=-87.6046

>>> from rucio.common.utils import detect_client_location
>>> detect_client_location()
{'latitude': 41.7821, 'ip': '2001:1458:d00:2f::100:2db', 'fqdn': 'lxplus734.cern.ch', 'longitude': -87.6046, 'site': 'CERN-PROD'}

Asking again for file replicas still gives me FZK as a first file replica...

When doing it from python the same happens:

from rucio.common.utils import detect_client_location from rucio.client.didclient import DIDClient from rucio.client.replicaclient import ReplicaClient

did_client = DIDClient()
replica_client = ReplicaClient()
detect_client_location()

scope='data17_13TeV'
ds='data17_13TeV.00339070.physics_Main.deriv.DAOD_PHYS.r10258_p3399_p4356_tid23589107_00'
fl=[]
count=0

res = did_client.list_files(scope, ds)
for f in res:
    fl.append({'scope':scope, 'name':f['name']})
    count+=1
    if count==10:
        break

reps=replica_client.list_replicas(dids=fl, schemes=['root']) for rep in reps:
    print(rep)

{u'adler32': u'4e9a36ae', u'name': u'DAOD_PHYS.23589107._000010.pool.root.1', u'rses': {u'FZK-LCG2_DATADISK': [u'root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1'], u'MWT2_DATADISK': [u'root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1']}, u'bytes': 1508349284, u'states': {u'FZK-LCG2_DATADISK': u'AVAILABLE', u'MWT2_DATADISK': u'AVAILABLE'}, u'pfns': {u'root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1': {u'domain': u'wan', u'rse': u'FZK-LCG2_DATADISK', u'priority': 1, u'volatile': False, u'client_extract': False, u'type': u'DISK', u'rse_id': u'a9dd4f46935d46ffa0962a9150235a67'}, u'root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlasdatadisk/rucio/data17_13TeV/0d/5d/DAOD_PHYS.23589107._000010.pool.root.1': {u'domain': u'wan', u'rse': u'MWT2_DATADISK', u'priority': 2, u'volatile': False, u'client_extract': False, u'type': u'DISK', u'rse_id': u'f4da89c7cef54133a45c22712dda45d5'}}, u'scope': u'data17_13TeV', u'md5': None}



so either is something wrong in calculation ,of  geoip distance or corrdinates  are not found correctly, or sorting is not done at all.  Or something completely different. I can't debug it  as I don't have access to these  k8s  pods to check what values they cache.  

I have checked that both MWT2 and LRZ-LMU ahave  reasonably good coordinates in maxmind database.

Changed files:
lib/rucio/core/replica_sorter.py

diff --git a/lib/rucio/core/replica_sorter.py b/lib/rucio/core/replica_sorter.py
index eaf4c8e72..36aff4b37 100644
--- a/lib/rucio/core/replica_sorter.py
+++ b/lib/rucio/core/replica_sorter.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2017-2020 CERN
+# Copyright 2020-2021 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,10 @@
 # limitations under the License.
 #
 # Authors:
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2017-2020
-# - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2018
-# - Vincent Garonne <vgaronne@gmail.com>, 2018
-# - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
-# - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2019
-# - Martin Barisits <martin.barisits@cern.ch>, 2019
-# - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
+# - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020-2021
+# - Ilija Vukotic <ivukotic@cern.ch>, 2021
+# - David Población Criado <david.poblacion.criado@cern.ch>, 2021
+# - Martin Barisits <martin.barisits@cern.ch>, 2021
 
 # This product includes GeoLite data created by MaxMind,
 # available from <a href="http://www.maxmind.com">http://www.maxmind.com</a>
@@ -119,7 +115,8 @@ def __get_distance(se1, client_location, ignore_error):
     :ignore_error: Ignore exception when the GeoLite DB cannot be retrieved
     """
     # does not cache ignore_error, str.lower on hostnames/ips is fine
-    canonical_parties = list(map(lambda x: str(x).lower(), [se1, client_location['ip']])).sort()
+    canonical_parties = list(map(lambda x: str(x).lower(), [se1, client_location['ip'], client_location['latitude'], client_location['longitude']]))
+    canonical_parties.sort()
     cache_key = f'replica_sorter:__get_distance|site_distance|{canonical_parties}'
     cache_val = REGION.get(cache_key)
     if cache_val is NO_VALUE: