Repository: freqtrade/freqtrade

Issue:
fiat-convert: coingecko symbols are not unique
## Describe your environment

  * Operating system: Debian testing
  * Python Version: 3.9.1 (`python -V`)
  * CCXT version: doesn't matter (`pip freeze | grep ccxt`)
  * Freqtrade Version: develop (`freqtrade -V` or `docker-compose run --rm freqtrade -V` for Freqtrade running in docker)
  
## Describe the problem:

While working on #5067 I've discovered that the coingecko coin list has duplicated symbols

```
In [7]: from pycoingecko import CoinGeckoAPI
   ...: cg = CoinGeckoAPI()

In [8]: a=cg.get_coins_list()

In [9]: b = [x['symbol'] for x in a]

In [10]: len(set([x for x in b if b.count(x) > 1]))
Out[10]: 747
```

https://github.com/freqtrade/freqtrade/blob/a0893b291a099b928ced567b9ef81f2d2795717a/freqtrade/rpc/fiat_convert.py#L54 also wrongfully assumes that the symbols are uniqe.

I've added a hand-hacked workaround for the symbols from binance in #5067 - but a more general solution might be needed.




Discussion:
I guess you should rather raise this against either the PyCoingecko repository - or better yet with support from coingecko (but i doubt either of them will do anything for this, as it's a problem of no standardization across the crypto space).

Crypto symbols are (in some cases) not unique - and the same coin can be called A on one exchange, and B in another exchange. 
This can happen with small tokens (atm - which can be "Africa to Mars" and "Atletico Madrid Fan Token") - but sometimes also affects bigger tokens, if they are listed with different names on different exchanges.

We will definitely not have this list hand-coded / hard-coded anywhere - as it will be outdated tomorrow, as exchanges list coins all the time.
If you think that's a requirement - will you commit to keeping this list uptodate (for all supported exchanges) ... otherwise, that's clearly a no-go.
Well clearly not PyCoingecko's fault.

What would be the best fix imho
- get the marketcap for duplicate symbols
- remove all with marketcap == none
- for those that are left, use the higher marketcap.

Thats a few lines of code, with a result that could be cached for a longer time.
I'd be happy to write that, just not sure what the best place for it is.

In the current state, fiat conversion for some well known coins will give awkward results.

where marketcap == none - sure - these can be dropped easily (you should also check for `market_cap == 0.0` - this seems to be prevelant ...)

I'm unsure about the 3rd point. 
Assuming we have 2 coins with different marketcaps - what makes the "higher" market-cap the right one?

In my understanding, it'll depend on the exchange - and picking the "wrong" one might lead you to trade a coin with a market-cap of a few 1000$ - instead of the one with millions - simply because the "guess" was wrong. 

It's safer to remove these completely if we can't be sure which one is traded on this particular exchange.

Resolving patch:
diff --git a/freqtrade/rpc/fiat_convert.py b/freqtrade/rpc/fiat_convert.py
--- a/freqtrade/rpc/fiat_convert.py
+++ b/freqtrade/rpc/fiat_convert.py
@@ -5,7 +5,7 @@
 
 import datetime
 import logging
-from typing import Dict
+from typing import Dict, List
 
 from cachetools.ttl import TTLCache
 from pycoingecko import CoinGeckoAPI
@@ -25,8 +25,7 @@ class CryptoToFiatConverter:
     """
     __instance = None
     _coingekko: CoinGeckoAPI = None
-
-    _cryptomap: Dict = {}
+    _coinlistings: List[Dict] = []
     _backoff: float = 0.0
 
     def __new__(cls):
@@ -49,9 +48,8 @@ def __init__(self) -> None:
 
     def _load_cryptomap(self) -> None:
         try:
-            coinlistings = self._coingekko.get_coins_list()
-            # Create mapping table from symbol to coingekko_id
-            self._cryptomap = {x['symbol']: x['id'] for x in coinlistings}
+            # Use list-comprehension to ensure we get a list.
+            self._coinlistings = [x for x in self._coingekko.get_coins_list()]
         except RequestException as request_exception:
             if "429" in str(request_exception):
                 logger.warning(
@@ -69,6 +67,24 @@ def _load_cryptomap(self) -> None:
             logger.error(
                 f"Could not load FIAT Cryptocurrency map for the following problem: {exception}")
 
+    def _get_gekko_id(self, crypto_symbol):
+        if not self._coinlistings:
+            if self._backoff <= datetime.datetime.now().timestamp():
+                self._load_cryptomap()
+                # Still not loaded.
+                if not self._coinlistings:
+                    return None
+            else:
+                return None
+        found = [x for x in self._coinlistings if x['symbol'] == crypto_symbol]
+        if len(found) == 1:
+            return found[0]['id']
+
+        if len(found) > 0:
+            # Wrong!
+            logger.warning(f"Found multiple mappings in goingekko for {crypto_symbol}.")
+            return None
+
     def convert_amount(self, crypto_amount: float, crypto_symbol: str, fiat_symbol: str) -> float:
         """
         Convert an amount of crypto-currency to fiat
@@ -143,22 +159,14 @@ def _find_price(self, crypto_symbol: str, fiat_symbol: str) -> float:
         if crypto_symbol == fiat_symbol:
             return 1.0
 
-        if self._cryptomap == {}:
-            if self._backoff <= datetime.datetime.now().timestamp():
-                self._load_cryptomap()
-                # return 0.0 if we still don't have data to check, no reason to proceed
-                if self._cryptomap == {}:
-                    return 0.0
-            else:
-                return 0.0
+        _gekko_id = self._get_gekko_id(crypto_symbol)
 
-        if crypto_symbol not in self._cryptomap:
+        if not _gekko_id:
             # return 0 for unsupported stake currencies (fiat-convert should not break the bot)
             logger.warning("unsupported crypto-symbol %s - returning 0.0", crypto_symbol)
             return 0.0
 
         try:
-            _gekko_id = self._cryptomap[crypto_symbol]
             return float(
                 self._coingekko.get_price(
                     ids=_gekko_id,


Test patch:
diff --git a/tests/rpc/test_fiat_convert.py b/tests/rpc/test_fiat_convert.py
--- a/tests/rpc/test_fiat_convert.py
+++ b/tests/rpc/test_fiat_convert.py
@@ -22,7 +22,7 @@ def test_fiat_convert_is_supported(mocker):
 def test_fiat_convert_find_price(mocker):
     fiat_convert = CryptoToFiatConverter()
 
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = {}
     fiat_convert._backoff = 0
     mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._load_cryptomap',
                  return_value=None)
@@ -44,7 +44,7 @@ def test_fiat_convert_find_price(mocker):
 
 
 def test_fiat_convert_unsupported_crypto(mocker, caplog):
-    mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._cryptomap', return_value=[])
+    mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._coinlistings', return_value=[])
     fiat_convert = CryptoToFiatConverter()
     assert fiat_convert._find_price(crypto_symbol='CRYPTO_123', fiat_symbol='EUR') == 0.0
     assert log_has('unsupported crypto-symbol CRYPTO_123 - returning 0.0', caplog)
@@ -88,9 +88,9 @@ def test_fiat_convert_two_FIAT(mocker):
 def test_loadcryptomap(mocker):
 
     fiat_convert = CryptoToFiatConverter()
-    assert len(fiat_convert._cryptomap) == 2
+    assert len(fiat_convert._coinlistings) == 2
 
-    assert fiat_convert._cryptomap["btc"] == "bitcoin"
+    assert fiat_convert._get_gekko_id("btc") == "bitcoin"
 
 
 def test_fiat_init_network_exception(mocker):
@@ -102,11 +102,10 @@ def test_fiat_init_network_exception(mocker):
     )
     # with pytest.raises(RequestEsxception):
     fiat_convert = CryptoToFiatConverter()
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = {}
     fiat_convert._load_cryptomap()
 
-    length_cryptomap = len(fiat_convert._cryptomap)
-    assert length_cryptomap == 0
+    assert len(fiat_convert._coinlistings) == 0
 
 
 def test_fiat_convert_without_network(mocker):
@@ -132,11 +131,10 @@ def test_fiat_too_many_requests_response(mocker, caplog):
     )
     # with pytest.raises(RequestEsxception):
     fiat_convert = CryptoToFiatConverter()
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = {}
     fiat_convert._load_cryptomap()
 
-    length_cryptomap = len(fiat_convert._cryptomap)
-    assert length_cryptomap == 0
+    assert len(fiat_convert._coinlistings) == 0
     assert fiat_convert._backoff > datetime.datetime.now().timestamp()
     assert log_has(
         'Too many requests for Coingecko API, backing off and trying again later.',
@@ -144,20 +142,33 @@ def test_fiat_too_many_requests_response(mocker, caplog):
     )
 
 
+def test_fiat_multiple_coins(mocker, caplog):
+    fiat_convert = CryptoToFiatConverter()
+    fiat_convert._coinlistings = [
+        {'id': 'helium', 'symbol': 'hnt', 'name': 'Helium'},
+        {'id': 'hymnode', 'symbol': 'hnt', 'name': 'Hymnode'},
+        {'id': 'bitcoin', 'symbol': 'btc', 'name': 'Bitcoin'},
+    ]
+
+    assert fiat_convert._get_gekko_id('btc') == 'bitcoin'
+    assert fiat_convert._get_gekko_id('hnt') is None
+
+    assert log_has('Found multiple mappings in goingekko for hnt.', caplog)
+
+
 def test_fiat_invalid_response(mocker, caplog):
     # Because CryptoToFiatConverter is a Singleton we reset the listings
-    listmock = MagicMock(return_value="{'novalidjson':DEADBEEFf}")
+    listmock = MagicMock(return_value=None)
     mocker.patch.multiple(
         'freqtrade.rpc.fiat_convert.CoinGeckoAPI',
         get_coins_list=listmock,
     )
     # with pytest.raises(RequestEsxception):
     fiat_convert = CryptoToFiatConverter()
-    fiat_convert._cryptomap = {}
+    fiat_convert._coinlistings = []
     fiat_convert._load_cryptomap()
 
-    length_cryptomap = len(fiat_convert._cryptomap)
-    assert length_cryptomap == 0
+    assert len(fiat_convert._coinlistings) == 0
     assert log_has_re('Could not load FIAT Cryptocurrency map for the following problem: .*',
                       caplog)