Repository: freqtrade/freqtrade
Pull request: feat: enhance data fetching logic with pagination to increase market …

<!-- Thank you for sending your pull request. But first, have you included
unit tests, and is your code PEP8 conformant? [More details](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md)
-->
## Summary

enhance data fetching logic with pagination to increase market cap rank limit

closes #10742 
## Quick changelog

feat: enhance data fetching logic with pagination to increase market cap rank limit

## What's new?

enhance data fetching logic with pagination to increase market cap rank limit


Problem statement:
Unlimited Marketcap Pairlist?
<!-- 
Note: this section will not show up in the issue.
Have you search for this feature before requesting it? It's highly likely that a similar request was already filed.
-->

## Describe your environment
(if applicable)

  * Operating system: Windows Server 2022
  * Python Version: Current Miniconda (`python -V`)
  * CCXT version: _____ (`pip freeze | grep ccxt`)
  * Freqtrade Version: 2024.9 (`freqtrade -V` or `docker compose run --rm freqtrade -V` for Freqtrade running in docker)


## Describe the enhancement

Is it technically possible for you, to also make the marketcap pairlist unlimited, like the volume pairlist, going beyond the position of 250? This approach moves the responsibility (usefull or useless) from you as the programmer team to the trader. What do you think about this? 

*Explain the enhancement you would like*

Hints from issue discussion:
Is it technically possible? sure - it's pagination.
is it worth our time to implement, ensure it's correctly working and maintaining it in the future? 
I don't think so - unless you can provide a _very compelling_ reason to do so.

It's not a responsability question either - but simply a "it's a pointless feature we'd be wasting our time" decision (which can be changed, but in this issue, i currently don't see arguments for).

if you're using a Pairlist beyond 250 (so the market-cap of that coin is 250 positions below the top coin) - you can just as well go with a "all" pairlist or Volumepairlist.
Tokens past 250 are rather unlikely to be listed on exchanges anyway - which auto-excludes them (you can't have a pair in your whitelist you can't trade on the exchange).

If you're looking for a specific pair you see missing, you can use the  new "category" feature - but again, i don't see why it should paginate beyond the initial response.

Unless you'll have some compelling reason as to why we should complicate our code with this - I'm going to close this as a question which has been answered.
Since there is no more follow-up, I assume you have your question answered.

Commit message: chore: Convert page number to string to ensure type compatibility

Patch to review:
diff --git a/freqtrade/plugins/pairlist/MarketCapPairList.py b/freqtrade/plugins/pairlist/MarketCapPairList.py
index 2394e910ec7..b925d0b21e6 100644
--- a/freqtrade/plugins/pairlist/MarketCapPairList.py
+++ b/freqtrade/plugins/pairlist/MarketCapPairList.py
@@ -5,6 +5,7 @@
 """
 
 import logging
+import math
 
 from cachetools import TTLCache
 
@@ -56,8 +57,8 @@ def __init__(self, *args, **kwargs) -> None:
                         f"You can choose from {category_ids}"
                     )
 
-        if self._max_rank > 250:
-            raise OperationalException("This filter only support marketcap rank up to 250.")
+        if self._max_rank > 1000:
+            raise OperationalException("This filter only support marketcap rank up to 1000.")
 
     @property
     def needstickers(self) -> bool:
@@ -165,7 +166,11 @@ def filter_pairlist(self, pairlist: list[str], tickers: dict) -> list[str]:
             data = []
 
             if not self._categories:
-                data = self._coingecko.get_coins_markets(**default_kwargs)
+                pages_required = math.ceil(self._max_rank / 250)
+                for page in range(1, pages_required + 1):
+                    default_kwargs["page"] = str(page)
+                    page_data = self._coingecko.get_coins_markets(**default_kwargs)
+                    data.extend(page_data)
             else:
                 for category in self._categories:
                     category_data = self._coingecko.get_coins_markets(
diff --git a/tests/plugins/test_pairlist.py b/tests/plugins/test_pairlist.py
index 1c138cc559e..0cb1df32324 100644
--- a/tests/plugins/test_pairlist.py
+++ b/tests/plugins/test_pairlist.py
@@ -2458,10 +2458,10 @@ def test_MarketCapPairList_exceptions(mocker, default_conf_usdt):
         PairListManager(exchange, default_conf_usdt)
 
     default_conf_usdt["pairlists"] = [
-        {"method": "MarketCapPairList", "number_assets": 20, "max_rank": 260}
+        {"method": "MarketCapPairList", "number_assets": 20, "max_rank": 1010}
     ]
     with pytest.raises(
-        OperationalException, match="This filter only support marketcap rank up to 250."
+        OperationalException, match="This filter only support marketcap rank up to 1000."
     ):
         PairListManager(exchange, default_conf_usdt)
 


Code review comments:
diff_hunk: @@ -56,8 +57,8 @@ def __init__(self, *args, **kwargs) -> None:
                         f"You can choose from {category_ids}"
                     )
 
-        if self._max_rank > 250:
-            raise OperationalException("This filter only support marketcap rank up to 250.")
+        if self._max_rank > 1000:
path: freqtrade/plugins/pairlist/MarketCapPairList.py
text: @user1:
Is there a particular reason we should still care about this? (maybe it should be converted to a warning instead of an error?)

it made sense while we didn't paginate - but it seems like an arbitrary limit now as it's really irrelevant if you're running with the top 1_000, or top 15_000 pairs.
In the end, it's the user's responsibility (we can add a note to this) to keep this at a sane level? 


@author:
I have added a warning when max_rank exceeds 250 in MarketCapPairList.