Repository: freqtrade/freqtrade

Issue:
AgeFilter fetches 1d candle data in every loop
  * Operating system: Ubuntu 20.04.3 LTS
  * Python Version: 3.8.10
  * CCXT version: 1.55.28
  * Freqtrade Version: 2021.8 stable

First a big thanks for your great work!

I have a question regarding the AgeFilter.
After the AgeFilter removes some newly listed coins it will check these coins in every loop. Is behavior intended?
Because with the check freqtrade also fetches the 1d candle data for these coins, which means more request weight on the exchange. Right now I use "process_throttle_secs": 1 and have 6 new coins removed. This results in an additional request weight of 360/min.
Is it possible that the AgeFilter only fetches the coins, when the VolumePairsList gets refreshed and uses the cached data in the other loops?

Beste regards
Borsti 


Log:
```
2021-09-07 14:49:55,730 - freqtrade.worker - DEBUG - Throttling with '_process_running()': sleep for 0.1 s, last iteration took 0.9 s.
2021-09-07 14:49:55,730 - freqtrade.worker - DEBUG - ========================================
2021-09-07 14:49:55,734 - freqtrade.exchange.exchange - DEBUG - Refreshing candle (OHLCV) data for 6 pairs
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair MBOX/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair KEEP/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair RAY/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair XEC/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,736 - freqtrade.exchange.exchange - DEBUG - Fetching pair QNT/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,736 - freqtrade.exchange.exchange - DEBUG - Fetching pair C98/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:56,035 - freqtrade.exchange.exchange - DEBUG - Done fetching pair MBOX/USDT, interval 1d ...
2021-09-07 14:49:56,154 - freqtrade.exchange.exchange - DEBUG - Done fetching pair KEEP/USDT, interval 1d ...
2021-09-07 14:49:56,156 - freqtrade.exchange.exchange - DEBUG - Done fetching pair RAY/USDT, interval 1d ...
2021-09-07 14:49:56,188 - freqtrade.exchange.exchange - DEBUG - Done fetching pair XEC/USDT, interval 1d ...
2021-09-07 14:49:56,274 - freqtrade.exchange.exchange - DEBUG - Done fetching pair QNT/USDT, interval 1d ...
2021-09-07 14:49:56,298 - freqtrade.exchange.exchange - DEBUG - Done fetching pair C98/USDT, interval 1d ...
2021-09-07 14:49:56,299 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair MBOX/USDT.
2021-09-07 14:49:56,309 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,320 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair KEEP/USDT.
2021-09-07 14:49:56,330 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,338 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair RAY/USDT.
2021-09-07 14:49:56,346 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,354 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair XEC/USDT.
2021-09-07 14:49:56,361 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,369 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair QNT/USDT.
2021-09-07 14:49:56,377 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,385 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair C98/USDT.
2021-09-07 14:49:56,393 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,405 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed RAY/USDT from whitelist, because age 28 is less than 90 days
2021-09-07 14:49:56,406 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed XEC/USDT from whitelist, because age 4 is less than 90 days
2021-09-07 14:49:56,406 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed QNT/USDT from whitelist, because age 40 is less than 90 days
2021-09-07 14:49:56,407 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed C98/USDT from whitelist, because age 46 is less than 90 days
2021-09-07 14:49:56,407 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed KEEP/USDT from whitelist, because age 82 is less than 90 days
2021-09-07 14:49:56,407 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed MBOX/USDT from whitelist, because age 19 is less than 90 days
2021-09-07 14:49:56,408 - freqtrade.plugins.pairlist.IPairList - DEBUG - Validated 74 pairs.
2021-09-07 14:49:56,430 - freqtrade.exchange.exchange - DEBUG - Refreshing candle (OHLCV) data for 207 pairs
2021-09-07 14:49:56,431 - freqtrade.exchange.exchange - DEBUG - Using cached candle (OHLCV) data for pair CHR/USDT, timeframe 1d ...
2021-09-07 14:49:56,431 - freqtrade.exchange.exchange - DEBUG - Using cached candle (OHLCV) data for pair BTT/USDT, timeframe 5m ...
2021-09-07 14:49:56,432 - freqtrade.exchange.exchange - DEBUG - Using cached candle (OHLCV) data for pair LINK/USDT, timeframe 1h ...
.
.
.
```
Edit: Formatting Log

AgeFilter fetches 1d candle data in every loop
  * Operating system: Ubuntu 20.04.3 LTS
  * Python Version: 3.8.10
  * CCXT version: 1.55.28
  * Freqtrade Version: 2021.8 stable

First a big thanks for your great work!

I have a question regarding the AgeFilter.
After the AgeFilter removes some newly listed coins it will check these coins in every loop. Is behavior intended?
Because with the check freqtrade also fetches the 1d candle data for these coins, which means more request weight on the exchange. Right now I use "process_throttle_secs": 1 and have 6 new coins removed. This results in an additional request weight of 360/min.
Is it possible that the AgeFilter only fetches the coins, when the VolumePairsList gets refreshed and uses the cached data in the other loops?

Beste regards
Borsti 


Log:
```
2021-09-07 14:49:55,730 - freqtrade.worker - DEBUG - Throttling with '_process_running()': sleep for 0.1 s, last iteration took 0.9 s.
2021-09-07 14:49:55,730 - freqtrade.worker - DEBUG - ========================================
2021-09-07 14:49:55,734 - freqtrade.exchange.exchange - DEBUG - Refreshing candle (OHLCV) data for 6 pairs
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair MBOX/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair KEEP/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair RAY/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,735 - freqtrade.exchange.exchange - DEBUG - Fetching pair XEC/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,736 - freqtrade.exchange.exchange - DEBUG - Fetching pair QNT/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:55,736 - freqtrade.exchange.exchange - DEBUG - Fetching pair C98/USDT, interval 1d, since 1623110400000 (2021-06-08T00:00:00+00:00) ...
2021-09-07 14:49:56,035 - freqtrade.exchange.exchange - DEBUG - Done fetching pair MBOX/USDT, interval 1d ...
2021-09-07 14:49:56,154 - freqtrade.exchange.exchange - DEBUG - Done fetching pair KEEP/USDT, interval 1d ...
2021-09-07 14:49:56,156 - freqtrade.exchange.exchange - DEBUG - Done fetching pair RAY/USDT, interval 1d ...
2021-09-07 14:49:56,188 - freqtrade.exchange.exchange - DEBUG - Done fetching pair XEC/USDT, interval 1d ...
2021-09-07 14:49:56,274 - freqtrade.exchange.exchange - DEBUG - Done fetching pair QNT/USDT, interval 1d ...
2021-09-07 14:49:56,298 - freqtrade.exchange.exchange - DEBUG - Done fetching pair C98/USDT, interval 1d ...
2021-09-07 14:49:56,299 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair MBOX/USDT.
2021-09-07 14:49:56,309 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,320 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair KEEP/USDT.
2021-09-07 14:49:56,330 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,338 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair RAY/USDT.
2021-09-07 14:49:56,346 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,354 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair XEC/USDT.
2021-09-07 14:49:56,361 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,369 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair QNT/USDT.
2021-09-07 14:49:56,377 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,385 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair C98/USDT.
2021-09-07 14:49:56,393 - freqtrade.data.converter - DEBUG - Dropping last candle
2021-09-07 14:49:56,405 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed RAY/USDT from whitelist, because age 28 is less than 90 days
2021-09-07 14:49:56,406 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed XEC/USDT from whitelist, because age 4 is less than 90 days
2021-09-07 14:49:56,406 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed QNT/USDT from whitelist, because age 40 is less than 90 days
2021-09-07 14:49:56,407 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed C98/USDT from whitelist, because age 46 is less than 90 days
2021-09-07 14:49:56,407 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed KEEP/USDT from whitelist, because age 82 is less than 90 days
2021-09-07 14:49:56,407 - freqtrade.plugins.pairlist.IPairList - DEBUG - Removed MBOX/USDT from whitelist, because age 19 is less than 90 days
2021-09-07 14:49:56,408 - freqtrade.plugins.pairlist.IPairList - DEBUG - Validated 74 pairs.
2021-09-07 14:49:56,430 - freqtrade.exchange.exchange - DEBUG - Refreshing candle (OHLCV) data for 207 pairs
2021-09-07 14:49:56,431 - freqtrade.exchange.exchange - DEBUG - Using cached candle (OHLCV) data for pair CHR/USDT, timeframe 1d ...
2021-09-07 14:49:56,431 - freqtrade.exchange.exchange - DEBUG - Using cached candle (OHLCV) data for pair BTT/USDT, timeframe 5m ...
2021-09-07 14:49:56,432 - freqtrade.exchange.exchange - DEBUG - Using cached candle (OHLCV) data for pair LINK/USDT, timeframe 1h ...
.
.
.
```
Edit: Formatting Log



Discussion:
surely it'd be possible to change it - but it's currently not implemented - and is kindof working intended as is (although it was designed with StaticPairlist in mind - which means you can simply remove these pairs).


surely it'd be possible to change it - but it's currently not implemented - and is kindof working intended as is (although it was designed with StaticPairlist in mind - which means you can simply remove these pairs).



Resolving patch:
diff --git a/freqtrade/configuration/PeriodicCache.py b/freqtrade/configuration/PeriodicCache.py
new file mode 100644
--- /dev/null
+++ b/freqtrade/configuration/PeriodicCache.py
@@ -0,0 +1,19 @@
+from datetime import datetime, timezone
+
+from cachetools.ttl import TTLCache
+
+
+class PeriodicCache(TTLCache):
+    """
+    Special cache that expires at "straight" times
+    A timer with ttl of 3600 (1h) will expire at every full hour (:00).
+    """
+
+    def __init__(self, maxsize, ttl, getsizeof=None):
+        def local_timer():
+            ts = datetime.now(timezone.utc).timestamp()
+            offset = (ts % ttl)
+            return ts - offset
+
+        # Init with smlight offset
+        super().__init__(maxsize=maxsize, ttl=ttl-1e-5, timer=local_timer, getsizeof=getsizeof)
diff --git a/freqtrade/configuration/__init__.py b/freqtrade/configuration/__init__.py
--- a/freqtrade/configuration/__init__.py
+++ b/freqtrade/configuration/__init__.py
@@ -4,4 +4,5 @@
 from freqtrade.configuration.config_setup import setup_utils_configuration
 from freqtrade.configuration.config_validation import validate_config_consistency
 from freqtrade.configuration.configuration import Configuration
+from freqtrade.configuration.PeriodicCache import PeriodicCache
 from freqtrade.configuration.timerange import TimeRange
diff --git a/freqtrade/plugins/pairlist/AgeFilter.py b/freqtrade/plugins/pairlist/AgeFilter.py
--- a/freqtrade/plugins/pairlist/AgeFilter.py
+++ b/freqtrade/plugins/pairlist/AgeFilter.py
@@ -8,6 +8,7 @@
 import arrow
 from pandas import DataFrame
 
+from freqtrade.configuration import PeriodicCache
 from freqtrade.exceptions import OperationalException
 from freqtrade.misc import plural
 from freqtrade.plugins.pairlist.IPairList import IPairList
@@ -18,14 +19,15 @@
 
 class AgeFilter(IPairList):
 
-    # Checked symbols cache (dictionary of ticker symbol => timestamp)
-    _symbolsChecked: Dict[str, int] = {}
-
     def __init__(self, exchange, pairlistmanager,
                  config: Dict[str, Any], pairlistconfig: Dict[str, Any],
                  pairlist_pos: int) -> None:
         super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos)
 
+        # Checked symbols cache (dictionary of ticker symbol => timestamp)
+        self._symbolsChecked: Dict[str, int] = {}
+        self._symbolsCheckFailed = PeriodicCache(maxsize=1000, ttl=86_400)
+
         self._min_days_listed = pairlistconfig.get('min_days_listed', 10)
         self._max_days_listed = pairlistconfig.get('max_days_listed', None)
 
@@ -69,10 +71,13 @@ def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]:
         :param tickers: Tickers (from exchange.get_tickers()). May be cached.
         :return: new allowlist
         """
-        needed_pairs = [(p, '1d') for p in pairlist if p not in self._symbolsChecked]
+        needed_pairs = [
+            (p, '1d') for p in pairlist
+            if p not in self._symbolsChecked and p not in self._symbolsCheckFailed]
         if not needed_pairs:
-            return pairlist
-
+            # Remove pairs that have been removed before
+            return [p for p in pairlist if p not in self._symbolsCheckFailed]
+        logger.info(f"needed pairs {needed_pairs}")
         since_days = -(
             self._max_days_listed if self._max_days_listed else self._min_days_listed
         ) - 1
@@ -118,5 +123,6 @@ def _validate_pair_loc(self, pair: str, daily_candles: Optional[DataFrame]) -> b
                     " or more than "
                     f"{self._max_days_listed} {plural(self._max_days_listed, 'day')}"
                 ) if self._max_days_listed else ''), logger.info)
+                self._symbolsCheckFailed[pair] = arrow.utcnow().int_timestamp * 1000
                 return False
         return False


Test patch:
diff --git a/tests/plugins/test_pairlist.py b/tests/plugins/test_pairlist.py
--- a/tests/plugins/test_pairlist.py
+++ b/tests/plugins/test_pairlist.py
@@ -4,6 +4,7 @@
 from unittest.mock import MagicMock, PropertyMock
 
 import pytest
+import time_machine
 
 from freqtrade.constants import AVAILABLE_PAIRLISTS
 from freqtrade.exceptions import OperationalException
@@ -815,32 +816,63 @@ def test_agefilter_min_days_listed_too_large(mocker, default_conf, markets, tick
 
 
 def test_agefilter_caching(mocker, markets, whitelist_conf_agefilter, tickers, ohlcv_history):
-    ohlcv_data = {
-        ('ETH/BTC', '1d'): ohlcv_history,
-        ('TKN/BTC', '1d'): ohlcv_history,
-        ('LTC/BTC', '1d'): ohlcv_history,
-    }
-    mocker.patch.multiple('freqtrade.exchange.Exchange',
-                          markets=PropertyMock(return_value=markets),
-                          exchange_has=MagicMock(return_value=True),
-                          get_tickers=tickers
-                          )
-    mocker.patch.multiple(
-        'freqtrade.exchange.Exchange',
-        refresh_latest_ohlcv=MagicMock(return_value=ohlcv_data),
-    )
+    with time_machine.travel("2021-09-01 05:00:00 +00:00") as t:
+        ohlcv_data = {
+            ('ETH/BTC', '1d'): ohlcv_history,
+            ('TKN/BTC', '1d'): ohlcv_history,
+            ('LTC/BTC', '1d'): ohlcv_history,
+        }
+        mocker.patch.multiple(
+            'freqtrade.exchange.Exchange',
+            markets=PropertyMock(return_value=markets),
+            exchange_has=MagicMock(return_value=True),
+            get_tickers=tickers,
+            refresh_latest_ohlcv=MagicMock(return_value=ohlcv_data),
+        )
 
-    freqtrade = get_patched_freqtradebot(mocker, whitelist_conf_agefilter)
-    assert freqtrade.exchange.refresh_latest_ohlcv.call_count == 0
-    freqtrade.pairlists.refresh_pairlist()
-    assert len(freqtrade.pairlists.whitelist) == 3
-    assert freqtrade.exchange.refresh_latest_ohlcv.call_count > 0
+        freqtrade = get_patched_freqtradebot(mocker, whitelist_conf_agefilter)
+        assert freqtrade.exchange.refresh_latest_ohlcv.call_count == 0
+        freqtrade.pairlists.refresh_pairlist()
+        assert len(freqtrade.pairlists.whitelist) == 3
+        assert freqtrade.exchange.refresh_latest_ohlcv.call_count > 0
 
-    previous_call_count = freqtrade.exchange.refresh_latest_ohlcv.call_count
-    freqtrade.pairlists.refresh_pairlist()
-    assert len(freqtrade.pairlists.whitelist) == 3
-    # Called once for XRP/BTC
-    assert freqtrade.exchange.refresh_latest_ohlcv.call_count == previous_call_count + 1
+        freqtrade.pairlists.refresh_pairlist()
+        assert len(freqtrade.pairlists.whitelist) == 3
+        # Call to XRP/BTC cached
+        assert freqtrade.exchange.refresh_latest_ohlcv.call_count == 2
+
+        ohlcv_data = {
+            ('ETH/BTC', '1d'): ohlcv_history,
+            ('TKN/BTC', '1d'): ohlcv_history,
+            ('LTC/BTC', '1d'): ohlcv_history,
+            ('XRP/BTC', '1d'): ohlcv_history.iloc[[0]],
+        }
+        mocker.patch('freqtrade.exchange.Exchange.refresh_latest_ohlcv', return_value=ohlcv_data)
+        freqtrade.pairlists.refresh_pairlist()
+        assert len(freqtrade.pairlists.whitelist) == 3
+        assert freqtrade.exchange.refresh_latest_ohlcv.call_count == 1
+
+        # Move to next day
+        t.move_to("2021-09-02 01:00:00 +00:00")
+        mocker.patch('freqtrade.exchange.Exchange.refresh_latest_ohlcv', return_value=ohlcv_data)
+        freqtrade.pairlists.refresh_pairlist()
+        assert len(freqtrade.pairlists.whitelist) == 3
+        assert freqtrade.exchange.refresh_latest_ohlcv.call_count == 1
+
+        # Move another day with fresh mocks (now the pair is old enough)
+        t.move_to("2021-09-03 01:00:00 +00:00")
+        # Called once for XRP/BTC
+        ohlcv_data = {
+            ('ETH/BTC', '1d'): ohlcv_history,
+            ('TKN/BTC', '1d'): ohlcv_history,
+            ('LTC/BTC', '1d'): ohlcv_history,
+            ('XRP/BTC', '1d'): ohlcv_history,
+        }
+        mocker.patch('freqtrade.exchange.Exchange.refresh_latest_ohlcv', return_value=ohlcv_data)
+        freqtrade.pairlists.refresh_pairlist()
+        assert len(freqtrade.pairlists.whitelist) == 4
+        # Called once (only for XRP/BTC)
+        assert freqtrade.exchange.refresh_latest_ohlcv.call_count == 1
 
 
 def test_OffsetFilter_error(mocker, whitelist_conf) -> None:
diff --git a/tests/test_periodiccache.py b/tests/test_periodiccache.py
new file mode 100644
--- /dev/null
+++ b/tests/test_periodiccache.py
@@ -0,0 +1,32 @@
+import time_machine
+
+from freqtrade.configuration import PeriodicCache
+
+
+def test_ttl_cache():
+
+    with time_machine.travel("2021-09-01 05:00:00 +00:00") as t:
+
+        cache = PeriodicCache(5, ttl=60)
+        cache1h = PeriodicCache(5, ttl=3600)
+
+        assert cache.timer() == 1630472400.0
+        cache['a'] = 1235
+        cache1h['a'] = 555123
+        assert 'a' in cache
+        assert 'a' in cache1h
+
+        t.move_to("2021-09-01 05:00:59 +00:00")
+        assert 'a' in cache
+        assert 'a' in cache1h
+
+        # Cache expired
+        t.move_to("2021-09-01 05:01:00 +00:00")
+        assert 'a' not in cache
+        assert 'a' in cache1h
+
+        t.move_to("2021-09-01 05:59:59 +00:00")
+        assert 'a' in cache1h
+
+        t.move_to("2021-09-01 06:00:00 +00:00")
+        assert 'a' not in cache1h