Repository: freqtrade/freqtrade

Issue:
specifying buy signals timeouts during live
An option for configuring signals timeouts is useful for longer timeframes.
1. You have open orders
2. a sell is triggered at 3/4 of the timespan of timeframe
3. the bot processed a buy signals, and after the sell executed a new buy order, but that signal might be old already and not worth being honored.

This is true also when the bot is started with initial state running which of course can be worked around by starting stopped state and running it only after new candles are fetched.

This can also be specified in the strategy itself by guarding against time deltas.


Discussion:
There's 2 parts to this, both together should suffice your points:

* After a sell, that pair is locked for this candle - so the same pair will not be bought within this candle (it could be bought at the next candle though).
* You can have the bot analyze buy / sell signals only once new candles arrive with 
 `process_only_new_candles`.
This will prevent buy-signals "intra-candle" - which can happen if a slot is opened after "half a candle". ROI and Stoploss are still processed once every iteration (5s - by default).

Its independent of the timeframe, since the same can be problematic for short timeframes too.

In my opinion, you are right that the "signal might be old already and not worth being honored."

However, the bot assumes that the user is smarter than the bot itself and if you issue the buy/sell signal from the strategy, this is still active and should be handled. Since only you know when to sell and when to buy.

This can be mitigated by using the `crossed_above/crossed_below` functions in the strategy, but they are not always easily applicable. And simple usage of comparisons in the strategy, yes, often creates the signals that are old, outdated and should nor be taken.

I've been thinking about this for long time and I can propose the following solution:

I can add a new helper function (presumably into freqtrade/technical), say, named `signal_delta()`, which will produce True on the heading front of the trading signal. For example, if we have the `buy` column as
```
False
False
True
True
True
True
True
False
```
`signal_delta(dataframe['buy'])` will result in
```
False
False
True
False
False
False
False
False
```
leaving, as `crossed_above/crossed_below` only one candle with True, where the signal was actually raised first time.

Then, a new helper function named, say, `signal_trail(N)` can be added, which will prolong the signal for N next candles.

So the usage of those together can be
```
dataframe['buy'] = signal_trail(signal_delta(dataframe['buy']), 3)
```
and/or
```
signal_delta(dataframe['buy'], inplace=True)
signal_trail(dataframe['buy'], 3, inplace=True)
```
which will produce 3 candles with the buy signal, removing the outdated signals where the signal is old:
```
False
False
True
True
True
False
False
False
```

Thus, no changes into the bot trading logic is required.

@xmatthias @orehunt what do you think?

i think you misunderstood the issue .. it's about intra-candle.

say i use 1h candles, and get a buy-signal at 8:00am
that signal will "remain there" until 8:59 am even with `crossed_above()` - since this is based on the calculation done for the previous candle.

Now if my bot sells another pair at 8:30 - it'll buy this pair at 8:31 - which is wrong (the signal is half an hour old).
The only mitigation for this is `process_only_new_candles`.

yes @xmatthias  is correct, I am referring to the actual signals that trigger buy trades. I did think about `process_only_new_candles` however I was not sure how are signals stored or queued buy the bot. 

If signals are produced, but no more trades can be opened (be it either because of max number of open trades, or not enough balance), are they all discarded? Or queued? If the bot discards all signals if it can't open new trades then it helps. 

However that is also too extreme, because maybe you are fine opening trades for signals not older then half the current timeframe.
By default, the bot analyzes the dataframe for candles on each iteration (~ every 5s).
The dataframe is "reset" between iterations.

with `process_only_new_candles` this is not the case, and it'll only analyze when a new candle arrives.

> However that is also too extreme, because maybe you are fine opening trades for signals not older then half the current timeframe.

in which case you need to handle this within the strategy - which shouldn't be too hard

something like this should do it 
```python
def is_buy_timeout():
    if (self.config['runmode'] in [RunMode.LIVE, RunMode.DRY_RUN]):
        time_buffer = Timedelta(self.config['ticker_interval']) / 2
        timeout = d.loc[d.index.max(), 'date'] + time_buffer
        return (timeout <= datetime.now(timeout.tzinfo))
    else:
        return False

if is_buy_timeout():
    d.loc['buy'] = 0
    return d
```
Experimenting with a couple of helper functions I wrote (they could probably be placed into technical):

```
def rising(series):
    return series.where(series.diff() > 0, 0.0)

def decay(series, n):
    return series.rolling(n, min_periods=1).max()
```

Usage (in `populate_buy_trend()` just before return):
```
        dataframe['buy'] = rising(dataframe['buy'])
```
and
```
        dataframe['buy'] = decay(rising(dataframe['buy']), 5)
```

First removes the long rows of buy signals and leave only first one. This is the replacement for crossed_above/below. Sometimes, when the conditions are tricky, it's difficult to understand, where crossed_above/below should be used. The `rising()` simply leaves first candle in each long row of signals.

Second one, adds a trail, a decay of 5 candles after that first signal, prolonging it for 5 next candles.

@orehunt is not it what you would like to achieve (buy signal active for only N candles since its first appearance...)?


no it is not...

Assume a 1h candle - he'd like the buy-signal using `crossed*()` at 1am- and then the buy-signal should stay "active" until 1:30 ... which has nothing to do with your solution...

@orehunt 
i think your solution works perfectly fine.

an alternative would be to use "half" candles.
so if you wanna trade at 1h candles with 30min timeout, you use 30min candles - and resample the dataframe to 1h for buy-signals (filling NA buy-signal with 0 after resampling). 
That'll give you a similar result (although limiting data history to half of the interval you'd get from 1h candles).
> no it is not...
> 
> Assume a 1h candle - he'd like the buy-signal using `crossed*()` at 1am- and then the buy-signal should stay "active" until 1:30 ... which has nothing to do with your solution...

yeah, got it
I read above in the details...
Now I do not understand the points in question of this issue. The pair is locked after buy till the end of the candle. Even without the use of `process_only_new_candles`.

So the signal can only be 'too old' at the next candles if the buy column contains the long row of the signals. This can easily be eliminated by the use of `crossed_above` family of functions or with the helper functions I stated above...

@hroff-1902 
- At time 1:00 you fill all your max trades (lets say 3)
- at time 1:35 a sell is executed, now you have one trade slot available
- If the strategy has more than 3 buy signals for the current period, there will be another buy trade executed for another ticker at 1:35 after the tickers are analyzed, of course it won't be the locked one, but others will
Exactly - and since we want buys to happen until 1:30 (assuming signal is still fresh) we cannot use `process_only_new_candles`.

I do think that the code posted before (https://github.com/freqtrade/freqtrade/issues/2423#issuecomment-546599603) will accomplish this.
However, i think this should be part of the strategy - not of freqtrade core code - since it appears to be very specific.
okay, I got the example,


should not `process_only_new_candles` help with this, preventing buys at 1:35? And using `crossed_above`, to make the signal at only first candle, preventing rows of signals?


Is this something that we should document / include as base functionality, or are the above snippets sufficient to get this achived?

If any, i can imagine a config option where you can specify "buy-signal valid for 10% of the candle" ... which makes it independent on the timeframe used.


the intention for this is prevent stale signals to execute trades, a better option could be:
don't buy if price is plus/minus the open price of the current (as in incomplete) candle

Deciding what is a good timeout is harder since it is dependent on liquidity, and the timeframe used, whereas a distance from the open price is easier to reason with and gives a simpler notion of the incurred risk compared to gauging risk from time/liquidity which can be unclear and blurry

Still, this could also be a reusable strategy component (many things could be reusable strategy components :P) and not a config option
Well you can do this calculation (decided price vs. candle open price) in the `confirm_trade_entry` [callback](https://www.freqtrade.io/en/latest/strategy-advanced/#trade-entry-buy-order-confirmation), so maybe that's a better way to do this than to use a time-based component (?) 


Resolving patch:
diff --git a/freqtrade/resolvers/strategy_resolver.py b/freqtrade/resolvers/strategy_resolver.py
--- a/freqtrade/resolvers/strategy_resolver.py
+++ b/freqtrade/resolvers/strategy_resolver.py
@@ -79,6 +79,7 @@ def load_strategy(config: Dict[str, Any] = None) -> IStrategy:
                       ("sell_profit_only",                False,       'ask_strategy'),
                       ("ignore_roi_if_buy_signal",        False,       'ask_strategy'),
                       ("disable_dataframe_checks",        False,       None),
+                      ("ignore_buying_expired_candle_after",  0,       'ask_strategy')
                       ]
         for attribute, default, subkey in attributes:
             if subkey:
diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py
--- a/freqtrade/strategy/interface.py
+++ b/freqtrade/strategy/interface.py
@@ -5,7 +5,7 @@
 import logging
 import warnings
 from abc import ABC, abstractmethod
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 from enum import Enum
 from typing import Dict, List, NamedTuple, Optional, Tuple
 
@@ -15,7 +15,7 @@
 from freqtrade.constants import ListPairsWithTimeframes
 from freqtrade.data.dataprovider import DataProvider
 from freqtrade.exceptions import OperationalException, StrategyError
-from freqtrade.exchange import timeframe_to_minutes
+from freqtrade.exchange import timeframe_to_minutes, timeframe_to_seconds
 from freqtrade.exchange.exchange import timeframe_to_next_date
 from freqtrade.persistence import PairLocks, Trade
 from freqtrade.strategy.strategy_wrapper import strategy_safe_wrapper
@@ -113,6 +113,9 @@ class IStrategy(ABC):
     # run "populate_indicators" only for new candle
     process_only_new_candles: bool = False
 
+    # Number of seconds after which the candle will no longer result in a buy on expired candles
+    ignore_buying_expired_candle_after: int = 0
+
     # Disable checking the dataframe (converts the error into a warning message)
     disable_dataframe_checks: bool = False
 
@@ -476,8 +479,22 @@ def get_signal(self, pair: str, timeframe: str, dataframe: DataFrame) -> Tuple[b
         (buy, sell) = latest[SignalType.BUY.value] == 1, latest[SignalType.SELL.value] == 1
         logger.debug('trigger: %s (pair=%s) buy=%s sell=%s',
                      latest['date'], pair, str(buy), str(sell))
+        timeframe_seconds = timeframe_to_seconds(timeframe)
+        if self.ignore_expired_candle(latest_date=latest_date,
+                                      current_time=datetime.now(timezone.utc),
+                                      timeframe_seconds=timeframe_seconds,
+                                      buy=buy):
+            return False, sell
         return buy, sell
 
+    def ignore_expired_candle(self, latest_date: datetime, current_time: datetime,
+                              timeframe_seconds: int, buy: bool):
+        if self.ignore_buying_expired_candle_after and buy:
+            time_delta = current_time - (latest_date + timedelta(seconds=timeframe_seconds))
+            return time_delta.total_seconds() > self.ignore_buying_expired_candle_after
+        else:
+            return False
+
     def should_sell(self, trade: Trade, rate: float, date: datetime, buy: bool,
                     sell: bool, low: float = None, high: float = None,
                     force_stoploss: float = 0) -> SellCheckTuple:
@@ -672,6 +689,7 @@ def advise_buy(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
         :return: DataFrame with buy column
         """
         logger.debug(f"Populating buy signals for pair {metadata.get('pair')}.")
+
         if self._buy_fun_len == 2:
             warnings.warn("deprecated - check out the Sample strategy to see "
                           "the current function headers!", DeprecationWarning)


Test patch:
diff --git a/tests/strategy/test_interface.py b/tests/strategy/test_interface.py
--- a/tests/strategy/test_interface.py
+++ b/tests/strategy/test_interface.py
@@ -106,6 +106,28 @@ def test_get_signal_old_dataframe(default_conf, mocker, caplog, ohlcv_history):
     assert log_has('Outdated history for pair xyz. Last tick is 16 minutes old', caplog)
 
 
+def test_ignore_expired_candle(default_conf):
+    default_conf.update({'strategy': 'DefaultStrategy'})
+    strategy = StrategyResolver.load_strategy(default_conf)
+    strategy.ignore_buying_expired_candle_after = 60
+
+    latest_date = datetime(2020, 12, 30, 7, 0, 0, tzinfo=timezone.utc)
+    # Add 1 candle length as the "latest date" defines candle open.
+    current_time = latest_date + timedelta(seconds=80 + 300)
+
+    assert strategy.ignore_expired_candle(latest_date=latest_date,
+                                          current_time=current_time,
+                                          timeframe_seconds=300,
+                                          buy=True) is True
+
+    current_time = latest_date + timedelta(seconds=30 + 300)
+
+    assert not strategy.ignore_expired_candle(latest_date=latest_date,
+                                              current_time=current_time,
+                                              timeframe_seconds=300,
+                                              buy=True) is True
+
+
 def test_assert_df_raise(mocker, caplog, ohlcv_history):
     ohlcv_history.loc[1, 'date'] = arrow.utcnow().shift(minutes=-16)
     # Take a copy to correctly modify the call