Repository: freqtrade/freqtrade
Pull request: Ignoring candles that have expired within timeframe
Signed-off-by: hoeckxer <hawkeyenl@yahoo.com>

Thank you for sending your pull request. But first, have you included
unit tests, and is your code PEP8 conformant? [More details](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md)

## Summary
With this implementation, functionality is added to ignore buy signals on candles during ticks while the candle hasn't refreshed yet.

closes #2423

## Quick changelog

- Added documentation
- Added test (if required I can add a test for get_signal as well)
- Added functionality ignore a buy signal when the candle is older than X seconds

## What's new?
When working with a low number of max trades and a higher timeframe, you can run into the situation that a trade slot becomes available, and buys into the last candle signal, which may not be valid (and therefore desired) anymore. This PR allows you to ignore buy signals on a candle if it's older than X (configurable) number of seconds


Problem statement:
specifying buy signals timeouts during live
An option for configuring signals timeouts is useful for longer timeframes.
1. You have open orders
2. a sell is triggered at 3/4 of the timespan of timeframe
3. the bot processed a buy signals, and after the sell executed a new buy order, but that signal might be old already and not worth being honored.

This is true also when the bot is started with initial state running which of course can be worked around by starting stopped state and running it only after new candles are fetched.

This can also be specified in the strategy itself by guarding against time deltas.

Hints from issue discussion:
There's 2 parts to this, both together should suffice your points:

* After a sell, that pair is locked for this candle - so the same pair will not be bought within this candle (it could be bought at the next candle though).
* You can have the bot analyze buy / sell signals only once new candles arrive with 
 `process_only_new_candles`.
This will prevent buy-signals "intra-candle" - which can happen if a slot is opened after "half a candle". ROI and Stoploss are still processed once every iteration (5s - by default).

Its independent of the timeframe, since the same can be problematic for short timeframes too.

In my opinion, you are right that the "signal might be old already and not worth being honored."

However, the bot assumes that the user is smarter than the bot itself and if you issue the buy/sell signal from the strategy, this is still active and should be handled. Since only you know when to sell and when to buy.

This can be mitigated by using the `crossed_above/crossed_below` functions in the strategy, but they are not always easily applicable. And simple usage of comparisons in the strategy, yes, often creates the signals that are old, outdated and should nor be taken.

I've been thinking about this for long time and I can propose the following solution:

I can add a new helper function (presumably into freqtrade/technical), say, named `signal_delta()`, which will produce True on the heading front of the trading signal. For example, if we have the `buy` column as
```
False
False
True
True
True
True
True
False
```
`signal_delta(dataframe['buy'])` will result in
```
False
False
True
False
False
False
False
False
```
leaving, as `crossed_above/crossed_below` only one candle with True, where the signal was actually raised first time.

Then, a new helper function named, say, `signal_trail(N)` can be added, which will prolong the signal for N next candles.

So the usage of those together can be
```
dataframe['buy'] = signal_trail(signal_delta(dataframe['buy']), 3)
```
and/or
```
signal_delta(dataframe['buy'], inplace=True)
signal_trail(dataframe['buy'], 3, inplace=True)
```
which will produce 3 candles with the buy signal, removing the outdated signals where the signal is old:
```
False
False
True
True
True
False
False
False
```

Thus, no changes into the bot trading logic is required.

@xmatthias @orehunt what do you think?

i think you misunderstood the issue .. it's about intra-candle.

say i use 1h candles, and get a buy-signal at 8:00am
that signal will "remain there" until 8:59 am even with `crossed_above()` - since this is based on the calculation done for the previous candle.

Now if my bot sells another pair at 8:30 - it'll buy this pair at 8:31 - which is wrong (the signal is half an hour old).
The only mitigation for this is `process_only_new_candles`.

yes @xmatthias  is correct, I am referring to the actual signals that trigger buy trades. I did think about `process_only_new_candles` however I was not sure how are signals stored or queued buy the bot. 

If signals are produced, but no more trades can be opened (be it either because of max number of open trades, or not enough balance), are they all discarded? Or queued? If the bot discards all signals if it can't open new trades then it helps. 

However that is also too extreme, because maybe you are fine opening trades for signals not older then half the current timeframe.
By default, the bot analyzes the dataframe for candles on each iteration (~ every 5s).
The dataframe is "reset" between iterations.

with `process_only_new_candles` this is not the case, and it'll only analyze when a new candle arrives.

> However that is also too extreme, because maybe you are fine opening trades for signals not older then half the current timeframe.

in which case you need to handle this within the strategy - which shouldn't be too hard

something like this should do it 
```python
def is_buy_timeout():
    if (self.config['runmode'] in [RunMode.LIVE, RunMode.DRY_RUN]):
        time_buffer = Timedelta(self.config['ticker_interval']) / 2
        timeout = d.loc[d.index.max(), 'date'] + time_buffer
        return (timeout <= datetime.now(timeout.tzinfo))
    else:
        return False

if is_buy_timeout():
    d.loc['buy'] = 0
    return d
```
Experimenting with a couple of helper functions I wrote (they could probably be placed into technical):

```
def rising(series):
    return series.where(series.diff() > 0, 0.0)

def decay(series, n):
    return series.rolling(n, min_periods=1).max()
```

Usage (in `populate_buy_trend()` just before return):
```
        dataframe['buy'] = rising(dataframe['buy'])
```
and
```
        dataframe['buy'] = decay(rising(dataframe['buy']), 5)
```

First removes the long rows of buy signals and leave only first one. This is the replacement for crossed_above/below. Sometimes, when the conditions are tricky, it's difficult to understand, where crossed_above/below should be used. The `rising()` simply leaves first candle in each long row of signals.

Second one, adds a trail, a decay of 5 candles after that first signal, prolonging it for 5 next candles.

@orehunt is not it what you would like to achieve (buy signal active for only N candles since its first appearance...)?


no it is not...

Assume a 1h candle - he'd like the buy-signal using `crossed*()` at 1am- and then the buy-signal should stay "active" until 1:30 ... which has nothing to do with your solution...

@orehunt 
i think your solution works perfectly fine.

an alternative would be to use "half" candles.
so if you wanna trade at 1h candles with 30min timeout, you use 30min candles - and resample the dataframe to 1h for buy-signals (filling NA buy-signal with 0 after resampling). 
That'll give you a similar result (although limiting data history to half of the interval you'd get from 1h candles).
> no it is not...
> 
> Assume a 1h candle - he'd like the buy-signal using `crossed*()` at 1am- and then the buy-signal should stay "active" until 1:30 ... which has nothing to do with your solution...

yeah, got it
I read above in the details...
Now I do not understand the points in question of this issue. The pair is locked after buy till the end of the candle. Even without the use of `process_only_new_candles`.

So the signal can only be 'too old' at the next candles if the buy column contains the long row of the signals. This can easily be eliminated by the use of `crossed_above` family of functions or with the helper functions I stated above...

Commit message: Simplified return statements

Signed-off-by: hoeckxer <hawkeyenl@yahoo.com>

Patch to review:
diff --git a/docs/configuration.md b/docs/configuration.md
index b70a85c0496..db078cba878 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -121,6 +121,8 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `user_data_dir` | Directory containing user data. <br> *Defaults to `./user_data/`*. <br> **Datatype:** String
 | `dataformat_ohlcv` | Data format to use to store historical candle (OHLCV) data. <br> *Defaults to `json`*. <br> **Datatype:** String
 | `dataformat_trades` | Data format to use to store historical trades data. <br> *Defaults to `jsongz`*. <br> **Datatype:** String
+| `ignore_buying_expired_candle` | Enables usage of skipping buys on candles that are older than a specified period. <br>*Defaults to `False`* <br> **Datatype:** Boolean 
+| `ignore_buying_expired_candle_after` | Specifies the number of seconds until a buy signal is no longer used when setting `ignore_buying_expired_candle`. <br> **Datatype:** Integer
 
 ### Parameters in the strategy
 
@@ -144,6 +146,8 @@ Values set in the configuration file always overwrite values set in the strategy
 * `use_sell_signal` (ask_strategy)
 * `sell_profit_only` (ask_strategy)
 * `ignore_roi_if_buy_signal` (ask_strategy)
+* `ignore_buying_expired_candle`
+* `ignore_buying_expired_candle_after`
 
 ### Configuring amount per trade
 
@@ -671,6 +675,19 @@ export HTTPS_PROXY="http://addr:port"
 freqtrade
 ```
 
+## Ignoring expired candles
+
+When working with larger timeframes (for example 1h or more) and using a low `max_trades` value, the last candle can be processed as soon as a trade slot becomes available. When processing the last candle, this can lead to a situation where it may not be desirable to use the buy signal on that candle. For example, when using a condition in your strategy where you use a cross-over, that point may have passed too long ago for you to start a trade on it.
+
+In these situations, you can enable the functionality to ignore candles that are beyond a specified period by setting `ignore_buying_expired_candle` to `True`. After this, you can set `ignore_buying_expired_candle_after` to the number of seconds after which the candle becomes expired.
+
+For example, if your strategy is using a 1h timeframe, and you only want to buy within the first 5 minutes when a new candle comes in, you can add the following configuration to your strategy:
+
+``` json
+ignore_buying_expired_candle = True
+ignore_buying_expired_candle_after = 300 # 5 minutes
+```
+
 ## Embedding Strategies
 
 Freqtrade provides you with with an easy way to embed the strategy into your configuration file.
diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py
index 1e4fc8b126e..a1da3681dd8 100644
--- a/freqtrade/strategy/interface.py
+++ b/freqtrade/strategy/interface.py
@@ -5,7 +5,7 @@
 import logging
 import warnings
 from abc import ABC, abstractmethod
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 from enum import Enum
 from typing import Dict, List, NamedTuple, Optional, Tuple
 
@@ -113,6 +113,11 @@ class IStrategy(ABC):
     # run "populate_indicators" only for new candle
     process_only_new_candles: bool = False
 
+    # Don't buy on expired candles
+    ignore_buying_expired_candle: bool = False
+    # Number of seconds after which the candle will no longer result in a buy
+    ignore_buying_expired_candle_after: int = 0
+
     # Disable checking the dataframe (converts the error into a warning message)
     disable_dataframe_checks: bool = False
 
@@ -476,8 +481,20 @@ def get_signal(self, pair: str, timeframe: str, dataframe: DataFrame) -> Tuple[b
         (buy, sell) = latest[SignalType.BUY.value] == 1, latest[SignalType.SELL.value] == 1
         logger.debug('trigger: %s (pair=%s) buy=%s sell=%s',
                      latest['date'], pair, str(buy), str(sell))
+        if self.ignore_expired_candle(dataframe=dataframe, buy=buy):
+            return False, sell
         return buy, sell
 
+    def ignore_expired_candle(self, dataframe: DataFrame, buy: bool):
+        if self.ignore_buying_expired_candle and buy:
+            current_time = datetime.now(timezone.utc) - timedelta(
+                seconds=self.ignore_buying_expired_candle_after)
+            candle_time = dataframe['date'].tail(1).iat[0]
+            time_delta = current_time - candle_time
+            return time_delta.total_seconds() > self.ignore_buying_expired_candle_after
+        else:
+            return False
+
     def should_sell(self, trade: Trade, rate: float, date: datetime, buy: bool,
                     sell: bool, low: float = None, high: float = None,
                     force_stoploss: float = 0) -> SellCheckTuple:
@@ -672,6 +689,7 @@ def advise_buy(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
         :return: DataFrame with buy column
         """
         logger.debug(f"Populating buy signals for pair {metadata.get('pair')}.")
+
         if self._buy_fun_len == 2:
             warnings.warn("deprecated - check out the Sample strategy to see "
                           "the current function headers!", DeprecationWarning)
diff --git a/tests/strategy/test_interface.py b/tests/strategy/test_interface.py
index 7eed433025f..f389be45b1e 100644
--- a/tests/strategy/test_interface.py
+++ b/tests/strategy/test_interface.py
@@ -106,6 +106,23 @@ def test_get_signal_old_dataframe(default_conf, mocker, caplog, ohlcv_history):
     assert log_has('Outdated history for pair xyz. Last tick is 16 minutes old', caplog)
 
 
+def test_ignore_expired_candle(default_conf, ohlcv_history):
+    default_conf.update({'strategy': 'DefaultStrategy'})
+    strategy = StrategyResolver.load_strategy(default_conf)
+    strategy.ignore_buying_expired_candle = True
+    strategy.ignore_buying_expired_candle_after = 60
+
+    ohlcv_history.loc[-1, 'date'] = arrow.utcnow().shift(minutes=-3)
+    # Take a copy to correctly modify the call
+    mocked_history = ohlcv_history.copy()
+    mocked_history['sell'] = 0
+    mocked_history['buy'] = 0
+    mocked_history.loc[1, 'buy'] = 1
+    mocked_history.loc[1, 'sell'] = 1
+
+    assert strategy.ignore_expired_candle(mocked_history, True) is True
+
+
 def test_assert_df_raise(mocker, caplog, ohlcv_history):
     ohlcv_history.loc[1, 'date'] = arrow.utcnow().shift(minutes=-16)
     # Take a copy to correctly modify the call


Code review comments:
diff_hunk: @@ -144,6 +146,8 @@ Values set in the configuration file always overwrite values set in the strategy
 * `use_sell_signal` (ask_strategy)
 * `sell_profit_only` (ask_strategy)
 * `ignore_roi_if_buy_signal` (ask_strategy)
+* `ignore_buying_expired_candle`
+* `ignore_buying_expired_candle_after`
path: docs/configuration.md
text: @user1:
should we allow this to be set in the configuration as well (otherwise this is the wrong place to document these settings).

If the answer is yes, then you must add them to the attributes in `strategy_resolver.py`.

@author:
My idea would be that this is closely tied to the timeframe, as it has a (somewhat) direct relationship with it. The timeframe is something that belongs with the strategy I believe so that's the reason I put it here. Do you think it should be in the configuration @user1 ?

@user1:
while timeframe CAN be set by the strategy, you can also have it in the configuration - which will then override what you had in the strategy.
I don't use it myself, but having it in the configuration is the default for all sample configurations iirc.

so while timeframe CAN come from the strategy, it can also come from the strategy - so allowing it in both places might be a consequence.


@author:
I moved them to strategy_resolver.py, but I'm not sure I did it right, so please check and let me know if it needs to be done differently!

diff_hunk: @@ -476,8 +481,20 @@ def get_signal(self, pair: str, timeframe: str, dataframe: DataFrame) -> Tuple[b
         (buy, sell) = latest[SignalType.BUY.value] == 1, latest[SignalType.SELL.value] == 1
         logger.debug('trigger: %s (pair=%s) buy=%s sell=%s',
                      latest['date'], pair, str(buy), str(sell))
+        if self.ignore_expired_candle(dataframe=dataframe, buy=buy):
path: freqtrade/strategy/interface.py
text: @user1:
Consider passing `latest` instead of the full dataframe - it'll contain the last row (the row that's the base for the buy signal) - but it's not calculated via `.tail()` - so allows reordering of the dataframe by the strategy (which i'd advise noone to do - but we're protected from it).

You will also need `timeframe_minutes` in the function (see comment on the function for an explanation why).

@author:
Correct, I adjusted the code due to these new insights, thanks!