Repository: freqtrade/freqtrade

Issue:
FTX - no errors using download-data but data does not make sense
I've been downloading data from FTX spot markets, specifically the leveraged tokens for backtesting;

Note that **FTX orderbooks only exist since sometime may or june 2019.** 
This is what i get: 

```
2020-02-13 18:14:44,189 - freqtrade.exchange.exchange - INFO - Using Exchange "FTX"
2020-02-13 18:14:45,215 - freqtrade.data.history - INFO - Downloading pair BULL/USD, interval 1h.
2020-02-13 18:14:45,215 - freqtrade.data.history - INFO - Download history data for pair: "BULL/USD", timeframe: 1h and store in /home/dante/freqtrade/user_data/data/ftx.
2020-02-13 18:14:45,215 - freqtrade.misc - DEBUG - Loading ticker data from file /home/dante/freqtrade/user_data/data/ftx/BULL_USD-1h.json
2020-02-13 18:14:45,342 - freqtrade.data.history - DEBUG - Current Start: None
2020-02-13 18:14:45,342 - freqtrade.data.history - DEBUG - Current End: None
2020-02-13 18:14:45,343 - freqtrade.exchange.exchange - DEBUG - one_call: 1800000000 msecs (2 weeks)
2020-02-13 18:14:45,343 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1519344000000 (2018-02-23T00:00:00+00:00) ...
2020-02-13 18:14:45,343 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1521144000000 (2018-03-15T20:00:00+00:00) ...
2020-02-13 18:14:45,343 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1522944000000 (2018-04-05T16:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1524744000000 (2018-04-26T12:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1526544000000 (2018-05-17T08:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1528344000000 (2018-06-07T04:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1530144000000 (2018-06-28T00:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1531944000000 (2018-07-18T20:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1533744000000 (2018-08-08T16:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1535544000000 (2018-08-29T12:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1537344000000 (2018-09-19T08:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1539144000000 (2018-10-10T04:00:00+00:00) ...
2020-02-13 18:14:45,344 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1540944000000 (2018-10-31T00:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1542744000000 (2018-11-20T20:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1544544000000 (2018-12-11T16:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1546344000000 (2019-01-01T12:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1548144000000 (2019-01-22T08:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1549944000000 (2019-02-12T04:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1551744000000 (2019-03-05T00:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1553544000000 (2019-03-25T20:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1555344000000 (2019-04-15T16:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1557144000000 (2019-05-06T12:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1558944000000 (2019-05-27T08:00:00+00:00) ...
2020-02-13 18:14:45,345 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1560744000000 (2019-06-17T04:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1562544000000 (2019-07-08T00:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1564344000000 (2019-07-28T20:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1566144000000 (2019-08-18T16:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1567944000000 (2019-09-08T12:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1569744000000 (2019-09-29T08:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1571544000000 (2019-10-20T04:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1573344000000 (2019-11-10T00:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1575144000000 (2019-11-30T20:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1576944000000 (2019-12-21T16:00:00+00:00) ...
2020-02-13 18:14:45,346 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1578744000000 (2020-01-11T12:00:00+00:00) ...
2020-02-13 18:14:45,347 - freqtrade.exchange.exchange - DEBUG - Fetching pair BULL/USD, interval 1h, since 1580544000000 (2020-02-01T08:00:00+00:00) ...
```
**NOTE, i tried uploading the JSON file but it was not supported by git, find attached as .txt file**

[BULL_USD-1h.txt](https://github.com/freqtrade/freqtrade/files/4200255/BULL_USD-1h.txt)




Discussion:
well then you should only provide it with a `--days` option to get back to june - that'll speed it up - since we do try it from the begining  (freqtrade does not know when the data did start).

Other than that - i think your data starts mid-december ... which to me suggests one of 2 things - either the API does not give you more history than a few 1000 datapoints (kraken handles it that way) - or ccxt has yet another bug.

Now for freqtrade, you can use `--dl-trades` - which will give you the full history for kraken - and i assume should work for FTX too. Note that when using `--dl-trades` it'll download all trades - which can amass to multiple G if done for a long enough period of time.

Now i'll try to do some digging to find where the problem is - but my best bet is currently on the api being limited in the amount of history it provides
Doing some (very quick) digging into the [OHLCV endpoint](https://docs.ftx.com/#get-historical-prices), it seems like the API does require an end-time to work correctly.

using the following url
`https://ftx.com/api/markets/BTC/USD/candles?resolution=300&start_time=1564332783&end_time=1566332783&limit=20`

i get data from 2019-08-20T20:25:00+00:00 - 2019-08-20T18:50:00+00:00 - that's 20 candles from the upper timelimit (end_time).

Now if i omit the end-time - 
`https://ftx.com/api/markets/BTC/USD/candles?resolution=300&start_time=1564332783&limit=20`
i get data from `2020-02-13T17:35:00+00:00` to `2020-02-13T19:10:00+00:00` - it's basically using "current date" as the upper limit.


Now while this could probably be fixed in ccxt - i think the issue here is either with the API documentation (which does not state this behaviour at all) - or with the API itself - which does ignore start_time completely.

If this is intended, we can report it against ccxt - where there should be a fix to it (i've had this before iirc).
However the documentation is wrong insofar as the start-time parameter is completely irrelevant.



same as for bitMEX (end-time of the candle)? (remember my old question regarding this? I think I have year-old changes in the stashes...)

PS: as I recall, I made an issue or even a PR to ccxt for bitmex to allow it accept candle times properly

PPS: similar to this: https://github.com/ccxt/ccxt/pull/5520 ?


"they only provide the last 1500 entries. If my since is in this range, it works and reduces the entries and begins with the correct time."
https://github.com/ccxt/ccxt/pull/6113#issuecomment-553951051
(that's for fetch_trades, which also has same behavior as I understood it)
Yes that's true - but that makes end-time a required argument in the API - which if not proivided defaults to current time - and open_time not. 
The documentation does not show this at all ... so i guess the expectation that it "may" be the other way around is valid
@xmatthias, FTX dev here: we can upate the docs, but it's intended that "it's basically using "current date" as the upper limit."
I can update the docs, and someone should bump CCXT about it. Thanks!
yeah, having the docs updated / clarified would definitely help.

I can report it to ccxt if you want ... 


Resolving patch:
diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py
--- a/freqtrade/exchange/__init__.py
+++ b/freqtrade/exchange/__init__.py
@@ -1,18 +1,20 @@
-from freqtrade.exchange.common import MAP_EXCHANGE_CHILDCLASS  # noqa: F401
-from freqtrade.exchange.exchange import Exchange  # noqa: F401
-from freqtrade.exchange.exchange import (get_exchange_bad_reason,  # noqa: F401
+# flake8: noqa: F401
+from freqtrade.exchange.common import MAP_EXCHANGE_CHILDCLASS
+from freqtrade.exchange.exchange import Exchange
+from freqtrade.exchange.exchange import (get_exchange_bad_reason,
                                          is_exchange_bad,
                                          is_exchange_known_ccxt,
                                          is_exchange_officially_supported,
                                          ccxt_exchanges,
                                          available_exchanges)
-from freqtrade.exchange.exchange import (timeframe_to_seconds,  # noqa: F401
+from freqtrade.exchange.exchange import (timeframe_to_seconds,
                                          timeframe_to_minutes,
                                          timeframe_to_msecs,
                                          timeframe_to_next_date,
                                          timeframe_to_prev_date)
-from freqtrade.exchange.exchange import (market_is_active,  # noqa: F401
+from freqtrade.exchange.exchange import (market_is_active,
                                          symbol_is_pair)
-from freqtrade.exchange.kraken import Kraken  # noqa: F401
-from freqtrade.exchange.binance import Binance  # noqa: F401
-from freqtrade.exchange.bibox import Bibox  # noqa: F401
+from freqtrade.exchange.kraken import Kraken
+from freqtrade.exchange.binance import Binance
+from freqtrade.exchange.bibox import Bibox
+from freqtrade.exchange.ftx import Ftx
diff --git a/freqtrade/exchange/ftx.py b/freqtrade/exchange/ftx.py
new file mode 100644
--- /dev/null
+++ b/freqtrade/exchange/ftx.py
@@ -0,0 +1,14 @@
+""" FTX exchange subclass """
+import logging
+from typing import Dict
+
+from freqtrade.exchange import Exchange
+
+logger = logging.getLogger(__name__)
+
+
+class Ftx(Exchange):
+
+    _ft_has: Dict = {
+        "ohlcv_candle_limit": 1500,
+    }


Test patch: