Repository: freqtrade/freqtrade

Issue:
binance sell order not working because of insufficient fund 
## Step 1: Have you search for this issue before posting it?

If you have discovered a bug in the bot, please [search our issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue). 
If it hasn't been reported, please create a new issue.

## Step 2: Describe your environment

  * Operating system: linux ubuntu 18.04
  * Python Version: Python 3.6.8
  * CCXT version: ccxt==1.19.14
  * Branch: Develop
  * Last Commit ID: ad2289c34cc938bc8b529f2676e71555fb216faa 
## Step 3: Describe the problem:

The order amount in DB is different from the actual amount in my binance account. I'm new to freqtrade, please help me and thanks!
  
### Observed Results:

  * What happened?
freqtrade tried to buy EOS/BTC with amount of 26.46 but after the order filled, the actual amount is 26.43 so I have to manually change the amount in DB.

### Relevant code exceptions or logs:
`2019-11-07 07:21:12,992 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=1, pair=EOS/BTC, amount=26.46202699, open_rate=0.00037790, open_since=2019-11-07 12:20:54)`
`2019-11-07 07:21:13,703 - freqtrade.freqtradebot - INFO - Applying fee on amount for Trade(id=1, pair=EOS/BTC, amount=26.46202699, open_rate=0.00037790, open_since=2019-11-07 12:20:54) failed: myTrade-Dict empty found`
`2019-11-07 07:21:13,704 - freqtrade.persistence - INFO - Updating trade (id=1) ...`
`2019-11-07 07:21:13,704 - freqtrade.persistence - INFO - LIMIT_BUY has been fulfilled for Trade(id=1, pair=EOS/BTC, amount=26.46000000, open_rate=0.00037800, open_since=2019-11-07 12:20:54).`



Discussion:
> 2019-11-07 07:21:13,703 - freqtrade.freqtradebot - INFO - Applying fee on amount for Trade(id=1, pair=EOS/BTC, amount=26.46202699, open_rate=0.00037790, open_since=2019-11-07 12:20:54) failed: myTrade-Dict empty found

Is the clock on your system accurate?
This does not just mean that you see the correct time - but that the information about the timezone is correct (so if you're in the CET timezone, the command `date` shows `Thu Nov  7 14:37:44 CET 2019`).

The above line is VERY suspicious - it shows a trade log from 07:21 - and the trade is supposed to be open since 12:20 - 4 hours into the future.

The "non-updated amount" happens because the trade on the exchange could not be found ... (`myTrade-Dict empty found`) 
however, since we search by time, having the time set wrongly can cause this to happen.
@xmatthias I am running the bot on the virtual machine. And command `date` show me `Thu Nov  7 08:45:23 EST 2019`, the time is correct but I am confused why open_since is totally different. Do you have any idea what might cause this issue?
> why open_since is totally different

It's in UTC. As all other calculations with time in Freqtrade and timezone used in communication with exchanges...

@hroff-1902  Thanks. I’ll try to change the time zone when I get back home. 
that shouldn't be necessary - it's just a mismatch of logs (in your timezone) vs. bot output (UTC).

What is the timestamp of the trade on the exchange?
> @hroff-1902 Thanks. I’ll try to change the time zone when I get back home.

I did not say that it will help neither I said it's necessary. I only explained why you see the timestamp that differs. (But you may try this, pls report back...)

@xmatthias there can be a bug here, should we add some tests for running the bot in the UTC-N timezones?

@xmatthias This is the transaction from binance
![2B6189E4-7E9C-49CA-BADE-391CB464AFF7](https://user-images.githubusercontent.com/32484941/68400165-94db6980-0145-11ea-8273-52fbf426eccf.jpeg)

> @xmatthias there can be a bug here, should we add some tests for running the bot in the UTC-N timezones?

I wouldn't know how to test this ...  we can't flip the system timezone for just one test?

also, consider that we use "trade.open_date" and then simply do a `(since.timestamp() -5) * 1000` (subtracting 5 seconds) as "since" to avoid fetching tons of old trades for that pair.

https://github.com/freqtrade/freqtrade/blob/ad2289c34cc938bc8b529f2676e71555fb216faa/freqtrade/exchange/exchange.py#L885



Let me correct myself - this IS related to the timezone ... please (for now) switch the time to UTC (or UTC+something) ... 

We'll work on a fix for this ...

2019-11-07 22:04:19,798 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=2, pair=AMB/BTC, amount=3012.04819277, open_rate=0.00000332, open_since=2019-11-07 22:00:35)
2019-11-07 22:04:24,257 - freqtrade.wallets - INFO - Wallets synced.
2019-11-07 22:04:24,791 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=2, pair=AMB/BTC, amount=3012.04819277, open_rate=0.00000332, open_since=2019-11-07 22:00:35)
2019-11-07 22:04:25,485 - freqtrade.freqtradebot - INFO - Applying fee on amount for Trade(id=2, pair=AMB/BTC, amount=3012.04819277, open_rate=0.00000332, open_since=2019-11-07 22:00:35) (from 3012.0 to 3008.988) from Trades
2019-11-07 22:04:25,485 - freqtrade.persistence - INFO - Updating trade (id=2) ...
2019-11-07 22:04:25,485 - freqtrade.persistence - INFO - LIMIT_BUY has been fulfilled for Trade(id=2, pair=AMB/BTC, amount=3008.98800000, open_rate=0.00000332, open_since=2019-11-07 22:00:35).

After I changed the time zone to UTC, the bot is working well.
Thanks a lot! @xmatthias @hroff-1902 
@edwardming789 thanks for reporting, @xmatthias has already localized the problem with timezones which caused this error for users from the Western hemisphere...

I had a similar issue with PERL and AMB .. so I sold those pairs manually .. now i don't have those coins in my wallet .. but the bot still updates shows those in my chat .. 
<img width="493" alt="Screen Shot 2019-11-07 at 7 14 47 PM" src="https://user-images.githubusercontent.com/20447610/68446912-fc6bd600-0192-11ea-8ba9-3752aae47cbf.png">
 
I have already sold these coins

How do I tell freqtrade to stop using them now ?

@shashrvacai You could either use sqlite3 or sqlite browser to open db and change it manually 
Unfortunately, you'll need to close this manually as documented [here](https://www.freqtrade.io/en/latest/sql_cheatsheet/#fix-trade-still-open-after-a-manual-sell-on-the-exchange).


Resolving patch:
diff --git a/freqtrade/data/btanalysis.py b/freqtrade/data/btanalysis.py
--- a/freqtrade/data/btanalysis.py
+++ b/freqtrade/data/btanalysis.py
@@ -7,7 +7,7 @@
 
 import numpy as np
 import pandas as pd
-import pytz
+from datetime import timezone
 
 from freqtrade import persistence
 from freqtrade.misc import json_load
@@ -106,8 +106,8 @@ def load_trades_from_db(db_url: str) -> pd.DataFrame:
                "stop_loss", "initial_stop_loss", "strategy", "ticker_interval"]
 
     trades = pd.DataFrame([(t.pair,
-                            t.open_date.replace(tzinfo=pytz.UTC),
-                            t.close_date.replace(tzinfo=pytz.UTC) if t.close_date else None,
+                            t.open_date.replace(tzinfo=timezone.utc),
+                            t.close_date.replace(tzinfo=timezone.utc) if t.close_date else None,
                             t.calc_profit(), t.calc_profit_percent(),
                             t.open_rate, t.close_rate, t.amount,
                             (round((t.close_date.timestamp() - t.open_date.timestamp()) / 60, 2)
diff --git a/freqtrade/data/history.py b/freqtrade/data/history.py
--- a/freqtrade/data/history.py
+++ b/freqtrade/data/history.py
@@ -9,12 +9,11 @@
 import logging
 import operator
 from copy import deepcopy
-from datetime import datetime
+from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple
 
 import arrow
-import pytz
 from pandas import DataFrame
 
 from freqtrade import OperationalException, misc
@@ -56,10 +55,10 @@ def trim_dataframe(df: DataFrame, timerange: TimeRange) -> DataFrame:
     Trim dataframe based on given timerange
     """
     if timerange.starttype == 'date':
-        start = datetime.fromtimestamp(timerange.startts, tz=pytz.utc)
+        start = datetime.fromtimestamp(timerange.startts, tz=timezone.utc)
         df = df.loc[df['date'] >= start, :]
     if timerange.stoptype == 'date':
-        stop = datetime.fromtimestamp(timerange.stopts, tz=pytz.utc)
+        stop = datetime.fromtimestamp(timerange.stopts, tz=timezone.utc)
         df = df.loc[df['date'] <= stop, :]
     return df
 
diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@@ -875,6 +875,22 @@ def get_order_book(self, pair: str, limit: int = 100) -> dict:
 
     @retrier
     def get_trades_for_order(self, order_id: str, pair: str, since: datetime) -> List:
+        """
+        Fetch Orders using the "fetch_my_trades" endpoint and filter them by order-id.
+        The "since" argument passed in is coming from the database and is in UTC,
+        as timezone-native datetime object.
+        From the python documentation:
+            > Naive datetime instances are assumed to represent local time
+        Therefore, calling "since.timestamp()" will get the UTC timestamp, after applying the
+        transformation from local timezone to UTC.
+        This works for timezones UTC+ since then the result will contain trades from a few hours
+        instead of from the last 5 seconds, however fails for UTC- timezones,
+        since we're then asking for trades with a "since" argument in the future.
+
+        :param order_id order_id: Order-id as given when creating the order
+        :param pair: Pair the order is for
+        :param since: datetime object of the order creation time. Assumes object is in UTC.
+        """
         if self._config['dry_run']:
             return []
         if not self.exchange_has('fetchMyTrades'):
@@ -882,7 +898,8 @@ def get_trades_for_order(self, order_id: str, pair: str, since: datetime) -> Lis
         try:
             # Allow 5s offset to catch slight time offsets (discovered in #1185)
             # since needs to be int in milliseconds
-            my_trades = self._api.fetch_my_trades(pair, int((since.timestamp() - 5) * 1000))
+            my_trades = self._api.fetch_my_trades(
+                pair, int((since.replace(tzinfo=timezone.utc).timestamp() - 5) * 1000))
             matched_trades = [trade for trade in my_trades if trade['order'] == order_id]
 
             return matched_trades


Test patch:
diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@@ -1586,8 +1586,9 @@ def test_name(default_conf, mocker, exchange_name):
 
 @pytest.mark.parametrize("exchange_name", EXCHANGES)
 def test_get_trades_for_order(default_conf, mocker, exchange_name):
+
     order_id = 'ABCD-ABCD'
-    since = datetime(2018, 5, 5, tzinfo=timezone.utc)
+    since = datetime(2018, 5, 5, 0, 0, 0)
     default_conf["dry_run"] = False
     mocker.patch('freqtrade.exchange.Exchange.exchange_has', return_value=True)
     api_mock = MagicMock()
@@ -1623,7 +1624,8 @@ def test_get_trades_for_order(default_conf, mocker, exchange_name):
     assert api_mock.fetch_my_trades.call_args[0][0] == 'LTC/BTC'
     # Same test twice, hardcoded number and doing the same calculation
     assert api_mock.fetch_my_trades.call_args[0][1] == 1525478395000
-    assert api_mock.fetch_my_trades.call_args[0][1] == int(since.timestamp() - 5) * 1000
+    assert api_mock.fetch_my_trades.call_args[0][1] == int(since.replace(
+        tzinfo=timezone.utc).timestamp() - 5) * 1000
 
     ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name,
                            'get_trades_for_order', 'fetch_my_trades',