Repository: rotki/rotki

Issue:
Binance: Too many requests; current limit is 1200 req/min
I got the following error/warning message when creating a tax report for 8/1/2018-8/26/2018. 

# Trade History Query Warning
>During trade history query we got: Binance API request https://api.binance.com/api/v3/myTrades?symbol=PIVXBTC&fromId=0&recvWindow=10000×tamp=1535324978160&signature=de264... for myTrades failed with HTTP status code: 429, error code: -1003 and error message: Too many requests; current limit is 1200 requests per minute. Please use the websocket for live updates to avoid polling the API.. History report is probably not complete.




Should there be a setting in rotkehlchen to limit the basic requests per minute for each exchange's API? 

It seems that different tiers of membership at each exchange might allow for a greater number of available requests/min. It would be best if rotkehlchen would support faster polling, when possible.




Discussion:
Where did you see this warning? Where was it shown?

 Binance does indeed have query limits as explained in their API [here](https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md#limits)
This was shown as a popup in rotkehlchen.

Thank you for the API link.

Is there any way to limit the requests per minute from rotkehlchen so that the error doesn't happen?
It keeps happening for you? What action are you taking when it happens?
Doesn't seem to happen anymore. If it happens again I'll open another issue.
I got a similar error again...
<img width="2004" alt="screen shot 2018-09-20 at 10 28 54 pm" src="https://user-images.githubusercontent.com/4165385/45915872-9fd41d00-be11-11e8-8ea1-3accfbabb286.png">

Hi @LefterisJP, I am again receiving a similar error about limit of requests per minute but this time I noticed the symbol (ex. above "STORMBTC") in the URI query string is not one that I have used. 

Why is Rotkehlchen querying the exchange for symbols that have no transactions?
Hmm I see what you mean. This limitation comes due to the way the binance API is written. I have to iterate through all existing markets to see which ones you have history for. Perhaps they have added a new endpoint since the time I wrote this. I will check.
No I dont' see any such endpoint: https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md
@kascheri12 I can try to handle the 429 response for binance and wait on it instead of failing like above. But how the heck did you manage to go over 1200 requests per minute? Can you perhaps describe exactly what you did or even send me logs?

Assuming you used the tax report query per the screenshot, that should use the `myTrades` endpoint which has a `weight` of `5` for  each query. So that leaves us `1200 / 5 =  240` requests per minute. For the symbols for which you have lots of trades we need to make multiple requests (only 500 trades are returned per request). So you must really have had a lot of binance trades and a very good connection to get that.

I think the right solution here is to handle 429 for binance and add a waiting factor.
@LefterisJP I have sent you an email with the goods. 

These are the steps taken for the test case:
- Run from latest master branch; macOS 10.14.2
- Login and wait for dashboard to load
- Select Tax Report
- Select May 1, 2018 00:00 & May 31, 2018 00:00 as begin/end dates
- Select Generate Report
- Wait, receive 429 response, select Close


I don't think 150Mbps is excessively high bandwidth... I'm using an i7 MacBook Pro..? I do believe I have a lot of (probably unsuccessful/unnecessary) Binance trades.

It's unfortunate that the Binance API has this limitation. It makes for lots of wasted resources for all parties involved.

Resolving patch:
diff --git a/rotkehlchen/binance.py b/rotkehlchen/binance.py
--- a/rotkehlchen/binance.py
+++ b/rotkehlchen/binance.py
@@ -5,6 +5,8 @@
 from typing import Any, Dict, List, NamedTuple, Optional, Tuple, Union, cast
 from urllib.parse import urlencode
 
+import gevent
+
 from rotkehlchen import typing
 from rotkehlchen.errors import RemoteError
 from rotkehlchen.exchange import Exchange
@@ -107,6 +109,8 @@ def __init__(
             secret: typing.ApiSecret,
             inquirer: Inquirer,
             data_dir: typing.FilePath,
+            initial_backoff: int = 4,
+            backoff_limit: int = 180,
     ):
         super(Binance, self).__init__('binance', api_key, secret, data_dir)
         self.apiversion = 'v3'
@@ -116,6 +120,8 @@ def __init__(
             'Accept': 'application/json',
             'X-MBX-APIKEY': self.api_key,
         })
+        self.initial_backoff = initial_backoff
+        self.backoff_limit = backoff_limit
 
     def first_connection(self):
         if self.first_connection_made:
@@ -164,44 +170,60 @@ def api_query(self, method: str, options: Optional[Dict] = None) -> Union[List,
         if not options:
             options = {}
 
-        with self.lock:
-            # Protect this region with a lock since binance will reject
-            # non-increasing nonces. So if two greenlets come in here at
-            # the same time one of them will fail
-            if method in V3_ENDPOINTS:
-                api_version = 3
-                # Recommended recvWindows is 5000 but we get timeouts with it
-                options['recvWindow'] = 10000
-                options['timestamp'] = str(int(time.time() * 1000))
-                signature = hmac.new(
-                    self.secret,
-                    urlencode(options).encode('utf-8'),
-                    hashlib.sha256,
-                ).hexdigest()
-                options['signature'] = signature
-            elif method in V1_ENDPOINTS:
-                api_version = 1
+        backoff = self.initial_backoff
+
+        while True:
+            with self.lock:
+                # Protect this region with a lock since binance will reject
+                # non-increasing nonces. So if two greenlets come in here at
+                # the same time one of them will fail
+                if method in V3_ENDPOINTS:
+                    api_version = 3
+                    # Recommended recvWindows is 5000 but we get timeouts with it
+                    options['recvWindow'] = 10000
+                    options['timestamp'] = str(int(time.time() * 1000))
+                    signature = hmac.new(
+                        self.secret,
+                        urlencode(options).encode('utf-8'),
+                        hashlib.sha256,
+                    ).hexdigest()
+                    options['signature'] = signature
+                elif method in V1_ENDPOINTS:
+                    api_version = 1
+                else:
+                    raise ValueError('Unexpected binance api method {}'.format(method))
+
+                request_url = self.uri + 'v' + str(api_version) + '/' + method + '?'
+                request_url += urlencode(options)
+
+                log.debug('Binance API request', request_url=request_url)
+
+                response = self.session.get(request_url)
+
+            limit_ban = response.status_code == 429 and backoff > self.backoff_limit
+            if limit_ban or response.status_code not in (200, 429):
+                result = rlk_jsonloads(response.text)
+                raise RemoteError(
+                    'Binance API request {} for {} failed with HTTP status '
+                    'code: {}, error code: {} and error message: {}'.format(
+                        response.url,
+                        method,
+                        response.status_code,
+                        result['code'],
+                        result['msg'],
+                    ))
+            elif response.status_code == 429:
+                if backoff > self.backoff_limit:
+                    break
+                # Binance has limits and if we hit them we should backoff
+                # https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md#limits
+                log.debug('Got 429 from Binance. Backing off', seconds=backoff)
+                gevent.sleep(backoff)
+                backoff = backoff * 2
+                continue
             else:
-                raise ValueError('Unexpected binance api method {}'.format(method))
-
-            request_url = self.uri + 'v' + str(api_version) + '/' + method + '?'
-            request_url += urlencode(options)
-
-            log.debug('Binance API request', request_url=request_url)
-
-            response = self.session.get(request_url)
-
-        if response.status_code != 200:
-            result = rlk_jsonloads(response.text)
-            raise RemoteError(
-                'Binance API request {} for {} failed with HTTP status '
-                'code: {}, error code: {} and error message: {}'.format(
-                    response.url,
-                    method,
-                    response.status_code,
-                    result['code'],
-                    result['msg'],
-                ))
+                # success
+                break
 
         json_ret = rlk_jsonloads(response.text)
         return json_ret
@@ -261,21 +283,30 @@ def query_trade_history(
             markets = self._symbols_to_pair.keys()
 
         all_trades_history = list()
+        # Limit of results to return. 1000 is max limit according to docs
+        limit = 1000
         for symbol in markets:
             last_trade_id = 0
-            len_result = 500
-            while len_result == 500:
+            len_result = limit
+            while len_result == limit:
                 result = self.api_query(
                     'myTrades',
                     options={
                         'symbol': symbol,
                         'fromId': last_trade_id,
+                        'limit': limit,
+                        # Not specifying them since binance does not seem to
+                        # respect them and always return all trades
+                        # 'startTime': start_ts * 1000,
+                        # 'endTime': end_ts * 1000,
                     })
+                if result:
+                    last_trade_id = result[-1]['id'] + 1
                 len_result = len(result)
                 log.debug('binance myTrades query result', results_num=len_result)
                 for r in result:
                     r['symbol'] = symbol
-            all_trades_history.extend(result)
+                all_trades_history.extend(result)
 
         all_trades_history.sort(key=lambda x: x['time'])
 


Test patch:
diff --git a/rotkehlchen/tests/test_binance.py b/rotkehlchen/tests/test_binance.py
--- a/rotkehlchen/tests/test_binance.py
+++ b/rotkehlchen/tests/test_binance.py
@@ -1,11 +1,14 @@
 import json
-import pytest
 import os
 from pathlib import Path
+from unittest.mock import patch
+
+import pytest
 
+from rotkehlchen.binance import Binance, trade_from_binance
+from rotkehlchen.errors import RemoteError
 from rotkehlchen.fval import FVal
 from rotkehlchen.order_formatting import Trade
-from rotkehlchen.binance import Binance, trade_from_binance
 
 
 @pytest.fixture
@@ -127,3 +130,50 @@ def test_trade_from_binance(mock_binance):
     for idx, binance_trade in enumerate(binance_trades_list):
         our_trade = trade_from_binance(binance_trade, mock_binance.symbols_to_pair)
         assert our_trade == our_expected_list[idx]
+
+
+exchange_info_mock_text = '''{
+  "timezone": "UTC",
+  "serverTime": 1508631584636,
+  "symbols": [{
+    "symbol": "ETHBTC",
+    "status": "TRADING",
+    "baseAsset": "ETH",
+    "baseAssetPrecision": 8,
+    "quoteAsset": "BTC",
+    "quotePrecision": 8,
+    "icebergAllowed": false
+    }]
+}'''
+
+
+def test_binance_backoff_after_429(mock_binance):
+    class MockResponse():
+
+        def __init__(self, status_code, text):
+            self.status_code = status_code
+            self.text = text
+            self.url = 'http://someurl.com'
+
+    count = 0
+
+    def mock_429(url):
+        nonlocal count
+        if count < 2:
+            response = MockResponse(429, '{"code": -1103, "msg": "Too many requests"}')
+        else:
+            response = MockResponse(200, exchange_info_mock_text)
+        count += 1
+        return response
+
+    mock_binance.initial_backoff = 0.5
+    mock_binance.backoff_limit = 2
+    with patch.object(mock_binance.session, 'get', side_effect=mock_429):
+        # test that after 2 429 cals we finally succeed in the API call
+        result = mock_binance.api_query('exchangeInfo')
+        assert 'timezone' in result
+
+        # Test the backoff_limit properly returns an error when hit
+        count = -9999999
+        with pytest.raises(RemoteError):
+            mock_binance.api_query('exchangeInfo')