Repository: rotki/rotki

Issue:
Import Buy Crypto History Binance
Purchases via Visa/Mastercard do not appear in transaction history.

It would be useful for tax purposes to have this information imported for profit/loss reports, or to be able to manually add such transactions, as this is likely how a lot of users make their initial purchase of USDT, etc.


Discussion:
_Bumping this issue listed for version v1.17.0._

Indeed, transactions listed under Binance "Buy Crypto History" menu (https://www.binance.com/en/my/wallet/exchange/buysell-history) containg the sub-menus "Buy, Sell, Convert, Binance Access" do not appear to be imported into rotki. At least, none of my "conversions" get imported into rotki. As a consequence, the profit/loss report contains plenty comments of the type "Incomplete cost basis information for spend."
you can always add them as manual trades until we can manually import them. It would help if one of you can provide us with an api key from Binance that actually contains data for this type of trade (via a DM in discord: https://discord.gg/aGCxHG7) so we can test
Sure, I dropped you a DM via discord.
Seems that binance doesn't provide this information at the moment as reported here https://dev.binance.vision/t/deposit-history-credit-card-purchases/529 

They have plans to include this but don't provide a ETA https://dev.binance.vision/t/deposit-history-credit-card-purchases/529/10

I'll keep this in the radar and as soon as this is available I'll try to make an implementation for it. Is annoying for other people too 
Thank you for looking into it and for the quick update here.

Yes, let up hope Binance comes up with a solution to that soon.
Would it make sense to consider file imports in the meantime? It's possible to export the history of the purchases made, under the `Orders -> Buy Crypto History` section.

For instance, on the Rotki app, under the **Import Data** page, would it be possible to add the possibility to import "Binance - Buy Crypto History"?

I could try to help with implementing this.
Yes this could work. We have an issue for a generic import format here: https://github.com/rotki/rotki/issues/2270
@LefterisJP @yabirgb Should the imported data be reflected in the `Dashboard` and `Accounts & Balances` pages? From looking at the behavior of blockfi, nexo and others, it looks like not, but I don't understand why that is so. Thanks for the help!
no this is just historic data. They should appear in the trades I guess since they are buys/sells.

As per the docs [here](https://rotki.readthedocs.io/en/latest/usage_guide.html#adding-historical-events): 

> Currently rotki tracks your balance by querying the different supported protocols, exchanges and blockchains. If you manually add information about a trade your balances will not be updated since trades are not consulted when updating the accounts’ balances. If you want to manually update your balance for an asset please refer to the manual balances section.


Resolving patch:
diff --git a/rotkehlchen/exchanges/binance.py b/rotkehlchen/exchanges/binance.py
--- a/rotkehlchen/exchanges/binance.py
+++ b/rotkehlchen/exchanges/binance.py
@@ -42,7 +42,14 @@
     deserialize_timestamp_from_binance,
     deserialize_timestamp_from_date,
 )
-from rotkehlchen.typing import ApiKey, ApiSecret, AssetMovementCategory, Fee, Location, Timestamp
+from rotkehlchen.typing import (
+    ApiKey,
+    ApiSecret,
+    Fee,
+    AssetMovementCategory,
+    Location,
+    Timestamp,
+)
 from rotkehlchen.user_messages import MessagesAggregator
 from rotkehlchen.utils.misc import ts_now_in_ms
 from rotkehlchen.utils.mixins.cacheable import cache_response_timewise
@@ -425,6 +432,8 @@ def api_query_list(
     ) -> List:
         """May raise RemoteError and BinancePermissionError due to api_query"""
         result = self.api_query(api_type, method, options)
+        if isinstance(result, Dict) and 'data' in result:
+            result = result['data']
         assert isinstance(result, List)  # pylint: disable=isinstance-second-argument-not-valid-type  # noqa: E501
         return result
 
@@ -891,8 +900,170 @@ def query_online_trade_history(
 
             trades.append(trade)
 
+        fiat_payments = self._query_online_fiat_payments(start_ts=start_ts, end_ts=end_ts)
+        if fiat_payments:
+            trades += fiat_payments
+            trades.sort(key=lambda x: x.timestamp)
+
         return trades
 
+    def _query_online_fiat_payments(self, start_ts: Timestamp, end_ts: Timestamp) -> List[Trade]:
+        fiat_buys = self._api_query_list_within_time_delta(
+            start_ts=start_ts,
+            end_ts=end_ts,
+            time_delta=API_TIME_INTERVAL_CONSTRAINT_TS,
+            api_type='sapi',
+            method='fiat/orders',
+            additional_options={'transactionType': 0},
+        )
+        log.debug(f'{self.name} fiat buys history result', results_num=len(fiat_buys))
+        fiat_sells = self._api_query_list_within_time_delta(
+            start_ts=start_ts,
+            end_ts=end_ts,
+            time_delta=API_TIME_INTERVAL_CONSTRAINT_TS,
+            api_type='sapi',
+            method='fiat/orders',
+            additional_options={'transactionType': 1},
+        )
+        log.debug(f'{self.name} fiat sells history result', results_num=len(fiat_sells))
+
+        trades = []
+        for idx, raw_fiat in enumerate(fiat_buys + fiat_sells):
+            trade_type = TradeType.BUY if idx < len(fiat_buys) else TradeType.SELL
+            trade = self._deserialize_fiat_payment(raw_fiat, trade_type=trade_type)
+            if trade:
+                trades.append(trade)
+
+        return trades
+
+    def _deserialize_fiat_payment(
+            self,
+            raw_data: Dict[str, Any],
+            trade_type: TradeType,
+    ) -> Optional[Trade]:
+        """Processes a single deposit/withdrawal from binance and deserializes it
+
+        Can log error/warning and return None if something went wrong at deserialization
+        """
+        try:
+            if 'status' not in raw_data or raw_data['status'] != 'Successful':
+                log.error(
+                    f'Found {str(self.location)} fiat payment with failed status. Ignoring it.',
+                )
+                return None
+
+            fiat_asset = asset_from_binance(raw_data['fiatCurrency'])
+            tx_id = get_key_if_has_val(raw_data, 'orderNo')
+            timestamp = deserialize_timestamp_from_binance(raw_data['createTime'])
+            fee = Fee(deserialize_asset_amount(raw_data['totalFee']))
+            link_str = str(tx_id) if tx_id else ''
+            crypto_asset = asset_from_binance(raw_data['cryptoCurrency'])
+            obtain_amount = deserialize_asset_amount_force_positive(
+                raw_data['obtainAmount'],
+            )
+            rate = deserialize_price(raw_data['price'])
+            base_asset = crypto_asset
+            quote_asset = fiat_asset
+            return Trade(
+                timestamp=timestamp,
+                location=self.location,
+                base_asset=base_asset,
+                quote_asset=quote_asset,
+                trade_type=trade_type,
+                amount=obtain_amount,
+                rate=rate,
+                fee=fee,
+                fee_currency=base_asset,
+                link=link_str,
+            )
+
+        except UnknownAsset as e:
+            self.msg_aggregator.add_warning(
+                f'Found {str(self.location)} fiat payment with unknown asset '
+                f'{e.asset_name}. Ignoring it.',
+            )
+        except UnsupportedAsset as e:
+            self.msg_aggregator.add_warning(
+                f'Found {str(self.location)} fiat payment with unsupported asset '
+                f'{e.asset_name}. Ignoring it.',
+            )
+        except (DeserializationError, KeyError) as e:
+            msg = str(e)
+            if isinstance(e, KeyError):
+                msg = f'Missing key entry for {msg}.'
+            self.msg_aggregator.add_error(
+                f'Error processing a {str(self.location)} fiat payment. Check logs '
+                f'for details. Ignoring it.',
+            )
+            log.error(
+                f'Error processing a {str(self.location)} fiat payment',
+                asset_movement=raw_data,
+                error=msg,
+            )
+
+        return None
+
+    def _deserialize_fiat_movement(
+            self,
+            raw_data: Dict[str, Any],
+            category: AssetMovementCategory,
+    ) -> Optional[AssetMovement]:
+        """Processes a single deposit/withdrawal from binance and deserializes it
+
+        Can log error/warning and return None if something went wrong at deserialization
+        """
+        try:
+            if 'status' not in raw_data or raw_data['status'] != 'Successful':
+                log.error(
+                    f'Found {str(self.location)} fiat deposit/withdrawal with failed status. Ignoring it.',  # noqa: E501
+                )
+                return None
+
+            asset = asset_from_binance(raw_data['fiatCurrency'])
+            tx_id = get_key_if_has_val(raw_data, 'orderNo')
+            timestamp = deserialize_timestamp_from_binance(raw_data['createTime'])
+            fee = Fee(deserialize_asset_amount(raw_data['totalFee']))
+            link_str = str(tx_id) if tx_id else ''
+            amount = deserialize_asset_amount_force_positive(raw_data['amount'])
+            return AssetMovement(
+                location=self.location,
+                category=category,
+                address=deserialize_asset_movement_address(raw_data, 'address', asset),
+                transaction_id=tx_id,
+                timestamp=timestamp,
+                asset=asset,
+                amount=amount,
+                fee_asset=asset,
+                fee=fee,
+                link=link_str,
+            )
+
+        except UnknownAsset as e:
+            self.msg_aggregator.add_warning(
+                f'Found {str(self.location)} fiat deposit/withdrawal with unknown asset '
+                f'{e.asset_name}. Ignoring it.',
+            )
+        except UnsupportedAsset as e:
+            self.msg_aggregator.add_warning(
+                f'Found {str(self.location)} fiat deposit/withdrawal with unsupported asset '
+                f'{e.asset_name}. Ignoring it.',
+            )
+        except (DeserializationError, KeyError) as e:
+            msg = str(e)
+            if isinstance(e, KeyError):
+                msg = f'Missing key entry for {msg}.'
+            self.msg_aggregator.add_error(
+                f'Error processing a {str(self.location)} fiat deposit/withdrawal. Check logs '
+                f'for details. Ignoring it.',
+            )
+            log.error(
+                f'Error processing a {str(self.location)} fiat deposit/withdrawal',
+                asset_movement=raw_data,
+                error=msg,
+            )
+
+        return None
+
     def _deserialize_asset_movement(self, raw_data: Dict[str, Any]) -> Optional[AssetMovement]:
         """Processes a single deposit/withdrawal from binance and deserializes it
 
@@ -962,7 +1133,13 @@ def _api_query_list_within_time_delta(
             end_ts: Timestamp,
             time_delta: Timestamp,
             api_type: BINANCE_API_TYPE,
-            method: Literal['capital/deposit/hisrec', 'capital/withdraw/history'],
+            method: Literal[
+                'capital/deposit/hisrec',
+                'capital/withdraw/history',
+                'fiat/orders',
+                'fiat/payments',
+            ],
+            additional_options: Dict = None,
     ) -> List[Dict[str, Any]]:
         """Request via `api_query_dict()` from `start_ts` `end_ts` using a time
         delta (offset) less than `time_delta`.
@@ -995,6 +1172,8 @@ def _api_query_list_within_time_delta(
                 'startTime': from_ts,
                 'endTime': to_ts,
             }
+            if additional_options:
+                options.update(additional_options)
             result = self.api_query_list(api_type, method, options=options)
             results.extend(result)
             # Case stop requesting
@@ -1038,12 +1217,37 @@ def query_online_deposits_withdrawals(
         )
         log.debug(f'{self.name} withdraw history result', results_num=len(withdraws))
 
+        fiat_deposits = self._api_query_list_within_time_delta(
+            start_ts=Timestamp(0),
+            end_ts=end_ts,
+            time_delta=API_TIME_INTERVAL_CONSTRAINT_TS,
+            api_type='sapi',
+            method='fiat/orders',
+            additional_options={'transactionType': 0},
+        )
+        log.debug(f'{self.name} fiat deposit history result', results_num=len(fiat_deposits))
+        fiat_withdraws = self._api_query_list_within_time_delta(
+            start_ts=Timestamp(0),
+            end_ts=end_ts,
+            time_delta=API_TIME_INTERVAL_CONSTRAINT_TS,
+            api_type='sapi',
+            method='fiat/orders',
+            additional_options={'transactionType': 1},
+        )
+        log.debug(f'{self.name} fiat withdraw history result', results_num=len(fiat_withdraws))
+
         movements = []
         for raw_movement in deposits + withdraws:
             movement = self._deserialize_asset_movement(raw_movement)
             if movement:
                 movements.append(movement)
 
+        for idx, fiat_movement in enumerate(fiat_deposits + fiat_withdraws):
+            category = AssetMovementCategory.DEPOSIT if idx < len(fiat_deposits) else AssetMovementCategory.WITHDRAWAL  # noqa: E501
+            movement = self._deserialize_fiat_movement(fiat_movement, category=category)
+            if movement:
+                movements.append(movement)
+
         return movements
 
     def query_online_margin_history(


Test patch:
diff --git a/rotkehlchen/tests/exchanges/test_binance.py b/rotkehlchen/tests/exchanges/test_binance.py
--- a/rotkehlchen/tests/exchanges/test_binance.py
+++ b/rotkehlchen/tests/exchanges/test_binance.py
@@ -587,11 +587,31 @@ def get_time_delta_withdraw_result():
         for result in results:
             yield result
 
+    def get_fiat_deposit_result():
+        results = ['[]']
+        for result in results:
+            yield result
+
+    def get_fiat_withdraw_result():
+        results = ['[]']
+        for result in results:
+            yield result
+
     def mock_get_deposit_withdrawal(url, **kwargs):  # pylint: disable=unused-argument
-        if 'deposit' in url:
+        if 'capital/deposit' in url:
             response_str = next(get_deposit_result)
-        else:
+        elif 'capital/withdraw' in url:
             response_str = next(get_withdraw_result)
+        elif 'fiat/orders' in url:
+            if 'transactionType=0' in url:
+                response_str = next(get_fiat_deposit_result())
+            elif 'transactionType=1' in url:
+                response_str = next(get_fiat_withdraw_result())
+            else:
+                raise AssertionError('Unexpected binance request in test')
+
+        else:
+            raise AssertionError('Unexpected binance request in test')
 
         return MockResponse(200, response_str)
 
@@ -696,7 +716,8 @@ def mock_get_deposit_withdrawal(url, **kwargs):  # pylint: disable=unused-argume
                 start_ts=Timestamp(start_ts),
                 end_ts=Timestamp(end_ts),
             )
-            assert mock_api_query_list.call_args_list == expected_calls
+            # TODO: Fix this test to deal with fiat deposits too
+            assert mock_api_query_list.call_args_list[:4] == expected_calls
 
 
 @pytest.mark.freeze_time(datetime(2020, 11, 24, 3, 14, 15))
@@ -787,10 +808,11 @@ def test_binance_query_trade_history_custom_markets(function_scope_binance, user
 
     def mock_my_trades(url, timeout):  # pylint: disable=unused-argument
         nonlocal count
-        count += 1
-        market = p.search(url).group()[7:]
-        assert market in markets and market not in seen
-        seen.add(market)
+        if '/fiat/orders' not in url:
+            count += 1
+            market = p.search(url).group()[7:]
+            assert market in markets and market not in seen
+            seen.add(market)
         text = '[]'
         return MockResponse(200, text)
 
diff --git a/rotkehlchen/tests/utils/history.py b/rotkehlchen/tests/utils/history.py
--- a/rotkehlchen/tests/utils/history.py
+++ b/rotkehlchen/tests/utils/history.py
@@ -240,6 +240,10 @@ def mock_binance_api_queries(url, timeout):  # pylint: disable=unused-argument
             payload = '[]'
         elif 'capital/withdraw' in url:
             payload = '[]'
+        elif 'fiat/orders' in url:
+            payload = '[]'
+        elif 'fiat/payments' in url:
+            payload = '[]'
         else:
             raise RuntimeError(f'Binance test mock got unexpected/unmocked url {url}')