Repository: rotki/rotki

Issue:
Include trades and asset movements from non-connected exchanges in history
## Problem Definition

If all trades are queried, only trades from connected exchanges are returned:

https://github.com/rotki/rotki/blob/f3a795e0a3061751db22e6ebfcf7d4701488878d/rotkehlchen/rotkehlchen.py#L603

If asset movements are queried, they are only queried from the connected exchanges or crypto.com:

https://github.com/rotki/rotki/blob/f3a795e0a3061751db22e6ebfcf7d4701488878d/rotkehlchen/rotkehlchen.py#L847-L888

That can miss:

- Trades/asset movements imported from cointracking
- Trades/asset movements from non-connected exchanges that are in the DB. They would most probably be from cointracking though since when we delete an api key connection we clean the DB.

## Task

Make it also fetch any of the non-connected location entries from the DB and include them in the result


Discussion:
@LefterisJP the way we do the checks at the moment requires this to also be added on the frontend.
The current approach though it's not perfect since for example crypto.com is queried every single time even for users that don't have any data.

For normal exchanges, we have the connected exchanges property so only these are queried by the frontend for things like crypto.com and cointracking we don't have anything though. How easy or hard would be to notify the frontend that something exists there? to avoid having hardcoded always performed requests?
@yabirgb don't you actually solve this already in v1.17.0?
@LefterisJP yes should be fixed in my PR for blockfi. I have to check for cointracking and trades because I don't remember. This logic was already in there so probably what is missing is the cointracking location
All right, so please make sure to also close this issue then once you are done.
For the first point `Trades/asset movements imported from cointracking`they should be correctly queried if the location is activated. 

About how to handle the trades from exchanges not connected we'll spend more time since essentially we need to ask for every known exchange, and the frontend also needs to query for all exchanges. I believe a better solution will be to add a list of imported exchanges in addition to the existing `conneted_exchanges` and use it at the moment of querying trades (this list is not the same as the external locations since will only contain locations where we have data but not from connected exchanges)
Having the list of external locations with trades/asset movements would certainly be a better approach than having to maintain a list of external locations on the frontend. Also this way we won't do any unneeded queries.
So to close this issue after #3797 gets merged we need to include all the trades in the frontend queries. Currently the frontend creates different trades queries for the external locations (crypto.com, bisq, nexo...) and for the exchanges that it knows have keys.

The PR #3797 introduces a new endpoint that will return all the locations that contain information, this can be exchanges, external locations or some defi protocols like `uniswap`, `sushiswap` and `balancer`.

What the frontend should do now is instead of trusting the known list of exchanges and query all external locations call this endpoint instead and query trades only for this locations. Take into account that it can include locations like `uniswap` and other so maybe when you query the trades you will need to apply some logic in the returned list

Resolving patch:
diff --git a/rotkehlchen/history/events.py b/rotkehlchen/history/events.py
--- a/rotkehlchen/history/events.py
+++ b/rotkehlchen/history/events.py
@@ -1,7 +1,7 @@
 import logging
 from collections import defaultdict
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast, overload
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set, Tuple, Union, cast, overload
 
 from typing_extensions import Literal
 
@@ -119,6 +119,11 @@ def __init__(
         self.datelocaltime = db_settings.display_date_in_localtime
         self._reset_variables()
 
+    def locations_from_csv(self) -> Set[Location]:
+        locations_with_keys = set(self.exchange_manager.connected_exchanges.keys())
+        associated_exchanges_locations = self.db.get_associated_locations().intersection(ALL_SUPPORTED_EXCHANGES)  # noqa: E501
+        return associated_exchanges_locations - locations_with_keys
+
     def timestamp_to_date(self, timestamp: Timestamp) -> str:
         return timestamp_to_date(
             timestamp,
@@ -291,7 +296,7 @@ def query_trades(
                 self.actions_per_location['trade'][given_location] = 0
             trades = self.query_location_trades(from_ts, to_ts, Location.EXTERNAL, only_cache)
             # Look for trades that might be imported from CSV files
-            for csv_location in EXTERNAL_EXCHANGES:
+            for csv_location in self.locations_from_csv():
                 trades.extend(self.query_location_trades(
                     from_ts=from_ts,
                     to_ts=to_ts,
@@ -366,26 +371,27 @@ def query_location_trades(
         else:
             # should only be an exchange
             exchanges_list = self.exchange_manager.connected_exchanges.get(location)
-            if exchanges_list is None:
-                log.warning(
-                    f'Tried to query trades from {str(location)} which is either not an '
-                    f'exchange or not an exchange the user has connected to',
-                )
-                return []
-
             location_trades = []
-            for exchange in exchanges_list:
-                all_set = {x.identifier for x in location_trades}
-                new_trades = exchange.query_trade_history(
-                    start_ts=from_ts,
-                    end_ts=to_ts,
-                    only_cache=only_cache,
+            if exchanges_list is None:
+                # Try to get imported trades for exchanges not connected
+                location_trades = self.db.get_trades(  # type: ignore  # list invariance
+                    from_ts=from_ts,
+                    to_ts=to_ts,
+                    location=location,
                 )
-                # TODO: Really dirty. Figure out a better way.
-                # Since some of the trades may already be in the DB if multiple
-                # keys are used for a single exchange.
-                new_trades = [x for x in new_trades if x.identifier not in all_set]
-                location_trades.extend(new_trades)
+            else:
+                for exchange in exchanges_list:
+                    all_set = {x.identifier for x in location_trades}
+                    new_trades = exchange.query_trade_history(
+                        start_ts=from_ts,
+                        end_ts=to_ts,
+                        only_cache=only_cache,
+                    )
+                    # TODO: Really dirty. Figure out a better way.
+                    # Since some of the trades may already be in the DB if multiple
+                    # keys are used for a single exchange.
+                    new_trades = [x for x in new_trades if x.identifier not in all_set]
+                    location_trades.extend(new_trades)
 
         trades: TRADES_LIST = []
         if self.chain_manager.premium is None:
@@ -423,7 +429,6 @@ def _query_and_populate_exchange_asset_movements(
             location_movements = [x for x in location_movements if x.identifier not in all_set]
         else:
             assert isinstance(exchange, Location), 'only a location should make it here'
-            assert exchange in EXTERNAL_EXCHANGES, 'only csv supported exchanges should get here'  # noqa : E501
             location = exchange
             # We might have no exchange information but CSV imported information
             self.actions_per_location['asset_movement'][location] = 0
@@ -466,7 +471,7 @@ def query_asset_movements(
         if location is not None:
             # clear the asset movements queried for this exchange
             self.actions_per_location['asset_movement'][location] = 0
-            if location in EXTERNAL_EXCHANGES:
+            if location in self.locations_from_csv():
                 movements = self._query_and_populate_exchange_asset_movements(
                     from_ts=from_ts,
                     to_ts=to_ts,


Test patch:
diff --git a/rotkehlchen/tests/api/test_trades.py b/rotkehlchen/tests/api/test_trades.py
--- a/rotkehlchen/tests/api/test_trades.py
+++ b/rotkehlchen/tests/api/test_trades.py
@@ -6,6 +6,7 @@
 
 from rotkehlchen.api.v1.encoding import TradeSchema
 from rotkehlchen.constants.assets import A_AAVE, A_BTC, A_DAI, A_EUR, A_WETH
+from rotkehlchen.constants.misc import ZERO
 from rotkehlchen.exchanges.data_structures import Trade
 from rotkehlchen.fval import FVal
 from rotkehlchen.history.events import FREE_TRADES_LIMIT
@@ -20,7 +21,7 @@
     assert_poloniex_trades_result,
     mock_history_processing_and_exchanges,
 )
-from rotkehlchen.typing import Location, TradeType
+from rotkehlchen.typing import AssetAmount, Fee, Location, Price, Timestamp, TradeType
 
 
 @pytest.mark.parametrize('added_exchanges', [(Location.BINANCE, Location.POLONIEX)])
@@ -786,3 +787,116 @@ def test_delete_trades_trades_errors(rotkehlchen_api_server):
         contained_in_msg="Tried to delete non-existing trade",
         status_code=HTTPStatus.CONFLICT,
     )
+
+
+@pytest.mark.parametrize('added_exchanges', [(Location.BINANCE, Location.POLONIEX)])
+def test_query_trades_associated_locations(rotkehlchen_api_server_with_exchanges):
+    """Test that querying the trades endpoint works as expected when we have associated
+    locations including exchanges asociated and imported locations.
+    """
+    rotki = rotkehlchen_api_server_with_exchanges.rest_api.rotkehlchen
+    setup = mock_history_processing_and_exchanges(rotki)
+
+    trades = [Trade(
+        timestamp=Timestamp(1596429934),
+        location=Location.EXTERNAL,
+        base_asset=A_WETH,
+        quote_asset=A_EUR,
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('1')),
+        rate=Price(FVal('320')),
+        fee=Fee(ZERO),
+        fee_currency=A_EUR,
+        link='',
+        notes='',
+    ), Trade(
+        timestamp=Timestamp(1596429934),
+        location=Location.KRAKEN,
+        base_asset=A_WETH,
+        quote_asset=A_EUR,
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('1')),
+        rate=Price(FVal('320')),
+        fee=Fee(ZERO),
+        fee_currency=A_EUR,
+        link='',
+        notes='',
+    ), Trade(
+        timestamp=Timestamp(1596429934),
+        location=Location.BISQ,
+        base_asset=A_WETH,
+        quote_asset=A_EUR,
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('1')),
+        rate=Price(FVal('320')),
+        fee=Fee(ZERO),
+        fee_currency=A_EUR,
+        link='',
+        notes='',
+    ), Trade(
+        timestamp=Timestamp(1596429934),
+        location=Location.BINANCE,
+        base_asset=A_WETH,
+        quote_asset=A_EUR,
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('1')),
+        rate=Price(FVal('320')),
+        fee=Fee(ZERO),
+        fee_currency=A_EUR,
+        link='',
+        notes='',
+    )]
+
+    # Add multiple entries for same exchange + connected exchange
+    rotki.data.db.add_trades(trades)
+
+    # Simply get all trades without any filtering
+    with setup.binance_patch, setup.polo_patch:
+        response = requests.get(
+            api_url_for(
+                rotkehlchen_api_server_with_exchanges,
+                'tradesresource',
+            ),
+        )
+    result = assert_proper_response_with_result(response)
+    result = result['entries']
+    assert len(result) == 9  # 3 polo, (2 + 1) binance trades, 1 kraken, 1 external, 1 BISQ
+    expected_locations = (
+        Location.KRAKEN,
+        Location.POLONIEX,
+        Location.BINANCE,
+        Location.BISQ,
+        Location.EXTERNAL,
+    )
+    returned_locations = {x['entry']['location'] for x in result}
+    assert returned_locations == set(map(str, expected_locations))
+
+    response = requests.get(
+        api_url_for(
+            rotkehlchen_api_server_with_exchanges,
+            "tradesresource",
+        ), json={'location': 'kraken'},
+    )
+    result = assert_proper_response_with_result(response)
+    result = result['entries']
+    assert len(result) == 1
+
+    response = requests.get(
+        api_url_for(
+            rotkehlchen_api_server_with_exchanges,
+            "tradesresource",
+        ), json={'location': 'binance'},
+    )
+    result = assert_proper_response_with_result(response)
+    result = result['entries']
+    assert len(result) == 3
+
+    response = requests.get(
+        api_url_for(
+            rotkehlchen_api_server_with_exchanges,
+            "tradesresource",
+        ), json={'location': 'nexo'},
+    )
+    result = assert_proper_response_with_result(response)
+    result = result['entries']
+    assert len(result) == 0