Repository: rotki/rotki

Issue:
Frontend timeouts while trying to parse very big json response
## Problem Definition

While trying to create the PnL report for a big number of events the frontend hangs and no output is produced

## How to reproduce

In the backend change https://github.com/rotki/rotki/blob/48589c7d79e8a464c9f44084bd6ce4a1a3d9730c/rotkehlchen/accounting/accountant.py#L466 to something like `'all_events': self.csvexporter.all_events*10000,` and request a full year of events (or a period with enough events).

## Observable behavior

The frontend successfully retrieves the task response from the backend but it timeouts trying to read the response with errors in the console about the request failing but not providing any further information  



Discussion:
This happens due to the sheer size of the returned data. The frontend takes too long to parse the json (0.5-1GB) and even when it does it will blow up the memory of the frontend.

The fact this can happen shows a big weakness in our current approach and this bugfix requires the implementation of https://github.com/rotki/rotki/issues/1214 too.

How we can solve it properly:

## Save PnL reports in the DB

Fix #1214

1. Create a new user DB, encrypted with the same password as the normal user DB. Create a table there to save user PNL reports. The reason for a new DB is that this DB should hold data that is not synced with the server. Due to the sheer size of the PnL reports we can't afford to sync it with our server for premium users.
2. Save each PnL report in a table in that DB
3. Create API endpoints to manage existing PnL reports. We should have endpoints to (1) retrieve metadata and full report for each report (2) delete report to save space and (3) get all reports data (number of saved reports, size of each, date range, PnL overview)
 
## Change the history result endpoint

The history result endpoint should no longer exist. Instead what we should do is check when a PnL report finished (same way as we do now) and once it's been saved in the special DB retrieve it.

It would be retrieved through the aforementioned "get metadata and full report" for a report. This should have pagination. Probaly very simple one with offset and limit and it should be sorted by timestamp.

What this would allow is retrieving of any report in the frontend and paginating it without keeping (1) the entire report in memory in the frontend AND (2) the entire report in memory in the backend


-----

Implementing both those things would essentially fix the bug here.


Resolving patch:
diff --git a/rotkehlchen/chain/ethereum/manager.py b/rotkehlchen/chain/ethereum/manager.py
--- a/rotkehlchen/chain/ethereum/manager.py
+++ b/rotkehlchen/chain/ethereum/manager.py
@@ -26,6 +26,7 @@
 from rotkehlchen.chain.ethereum.contracts import EthereumContract
 from rotkehlchen.chain.ethereum.graph import Graph
 from rotkehlchen.chain.ethereum.modules.eth2 import ETH2_DEPOSIT
+from rotkehlchen.chain.ethereum.typing import string_to_ethereum_address
 from rotkehlchen.chain.ethereum.utils import multicall_2
 from rotkehlchen.constants.ethereum import ERC20TOKEN_ABI, ETH_SCAN
 from rotkehlchen.errors import (
@@ -217,6 +218,8 @@ def __init__(
         self.etherscan = etherscan
         self.msg_aggregator = msg_aggregator
         self.eth_rpc_timeout = eth_rpc_timeout
+        self.archive_connection = False
+        self.queried_archive_connection = False
         for node in connect_at_start:
             self.greenlet_manager.spawn_and_track(
                 after_seconds=None,
@@ -406,8 +409,8 @@ def query(self, method: Callable, call_order: Sequence[NodeName], **kwargs: Any)
                 result = method(web3, **kwargs)
             except (
                     RemoteError,
-                    BlockchainQueryError,
                     requests.exceptions.RequestException,
+                    BlockchainQueryError,
                     TransactionNotFound,
                     KeyError,  # saw this happen inside web3.py if resulting json contains unexpected key. Happened with mycrypto's node  # noqa: E501
             ) as e:  # noqa: E501
@@ -436,6 +439,51 @@ def get_latest_block_number(self, call_order: Optional[Sequence[NodeName]] = Non
             call_order=call_order if call_order is not None else self.default_call_order(),
         )
 
+    def get_historical_eth_balance(
+            self,
+            address: ChecksumEthAddress,
+            block_number: int,
+    ) -> Optional[FVal]:
+        """Attempts to get a historical eth balance from the local own node only.
+        If there is no node or the node can't query historical balance (not archive) then
+        returns None"""
+        web3 = self.web3_mapping.get(NodeName.OWN)
+        if web3 is None:
+            return None
+
+        try:
+            result = web3.eth.get_balance(address, block_identifier=block_number)
+        except (
+                requests.exceptions.RequestException,
+                BlockchainQueryError,
+                TransactionNotFound,
+                KeyError,  # saw this happen inside web3.py if resulting json contains unexpected key. Happened with mycrypto's node  # noqa: E501
+        ):
+            return None
+
+        try:
+            balance = from_wei(FVal(result))
+        except ValueError:
+            return None
+
+        return balance
+
+    def have_archive(self, requery: bool = False) -> bool:
+        """Checks to see if our own connected node is an archive node
+
+        If requery is True it always queries the node. Otherwise it remembers last query.
+        """
+        if self.queried_archive_connection and requery is False:
+            return self.archive_connection
+
+        balance = self.get_historical_eth_balance(
+            address=string_to_ethereum_address('0x50532e4Be195D1dE0c2E6DfA46D9ec0a4Fee6861'),
+            block_number=87042,
+        )
+        self.archive_connection = balance is not None and balance == FVal('5.1063307')
+        self.queried_archive_connection = True
+        return self.archive_connection
+
     def query_eth_highest_block(self) -> BlockNumber:
         """ Attempts to query an external service for the block height
 


Test patch: