Repository: rotki/rotki

Issue:
Compound backend problems
## Problem Definition

1. Exception while querying compound history

```
[11/09/2020 17:33:58 CEST] web3.providers.HTTPProvider: Getting response HTTP. URI: https://api.mycryptoapi.com/eth, Method: eth_call, Response: {'jsonrpc': '2.0', 'result': '0x0000000000000000000000000000000000000000000000000000000af3a37c57', 'id': 18}                    
[11/09/2020 17:33:58 CEST] rotkehlchen.externalapis.etherscan: Querying etherscan: https://api.etherscan.io/api?module=proxy&action=eth_call&to=0x39AA39c021dfbaE8faC545936693aC917d5E7563&data=0xae9d70b0&apikey=YKABHR9XM65MI3GPK2JE3VFJAFXZUQJKNN                             
[11/09/2020 17:33:58 CEST] web3.RequestManager: Making request. Method: eth_call                                                                                                                                                                                                 
[11/09/2020 17:33:58 CEST] web3.providers.HTTPProvider: Making request HTTP. URI: https://mainnet-nethermind.blockscout.com/, Method: eth_call                                                                                                                                   
[11/09/2020 17:33:59 CEST] rotkehlchen.externalapis.etherscan: Querying etherscan: https://api.etherscan.io/api?module=logs&action=getLogs&fromBlock=9828155&toBlock=10128155&address=0x35D1b3F3D7966A1DFe207aa4514C12a259A0492B&topic0=0x760887030000000000000000000000000000000
0000000000000000000000000&topic0_1opr=and&topic1=0x4554482d41000000000000000000000000000000000000000000000000000000&topic1_2opr=and&topic2=0x000000000000000000000000cd411cffba79f6acd475e8cfaea2c69d81ad479b&topic2_3opr=and&apikey=YKABHR9XM65MI3GPK2JE3VFJAFXZUQJKNN          
[11/09/2020 17:33:59 CEST] rotkehlchen.externalapis.etherscan: Querying etherscan: https://api.etherscan.io/api?module=logs&action=getLogs&fromBlock=10128156&toBlock=10428156&address=0x35D1b3F3D7966A1DFe207aa4514C12a259A0492B&topic0=0x76088703000000000000000000000000000000
00000000000000000000000000&topic0_1opr=and&topic1=0x4554482d41000000000000000000000000000000000000000000000000000000&topic1_2opr=and&topic2=0x000000000000000000000000cd411cffba79f6acd475e8cfaea2c69d81ad479b&topic2_3opr=and&apikey=YKABHR9XM65MI3GPK2JE3VFJAFXZUQJKNN         
[11/09/2020 17:33:59 CEST] web3.providers.HTTPProvider: Getting response HTTP. URI: https://mainnet-nethermind.blockscout.com/, Method: eth_call, Response: {'jsonrpc': '2.0', 'result': '0x000000000000000000000000000000000000000000000000000000033b24902b', 'id': 23}         
[11/09/2020 17:33:59 CEST] rotkehlchen.api.rest: Greenlet for task 14 dies with exception: repay events should have a to_asset.                                                                                                                                                  
Exception Name: <class 'AssertionError'>                                                                                                                                                                                                                                         
Exception Info: repay events should have a to_asset                                                                                                                                                                                                                              
Traceback:                                                                                                                                                                                                                                                                       
   File "src/gevent/greenlet.py", line 818, in gevent._greenlet.Greenlet.run                                                                                                                                                                                                     
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/api/rest.py", line 230, in _do_query_async                                                                                                                                                                                      
    result = getattr(self, command)(**kwargs)                                                                                                                                                                                                                                    
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/api/rest.py", line 1453, in _eth_module_query                                                                                                                                                                                   
    result = getattr(module_obj, method)(**kwargs)                                                                                                                                                                                                                               
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/chain/ethereum/compound.py", line 638, in get_history                                                                                                                                                                           
    profit, loss, rewards = self._process_events(events, given_defi_balances)                                                                                                                                                                                                    
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/chain/ethereum/compound.py", line 579, in _process_events                                                                                                                                                                       
    assert event.to_asset, 'repay events should have a to_asset'                                                                                                                                                                                                                 
```

2. Negative numbers usd_value part of the profit.

## Task

Solve them


Discussion:

Resolving patch:
diff --git a/rotkehlchen/accounting/structures.py b/rotkehlchen/accounting/structures.py
--- a/rotkehlchen/accounting/structures.py
+++ b/rotkehlchen/accounting/structures.py
@@ -79,7 +79,7 @@ def __add__(self, other: Any) -> 'Balance':
         )
 
     def __sub__(self, other: Any) -> 'Balance':
-        other = _evaluate_balance_input(other, 'addition')
+        other = _evaluate_balance_input(other, 'subtraction')
         return Balance(
             amount=self.amount - other.amount,
             usd_value=self.usd_value - other.usd_value,
diff --git a/rotkehlchen/chain/ethereum/compound.py b/rotkehlchen/chain/ethereum/compound.py
--- a/rotkehlchen/chain/ethereum/compound.py
+++ b/rotkehlchen/chain/ethereum/compound.py
@@ -83,7 +83,7 @@ class CompoundEvent(NamedTuple):
     event_type: Literal['mint', 'redeem', 'borrow', 'repay', 'liquidation', 'comp']
     address: ChecksumEthAddress
     block_number: int
-    timestamp: int
+    timestamp: Timestamp
     asset: Asset
     value: Balance
     to_asset: Optional[Asset]
@@ -562,52 +562,74 @@ def _process_events(
                 assets[event.address][event.asset] -= event.value
             elif event.event_type == 'redeem':
                 assert event.to_asset, 'redeem events should have a to_asset'
-                e_profit = (
-                    assets[event.address][event.to_asset] +
-                    event.to_value -
-                    profit_so_far[event.address][event.to_asset]
+                assert event.to_value, 'redeem events should have a to_value'
+                profit_amount = (
+                    assets[event.address][event.to_asset].amount +
+                    event.to_value.amount -
+                    profit_so_far[event.address][event.to_asset].amount
                 )
-                profit = e_profit if e_profit.amount >= 0 else None  # not realized profit yet
-                if profit:
+                profit: Optional[Balance]
+                if profit_amount >= 0:
+                    usd_price = query_usd_price_zero_if_error(
+                        asset=event.to_asset,
+                        time=event.timestamp,
+                        location='comp redeem event processing',
+                        msg_aggregator=self.msg_aggregator,
+                    )
+                    profit = Balance(profit_amount, profit_amount * usd_price)
                     profit_so_far[event.address][event.to_asset] += profit
+                else:
+                    profit = None
+
                 assets[event.address][event.to_asset] += event.to_value
                 events[idx] = event._replace(realized_pnl=profit)  # TODO: maybe not named tuple?
 
             elif event.event_type == 'borrow':
                 loss_assets[event.address][event.asset] -= event.value
             elif event.event_type == 'repay':
-                assert event.to_asset, 'repay events should have a to_asset'
-                e_loss = (
-                    loss_assets[event.address][event.asset] +
-                    event.value -
-                    loss_so_far[event.address][event.to_asset]
+                loss_amount = (
+                    loss_assets[event.address][event.asset].amount +
+                    event.value.amount -
+                    loss_so_far[event.address][event.asset].amount
                 )
-                loss = e_loss if e_loss.amount >= 0 else None  # not realized loss yet
-                if loss:
-                    loss_so_far[event.address][event.to_asset] += loss
+                loss: Optional[Balance]
+                if loss_amount >= 0:
+                    usd_price = query_usd_price_zero_if_error(
+                        asset=event.asset,
+                        time=event.timestamp,
+                        location='comp repay event processing',
+                        msg_aggregator=self.msg_aggregator,
+                    )
+                    loss = Balance(loss_amount, loss_amount * usd_price)
+                    loss_so_far[event.address][event.asset] += loss
+                else:
+                    loss = None
+
                 loss_assets[event.address][event.asset] += event.value
                 events[idx] = event._replace(realized_pnl=loss)  # TODO: maybe not named tuple?
             elif event.event_type == 'liquidation':
                 assert event.to_asset, 'liquidation events should have a to_asset'
                 # Liquidator covers part of the borrowed amount
                 loss_assets[event.address][event.asset] += event.value
+                loss_so_far[event.address][event.asset] += event.value
                 # Liquidator receives discounted to_asset
                 loss_assets[event.address][event.to_asset] += event.to_value
+                loss_so_far[event.address][event.to_asset] += event.to_value
             elif event.event_type == 'comp':
                 rewards_assets[event.address][A_COMP] += event.value
 
         for address, bentry in balances.items():
             for asset, entry in bentry['lending'].items():
                 # get the underlying
-                assets[address][asset] += entry.balance
+                profit_so_far[address][asset] += entry.balance
 
             for asset, entry in bentry['borrowing'].items():
-                loss_assets[address][asset] += entry.balance
+                loss_so_far[address][asset] += entry.balance
 
             for asset, entry in bentry['rewards'].items():
                 rewards_assets[address][asset] += entry.balance
 
-        return assets, loss_assets, rewards_assets
+        return profit_so_far, loss_so_far, rewards_assets
 
     def get_history(
             self,


Test patch:
diff --git a/rotkehlchen/tests/api/test_compound.py b/rotkehlchen/tests/api/test_compound.py
--- a/rotkehlchen/tests/api/test_compound.py
+++ b/rotkehlchen/tests/api/test_compound.py
@@ -22,6 +22,7 @@
 )
 from rotkehlchen.tests.utils.constants import A_CDAI, A_CUSDC, A_USDC
 from rotkehlchen.tests.utils.rotkehlchen import setup_balances
+from rotkehlchen.typing import Timestamp
 
 TEST_ACC1 = '0x65304d6aff5096472519ca86a6a1fea31cb47Ced'
 
@@ -163,13 +164,15 @@ def test_query_compound_balances_module_not_activated(
     '0xC440f3C87DC4B6843CABc413916220D4f4FeD117',
     # For mint/redeem + comp
     '0xF59D4937BF1305856C3a267bB07791507a3377Ee',
+    # For repay
+    '0x65304d6aff5096472519ca86a6a1fea31cb47Ced',
 ]
 
 EXPECTED_EVENTS = [CompoundEvent(
     event_type='mint',
     address=deserialize_ethereum_address('0x2B888954421b424C5D3D9Ce9bB67c9bD47537d12'),
     block_number=9443573,
-    timestamp=1581184577,
+    timestamp=Timestamp(1581184577),
     asset=A_DAI,
     value=Balance(amount=FVal('2988.4343'), usd_value=FVal('3012.3417744')),
     to_asset=A_CDAI,
@@ -181,7 +184,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='redeem',
     address=deserialize_ethereum_address('0x2B888954421b424C5D3D9Ce9bB67c9bD47537d12'),
     block_number=9533397,
-    timestamp=1582378248,
+    timestamp=Timestamp(1582378248),
     asset=A_CDAI,
     value=Balance(amount=FVal('148015.6966153'), usd_value=FVal('3075.319825609865034570156')),
     to_asset=A_DAI,
@@ -191,7 +194,7 @@ def test_query_compound_balances_module_not_activated(
     ),
     realized_pnl=Balance(
         amount=FVal('8.953444259127714006'),
-        usd_value=FVal('62.978051209865034570156'),
+        usd_value=FVal('9.186233809865034570156'),
     ),
     tx_hash='0xd88138b22ef340f9ce572408b8cef10ea8df91768aee5205d5edbdb6fca76665',
     log_index=178,
@@ -199,7 +202,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='redeem',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9772026,
-    timestamp=1585558039,
+    timestamp=Timestamp(1585558039),
     asset=A_CUSDC,
     value=Balance(amount=FVal('95.18807265'), usd_value=FVal('2')),
     to_asset=A_USDC,
@@ -211,7 +214,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='borrow',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9772043,
-    timestamp=1585558230,
+    timestamp=Timestamp(1585558230),
     asset=A_ETH,
     value=Balance(amount=FVal('0.0108'), usd_value=FVal('1.428948')),
     to_asset=None,
@@ -223,7 +226,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='borrow',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9785337,
-    timestamp=1585735112,
+    timestamp=Timestamp(1585735112),
     asset=A_ETH,
     value=Balance(amount=FVal('0.0001'), usd_value=FVal('0.013605')),
     to_asset=None,
@@ -235,7 +238,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='liquidation',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9789509,
-    timestamp=1585790265,
+    timestamp=Timestamp(1585790265),
     asset=A_ETH,
     value=Balance(amount=FVal('0.005450929782112544'), usd_value=FVal('0.77179714784931510496')),
     to_asset=A_CUSDC,
@@ -247,7 +250,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='borrow',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9791832,
-    timestamp=1585820470,
+    timestamp=Timestamp(1585820470),
     asset=A_ETH,
     value=Balance(amount=FVal('0.00065'), usd_value=FVal('0.0920335')),
     to_asset=None,
@@ -259,7 +262,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='liquidation',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9793584,
-    timestamp=1585844643,
+    timestamp=Timestamp(1585844643),
     asset=A_ETH,
     value=Balance(amount=FVal('0.003050579405059551'), usd_value=FVal('0.43193153796238182609')),
     to_asset=A_CUSDC,
@@ -271,7 +274,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='borrow',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9798001,
-    timestamp=1585903297,
+    timestamp=Timestamp(1585903297),
     asset=A_ETH,
     value=Balance(amount=FVal('0.000326'), usd_value=FVal('0.04611270')),
     to_asset=None,
@@ -283,7 +286,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='liquidation',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9817257,
-    timestamp=1586159213,
+    timestamp=Timestamp(1586159213),
     asset=A_ETH,
     value=Balance(amount=FVal('0.00168867571834735'), usd_value=FVal('0.2898780738115061010')),
     to_asset=A_CUSDC,
@@ -295,7 +298,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='mint',
     address=deserialize_ethereum_address('0x2B888954421b424C5D3D9Ce9bB67c9bD47537d12'),
     block_number=9925459,
-    timestamp=1587601131,
+    timestamp=Timestamp(1587601131),
     asset=A_DAI,
     value=Balance(
         amount=FVal('1275.827510923051483475'),
@@ -310,7 +313,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='redeem',
     address=deserialize_ethereum_address('0x2B888954421b424C5D3D9Ce9bB67c9bD47537d12'),
     block_number=9937855,
-    timestamp=1587766729,
+    timestamp=Timestamp(1587766729),
     asset=A_CDAI,
     value=Balance(
         amount=FVal('62413.91974005'),
@@ -331,7 +334,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='liquidation',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9989623,
-    timestamp=1588459991,
+    timestamp=Timestamp(1588459991),
     asset=A_ETH,
     value=Balance(amount=FVal('0.0005'), usd_value=FVal('0.107075')),
     to_asset=A_CUSDC,
@@ -343,7 +346,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='liquidation',
     address=deserialize_ethereum_address('0xC440f3C87DC4B6843CABc413916220D4f4FeD117'),
     block_number=9989922,
-    timestamp=1588464109,
+    timestamp=Timestamp(1588464109),
     asset=A_ETH,
     value=Balance(amount=FVal('0.00053619065955'), usd_value=FVal('0.1126322099450730')),
     to_asset=A_CUSDC,
@@ -355,7 +358,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='mint',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10649153,
-    timestamp=1597288823,
+    timestamp=Timestamp(1597288823),
     asset=A_DAI,
     value=Balance(
         amount=FVal('738.31900592'),
@@ -370,7 +373,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='mint',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10655225,
-    timestamp=1597369409,
+    timestamp=Timestamp(1597369409),
     asset=A_DAI,
     value=Balance(
         amount=FVal('1155.06093'),
@@ -385,7 +388,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='redeem',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10661485,
-    timestamp=1597452580,
+    timestamp=Timestamp(1597452580),
     asset=A_CDAI,
     value=Balance(
         amount=FVal('91971.75759361'),
@@ -398,7 +401,7 @@ def test_query_compound_balances_module_not_activated(
     ),
     realized_pnl=Balance(
         amount=FVal('0.298272295722372195'),
-        usd_value=FVal('-1.401496960464681710855'),
+        usd_value=FVal('0.301553290975318289145'),
     ),
     tx_hash='0x2bbb296ebf1d94ad28d54c446cb23709b3463c4a43d8b5b8438ff39b2b985e1c',
     log_index=33,
@@ -406,7 +409,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='comp',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10661485,
-    timestamp=1597452580,
+    timestamp=Timestamp(1597452580),
     asset=A_COMP,
     value=Balance(
         amount=FVal('0.002931620367040859'),
@@ -424,7 +427,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='mint',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10699185,
-    timestamp=1597954197,
+    timestamp=Timestamp(1597954197),
     asset=A_DAI,
     value=Balance(
         amount=FVal('1164.9182'),
@@ -439,7 +442,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='redeem',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10701348,
-    timestamp=1597982781,
+    timestamp=Timestamp(1597982781),
     asset=A_CDAI,
     value=Balance(
         amount=FVal('56547.38087342'),
@@ -452,7 +455,7 @@ def test_query_compound_balances_module_not_activated(
     ),
     realized_pnl=Balance(
         amount=FVal('0.035281284216542725'),
-        usd_value=FVal('1.200411171921841981350'),
+        usd_value=FVal('0.035492971921841981350'),
     ),
     tx_hash='0xa931709c3f2e6a53c0cfe7e65a9b73d1e6cfe2f6a1dec907578c443d66130cb6',
     log_index=69,
@@ -460,7 +463,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='mint',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10705474,
-    timestamp=1598038125,
+    timestamp=Timestamp(1598038125),
     asset=A_DAI,
     value=Balance(
         amount=FVal('157.51085'),
@@ -475,7 +478,7 @@ def test_query_compound_balances_module_not_activated(
     event_type='redeem',
     address=deserialize_ethereum_address('0xF59D4937BF1305856C3a267bB07791507a3377Ee'),
     block_number=10712981,
-    timestamp=1598138282,
+    timestamp=Timestamp(1598138282),
     asset=A_CDAI,
     value=Balance(
         amount=FVal('7645.20779168'),
@@ -488,7 +491,7 @@ def test_query_compound_balances_module_not_activated(
     ),
     realized_pnl=Balance(
         amount=FVal('0.016634036276960755'),
-        usd_value=FVal('-0.455848611614208362735'),
+        usd_value=FVal('0.016683938385791637265'),
     ),
     tx_hash='0x667f4eb9952ffdb5141741fecb8a798f207b02adc480bb8063b055c4b10ad1dd',
     log_index=38,
@@ -502,12 +505,12 @@ def test_query_compound_balances_module_not_activated(
 @pytest.mark.parametrize('mocked_current_prices', [mocked_current_prices])
 @pytest.mark.parametrize('default_mock_price_value', [FVal(1)])
 def test_query_compound_history(rotkehlchen_api_server, ethereum_accounts):  # pylint: disable=unused-argument  # noqa: E501
-    """Check querying the compound histoy endpoint works. Uses real data"""
+    """Check querying the compound history endpoint works. Uses real data"""
     rotki = rotkehlchen_api_server.rest_api.rotkehlchen
     setup = setup_balances(
         rotki,
         ethereum_accounts=ethereum_accounts,
-        eth_balances=['1000000', '2000000', '33000030003'],
+        eth_balances=['1000000', '2000000', '33000030003', '42323213'],
         token_balances={},
         btc_accounts=None,
         original_queries=['zerion'],
@@ -533,11 +536,22 @@ def test_query_compound_history(rotkehlchen_api_server, ethereum_accounts):  # p
             result = assert_proper_response_with_result(response)
 
     assert len(result) == 4
-
     expected_events = process_result_list(EXPECTED_EVENTS)
     # Check only 22 first events, since this is how many there were in the time of
-    # the writing of the test
-    assert result['events'][:22] == expected_events
+    # the writing of the test. Also don't check events for one of the addresses
+    # as it's added later, has many events and it's only to see we handle repay correctly
+    to_check_events = [
+        x for x in result['events'] if x['address'] != '0x65304d6aff5096472519ca86a6a1fea31cb47Ced'
+    ]
+    assert to_check_events[:22] == expected_events
+    # Check one repay event
+    other_events = [
+        x for x in result['events'] if x['address'] == '0x65304d6aff5096472519ca86a6a1fea31cb47Ced'
+    ]
+    assert other_events[12]['event_type'] == 'repay'
+    expected_hash = '0x48a3e2ef8a746383deac34d74f2f0ea0451b2047701fbed4b9d769a782888eea'
+    assert other_events[12]['tx_hash'] == expected_hash
+    assert other_events[12]['value']['amount'] == '0.55064402'
 
     # Check interest profit mappings
     profit_0 = result['interest_profit']['0x2B888954421b424C5D3D9Ce9bB67c9bD47537d12']