Repository: rotki/rotki

Issue:
PnL reports "incomplete cost basis for spend" on trades based on profit currency
## Problem Definition

The PnL report throws the following message when computing cost basis related to 5 USD->ADA buys and 1 ADA->SOL sell. Investigating further it looks like the issue may be related to rates derived from the price oracle rather than being derived from the trades themselves.

> Not enough documented acquisitions found for ADA(Cardano) before 12/10/2021 08:23:56. Only found acquisitions for 37004.53663936611576641883254 ADA(Cardano) and miss 729.49792163388423358116746 ADA(Cardano).Let rotki know how you acquired it via a ledger action

> Incomplete cost basis information for spend. Used: 2677.374566366115766418832538 / 2951.286057 acquired in coinbase at 15/05/2021 11:59:18|4123.161526 / 4123.161526 acquired in coinbase at 19/05/2021 11:13:25|4125.287831 / 4125.287831 acquired in coinbase at 20/05/2021 07:22:16|26078.712716 / 26078.712716 acquired in coinbase at 20/05/2021 07:32:58

| type | location | paid_asset                | paid_in_asset                 | taxable_amount | received_asset | received_in_asset             | net_profit_or_loss      | time                | is_virtual | paid_in_USD                   | taxable_received_in_USD       | taxable_bought_cost_in_USD    | cost_basis                                                                                                                                                                                                                                                                                                                                                        | total_bought_cost_in_USD      | total_received_in_USD         | link                                 | notes | 
|------|----------|---------------------------|-------------------------------|----------------|----------------|-------------------------------|-------------------------|---------------------|------------|-------------------------------|-------------------------------|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|-------------------------------|--------------------------------------|-------| 
| buy  | coinbase | USD(United States Dollar) | 1050.540000000000000000000000 | 0              | ADA(Cardano)   | 455.586431                    | 0                       | 15/05/2021 11:56:48 | FALSE      | 1066.190000000000000000000000 | 0                             | 0                             |                                                                                                                                                                                                                                                                                                                                                                   | 0                             | 0                             | aecb3909-42d1-53b8-98fd-9e6b5ba9db26 |       | 
| buy  | coinbase | USD(United States Dollar) | 6897.230000000000000000000000 | 0              | ADA(Cardano)   | 2951.286057                   | 0                       | 15/05/2021 11:59:18 | FALSE      | 7000.000000000000000000000000 | 0                             | 0                             |                                                                                                                                                                                                                                                                                                                                                                   | 0                             | 0                             | 20cb0018-08fd-5525-8497-79abe0d5da98 |       | 
| buy  | coinbase | USD(United States Dollar) | 6897.230000000000000000000000 | 0              | ADA(Cardano)   | 4123.161526                   | 0                       | 19/05/2021 11:13:25 | FALSE      | 7000.000000000000000000000000 | 0                             | 0                             |                                                                                                                                                                                                                                                                                                                                                                   | 0                             | 0                             | 46503731-b033-517c-b9db-36e7cb6fc7ed |       | 
| buy  | coinbase | USD(United States Dollar) | 7640.920000000000000000000000 | 0              | ADA(Cardano)   | 4125.287831                   | 0                       | 20/05/2021 07:22:16 | FALSE      | 7754.770000000000000000000000 | 0                             | 0                             |                                                                                                                                                                                                                                                                                                                                                                   | 0                             | 0                             | 559d2b26-e4dc-5ac6-a3bc-e09777653e48 |       | 
| buy  | coinbase | USD(United States Dollar) | 49265.94000000000000000000000 | 0              | ADA(Cardano)   | 26078.712716                  | 0                       | 20/05/2021 07:32:58 | FALSE      | 50000.00000000000000000000000 | 0                             | 0                             |                                                                                                                                                                                                                                                                                                                                                                   | 0                             | 0                             | 186ae115-5aa7-5b50-89a5-2af158171d00 |       | 
| sell | coinbase | ADA(Cardano)              | 37734.034561                  | 37734.034561   | SOL(Solana)    | 552.3158858360000000000000000 | =IF(E218=0;0;L218-M218) | 12/10/2021 08:23:56 | FALSE      | 0                             | 76645.34552515690000000000000 | 71105.09375804800000000000000 | Incomplete cost basis information for spend. Used: 2677.374566366115766418832538 / 2951.286057 acquired in coinbase at 15/05/2021 11:59:18\|4123.161526 / 4123.161526 acquired in coinbase at 19/05/2021 11:13:25\|4125.287831 / 4125.287831 acquired in coinbase at 20/05/2021 07:22:16\|26078.712716 / 26078.712716 acquired in coinbase at 20/05/2021 07:32:58 | 72648.34661126450000000000000 | 78188.59837837330000000000000 | d97ccd41-13ab-51b7-83d1-2ef0db3e7a03 |       | 
| buy  | coinbase | ADA(Cardano)              | 37734.03456100000000000000000 | 0              | SOL(Solana)    | 552.3158858360000000000000000 | 0                       | 12/10/2021 08:23:56 | TRUE       | 79826.35011379550000000000000 | 0                             | 0                             |                                                                                                                                                                                                                                                                                                                                                                   | 0                             | 0                             | d97ccd41-13ab-51b7-83d1-2ef0db3e7a03 |       | 

## Logs

> [15/01/2022 19:34:15 CST] DEBUG rotkehlchen.history.price Greenlet-6: Querying historical price from_asset=SOL(Solana), to_asset=USD(United States Dollar), timestamp=1634045036
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.externalapis.cryptocompare Greenlet-6: Will query Cryptocompare history for SOL-2 -> USD @ 1634045036. Cached data: True rate_limited in last 60 seconds: False
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.externalapis.cryptocompare Greenlet-6: Got historical price from cryptocompare from_asset=SOL(Solana), to_asset=USD(United States Dollar), timestamp=1634045036, price=0
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.history.price Greenlet-6: Historical price oracle cryptocompare got price price=141.565, from_asset=SOL(Solana), to_asset=USD(United States Dollar), timestamp=1634045036
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.history.price Greenlet-6: Querying historical price from_asset=ADA(Cardano), to_asset=USD(United States Dollar), timestamp=1634045036
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.externalapis.cryptocompare Greenlet-6: Will query Cryptocompare history for ADA -> USD @ 1634045036. Cached data: True rate_limited in last 60 seconds: False
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.externalapis.cryptocompare Greenlet-6: Got historical price from cryptocompare from_asset=ADA(Cardano), to_asset=USD(United States Dollar), timestamp=1634045036, price=0
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.history.price Greenlet-6: Historical price oracle cryptocompare got price price=2.1155, from_asset=ADA(Cardano), to_asset=USD(United States Dollar), timestamp=1634045036
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.events Greenlet-6: Processing sell trade of ADA for SOL-2 at 1634045036
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.cost_basis Greenlet-6: AssetSpendEvent in coinbase @ 1634045036.amount: 37734.034561 rate: 2.072097492039325744126330557 fee_rate: 0.04089816716316655456462785440 gain_in_profit_currency: 78188.59837837333999999999999
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.cost_basis Greenlet-6: Spend uses up entire historical acquisition tax_status=TAXABLE, bought_amount=455.586431, asset=ADA(Cardano), acquisition_rate=2.305907130934722680535672056, profit_currency=USD(United States Dollar), time=15/05/2021 11:56:48
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.cost_basis Greenlet-6: Spend uses up part of historical acquisition tax_status=TAXABLE, used_amount=273.9114906338842335811674623, from_amount=2951.286057, asset=ADA(Cardano), acquisition_rate=2.337025238079115825945163539, profit_currency=USD(United States Dollar), time=15/05/2021 11:59:18
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.cost_basis Greenlet-6: Spend uses up entire historical acquisition tax_status=TAXABLE, bought_amount=2677.374566366115766418832538, asset=ADA(Cardano), acquisition_rate=2.337025238079115825945163539, profit_currency=USD(United States Dollar), time=15/05/2021 11:59:18
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.cost_basis Greenlet-6: Spend uses up entire historical acquisition tax_status=TAXABLE, bought_amount=4123.161526, asset=ADA(Cardano), acquisition_rate=1.672801309506592441947422266, profit_currency=USD(United States Dollar), time=19/05/2021 11:13:25
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.cost_basis Greenlet-6: Spend uses up entire historical acquisition tax_status=TAXABLE, bought_amount=4125.287831, asset=ADA(Cardano), acquisition_rate=1.852215000025291568558189170, profit_currency=USD(United States Dollar), time=20/05/2021 07:22:16
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.cost_basis Greenlet-6: Spend uses up entire historical acquisition tax_status=TAXABLE, bought_amount=26078.712716, asset=ADA(Cardano), acquisition_rate=1.889124687115940542806967282, profit_currency=USD(United States Dollar), time=20/05/2021 07:32:58
[15/01/2022 19:34:15 CST] ERROR rotkehlchen.user_messages Greenlet-6: Not enough documented acquisitions found for ADA(Cardano) before 12/10/2021 08:23:56. Only found acquisitions for 37004.53663936611576641883254 ADA(Cardano) and miss 729.49792163388423358116746 ADA(Cardano).Let rotki know how you acquired it via a ledger action
[15/01/2022 19:34:15 CST] DEBUG rotkehlchen.accounting.events Greenlet-6: After Sell Profit/Loss taxable_profit_loss=5540.25176710888960929265968, general_profit_loss=5540.25176710888960929265968, profit_currency=USD(United States Dollar)

<img width="742" alt="Rotki Rates" src="https://user-images.githubusercontent.com/882552/149674632-9eb22ebf-ddd2-4b63-99e3-1b2ba8835dba.png">

<img width="466" alt="cointracker.io rates" src="https://user-images.githubusercontent.com/882552/149674631-a624cbb8-32b0-492d-a794-becbb5b0676b.png">

[related_events.csv](https://github.com/rotki/rotki/files/7877736/related_events.csv)

### System Description


Operating system: macOS Big Sur 11.6 
Rotki version: 1.23.1



Discussion:
I think I made sense of the error notification reported in the app:
> Not enough documented acquisitions found for ADA(Cardano) before 12/10/2021 08:23:56. Only found acquisitions for 37004.53663936611576641883254 ADA(Cardano) and miss 729.49792163388423358116746 ADA(Cardano).Let rotki know how you acquired it via a ledger action

The 729.49792163388423358116746 ADA(Cardano) corresponds to the fee associated with the trade from ADA to SOL. I confirmed this by looking at the entries in the trades table. Should this be accounted for in the documented acquisitions?

id | time | location | base_asset | quote_asset | type | amount | rate | fee | fee_currency | link | notes
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
e3bca1667d120bb37599b7958fd591060dec96cf422fb003d52da395bf084bcd | 1621513978 | G | ADA | USD | A | 26078.712716 | 1.889124687115940542806967282 | 734.06 | USD | 186ae115-5aa7-5b50-89a5-2af158171d00 |  
7d80409a3822944376e06c23a26aa3e181e09786276b65b44c926520f0d769f2 | 1621513336 | G | ADA | USD | A | 4125.287831 | 1.852215000025291568558189170 | 113.85 | USD | 559d2b26-e4dc-5ac6-a3bc-e09777653e48 |  
0b7f228eb64cb4ce2b7a20b76cadaa17c29e9acdebe6133cbedfae17966dd4a0 | 1621440805 | G | ADA | USD | A | 4123.161526 | 1.672801309506592441947422266 | 102.77 | USD | 46503731-b033-517c-b9db-36e7cb6fc7ed |  
99aeac90a50a41f229b86a7417b7040edcdcb195078ba7c1db19c25f3ee70ab4 | 1621097958 | G | ADA | USD | A | 2951.286057 | 2.337025238079115825945163539 | 102.77 | USD | 20cb0018-08fd-5525-8497-79abe0d5da98 |  
22a15578cd16fdc5165bb23ccc98278fcf87a34da10f96a2ad60d2a8b2abc427 | 1621097808 | G | ADA | USD | A | 455.586431 | 2.305907130934722680535672056 | 15.65 | USD | aecb3909-42d1-53b8-98fd-9e6b5ba9db26 |  
04d0b32b111e3beb995fb6bd10922eeddeb1fe31ea79d6b8ab063b8aec6a14f7 | 1634045036 | G | ADA | SOL-2 | B | 37734.034561 | 0.01463707478571204566189616471 | 729.4979216338842335811674623 | ADA | d97ccd41-13ab-51b7-83d1-2ef0db3e7a03 |  

The discrepancy in the rates between rotki and cointracking turns out to be that rotki computes the rate excluding the fee ((amount * rate) + fee = USD spent) while cointracking computes including the fee ((amount + fee) *  rate = USD spent).

I'm still trying to make sense of the other message related to cost basis.


I determined that the cost basis error stems from the fact that acquisitions are deducted for the fee_amount. However, when computing the remaining acquisitions this fee_amount is not deducted from the selling_amount. This causes the acquisition to be short by the fee_amount as described in the error message:

> Not enough documented acquisitions found for ADA(Cardano) before 12/10/2021 08:23:56. Only found acquisitions for 37004.53663936611576641883254 ADA(Cardano) and miss 729.49792163388423358116746 ADA(Cardano).Let rotki know how you acquired it via a ledger action

I tried the following and for this case it seemed to have resolved the issue. I'm still validating this atm but thought i'd share what i've found so far.

```diff
diff --git a/rotkehlchen/accounting/events.py b/rotkehlchen/accounting/events.py
index 1892df4fe..28fddfc01 100644
--- a/rotkehlchen/accounting/events.py
+++ b/rotkehlchen/accounting/events.py
@@ -583,6 +583,9 @@ class TaxableEvents():
                 spending_asset=fee_currency,
                 timestamp=timestamp,
             )
+            if fee_currency == selling_asset:
+                # deduct the fee_amount
+                selling_amount = selling_amount - fee_amount
         self.handle_prefork_asset_sells(selling_asset, selling_amount, timestamp)
 
         # now search the acquisitions for `paid_with_asset` and calculate profit/loss
```
> I determined that the cost basis error stems from the fact that acquisitions are deducted for the fee_amount. However, when computing the remaining acquisitions this fee_amount is not deducted from the selling_amount. This causes the acquisition to be short by the fee_amount as described in the error message:
> 
> > Not enough documented acquisitions found for ADA(Cardano) before 12/10/2021 08:23:56. Only found acquisitions for 37004.53663936611576641883254 ADA(Cardano) and miss 729.49792163388423358116746 ADA(Cardano).Let rotki know how you acquired it via a ledger action
> 
> I tried the following and for this case it seemed to have resolved the issue. I'm still validating this atm but thought i'd share what i've found so far.
> 
> ```diff
> diff --git a/rotkehlchen/accounting/events.py b/rotkehlchen/accounting/events.py
> index 1892df4fe..28fddfc01 100644
> --- a/rotkehlchen/accounting/events.py
> +++ b/rotkehlchen/accounting/events.py
> @@ -583,6 +583,9 @@ class TaxableEvents():
>                  spending_asset=fee_currency,
>                  timestamp=timestamp,
>              )
> +            if fee_currency == selling_asset:
> +                # deduct the fee_amount
> +                selling_amount = selling_amount - fee_amount
>          self.handle_prefork_asset_sells(selling_asset, selling_amount, timestamp)
>  
>          # now search the acquisitions for `paid_with_asset` and calculate profit/loss
> ```

Hola @idelgado ! The situation you want to fix in this comment was recently handled by https://github.com/rotki/rotki/pull/3933/ . 
This might be a special case, working on this issue now
Wrote you a DM on discord @idelgado !

Resolving patch:
diff --git a/rotkehlchen/exchanges/coinbase.py b/rotkehlchen/exchanges/coinbase.py
--- a/rotkehlchen/exchanges/coinbase.py
+++ b/rotkehlchen/exchanges/coinbase.py
@@ -132,14 +132,34 @@ def trade_from_conversion(trade_a: Dict[str, Any], trade_b: Dict[str, Any]) -> O
     amount_before_fee = deserialize_asset_amount(trade_a['native_amount']['amount'])
     # amount_after_fee + amount_before_fee is a negative amount and the fee needs to be positive
     conversion_native_fee_amount = abs(amount_after_fee + amount_before_fee)
-    if ZERO not in (tx_amount, conversion_native_fee_amount, amount_before_fee):
-        # We have the fee amount in the native currency. To get it in the
-        # converted asset we have to get the rate
-        asset_native_rate = tx_amount / abs(amount_before_fee)
-        fee_amount = Fee(conversion_native_fee_amount * asset_native_rate)
+    if ZERO not in (tx_amount, conversion_native_fee_amount, amount_before_fee, amount_after_fee):
+        # To get the asset in wich the fee is nominated we pay attention to the creation
+        # date of each event. As per our hypothesis the fee is nominated in the asset
+        # for which the first transaction part was initialized
+        time_created_a = deserialize_timestamp_from_date(
+            date=trade_a['created_at'],
+            formatstr='iso8601',
+            location='coinbase',
+        )
+        time_created_b = deserialize_timestamp_from_date(
+            date=trade_b['created_at'],
+            formatstr='iso8601',
+            location='coinbase',
+        )
+        if time_created_a < time_created_b:
+            # We have the fee amount in the native currency. To get it in the
+            # converted asset we have to get the rate
+            asset_native_rate = tx_amount / abs(amount_before_fee)
+            fee_amount = Fee(conversion_native_fee_amount * asset_native_rate)
+            fee_asset = asset_from_coinbase(trade_a['amount']['currency'], time=timestamp)
+        else:
+            trade_b_amount = abs(deserialize_asset_amount(trade_b['amount']['amount']))
+            asset_native_rate = trade_b_amount / abs(amount_after_fee)
+            fee_amount = Fee(conversion_native_fee_amount * asset_native_rate)
+            fee_asset = asset_from_coinbase(trade_b['amount']['currency'], time=timestamp)
     else:
         fee_amount = Fee(ZERO)
-    fee_asset = asset_from_coinbase(trade_a['amount']['currency'], time=timestamp)
+        fee_asset = asset_from_coinbase(trade_a['amount']['currency'], time=timestamp)
 
     return Trade(
         timestamp=timestamp,


Test patch:
diff --git a/rotkehlchen/tests/exchanges/test_coinbase.py b/rotkehlchen/tests/exchanges/test_coinbase.py
--- a/rotkehlchen/tests/exchanges/test_coinbase.py
+++ b/rotkehlchen/tests/exchanges/test_coinbase.py
@@ -1088,7 +1088,7 @@ def test_asset_conversion():
             "currency": "USD",
         },
         "description": None,
-        "created_at": "2020-06-08T02:32:16Z",
+        "created_at": "2020-06-08T02:32:15Z",
         "updated_at": "2021-06-08T02:32:16Z",
         "resource": "transaction",
         "resource_path": "/v2/accounts/sd5af/transactions/77c5ad72-764e-414b-8bdb-b5aed20fb4b1",
@@ -1170,7 +1170,7 @@ def test_asset_conversion_not_stable_coin():
             "currency": "USD",
         },
         "description": None,
-        "created_at": "2020-06-08T02:32:16Z",
+        "created_at": "2020-06-08T02:32:15Z",
         "updated_at": "2021-06-08T02:32:16Z",
         "resource": "transaction",
         "resource_path": "/v2/accounts/sd5af/transactions/77c5ad72-764e-414b-8bdb-b5aed20fb4b1",
@@ -1239,7 +1239,6 @@ def test_asset_conversion_not_stable_coin():
 
 def test_asset_conversion_zero_fee():
     """Test a conversion with 0 fee"""
-
     trade_a = {
         "id": "77c5ad72-764e-414b-8bdb-b5aed20fb4b1",
         "type": "trade",
@@ -1318,3 +1317,90 @@ def test_asset_conversion_zero_fee():
         link='5dceef97-ef34-41e6-9171-3e60cd01639e',
     )
     assert trade == expected_trade
+
+
+def test_asset_conversion_choosing_fee_asset():
+    """Test that the fee asset is correctly choosen when the received asset transaction
+    is created before the giving transaction.
+    """
+
+    trade_a = {
+        "id": "REDACTED",
+        "type": "trade",
+        "status": "completed",
+        "amount": {
+            "amount": "-37734.034561",
+            "currency": "ETH",
+        },
+        "native_amount": {
+            "amount": "-79362.22",
+            "currency": "USD",
+        },
+        "description": None,
+        "created_at": "2021-10-12T13:23:56Z",
+        "updated_at": "2021-10-12T13:23:56Z",
+        "resource": "transaction",
+        "resource_path": "/v2/accounts/REDACTED/transactions/REDACTED",
+        "instant_exchange": False,
+        "trade": {
+            "id": "id_of_trade",
+            "resource": "trade",
+            "resource_path": "/v2/accounts/REDACTED/trades/id_of_trade",
+        },
+        "details": {
+            "title": "Converted from ETH",
+            "subtitle": "Using ETH Wallet",
+            "header": "Converted 37,734.034561 ETH ($79,362.22)",
+            "health": "positive",
+            "payment_method_name": "ETH Wallet",
+        },
+        "hide_native_amount": False,
+    }
+
+    trade_b = {
+        "id": "REDACTED",
+        "type": "trade",
+        "status": "completed",
+        "amount": {
+            "amount": "552.315885836",
+            "currency": "BTC",
+        },
+        "native_amount": {
+            "amount": "77827.94",
+            "currency": "USD",
+        },
+        "description": None,
+        "created_at": "2021-10-12T13:23:55Z",
+        "updated_at": "2021-10-12T13:23:57Z",
+        "resource": "transaction",
+        "resource_path": "/v2/accounts/REDACTED/transactions/REDACTED",
+        "instant_exchange": False,
+        "trade": {
+            "id": "id_of_trade",
+            "resource": "trade",
+            "resource_path": "/v2/accounts/REDACTED/trades/id_of_trade",
+        },
+        "details": {
+            "title": "Converted to BTC",
+            "subtitle": "Using ETH Wallet",
+            "header": "Converted 552.31588584 BTC ($77,827.94)",
+            "health": "positive",
+            "payment_method_name": "ETH Wallet",
+        },
+        "hide_native_amount": False,
+    }
+
+    trade = trade_from_conversion(trade_a, trade_b)
+    expected_trade = Trade(
+        timestamp=1634045036,
+        location=Location.COINBASE,
+        base_asset=A_ETH,
+        quote_asset=A_BTC,
+        trade_type=TradeType.SELL,
+        amount=FVal('37734.034561'),
+        rate=FVal('0.01463707478571204566189616471'),
+        fee=FVal('10.88821337581925051594581586'),
+        fee_currency=A_BTC,
+        link='id_of_trade',
+    )
+    assert trade == expected_trade