Repository: rotki/rotki

Issue:
Utilize Cryptocompare historical price API v2
## Problem Definition

Current cryptocompare v1 historical price API has had problems of incosistent historical data. See https://github.com/rotkehlchenio/rotkehlchen/issues/377

To solve that problem cryptocompare has has a revamp of that API and introduced a [new version](https://blog.cryptocompare.com/historical-data-api-update-4ee44c549a8f)

## Task

Utilize v2 of cryptocompare historical price API.

Should also fix #377 

The task basically consists of switching to v2 of the API and slightly adjusting the response processing to that the response can be processed.

For v1 it was response -> Data, and with v2 it should be response -> Data -> Data.

In addition after this is implemented the expected values for the following tests should be adjusted and the tests be unskipped:

- https://github.com/rotkehlchenio/rotkehlchen/blob/fe9639c5eaa5d83d0efc3e2c272fdd0e4ec194ea/rotkehlchen/tests/test_cryptocompare.py#L59

- https://github.com/rotkehlchenio/rotkehlchen/blob/fe9639c5eaa5d83d0efc3e2c272fdd0e4ec194ea/rotkehlchen/tests/test_price_history.py#L11

- https://github.com/rotkehlchenio/rotkehlchen/blob/fe9639c5eaa5d83d0efc3e2c272fdd0e4ec194ea/rotkehlchen/tests/test_price_history.py#L52



Discussion:
Working on this, should have a pull request within a couple days hopefully.  Seems pretty straightforward.  For test_cryptocompare.py, do you want that unskipped?  It claims to be skipped as a duplicate of another test, but that other test doesn't seem to exist.  Also, you have some documentation in the comments about the v1 issue, do you want that kept and added to, or removed?
Hey @AndrewBezold thanks!

> For test_cryptocompare.py, do you want that unskipped? It claims to be skipped as a duplicate of another test, but that other test doesn't seem to exist

It does exist here:

https://github.com/rotkehlchenio/rotkehlchen/blob/1dea30e4578803939bd13ee240349d227fbdee18/rotkehlchen/tests/test_end_to_end_tax_report.py#L581

> Also, you have some documentation in the comments about the v1 issue, do you want that kept and added to, or removed?

I guess removed if we switch to v2.


Also regarding the unskipped tests I believe it was these ones that need to be unskipped when switching to v2: https://github.com/rotkehlchenio/rotkehlchen/blob/1dea30e4578803939bd13ee240349d227fbdee18/rotkehlchen/tests/test_price_history.py#L11

And for which the new numbers (whatever they may be) used.
Hey @AndrewBezold question. Would it be possile to have this done by Friday? If this and 2 other issues are done perhaps a release can be made before my flying to Devcon.
Yes, that should be doable.  I've got most of it finished, just working on fixing the last test.

Resolving patch:
diff --git a/rotkehlchen/externalapis/cryptocompare.py b/rotkehlchen/externalapis/cryptocompare.py
--- a/rotkehlchen/externalapis/cryptocompare.py
+++ b/rotkehlchen/externalapis/cryptocompare.py
@@ -124,11 +124,7 @@ def _api_query(self, path: str) -> Dict[str, Any]:
 
                 log.error('Cryptocompare query failure', url=querystr, error=error_message)
                 raise ValueError(error_message)
-
-            # for histohour we want all the data, including the wrapper to get TimeFrom
-            # and TimeTo
-            if 'histohour' not in path:
-                return resp['Data']
+            return resp['Data']
 
         # else not a wrapped response
         return resp
@@ -145,7 +141,7 @@ def query_endpoint_histohour(
         cc_from_asset_symbol = from_asset.to_cryptocompare()
         cc_to_asset_symbol = to_asset.to_cryptocompare()
         query_path = (
-            f'histohour?fsym={cc_from_asset_symbol}&tsym={cc_to_asset_symbol}'
+            f'v2/histohour?fsym={cc_from_asset_symbol}&tsym={cc_to_asset_symbol}'
             f'&limit={limit}&toTs={to_timestamp}'
         )
         result = self._api_query(path=query_path)


Test patch:
diff --git a/rotkehlchen/tests/test_accounting.py b/rotkehlchen/tests/test_accounting.py
--- a/rotkehlchen/tests/test_accounting.py
+++ b/rotkehlchen/tests/test_accounting.py
@@ -32,7 +32,7 @@
         'location': 'external',
     }, {
         'timestamp': 1473505138,  # cryptocompare hourly BTC/EUR price: 556.435
-        'pair': 'ETH_BTC',  # cryptocompare hourly ETH/EUR price: 10.365
+        'pair': 'ETH_BTC',  # cryptocompare hourly ETH/EUR price: 10.36
         'trade_type': 'buy',  # Buy ETH with BTC
         'rate': 0.01858275,
         'fee': 0.06999999999999999,
@@ -68,8 +68,9 @@
 @pytest.mark.parametrize('mocked_price_queries', [prices])
 def test_simple_accounting(accountant):
     accounting_history_process(accountant, 1436979735, 1495751688, history1)
-    assert accountant.general_trade_pl.is_close("557.528104903")
-    assert accountant.taxable_trade_pl.is_close("557.528104903")
+    # assert accountant.general_trade_pl.is_close("557.528104903")
+    assert accountant.general_trade_pl.is_close("557.5284549025")
+    assert accountant.taxable_trade_pl.is_close("557.5284549025")
 
 
 @pytest.mark.parametrize('mocked_price_queries', [prices])
@@ -82,8 +83,8 @@ def test_simple_accounting_with_unknown_and_unsupported_assets(accountant):
     out then"""
     history = history1 + history_unknown_assets
     accounting_history_process(accountant, 1436979735, 1495751688, history)
-    assert accountant.general_trade_pl.is_close("557.528104903")
-    assert accountant.taxable_trade_pl.is_close("557.528104903")
+    assert accountant.general_trade_pl.is_close('557.52845490257')
+    assert accountant.taxable_trade_pl.is_close('557.52845490257')
     warnings = accountant.msg_aggregator.consume_warnings()
     assert len(warnings) == 1
     assert 'found a trade containing unknown asset UNKNOWNASSET. Ignoring it.' in warnings[0]
@@ -102,7 +103,7 @@ def test_selling_crypto_bought_with_crypto(accountant):
         'location': 'external',
     }, {
         'timestamp': 1449809536,  # cryptocompare hourly BTC/EUR price: 386.175
-        'pair': 'XMR_BTC',  # cryptocompare hourly XMR/EUR price: 0.396987900
+        'pair': 'XMR_BTC',  # cryptocompare hourly XMR/EUR price: 0.39665
         'trade_type': 'buy',  # Buy XMR with BTC
         'rate': 0.0010275,
         'fee': 0.9375,
@@ -125,12 +126,12 @@ def test_selling_crypto_bought_with_crypto(accountant):
     assert len(sells) == 1
     assert sells[0].amount == FVal('0.3853125')
     assert sells[0].timestamp == 1449809536
-    assert sells[0].rate == FVal('386.175')
-    assert sells[0].fee_rate.is_close(FVal('0.96590729927'))
-    assert sells[0].gain.is_close(FVal('148.798054688'))
+    assert sells[0].rate.is_close(FVal('386.0340632603'))
+    assert sells[0].fee_rate.is_close(FVal('0.96508515815085'))
+    assert sells[0].gain.is_close(FVal('148.74375'))
 
-    assert accountant.general_trade_pl.is_close("73.8764769569")
-    assert accountant.taxable_trade_pl.is_close("73.8764769569")
+    assert accountant.general_trade_pl.is_close('73.8225270636')
+    assert accountant.taxable_trade_pl.is_close('73.8225270636')
 
 
 @pytest.mark.parametrize('mocked_price_queries', [prices])
@@ -262,16 +263,16 @@ def test_nocrypto2crypto(accountant):
 @pytest.mark.parametrize('accounting_taxfree_after_period', [None])
 def test_no_taxfree_period(accountant):
     accounting_history_process(accountant, 1436979735, 1519693374, history5)
-    assert accountant.general_trade_pl.is_close("265250.961748")
-    assert accountant.taxable_trade_pl.is_close("265250.961748")
+    assert accountant.general_trade_pl.is_close('265250.9620977')
+    assert accountant.taxable_trade_pl.is_close('265250.9620977')
 
 
 @pytest.mark.parametrize('mocked_price_queries', [prices])
 @pytest.mark.parametrize('accounting_taxfree_after_period', [86400])
 def test_big_taxfree_period(accountant):
     accounting_history_process(accountant, 1436979735, 1519693374, history5)
-    assert accountant.general_trade_pl.is_close("265250.961748")
-    assert accountant.taxable_trade_pl.is_close("0")
+    assert accountant.general_trade_pl.is_close('265250.9620977')
+    assert accountant.taxable_trade_pl.is_close('0')
 
 
 @pytest.mark.parametrize('mocked_price_queries', [prices])
@@ -384,7 +385,7 @@ def test_ignored_assets(accountant):
         'location': 'kraken',
     }]
     result = accounting_history_process(accountant, 1436979735, 1519693374, history)
-    assert FVal(result['overview']['total_taxable_profit_loss']).is_close("557.528104903")
+    assert FVal(result['overview']['total_taxable_profit_loss']).is_close('557.5284549025')
 
 
 @pytest.mark.parametrize('mocked_price_queries', [prices])
@@ -400,7 +401,7 @@ def test_settlement_buy(accountant):
         'location': 'kraken',
     }, {  # 0.0079275 * 810.49 + 0.15 * 12.4625608386372145 = 8.29454360079
         'timestamp': 1484629704,  # 17/01/2017
-        'pair': 'DASH_BTC',  # DASH/EUR price: 12.4625608386372145
+        'pair': 'DASH_BTC',  # DASH/EUR price: 12.88
         'trade_type': 'settlement_buy',  # Buy DASH with BTC to settle. Essentially BTC loss
         'rate': 0.015855,  # BTC/EUR price: 810.49
         'fee': 0.15,
@@ -425,8 +426,8 @@ def test_settlement_buy(accountant):
         history,
     )
     assert accountant.get_calculated_asset_amount('BTC').is_close('3.9920725')
-    assert FVal(result['overview']['total_taxable_profit_loss']).is_close('1932.6616152')
-    assert FVal(result['overview']['settlement_losses']).is_close('8.29454360079')
+    assert FVal(result['overview']['total_taxable_profit_loss']).is_close('1932.598999')
+    assert FVal(result['overview']['settlement_losses']).is_close('8.357159475')
 
 
 @pytest.mark.parametrize('mocked_price_queries', [prices])
diff --git a/rotkehlchen/tests/test_cryptocompare.py b/rotkehlchen/tests/test_cryptocompare.py
--- a/rotkehlchen/tests/test_cryptocompare.py
+++ b/rotkehlchen/tests/test_cryptocompare.py
@@ -58,7 +58,7 @@ def test_cryptocompare_historical_data_use_cached_price(accounting_data_dir):
 
 @pytest.mark.skip(
     'Same test as test_end_to_end_tax_report::'
-    'test_unknown_cryptocompare_asset_and_price_not_found_in_history',
+    'test_cryptocompare_asset_and_price_not_found_in_history_processing',
 )
 @pytest.mark.parametrize('use_clean_caching_directory', [True])
 @pytest.mark.parametrize('should_mock_price_queries', [False])
@@ -72,7 +72,7 @@ def test_cryptocompare_histohour_query_old_ts_xcp(
     Unfortunately still no price is found so we have to expect a NoPriceForGivenTimestamp
 
     This test is now skipped since it's a subset of:
-    test_end_to_end_tax_report::test_unknown_cryptocompare_asset_and_price_not_found_in_history
+    test_end_to_end_tax_report::test_cryptocompare_asset_and_price_not_found_in_history_processing
 
     When more price data sources are introduced then this should probably be unskipped
     to focus on the cryptocompare case. But at the moment both tests follow the same
diff --git a/rotkehlchen/tests/test_end_to_end_tax_report.py b/rotkehlchen/tests/test_end_to_end_tax_report.py
--- a/rotkehlchen/tests/test_end_to_end_tax_report.py
+++ b/rotkehlchen/tests/test_end_to_end_tax_report.py
@@ -364,20 +364,19 @@ def test_end_to_end_tax_report(accountant):
     # action seen in history before end_ts
     assert accountant.currently_processing_timestamp == 1511626623
     general_trade_pl = FVal(result['general_trade_profit_loss'])
-    assert general_trade_pl.is_close('5032.272105644')
+    assert general_trade_pl.is_close('5032.272455644')
     taxable_trade_pl = FVal(result['taxable_trade_profit_loss'])
-    assert taxable_trade_pl.is_close('3954.996589709')
+    assert taxable_trade_pl.is_close('3954.996939709')
     loan_profit = FVal(result['loan_profit'])
-    assert loan_profit.is_close('0.114027511004')
+    assert loan_profit.is_close('0.116193915')
     settlement_losses = FVal(result['settlement_losses'])
-    assert settlement_losses.is_close('11.8554392326')
+    assert settlement_losses.is_close('11.91758472725')
     asset_movement_fees = FVal(result['asset_movement_fees'])
-    assert asset_movement_fees.is_close('2.39417915')
+    assert asset_movement_fees.is_close('2.39096865')
     ethereum_transaction_gas_costs = FVal(result['ethereum_transaction_gas_costs'])
-    assert ethereum_transaction_gas_costs.is_close('2.7210025')
+    assert ethereum_transaction_gas_costs.is_close('2.736160')
     margin_pl = FVal(result['margin_positions_profit_loss'])
-    # assert margin_pl.is_close('232.95481')
-    assert margin_pl.is_close('232.7965295')
+    assert margin_pl.is_close('232.8225695')
     expected_total_taxable_pl = (
         taxable_trade_pl +
         margin_pl +
@@ -424,15 +423,15 @@ def test_end_to_end_tax_report_in_period(accountant):
     taxable_trade_pl = FVal(result['taxable_trade_profit_loss'])
     assert taxable_trade_pl.is_close('642.7084519791')
     loan_profit = FVal(result['loan_profit'])
-    assert loan_profit.is_close('0.111881296004')
+    assert loan_profit.is_close('0.1140477')
     settlement_losses = FVal(result['settlement_losses'])
-    assert settlement_losses.is_close('10.7553789375')
+    assert settlement_losses.is_close('10.81727783')
     asset_movement_fees = FVal(result['asset_movement_fees'])
-    assert asset_movement_fees.is_close('2.38526415')
+    assert asset_movement_fees.is_close('2.38205415')
     ethereum_transaction_gas_costs = FVal(result['ethereum_transaction_gas_costs'])
-    assert ethereum_transaction_gas_costs.is_close('2.2617525')
+    assert ethereum_transaction_gas_costs.is_close('2.276810')
     margin_pl = FVal(result['margin_positions_profit_loss'])
-    assert margin_pl.is_close('234.5063565')
+    assert margin_pl.is_close('234.5323965')
     expected_total_taxable_pl = (
         taxable_trade_pl +
         margin_pl +
@@ -578,7 +577,7 @@ def test_end_to_end_tax_report_in_period(accountant):
 
 @pytest.mark.parametrize('use_clean_caching_directory', [True])
 @pytest.mark.parametrize('should_mock_price_queries', [False])
-def test_cryptocompare_asset_and_price_not_found_in_history_sprocessing(accountant):
+def test_cryptocompare_asset_and_price_not_found_in_history_processing(accountant):
     """
     Make sure that in history processing if no price is found for a trade it's skipped
     and an error is logged. Same for price query of an unknown asset.
diff --git a/rotkehlchen/tests/test_price_history.py b/rotkehlchen/tests/test_price_history.py
--- a/rotkehlchen/tests/test_price_history.py
+++ b/rotkehlchen/tests/test_price_history.py
@@ -8,10 +8,9 @@
 from rotkehlchen.tests.utils.history import prices
 
 
-@pytest.mark.skip("https://github.com/rotkehlchenio/rotkehlchen/issues/377")
 @pytest.mark.parametrize('should_mock_price_queries', [False])
-def test_incosistent_prices_double_checking(price_historian):
-    """ This is a regression test for the incosistent DASH/EUR and DASH/USD prices
+def test_inconsistent_prices_double_checking(price_historian):
+    """ This is a regression test for the inconsistent DASH/EUR and DASH/USD prices
     that were returned on 02/12/2018. Issue:
     https://github.com/rotkehlchenio/rotkehlchen/issues/221
     """
@@ -19,20 +18,14 @@ def test_incosistent_prices_double_checking(price_historian):
     # Note the prices are not the same as in the issue because rotkehlchen uses
     # hourly rates while in the issue we showcased query of daily average
     usd_price = price_historian.query_historical_price(A_DASH, A_USD, 1479200704)
-    assert usd_price.is_close(FVal('9.63'))
+    assert usd_price.is_close(FVal('9.6125'))
     eur_price = price_historian.query_historical_price(A_DASH, A_EUR, 1479200704)
-    # 13/01/19 Cryptocompare fixed the error so our incosistency adjustment code that would
-    # give the price of 8.945657222 is not hit.
-    # 16/01/2019 They just reintroduced the error. So 9.0 is no longer returned
-    assert eur_price.is_close(FVal('8.945657222'), max_diff=0.001)
+    assert eur_price.is_close(FVal('9.0015'), max_diff=0.001)
 
     inv_usd_price = price_historian.query_historical_price(A_USD, A_DASH, 1479200704)
-    assert inv_usd_price.is_close(FVal('0.103842'), max_diff=0.0001)
+    assert inv_usd_price.is_close(FVal('0.10385'), max_diff=0.0001)
     inv_eur_price = price_historian.query_historical_price(A_EUR, A_DASH, 1479200704)
-    # 13/01/19 Cryptocompare fixed the error so our incosistency adjustment code
-    # that would give the price of 0.11179 is not hit
-    # 16/01/2019 They just reintroduced the error. So 0.11115 is no longer returned
-    assert inv_eur_price.is_close(FVal('0.11179'), max_diff=0.0001)
+    assert inv_eur_price.is_close(FVal('0.1111'), max_diff=0.0001)
 
 
 def do_queries_for(from_asset, to_asset, price_historian):
@@ -49,7 +42,6 @@ def do_queries_for(from_asset, to_asset, price_historian):
         assert price.is_close(pair_map[timestamp]), msg
 
 
-@pytest.mark.skip("https://github.com/rotkehlchenio/rotkehlchen/issues/377")
 @pytest.mark.parametrize('should_mock_price_queries', [False])
 def test_price_queries(price_historian):
     """Test some cryptocompare price queries making sure our querying mechanism works"""
diff --git a/rotkehlchen/tests/utils/history.py b/rotkehlchen/tests/utils/history.py
--- a/rotkehlchen/tests/utils/history.py
+++ b/rotkehlchen/tests/utils/history.py
@@ -24,7 +24,7 @@
 TEST_END_TS = 1559427707
 
 
-# Prices queried by cryptocompare  @ 12/01/2019
+# Prices queried by cryptocompare  @ 02/10/2019
 prices = {
     'BTC': {
         'EUR': {
@@ -43,56 +43,39 @@
             1487289600: FVal(979.39),
             1491177600: FVal(1039.935),
             1495969504: FVal(1964.685),
-            1498694400: FVal(2244.255),
+            1498694400: FVal(2244.465),
             1512693374: FVal(14415.365),
         },
     },
     'ETH': {
         'EUR': {
             1446979735: FVal(0.8583),
-            1463184190: FVal(9.185),
+            1463184190: FVal(9.187),
             1463508234: FVal(10.785),
-            1473505138: FVal(10.365),
+            1473505138: FVal(10.36),
             1475042230: FVal(11.925),
             1476536704: FVal(10.775),
-            1479510304: FVal(8.915),
-            1491062063: FVal(47.5),
-            1493291104: FVal(52.885),
-            1511626623: FVal(393.955),
+            1479510304: FVal(8.9145),
+            1491062063: FVal(47.865),
+            1493291104: FVal(53.175),
+            1511626623: FVal(396.56),
         },
     },
     'XMR': {
         'EUR': {
-            1449809536: FVal(0.396987900),  # BTC adjusted price
+            1449809536: FVal(0.39665),
         },
     },
     'DASH': {
         'EUR': {
-            # TODO: Switch to the DASH non-usd adjusted prices since cryptocompare
-            # starting returning correct results and adjust the tests accordingly
-            # 1479200704: FVal(9),
-            1479200704: FVal(8.9456),  # old USD adjusted price
-
-            # 1480683904: FVal(8.155),
-            1480683904: FVal(8.104679571509114828039),  # old USD adjusted price
-
-            # 1483351504: FVal(11.115),
-            1483351504: FVal(10.9698996),  # old USD adjusted price
-
-            # 1484629704: FVal(12.89),  # found in historical hourly
-            1484629704: FVal(12.4625608386372145),  # old USD adjusted price
-
-            # 1485252304: FVal(13.48),
-            1485252304: FVal(13.22106438),  # old USD adjusted price
-
-            # 1486299904: FVal(15.29),
-            1486299904: FVal(15.36169816590634019),  # old USD adjusted price
-
-            # 1487027104: FVal(16.08)
-            1487027104: FVal(15.73995672),  # old USD adjusted price
-
-            # 1502715904: FVal(173.035),
-            1502715904: FVal(173.77),  # old USD adjusted price
+            1479200704: FVal(9.0015),
+            1480683904: FVal(8.154),
+            1483351504: FVal(11.115),
+            1484629704: FVal(12.88),
+            1485252304: FVal(13.48),
+            1486299904: FVal(15.29),
+            1487027104: FVal(16.08),
+            1502715904: FVal(173.035),
         },
     },
 }