internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rotki/rotki

Issue:
No user feedback if saving snapshot task fails
## Abstract

If the task to save the snapshot (either the automatic one or the "Force Save" one) fails for whatever reason, let's say an unhandled exception there is an entry in the error logs but there is no user feedback. No messages sent via the `msg_aggregator`.

## Task

The task retrieval should show either success or failure, when the frontend queries for the task. So I think the frontend should be able to show a message when either the saving balances snapshot from "Force Save" or the scheduled one fails. 


Discussion:
@LefterisJP I am guessing that this is not a frontend issue since it includes the `msg_aggregator` should I update the issue to indicate this is a backend issue?
> The task retrieval should show either success or failure, when the frontend queries for the task. So I think the frontend should be able to show a message when either the saving balances snapshot from "Force Save" or the scheduled one fails.

I was thinking more of a message in the UI. But can also be a message by the msg_aggregator. Have not thought of this issue in detail yet.
I am also adding the backend label since we need to provide a way for the backend to notify the frontend when the snapshot fails.
We can do so probably via websockets?
Certainly, that would make sense. 
Backend part done by https://github.com/rotki/rotki/pull/3680

Now need frontend + changelog. So assigning to @kelsos 

Resolving patch:
diff --git a/rotkehlchen/api/websockets/notifier.py b/rotkehlchen/api/websockets/notifier.py
--- a/rotkehlchen/api/websockets/notifier.py
+++ b/rotkehlchen/api/websockets/notifier.py
@@ -71,6 +71,7 @@ def broadcast(
         message_data = {'type': str(message_type), 'data': to_send_data}
         message = json.dumps(message_data)  # TODO: Check for dumps error
         to_remove_indices = set()
+        spawned_one_broadcast = False
         for idx, websocket in enumerate(self.subscribers):
             if websocket.closed is True:
                 to_remove_indices.add(idx)
@@ -88,11 +89,15 @@ def broadcast(
                 failure_callback=failure_callback,
                 failure_callback_args=failure_callback_args,
             )
+            spawned_one_broadcast = True
 
         if len(to_remove_indices) != 0:  # removed closed websockets from the list
             self.subscribers = [
                 i for j, i in enumerate(self.subscribers) if j not in to_remove_indices
             ]
+        if spawned_one_broadcast is False and failure_callback is not None:
+            failure_callback_args = {} if failure_callback_args is None else failure_callback_args  # noqa: E501
+            failure_callback(**failure_callback_args)
 
 
 class RotkiWSApp(WebSocketApplication):


Test patch:
Repository: rotki/rotki

Issue:
Add support for the paraswap airdrop
More information at 

https://medium.com/paraswap/introducing-the-psp-token-to-make-paraswap-even-more-efficient-and-decentralized-c4730e50be72


Discussion:
This seems to be the list of users https://raw.githubusercontent.com/paraswap/paraswap-rewards-snapshot/master/data/airdrop-users.json
Also @banteg made a CSV with the right format ready to be put into our [data repo](https://github.com/rotki/data): https://gist.github.com/banteg/f3a1bdb73ec49689823d20e38387a696

Should be really easy for anyone to do this. The only work required is seen in the similar ens airdrop PR: https://github.com/rotki/rotki/pull/3690

That PR also contains unrelated changes (which should be pretty obvious)
@yabirgb I'd like to take a crack at this. Is there a deadline or just asap?
Hello @Aragar199! We want it for next release but it won't by before 1-2 weeks I believe. Just work on it as soon as you can :)
Also you will probably need to have the PSP asset added. But you can work on this assuming PSP is added and we can add it in another PR.
Understood! Seems straightforward enough, you can assign it to me. I'm in the discord as well.

Resolving patch:
diff --git a/rotkehlchen/chain/ethereum/airdrops.py b/rotkehlchen/chain/ethereum/airdrops.py
--- a/rotkehlchen/chain/ethereum/airdrops.py
+++ b/rotkehlchen/chain/ethereum/airdrops.py
@@ -18,6 +18,7 @@
     A_FOX,
     A_GRAIN,
     A_LDO,
+    A_PSP,
     A_TORN,
     A_UNI,
 )
@@ -95,6 +96,11 @@
         A_ENS,
         'https://claim.ens.domains/',
     ),
+    'psp': (
+        'https://raw.githubusercontent.com/rotki/data/main/airdrops/psp.csv',
+        A_PSP,
+        'https://paraswap.io/',
+    ),
 }
 
 POAP_AIRDROPS = {
diff --git a/rotkehlchen/constants/assets.py b/rotkehlchen/constants/assets.py
--- a/rotkehlchen/constants/assets.py
+++ b/rotkehlchen/constants/assets.py
@@ -2209,3 +2209,15 @@
     protocol=None,
 )
 CONSTANT_ASSETS.append(A_ENS)
+A_PSP = EthereumToken.initialize(
+    address=string_to_ethereum_address('0xcAfE001067cDEF266AfB7Eb5A286dCFD277f3dE5'),
+    decimals=18,
+    name='ParaSwap',
+    symbol='PSP',
+    started=Timestamp(1636966698),
+    swapped_for=None,
+    coingecko='paraswap',
+    cryptocompare=None,
+    protocol=None,
+)
+CONSTANT_ASSETS.append(A_PSP)


Test patch:
Repository: rotki/rotki

Issue:
Nexo Fixed term deposit is imported as a duplicate deposit
Specifically, entries that have a value of `LockingTermDeposit` are interpreted as another deposit. But that actually only means that the coins are locked for the chosen duration.


Discussion:
Hello @AdalLS! thanks for reporting it. Will see why and fix it :)

Resolving patch:
diff --git a/rotkehlchen/data/importer.py b/rotkehlchen/data/importer.py
--- a/rotkehlchen/data/importer.py
+++ b/rotkehlchen/data/importer.py
@@ -980,7 +980,7 @@ def _consume_nexo(self, csv_row: Dict[str, Any], **kwargs: Any) -> None:
                 'since not enough information is provided about the trade.',
             )
             return
-        if entry_type in ('Deposit', 'ExchangeDepositedOn', 'LockingTermDeposit'):
+        if entry_type in ('Deposit', 'ExchangeDepositedOn'):
             asset_movement = AssetMovement(
                 location=Location.NEXO,
                 category=AssetMovementCategory.DEPOSIT,


Test patch:
diff --git a/rotkehlchen/tests/utils/dataimport.py b/rotkehlchen/tests/utils/dataimport.py
--- a/rotkehlchen/tests/utils/dataimport.py
+++ b/rotkehlchen/tests/utils/dataimport.py
@@ -784,17 +784,6 @@ def assert_nexo_results(rotki: Rotkehlchen):
         fee_asset=A_USD,
         fee=Fee(ZERO),
         link='NXT0000000003',
-    ), AssetMovement(
-        location=Location.NEXO,
-        category=AssetMovementCategory.DEPOSIT,
-        timestamp=Timestamp(1615024314),
-        address=None,
-        transaction_id=None,
-        asset=symbol_to_asset_or_token('USDC'),
-        amount=AssetAmount(FVal('1')),
-        fee_asset=A_USD,
-        fee=Fee(ZERO),
-        link='NXT0000000001',
     )]
 
     assert ledger_actions == expected_actions
Repository: rotki/rotki

Issue:
Both AAVE and stkAAVE are counted in balances
## Problem Definition

If an account is staking AAVE and has stkAAVE, then it seems that both stkAAVE is now seen by rotki thus almost double counting the amount of owned AAVE.

In the past and until February 6 we were turning stkAAVE + claimable stkAAVE as AAVE.

Since February 6th 2022 it seems that stkAAVE is now also shown, presumably due to a change in the zerion contract so we need to make some changes here.

## Task

Fix it.


Discussion:

Resolving patch:
diff --git a/rotkehlchen/chain/ethereum/tokens.py b/rotkehlchen/chain/ethereum/tokens.py
--- a/rotkehlchen/chain/ethereum/tokens.py
+++ b/rotkehlchen/chain/ethereum/tokens.py
@@ -128,6 +128,8 @@ def query_tokens_for_addresses(
             # since the SDK entry might return other tokens from sushi and we don't
             # fully support sushi now.
             string_to_ethereum_address('0x8798249c2E607446EfB7Ad49eC89dD1865Ff4272'),
+            # Ignore stkAave since it's queried by defi SDK.
+            string_to_ethereum_address('0x4da27a545c0c5B758a6BA100e3a049001de870f5'),
             # Ignore the following tokens. They are old tokens of upgraded contracts which
             # duplicated the balances at upgrade instead of doing a token swap.
             # e.g.: https://github.com/rotki/rotki/issues/3548


Test patch:
Repository: rotki/rotki

Issue:
Docker container's /tmp doesn't get automatically cleaned
## Problem Definition

PyInstaller extracts the files in /tmp every time the backend starts
In the docker container /tmp is never cleaned which results in an ever-increasing size on every application restart

## TODO

- [ ] Add /tmp cleanup on start




Discussion:

Resolving patch:
diff --git a/packaging/docker/entrypoint.py b/packaging/docker/entrypoint.py
--- a/packaging/docker/entrypoint.py
+++ b/packaging/docker/entrypoint.py
@@ -2,10 +2,12 @@
 import json
 import logging
 import os
+import shutil
 import subprocess
 import time
+from datetime import datetime, timedelta
 from pathlib import Path
-from typing import Dict, Optional, Any, List
+from typing import Any, Dict, List, Optional
 
 logger = logging.getLogger('monitor')
 logging.basicConfig(level=logging.DEBUG)
@@ -13,6 +15,41 @@
 DEFAULT_LOG_LEVEL = 'critical'
 
 
+def can_delete(file: Path, cutoff: int) -> bool:
+    return int(os.stat(file).st_mtime) <= cutoff or file.name.startswith('_MEI')
+
+
+def cleanup_tmp() -> None:
+    logger.info('Preparing to cleanup tmp directory')
+    tmp_dir = Path('/tmp/').glob('*')
+    cache_cutoff = datetime.today() - timedelta(hours=6)
+    cutoff_epoch = int(cache_cutoff.strftime("%s"))
+    to_delete = filter(lambda x: can_delete(x, cutoff_epoch), tmp_dir)
+
+    deleted = 0
+    skipped = 0
+
+    for item in to_delete:
+        path = Path(item)
+        if path.is_file():
+            try:
+                path.unlink()
+                deleted += 1
+                continue
+            except PermissionError:
+                skipped += 1
+                continue
+
+        try:
+            shutil.rmtree(item)
+            deleted += 1
+        except OSError:
+            skipped += 1
+            continue
+
+    logger.info(f'Deleted {deleted} files or directories, skipped {skipped} from /tmp')
+
+
 def load_config_from_file() -> Optional[Dict[str, Any]]:
     config_file = Path('/config/rotki_config.json')
     if not config_file.exists():
@@ -101,6 +138,8 @@ def load_config() -> List[str]:
     return args
 
 
+cleanup_tmp()
+
 base_args = [
     '/usr/sbin/rotki',
     '--rest-api-port',


Test patch:
Repository: rotki/rotki

Issue:
Allow to filter decoded events by different parameters
## Abstract

Currently is not possible to filter transactions in the frontend by protocol, timestamp or other params

## Motivation

Improve the quality of the search for ethereum transactions

## Specification

The endpoint to query ethereum transactions should accept parameters for:

- protocol (counterparty)
- timestamp (from and to, we already do this in other places irc)
- address
- asset

at least



Discussion:
The reason why they do not get filtered like this by the way is that the order that they appear in is that of the ethereum transacitons the events are in
@lukicenturi @kelsos This just needs frontend work and as such would be nice to get it into the release.

With the current implementation you should be able to filter the ethereum transaction and their events right now by:

**These will filter on transactions**

- Time range
- Address

**These will filter on events**

- Asset
- List of counterparties (protocols)

Resolving patch:
diff --git a/rotkehlchen/api/v1/schemas.py b/rotkehlchen/api/v1/schemas.py
--- a/rotkehlchen/api/v1/schemas.py
+++ b/rotkehlchen/api/v1/schemas.py
@@ -166,7 +166,7 @@ class EthereumTransactionQuerySchema(
     address = EthereumAddressField(load_default=None)
     from_timestamp = TimestampField(load_default=Timestamp(0))
     to_timestamp = TimestampField(load_default=ts_now)
-    protocols = fields.List(fields.String(), load_default=None)
+    protocols = DelimitedOrNormalList(fields.String(), load_default=None)
     asset = AssetField(load_default=None)
     exclude_ignored_assets = fields.Boolean(load_default=True)
 


Test patch:
Repository: rotki/rotki

Issue:
Extract SQLCipher and pysqlcipher building to different repo
## Problem Definition

We have pinned versions of SQLCipher, and pysqlcipher that we use.

The build of SQLCipher happens on every build, docker, windows, macos, linux, arm64.
Since we use pinned versions we should create a new repo that builds sqlcipher for all the supported OSes/architectures and maybe publishes the wheels/packages to PyPI

We only need to build these dependencies when there is a change in version, otherwise there is no need to build them every single time since this increases the build times everywhere and complicates the windows development part.

Ideally, it would be nice to include SQLcipher in the python package to make things easier

### Task
- Create a separate repo to handle the building and publishing





Discussion:
Changed the title after chatting with @kelsos. He knows what needs to be done and will update the description.

Resolving patch:
diff --git a/tools/pyinstaller_hooks/hook-pysqlcipher3.py b/tools/pyinstaller_hooks/hook-pysqlcipher3.py
--- a/tools/pyinstaller_hooks/hook-pysqlcipher3.py
+++ b/tools/pyinstaller_hooks/hook-pysqlcipher3.py
@@ -1,3 +1,3 @@
 from PyInstaller.utils.hooks import copy_metadata
 
-datas = copy_metadata("pysqlcipher3")
+datas = copy_metadata("rotki-pysqlcipher3")


Test patch:
Repository: rotki/rotki

Issue:
Add contribution guide for CEX and DeFi modules
## Problem Definition

There is no easy way for external people to see how to create rotki modules. We need to change that. Not all parts of the code are super modular but for those that are there is no way other than reading the code to understand how to contribute

## Task

- [x] Add a guide in the contribution docs about how can a user add support for a new CEX.
- [x] Add a guide in the contributions docs about how can a user add support for a new DeFi module

Note for future: Iterating through these guides we should also find ways to make the tool more modular.


Discussion:

Resolving patch:
diff --git a/rotkehlchen/chain/ethereum/accounting/structures.py b/rotkehlchen/chain/ethereum/accounting/structures.py
--- a/rotkehlchen/chain/ethereum/accounting/structures.py
+++ b/rotkehlchen/chain/ethereum/accounting/structures.py
@@ -29,5 +29,4 @@ class TxEventSettings(NamedTuple):
     take: int
     method: Literal['acquisition', 'spend']
     multitake_treatment: Optional[TxMultitakeTreatment] = None
-    # accountant_cb: Optional[Callable[['AccountingPot', 'HistoryBaseEntry', List['HistoryBaseEntry']], None]] = None  # noqa: E501
     accountant_cb: Optional[AccountantCallback] = None


Test patch:
Repository: rotki/rotki

Issue:
Unicode logging entries in Windows
## Problem Definition

While checking the logs of a Windows user I noticed the following repeating multiple times inside rotki-electron.log

```--- Logging error ---
Traceback (most recent call last):
  File "logging\__init__.py", line 1086, in emit
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character '\u2728' in position 101: character maps to <undefined>
Call stack:
  File "rotkehlchen\api\rest.py", line 298, in _do_query_async
  File "rotkehlchen\api\rest.py", line 664, in _query_blockchain_balances
  File "rotkehlchen\utils\mixins\lockable.py", line 46, in wrapper
  File "rotkehlchen\utils\mixins\cacheable.py", line 97, in wrapper
  File "rotkehlchen\chain\manager.py", line 616, in query_balances
  File "rotkehlchen\utils\mixins\lockable.py", line 46, in wrapper
  File "rotkehlchen\utils\mixins\cacheable.py", line 97, in wrapper
  File "rotkehlchen\chain\manager.py", line 1150, in query_ethereum_balances
  File "rotkehlchen\chain\manager.py", line 1083, in query_ethereum_tokens
  File "rotkehlchen\chain\evm\tokens.py", line 341, in query_tokens_for_addresses
  File "rotkehlchen\chain\evm\tokens.py", line 181, in _get_multicall_token_balances
  File "logging\__init__.py", line 1800, in debug
  File "logging\__init__.py", line 1844, in log
  File "logging\__init__.py", line 1512, in log
  File "logging\__init__.py", line 1589, in _log
  File "logging\__init__.py", line 1599, in handle
  File "logging\__init__.py", line 1661, in callHandlers
  File "logging\__init__.py", line 952, in handle
  File "logging\handlers.py", line 75, in emit
  File "logging\__init__.py", line 1187, in emit
  File "logging\__init__.py", line 1091, in emit
Message: 'Greenlet-4: Found \u2728(0x714a91578624d76A209E9aA4aF9a1F55d78f9190) token balance for 0x<redacted> and balance 6.434434324'
Arguments: ()
```

I suspect there may be an issue making unicode log entries to a windows file

## Task

Reproduce and fix


Discussion:

Resolving patch:
diff --git a/rotkehlchen/logging.py b/rotkehlchen/logging.py
--- a/rotkehlchen/logging.py
+++ b/rotkehlchen/logging.py
@@ -176,6 +176,7 @@ def configure_logging(args: argparse.Namespace) -> None:
             'backupCount': backups_num,
             'level': loglevel,
             'formatter': 'default',
+            'encoding': 'utf-8',
         }
     else:
         selected_handlers = ['console']


Test patch:
Repository: rotki/rotki

Issue:
Error accessing database during exchange query (kraken)
rotki version: develop

logs

```
[31/10/2022 20:13:38 CET] ERROR rotkehlchen.greenlets Greenlet with id 140129837313536: Query history of Kraken exchange died with exception: 'NoneType' object has no attribute 'read_ctx'.
Exception Name: <class 'AttributeError'>
Exception Info: 'NoneType' object has no attribute 'read_ctx'
Traceback:
   File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/yabirgb/work/rotki/rotkehlchen/exchanges/exchange.py", line 489, in query_history_with_callbacks
    trades_history = self.query_trade_history(
  File "/home/yabirgb/work/rotki/rotkehlchen/utils/mixins/lockable.py", line 46, in wrapper
    result = f(wrappingobj, *args, **kwargs)
  File "/home/yabirgb/work/rotki/rotkehlchen/exchanges/exchange.py", line 267, in query_trade_history
    with self.db.conn.read_ctx() as cursor:
```

It happened to me with a db that I just pulled. What I did is create a new account syncing with the database in the rotki servers and let it query. I got the error shown previously. Tried to reproduce but It didn't show again


Discussion:
yeah but as I said in the discord this makes no sense. Would really need more information to reproduce an issue like this

Resolving patch:
diff --git a/rotkehlchen/rotkehlchen.py b/rotkehlchen/rotkehlchen.py
--- a/rotkehlchen/rotkehlchen.py
+++ b/rotkehlchen/rotkehlchen.py
@@ -171,6 +171,7 @@ def reset_after_failed_account_creation_or_login(self) -> None:
         to sync premium databases we relogged in
         """
         self.cryptocompare.db = None
+        self.exchange_manager.delete_all_exchanges()
         self.data.logout()
 
     def unlock_user(


Test patch:
Repository: rotki/rotki

Issue:
Think of a way to keep development accounts separately 
## Abstract

Between multiple development machines and between `production`/`develop` it becomes harder to keep track of which database has been used by which branch. This can lead to partially updated databases being used.

As a proposal, we could somehow separate where the `develop` accounts and the production accounts are stored so that they are not sharing the same place.

We can always copy accounts from production to develop manually (this can go to the guide).

We should also consider how this affects nightlies and how nightlies are treated. For example we might want to treat nightlies as development to avoid having users mess accidentally with their production accounts.  

## Motivation

Helps better track which accounts are used in `develop`/`production`.

## Specification

- TDB



Discussion:

Resolving patch:
diff --git a/rotkehlchen/config.py b/rotkehlchen/config.py
--- a/rotkehlchen/config.py
+++ b/rotkehlchen/config.py
@@ -2,6 +2,7 @@
 import os
 import platform
 import shutil
+import sys
 from pathlib import Path
 
 from rotkehlchen.logging import RotkehlchenLogsAdapter
@@ -42,14 +43,18 @@ def default_data_directory() -> Path:
     An interesting lirary that finds the data directories per OS is this:
     https://github.com/ActiveState/appdirs/blob/master/appdirs.py
     """
+    data_dir_name = 'data'
+    if getattr(sys, 'frozen', False) is False:
+        data_dir_name = 'develop_data'
+
     if platform.system() == 'Linux':
         xdgconfig = get_xdg_data_home()
-        datadir = xdgconfig / 'rotki' / 'data'
+        datadir = xdgconfig / 'rotki' / data_dir_name
     elif platform.system() == 'Windows':
         appdata = get_win32_appdata()
-        datadir = appdata / 'rotki' / 'data'
+        datadir = appdata / 'rotki' / data_dir_name
     elif platform.system() == 'Darwin':
-        datadir = Path(os.path.expanduser('~/Library/Application Support/rotki/data'))
+        datadir = Path(os.path.expanduser(f'~/Library/Application Support/rotki/{data_dir_name}'))  # noqa: E501
     else:
         raise AssertionError(f'rotki running in unknown system: {platform.system()}')
 


Test patch:
Repository: rotki/rotki

Issue:
Make sure that Docker image can properly shutdown
## Problem Definition

While trying to set up the proxy I noticed that the proxy didn't exit cleanly. 
Instead, it exited with (code=137). 

This might be because there is an infinite loop (intentional) monitoring the subprocesses, and this loop only exits if one of the subprocesses stops running.

This will cause the script to exit with code 1.

https://github.com/rotki/rotki/blob/53d36bc1f5aa0ff64fcfaf21f0ba6ca005a032eb/packaging/docker/entrypoint.py#L173

## Requirements

- Handle `SIGINT`/`SIGTERM` normally and exit cleanly (code=0)
- Make sure that if the subprocesses die we exit (code=1)



Discussion:

Resolving patch:
diff --git a/packaging/docker/entrypoint.py b/packaging/docker/entrypoint.py
--- a/packaging/docker/entrypoint.py
+++ b/packaging/docker/entrypoint.py
@@ -3,6 +3,7 @@
 import logging
 import os
 import shutil
+from signal import signal, SIGINT, SIGTERM, SIGQUIT
 import subprocess
 import time
 from datetime import datetime, timedelta
@@ -170,6 +171,30 @@ def load_config() -> List[str]:
     logger.error('Failed to start nginx')
     exit(1)
 
+
+def terminate_process(process_name: str, process: subprocess.Popen) -> None:
+    logger.info(f'Terminating {process_name}')
+    if process.poll() is not None:
+        logger.error(f'{process_name} was not running. This means that some error occurred.')
+        exit(1)
+
+    process.terminate()
+    process.wait()  # wait untill the process terminates
+
+
+def graceful_exit(signal, frame):
+    logger.info(f'Received signal {signal}. Exiting gracefully')
+    terminate_process('rotki', rotki)
+    terminate_process('nginx', nginx)
+    exit(0)
+
+
+# Handle exits via ctrl+c or via `docker stop` gracefully
+signal(SIGINT, graceful_exit)
+signal(SIGTERM, graceful_exit)
+signal(SIGQUIT, graceful_exit)
+
+
 while True:
     time.sleep(60)
 


Test patch: