Repository: Chia-Network/chia-blockchain
Pull request: Resend transactions
This PR periodically resends unconfirmed transactions to peers. This is needed because acceptance of a SpendBundle into the Mempool by a peer does not guarantee that it will be later confirmed, or even transmitted to other peers.

It does this by resetting the remembered results from add to mempool requests that were successful but not confirmed.

Fixes https://github.com/Chia-Network/chia-blockchain/issues/10873




Problem statement:
[Bug] Wallet gets stuck on sending out CAT
### What happened?

Sending out a CAT can sometimes block the wallet and only solution is to sync the wallet DB from 0.

Sending out a cat using CLI "chia wallet send -i 2 ..." sometimes creates a transaction that is stuck in "Status: Pending" forever, when checking it with "chia wallet get_transaction -tx"

Looking into wallet DB, the transaction is marked as sent = 1, confirmed = 1

Confirmed_at_hight is a block number before the transaction was even initiated, so not possible. The confirmed_at_height is same as the last block that the same CAT was successfully send out.

After this happens, sending out of that CAT is impossible, it is stuck. Receiving it appears to be read correctly.

### Version

1.3.1

### What platform are you using?

Linux

### What ui mode are you using?

CLI

### Relevant log output

_No response_

[Bug] transactions gets stuck in "pending" and are not retried
### What happened?

### Prerequisites
- 1.3.x wallet (1.3.0 or 1.3.1)
- Mempool is full

### How to reproduce
- Send a transaction, preferrably a multi out which wont be included due to the full mempool straight away
  - Might need to repeat the step above a few times to get a stuck transaction

### What happens
When a transaction is stuck, it will show its state as "Pending" via `chia wallet get_transaction`. The tx will never be included in the mempool and thus never confirm.

### Current workaround
Restarting the wallet service (`chia start -r wallet`) sometimes results in the tx transitioning to "In mempool" state and confirm within a minute. In the past few weeks the amount of wallet service restarts required to get a stuck transaction "unstuck" this way was 1-7.

### Version

1.3.1

### What platform are you using?

Linux

### What ui mode are you using?

CLI

### Relevant log output

_No response_

cannot claim Claim Rewards until now.[Bug] 
### What happened?

I'm still having trouble with Claim Rewards.
From version 1.3.0 until now, 1.3.4 cannot claim Claim Rewards until now.

### Version

1.3.4

### What platform are you using?

Windows

### What ui mode are you using?

GUI

### Relevant log output

```shell
2022-04-20T09:57:17.911 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x97ffc8db7395efb4fb334f95ff7070ccf83a58f0243c16db327a12266a388387'}
2022-04-20T09:57:17.917 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x97ffc8db7395efb4fb334f95ff7070ccf83a58f0243c16db327a12266a388387'}
2022-04-20T10:10:59.645 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'MEMPOOL_NOT_INITIALIZED',
 'status': 3,
 'txid': '0x97ffc8db7395efb4fb334f95ff7070ccf83a58f0243c16db327a12266a388387'}
2022-04-20T10:18:01.044 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x3bff8035471e0cb84aa876c90b4bb0ba4100178ca863d4036bb6e16ca906b885'}
2022-04-20T10:18:01.050 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x3bff8035471e0cb84aa876c90b4bb0ba4100178ca863d4036bb6e16ca906b885'}
```
2022-04-20T11:17:12.148 wallet chia.rpc.rpc_server        : WARNING  Error while handling message: Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 225, in safe_handle
  File "chia\rpc\rpc_server.py", line 216, in ws_api
  File "chia\rpc\wallet_rpc_api.py", line 618, in get_wallet_balance
KeyError: 0

Hints from issue discussion:
This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.
Still active
Here is another instance of this happening in which restarting chia solved the problem https://www.reddit.com/r/chia/comments/tnp1v4/24_hours_pending_transaction_dust_storm/
I have a similar problem. Accepted an offer and forgot to add a fee. After 12h pending I canceled it. 2 days later I have still peding balance of a CH21 token (-1) and XCH (+0.25). The offer is listed as "canceled".

Edit: 11h later. still pending, even after two restarts (current mempool 56).

![image](https://user-images.githubusercontent.com/14164004/160702664-be16a3da-0f28-41ab-897d-6269d559b6cd.png)
![image](https://user-images.githubusercontent.com/14164004/160702764-0d474d98-9cd4-47bd-b14c-88f20b3a6557.png)
![image](https://user-images.githubusercontent.com/14164004/160702971-b3780aa5-4823-40ec-94d1-c05627f4374c.png)
![image](https://user-images.githubusercontent.com/14164004/160703071-5f71bdae-ba41-4799-9549-f45007a42342.png)

### Version
1.3.1

### What platform are you using?
Windows

### What ui mode are you using?
GUI

Something new on this topic? It is still pending and I can't access my CH21 token.^^
Annoying. Why does nobody write "delete unconfirmed transactions" (which is hidden in that 3dots menu). And why is there no info in the UI what to do and what problems it can cause If someone deletes the unconfirmed transactions? It makes no sense that it stays unconfirmed for days. And why is there no standard fee. Or better like in my Ethereum Wallet where I can choose from three or so options? Not even a current USD price is shown. This software is very alpha and absolute not ready for market. 
Fix here: https://github.com/Chia-Network/chia-blockchain/pull/11167
And it has a very strange way of displaying the values ​​in the wallet window. @wallentx 

[https://www.img.in.th/image/79PkA7](url)
When the wallet is connected to multiple nodes, getting DOUBLE_SPEND can be normal as you send the same transaction to multiple nodes, so some of those nodes may see it already in the mempool as a double spend. So the claim transaction may go through just fine despite this error.
> When the wallet is connected to multiple nodes, getting DOUBLE_SPEND can be normal as you send the same transaction to multiple nodes, so some of those nodes may see it already in the mempool as a double spend. So the claim transaction may go through just fine despite this error.

But now I can't get the reward for a very long time. Is there any reason that can help me to fix it, how to do it? I think I have a problem with my wallet.
@emlowe  Do I have a way to contact you personally? I'm sure there's a problem with my wallet.
I would recommend to delete or rename your wallet DB and resync your wallet. You can get more real-time support in keybase in the support channel.
As you suggested, let's remove db and resync until I succeed. I've already done it. Remove everything, reinstall chia 1.3.4, but this problem is the same. No matter how much mojo I wear,
![1650593325782](https://user-images.githubusercontent.com/103640664/164583334-e0dfe393-07a7-4388-bfd7-ee396d07aaf6.jpg)
![1650428340900](https://user-images.githubusercontent.com/103640664/164583421-40ac3c19-13a1-4b22-97da-9f7373b20bd3.jpg)
log
 'status': 3,
 'txid': '0x37e18ec3fb4f247c90272df5e0c454a87444c7c35c079efc62569b50897bb814'}
2022-04-19T11:11:43.674 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'ASSERT_ANNOUNCE_CONSUMED_FAILED',
 'status': 3,
 'txid': '0x37e18ec3fb4f247c90272df5e0c454a87444c7c35c079efc62569b50897bb814'}

2022-04-22T01:46:54.580 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x918da07ac2720f993a2272686631fdf33c6dfb223570d183e8fecb9c44b9846b'}
2022-04-22T01:46:54.713 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'DOUBLE_SPEND', 'status': 3, 'txid': '0x918da07ac2720f993a2272686631fdf33c6dfb223570d183e8fecb9c44b9846b'}

2022-04-22T09:39:58.817 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'INVALID_FEE_TOO_CLOSE_TO_ZERO',
 'status': 3,
 'txid': '0xeaf25e88c97ac58e6a39c5d5a3594c20d3b6083a2ad305031caa7eb6ad38ce28'}
2022-04-22T09:39:58.822 wallet chia.wallet.wallet_node    : WARNING  SpendBundle has been rejected by the FullNode. {'error': 'INVALID_FEE_TOO_CLOSE_TO_ZERO',
 'status': 3,
 'txid': '0xeaf25e88c97ac58e6a39c5d5a3594c20d3b6083a2ad305031caa7eb6ad38ce28'}

This is a log error each time I press Confirm, there is a different error, but every time it is completely steady until I press delete the bank. 

@emlowe

Commit message: Add wallet resend parameter to config, move timeout code out of tx store, but close to call site

Patch to review:
diff --git a/chia/util/initial-config.yaml b/chia/util/initial-config.yaml
index ef6aa9874c35..ffafa9bd52e0 100644
--- a/chia/util/initial-config.yaml
+++ b/chia/util/initial-config.yaml
@@ -542,3 +542,6 @@ wallet:
   # if an unknown CAT belonging to us is seen, a wallet will be automatically created
   # the user accepts the risk/responsibility of verifying the authenticity and origin of unknown CATs
   automatically_add_unknown_cats: False
+
+  # Interval to resend unconfirmed transactions, even if previously accepted into Mempool
+  tx_resend_timeout_secs: 1800
diff --git a/chia/wallet/wallet_node.py b/chia/wallet/wallet_node.py
index f417e58d7a4a..bea9834426ac 100644
--- a/chia/wallet/wallet_node.py
+++ b/chia/wallet/wallet_node.py
@@ -141,6 +141,8 @@ def __init__(
         self.validation_semaphore = None
         self.local_node_synced = False
         self.LONG_SYNC_THRESHOLD = 200
+        self.last_wallet_tx_resend_time: int = 0
+        self.wallet_tx_resend_timeout_secs: int = 60
 
     async def ensure_keychain_proxy(self) -> KeychainProxy:
         if self.keychain_proxy is None:
@@ -230,6 +232,9 @@ async def _start(
         if self.state_changed_callback is not None:
             self.wallet_state_manager.set_callback(self.state_changed_callback)
 
+        self.last_wallet_tx_resend_time = int(time.time())
+        if "tx_resend_timeout_secs" in self.config:
+            self.wallet_tx_resend_timeout_secs = self.config["tx_resend_timeout_secs"]
         self.wallet_state_manager.set_pending_callback(self._pending_tx_handler)
         self._shut_down = False
         self._process_new_subscriptions_task = asyncio.create_task(self._process_new_subscriptions())
@@ -332,7 +337,14 @@ async def _messages_to_resend(self) -> List[Tuple[Message, Set[bytes32]]]:
             return []
         messages: List[Tuple[Message, Set[bytes32]]] = []
 
-        records: List[TransactionRecord] = await self.wallet_state_manager.tx_store.get_not_sent()
+        current_time = int(time.time())
+        retry_accepted_txs = False
+        if self.last_wallet_tx_resend_time < current_time - self.wallet_tx_resend_timeout_secs:
+            self.last_wallet_tx_resend_time = current_time
+            retry_accepted_txs = True
+        records: List[TransactionRecord] = await self.wallet_state_manager.tx_store.get_not_sent(
+            include_accepted_txs=retry_accepted_txs
+        )
 
         for record in records:
             if record.spend_bundle is None:
@@ -1044,7 +1056,9 @@ async def new_peak_wallet(self, new_peak: wallet_protocol.NewPeakWallet, peer: W
 
         if peer.peer_node_id in self.synced_peers:
             await self.wallet_state_manager.blockchain.set_finished_sync_up_to(new_peak.height)
-        await self.wallet_state_manager.new_peak(new_peak)
+
+        async with self.wallet_state_manager.lock:
+            await self.wallet_state_manager.new_peak(new_peak)
 
     async def wallet_short_sync_backtrack(self, header_block: HeaderBlock, peer: WSChiaConnection) -> int:
         assert self.wallet_state_manager is not None
diff --git a/chia/wallet/wallet_state_manager.py b/chia/wallet/wallet_state_manager.py
index 84d31c2af203..a7497f289851 100644
--- a/chia/wallet/wallet_state_manager.py
+++ b/chia/wallet/wallet_state_manager.py
@@ -1212,6 +1212,10 @@ async def new_peak(self, peak: wallet_protocol.NewPeakWallet):
         for wallet_id, wallet in self.wallets.items():
             if wallet.type() == uint8(WalletType.POOLING_WALLET):
                 await wallet.new_peak(peak.height)
+        current_time = int(time.time())
+
+        if self.wallet_node.last_wallet_tx_resend_time < current_time - self.wallet_node.wallet_tx_resend_timeout_secs:
+            self.tx_pending_changed()
 
     async def add_interested_puzzle_hashes(
         self, puzzle_hashes: List[bytes32], wallet_ids: List[int], in_transaction: bool = False
diff --git a/chia/wallet/wallet_transaction_store.py b/chia/wallet/wallet_transaction_store.py
index 6d9b91d2301b..b6d8b36da1d1 100644
--- a/chia/wallet/wallet_transaction_store.py
+++ b/chia/wallet/wallet_transaction_store.py
@@ -1,3 +1,4 @@
+import dataclasses
 import time
 from typing import Dict, List, Optional, Tuple
 
@@ -13,6 +14,15 @@
 from chia.wallet.util.transaction_type import TransactionType
 
 
+def filter_ok_mempool_status(sent_to: List[Tuple[str, uint8, Optional[str]]]) -> List[Tuple[str, uint8, Optional[str]]]:
+    """Remove SUCCESS and PENDING status records from a TransactionRecord sent_to field"""
+    new_sent_to = []
+    for peer, status, err in sent_to:
+        if status == MempoolInclusionStatus.FAILED.value:
+            new_sent_to.append((peer, status, err))
+    return new_sent_to
+
+
 class WalletTransactionStore:
     """
     WalletTransactionStore stores transaction history for the wallet.
@@ -23,6 +33,8 @@ class WalletTransactionStore:
     tx_record_cache: Dict[bytes32, TransactionRecord]
     tx_submitted: Dict[bytes32, Tuple[int, int]]  # tx_id: [time submitted: count]
     unconfirmed_for_wallet: Dict[int, Dict[bytes32, TransactionRecord]]
+    last_wallet_tx_resend_time: int  # Epoch time in seconds
+    wallet_tx_resend_timeout_secs: int  # Duration in seconds
 
     @classmethod
     async def create(cls, db_wrapper: DBWrapper):
@@ -79,6 +91,8 @@ async def create(cls, db_wrapper: DBWrapper):
         self.tx_record_cache = {}
         self.tx_submitted = {}
         self.unconfirmed_for_wallet = {}
+        self.last_wallet_tx_resend_time = int(time.time())
+        self.wallet_tx_resend_timeout_secs = 60 * 60
         await self.rebuild_tx_cache()
         return self
 
@@ -277,9 +291,9 @@ async def get_transaction_record(self, tx_id: bytes32) -> Optional[TransactionRe
             return record
         return None
 
-    async def get_not_sent(self) -> List[TransactionRecord]:
+    async def get_not_sent(self, *, include_accepted_txs=False) -> List[TransactionRecord]:
         """
-        Returns the list of transaction that have not been received by full node yet.
+        Returns the list of transactions that have not been received by full node yet.
         """
         current_time = int(time.time())
         cursor = await self.db_connection.execute(
@@ -289,9 +303,16 @@ async def get_not_sent(self) -> List[TransactionRecord]:
         rows = await cursor.fetchall()
         await cursor.close()
         records = []
+
         for row in rows:
             record = TransactionRecord.from_bytes(row[0])
-            if record.name in self.tx_submitted:
+            if include_accepted_txs:
+                # Reset the "sent" state for peers that have replied about this transaction. Retain errors.
+                record = dataclasses.replace(record, sent=1, sent_to=filter_ok_mempool_status(record.sent_to))
+                await self.add_transaction_record(record, False)
+                self.tx_submitted[record.name] = current_time, 1
+                records.append(record)
+            elif record.name in self.tx_submitted:
                 time_submitted, count = self.tx_submitted[record.name]
                 if time_submitted < current_time - (60 * 10):
                     records.append(record)


Code review comments:
diff_hunk: @@ -141,6 +141,8 @@ def __init__(
         self.validation_semaphore = None
         self.local_node_synced = False
         self.LONG_SYNC_THRESHOLD = 200
+        self.last_wallet_tx_resend_time: int = 0
+        self.wallet_tx_resend_timeout_secs: int = 60
path: chia/wallet/wallet_node.py
text: @user1:
it seems you meant to set this to 1800

@author:
Thanks - I originally had this code in the WalletTransactionManager, but I think this version is better.

diff_hunk: @@ -230,6 +232,9 @@ async def _start(
         if self.state_changed_callback is not None:
             self.wallet_state_manager.set_callback(self.state_changed_callback)
 
+        self.last_wallet_tx_resend_time = int(time.time())
+        if "tx_resend_timeout_secs" in self.config:
+            self.wallet_tx_resend_timeout_secs = self.config["tx_resend_timeout_secs"]
path: chia/wallet/wallet_node.py
text: @user1:
another way of writing this would be:

```
self.wallet_tx_resend_timeout_secs = self.config.get("tx_resend_timeout_secs", 1800)
```

which I think would be a bit more robust