Repository: spesmilo/electrum

Issue:
Cannot send lightning funds: Too many HTLCs already in channel 
I decided to try lightning, and it automatically opened a channel for me. Everything seems correct, node is ONLINE, can send: shows correct amount. But every time i try to send i get this error:

Payment failed
Too many HTLCs already in channel

electrum 4.5.2




Discussion:
Is this with trampoline routing enabled or disabled?
note: https://github.com/spesmilo/electrum/issues/8533 is conceptually similar
No trampoline in use. But i remember trying clicking it, and sending few payments, but it said node does not support it, so i turned it off again.

I checked transaction history, and only transaction is the channel founding tx. So where are these HTLC coming from? Could the command line option enable_htlc_settle be of help here? (seems undocumented, as is the error code!?)
lightning transactions that are 'in progress' are not displayed in your history. they only show up once the HTLC is settled.

edit: it would probably be good to display those in progress transactions in the history tab.

> Could the command line option enable_htlc_settle be of help here? (seems undocumented, as is the error code!?)

no, that command is only useful for testing. 

When clicking on the node it shows only failed HTLCs. Is this what you refer to as transactions in progress? If so, can they still succeed? This would seem like extremely dangerous having transactions hidden that can still pay old invoices that are long gone. And if they cannot succeed, why keep them around, is it just to wait the CLTV timeout? If so, there certainly should be some indication of what is going on (besides a 5 second system popup with undocumented error code)
if you have sent an HTLC, you cannot cancel it, you have to keep it around until it either succeeds or is failed. 
there is no danger of paying an old invoice twice, because the software won't attempt a new payment. 

> When clicking on the node it shows only failed HTLCs. Is this what you refer to as transactions in progress? 

no, if they failed, they will not succeed. in progress HTLCs should be shown as "in current commitment transaction"

since you mention that you are not using trampoline, can you explain what you mean by "it automatically opened a channel for me." ? Also, which node did you open a channel with? 

It would also be nice to know what the value of `max_accepted_htlcs` of the remote node config.
In the console tab, type the following: 
`[c.config[1].max_accepted_htlcs for c in channels]`

> It would also be nice to know what the value of `max_accepted_htlcs` of the remote node config.
In the console tab, type the following: 
`[c.config[1].max_accepted_htlcs for c in channels]`

It would be useful to know both local and remote values (btw `1` is local). So please run this in the console instead:
```
[(c.config[1].max_accepted_htlcs, c.config[-1].max_accepted_htlcs) for c in channels]
```
> > It would also be nice to know what the value of `max_accepted_htlcs` of the remote node config.
> > In the console tab, type the following:
> > `[c.config[1].max_accepted_htlcs for c in channels]`
> 
> It would be useful to know both local and remote values (btw `1` is local). So please run this in the console instead:
> 
> ```
> [(c.config[1].max_accepted_htlcs, c.config[-1].max_accepted_htlcs) for c in channels]
> ```

[(5, 483)]

With automatically opened a channel i meant that i just clicked in the gui and it did everything automatically, i did not specify node.

The "in current commitment transaction" is empty.
> [(5, 483)]

Hmm you must have opened that channel a long time ago, we increased the locally chosen value from 5 to 30 in version 4.1 (https://github.com/spesmilo/electrum/commit/762ebb12b253dd53517dbd99b13efcc0e0168b05).
yes
@reivanen you should close that channel, and open a new one with 4.5.2. It will use the increased value for that parameter.
We should make the code smarter though, to realise that it should not split the payment into more parts if the channel cannot carry more HTLCs. And at the same time, probably also consider https://github.com/spesmilo/electrum/issues/8533

It is probably https://github.com/spesmilo/electrum/commit/35c9ac8f3173deaac14230ae1991a19d0d10987d that started causing the issue btw.
is it a fix if i use electrum version before 4.5.2, or whatever included https://github.com/spesmilo/electrum/commit/35c9ac8f3173deaac14230ae1991a19d0d10987d ?

Sure, channel close&open new would work, but at current fee levels such unnecessary churn becomes pretty expensive.
in general it is not possible to downgrade; older versions of the wallet use older formats for the wallet file, and cannot read newer versions.

if want to keep using this channel, you might have to wait until we have a proper fix, as @SomberNight suggested
note: as temporary workaround, you can effectively disable the too eager splitting introduced in that commit by running `wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1` in the console
(but this does not persist through application restarts)
your suggestion resulted in electrum crashing while sending, and it offering me the privilege to send bug report. Which i did.

It took a long time to return the HTLC from in current commitment to available (failed) but eventually it did 

edit: does the command expect spaces before and after the = ? This is how i copypasted it from here.
> your suggestion resulted in electrum crashing while sending, and it offering me the privilege to send bug report. Which i did.

I can't see a related new issue, it must have been merged into an existing one. Do you know have a link?
Or if you have [debug logs](https://electrum.readthedocs.io/en/latest/faq.html#how-to-enable-debug-logging) those would contain the traceback; or it is also possible to copy the traceback when sending the bug report (though at that point, you could just copy the link to the report).

> edit: does the command expect spaces before and after the = ?

It does not matter.
I did not open new issue thread since it was directly related to this thread.

this is what i found in log:

```
20240207T182433.564367Z |    ERROR | gui.qt.exception_window.Exception_Hook | exception caught by crash reporter
Traceback (most recent call last):
  File "/tmp/.mount_electrW4swOi/usr/lib/python3.10/site-packages/electrum/gui/qt/util.py", line 1426, in on_qt_callback_signal
    return func(self, *args[1:])
  File "/tmp/.mount_electrW4swOi/usr/lib/python3.10/site-packages/electrum/gui/qt/channel_details.py", line 156, in on_event_htlc_failed
    self.move('inflight', 'failed', payment_hash)
  File "/tmp/.mount_electrW4swOi/usr/lib/python3.10/site-packages/electrum/gui/qt/channel_details.py", line 126, in move
    row_idx = self.keyname_rows[fro].pop(payment_hash)
KeyError: b'\xc1\xb7\x0f\xfd9en%\xe4\xe3\x10C\xa5\x84\xc9\xd4\xda4>\xdaUT^\xcb&2B\xd8\xef\xe5k?'
```

but should this not already be in your knowledge as i sent the bug report electrum asked me to send?
Actually digging through the log further, i saw this link mentioned by the crash reporter:; https://github.com/spesmilo/electrum/issues/5728

sad to see it is a 5 year old bug
> but should this not already be in your knowledge as i sent the bug report electrum asked me to send?

There are a lot of reports, which then get collated by a bot into issues. Sometimes the bot makes mistakes and merges reports into semantically different but similar-looking bugs. I can't know which report was sent by you.

> Actually digging through the log further, i saw this link mentioned by the crash reporter:; https://github.com/spesmilo/electrum/issues/5728

Ah, yeah ok, that's a completely unrelated bug from this issue. That's just a refresh bug with the channel_details dialog in the gui.
It does not explain why your payment failed.
Is it safe to upload the whole log file somewhere or does it contain sensitive information?
It is privacy-sensitive, but not security-sensitive. You can send email it to me if you want.
I sent it.
> It is privacy-sensitive, but not security-sensitive. You can send email it to me if you want.

So, did you find anything interesting? (besides my privacy-sensitive information :)
I had looked at your log but I could not really see anything interesting in it :/
You are trying to send 10 mBTC, which is quite a lot for a single HTLC. It might succeed if you retried several times. However for this kind of amount MPP really helps, and that is exactly at the heart of the issue here.

Setting `wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1` like I suggested, mostly disables MPP (hence single large HTLC). The code normally would try to split it into many parts, but this is not possible with your channel (as the old code set the parameters to too low when the channel was opened). So the best that can be done is what I said in https://github.com/spesmilo/electrum/issues/8857#issuecomment-1918354928, to make the code smarter to still split the HTLC, but only into as many parts as the channel allows.
well, the problem does not seem to be the send amount, as 0.001 BTC results in same.

I will wait for next version and hope the fix is in by then, otherwise i will eat the loss and close the channel. (and truly hope it works unlike this first step into lightning using electrum). When i opened the channel it was just decent spending money, now it is more than a month's salary.
If

wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1

Does not solve the problem, is it expected for the next update to solve it? Seeing how ever more popular bitcoin is becoming day by day and how people pay ever higher fees, i'm thinking maybe eat the loss now and not wish on the next ethereum version fixing this?

Or is there someting more than the above mentioned command coming that could save this channel?
with 4.5.5 the same error happened. As i expected because i did not see anything obviously related in the changelog. But after setting wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1 i was able to send both 0.001 and 0.01 amounts. So either this is just random luck or there after all was some improvements made.



Patch:
diff --git a/electrum/lnchannel.py b/electrum/lnchannel.py
index 9336be48472e..2848cf6fd786 100644
--- a/electrum/lnchannel.py
+++ b/electrum/lnchannel.py
@@ -1095,7 +1095,7 @@ def _assert_can_add_htlc(self, *, htlc_proposer: HTLCOwner, amount_msat: int,
             if amount_msat < chan_config.htlc_minimum_msat:
                 raise PaymentFailure(f'HTLC value too small: {amount_msat} msat')
 
-        if self.too_many_htlcs(htlc_proposer):
+        if self.htlc_slots_left(htlc_proposer) == 0:
             raise PaymentFailure('Too many HTLCs already in channel')
 
         if amount_msat > self.remaining_max_inflight(htlc_receiver):
@@ -1108,7 +1108,7 @@ def _assert_can_add_htlc(self, *, htlc_proposer: HTLCOwner, amount_msat: int,
         if max_can_send_msat < amount_msat:
             raise PaymentFailure(f'Not enough balance. can send: {max_can_send_msat}, tried: {amount_msat}')
 
-    def too_many_htlcs(self, htlc_proposer: HTLCOwner) -> bool:
+    def htlc_slots_left(self, htlc_proposer: HTLCOwner) -> int:
         # check "max_accepted_htlcs"
         htlc_receiver = htlc_proposer.inverted()
         ctn = self.get_next_ctn(htlc_receiver)
@@ -1116,17 +1116,16 @@ def too_many_htlcs(self, htlc_proposer: HTLCOwner) -> bool:
         # If proposer is LOCAL we apply stricter checks as that is behaviour we can control.
         # This should lead to fewer disagreements (i.e. channels failing).
         strict = (htlc_proposer == LOCAL)
-        # this is the loose check BOLT-02 specifies:
-        if len(self.hm.htlcs_by_direction(htlc_receiver, direction=RECEIVED, ctn=ctn)) + 1 > chan_config.max_accepted_htlcs:
-            return True
-        # however, c-lightning is a lot stricter, so extra checks:
-        # https://github.com/ElementsProject/lightning/blob/4dcd4ca1556b13b6964a10040ba1d5ef82de4788/channeld/full_channel.c#L581
-        if strict:
-            max_concurrent_htlcs = min(self.config[htlc_proposer].max_accepted_htlcs,
-                                       self.config[htlc_receiver].max_accepted_htlcs)
-            if len(self.hm.htlcs(htlc_receiver, ctn=ctn)) + 1 > max_concurrent_htlcs:
-                return True
-        return False
+        if not strict:
+            # this is the loose check BOLT-02 specifies:
+            return chan_config.max_accepted_htlcs - len(self.hm.htlcs_by_direction(htlc_receiver, direction=RECEIVED, ctn=ctn))
+        else:
+            # however, c-lightning is a lot stricter, so extra checks:
+            # https://github.com/ElementsProject/lightning/blob/4dcd4ca1556b13b6964a10040ba1d5ef82de4788/channeld/full_channel.c#L581
+            max_concurrent_htlcs = min(
+                self.config[htlc_proposer].max_accepted_htlcs,
+                self.config[htlc_receiver].max_accepted_htlcs)
+            return max_concurrent_htlcs - len(self.hm.htlcs(htlc_receiver, ctn=ctn))
 
     def remaining_max_inflight(self, htlc_receiver: HTLCOwner) -> int:
         # check "max_htlc_value_in_flight_msat"
@@ -1553,7 +1552,7 @@ def consider_ctx(*, ctx_owner: HTLCOwner, is_htlc_dust: bool) -> int:
         )
 
         max_send_msat = min(max_send_msat, self.remaining_max_inflight(receiver))
-        if self.too_many_htlcs(sender):
+        if self.htlc_slots_left(sender) == 0:
             max_send_msat = 0
 
         max_send_msat = max(max_send_msat, 0)
diff --git a/electrum/lnworker.py b/electrum/lnworker.py
index 77e390af4f1d..fe3ef9d45180 100644
--- a/electrum/lnworker.py
+++ b/electrum/lnworker.py
@@ -1927,7 +1927,7 @@ def suggest_payment_splits(
         receiver_pubkey: bytes,
     ) -> List['SplitConfigRating']:
         channels_with_funds = {
-            (chan.channel_id, chan.node_id): int(chan.available_to_spend(HTLCOwner.LOCAL))
+            (chan.channel_id, chan.node_id): ( int(chan.available_to_spend(HTLCOwner.LOCAL)), chan.htlc_slots_left(HTLCOwner.LOCAL))
             for chan in my_active_channels
         }
         # if we have a direct channel it's preferrable to send a single part directly through this
diff --git a/electrum/mpp_split.py b/electrum/mpp_split.py
index c7ea3663ab81..3b0aa64177b7 100644
--- a/electrum/mpp_split.py
+++ b/electrum/mpp_split.py
@@ -15,7 +15,7 @@
 
 
 # maps a channel (channel_id, node_id) to the funds it has available
-ChannelsFundsInfo = Dict[Tuple[bytes, bytes], int]
+ChannelsFundsInfo = Dict[Tuple[bytes, bytes], Tuple[int, int]]
 
 
 class SplitConfig(dict, Dict[Tuple[bytes, bytes], List[int]]):
@@ -105,7 +105,7 @@ def rate_config(
     total_amount = config.total_config_amount()
 
     for channel, amounts in config.items():
-        funds = channels_with_funds[channel]
+        funds, slots = channels_with_funds[channel]
         if amounts:
             for amount in amounts:
                 rating += amount * amount / (total_amount * total_amount)  # penalty to favor equal distribution of amounts
@@ -116,7 +116,8 @@ def rate_config(
 
 
 def suggest_splits(
-        amount_msat: int, channels_with_funds: ChannelsFundsInfo,
+        amount_msat: int,
+        channels_with_funds: ChannelsFundsInfo,
         exclude_single_part_payments=False,
         exclude_multinode_payments=False,
         exclude_single_channel_splits=False
@@ -132,32 +133,37 @@ def suggest_splits(
     """
 
     configs = []
-    channels_order = list(channels_with_funds.keys())
+    channel_keys = list(channels_with_funds.keys())
 
     # generate multiple configurations to get more configurations (there is randomness in this loop)
     for _ in range(CANDIDATES_PER_LEVEL):
         # we want to have configurations with no splitting to many splittings
         for target_parts in range(1, MAX_PARTS):
             config = SplitConfig()
-
             # randomly split amount into target_parts chunks
             split_amounts = split_amount_normal(amount_msat, target_parts)
             # randomly distribute amounts over channels
             for amount in split_amounts:
-                random.shuffle(channels_order)
+                random.shuffle(channel_keys)
                 # we check each channel and try to put the funds inside, break if we succeed
-                for c in channels_order:
+                for c in channel_keys:
                     if c not in config:
                         config[c] = []
-                    if sum(config[c]) + amount <= channels_with_funds[c]:
+                    channel_funds, channel_slots = channels_with_funds[c]
+                    if sum(config[c]) + amount <= channel_funds and len(config[c]) < channel_slots:
                         config[c].append(amount)
                         break
                 # if we don't succeed to put the amount anywhere,
                 # we try to fill up channels and put the rest somewhere else
                 else:
                     distribute_amount = amount
-                    for c in channels_order:
-                        funds_left = channels_with_funds[c] - sum(config[c])
+                    for c in channel_keys:
+                        channel_funds, channel_slots = channels_with_funds[c]
+                        slots_left = channel_slots - len(config[c])
+                        if slots_left == 0:
+                            # no slot left in that channel
+                            continue
+                        funds_left = channel_funds - sum(config[c])
                         # it would be good to not fill the full channel if possible
                         add_amount = min(funds_left, distribute_amount)
                         config[c].append(add_amount)
@@ -165,7 +171,7 @@ def suggest_splits(
                         if distribute_amount == 0:
                             break
             if config.total_config_amount() != amount_msat:
-                raise NoPathFound('Cannot distribute payment over channels.')
+                continue
             if target_parts > 1 and config.is_any_amount_smaller_than_min_part_size():
                 if target_parts == 2:
                     # if there are already too small parts at the first split excluding single
@@ -175,6 +181,8 @@ def suggest_splits(
                 continue
             assert config.total_config_amount() == amount_msat
             configs.append(config)
+        if not configs:
+            raise NoPathFound('Cannot distribute payment over channels.')
 
     configs = remove_duplicates(configs)
 


Test patch:
diff --git a/tests/test_mpp_split.py b/tests/test_mpp_split.py
index 77c5e08b8c2c..2c6bfe244ac6 100644
--- a/tests/test_mpp_split.py
+++ b/tests/test_mpp_split.py
@@ -15,10 +15,10 @@ def setUp(self):
         random.seed(0)
         # key tuple denotes (channel_id, node_id)
         self.channels_with_funds = {
-            (b"0", b"0"): 1_000_000_000,
-            (b"1", b"1"): 500_000_000,
-            (b"2", b"0"): 302_000_000,
-            (b"3", b"2"): 101_000_000,
+            (b"0", b"0"): (1_000_000_000, 3),
+            (b"1", b"1"): (500_000_000, 2),
+            (b"2", b"0"): (302_000_000, 2),
+            (b"3", b"2"): (101_000_000, 1),
         }
 
     def tearDown(self):
@@ -51,7 +51,7 @@ def test_suggest_splits(self):
 
         with self.subTest(msg="do a payment with the maximal amount spendable over all channels"):
             splits = mpp_split.suggest_splits(
-                sum(self.channels_with_funds.values()), self.channels_with_funds, exclude_single_part_payments=True)
+                sum([x[0] for x in self.channels_with_funds.values()]), self.channels_with_funds, exclude_single_part_payments=True)
             self.assertEqual({
                 (b"0", b"0"): [1_000_000_000],
                 (b"1", b"1"): [500_000_000],
@@ -68,6 +68,25 @@ def test_suggest_splits(self):
             # a splitting of the parts into two
             self.assertEqual(2, splits[4].config.number_parts())
 
+        with self.subTest(msg="no htlc slots available"):
+            channels = self.channels_with_funds.copy()
+            # set all available slots to 0
+            for chan, (amount, _slots) in channels.items():
+                channels[chan] = (amount, 0)
+            with self.assertRaises(NoPathFound):
+                mpp_split.suggest_splits(20_000_000, channels, exclude_single_part_payments=False)
+
+        with self.subTest(msg="only one channel can add htlcs"):
+            channels = self.channels_with_funds.copy()
+            # set all available slots to 0 except for the first channel
+            for chan, (amount, _slots) in channels.items():
+                if chan != (b"0", b"0"):
+                    channels[chan] = (amount, 0)
+            splits = mpp_split.suggest_splits(1_000_000_000, channels, exclude_single_part_payments=True)
+            for split in splits:
+                # check that the whole amount has been split on this channel
+                self.assertEqual(sum(split.config[(b"0", b"0")]), 1_000_000_000)
+
     def test_send_to_single_node(self):
         splits = mpp_split.suggest_splits(1_000_000_000, self.channels_with_funds, exclude_single_part_payments=False, exclude_multinode_payments=True)
         for split in splits:
@@ -75,7 +94,10 @@ def test_send_to_single_node(self):
 
     def test_saturation(self):
         """Split configurations which spend the full amount in a channel should be avoided."""
-        channels_with_funds = {(b"0", b"0"): 159_799_733_076, (b"1", b"1"): 499_986_152_000}
+        channels_with_funds = {
+            (b"0", b"0"): (159_799_733_076, 1),
+            (b"1", b"1"): (499_986_152_000, 1)
+        }
         splits = mpp_split.suggest_splits(600_000_000_000, channels_with_funds, exclude_single_part_payments=True)
 
         uses_full_amount = False
@@ -114,18 +136,21 @@ def test_suggest_part_penalty(self):
 
     def test_suggest_splits_single_channel(self):
         channels_with_funds = {
-            (b"0", b"0"): 1_000_000_000,
+            (b"0", b"0"): (1_000_000_000, 3),
         }
-
         with self.subTest(msg="do a payment with the maximal amount spendable on a single channel"):
             splits = mpp_split.suggest_splits(1_000_000_000, channels_with_funds, exclude_single_part_payments=False)
+            self.assertEqual(1, len(splits[0].config[(b"0", b"0")]))
             self.assertEqual({(b"0", b"0"): [1_000_000_000]}, splits[0].config)
+
         with self.subTest(msg="test sending an amount greater than what we have available"):
             self.assertRaises(NoPathFound, mpp_split.suggest_splits, *(1_100_000_000, channels_with_funds))
+
         with self.subTest(msg="test sending a large amount over a single channel in chunks"):
             mpp_split.PART_PENALTY = 0.5
             splits = mpp_split.suggest_splits(1_000_000_000, channels_with_funds, exclude_single_part_payments=False)
             self.assertEqual(2, len(splits[0].config[(b"0", b"0")]))
+
         with self.subTest(msg="test sending a large amount over a single channel in chunks"):
             mpp_split.PART_PENALTY = 0.3
             splits = mpp_split.suggest_splits(1_000_000_000, channels_with_funds, exclude_single_part_payments=False)