Repository: bitcoindevkit/bdk
Pull request #1798: Refactor/use iterators to preselect utxos
Author: login: nymius
Labels: name: module-wallet
Created: 2025-01-13T20:32:28Z | Merged: 2025-03-06T00:06:13Z

### Description

There were multiple calls for de-duplication of selected UTxOs in `Wallet::create_tx`: ([1](https://github.com/bitcoindevkit/bdk/blob/abc305612160c0e2ce85c7bba4c2c162ff488adc/crates/wallet/src/wallet/mod.rs#L2016-L2020)) and ([2](https://github.com/bitcoindevkit/bdk/blob/abc305612160c0e2ce85c7bba4c2c162ff488adc/crates/wallet/src/wallet/mod.rs#L1452)).

As the test [`test_filter_duplicates`](https://github.com/bitcoindevkit/bdk/blob/master/crates/wallet/src/wallet/coin_selection.rs#L1666-L1695) shows, there are four possible cases for duplication of UTxOs while feeding the coin selection algorithms.

1. no duplication: out of concern
2. duplication in the required utxos only: covered by the source of `required_utxos`, `Wallet::list_unspent`, which [roots back the provided `UTxOs` to a `HashMap`](https://github.com/bitcoindevkit/bdk/blob/a5335a18431dfecea1f524af44c953b79a96867c/crates/chain/src/tx_graph.rs#L911-L912) which should avoid any duplication by definition
3. duplication in the optional utxos only: is the only one possible as optional `UTxOs` are stored in a `Vec` and no checks are performed about the duplicity of their members.
4. duplication across the required and optional utxos: is already covered by `Wallet::preselect_utxos`, which avoid the processing of required UTxOs while listing the unspent available UTxOs in the wallet.

This refactor does the following:
- Refactors `TxParams::utxos` type to be `HashSet<LocalOutput>` avoiding the duplication case 3
- Keeps avoiding case 4 without overhead and allowing a query closer to O(1) on avg. to cover duplication case 4 (before was O(n) where n is the size of required utxos) thanks to the above change.
- Still covers case 2 because the [source of `required_utxos`, `Wallet::list_unspent`](https://github.com/bitcoindevkit/bdk/blob/a5335a18431dfecea1f524af44c953b79a96867c/crates/chain/src/tx_graph.rs#L911-L912) comes from a `HashMap` which should avoid duplication by definition.
- Moves the computation of the `WeightedUtxos` to the last part of UTxO filtering, allowing the unification of the computation for local outputs.
- Removes some extra allocations done for helpers structures or intermediate results while filtering UTxOs.
- Allows for future integration of UTxO filtering methods for other utilities, e.g.: filter locked utxos
  ```rust
  .filter(|utxo| !self.is_utxo_locked(utxo.outpoint, self.latest_checkpoint().height()))
  ```
- Adds more comments for each filtering step.
- Differentiates `foreign_utxos`, which should include a provided satisfation weight to use them effectively, and `utxos`, manually selected UTxOs for which the wallet can compute their satisfaction weight without external resources.

With these changes all four cases would be covered, and `coin_selection::filter_duplicates` is no longer needed.

Fixes bitcoindevkit/bdk#1794.

### Notes to the reviewers

I added three test to cover the interesting cases for duplication:
    - there are no duplicates in required utxos
    - there are no duplicates in optional utxos
    - there are no duplicates across optional and required utxos
the three of them have been prefixed with `not_duplicated_utxos*` to allow its joint execution under the command:
```bash
cargo test -- not_duplicated_utxos
```
because the guarantees for the three conditions above are spread in different parts of the code.

### Changelog notice

No changes to public APIs.

### Checklists

* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
* [x] This pull request does not break the existing API
* [x] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR



Diff:
diff --git a/crates/wallet/src/wallet/coin_selection.rs b/crates/wallet/src/wallet/coin_selection.rs
index 4429fae73..e5654fb4d 100644
--- a/crates/wallet/src/wallet/coin_selection.rs
+++ b/crates/wallet/src/wallet/coin_selection.rs
@@ -101,7 +101,6 @@
 //! # Ok::<(), anyhow::Error>(())
 //! ```
 
-use crate::chain::collections::HashSet;
 use crate::wallet::utils::IsDust;
 use crate::Utxo;
 use crate::WeightedUtxo;
@@ -109,7 +108,6 @@ use bitcoin::{Amount, FeeRate, SignedAmount};
 
 use alloc::vec::Vec;
 use bitcoin::consensus::encode::serialize;
-use bitcoin::OutPoint;
 use bitcoin::TxIn;
 use bitcoin::{Script, Weight};
 
@@ -720,30 +718,12 @@ fn calculate_cs_result(
     }
 }
 
-/// Remove duplicate UTXOs.
-///
-/// If a UTXO appears in both `required` and `optional`, the appearance in `required` is kept.
-pub(crate) fn filter_duplicates<I>(required: I, optional: I) -> (I, I)
-where
-    I: IntoIterator<Item = WeightedUtxo> + FromIterator<WeightedUtxo>,
-{
-    let mut visited = HashSet::<OutPoint>::new();
-    let required = required
-        .into_iter()
-        .filter(|utxo| visited.insert(utxo.utxo.outpoint()))
-        .collect::<I>();
-    let optional = optional
-        .into_iter()
-        .filter(|utxo| visited.insert(utxo.utxo.outpoint()))
-        .collect::<I>();
-    (required, optional)
-}
-
 #[cfg(test)]
 mod test {
     use assert_matches::assert_matches;
     use bitcoin::hashes::Hash;
-    use chain::{BlockId, ChainPosition, ConfirmationBlockTime};
+    use bitcoin::OutPoint;
+    use chain::{ChainPosition, ConfirmationBlockTime};
     use core::str::FromStr;
     use rand::rngs::StdRng;
 
@@ -751,7 +731,6 @@ mod test {
 
     use super::*;
     use crate::types::*;
-    use crate::wallet::coin_selection::filter_duplicates;
 
     use rand::prelude::SliceRandom;
     use rand::{thread_rng, Rng, RngCore, SeedableRng};
@@ -1618,102 +1597,6 @@ mod test {
         assert_eq!(res.selected_amount(), Amount::from_sat(200_000));
     }
 
-    #[test]
-    fn test_filter_duplicates() {
-        fn utxo(txid: &str, value: u64) -> WeightedUtxo {
-            WeightedUtxo {
-                satisfaction_weight: Weight::ZERO,
-                utxo: Utxo::Local(LocalOutput {
-                    outpoint: OutPoint::new(bitcoin::hashes::Hash::hash(txid.as_bytes()), 0),
-                    txout: TxOut {
-                        value: Amount::from_sat(value),
-                        script_pubkey: ScriptBuf::new(),
-                    },
-                    keychain: KeychainKind::External,
-                    is_spent: false,
-                    derivation_index: 0,
-                    chain_position: ChainPosition::Confirmed {
-                        anchor: ConfirmationBlockTime {
-                            block_id: BlockId {
-                                height: 12345,
-                                hash: BlockHash::all_zeros(),
-                            },
-                            confirmation_time: 12345,
-                        },
-                        transitively: None,
-                    },
-                }),
-            }
-        }
-
-        fn to_utxo_vec(utxos: &[(&str, u64)]) -> Vec<WeightedUtxo> {
-            let mut v = utxos
-                .iter()
-                .map(|&(txid, value)| utxo(txid, value))
-                .collect::<Vec<_>>();
-            v.sort_by_key(|u| u.utxo.outpoint());
-            v
-        }
-
-        struct TestCase<'a> {
-            name: &'a str,
-            required: &'a [(&'a str, u64)],
-            optional: &'a [(&'a str, u64)],
-            exp_required: &'a [(&'a str, u64)],
-            exp_optional: &'a [(&'a str, u64)],
-        }
-
-        let test_cases = [
-            TestCase {
-                name: "no_duplicates",
-                required: &[("A", 1000), ("B", 2100)],
-                optional: &[("C", 1000)],
-                exp_required: &[("A", 1000), ("B", 2100)],
-                exp_optional: &[("C", 1000)],
-            },
-            TestCase {
-                name: "duplicate_required_utxos",
-                required: &[("A", 3000), ("B", 1200), ("C", 1234), ("A", 3000)],
-                optional: &[("D", 2100)],
-                exp_required: &[("A", 3000), ("B", 1200), ("C", 1234)],
-                exp_optional: &[("D", 2100)],
-            },
-            TestCase {
-                name: "duplicate_optional_utxos",
-                required: &[("A", 3000), ("B", 1200)],
-                optional: &[("C", 5000), ("D", 1300), ("C", 5000)],
-                exp_required: &[("A", 3000), ("B", 1200)],
-                exp_optional: &[("C", 5000), ("D", 1300)],
-            },
-            TestCase {
-                name: "duplicate_across_required_and_optional_utxos",
-                required: &[("A", 3000), ("B", 1200), ("C", 2100)],
-                optional: &[("A", 3000), ("D", 1200), ("E", 5000)],
-                exp_required: &[("A", 3000), ("B", 1200), ("C", 2100)],
-                exp_optional: &[("D", 1200), ("E", 5000)],
-            },
-        ];
-
-        for (i, t) in test_cases.into_iter().enumerate() {
-            let (required, optional) =
-                filter_duplicates(to_utxo_vec(t.required), to_utxo_vec(t.optional));
-            assert_eq!(
-                required,
-                to_utxo_vec(t.exp_required),
-                "[{}:{}] unexpected `required` result",
-                i,
-                t.name
-            );
-            assert_eq!(
-                optional,
-                to_utxo_vec(t.exp_optional),
-                "[{}:{}] unexpected `optional` result",
-                i,
-                t.name
-            );
-        }
-    }
-
     #[test]
     fn test_deterministic_coin_selection_picks_same_utxos() {
         enum CoinSelectionAlgo {
diff --git a/crates/wallet/src/wallet/mod.rs b/crates/wallet/src/wallet/mod.rs
index 6e00b1094..e4dc6d056 100644
--- a/crates/wallet/src/wallet/mod.rs
+++ b/crates/wallet/src/wallet/mod.rs
@@ -36,7 +36,7 @@ use bdk_chain::{
 use bitcoin::{
     absolute,
     consensus::encode::serialize,
-    constants::{genesis_block, COINBASE_MATURITY},
+    constants::genesis_block,
     psbt,
     secp256k1::Secp256k1,
     sighash::{EcdsaSighashType, TapSighashType},
@@ -1417,8 +1417,19 @@ impl Wallet {
 
         fee_amount += fee_rate * tx.weight();
 
-        let (required_utxos, optional_utxos) =
-            self.preselect_utxos(&params, Some(current_height.to_consensus_u32()));
+        let (required_utxos, optional_utxos) = {
+            // NOTE: manual selection overrides unspendable
+            let mut required: Vec<WeightedUtxo> = params.utxos.values().cloned().collect();
+            let optional = self.filter_utxos(&params, current_height.to_consensus_u32());
+
+            // if drain_wallet is true, all UTxOs are required
+            if params.drain_wallet {
+                required.extend(optional);
+                (required, vec![])
+            } else {
+                (required, optional)
+            }
+        };
 
         // get drain script
         let mut drain_index = Option::<(KeychainKind, u32)>::None;
@@ -1447,9 +1458,6 @@ impl Wallet {
             }
         };
 
-        let (required_utxos, optional_utxos) =
-            coin_selection::filter_duplicates(required_utxos, optional_utxos);
-
         let coin_selection = coin_selection
             .coin_select(
                 required_utxos,
@@ -1618,60 +1626,71 @@ impl Wallet {
             .map_err(|_| BuildFeeBumpError::FeeRateUnavailable)?;
 
         // remove the inputs from the tx and process them
-        let original_txin = tx.input.drain(..).collect::<Vec<_>>();
-        let original_utxos = original_txin
-            .iter()
+        let utxos = tx
+            .input
+            .drain(..)
             .map(|txin| -> Result<_, BuildFeeBumpError> {
-                let prev_tx = graph
+                graph
+                    // Get previous transaction
                     .get_tx(txin.previous_output.txid)
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-                let txout = &prev_tx.output[txin.previous_output.vout as usize];
-
-                let chain_position = chain_positions
-                    .get(&txin.previous_output.txid)
-                    .cloned()
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-
-                let weighted_utxo = match txout_index.index_of_spk(txout.script_pubkey.clone()) {
-                    Some(&(keychain, derivation_index)) => {
-                        let satisfaction_weight = self
-                            .public_descriptor(keychain)
-                            .max_weight_to_satisfy()
-                            .unwrap();
-                        WeightedUtxo {
-                            utxo: Utxo::Local(LocalOutput {
-                                outpoint: txin.previous_output,
-                                txout: txout.clone(),
-                                keychain,
-                                is_spent: true,
-                                derivation_index,
-                                chain_position,
-                            }),
-                            satisfaction_weight,
-                        }
-                    }
-                    None => {
-                        let satisfaction_weight = Weight::from_wu_usize(
-                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
-                        );
-                        WeightedUtxo {
-                            utxo: Utxo::Foreign {
-                                outpoint: txin.previous_output,
-                                sequence: txin.sequence,
-                                psbt_input: Box::new(psbt::Input {
-                                    witness_utxo: Some(txout.clone()),
-                                    non_witness_utxo: Some(prev_tx.as_ref().clone()),
-                                    ..Default::default()
-                                }),
-                            },
-                            satisfaction_weight,
+                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))
+                    // Get chain position
+                    .and_then(|prev_tx| {
+                        chain_positions
+                            .get(&txin.previous_output.txid)
+                            .cloned()
+                            .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))
+                            .map(|chain_position| (prev_tx, chain_position))
+                    })
+                    .map(|(prev_tx, chain_position)| {
+                        let txout = prev_tx.output[txin.previous_output.vout as usize].clone();
+                        match txout_index.index_of_spk(txout.script_pubkey.clone()) {
+                            Some(&(keychain, derivation_index)) => (
+                                txin.previous_output,
+                                WeightedUtxo {
+                                    satisfaction_weight: self
+                                        .public_descriptor(keychain)
+                                        .max_weight_to_satisfy()
+                                        .unwrap(),
+                                    utxo: Utxo::Local(LocalOutput {
+                                        outpoint: txin.previous_output,
+                                        txout: txout.clone(),
+                                        keychain,
+                                        is_spent: true,
+                                        derivation_index,
+                                        chain_position,
+                                    }),
+                                },
+                            ),
+                            None => {
+                                let satisfaction_weight = Weight::from_wu_usize(
+                                    serialize(&txin.script_sig).len() * 4
+                                        + serialize(&txin.witness).len(),
+                                );
+
+                                (
+                                    txin.previous_output,
+                                    WeightedUtxo {
+                                        utxo: Utxo::Foreign {
+                                            outpoint: txin.previous_output,
+                                            sequence: txin.sequence,
+                                            psbt_input: Box::new(psbt::Input {
+                                                witness_utxo: txout
+                                                    .script_pubkey
+                                                    .witness_version()
+                                                    .map(|_| txout.clone()),
+                                                non_witness_utxo: Some(prev_tx.as_ref().clone()),
+                                                ..Default::default()
+                                            }),
+                                        },
+                                        satisfaction_weight,
+                                    },
+                                )
+                            }
                         }
-                    }
-                };
-
-                Ok(weighted_utxo)
+                    })
             })
-            .collect::<Result<Vec<_>, _>>()?;
+            .collect::<Result<HashMap<OutPoint, WeightedUtxo>, BuildFeeBumpError>>()?;
 
         if tx.output.len() > 1 {
             let mut change_index = None;
@@ -1698,7 +1717,7 @@ impl Wallet {
                 .into_iter()
                 .map(|txout| (txout.script_pubkey, txout.value))
                 .collect(),
-            utxos: original_utxos,
+            utxos,
             bumping_fee: Some(tx_builder::PreviousFee {
                 absolute: fee,
                 rate: fee_rate,
@@ -1976,117 +1995,52 @@ impl Wallet {
         descriptor.at_derivation_index(child).ok()
     }
 
-    fn get_available_utxos(&self) -> Vec<(LocalOutput, Weight)> {
-        self.list_unspent()
-            .map(|utxo| {
-                let keychain = utxo.keychain;
-                (utxo, {
-                    self.public_descriptor(keychain)
-                        .max_weight_to_satisfy()
-                        .unwrap()
-                })
-            })
-            .collect()
-    }
-
     /// Given the options returns the list of utxos that must be used to form the
     /// transaction and any further that may be used if needed.
-    fn preselect_utxos(
-        &self,
-        params: &TxParams,
-        current_height: Option<u32>,
-    ) -> (Vec<WeightedUtxo>, Vec<WeightedUtxo>) {
-        let TxParams {
-            change_policy,
-            unspendable,
-            utxos,
-            drain_wallet,
-            manually_selected_only,
-            bumping_fee,
-            ..
-        } = params;
-
-        let manually_selected = utxos.clone();
-        // we mandate confirmed transactions if we're bumping the fee
-        let must_only_use_confirmed_tx = bumping_fee.is_some();
-        let must_use_all_available = *drain_wallet;
-
-        //    must_spend <- manually selected utxos
-        //    may_spend  <- all other available utxos
-        let mut may_spend = self.get_available_utxos();
-
-        may_spend.retain(|may_spend| {
-            !manually_selected
-                .iter()
-                .any(|manually_selected| manually_selected.utxo.outpoint() == may_spend.0.outpoint)
-        });
-        let mut must_spend = manually_selected;
-
-        // NOTE: we are intentionally ignoring `unspendable` here. i.e manual
-        // selection overrides unspendable.
-        if *manually_selected_only {
-            return (must_spend, vec![]);
-        }
-
-        let satisfies_confirmed = may_spend
-            .iter()
-            .map(|u| -> bool {
-                let txid = u.0.outpoint.txid;
-                let tx = match self.indexed_graph.graph().get_tx(txid) {
-                    Some(tx) => tx,
-                    None => return false,
-                };
-
-                // Whether the UTXO is mature and, if needed, confirmed
-                let mut spendable = true;
-                let chain_position = u.0.chain_position;
-                if must_only_use_confirmed_tx && !chain_position.is_confirmed() {
-                    return false;
-                }
-                if tx.is_coinbase() {
-                    debug_assert!(
-                        chain_position.is_confirmed(),
-                        "coinbase must always be confirmed"
-                    );
-                    if let Some(current_height) = current_height {
-                        match chain_position {
-                            ChainPosition::Confirmed { anchor, .. } => {
-                                // https://github.com/bitcoin/bitcoin/blob/c5e67be03bb06a5d7885c55db1f016fbf2333fe3/src/validation.cpp#L373-L375
-                                let spend_height = current_height + 1;
-                                let coin_age_at_spend_height =
-                                    spend_height.saturating_sub(anchor.block_id.height);
-                                spendable &= coin_age_at_spend_height >= COINBASE_MATURITY;
-                            }
-                            ChainPosition::Unconfirmed { .. } => spendable = false,
-                        }
-                    }
-                }
-                spendable
-            })
-            .collect::<Vec<_>>();
-
-        let mut i = 0;
-        may_spend.retain(|u| {
-            let retain = (self.keychains().count() == 1 || change_policy.is_satisfied_by(&u.0))
-                && !unspendable.contains(&u.0.outpoint)
-                && satisfies_confirmed[i];
-            i += 1;
-            retain
-        });
-
-        let mut may_spend = may_spend
-            .into_iter()
-            .map(|(local_utxo, satisfaction_weight)| WeightedUtxo {
-                satisfaction_weight,
-                utxo: Utxo::Local(local_utxo),
-            })
-            .collect();
-
-        if must_use_all_available {
-            must_spend.append(&mut may_spend);
+    fn filter_utxos(&self, params: &TxParams, current_height: u32) -> Vec<WeightedUtxo> {
+        if params.manually_selected_only {
+            vec![]
+        // only process optional UTxOs if manually_selected_only is false
+        } else {
+            self.indexed_graph
+                .graph()
+                // get all unspent UTxOs from wallet
+                // NOTE: the UTxOs returned by the following method already belong to wallet as the
+                // call chain uses get_tx_node infallibly
+                .filter_chain_unspents(
+                    &self.chain,
+                    self.chain.tip().block_id(),
+                    self.indexed_graph.index.outpoints().iter().cloned(),
+                )
+                // only create LocalOutput if UTxO is mature
+                .filter_map(move |((k, i), full_txo)| {
+                    full_txo
+                        .is_mature(current_height)
+                        .then(|| new_local_utxo(k, i, full_txo))
+                })
+                // only process UTxOs not selected manually, they will be considered later in the chain
+                // NOTE: this avoid UTxOs in both required and optional list
+                .filter(|may_spend| !params.utxos.contains_key(&may_spend.outpoint))
+                // only add to optional UTxOs those which satisfy the change policy if we reuse change
+                .filter(|local_output| {
+                    self.keychains().count() == 1
+                        || params.change_policy.is_satisfied_by(local_output)
+                })
+                // only add to optional UTxOs those marked as spendable
+                .filter(|local_output| !params.unspendable.contains(&local_output.outpoint))
+                // if bumping fees only add to optional UTxOs those confirmed
+                .filter(|local_output| {
+                    params.bumping_fee.is_none() || local_output.chain_position.is_confirmed()
+                })
+                .map(|utxo| WeightedUtxo {
+                    satisfaction_weight: self
+                        .public_descriptor(utxo.keychain)
+                        .max_weight_to_satisfy()
+                        .unwrap(),
+                    utxo: Utxo::Local(utxo),
+                })
+                .collect()
         }
-
-        (must_spend, may_spend)
     }
 
     fn complete_transaction(
@@ -2605,3 +2559,73 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let mut params = TxParams::default();
+        let output = wallet.get_utxo(OutPoint { txid, vout: 0 }).unwrap();
+        params.utxos.insert(
+            output.outpoint,
+            WeightedUtxo {
+                satisfaction_weight: wallet
+                    .public_descriptor(output.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(output),
+            },
+        );
+        // enforce selection of first output in transaction
+        let received = wallet.filter_utxos(&params, wallet.latest_checkpoint().block_id().height);
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![wallet
+            .get_utxo(OutPoint { txid, vout: 1 })
+            .map(|utxo| WeightedUtxo {
+                satisfaction_weight: wallet
+                    .public_descriptor(utxo.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo),
+            })
+            .unwrap()];
+
+        assert_eq!(expected, received);
+    }
+}
diff --git a/crates/wallet/src/wallet/tx_builder.rs b/crates/wallet/src/wallet/tx_builder.rs
index da41c6b00..235987fc0 100644
--- a/crates/wallet/src/wallet/tx_builder.rs
+++ b/crates/wallet/src/wallet/tx_builder.rs
@@ -52,7 +52,7 @@ use rand_core::RngCore;
 use super::coin_selection::CoinSelectionAlgorithm;
 use super::utils::shuffle_slice;
 use super::{CreateTxError, Wallet};
-use crate::collections::{BTreeMap, HashSet};
+use crate::collections::{BTreeMap, HashMap, HashSet};
 use crate::{KeychainKind, LocalOutput, Utxo, WeightedUtxo};
 
 /// A transaction builder
@@ -125,7 +125,7 @@ pub(crate) struct TxParams {
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) utxos: HashMap<OutPoint, WeightedUtxo>,
     pub(crate) unspendable: HashSet<OutPoint>,
     pub(crate) manually_selected_only: bool,
     pub(crate) sighash: Option<psbt::PsbtSighashType>,
@@ -274,26 +274,28 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
-        {
-            let wallet = &mut self.wallet;
-            let utxos = outpoints
-                .iter()
-                .map(|outpoint| {
-                    wallet
-                        .get_utxo(*outpoint)
-                        .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
-                })
-                .collect::<Result<Vec<_>, _>>()?;
-
-            for utxo in utxos {
-                let descriptor = wallet.public_descriptor(utxo.keychain);
-                let satisfaction_weight = descriptor.max_weight_to_satisfy().unwrap();
-                self.params.utxos.push(WeightedUtxo {
-                    satisfaction_weight,
-                    utxo: Utxo::Local(utxo),
-                });
-            }
-        }
+        let utxo_batch = outpoints
+            .iter()
+            .map(|outpoint| {
+                self.wallet
+                    .get_utxo(*outpoint)
+                    .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
+                    .map(|output| {
+                        (
+                            *outpoint,
+                            WeightedUtxo {
+                                satisfaction_weight: self
+                                    .wallet
+                                    .public_descriptor(output.keychain)
+                                    .max_weight_to_satisfy()
+                                    .unwrap(),
+                                utxo: Utxo::Local(output),
+                            },
+                        )
+                    })
+            })
+            .collect::<Result<HashMap<OutPoint, WeightedUtxo>, AddUtxoError>>()?;
+        self.params.utxos.extend(utxo_batch);
 
         Ok(self)
     }
@@ -306,7 +308,14 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
         self.add_utxos(&[outpoint])
     }
 
-    /// Add a foreign UTXO i.e. a UTXO not owned by this wallet.
+    /// Add a foreign UTXO i.e. a UTXO not known by this wallet.
+    ///
+    /// There might be cases where the UTxO belongs to the wallet but it doesn't have knowledge of
+    /// it. This is possible if the wallet is not synced or its not being use to track
+    /// transactions. In those cases is the responsibility of the user to add any possible local
+    /// UTxOs through the [`TxBuilder::add_utxo`] method.
+    /// A manually added local UTxO will always have greater precedence than a foreign utxo. No
+    /// matter if it was added before or after the foreign UTxO.
     ///
     /// At a minimum to add a foreign UTXO we need:
     ///
@@ -393,14 +402,25 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
             }
         }
 
-        self.params.utxos.push(WeightedUtxo {
-            satisfaction_weight,
-            utxo: Utxo::Foreign {
+        if let Some(WeightedUtxo {
+            utxo: Utxo::Local { .. },
+            ..
+        }) = self.params.utxos.get(&outpoint)
+        {
+            None
+        } else {
+            self.params.utxos.insert(
                 outpoint,
-                sequence,
-                psbt_input: Box::new(psbt_input),
-            },
-        });
+                WeightedUtxo {
+                    satisfaction_weight,
+                    utxo: Utxo::Foreign {
+                        outpoint,
+                        sequence,
+                        psbt_input: Box::new(psbt_input),
+                    },
+                },
+            )
+        };
 
         Ok(self)
     }
@@ -1098,4 +1118,228 @@ mod test {
         builder.fee_rate(FeeRate::from_sat_per_kwu(feerate + 250));
         let _ = builder.finish().unwrap();
     }
+
+    #[test]
+    fn not_duplicated_utxos_in_required_list() {
+        let mut params = TxParams::default();
+        let test_utxos = get_test_utxos();
+        let fake_weighted_utxo = WeightedUtxo {
+            satisfaction_weight: Weight::from_wu(0),
+            utxo: Utxo::Local(test_utxos[0].clone()),
+        };
+        for _ in 0..3 {
+            params
+                .utxos
+                .insert(test_utxos[0].outpoint, fake_weighted_utxo.clone());
+        }
+        assert_eq!(
+            vec![(test_utxos[0].outpoint, fake_weighted_utxo)],
+            params.utxos.into_iter().collect::<Vec<_>>()
+        );
+    }
+
+    #[test]
+    fn not_duplicated_foreign_utxos_with_same_outpoint_but_different_weight() {
+        use crate::test_utils::{get_funded_wallet_single, get_funded_wallet_wpkh, get_test_wpkh};
+
+        // Use two different wallets to avoid adding local utxos
+        let (wallet1, txid1) = get_funded_wallet_wpkh();
+        let (mut wallet2, txid2) = get_funded_wallet_single(get_test_wpkh());
+
+        // if the transactions were produced by the same wallet the following assert should fail
+        assert_ne!(txid1, txid2);
+
+        let utxo1 = wallet1.list_unspent().next().unwrap();
+        let tx1 = wallet1.get_tx(txid1).unwrap().tx_node.tx.clone();
+
+        let satisfaction_weight = wallet1
+            .public_descriptor(KeychainKind::External)
+            .max_weight_to_satisfy()
+            .unwrap();
+
+        let mut builder = wallet2.build_tx();
+
+        // add foreign utxo with satisfaction weight x
+        assert!(builder
+            .add_foreign_utxo(
+                utxo1.outpoint,
+                psbt::Input {
+                    non_witness_utxo: Some(tx1.as_ref().clone()),
+                    ..Default::default()
+                },
+                satisfaction_weight,
+            )
+            .is_ok());
+
+        let modified_satisfaction_weight = satisfaction_weight - Weight::from_wu(6);
+
+        assert_ne!(satisfaction_weight, modified_satisfaction_weight);
+
+        // add foreign utxo with same outpoint but satisfaction weight x - 6wu
+        assert!(builder
+            .add_foreign_utxo(
+                utxo1.outpoint,
+                psbt::Input {
+                    non_witness_utxo: Some(tx1.as_ref().clone()),
+                    ..Default::default()
+                },
+                modified_satisfaction_weight,
+            )
+            .is_ok());
+
+        let foreign_utxo_with_modified_weight =
+            builder.params.utxos.values().collect::<Vec<_>>()[0];
+
+        assert_eq!(builder.params.utxos.len(), 1);
+        assert_eq!(
+            foreign_utxo_with_modified_weight.satisfaction_weight,
+            modified_satisfaction_weight
+        );
+    }
+
+    #[test]
+    fn test_prexisting_local_utxo_have_precedence_over_foreign_utxo_with_same_outpoint() {
+        // In this test we are assuming a setup where there are two wallets using the same
+        // descriptor, but only one is tracking transactions, while the other is not.
+        // Within this conditions we want the second wallet to be able to consider the unknown
+        // LocalOutputs provided by the first wallet with greater precedence than any foreign utxo,
+        // even if the foreign utxo shares the same outpoint Remember the second wallet does not
+        // know about any UTxOs, so in principle, an unknown local utxo could be added as foreign.
+        //
+        // In this case, somehow the wallet has knowledge of one local utxo and it tries to add the
+        // same utxo as a foreign one, but the API ignores this, because local utxos have higher
+        // precedence.
+        use crate::test_utils::{get_funded_wallet_wpkh, get_test_wpkh_and_change_desc};
+        use bitcoin::Network;
+
+        // Use the same wallet twice
+        let (wallet1, txid1) = get_funded_wallet_wpkh();
+        // But the second one has no knowledge of tx associated with txid1
+        let (main_descriptor, change_descriptor) = get_test_wpkh_and_change_desc();
+        let mut wallet2 = Wallet::create(main_descriptor, change_descriptor)
+            .network(Network::Regtest)
+            .create_wallet_no_persist()
+            .expect("descriptors must be valid");
+
+        let utxo1 = wallet1.list_unspent().next().unwrap();
+        let tx1 = wallet1.get_tx(txid1).unwrap().tx_node.tx.clone();
+
+        let satisfaction_weight = wallet1
+            .public_descriptor(KeychainKind::External)
+            .max_weight_to_satisfy()
+            .unwrap();
+
+        let mut builder = wallet2.build_tx();
+
+        // Add local UTxO manually, through tx_builder private parameters and not through
+        // add_utxo method because we are assuming wallet2 has not knowledge of utxo1 yet
+        builder.params.utxos.insert(
+            utxo1.outpoint,
+            WeightedUtxo {
+                satisfaction_weight: wallet1
+                    .public_descriptor(utxo1.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo1.clone()),
+            },
+        );
+
+        // add foreign utxo
+        assert!(builder
+            .add_foreign_utxo(
+                utxo1.outpoint,
+                psbt::Input {
+                    non_witness_utxo: Some(tx1.as_ref().clone()),
+                    ..Default::default()
+                },
+                satisfaction_weight,
+            )
+            .is_ok());
+
+        let utxo_should_still_be_local = builder.params.utxos.values().collect::<Vec<_>>()[0];
+
+        assert_eq!(builder.params.utxos.len(), 1);
+        assert_eq!(utxo_should_still_be_local.utxo.outpoint(), utxo1.outpoint);
+        // UTxO should still be LocalOutput
+        assert!(matches!(
+            utxo_should_still_be_local,
+            WeightedUtxo {
+                utxo: Utxo::Local(..),
+                ..
+            }
+        ));
+    }
+
+    #[test]
+    fn test_prexisting_foreign_utxo_have_no_precedence_over_local_utxo_with_same_outpoint() {
+        // In this test we are assuming a setup where there are two wallets using the same
+        // descriptor, but only one is tracking transactions, while the other is not.
+        // Within this conditions we want the second wallet to be able to consider the unknown
+        // LocalOutputs provided by the first wallet with greater precedence than any foreign utxo,
+        // even if the foreign utxo shares the same outpoint Remember the second wallet does not
+        // know about any UTxOs, so in principle, an unknown local utxo could be added as foreign.
+        //
+        // In this case, the wallet adds a local utxo as if it were foreign and after this it adds
+        // it as local utxo. In this case the local utxo should still have precedence over the
+        // foreign utxo.
+        use crate::test_utils::{get_funded_wallet_wpkh, get_test_wpkh_and_change_desc};
+        use bitcoin::Network;
+
+        // Use the same wallet twice
+        let (wallet1, txid1) = get_funded_wallet_wpkh();
+        // But the second one has no knowledge of tx associated with txid1
+        let (main_descriptor, change_descriptor) = get_test_wpkh_and_change_desc();
+        let mut wallet2 = Wallet::create(main_descriptor, change_descriptor)
+            .network(Network::Regtest)
+            .create_wallet_no_persist()
+            .expect("descriptors must be valid");
+
+        let utxo1 = wallet1.list_unspent().next().unwrap();
+        let tx1 = wallet1.get_tx(txid1).unwrap().tx_node.tx.clone();
+
+        let satisfaction_weight = wallet1
+            .public_descriptor(KeychainKind::External)
+            .max_weight_to_satisfy()
+            .unwrap();
+
+        let mut builder = wallet2.build_tx();
+
+        // add foreign utxo
+        assert!(builder
+            .add_foreign_utxo(
+                utxo1.outpoint,
+                psbt::Input {
+                    non_witness_utxo: Some(tx1.as_ref().clone()),
+                    ..Default::default()
+                },
+                satisfaction_weight,
+            )
+            .is_ok());
+
+        // Add local UTxO manually, through tx_builder private parameters and not through
+        // add_utxo method because we are assuming wallet2 has not knowledge of utxo1 yet
+        builder.params.utxos.insert(
+            utxo1.outpoint,
+            WeightedUtxo {
+                satisfaction_weight: wallet1
+                    .public_descriptor(utxo1.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo1.clone()),
+            },
+        );
+
+        let utxo_should_still_be_local = builder.params.utxos.values().collect::<Vec<_>>()[0];
+
+        assert_eq!(builder.params.utxos.len(), 1);
+        assert_eq!(utxo_should_still_be_local.utxo.outpoint(), utxo1.outpoint);
+        // UTxO should still be LocalOutput
+        assert!(matches!(
+            utxo_should_still_be_local,
+            WeightedUtxo {
+                utxo: Utxo::Local(..),
+                ..
+            }
+        ));
+    }
 }


Reviews:
user: login: evanlinjin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-01-14T06:29:36Z
user: login: ValuedMammal
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-03T23:43:22Z
user: login: ValuedMammal
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-04T00:14:17Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T15:21:39Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T17:04:29Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T18:20:47Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T20:18:47Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T20:19:15Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T20:19:44Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T20:20:30Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-05T20:23:07Z
user: login: ValuedMammal
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-12T16:30:41Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-12T19:15:15Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-12T20:23:31Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-13T12:52:36Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-14T21:02:16Z
user: login: ValuedMammal
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-25T15:17:22Z
user: login: evanlinjin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T05:49:00Z
user: login: evanlinjin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T04:29:29Z
user: login: evanlinjin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T04:45:18Z
user: login: evanlinjin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T04:48:13Z
user: login: evanlinjin
body: Documentation suggestions. Otherwise, LGTM
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T04:59:47Z
user: login: evanlinjin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T05:03:48Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T06:29:36Z
user: login: evanlinjin
body: ACK 2f83b4508fe2369c6336c9579517f3a4eb25d923

Just a nit for future reference.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T00:05:17Z
user: login: nymius
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T00:13:24Z

Review comments:
diff_hunk: @@ -1976,116 +1968,85 @@ impl Wallet {
         descriptor.at_derivation_index(child).ok()
     }
 
-    fn get_available_utxos(&self) -> Vec<(LocalOutput, Weight)> {
-        self.list_unspent()
-            .map(|utxo| {
-                let keychain = utxo.keychain;
-                (utxo, {
-                    self.public_descriptor(keychain)
-                        .max_weight_to_satisfy()
-                        .unwrap()
-                })
-            })
-            .collect()
-    }
-
     /// Given the options returns the list of utxos that must be used to form the
     /// transaction and any further that may be used if needed.
     fn preselect_utxos(
         &self,
         params: &TxParams,
         current_height: Option<u32>,
     ) -> (Vec<WeightedUtxo>, Vec<WeightedUtxo>) {
-        let TxParams {
-            change_policy,
-            unspendable,
-            utxos,
-            drain_wallet,
-            manually_selected_only,
-            bumping_fee,
-            ..
-        } = params;
-
-        let manually_selected = utxos.clone();
-        // we mandate confirmed transactions if we're bumping the fee
-        let must_only_use_confirmed_tx = bumping_fee.is_some();
-        let must_use_all_available = *drain_wallet;
-
-        //    must_spend <- manually selected utxos
-        //    may_spend  <- all other available utxos
-        let mut may_spend = self.get_available_utxos();
-
-        may_spend.retain(|may_spend| {
-            !manually_selected
-                .iter()
-                .any(|manually_selected| manually_selected.utxo.outpoint() == may_spend.0.outpoint)
-        });
-        let mut must_spend = manually_selected;
-
-        // NOTE: we are intentionally ignoring `unspendable` here. i.e manual
-        // selection overrides unspendable.
-        if *manually_selected_only {
-            return (must_spend, vec![]);
-        }
-
-        let satisfies_confirmed = may_spend
-            .iter()
-            .map(|u| -> bool {
-                let txid = u.0.outpoint.txid;
-                let tx = match self.indexed_graph.graph().get_tx(txid) {
-                    Some(tx) => tx,
-                    None => return false,
-                };
-
-                // Whether the UTXO is mature and, if needed, confirmed
-                let mut spendable = true;
-                let chain_position = u.0.chain_position;
-                if must_only_use_confirmed_tx && !chain_position.is_confirmed() {
-                    return false;
-                }
-                if tx.is_coinbase() {
-                    debug_assert!(
-                        chain_position.is_confirmed(),
-                        "coinbase must always be confirmed"
-                    );
-                    if let Some(current_height) = current_height {
-                        match chain_position {
-                            ChainPosition::Confirmed { anchor, .. } => {
-                                // https://github.com/bitcoin/bitcoin/blob/c5e67be03bb06a5d7885c55db1f016fbf2333fe3/src/validation.cpp#L373-L375
-                                spendable &= (current_height
-                                    .saturating_sub(anchor.block_id.height))
-                                    >= COINBASE_MATURITY;
-                            }
-                            ChainPosition::Unconfirmed { .. } => spendable = false,
-                        }
-                    }
+        self
+            // get all unspent UTxOs from wallet
+            .list_unspent()
+            // only process UTxOs not selected manually, they will be considered later in the chain
+            // NOTE: this avoid UTxOs in both required and optional list
+            .filter(|may_spend| !params.utxos.contains(may_spend))
+            // only process optional UTxOs if manually_selected_only is false
+            .take_while(|_| !params.manually_selected_only)
path: crates/wallet/src/wallet/mod.rs
user: login: evanlinjin
body: I think you are trying to be too fancy here. It'll be easier to read if we use an if statement and have `may_spend` as a separate variable.

```rust

let may_spend = if params.manually_selected_only {
    // This will be empty
} else {
    self.list_unspent()
    // etc...
};
```
created_at: 2025-01-14T06:29:36Z
author_association: MEMBER
diff_hunk: @@ -1976,116 +1993,52 @@ impl Wallet {
         descriptor.at_derivation_index(child).ok()
     }
 
-    fn get_available_utxos(&self) -> Vec<(LocalOutput, Weight)> {
-        self.list_unspent()
-            .map(|utxo| {
-                let keychain = utxo.keychain;
-                (utxo, {
-                    self.public_descriptor(keychain)
-                        .max_weight_to_satisfy()
-                        .unwrap()
-                })
-            })
-            .collect()
-    }
-
     /// Given the options returns the list of utxos that must be used to form the
     /// transaction and any further that may be used if needed.
-    fn preselect_utxos(
-        &self,
-        params: &TxParams,
-        current_height: Option<u32>,
-    ) -> (Vec<WeightedUtxo>, Vec<WeightedUtxo>) {
-        let TxParams {
-            change_policy,
-            unspendable,
-            utxos,
-            drain_wallet,
-            manually_selected_only,
-            bumping_fee,
-            ..
-        } = params;
-
-        let manually_selected = utxos.clone();
-        // we mandate confirmed transactions if we're bumping the fee
-        let must_only_use_confirmed_tx = bumping_fee.is_some();
-        let must_use_all_available = *drain_wallet;
-
-        //    must_spend <- manually selected utxos
-        //    may_spend  <- all other available utxos
-        let mut may_spend = self.get_available_utxos();
-
-        may_spend.retain(|may_spend| {
-            !manually_selected
-                .iter()
-                .any(|manually_selected| manually_selected.utxo.outpoint() == may_spend.0.outpoint)
-        });
-        let mut must_spend = manually_selected;
-
-        // NOTE: we are intentionally ignoring `unspendable` here. i.e manual
-        // selection overrides unspendable.
-        if *manually_selected_only {
-            return (must_spend, vec![]);
-        }
-
-        let satisfies_confirmed = may_spend
-            .iter()
-            .map(|u| -> bool {
-                let txid = u.0.outpoint.txid;
-                let tx = match self.indexed_graph.graph().get_tx(txid) {
-                    Some(tx) => tx,
-                    None => return false,
-                };
-
-                // Whether the UTXO is mature and, if needed, confirmed
-                let mut spendable = true;
-                let chain_position = u.0.chain_position;
-                if must_only_use_confirmed_tx && !chain_position.is_confirmed() {
-                    return false;
-                }
-                if tx.is_coinbase() {
-                    debug_assert!(
-                        chain_position.is_confirmed(),
-                        "coinbase must always be confirmed"
-                    );
-                    if let Some(current_height) = current_height {
-                        match chain_position {
-                            ChainPosition::Confirmed { anchor, .. } => {
-                                // https://github.com/bitcoin/bitcoin/blob/c5e67be03bb06a5d7885c55db1f016fbf2333fe3/src/validation.cpp#L373-L375
-                                spendable &= (current_height
-                                    .saturating_sub(anchor.block_id.height))
-                                    >= COINBASE_MATURITY;
-                            }
-                            ChainPosition::Unconfirmed { .. } => spendable = false,
-                        }
-                    }
-                }
-                spendable
-            })
-            .collect::<Vec<_>>();
-
-        let mut i = 0;
-        may_spend.retain(|u| {
-            let retain = (self.keychains().count() == 1 || change_policy.is_satisfied_by(&u.0))
-                && !unspendable.contains(&u.0.outpoint)
-                && satisfies_confirmed[i];
-            i += 1;
-            retain
-        });
-
-        let mut may_spend = may_spend
-            .into_iter()
-            .map(|(local_utxo, satisfaction_weight)| WeightedUtxo {
-                satisfaction_weight,
-                utxo: Utxo::Local(local_utxo),
-            })
-            .collect();
-
-        if must_use_all_available {
-            must_spend.append(&mut may_spend);
+    fn filter_utxos<'a>(&'a self, params: &'a TxParams, current_height: u32) -> Vec<WeightedUtxo> {
path: crates/wallet/src/wallet/mod.rs
user: login: ValuedMammal
body: The lifetime `'a` doesn't seem to be needed.
created_at: 2025-02-03T23:26:48Z
author_association: COLLABORATOR
diff_hunk: @@ -2604,3 +2557,137 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_anchor;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        let mut params = TxParams::default();
+        wallet
+            .get_utxo(OutPoint { txid, vout: 0 })
+            .map(|outpoint| {
+                params.utxos.insert(outpoint);
+            })
+            .unwrap();
+        // enforce selection of first output in transaction
+        let received = wallet.filter_utxos(&params, wallet.latest_checkpoint().block_id().height);
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![wallet
+            .get_utxo(OutPoint { txid, vout: 1 })
+            .map(|utxo| WeightedUtxo {
+                satisfaction_weight: wallet
+                    .public_descriptor(utxo.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo),
+            })
+            .unwrap()];
+
+        assert_eq!(expected, received);
+    }
+
+    #[test]
+    fn not_duplicated_utxos_in_optional_list() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+
+        for _ in 0..3 {
+            insert_tx(&mut wallet, two_output_tx.clone());
+        }
path: crates/wallet/src/wallet/mod.rs
user: login: ValuedMammal
body: Inserting the same tx again doesn't add more txs to the wallet.
created_at: 2025-02-03T23:29:17Z
author_association: COLLABORATOR
diff_hunk: @@ -2604,3 +2557,137 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_anchor;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        let mut params = TxParams::default();
+        wallet
+            .get_utxo(OutPoint { txid, vout: 0 })
+            .map(|outpoint| {
+                params.utxos.insert(outpoint);
+            })
+            .unwrap();
+        // enforce selection of first output in transaction
+        let received = wallet.filter_utxos(&params, wallet.latest_checkpoint().block_id().height);
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![wallet
+            .get_utxo(OutPoint { txid, vout: 1 })
+            .map(|utxo| WeightedUtxo {
+                satisfaction_weight: wallet
+                    .public_descriptor(utxo.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo),
+            })
+            .unwrap()];
+
+        assert_eq!(expected, received);
+    }
+
+    #[test]
+    fn not_duplicated_utxos_in_optional_list() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+
+        for _ in 0..3 {
+            insert_tx(&mut wallet, two_output_tx.clone());
+        }
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        // enforce selection of first output in transaction
+        let received = wallet
+            .indexed_graph
+            .graph()
+            .filter_chain_unspents(
+                &wallet.chain,
+                wallet.chain.tip().block_id(),
+                wallet.indexed_graph.index.outpoints().iter().cloned(),
+            )
+            .map(|(_, full_txout)| full_txout.outpoint)
+            .collect::<Vec<_>>();
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![OutPoint { txid, vout: 0 }, OutPoint { txid, vout: 1 }];
path: crates/wallet/src/wallet/mod.rs
user: login: ValuedMammal
body: I'm confused by this comment, is it left over from the other test?

I think this test would be better if we add a bunch of utxos (~10) to the wallet, then call `filter_utxos` and check that 1) none are duplicated and 2) none that we expect to be present are filtered out by accident.
created_at: 2025-02-03T23:37:29Z
author_association: COLLABORATOR
diff_hunk: @@ -3875,8 +3875,8 @@ fn test_spend_coinbase() {
     };
     insert_anchor(&mut wallet, txid, anchor);
 
-    let not_yet_mature_time = confirmation_height + COINBASE_MATURITY - 1;
-    let maturity_time = confirmation_height + COINBASE_MATURITY;
+    let not_yet_mature_time = confirmation_height + COINBASE_MATURITY - 2;
+    let maturity_time = confirmation_height + COINBASE_MATURITY - 1;
path: crates/wallet/tests/wallet.rs
user: login: ValuedMammal
body: A code comment might help to explain why this change
created_at: 2025-02-03T23:39:12Z
author_association: COLLABORATOR
diff_hunk: @@ -1618,60 +1640,54 @@ impl Wallet {
             .map_err(|_| BuildFeeBumpError::FeeRateUnavailable)?;
 
         // remove the inputs from the tx and process them
-        let original_txin = tx.input.drain(..).collect::<Vec<_>>();
-        let original_utxos = original_txin
-            .iter()
-            .map(|txin| -> Result<_, BuildFeeBumpError> {
-                let prev_tx = graph
-                    .get_tx(txin.previous_output.txid)
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-                let txout = &prev_tx.output[txin.previous_output.vout as usize];
-
-                let chain_position = chain_positions
-                    .get(&txin.previous_output.txid)
-                    .cloned()
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-
-                let weighted_utxo = match txout_index.index_of_spk(txout.script_pubkey.clone()) {
-                    Some(&(keychain, derivation_index)) => {
-                        let satisfaction_weight = self
-                            .public_descriptor(keychain)
-                            .max_weight_to_satisfy()
-                            .unwrap();
-                        WeightedUtxo {
-                            utxo: Utxo::Local(LocalOutput {
-                                outpoint: txin.previous_output,
-                                txout: txout.clone(),
-                                keychain,
-                                is_spent: true,
-                                derivation_index,
-                                chain_position,
-                            }),
-                            satisfaction_weight,
-                        }
-                    }
-                    None => {
-                        let satisfaction_weight = Weight::from_wu_usize(
-                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
-                        );
-                        WeightedUtxo {
-                            utxo: Utxo::Foreign {
-                                outpoint: txin.previous_output,
-                                sequence: txin.sequence,
-                                psbt_input: Box::new(psbt::Input {
-                                    witness_utxo: Some(txout.clone()),
-                                    non_witness_utxo: Some(prev_tx.as_ref().clone()),
-                                    ..Default::default()
-                                }),
-                            },
-                            satisfaction_weight,
+        let (local, foreign): (HashSet<LocalOutput>, Vec<WeightedUtxo>) =
+            tx.input.drain(..).try_fold(
+                (HashSet::<LocalOutput>::new(), Vec::<WeightedUtxo>::new()),
+                |(mut local, mut foreign), txin| -> Result<_, BuildFeeBumpError> {
+                    let prev_tx = graph
+                        .get_tx(txin.previous_output.txid)
+                        .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+                    let txout = &prev_tx.output[txin.previous_output.vout as usize];
+
+                    let chain_position =
+                        chain_positions
+                            .get(&txin.previous_output.txid)
+                            .cloned()
+                            .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+
+                    match txout_index.index_of_spk(txout.script_pubkey.clone()) {
+                        Some(&(keychain, derivation_index)) => local.insert(LocalOutput {
+                            outpoint: txin.previous_output,
+                            txout: txout.clone(),
+                            keychain,
+                            is_spent: true,
+                            derivation_index,
+                            chain_position,
+                        }),
+                        None => {
+                            let satisfaction_weight = Weight::from_wu_usize(
+                                serialize(&txin.script_sig).len() * 4
+                                    + serialize(&txin.witness).len(),
+                            );
+                            foreign.push(WeightedUtxo {
+                                utxo: Utxo::Foreign {
+                                    outpoint: txin.previous_output,
+                                    sequence: txin.sequence,
+                                    psbt_input: Box::new(psbt::Input {
+                                        witness_utxo: Some(txout.clone()),
path: crates/wallet/src/wallet/mod.rs
user: login: ValuedMammal
body: I know this was here already but shouldn't we only set `witness_utxo` if `txout.script_pubkey.witness_version().is_some()` ?
created_at: 2025-02-04T00:14:17Z
author_association: COLLABORATOR
diff_hunk: @@ -2604,3 +2557,137 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_anchor;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        let mut params = TxParams::default();
+        wallet
+            .get_utxo(OutPoint { txid, vout: 0 })
+            .map(|outpoint| {
+                params.utxos.insert(outpoint);
+            })
+            .unwrap();
+        // enforce selection of first output in transaction
+        let received = wallet.filter_utxos(&params, wallet.latest_checkpoint().block_id().height);
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![wallet
+            .get_utxo(OutPoint { txid, vout: 1 })
+            .map(|utxo| WeightedUtxo {
+                satisfaction_weight: wallet
+                    .public_descriptor(utxo.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo),
+            })
+            .unwrap()];
+
+        assert_eq!(expected, received);
+    }
+
+    #[test]
+    fn not_duplicated_utxos_in_optional_list() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+
+        for _ in 0..3 {
+            insert_tx(&mut wallet, two_output_tx.clone());
+        }
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        // enforce selection of first output in transaction
+        let received = wallet
+            .indexed_graph
+            .graph()
+            .filter_chain_unspents(
+                &wallet.chain,
+                wallet.chain.tip().block_id(),
+                wallet.indexed_graph.index.outpoints().iter().cloned(),
+            )
+            .map(|(_, full_txout)| full_txout.outpoint)
+            .collect::<Vec<_>>();
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![OutPoint { txid, vout: 0 }, OutPoint { txid, vout: 1 }];
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: 
> I'm confused by this comment, is it left over from the other test?

I'm confused too, probably I changed the test after the comment.

> 
> I think this test would be better if we add a bunch of utxos (~10) to the wallet, then call `filter_utxos` and check that 1) none are duplicated and 2) none that we expect to be present are filtered out by accident.

What I wanted to prove with the tests I implemented is that the only needed test case for `filter_utxos` is when there are duplicated utxos across optional and required.
In order to do that, here, I wanted to prove is impossible for `filter_chain_unspent` to produce duplicated optional utxos. That's why I mistakenly tried to insert the same tx multiple times. The thing is I have no knowledge of any way to incorporate duplicated utxos in the wallet bypassing those checks, and as the list is generated from an iterator on demand, the only way I can think right now is some kind of mock for the method, as it isn't an input of `filter_utxos`.
created_at: 2025-02-05T15:21:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1618,60 +1640,54 @@ impl Wallet {
             .map_err(|_| BuildFeeBumpError::FeeRateUnavailable)?;
 
         // remove the inputs from the tx and process them
-        let original_txin = tx.input.drain(..).collect::<Vec<_>>();
-        let original_utxos = original_txin
-            .iter()
-            .map(|txin| -> Result<_, BuildFeeBumpError> {
-                let prev_tx = graph
-                    .get_tx(txin.previous_output.txid)
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-                let txout = &prev_tx.output[txin.previous_output.vout as usize];
-
-                let chain_position = chain_positions
-                    .get(&txin.previous_output.txid)
-                    .cloned()
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-
-                let weighted_utxo = match txout_index.index_of_spk(txout.script_pubkey.clone()) {
-                    Some(&(keychain, derivation_index)) => {
-                        let satisfaction_weight = self
-                            .public_descriptor(keychain)
-                            .max_weight_to_satisfy()
-                            .unwrap();
-                        WeightedUtxo {
-                            utxo: Utxo::Local(LocalOutput {
-                                outpoint: txin.previous_output,
-                                txout: txout.clone(),
-                                keychain,
-                                is_spent: true,
-                                derivation_index,
-                                chain_position,
-                            }),
-                            satisfaction_weight,
-                        }
-                    }
-                    None => {
-                        let satisfaction_weight = Weight::from_wu_usize(
-                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
-                        );
-                        WeightedUtxo {
-                            utxo: Utxo::Foreign {
-                                outpoint: txin.previous_output,
-                                sequence: txin.sequence,
-                                psbt_input: Box::new(psbt::Input {
-                                    witness_utxo: Some(txout.clone()),
-                                    non_witness_utxo: Some(prev_tx.as_ref().clone()),
-                                    ..Default::default()
-                                }),
-                            },
-                            satisfaction_weight,
+        let (local, foreign): (HashSet<LocalOutput>, Vec<WeightedUtxo>) =
+            tx.input.drain(..).try_fold(
+                (HashSet::<LocalOutput>::new(), Vec::<WeightedUtxo>::new()),
+                |(mut local, mut foreign), txin| -> Result<_, BuildFeeBumpError> {
+                    let prev_tx = graph
+                        .get_tx(txin.previous_output.txid)
+                        .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+                    let txout = &prev_tx.output[txin.previous_output.vout as usize];
+
+                    let chain_position =
+                        chain_positions
+                            .get(&txin.previous_output.txid)
+                            .cloned()
+                            .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+
+                    match txout_index.index_of_spk(txout.script_pubkey.clone()) {
+                        Some(&(keychain, derivation_index)) => local.insert(LocalOutput {
+                            outpoint: txin.previous_output,
+                            txout: txout.clone(),
+                            keychain,
+                            is_spent: true,
+                            derivation_index,
+                            chain_position,
+                        }),
+                        None => {
+                            let satisfaction_weight = Weight::from_wu_usize(
+                                serialize(&txin.script_sig).len() * 4
+                                    + serialize(&txin.witness).len(),
+                            );
+                            foreign.push(WeightedUtxo {
+                                utxo: Utxo::Foreign {
+                                    outpoint: txin.previous_output,
+                                    sequence: txin.sequence,
+                                    psbt_input: Box::new(psbt::Input {
+                                        witness_utxo: Some(txout.clone()),
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: Is a good check. I assume `witness_utxos` and `non_witness_utxos` are kind of exclusive, unless we don't know the nature of the `UTxO` being spend, right?
created_at: 2025-02-05T17:04:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2604,3 +2557,137 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_anchor;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        let mut params = TxParams::default();
+        wallet
+            .get_utxo(OutPoint { txid, vout: 0 })
+            .map(|outpoint| {
+                params.utxos.insert(outpoint);
+            })
+            .unwrap();
+        // enforce selection of first output in transaction
+        let received = wallet.filter_utxos(&params, wallet.latest_checkpoint().block_id().height);
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![wallet
+            .get_utxo(OutPoint { txid, vout: 1 })
+            .map(|utxo| WeightedUtxo {
+                satisfaction_weight: wallet
+                    .public_descriptor(utxo.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo),
+            })
+            .unwrap()];
+
+        assert_eq!(expected, received);
+    }
+
+    #[test]
+    fn not_duplicated_utxos_in_optional_list() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+
+        for _ in 0..3 {
+            insert_tx(&mut wallet, two_output_tx.clone());
+        }
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        // enforce selection of first output in transaction
+        let received = wallet
+            .indexed_graph
+            .graph()
+            .filter_chain_unspents(
+                &wallet.chain,
+                wallet.chain.tip().block_id(),
+                wallet.indexed_graph.index.outpoints().iter().cloned(),
+            )
+            .map(|(_, full_txout)| full_txout.outpoint)
+            .collect::<Vec<_>>();
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![OutPoint { txid, vout: 0 }, OutPoint { txid, vout: 1 }];
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: Will remove the test for now, as I don't find a deterministic way to produce duplicated utxos which will be de duplicated later by wallet logic.
In relation to 2) I think [this line](https://github.com/bitcoindevkit/bdk/pull/1798/files#diff-641f4600ade27412ccd63976bcbff11451a7b34703e84d953f864d6cb02a2290R2616) should be enough to assert that.
created_at: 2025-02-05T18:20:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2604,3 +2557,137 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_anchor;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        let mut params = TxParams::default();
+        wallet
+            .get_utxo(OutPoint { txid, vout: 0 })
+            .map(|outpoint| {
+                params.utxos.insert(outpoint);
+            })
+            .unwrap();
+        // enforce selection of first output in transaction
+        let received = wallet.filter_utxos(&params, wallet.latest_checkpoint().block_id().height);
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![wallet
+            .get_utxo(OutPoint { txid, vout: 1 })
+            .map(|utxo| WeightedUtxo {
+                satisfaction_weight: wallet
+                    .public_descriptor(utxo.keychain)
+                    .max_weight_to_satisfy()
+                    .unwrap(),
+                utxo: Utxo::Local(utxo),
+            })
+            .unwrap()];
+
+        assert_eq!(expected, received);
+    }
+
+    #[test]
+    fn not_duplicated_utxos_in_optional_list() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+
+        for _ in 0..3 {
+            insert_tx(&mut wallet, two_output_tx.clone());
+        }
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
+
+        // enforce selection of first output in transaction
+        let received = wallet
+            .indexed_graph
+            .graph()
+            .filter_chain_unspents(
+                &wallet.chain,
+                wallet.chain.tip().block_id(),
+                wallet.indexed_graph.index.outpoints().iter().cloned(),
+            )
+            .map(|(_, full_txout)| full_txout.outpoint)
+            .collect::<Vec<_>>();
+        // notice expected doesn't include the first output from two_output_tx as it should be
+        // filtered out
+        let expected = vec![OutPoint { txid, vout: 0 }, OutPoint { txid, vout: 1 }];
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: Test removed in fe3c17e513b41ab3e5933e7bc45b8e1e865931ec
created_at: 2025-02-05T20:18:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1618,60 +1640,54 @@ impl Wallet {
             .map_err(|_| BuildFeeBumpError::FeeRateUnavailable)?;
 
         // remove the inputs from the tx and process them
-        let original_txin = tx.input.drain(..).collect::<Vec<_>>();
-        let original_utxos = original_txin
-            .iter()
-            .map(|txin| -> Result<_, BuildFeeBumpError> {
-                let prev_tx = graph
-                    .get_tx(txin.previous_output.txid)
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-                let txout = &prev_tx.output[txin.previous_output.vout as usize];
-
-                let chain_position = chain_positions
-                    .get(&txin.previous_output.txid)
-                    .cloned()
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-
-                let weighted_utxo = match txout_index.index_of_spk(txout.script_pubkey.clone()) {
-                    Some(&(keychain, derivation_index)) => {
-                        let satisfaction_weight = self
-                            .public_descriptor(keychain)
-                            .max_weight_to_satisfy()
-                            .unwrap();
-                        WeightedUtxo {
-                            utxo: Utxo::Local(LocalOutput {
-                                outpoint: txin.previous_output,
-                                txout: txout.clone(),
-                                keychain,
-                                is_spent: true,
-                                derivation_index,
-                                chain_position,
-                            }),
-                            satisfaction_weight,
-                        }
-                    }
-                    None => {
-                        let satisfaction_weight = Weight::from_wu_usize(
-                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
-                        );
-                        WeightedUtxo {
-                            utxo: Utxo::Foreign {
-                                outpoint: txin.previous_output,
-                                sequence: txin.sequence,
-                                psbt_input: Box::new(psbt::Input {
-                                    witness_utxo: Some(txout.clone()),
-                                    non_witness_utxo: Some(prev_tx.as_ref().clone()),
-                                    ..Default::default()
-                                }),
-                            },
-                            satisfaction_weight,
+        let (local, foreign): (HashSet<LocalOutput>, Vec<WeightedUtxo>) =
+            tx.input.drain(..).try_fold(
+                (HashSet::<LocalOutput>::new(), Vec::<WeightedUtxo>::new()),
+                |(mut local, mut foreign), txin| -> Result<_, BuildFeeBumpError> {
+                    let prev_tx = graph
+                        .get_tx(txin.previous_output.txid)
+                        .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+                    let txout = &prev_tx.output[txin.previous_output.vout as usize];
+
+                    let chain_position =
+                        chain_positions
+                            .get(&txin.previous_output.txid)
+                            .cloned()
+                            .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+
+                    match txout_index.index_of_spk(txout.script_pubkey.clone()) {
+                        Some(&(keychain, derivation_index)) => local.insert(LocalOutput {
+                            outpoint: txin.previous_output,
+                            txout: txout.clone(),
+                            keychain,
+                            is_spent: true,
+                            derivation_index,
+                            chain_position,
+                        }),
+                        None => {
+                            let satisfaction_weight = Weight::from_wu_usize(
+                                serialize(&txin.script_sig).len() * 4
+                                    + serialize(&txin.witness).len(),
+                            );
+                            foreign.push(WeightedUtxo {
+                                utxo: Utxo::Foreign {
+                                    outpoint: txin.previous_output,
+                                    sequence: txin.sequence,
+                                    psbt_input: Box::new(psbt::Input {
+                                        witness_utxo: Some(txout.clone()),
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: Check added in c4f7c0e273bef379bf76469db7921649288897eb
created_at: 2025-02-05T20:19:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1976,116 +1993,52 @@ impl Wallet {
         descriptor.at_derivation_index(child).ok()
     }
 
-    fn get_available_utxos(&self) -> Vec<(LocalOutput, Weight)> {
-        self.list_unspent()
-            .map(|utxo| {
-                let keychain = utxo.keychain;
-                (utxo, {
-                    self.public_descriptor(keychain)
-                        .max_weight_to_satisfy()
-                        .unwrap()
-                })
-            })
-            .collect()
-    }
-
     /// Given the options returns the list of utxos that must be used to form the
     /// transaction and any further that may be used if needed.
-    fn preselect_utxos(
-        &self,
-        params: &TxParams,
-        current_height: Option<u32>,
-    ) -> (Vec<WeightedUtxo>, Vec<WeightedUtxo>) {
-        let TxParams {
-            change_policy,
-            unspendable,
-            utxos,
-            drain_wallet,
-            manually_selected_only,
-            bumping_fee,
-            ..
-        } = params;
-
-        let manually_selected = utxos.clone();
-        // we mandate confirmed transactions if we're bumping the fee
-        let must_only_use_confirmed_tx = bumping_fee.is_some();
-        let must_use_all_available = *drain_wallet;
-
-        //    must_spend <- manually selected utxos
-        //    may_spend  <- all other available utxos
-        let mut may_spend = self.get_available_utxos();
-
-        may_spend.retain(|may_spend| {
-            !manually_selected
-                .iter()
-                .any(|manually_selected| manually_selected.utxo.outpoint() == may_spend.0.outpoint)
-        });
-        let mut must_spend = manually_selected;
-
-        // NOTE: we are intentionally ignoring `unspendable` here. i.e manual
-        // selection overrides unspendable.
-        if *manually_selected_only {
-            return (must_spend, vec![]);
-        }
-
-        let satisfies_confirmed = may_spend
-            .iter()
-            .map(|u| -> bool {
-                let txid = u.0.outpoint.txid;
-                let tx = match self.indexed_graph.graph().get_tx(txid) {
-                    Some(tx) => tx,
-                    None => return false,
-                };
-
-                // Whether the UTXO is mature and, if needed, confirmed
-                let mut spendable = true;
-                let chain_position = u.0.chain_position;
-                if must_only_use_confirmed_tx && !chain_position.is_confirmed() {
-                    return false;
-                }
-                if tx.is_coinbase() {
-                    debug_assert!(
-                        chain_position.is_confirmed(),
-                        "coinbase must always be confirmed"
-                    );
-                    if let Some(current_height) = current_height {
-                        match chain_position {
-                            ChainPosition::Confirmed { anchor, .. } => {
-                                // https://github.com/bitcoin/bitcoin/blob/c5e67be03bb06a5d7885c55db1f016fbf2333fe3/src/validation.cpp#L373-L375
-                                spendable &= (current_height
-                                    .saturating_sub(anchor.block_id.height))
-                                    >= COINBASE_MATURITY;
-                            }
-                            ChainPosition::Unconfirmed { .. } => spendable = false,
-                        }
-                    }
-                }
-                spendable
-            })
-            .collect::<Vec<_>>();
-
-        let mut i = 0;
-        may_spend.retain(|u| {
-            let retain = (self.keychains().count() == 1 || change_policy.is_satisfied_by(&u.0))
-                && !unspendable.contains(&u.0.outpoint)
-                && satisfies_confirmed[i];
-            i += 1;
-            retain
-        });
-
-        let mut may_spend = may_spend
-            .into_iter()
-            .map(|(local_utxo, satisfaction_weight)| WeightedUtxo {
-                satisfaction_weight,
-                utxo: Utxo::Local(local_utxo),
-            })
-            .collect();
-
-        if must_use_all_available {
-            must_spend.append(&mut may_spend);
+    fn filter_utxos<'a>(&'a self, params: &'a TxParams, current_height: u32) -> Vec<WeightedUtxo> {
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: Removed in 4c28ba52c60415f66b2c32238bc83c61d5a18df3
created_at: 2025-02-05T20:19:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3875,8 +3875,8 @@ fn test_spend_coinbase() {
     };
     insert_anchor(&mut wallet, txid, anchor);
 
-    let not_yet_mature_time = confirmation_height + COINBASE_MATURITY - 1;
-    let maturity_time = confirmation_height + COINBASE_MATURITY;
+    let not_yet_mature_time = confirmation_height + COINBASE_MATURITY - 2;
+    let maturity_time = confirmation_height + COINBASE_MATURITY - 1;
path: crates/wallet/tests/wallet.rs
user: login: nymius
body: Commented at e8e21e1178a8b7bd781611b50d3ac5962135c241
created_at: 2025-02-05T20:20:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1976,116 +1968,85 @@ impl Wallet {
         descriptor.at_derivation_index(child).ok()
     }
 
-    fn get_available_utxos(&self) -> Vec<(LocalOutput, Weight)> {
-        self.list_unspent()
-            .map(|utxo| {
-                let keychain = utxo.keychain;
-                (utxo, {
-                    self.public_descriptor(keychain)
-                        .max_weight_to_satisfy()
-                        .unwrap()
-                })
-            })
-            .collect()
-    }
-
     /// Given the options returns the list of utxos that must be used to form the
     /// transaction and any further that may be used if needed.
     fn preselect_utxos(
         &self,
         params: &TxParams,
         current_height: Option<u32>,
     ) -> (Vec<WeightedUtxo>, Vec<WeightedUtxo>) {
-        let TxParams {
-            change_policy,
-            unspendable,
-            utxos,
-            drain_wallet,
-            manually_selected_only,
-            bumping_fee,
-            ..
-        } = params;
-
-        let manually_selected = utxos.clone();
-        // we mandate confirmed transactions if we're bumping the fee
-        let must_only_use_confirmed_tx = bumping_fee.is_some();
-        let must_use_all_available = *drain_wallet;
-
-        //    must_spend <- manually selected utxos
-        //    may_spend  <- all other available utxos
-        let mut may_spend = self.get_available_utxos();
-
-        may_spend.retain(|may_spend| {
-            !manually_selected
-                .iter()
-                .any(|manually_selected| manually_selected.utxo.outpoint() == may_spend.0.outpoint)
-        });
-        let mut must_spend = manually_selected;
-
-        // NOTE: we are intentionally ignoring `unspendable` here. i.e manual
-        // selection overrides unspendable.
-        if *manually_selected_only {
-            return (must_spend, vec![]);
-        }
-
-        let satisfies_confirmed = may_spend
-            .iter()
-            .map(|u| -> bool {
-                let txid = u.0.outpoint.txid;
-                let tx = match self.indexed_graph.graph().get_tx(txid) {
-                    Some(tx) => tx,
-                    None => return false,
-                };
-
-                // Whether the UTXO is mature and, if needed, confirmed
-                let mut spendable = true;
-                let chain_position = u.0.chain_position;
-                if must_only_use_confirmed_tx && !chain_position.is_confirmed() {
-                    return false;
-                }
-                if tx.is_coinbase() {
-                    debug_assert!(
-                        chain_position.is_confirmed(),
-                        "coinbase must always be confirmed"
-                    );
-                    if let Some(current_height) = current_height {
-                        match chain_position {
-                            ChainPosition::Confirmed { anchor, .. } => {
-                                // https://github.com/bitcoin/bitcoin/blob/c5e67be03bb06a5d7885c55db1f016fbf2333fe3/src/validation.cpp#L373-L375
-                                spendable &= (current_height
-                                    .saturating_sub(anchor.block_id.height))
-                                    >= COINBASE_MATURITY;
-                            }
-                            ChainPosition::Unconfirmed { .. } => spendable = false,
-                        }
-                    }
+        self
+            // get all unspent UTxOs from wallet
+            .list_unspent()
+            // only process UTxOs not selected manually, they will be considered later in the chain
+            // NOTE: this avoid UTxOs in both required and optional list
+            .filter(|may_spend| !params.utxos.contains(may_spend))
+            // only process optional UTxOs if manually_selected_only is false
+            .take_while(|_| !params.manually_selected_only)
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: Done at e78b0e60b0b43579eb87ee3b4c746f2d3e62d65a
created_at: 2025-02-05T20:23:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,7 +125,8 @@ pub(crate) struct TxParams {
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) utxos: HashSet<LocalOutput>,
+    pub(crate) foreign_utxos: HashSet<WeightedUtxo>,
path: crates/wallet/src/wallet/tx_builder.rs
user: login: ValuedMammal
body: I'm worried that this won't be sufficient to check that the _outpoint_ is unique. For instance, one could change the satisfaction weight and get a different `WeightedUtxo` while using the same outpoint.
created_at: 2025-02-12T16:30:41Z
author_association: COLLABORATOR
diff_hunk: @@ -125,7 +125,8 @@ pub(crate) struct TxParams {
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) utxos: HashSet<LocalOutput>,
+    pub(crate) foreign_utxos: HashSet<WeightedUtxo>,
path: crates/wallet/src/wallet/tx_builder.rs
user: login: nymius
body: Good point, will think more about this. Will reproduce your case in a test and use a `HashMap` with outpoint as key in the meantime.
created_at: 2025-02-12T19:15:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,7 +125,8 @@ pub(crate) struct TxParams {
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) utxos: HashSet<LocalOutput>,
+    pub(crate) foreign_utxos: HashSet<WeightedUtxo>,
path: crates/wallet/src/wallet/tx_builder.rs
user: login: nymius
body: I modified the test slightly to cover that case. I didn't add a different one because it is not modifying the logic being tested, but just being more precise with the assertions.

I used `HashMap` instead of `BTreeMap` because we don't need the ordering features, however it can be argued that `BTreeMap` is already use it in `crates/wallet/src/wallet/tx_builder.rs` and we are adding a new import.
I prefered the former over the later.
created_at: 2025-02-12T20:23:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,7 +125,8 @@ pub(crate) struct TxParams {
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) utxos: HashSet<LocalOutput>,
+    pub(crate) foreign_utxos: HashSet<WeightedUtxo>,
path: crates/wallet/src/wallet/tx_builder.rs
user: login: nymius
body: To avoid the duplication concern we had for #1582, we can implement `Hash` manually for `WeightedUTxO` and make it use the `Hash` implementation of `Outpoint`.
created_at: 2025-02-13T12:52:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -125,7 +125,8 @@ pub(crate) struct TxParams {
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) utxos: HashSet<LocalOutput>,
+    pub(crate) foreign_utxos: HashSet<WeightedUtxo>,
path: crates/wallet/src/wallet/tx_builder.rs
user: login: nymius
body: I realized this won't be enough, as `HashSet` requires the following for keys `k1`, `k2`:
$k1 == k2 \Rightarrow \text{hash}(k1) == \text{hash}(k2)$.
But this should enforced by us. If we implement `Hash` as I said above and don't also modify `PartialEq` implementation to be compatible with this, the case where outpoint is the same but weight is different will introduce two different utxos in foreign utxos set.

I wouldn't modify `PartialEq` because is probably going to break other logic depending of the usual way of comparing two elements (i.e. considering all its fields).

`HashMap` is still the best solution so far.

created_at: 2025-02-14T21:02:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1618,60 +1640,70 @@ impl Wallet {
             .map_err(|_| BuildFeeBumpError::FeeRateUnavailable)?;
 
         // remove the inputs from the tx and process them
-        let original_txin = tx.input.drain(..).collect::<Vec<_>>();
-        let original_utxos = original_txin
-            .iter()
-            .map(|txin| -> Result<_, BuildFeeBumpError> {
-                let prev_tx = graph
-                    .get_tx(txin.previous_output.txid)
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-                let txout = &prev_tx.output[txin.previous_output.vout as usize];
-
-                let chain_position = chain_positions
-                    .get(&txin.previous_output.txid)
-                    .cloned()
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-
-                let weighted_utxo = match txout_index.index_of_spk(txout.script_pubkey.clone()) {
-                    Some(&(keychain, derivation_index)) => {
-                        let satisfaction_weight = self
-                            .public_descriptor(keychain)
-                            .max_weight_to_satisfy()
-                            .unwrap();
-                        WeightedUtxo {
-                            utxo: Utxo::Local(LocalOutput {
-                                outpoint: txin.previous_output,
-                                txout: txout.clone(),
-                                keychain,
-                                is_spent: true,
-                                derivation_index,
-                                chain_position,
-                            }),
-                            satisfaction_weight,
-                        }
-                    }
-                    None => {
-                        let satisfaction_weight = Weight::from_wu_usize(
-                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
-                        );
-                        WeightedUtxo {
-                            utxo: Utxo::Foreign {
-                                outpoint: txin.previous_output,
-                                sequence: txin.sequence,
-                                psbt_input: Box::new(psbt::Input {
+        let (local, foreign): (HashSet<LocalOutput>, HashMap<OutPoint, WeightedUtxo>) =
+            tx.input.drain(..).try_fold(
+                (
+                    HashSet::<LocalOutput>::new(),
+                    HashMap::<OutPoint, WeightedUtxo>::new(),
+                ),
+                |(mut local, mut foreign), txin| -> Result<_, BuildFeeBumpError> {
+                    let prev_tx = graph
+                        .get_tx(txin.previous_output.txid)
+                        .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+                    let txout = &prev_tx.output[txin.previous_output.vout as usize];
+
+                    let chain_position =
+                        chain_positions
+                            .get(&txin.previous_output.txid)
+                            .cloned()
+                            .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
+
+                    match txout_index.index_of_spk(txout.script_pubkey.clone()) {
+                        Some(&(keychain, derivation_index)) => local.insert(LocalOutput {
+                            outpoint: txin.previous_output,
+                            txout: txout.clone(),
+                            keychain,
+                            is_spent: true,
+                            derivation_index,
+                            chain_position,
+                        }),
+                        None => {
+                            let satisfaction_weight = Weight::from_wu_usize(
+                                serialize(&txin.script_sig).len() * 4
+                                    + serialize(&txin.witness).len(),
+                            );
+
+                            let psbt_input_data = if txout.script_pubkey.witness_version().is_some()
+                            {
+                                Box::new(psbt::Input {
                                     witness_utxo: Some(txout.clone()),
                                     non_witness_utxo: Some(prev_tx.as_ref().clone()),
                                     ..Default::default()
-                                }),
-                            },
-                            satisfaction_weight,
+                                })
+                            } else {
+                                Box::new(psbt::Input {
+                                    non_witness_utxo: Some(prev_tx.as_ref().clone()),
+                                    ..Default::default()
+                                })
+                            };
+                            foreign.insert(
+                                txin.previous_output,
+                                WeightedUtxo {
+                                    utxo: Utxo::Foreign {
+                                        outpoint: txin.previous_output,
+                                        sequence: txin.sequence,
+                                        psbt_input: psbt_input_data,
path: crates/wallet/src/wallet/mod.rs
user: login: ValuedMammal
body: I think this could also work, i.e. just set the `witness_utxo` based on the value of `witness_version`

```rust
    psbt_input: Box::new(psbt::Input {
        witness_utxo: txout
            .script_pubkey
            .witness_version()
            .map(|_| txout.clone()),
        non_witness_utxo: Some(prev_tx.as_ref().clone()),
        ..Default::default()
    }),
```
created_at: 2025-02-25T15:06:01Z
author_association: COLLABORATOR
diff_hunk: @@ -2605,3 +2573,76 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_anchor;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let expected_anchor = ConfirmationBlockTime {
+            block_id: wallet.latest_checkpoint().block_id(),
+            confirmation_time: 200,
+        };
+
+        insert_anchor(&mut wallet, txid, expected_anchor);
path: crates/wallet/src/wallet/mod.rs
user: login: ValuedMammal
body: nit: It doesn't really make sense to anchor the txid to block 0. `insert_tx` should be enough to add the utxos to the wallet.
created_at: 2025-02-25T15:06:10Z
author_association: COLLABORATOR
diff_hunk: @@ -274,26 +275,14 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
-        {
-            let wallet = &mut self.wallet;
-            let utxos = outpoints
-                .iter()
+        let _ = outpoints.iter().try_for_each(|outpoint| {
+            self.wallet
+                .get_utxo(*outpoint)
                 .map(|outpoint| {
-                    wallet
-                        .get_utxo(*outpoint)
-                        .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
+                    self.params.utxos.insert(outpoint);
path: crates/wallet/src/wallet/tx_builder.rs
user: login: ValuedMammal
body: nit: I would call this variable "output"
```suggestion
                    self.params.utxos.insert(output);
```
created_at: 2025-02-25T15:08:25Z
author_association: COLLABORATOR
diff_hunk: @@ -1098,4 +1090,61 @@ mod test {
         builder.fee_rate(FeeRate::from_sat_per_kwu(feerate + 250));
         let _ = builder.finish().unwrap();
     }
+
+    #[test]
+    fn not_duplicated_utxos_in_required_list() {
+        let mut params = TxParams::default();
+        let test_utxos = get_test_utxos();
+        for _ in 0..3 {
+            params.utxos.insert(test_utxos[0].clone());
+        }
+        assert_eq!(
+            vec![test_utxos[0].clone()],
+            params.utxos.into_iter().collect::<Vec<_>>()
+        );
+    }
+
+    #[test]
+    fn not_duplicated_foreign_utxos_with_same_outpoint_but_different_weight() {
+        use crate::test_utils::get_funded_wallet_wpkh;
+        // Use two different wallets to avoid adding local utxos
+        let (wallet1, txid1) = get_funded_wallet_wpkh();
path: crates/wallet/src/wallet/tx_builder.rs
user: login: ValuedMammal
body: You may want to use wallets with different descriptors.
```suggestion
        let (wallet1, txid1) = get_funded_wallet_single(get_test_wpkh());
```
created_at: 2025-02-25T15:13:24Z
author_association: COLLABORATOR
diff_hunk: @@ -274,26 +275,14 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
-        {
-            let wallet = &mut self.wallet;
-            let utxos = outpoints
-                .iter()
-                .map(|outpoint| {
-                    wallet
-                        .get_utxo(*outpoint)
-                        .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
+        let _ = outpoints.iter().try_for_each(|outpoint| {
+            self.wallet
+                .get_utxo(*outpoint)
+                .map(|output| {
+                    self.params.utxos.insert(output);
                 })
-                .collect::<Result<Vec<_>, _>>()?;
-
-            for utxo in utxos {
-                let descriptor = wallet.public_descriptor(utxo.keychain);
-                let satisfaction_weight = descriptor.max_weight_to_satisfy().unwrap();
-                self.params.utxos.push(WeightedUtxo {
-                    satisfaction_weight,
-                    utxo: Utxo::Local(utxo),
-                });
-            }
-        }
+                .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
+        });
path: crates/wallet/src/wallet/tx_builder.rs
user: login: evanlinjin
body: Styling nit: Just use a for loop.
created_at: 2025-02-27T05:27:21Z
author_association: MEMBER
diff_hunk: @@ -274,26 +275,14 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
-        {
-            let wallet = &mut self.wallet;
-            let utxos = outpoints
-                .iter()
-                .map(|outpoint| {
-                    wallet
-                        .get_utxo(*outpoint)
-                        .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
+        let _ = outpoints.iter().try_for_each(|outpoint| {
+            self.wallet
+                .get_utxo(*outpoint)
+                .map(|output| {
+                    self.params.utxos.insert(output);
                 })
-                .collect::<Result<Vec<_>, _>>()?;
-
-            for utxo in utxos {
-                let descriptor = wallet.public_descriptor(utxo.keychain);
-                let satisfaction_weight = descriptor.max_weight_to_satisfy().unwrap();
-                self.params.utxos.push(WeightedUtxo {
-                    satisfaction_weight,
-                    utxo: Utxo::Local(utxo),
-                });
-            }
-        }
+                .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
+        });
path: crates/wallet/src/wallet/tx_builder.rs
user: login: evanlinjin
body: I think it's better to not insert anything when this errors.
created_at: 2025-02-27T05:28:11Z
author_association: MEMBER
diff_hunk: @@ -125,7 +125,8 @@ pub(crate) struct TxParams {
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) utxos: HashSet<LocalOutput>,
+    pub(crate) foreign_utxos: HashMap<OutPoint, WeightedUtxo>,
path: crates/wallet/src/wallet/tx_builder.rs
user: login: evanlinjin
body: What would happen if we use a shared `HashMap`?

```rust
pub(crate) utxos: HashMap<OutPoint, WeightedUtxo>,
```
created_at: 2025-02-27T05:29:56Z
author_association: MEMBER
diff_hunk: @@ -1976,117 +2003,52 @@ impl Wallet {
         descriptor.at_derivation_index(child).ok()
     }
 
-    fn get_available_utxos(&self) -> Vec<(LocalOutput, Weight)> {
-        self.list_unspent()
-            .map(|utxo| {
-                let keychain = utxo.keychain;
-                (utxo, {
-                    self.public_descriptor(keychain)
-                        .max_weight_to_satisfy()
-                        .unwrap()
-                })
-            })
-            .collect()
-    }
-
     /// Given the options returns the list of utxos that must be used to form the
     /// transaction and any further that may be used if needed.
-    fn preselect_utxos(
-        &self,
-        params: &TxParams,
-        current_height: Option<u32>,
-    ) -> (Vec<WeightedUtxo>, Vec<WeightedUtxo>) {
-        let TxParams {
-            change_policy,
-            unspendable,
-            utxos,
-            drain_wallet,
-            manually_selected_only,
-            bumping_fee,
-            ..
-        } = params;
-
-        let manually_selected = utxos.clone();
-        // we mandate confirmed transactions if we're bumping the fee
-        let must_only_use_confirmed_tx = bumping_fee.is_some();
-        let must_use_all_available = *drain_wallet;
-
-        //    must_spend <- manually selected utxos
-        //    may_spend  <- all other available utxos
-        let mut may_spend = self.get_available_utxos();
-
-        may_spend.retain(|may_spend| {
-            !manually_selected
-                .iter()
-                .any(|manually_selected| manually_selected.utxo.outpoint() == may_spend.0.outpoint)
-        });
-        let mut must_spend = manually_selected;
-
-        // NOTE: we are intentionally ignoring `unspendable` here. i.e manual
-        // selection overrides unspendable.
-        if *manually_selected_only {
-            return (must_spend, vec![]);
-        }
-
-        let satisfies_confirmed = may_spend
-            .iter()
-            .map(|u| -> bool {
-                let txid = u.0.outpoint.txid;
-                let tx = match self.indexed_graph.graph().get_tx(txid) {
-                    Some(tx) => tx,
-                    None => return false,
-                };
-
-                // Whether the UTXO is mature and, if needed, confirmed
-                let mut spendable = true;
-                let chain_position = u.0.chain_position;
-                if must_only_use_confirmed_tx && !chain_position.is_confirmed() {
-                    return false;
-                }
-                if tx.is_coinbase() {
-                    debug_assert!(
-                        chain_position.is_confirmed(),
-                        "coinbase must always be confirmed"
-                    );
-                    if let Some(current_height) = current_height {
-                        match chain_position {
-                            ChainPosition::Confirmed { anchor, .. } => {
-                                // https://github.com/bitcoin/bitcoin/blob/c5e67be03bb06a5d7885c55db1f016fbf2333fe3/src/validation.cpp#L373-L375
-                                let spend_height = current_height + 1;
-                                let coin_age_at_spend_height =
-                                    spend_height.saturating_sub(anchor.block_id.height);
-                                spendable &= coin_age_at_spend_height >= COINBASE_MATURITY;
-                            }
-                            ChainPosition::Unconfirmed { .. } => spendable = false,
-                        }
-                    }
-                }
-                spendable
-            })
-            .collect::<Vec<_>>();
-
-        let mut i = 0;
-        may_spend.retain(|u| {
-            let retain = (self.keychains().count() == 1 || change_policy.is_satisfied_by(&u.0))
-                && !unspendable.contains(&u.0.outpoint)
-                && satisfies_confirmed[i];
-            i += 1;
-            retain
-        });
-
-        let mut may_spend = may_spend
-            .into_iter()
-            .map(|(local_utxo, satisfaction_weight)| WeightedUtxo {
-                satisfaction_weight,
-                utxo: Utxo::Local(local_utxo),
-            })
-            .collect();
-
-        if must_use_all_available {
-            must_spend.append(&mut may_spend);
+    fn filter_utxos(&self, params: &TxParams, current_height: u32) -> Vec<WeightedUtxo> {
+        if params.manually_selected_only {
+            vec![]
+        // only process optional UTxOs if manually_selected_only is false
+        } else {
+            self.indexed_graph
+                .graph()
+                // get all unspent UTxOs from wallet
+                // NOTE: the UTxOs returned by the following method already belong to wallet as the
+                // call chain uses get_tx_node infallibly
+                .filter_chain_unspents(
+                    &self.chain,
+                    self.chain.tip().block_id(),
+                    self.indexed_graph.index.outpoints().iter().cloned(),
+                )
+                // only create LocalOutput if UTxO is mature
+                .filter_map(move |((k, i), full_txo)| {
+                    full_txo
+                        .is_mature(current_height)
+                        .then(|| new_local_utxo(k, i, full_txo))
+                })
+                // only process UTxOs not selected manually, they will be considered later in the chain
+                // NOTE: this avoid UTxOs in both required and optional list
+                .filter(|may_spend| !params.utxos.contains(may_spend))
path: crates/wallet/src/wallet/mod.rs
user: login: evanlinjin
body: Why don't we filter out foreign utxos as well? It seems to make sense to put `utxos` and `foreign_utxos` in the same map?
created_at: 2025-02-27T05:48:47Z
author_association: MEMBER
diff_hunk: @@ -274,26 +274,28 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
-        {
-            let wallet = &mut self.wallet;
-            let utxos = outpoints
-                .iter()
-                .map(|outpoint| {
-                    wallet
-                        .get_utxo(*outpoint)
-                        .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
-                })
-                .collect::<Result<Vec<_>, _>>()?;
-
-            for utxo in utxos {
-                let descriptor = wallet.public_descriptor(utxo.keychain);
-                let satisfaction_weight = descriptor.max_weight_to_satisfy().unwrap();
-                self.params.utxos.push(WeightedUtxo {
-                    satisfaction_weight,
-                    utxo: Utxo::Local(utxo),
-                });
-            }
-        }
+        outpoints
+            .iter()
+            .map(|outpoint| {
+                self.wallet
+                    .get_utxo(*outpoint)
+                    .ok_or(AddUtxoError::UnknownUtxo(*outpoint))
+                    .map(|output| {
+                        (
+                            *outpoint,
+                            WeightedUtxo {
+                                satisfaction_weight: self
+                                    .wallet
+                                    .public_descriptor(output.keychain)
+                                    .max_weight_to_satisfy()
+                                    .unwrap(),
+                                utxo: Utxo::Local(output),
+                            },
+                        )
+                    })
+            })
+            .collect::<Result<HashMap<OutPoint, WeightedUtxo>, AddUtxoError>>()
path: crates/wallet/src/wallet/tx_builder.rs
user: login: evanlinjin
body: Nit: Can we collect this into a variable first? I think it'll make it easier to read.
created_at: 2025-02-28T04:29:29Z
author_association: MEMBER
diff_hunk: @@ -2605,3 +2559,77 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let mut params = TxParams::default();
+        wallet
+            .get_utxo(OutPoint { txid, vout: 0 })
+            .map(|output| {
+                params.utxos.insert(
+                    output.outpoint,
+                    WeightedUtxo {
+                        satisfaction_weight: wallet
+                            .public_descriptor(output.keychain)
+                            .max_weight_to_satisfy()
+                            .unwrap(),
+                        utxo: Utxo::Local(output),
+                    },
+                )
+            })
+            .unwrap();
path: crates/wallet/src/wallet/mod.rs
user: login: evanlinjin
body: What is this for?
created_at: 2025-02-28T04:45:18Z
author_association: MEMBER
diff_hunk: @@ -2605,3 +2559,77 @@ macro_rules! doctest_wallet {
         wallet
     }}
 }
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::test_utils::get_test_tr_single_sig_xprv_and_change_desc;
+    use crate::test_utils::insert_tx;
+
+    #[test]
+    fn not_duplicated_utxos_across_optional_and_required() {
+        let (external_desc, internal_desc) = get_test_tr_single_sig_xprv_and_change_desc();
+
+        // create new wallet
+        let mut wallet = Wallet::create(external_desc, internal_desc)
+            .network(Network::Testnet)
+            .create_wallet_no_persist()
+            .unwrap();
+
+        let two_output_tx = Transaction {
+            input: vec![],
+            output: vec![
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(25_000),
+                },
+                TxOut {
+                    script_pubkey: wallet
+                        .next_unused_address(KeychainKind::External)
+                        .script_pubkey(),
+                    value: Amount::from_sat(75_000),
+                },
+            ],
+            version: transaction::Version::non_standard(0),
+            lock_time: absolute::LockTime::ZERO,
+        };
+
+        let txid = two_output_tx.compute_txid();
+        insert_tx(&mut wallet, two_output_tx);
+
+        let mut params = TxParams::default();
+        wallet
+            .get_utxo(OutPoint { txid, vout: 0 })
+            .map(|output| {
+                params.utxos.insert(
+                    output.outpoint,
+                    WeightedUtxo {
+                        satisfaction_weight: wallet
+                            .public_descriptor(output.keychain)
+                            .max_weight_to_satisfy()
+                            .unwrap(),
+                        utxo: Utxo::Local(output),
+                    },
+                )
+            })
+            .unwrap();
path: crates/wallet/src/wallet/mod.rs
user: login: evanlinjin
body: `params.utxos.insert` should be on the outside otherwise it makes it hard to read.
created_at: 2025-02-28T04:48:13Z
author_association: MEMBER
diff_hunk: @@ -274,26 +274,28 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
path: crates/wallet/src/wallet/tx_builder.rs
user: login: evanlinjin
body: Ideally, we would error here if an `OutPoint` conflicts with a foreign UTXO. However, we will need to add new error variants (which is a MAJOR change since the error type does not have `non-exhaustive`).

For now, how about we make a note in the docs that if you add the same UTXO as both foreign and local, the latest change has precedence (please word that better).

Same goes for the add-foreign-utxo methods.
created_at: 2025-02-28T04:57:45Z
author_association: MEMBER
diff_hunk: @@ -274,26 +274,28 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
path: crates/wallet/src/wallet/tx_builder.rs
user: login: evanlinjin
body: For the add-foreign-utxo methods, ideally we will error out if it conflicts with pre-existing in `TxParams::utxos` or utxos in `TxBuilder::wallet`. However, the error variant is not `non-exhaustive` so that will be breaking.

For now, how about we just note that it would be unexpected behavior in the docs?
created_at: 2025-02-28T04:59:26Z
author_association: MEMBER
diff_hunk: @@ -274,26 +274,28 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
path: crates/wallet/src/wallet/tx_builder.rs
user: login: evanlinjin
body: @nymius had a better idea of making local UTXOs have precedence over foreign UTXOs. Let's do this and also update the documentation.
created_at: 2025-02-28T05:03:48Z
author_association: MEMBER
diff_hunk: @@ -274,26 +274,28 @@ impl<'a, Cs> TxBuilder<'a, Cs> {
     /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
     /// the "utxos" and the "unspendable" list, it will be spent.
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, AddUtxoError> {
path: crates/wallet/src/wallet/tx_builder.rs
user: login: nymius
body: I've added the documentation requested and two tests to address the discussed cases. #1823 will fix the issue against tracked local UTxOs (the one the wallet already has knowledge about, aka `optional`).
created_at: 2025-03-03T06:29:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1618,60 +1626,71 @@ impl Wallet {
             .map_err(|_| BuildFeeBumpError::FeeRateUnavailable)?;
 
         // remove the inputs from the tx and process them
-        let original_txin = tx.input.drain(..).collect::<Vec<_>>();
-        let original_utxos = original_txin
-            .iter()
+        let utxos = tx
+            .input
+            .drain(..)
             .map(|txin| -> Result<_, BuildFeeBumpError> {
-                let prev_tx = graph
+                graph
+                    // Get previous transaction
                     .get_tx(txin.previous_output.txid)
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-                let txout = &prev_tx.output[txin.previous_output.vout as usize];
-
-                let chain_position = chain_positions
-                    .get(&txin.previous_output.txid)
-                    .cloned()
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-
-                let weighted_utxo = match txout_index.index_of_spk(txout.script_pubkey.clone()) {
-                    Some(&(keychain, derivation_index)) => {
-                        let satisfaction_weight = self
-                            .public_descriptor(keychain)
-                            .max_weight_to_satisfy()
-                            .unwrap();
-                        WeightedUtxo {
-                            utxo: Utxo::Local(LocalOutput {
-                                outpoint: txin.previous_output,
-                                txout: txout.clone(),
-                                keychain,
-                                is_spent: true,
-                                derivation_index,
-                                chain_position,
-                            }),
-                            satisfaction_weight,
-                        }
-                    }
-                    None => {
-                        let satisfaction_weight = Weight::from_wu_usize(
-                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
-                        );
-                        WeightedUtxo {
-                            utxo: Utxo::Foreign {
-                                outpoint: txin.previous_output,
-                                sequence: txin.sequence,
-                                psbt_input: Box::new(psbt::Input {
-                                    witness_utxo: Some(txout.clone()),
-                                    non_witness_utxo: Some(prev_tx.as_ref().clone()),
-                                    ..Default::default()
-                                }),
-                            },
-                            satisfaction_weight,
+                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))
+                    // Get chain position
+                    .and_then(|prev_tx| {
+                        chain_positions
+                            .get(&txin.previous_output.txid)
+                            .cloned()
+                            .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))
+                            .map(|chain_position| (prev_tx, chain_position))
+                    })
+                    .map(|(prev_tx, chain_position)| {
+                        let txout = prev_tx.output[txin.previous_output.vout as usize].clone();
+                        match txout_index.index_of_spk(txout.script_pubkey.clone()) {
+                            Some(&(keychain, derivation_index)) => (
+                                txin.previous_output,
+                                WeightedUtxo {
+                                    satisfaction_weight: self
+                                        .public_descriptor(keychain)
+                                        .max_weight_to_satisfy()
+                                        .unwrap(),
+                                    utxo: Utxo::Local(LocalOutput {
+                                        outpoint: txin.previous_output,
+                                        txout: txout.clone(),
+                                        keychain,
+                                        is_spent: true,
+                                        derivation_index,
+                                        chain_position,
+                                    }),
+                                },
+                            ),
+                            None => {
+                                let satisfaction_weight = Weight::from_wu_usize(
+                                    serialize(&txin.script_sig).len() * 4
+                                        + serialize(&txin.witness).len(),
+                                );
+
+                                (
+                                    txin.previous_output,
+                                    WeightedUtxo {
+                                        utxo: Utxo::Foreign {
+                                            outpoint: txin.previous_output,
+                                            sequence: txin.sequence,
+                                            psbt_input: Box::new(psbt::Input {
+                                                witness_utxo: txout
+                                                    .script_pubkey
+                                                    .witness_version()
+                                                    .map(|_| txout.clone()),
+                                                non_witness_utxo: Some(prev_tx.as_ref().clone()),
+                                                ..Default::default()
+                                            }),
+                                        },
+                                        satisfaction_weight,
+                                    },
+                                )
+                            }
                         }
path: crates/wallet/src/wallet/mod.rs
user: login: evanlinjin
body: **Nit:** trying to make this section fully functional has made it harder to read. There is weird `map` statements, and it becomes hard to see whether we are `map`-ing on an iterator or option.

I would prefer the following:

```rust
                let prev_tx = graph
                    .get_tx(txin.previous_output.txid)
                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
                let chain_position = chain_positions
                    .get(&txin.previous_output.txid)
                    .cloned()
                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
                let txout = prev_tx.output[txin.previous_output.vout as usize].clone();
                let (outpoint, weighted_utxo) = match txout_index
                    .index_of_spk(txout.script_pubkey.clone())
                {
                    Some(&(keychain, derivation_index)) => (
                        txin.previous_output,
                        WeightedUtxo {
                            satisfaction_weight: self
                                .public_descriptor(keychain)
                                .max_weight_to_satisfy()
                                .unwrap(),
                            utxo: Utxo::Local(LocalOutput {
                                outpoint: txin.previous_output,
                                txout: txout.clone(),
                                keychain,
                                is_spent: true,
                                derivation_index,
                                chain_position,
                            }),
                        },
                    ),
                    None => {
                        let satisfaction_weight = Weight::from_wu_usize(
                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
                        );
                        (
                            txin.previous_output,
                            WeightedUtxo {
                                utxo: Utxo::Foreign {
                                    outpoint: txin.previous_output,
                                    sequence: txin.sequence,
                                    psbt_input: Box::new(psbt::Input {
                                        witness_utxo: txout
                                            .script_pubkey
                                            .witness_version()
                                            .map(|_| txout.clone()),
                                        non_witness_utxo: Some(prev_tx.as_ref().clone()),
                                        ..Default::default()
                                    }),
                                },
                                satisfaction_weight,
                            },
                        )
                    }
                };
                Ok((outpoint, weighted_utxo))
```
created_at: 2025-03-06T00:01:43Z
author_association: MEMBER
diff_hunk: @@ -1618,60 +1626,71 @@ impl Wallet {
             .map_err(|_| BuildFeeBumpError::FeeRateUnavailable)?;
 
         // remove the inputs from the tx and process them
-        let original_txin = tx.input.drain(..).collect::<Vec<_>>();
-        let original_utxos = original_txin
-            .iter()
+        let utxos = tx
+            .input
+            .drain(..)
             .map(|txin| -> Result<_, BuildFeeBumpError> {
-                let prev_tx = graph
+                graph
+                    // Get previous transaction
                     .get_tx(txin.previous_output.txid)
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-                let txout = &prev_tx.output[txin.previous_output.vout as usize];
-
-                let chain_position = chain_positions
-                    .get(&txin.previous_output.txid)
-                    .cloned()
-                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))?;
-
-                let weighted_utxo = match txout_index.index_of_spk(txout.script_pubkey.clone()) {
-                    Some(&(keychain, derivation_index)) => {
-                        let satisfaction_weight = self
-                            .public_descriptor(keychain)
-                            .max_weight_to_satisfy()
-                            .unwrap();
-                        WeightedUtxo {
-                            utxo: Utxo::Local(LocalOutput {
-                                outpoint: txin.previous_output,
-                                txout: txout.clone(),
-                                keychain,
-                                is_spent: true,
-                                derivation_index,
-                                chain_position,
-                            }),
-                            satisfaction_weight,
-                        }
-                    }
-                    None => {
-                        let satisfaction_weight = Weight::from_wu_usize(
-                            serialize(&txin.script_sig).len() * 4 + serialize(&txin.witness).len(),
-                        );
-                        WeightedUtxo {
-                            utxo: Utxo::Foreign {
-                                outpoint: txin.previous_output,
-                                sequence: txin.sequence,
-                                psbt_input: Box::new(psbt::Input {
-                                    witness_utxo: Some(txout.clone()),
-                                    non_witness_utxo: Some(prev_tx.as_ref().clone()),
-                                    ..Default::default()
-                                }),
-                            },
-                            satisfaction_weight,
+                    .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))
+                    // Get chain position
+                    .and_then(|prev_tx| {
+                        chain_positions
+                            .get(&txin.previous_output.txid)
+                            .cloned()
+                            .ok_or(BuildFeeBumpError::UnknownUtxo(txin.previous_output))
+                            .map(|chain_position| (prev_tx, chain_position))
+                    })
+                    .map(|(prev_tx, chain_position)| {
+                        let txout = prev_tx.output[txin.previous_output.vout as usize].clone();
+                        match txout_index.index_of_spk(txout.script_pubkey.clone()) {
+                            Some(&(keychain, derivation_index)) => (
+                                txin.previous_output,
+                                WeightedUtxo {
+                                    satisfaction_weight: self
+                                        .public_descriptor(keychain)
+                                        .max_weight_to_satisfy()
+                                        .unwrap(),
+                                    utxo: Utxo::Local(LocalOutput {
+                                        outpoint: txin.previous_output,
+                                        txout: txout.clone(),
+                                        keychain,
+                                        is_spent: true,
+                                        derivation_index,
+                                        chain_position,
+                                    }),
+                                },
+                            ),
+                            None => {
+                                let satisfaction_weight = Weight::from_wu_usize(
+                                    serialize(&txin.script_sig).len() * 4
+                                        + serialize(&txin.witness).len(),
+                                );
+
+                                (
+                                    txin.previous_output,
+                                    WeightedUtxo {
+                                        utxo: Utxo::Foreign {
+                                            outpoint: txin.previous_output,
+                                            sequence: txin.sequence,
+                                            psbt_input: Box::new(psbt::Input {
+                                                witness_utxo: txout
+                                                    .script_pubkey
+                                                    .witness_version()
+                                                    .map(|_| txout.clone()),
+                                                non_witness_utxo: Some(prev_tx.as_ref().clone()),
+                                                ..Default::default()
+                                            }),
+                                        },
+                                        satisfaction_weight,
+                                    },
+                                )
+                            }
                         }
path: crates/wallet/src/wallet/mod.rs
user: login: nymius
body: I agree, will try to control the bias towards functional style.
created_at: 2025-03-06T00:13:24Z
author_association: CONTRIBUTOR

PR comments:
user: login: evanlinjin
created_at: 2025-01-14T06:24:17Z
body: Hey @nymius, does this actually fix something or is this purely a refactor?

Note that we want to redo the tx building / creating logic to use `bdk_coin_select` and miniscript planning module so I think it's good policy to only do changes that are fixes or implement features that users are requesting.
author_association: MEMBER
user: login: nymius
created_at: 2025-01-14T21:07:26Z
body: > Hey @nymius, does this actually fix something or is this purely a refactor?

Hi @evanlinjin. Yes, maybe I went to far with the changes, still, it fixes the duplicate issue I open in bitcoindevkit/bdk#1794 by using a `HashSet` to handle manually selected UTxOs.

Then, about the refactor:

The research of the code push me to isolate the pre selection steps as items on a checklist and that's why It ended up as a refactor.

My idea is to further isolate each filter in their own iterator adaptor (a separated function for each one) that consumes `impl Iterator<Item = LocalOutputs>` and returns the same type, to later also allow the introduction of custom filters in `TxParams` which I think can be helpful to implement checks on the ability to spend a Taproot output, for example, in Silent Payments.

The heavy use of iterators came for trying to avoid the allocation of helpers, like `satisfies_confirmed`.

I envisioned something like this:
```rust
let optional_utxos = self
    .list_unspent()
    .check_are_not_already_manually_selected()
    .check_are_not_unspendable()
    .check_confirmed_only_if_RBF()
    .check_is_local_utxo()
    .check_is_mature_if_coinbase();
// then
let (required, optional) = optional_utxos.chain(required_utxos.iter().clone())
    .get_weighted_utxos()
    .chain(foreign_utxos.iter().clone())
    .apply_custom_validation_for_all_tx_inputs()
    .split_utxos_in_required_and_optional()
```

---

Discussing this today with @ValuedMammal, I decided to do the following changes:
1. [x] Just return optional utxos as they are the only needed values.
2. [x] Move the calculus of `WeightedUtxos` to another method. **Placed inside create_tx for now**
3. [x] Rename `preselect_utxos` to `filter_utxos` to correct the semantics the above point will change.
4. [x] Implement unit test for the feature to make the PR sound.
5. [x] Make `current_height` parameter not optional.

If we don't agree on the above, I propose the following alternatives:
- keep points 1 and 2 above.
- keep the old filtering implementation in place.
- keep the changes to the `build_fee_bump` (new `foreign_utxos` field in `TxParams`).
- drop all the other changes.



author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-01-28T16:56:37Z
body: Rebased
author_association: CONTRIBUTOR
user: login: ValuedMammal
created_at: 2025-02-04T00:14:33Z
body: 📌  I don't think we've considered what would happen if a third party provides us with a "foreign" utxo that actually duplicates a utxo owned by the wallet, potentially causing us to sign something we didn't agree to. It might need more research so I'll open an issue. bitcoindevkit/bdk_wallet#29 
author_association: COLLABORATOR
user: login: nymius
created_at: 2025-02-05T20:27:47Z
body: > 📌 I don't think we've considered what would happen if a third party provides us with a "foreign" utxo that actually duplicates a utxo owned by the wallet, potentially causing us to sign something we didn't agree to. It might need more research so I'll open an issue. bitcoindevkit/bdk_wallet#29

I created a separated draft PR bitcoindevkit/bdk#1823 to address the issue.
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-02-07T16:27:44Z
body: **review club notes 02/06/2025**

1. Include `psbt::Input::non_witness_utxo` field together with `psbt::Input::non_witness_utxo` when `txout.script_pubkey.witness_version().is_some()` while re constructing foreign UTxOs in `Wallet::build_fee_bump`.
3. The relevance of the integration of miniscript plan (#1786 ) for this PR was mentioned. `Wallet::get_available_utxos` obtained the satisfaction weight of UTxOs before passing them to `preselect_utxos`. As satisfaction weight is dependent of planning, there should be future work to integrate this, as it is a previous step to `Wallet::filter_utxos`.
4. The split of the PR was suggested, in two different ones to allow a faster review and merge of the fix for bitcoindevkit/bdk#1810.
5. As commits addressing suggested changes had been acknowledge by reviewers, the next step should be to squash them in four or less commits:
	1. Refactor
	2. `coin_selection::filter_duplicates` removal.
	3. Tests
	4. Miscellaneous not related to refactor
3. As part of the review we talked about bitcoindevkit/bdk_wallet#29  and its fix candidate bitcoindevkit/bdk#1823 :
	1. As bitcoindevkit/bdk#1823 introduces breaking changes, a compromise solution would be to add a silent check which works as a no-op in case transaction is found to be owned by the wallet. Also, one possible approach is to delay the introduction of the new `NotForeignUtxo` variant error (breaking change) until the release of the new major version, `2.0.0` with a `TODO` comment.
	2. It wasn't clear the ownership of the transactions stored in `TxGraph`. The terms `canonical` and `relevant` were mentioned as a good source to get an idea of the topic from the documentation.
	`Wallet::transactions` has a good explanation of this.
	`TxGraph`'s transactions don't have to belong to the wallet necessarily, and `get_txout` shouldn't be considered a good source of truth to establish if an `Outpoint` can be unlocked by `Wallet` or not. `Wallet::is_mine` is the correct method for this.

Thanks to @ValuedMammal, @oleonardolima, @LagginTimes and @evanlinjin for participating!
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-02-07T16:56:56Z
body: **Next steps based on review club**:
- [x] Include `psbt::Input::non_witness_utxo` field together with `psbt::Input::non_witness_utxo` when `txout.script_pubkey.witness_version().is_some()` while re constructing foreign UTxOs in `Wallet::build_fee_bump`.
- [x] Squash commits.
- [x] Create separated PR to fix bitcoindevkit/bdk#1810 
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-02-10T14:21:28Z
body: 905c8dd will be squashed to b1bff02fc3875bbb4342c3da694e26e9036bbb06 once bitcoindevkit/bdk#1830 is merged
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-02-10T18:52:41Z
body: Now that foreign UTxOs are separated from manually selected UTxOs there are new cases for duplicity:
1. Duplicated UTxOs inside `foreign_utxos` field.
2. Duplicated UTxOs across `foreign_utxos` and required/optional UTxOs. Shouldn't be a problem once bitcoindevkit/bdk#1823 is merged. Because of the new check which avoids the addition of local UTxOs for `Wallet` as foreign UTxOs.

**TODO**:
- [x] Ensure `foreign_utxos` does not contain duplicated UTxOs: `TxParams::foreign_utxos` transformed into `HashSet` in  7cb0243 and added test to confirm property. Will be squashed to b1bff02 once change is ACKnowledge. The avoidance of duplicity across foreign and optional/required UTxOs will be guaranteed by changes in bitcoindevkit/bdk#1823 
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-02-14T21:15:35Z
body: > [905c8dd](https://github.com/bitcoindevkit/bdk/commit/905c8dd663ee845911f52510fe40e58b22fed252) will be squashed to [b1bff02](https://github.com/bitcoindevkit/bdk/commit/b1bff02fc3875bbb4342c3da694e26e9036bbb06) once bitcoindevkit/bdk#1830 is merged

Squash done and rebased on master.

author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-02-27T01:16:37Z
body: Thanks @ValuedMammal! Updated and rebased
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-02-28T03:40:28Z
body: Addressed comments and rebased onto master.
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-03-03T06:30:20Z
body: Rebased onto b26ff891
author_association: CONTRIBUTOR
user: login: nymius
created_at: 2025-03-05T23:49:07Z
body: Rebased onto 362c3dc8
author_association: CONTRIBUTOR